Few days ago I was looking for a ways to speed up some of my tools. I was not stasifed with the content I had found on the web so I decieded to run some tests on my own. In the effect I created a small tool to validate different file write approach with different commandlets.

I took a small 24 MB text file with 2 million lines for my test:

PS C:\TMP> Get-ChildItem .\testNew

    Directory: C:\TMP

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       16.05.2019     15:59       25299718 testNew

PS C:\TMP> $x = Get-Content .\testNew  -ReadCount 0
PS C:\TMP> $x.count
2094232

I grabbed the file content to a $AllText variable and started testing.

According to my web search findings the fastest approach to write a file in PowerShell was to use .NET StreamWriter, so I applied it as follows:

Filter Write-Line{
    BEGIN{ $StreamWriter = New-Object System.IO.StreamWriter $Filename.newTestfile1.txt" }
    PROCESS{ $StreamWriter.WriteLine($_) }
    END{$StreamWriter.Close()}
}

I added a few more common and ‘PowerShellish’ tests like out-file, set-content and add-content.

Add-Content -Value $AllText -Path $Filename.newTestfile6.txt
$AllText | Add-Content -Path $Filename.newTestfile7.txt
Set-Content -value $AllText -Path $Filename.newTestfile2.txt
Out-File $Filename.newTestfile4.txt -inputobject $AllText -Encoding utf8
$AllText | Set-Content -Path $Filename.newTestfile3.txt
$AllText | Out-File $Filename.newTestfile5.txt -Encoding utf8

I decided to run tests 5 times to have the average measurement as well.

The results present execution time in seconds and they are really interesting:

In general many of us know that Out-File is quite slow. The same applies to Set-Content. What changes the game entirely is a way you deal with the Set-Content in your code. If instead of using a pipeline you use it as folows:

Set-Content -value $AllText -Path $Filename.newTestfile2.txt

You will get better results than System.IO.StreamWriter.

What is more suprising is that you can achieve slightly better results if you use Add-Content. This one works nicely regardless of pipeline usage.

To sum up, it appears that for the case of fast file writing we do not need to use .NET directly in PowerShell. The proper use of get-help and get-command should be enough.

In case someone wants to check on his/her own, I share the code of my function below:

Function Test-FileWriteSpeed{

    [CmdletBinding()]
    Param(
          [string]$Filename,
          [int]$AmountOfTests,
          [switch]$RemoveTestFilesAfterCalculation
          )
    $Tests = 1..$AmountOfTests
    $result = Foreach($test in $Tests){
            
            Write-Host -ForegroundColor yellow "Running test $test      $( (get-date).ToShortTimeString() )"
            $Filename = (get-item $Filename).FullName    
            $alltext = Get-Content -Path $Filename -readcount 0
            
            Filter Write-Line{
                BEGIN{ $StreamWriter = New-Object System.IO.StreamWriter "$Filename.newTestfile1.txt" }
                PROCESS{ $StreamWriter.WriteLine($_) }
                END{$StreamWriter.Close()}
                }
            
            Get-ChildItem *.newTestfile*.txt | Remove-Item
                
            $Ex = Measure-command { $AllText | Write-Line } | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "`$AllText | Write-Line" ; ExecutionTimeSeconds = $ex}    
            
            $Ex = Measure-command { Set-Content -value $AllText -Path "$Filename.newTestfile2.txt" } | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "Set-Content -value `$AllText -Path `$Filename.newTestfile2.txt" ; ExecutionTimeSeconds = $ex}    
            
            $Ex = Measure-command { $AllText | Set-Content -Path "$Filename.newTestfile3.txt" }  | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "`$AllText | Set-Content -Path `$Filename.newTestfile3.txt" ; ExecutionTimeSeconds = $ex}
            
            $Ex = Measure-command { Out-File "$Filename.newTestfile4.txt" -inputobject $AllText -Encoding utf8 } | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "Out-File `$Filename.newTestfile4.txt -inputobject `$AllText -Encoding utf8" ; ExecutionTimeSeconds = $ex}                    
            
            $Ex = Measure-command { $AllText | Out-File "$Filename.newTestfile5.txt" -Encoding utf8} | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "`$AllText | Out-File `$Filename.newTestfile5.txt -Encoding utf8" ; ExecutionTimeSeconds = $ex}
            
            $Ex = Measure-command { Add-Content -Value $AllText -Path "$Filename.newTestfile6.txt"} | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "Add-Content -Value `$AllText -Path `$Filename.newTestfile6.txt" ; ExecutionTimeSeconds = $ex}
            
            $Ex = Measure-command { Add-Content -Value $AllText -Path "$Filename.newTestfile7.txt"} | select-object -expandproperty TotalSeconds
            [PSCustomObject]@{Code = "`$AllText | Add-Content -Path `$Filename.newTestfile7.txt" ; ExecutionTimeSeconds = $ex}            
                
            }
    $result | Group-Object code | %{ $name = $_.name; $_.group | Measure-Object -Property ExecutionTimeSeconds -Sum -Average | select @{n='Code';e={$name}}, Average, Sum, @{n='Samples';e={$_.Count}}, Property  } | Sort-object -Property Sum
    IF($RemoveTestFilesAfterCalculation){Get-ChildItem *.newTestfile*.txt | Remove-Item}
}