{"id":59,"date":"2019-06-30T16:49:01","date_gmt":"2019-06-30T14:49:01","guid":{"rendered":"http:\/\/sys-code-alpha.com\/?p=59"},"modified":"2019-06-30T16:49:01","modified_gmt":"2019-06-30T14:49:01","slug":"powershell-and-writing-files-demystified","status":"publish","type":"post","link":"https:\/\/sys-code-alpha.com\/?p=59","title":{"rendered":"PowerShell and writing files demystified"},"content":{"rendered":"\n

Few days ago I was looking for a ways to speed up some of my tools. I was not stasifed with the content I had found on the web so I decieded to run some tests on my own. In the effect I created a small tool to validate different file write approach with different commandlets. <\/p>\n\n\n\n

I took a small 24 MB text file with 2 million lines for my test:<\/p>\n\n\n\n

PS C:\\TMP> Get-ChildItem .\\testNew\n\n    Directory: C:\\TMP\n\nMode                LastWriteTime         Length Name\n----                -------------         ------ ----\n-a----       16.05.2019     15:59       25299718 testNew\n\nPS C:\\TMP> $x = Get-Content .\\testNew  -ReadCount 0\nPS C:\\TMP> $x.count\n2094232<\/code><\/pre>\n\n\n\n
I grabbed the file content to a $AllText variable and started testing.<\/p>\n\n\n\n
According to my web search findings the fastest approach to write a file in PowerShell was to use .NET StreamWriter, so I applied it as follows:<\/p>\n\n\n\n
Filter Write-Line{\n    BEGIN{ $StreamWriter = New-Object System.IO.StreamWriter $Filename.newTestfile1.txt\" }\n    PROCESS{ $StreamWriter.WriteLine($_) }\n    END{$StreamWriter.Close()}\n}<\/code><\/pre>\n\n\n\n
I added a few more common and ‘PowerShellish’ tests like out-file, set-content and add-content.<\/p>\n\n\n\n
Add-Content -Value $AllText -Path $Filename.newTestfile6.txt
$AllText | Add-Content -Path $Filename.newTestfile7.txt
Set-Content -value $AllText -Path $Filename.newTestfile2.txt
Out-File $Filename.newTestfile4.txt -inputobject $AllText -Encoding utf8
$AllText | Set-Content -Path $Filename.newTestfile3.txt
$AllText | Out-File $Filename.newTestfile5.txt -Encoding utf8<\/p>\n\n\n\n
I decided to run tests 5 times to have the average measurement as well.<\/p>\n\n\n\n
The results present execution time in seconds and they are really interesting:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
In general many of us know that Out-File is quite slow. The same applies to Set-Content. What changes the game entirely is a way you deal with the Set-Content in your code. If instead of using a pipeline you use it as folows:<\/p>\n\n\n\n
Set-Content -value $AllText -Path $Filename.newTestfile2.txt<\/p>\n\n\n\n
You will get better results than System.IO.StreamWriter. <\/p>\n\n\n\n
What is more suprising is that you can achieve slightly better results if you use Add-Content. This one works nicely regardless of pipeline usage.<\/p>\n\n\n\n
To sum up, it appears that for the case of fast file writing we do not need to use .NET directly in PowerShell. The proper use of get-help and get-command should be enough.<\/p>\n\n\n\n
In case someone wants to check on his\/her own, I share the code of my function below:<\/p>\n\n\n\n
Function Test-FileWriteSpeed{\n\n    [CmdletBinding()]\n    Param(\n          [string]$Filename,\n          [int]$AmountOfTests,\n          [switch]$RemoveTestFilesAfterCalculation\n          )\n    $Tests = 1..$AmountOfTests\n    $result = Foreach($test in $Tests){\n            \n            Write-Host -ForegroundColor yellow \"Running test $test      $( (get-date).ToShortTimeString() )\"\n            $Filename = (get-item $Filename).FullName    \n            $alltext = Get-Content -Path $Filename -readcount 0\n            \n            Filter Write-Line{\n                BEGIN{ $StreamWriter = New-Object System.IO.StreamWriter \"$Filename.newTestfile1.txt\" }\n                PROCESS{ $StreamWriter.WriteLine($_) }\n                END{$StreamWriter.Close()}\n                }\n            \n            Get-ChildItem *.newTestfile*.txt | Remove-Item\n                \n            $Ex = Measure-command { $AllText | Write-Line } | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"`$AllText | Write-Line\" ; ExecutionTimeSeconds = $ex}    \n            \n            $Ex = Measure-command { Set-Content -value $AllText -Path \"$Filename.newTestfile2.txt\" } | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"Set-Content -value `$AllText -Path `$Filename.newTestfile2.txt\" ; ExecutionTimeSeconds = $ex}    \n            \n            $Ex = Measure-command { $AllText | Set-Content -Path \"$Filename.newTestfile3.txt\" }  | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"`$AllText | Set-Content -Path `$Filename.newTestfile3.txt\" ; ExecutionTimeSeconds = $ex}\n            \n            $Ex = Measure-command { Out-File \"$Filename.newTestfile4.txt\" -inputobject $AllText -Encoding utf8 } | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"Out-File `$Filename.newTestfile4.txt -inputobject `$AllText -Encoding utf8\" ; ExecutionTimeSeconds = $ex}                    \n            \n            $Ex = Measure-command { $AllText | Out-File \"$Filename.newTestfile5.txt\" -Encoding utf8} | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"`$AllText | Out-File `$Filename.newTestfile5.txt -Encoding utf8\" ; ExecutionTimeSeconds = $ex}\n            \n            $Ex = Measure-command { Add-Content -Value $AllText -Path \"$Filename.newTestfile6.txt\"} | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"Add-Content -Value `$AllText -Path `$Filename.newTestfile6.txt\" ; ExecutionTimeSeconds = $ex}\n            \n            $Ex = Measure-command { Add-Content -Value $AllText -Path \"$Filename.newTestfile7.txt\"} | select-object -expandproperty TotalSeconds\n            [PSCustomObject]@{Code = \"`$AllText | Add-Content -Path `$Filename.newTestfile7.txt\" ; ExecutionTimeSeconds = $ex}            \n                \n            }\n    $result | Group-Object code | %{ $name = $_.name; $_.group | Measure-Object -Property ExecutionTimeSeconds -Sum -Average | select @{n='Code';e={$name}}, Average, Sum, @{n='Samples';e={$_.Count}}, Property  } | Sort-object -Property Sum\n    IF($RemoveTestFilesAfterCalculation){Get-ChildItem *.newTestfile*.txt | Remove-Item}\n}<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
Few days ago I was looking for a ways to speed up some of my tools. I was not stasifed with the content I had found on the web so I decieded to run some tests on my own. In the effect I created a small tool to validate different file write approach with different […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=\/wp\/v2\/posts\/59"}],"collection":[{"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=59"}],"version-history":[{"count":13,"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=\/wp\/v2\/posts\/59\/revisions"}],"predecessor-version":[{"id":74,"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=\/wp\/v2\/posts\/59\/revisions\/74"}],"wp:attachment":[{"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=59"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=59"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sys-code-alpha.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=59"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}