Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support streaming hashes #747

Open
ruckc opened this issue May 25, 2016 · 14 comments · May be fixed by #2714
Open

Support streaming hashes #747

ruckc opened this issue May 25, 2016 · 14 comments · May be fixed by #2714

Comments

@ruckc
Copy link

ruckc commented May 25, 2016

What You Are Seeing?

Installing large packages fail when file size is larger than 2GB.

When looking at CryptoHashProvider, it appears it reads the entire file into memory and then compute the hash, and then throw the byte[] away.

It would make sense to change the IHashAlgorithm/HashAlgorithm to handle IO.Stream.

var hash = _hashAlgorithm.ComputeHash(_fileSystem.read_file_bytes(filePath));

What is Expected?

The ability to install large packages.

How Did You Get This To Happen? (Steps to Reproduce)

Build a large nupkg and install via chocolatey.

Output Log

Error computing hash for 'C:\ProgramData\chocolatey\lib\BigPackage\BigPackage.nupkg'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

Additional Information

Zendesk Ticket 1
Zendesk Ticket 2

@ferventcoder
Copy link
Member

Please submit the gist of the chocolatey log output. Then we can look from there.

@ferventcoder
Copy link
Member

Trying to determine where the actual issue occurs - more explanation: We only warn if we can't compute a hash. If you are seeing an error that is something else entirely.

@ruckc
Copy link
Author

ruckc commented May 25, 2016

Roger... looking further in the long and we found the reason the install failed. It would be nice if streaming hashes were supported though...

@ferventcoder
Copy link
Member

Was it a reason we could correct or is it something else?

@ferventcoder ferventcoder changed the title Error with packages larger than 2GB. Support streaming hashes May 25, 2016
@MichaelGerhart
Copy link

@ferventcoder was there ever any progress made on this? I am also having trouble deploying a package that is larger than 2GB with the same hash error, see the attached log file.
log.txt

@ferventcoder
Copy link
Member

Error or Warning?

@MichaelGerhart
Copy link

Its listed as a warning, but causes the package installation to fail.

@ferventcoder
Copy link
Member

Looks like warning

 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
2017-07-13 14:18:27,536 5380 [WARN ] - Error computing hash for 'C:\ProgramData\chocolatey\lib\MSSQL2014STD\tools\Sources\SQLServer2014x64Standard\STDISO.ISO'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.sh

And then I see the package fails. I see the package fails afterwards.

Can you open up the chocolatey.log instead of the chocolatey.summary.log and include that output instead? We'd need all of that to determine whether that is really causing the issue or not. Thanks!

@ferventcoder
Copy link
Member

Debug output is going to be much more helpful.

@KurtTheBerner
Copy link

I have exactly the same problem with a package larger than 2GB.
Gist Link with Debug Output:
https://gist.github.com/KurtTheBerner/6efdf08cbcac24a1f4ee2d28323383c0

Console Output:

Chocolatey v0.10.7
Installing the following packages:
MyBigPackage
By installing you accept licenses for the packages.

mybigpackage v1.2.3
mybigpackage package files install completed. Performing other installation steps.
Error computing hash for 'C:\ProgramData\chocolatey\lib\mybigpackage\mybigpackage.nupkg
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.
 The install of mybigpackage was successful.
  Software install location not explicitly set, could be in package or
  default install location if installer.
Chocolatey installed 1/1 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Although chocolatey says that The install of mybigpackage was successful, the package was not installed.
For us this is an important point for the use of Chocolatey, we would be glad if the problem were solved soon.

@KurtTheBerner
Copy link

@ferventcoder any news about this issue?

@ferventcoder
Copy link
Member

@KurtTheBerner I just reviewed your log, and the installation was successful for the package, but maybe not the software you were attempting to install. Perhaps there was some issue in the install-cmd for the software, but it did not report the error correctly back to to the powershell script which did not fail.

Looking at https://gist.githubusercontent.com/KurtTheBerner/6efdf08cbcac24a1f4ee2d28323383c0/raw/2522ba1cadfe8f6c11afd1f0e731661d21411ee1/chocolatey.log, you will see it reports the package install as successful, and it does see all of the files when it goes through and runs the checksums for the files (after the powerShell script runs). Finding the files and reporting success means the package installation was successful.

I would spend a closer look at the actual thing the installer is calling to see how it is not installing the software itself.

Here is the script you are running:

<#
==============================================================================
 FILE     : $chocolateyInstall.ps1 $
 PROJECT  : Install Script Library
 COPYRIGHT: Copyright (c) myCompany
 PURPOSE  : 
 COMMENT  : 
------------------------------------------------------------------------------
 HISTORY (latest entry first):
 2012-03-22  Version 1.0
==============================================================================
#>

$ErrorActionPreference = 'Stop'

$packageName = 'My big Package'
$toolsDir    = "$(Split-Path -parent $MyInvocation.MyCommand.Definition)"
$logDir      = $env:ALLUSERSPROFILE + "\MyCompany\InstallLogs\"
$logFileName = $logDir + "$packageName.log"

& $toolsDir\SQL-Install.cmd | Out-Null

Copy-Item -Path "$env:ProgramFiles\Microsoft SQL Server\120\Setup Bootstrap\Log\summary.txt" -Destination "$logDir\$packageName.log"

As you can see from the log, it finishes running that without any reported issues:

2017-08-11 14:36:26,081 7020 [DEBUG] - Built-in PowerShell host called with ['[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\mybigpackage\tools\chocolateyInstall.ps1' -installArguments '' -packageParameters '''] exited with '0'.

Since you are copying the log from the software install, I would take a closer look at that to see what errors it ran into. That's where you are going to likely find the issue. HTH

@ferventcoder
Copy link
Member

Calling a .cmd file may not produce an exit code like you want, but you can always check LastExitCode to see if it errored on install.

@KurtTheBerner
Copy link

@ferventcoder I found the issue, the installation was cancelled and the ExitCode was not reported correctly.
I fixed it and now the Package has been successfully installed, but still the following Warning occurs:

Error computing hash for 'C:\ProgramData\chocolatey\lib\mybigpackage\mybigpackage.nupkg'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.

This warning has irritated me and I thought the problem is caused by this Warning and prevents me from looking deeper in my Code.
Thanks for your Help.

TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue May 7, 2022
Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue May 7, 2022
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jun 27, 2022
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jun 27, 2022
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
@TheCakeIsNaOH TheCakeIsNaOH self-assigned this Jul 30, 2022
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Sep 23, 2022
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Dec 22, 2022
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jan 7, 2023
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jan 13, 2023
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jun 6, 2023
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jun 15, 2023
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jan 10, 2024
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Apr 28, 2024
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
TheCakeIsNaOH added a commit to TheCakeIsNaOH/choco that referenced this issue Jun 12, 2024
…files

Previously, when getting the checksum of a file, this read the file
into a byte array to pass to ComputeHash. However, this was limited to
files of 2gb or less, and was not efficent memory wise. This changes
the method to using a filestream which is passed to ComputeHash, which
should allow arbitrary file sizes with better memory usage.

The unit test for this method had to be removed because the mock
filesystem does not allow opening a file stream.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants