Buffering lambda execution time gets too high when number of pieces increase

[vasco-santos]

This caused the pipeline to get congested with timeouts recently. See aggregates from 6th and 7th January

Example aggregates to analyse:

• bafkzcibcaapjqqcrtp4zirffd53geuc6aobzacndjbqf6p7zbcrx77omccdrqoa
• bafkzcibcaapov2hh7lsjc5qyzng5uzejhl5omqzhzyjctjd6e7jrct6gtkj7yca

We should run the lambda code locally and see what makes the execution time spike

[alanshaw]

It's hashes. FML it's hashes.

Aggregates with 40,000+ pieces require 2,000,000+ hashes and there's simply not enough time to do that much work.

Things I have done:

• Increase time allocated to the lambda to maximum 15 minutes.
• Increase memory allocated to the lambda (apparently this increases CPU perf). This didn't really have any effect, and also I subsequently read that for single threaded lambdas (JS) adding memory beyond 4GB yields only marginal gains.
• Allow custom hasher to be used in aggregate building: feat: custom SHA256 hashing function for aggregate builder data-segment#42 and feat: pass custom hasher to index create data-segment#44
• Use Node.js native hasher perf: use Node.js crypto sha256 hasher for aggregate building #107
• Use new Node.js crypto.hash(...) which is faster than createHash+update+digest: feat: use nodejs 20 on buffer reducer #108
• Configure maximum aggregate pieces (effectively an upper bound on number of hashes that need to be done) fix: configure max pieces w3up#1566