Adding a quick split command implementation #1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've added two commands:
To try it out:
Clone
(The optional
--depth
and--single-branch
flags just reduce the download size and provides you with the essentials.)The Splitter
Build
If you don't have Go, you can get it from: https://go.dev/doc/install
Run
Example
./split -source fake-source.json -chunk-size 10_000 # 545.73s user 68.38s system 174% cpu 5:52.61 total for 60GiB
The Fakr
The utility
fakr
generates a fake structure of data. Nice to test withBuild
Run
ls -nh fake-source.json # -rw-r--r-- 1 501 20 683M Mar 4 15:22 fake-source.json