Testing with RClone #5828

sanderegg · 2024-05-15T09:44:38Z

sanderegg
May 15, 2024
Maintainer

RClone session

Created a EC2 g4dn.xlarge from the latest AMI without disk of 04.05.2024. then installed RClone using the script
sudo su
sudo apt install btop # to show what is going on while transferring

1.63.1

Install rclone 1.63.1

R_CLONE_VERSION="1.63.1"
curl --silent --location --remote-name "https://downloads.rclone.org/v${R_CLONE_VERSION}/rclone-v${R_CLONE_VERSION}- 
 linux-amd64.deb"
dpkg --install "rclone-v${R_CLONE_VERSION}-linux-amd64.deb"
rm "rclone-v${R_CLONE_VERSION}-linux-amd64.deb"
rclone --version

run original command from before version changes to download

rclone  --transfers 5 --use-mmap --buffer-size 0M --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

obtained transfer rate oscillate between 130-110MiB/s, total time 5m47.8s, 46 objects 30.838GiB
at the end of the transfer almost a minute goes by, probably computing the checksum?
run original command to upload the same to a different folder on S3

rclone  --transfers 5 --use-mmap --buffer-size 0M --use-json-log --stats 200ms --verbose sync test/ mys3:san-is-testing-here/upload-test/workspace/  --links

before transmitting, computing the checksum of the large file takes >1min20s, then uploads at around 100-108MiB/s, 1 CPU 90-100%, uses about 100MB, total time about 6min35

1.66.0

Install version 1.66.0

R_CLONE_VERSION="1.66.0"
curl --silent --location --remote-name "https://downloads.rclone.org/v${R_CLONE_VERSION}/rclone-v${R_CLONE_VERSION}-linux-amd64.deb"
dpkg --install "rclone-v${R_CLONE_VERSION}-linux-amd64.deb"
rm "rclone-v${R_CLONE_VERSION}-linux-amd64.deb"
rclone --version

run original command to download

rclone  --transfers 5 --use-mmap --buffer-size 0M --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

Cpu around 20%-30% while transfering, 60-80% during checksum at the end 63MB RAM, transfer rate around 130MiB/s, total time5m59.5s
run original command to upload the same to a different folder on S3

rclone  --transfers 5 --use-mmap --buffer-size 0M --use-json-log --stats 200ms --verbose sync test/ mys3:san-is-testing-here/upload-test2/workspace/  --links

running the command hanged the machine!!! going to EC2 serial console shows:
last screenshot
stopped the instance as it was not responding
stopping did not work -> terminated the instance

3rd trial with rclone 1.66.0

download

removing --buffer-size parameter, this is for async. default is to 16MB per transfer.

rclone  --transfers 5 --use-mmap --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

--> no changes
2. removing --mmap https://forum.rclone.org/t/what-is-the-impact-of-use-mmap-besides-reducing-memory-usage/40677/8

rclone  --transfers 5 --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

--> no changes
3.

rclone --transfers 5 --use-json-log --stats 200ms --verbose --disable-http2 --multi-thread-streams 16 --multi-thread-chunk-size 32Mi sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

--> 3x momentarily speed up, total time 4m43

use defaults --> 5min19

rclone --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

rclone --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

4m43

rclone --use-json-log --stats 200ms --verbose --disable-http2 --multi-thread-streams 10 sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

4m43 no change

rclone --use-json-log --stats 200ms --verbose --disable-http2 --multi-thread-streams 10 --multi-thread-chunk-size 32Mi sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

4m42 -- no change

rclone --use-json-log --stats 200ms --verbose --disable-http2 --multi-thread-streams 16 sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

4m43 -- no changes

rclone --use-json-log --stats 200ms --verbose --disable-http2 --multi-thread-streams 16 --checkers 32 sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

upload

rclone  --progress --stats 200ms --verbose sync test/ mys3:san-is-testing-here/upload-rclone/workspace/  --links

30 minutes???
tried a second time and the EC2 is unresponsive, no ssh, had to terminate it

test bandwith

rclone check --download mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ --progress

test aws CLI

aws s3 sync s3://san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test-aws/

download 3min13
2. with aws configure set default.s3.max_concurrent_requests 16 almot no changes

upload

aws s3 sync test-aws/ s3://san-is-testing-here/upload-aws/

3min38

Conclusion thus far:

RClone 1.66.0 is dangerous at least for uploads even with default settings. The EC2 machine just hangs forever.
AWs s3 sync is about 2x faster than RClone for our use-case both for upload/download.
The AWS sync feature is not part of boto3 and only available via CLI but this is similar to what we do with RClone which only provides a CLI as well.

sanderegg · 2024-05-15T14:40:15Z

sanderegg
May 15, 2024
Maintainer Author

Tested download with current command which does not kernel panic on upload:

rclone --transfers 5 --use-mmap --buffer-size 0M sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links --progress --multi-thread-streams 1

15 minutes!!!
super slow very bad. Let's try to downgrade.

0 replies

sanderegg · 2024-05-15T15:05:04Z

sanderegg
May 15, 2024
Maintainer Author

tested download using 1.63.1 removing `--buffer-size` flag (defaults to 16M)

rclone  --transfers 5 --use-mmap --use-json-log --stats 200ms --verbose sync mys3:san-is-testing-here/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --links

5min30s

1 reply

GitHK May 16, 2024
Collaborator

thanks for checking all possible options. Very well we can drop this if we have a better alternative!

matusdrobuliak66 · 2024-08-06T09:46:18Z

matusdrobuliak66
Aug 6, 2024
Collaborator

#6119

0 replies

matusdrobuliak66 · 2024-08-06T15:06:12Z

matusdrobuliak66
Aug 6, 2024
Collaborator

RClone vs AWS Cli manual test after implementation

Test with 1000 files with 100 MB size + 2 files with 10GB size

Staging (RClone with old behaviour)
- simcore/services/dynamic/jupyter-smash 3.0.21
Master (AWS Cli with EFS distributed file system)
- simcore/services/dynamic/jupyter-math 2.0.10
Master (AWS Cli without EFS) ~ around 167 MiB/s download speed
- simcore/services/dynamic/jupyter-math 2.0.10

Attempt	RClone	AWS CLI	AWS CLI with EFS
First Upload	9 mins	12 mins	10 mins
First Download	8 mins 46 secs	12 mins	6 secs
Second Upload (After modifying 3 files)	9 mins	3 secs	6 secs
Second Download	8 mins 46 secs	12 mins	6 secs

1 reply

YuryHrytsuk Aug 8, 2024
Collaborator

Wow. EFS is a real thing 🚀

sanderegg · 2024-08-07T06:23:35Z

sanderegg
Aug 7, 2024
Maintainer Author

After @matusdrobuliak66 comment above I did yet another trial:

5th trial 120.8 GB

rclone 1.63.1

download from s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

rclone sync mys3:sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test/ --progress --transfers 5 --use-mmap --buffer-size 0M --links --verbose

completed download (without checksum) after ~18minutes,
completed process after 32minutes 37 seconds (test 1)
completed after 32m36s (test 2)
upload to s3://san-testing-again/aws-rclone/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

rclone sync test/ mys3:san-testing-again/aws-rclone/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/  --progress --transfers 5 --use-mmap --buffer-size 0M --links --verbose

completed after 32m12.1s

AWS CLI S3 sync

download from s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

time aws s3 sync s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ test-aws/ --delete

completed after 20minutes 26 secs (test 1)
completed after 17m3.999s (test 2)
upload to s3://san-testing-again/aws-testing/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

  time aws s3 sync test-aws/ s3://san-testing-again/aws-testing/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ --delete

completed after 16m15s

Summary

Attempt	RClone	AWS CLI	S5cmd
Download 1	32m37s	20m26s	16m29.387s
Download 2	32m36s	17m3.999s	-
Upload	32m36s	16m15s	-

--> we need to understand why the improvement is not visible in oSparc

0 replies

sanderegg · 2024-08-07T07:41:30Z

sanderegg
Aug 7, 2024
Maintainer Author

here is yet another alternative: https://github.com/peak/s5cmd

early trials showed the same as aws-cli on the tested dataset for download: 16m29.387s

0 replies

sanderegg · 2024-08-07T09:59:52Z

sanderegg
Aug 7, 2024
Maintainer Author

g4dn.4xlarge with EBS disk at 1000Mbips transfer speed + 6000IOPS

AWS CLI

download from s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

time aws s3 sync s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ test-aws/ --delete

completed after 15m48.731s, transfer rate maxed out at 125MiB/s still

upload to s3://san-testing-again/aws-testing/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/

command:

  time aws s3 sync test-aws/ s3://san-testing-again/aws-testing/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ --delete

completed after 9m49.695s

tweaked AWS CLI

from https://stackoverflow.com/questions/9664904/best-way-to-move-files-between-s3-buckets

aws configure set default.s3.max_concurrent_requests 1000
aws configure set default.s3.max_queue_size 100000

"(The following commands will tell the AWS CLI to use 1,000 threads to execute jobs (each a small file or one part of a multipart copy) and look ahead 100,000 jobs)"

command:

time aws s3 sync s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ test-aws/ --delete

completed after 7m58.003s, transfer rate maxed out at 300MiB/s
upload to s3://san-testing-again/aws-testing-tweaked/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

  time aws s3 sync test-aws/ s3://san-testing-again/aws-testing-tweaked/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ --delete

completed after FAILURE too many open files

RClone 1.63.1

download from s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

rclone sync mys3:sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test-rclone/ --progress --transfers 5 --use-mmap --buffer-size 0M --links --verbose

completed after 13m4.0s, transfer rate around 230MiB/s (including checksum)
upload to san-testing-again/aws-rclone/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

rclone sync test-rclone/ mys3:san-testing-again/aws-rclone/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/  --progress --transfers 5 --use-mmap --buffer-size 0M --links --verbose

completed after 8m44.3s

tweaked RClone 1.63.1

removed checksum, mmap and buffer-size.
checksum computation speed is directly linked to drive speed.

rclone sync mys3:sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/workspace/ test-rclone/ --progress --transfers 5 --links --verbose --ignore-checksum

completed after 8m19.0s

upload to san-testing-again/aws-rclone/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/

command:

rclone sync test-rclone/ mys3:san-testing-again/aws-rclone-tweaked/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/  --progress --transfers 5 --links --verbose --ignore-checksum

completed after 8m30.6s

S5cmd

download from s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/

installation:

wget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz .
tar -xvf s5cmd_2.2.2_Linux-64bit.tar.gz
./s5cmd version

command:

time ./s5cmd --stat --json sync --delete "s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/*" test-s5cmd/

completed after 8m10.191s, transfer rate maxed out at 300MiB/s
upload to s3://san-testing-again/aws-s5cmd/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

  time ./s5cmd --stat --json sync --delete test-s5cmd/ s3://san-testing-again/aws-s5cmd/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/

completed after 4m50.391s

Summary

	RClone (as in code)	S5cmd	AWS CLI v1	RClone (tweaked-nochecksum)	AWS CLI v1 (tweaked)
Download	13m4s	8m10.191s	15m48.731s	8m19.0s	7m58.003s
Upload	8m44.3s	4m50.391s	9m49.695s	8m30.6s	FAILURE

AWS-cli v2

The AMI had an old version of the CLI
followed AWS info

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update
/usr/local/bin/aws --version # shows now aws-cli/2.17.25 Python/3.11.9 Linux/6.5.0-1022-aws exe/x86_64.ubuntu.22
aws --version # shows aws-cli/1.22.34 Python/3.10.12 Linux/6.5.0-1022-aws botocore/1.23.34 very old!

download from s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

time /usr/local/bin/aws s3 sync s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ test-aws-v2/ --delete

completed after 17m14.856s

upload to s3://san-testing-again/aws-testing/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/

command:

  time /usr/local/bin/aws s3 sync test-aws-v2/ s3://san-testing-again/aws-testing-v2/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ --delete

completed after 10m1.189s

tweaked AWS CLI v2

from https://stackoverflow.com/questions/9664904/best-way-to-move-files-between-s3-buckets

aws configure set default.s3.max_concurrent_requests 1000

"(The following commands will tell the AWS CLI to use 1,000 threads to execute jobs (each a small file or one part of a multipart copy) ~~and look ahead 100,000 jobs)"~~

command:

time /usr/local/bin/aws s3 sync s3://sim4life-simcore/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ test-aws-v2-tweaked/ --delete

completed after 7m27.752s
upload to s3://san-testing-again/aws-testing-tweaked/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/
command:

  time /usr/local/bin/aws s3 sync test-aws-v2-tweaked/ s3://san-testing-again/aws-testing-v2-tweaked/72a6031e-b9d5-11ee-aeb0-02420a0bcae9/8b5290bb-7ded-409a-bdd5-4c95250b12cd/ --delete

completed after FAILURE too many open files
--> the max_concurrent_requests on upload is a bad idea

0 replies

sanderegg · 2024-08-08T09:04:15Z

sanderegg
Aug 8, 2024
Maintainer Author

Results

SAN original tests on bare metal (g4dn.xlarge)

Standard GP3 (125MiB/s - 3000IOPS)	Download	Upload	Comments
RClone 1.66.0	5m59.5s	EC2 total breakdown, impossible to ssh, had to reboot	30GB dataset (46 objects)
RClone 1.63.1	5m47.8s	6min35	30GB dataset
AWS-CLI v1	3min13	3min38	30GB dataset

MD test on oSparc after AWS-CLI PR came in master/staging

Test with 1000 files with 100 MB size + 2 files with 10GB size
AWS-CLI was tested in AWS-master using jupyter-math, that means an autoscaled machine (g4dn.4xlarge, using instance storage)
RClone was tested in AWS-staging using jupyter-smash, which most probably ended up on a fixed machine (osparc-staging-GPU with st1 HDD 40-245 MiB/s - ??IOPS)

Instance storage (~330MiB/s - ?? IOPS)	Download	Upload	Comments
RClone 1.63.1	8min46	9min	staging
AWS-CLI v2	12min	12min	master

SAN subsequent tests on bare metal (g4dn.xlarge)

Same Dataset as done earlier but is now 120GB

Standard GP3 (125MiB/s - 3000IOPS)	Download	Upload
RClone 1.63.1	32min37s	32min36s
AWS-CLI v1	20min26s	16min15s
S5cmd	16m29s

SAN subsequent tests on bare metal (g4dn.4xlarge) with GP3 1000MiB/s - 6000IOPS

Same dataset as done earlier but is now 120GB

GP3 (1000MiB/s - 6000IOPS)	Download	Upload	Comments
RClone 1.63.1	13min4s	8min44s	checksum computation improvements!
AWS-CLI v1	15min48s	9min49s
AWS-CLI v2	17min14s	10min01s
S5cmd	8min10s	4min50s
AWS-CLI v2 tweaked	7min27s	FAILURE

GP3 disk costs

0.08/month-GB + 0.005/provisioned IOPS-month over 3,000 IOPS + 0.04/provisioned MiB/s-month over 125 MiB/s

	computation	costs/month
500GB Standard GP3 (125MiB/s - 3000IOPS)	500x0.08	40$
500GB GP3 (1000MiB/s - 6000IOPS)	500x0.08 + 875x0.04 + 3000x0.005	90$	a min of 4000IOPS is required to use 1000MiB/s
500GB GP3 (1000MiB/s - 4000IOPS)	500x0.08 + 875x0.04 + 1000x0.005	80$
maxed out 500GB GP3 (1000MiB/s - 16000IOPS)	500x0.08 + 875x0.04 + 13000x0.005	140$

Conclusion

Default installed AWS-cli on AMI is v1 instead of v2
Tweaking settings of AWS-CLI works for download but fails for uploads.
RClone 1.66.0 is unusable since it triggers a kernel panic
Early testing that lead to AWS-CLI being constantly better than RClone were done on EC2 with EBS GP3 standard (125 MiB/s - 3000IOPS), showed AWS-cli ~1-2x faster than RClone
Tests with RClone vs AWS-CLI in osparc with jupyters on master/staging were done on machines with hard-drives with different specs but showed RClone 30% faster than AWS-CLI,
EC2 GP3 default hard-drive IOPS/throughput favors AWS CLI vs RClone,
EC2 GP3 with maxed throughput of 1000MiB/s favors RClone over AWS CLI v1/v2 with default settings,
EC2 GP3 with maxed throughput shows a real throughput of ~300-400MiB/s using RClone and S5cmd, maybe due to reduced throughput using internet gateways? see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html
hard-drive throughput directly affects RClone checksum computation speed which went down from ~10minutes on standard GP3 to about ~1-2minutes on maxed out GP3 which seems proportional to throughput,
RClone computes a checksum additionally to AWS-CLI/S5cmd that is stored as metadata in S3, this brings some additional security for later downloads as it checks whether the files were changed,
Costs of improving EBS storage is not very high and could bring some speed up to download/upload and for runtime

1 reply

matusdrobuliak66 Aug 15, 2024
Collaborator

Decision: We will continue to use RClone for now

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing with RClone #5828

{{title}}

Replies: 8 comments 3 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Testing with RClone #5828

sanderegg May 15, 2024 Maintainer

RClone session

1.63.1

1.66.0

3rd trial with rclone 1.66.0

download

upload

test bandwith

test aws CLI

Conclusion thus far:

Replies: 8 comments · 3 replies

sanderegg May 15, 2024 Maintainer Author

Tested download with current command which does not kernel panic on upload:

sanderegg May 15, 2024 Maintainer Author

tested download using 1.63.1 removing --buffer-size flag (defaults to 16M)

GitHK May 16, 2024 Collaborator

matusdrobuliak66 Aug 6, 2024 Collaborator

matusdrobuliak66 Aug 6, 2024 Collaborator

RClone vs AWS Cli manual test after implementation

YuryHrytsuk Aug 8, 2024 Collaborator

sanderegg Aug 7, 2024 Maintainer Author

5th trial 120.8 GB

rclone 1.63.1

AWS CLI S3 sync

Summary

sanderegg Aug 7, 2024 Maintainer Author

sanderegg Aug 7, 2024 Maintainer Author

AWS CLI

tweaked AWS CLI

RClone 1.63.1

tweaked RClone 1.63.1

S5cmd

Summary

AWS-cli v2

tweaked AWS CLI v2

sanderegg Aug 8, 2024 Maintainer Author

Results

SAN original tests on bare metal (g4dn.xlarge)

MD test on oSparc after AWS-CLI PR came in master/staging

SAN subsequent tests on bare metal (g4dn.xlarge)

SAN subsequent tests on bare metal (g4dn.4xlarge) with GP3 1000MiB/s - 6000IOPS

GP3 disk costs

Conclusion

matusdrobuliak66 Aug 15, 2024 Collaborator

sanderegg
May 15, 2024
Maintainer

Replies: 8 comments 3 replies

sanderegg
May 15, 2024
Maintainer Author

sanderegg
May 15, 2024
Maintainer Author

tested download using 1.63.1 removing `--buffer-size` flag (defaults to 16M)

GitHK May 16, 2024
Collaborator

matusdrobuliak66
Aug 6, 2024
Collaborator

matusdrobuliak66
Aug 6, 2024
Collaborator

YuryHrytsuk Aug 8, 2024
Collaborator

sanderegg
Aug 7, 2024
Maintainer Author

sanderegg
Aug 7, 2024
Maintainer Author

sanderegg
Aug 7, 2024
Maintainer Author

sanderegg
Aug 8, 2024
Maintainer Author

matusdrobuliak66 Aug 15, 2024
Collaborator