Hi FoundationDB team,
First, thank you for maintaining such a well-documented project. We've been benchmarking FoundationDB on our own hardware and are trying to line our results up against the numbers published on the official Performance page (https://apple.github.io/foundationdb/performance.html), but we're having trouble reproducing them and would be grateful for some clarification.
The page documents the hardware clearly (e.g. 8.2M ops/sec on a 24-machine EC2 c3.8xlarge cluster, 384 server processes, 90% read / 10% write, 16-byte keys and 8-100 byte values; and the latency figures on a 12-machine E3-1240 / SATA SSD cluster with 48 processes). However, a few details that matter for reproduction don't seem to be covered:
- Benchmark tool - Which tool was used to generate the load? Was it a workload defined inside FDB's simulation/test harness,
mako, a custom client, or something else?
- Client-side configuration - How many load-generating client machines/processes/threads were used, and what concurrency (in-flight transactions) per client? Ops/sec is very sensitive to this and it isn't stated.
- Cluster configuration - What redundancy mode and storage engine (ssd/ssd-2/redwood) were used, and were any non-default knobs set?
- FDB version - Which release were these numbers measured on?
- Reproduction script - Is there a script or a documented procedure we could run to reproduce these results end-to-end? If one exists in the repo, a pointer would be very helpful.
Our goal isn't to dispute the numbers - we just want to make sure we're measuring the same thing so our own comparisons are fair. Any pointers would be much appreciated. Thanks!
Hi FoundationDB team,
First, thank you for maintaining such a well-documented project. We've been benchmarking FoundationDB on our own hardware and are trying to line our results up against the numbers published on the official Performance page (https://apple.github.io/foundationdb/performance.html), but we're having trouble reproducing them and would be grateful for some clarification.
The page documents the hardware clearly (e.g. 8.2M ops/sec on a 24-machine EC2 c3.8xlarge cluster, 384 server processes, 90% read / 10% write, 16-byte keys and 8-100 byte values; and the latency figures on a 12-machine E3-1240 / SATA SSD cluster with 48 processes). However, a few details that matter for reproduction don't seem to be covered:
mako, a custom client, or something else?Our goal isn't to dispute the numbers - we just want to make sure we're measuring the same thing so our own comparisons are fair. Any pointers would be much appreciated. Thanks!