Skip to content
This repository was archived by the owner on Mar 17, 2025. It is now read-only.

Commit 9dbdf9a

Browse files
beelchesterbeelchester
authored
reformat: benchmark Results (#271)
* reformat benchmarks Signed-off-by: Sahil Yeole <[email protected]> * update results.md Signed-off-by: Sahil Yeole <[email protected]> * update query columns Signed-off-by: Sahil Yeole <[email protected]> --------- Signed-off-by: Sahil Yeole <[email protected]>
1 parent 89efe36 commit 9dbdf9a

File tree

9 files changed

+90
-110
lines changed

9 files changed

+90
-110
lines changed

‎README.md

Lines changed: 25 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -44,73 +44,44 @@ Get started with the benchmarks:
4444

4545
## Benchmark Results
4646

47-
```graphql
48-
{
49-
posts {
50-
title
51-
}
52-
}
53-
```
47+
<!-- PERFORMANCE_RESULTS_START -->
5448

55-
<!-- PERFORMANCE_RESULTS_START_1 -->
49+
| Query | Server | Requests/sec | Latency (ms) | Relative |
50+
|-------:|--------:|--------------:|--------------:|---------:|
51+
| 1 | `{ posts { id userId title user { id name email }}}` |
52+
|| [Tailcall] | `20,459.40` | `4.82` | `38.67x` |
53+
|| [async-graphql] | `2,416.38` | `42.11` | `4.57x` |
54+
|| [Caliban] | `2,112.29` | `51.58` | `3.99x` |
55+
|| [Apollo GraphQL] | `2,013.53` | `49.98` | `3.81x` |
56+
|| [Gqlgen] | `1,427.79` | `76.83` | `2.70x` |
57+
|| [Netflix DGS] | `529.12` | `131.33` | `1.00x` |
58+
| 2 | `{ posts { title }}` |
59+
|| [Tailcall] | `49,734.70` | `2.04` | `18.15x` |
60+
|| [Caliban] | `13,721.40` | `11.63` | `5.01x` |
61+
|| [async-graphql] | `9,333.76` | `11.00` | `3.41x` |
62+
|| [Apollo GraphQL] | `4,299.68` | `25.27` | `1.57x` |
63+
|| [Netflix DGS] | `2,778.82` | `70.96` | `1.01x` |
64+
|| [Gqlgen] | `2,739.60` | `39.57` | `1.00x` |
5665

57-
| Server | Requests/sec | Latency (ms) |
58-
|--------:|--------------:|--------------:|
59-
| [Tailcall] | `59,446.30` | `1.67` |
60-
| [Caliban] | `9,322.02` | `11.04` |
61-
| [async-graphql] | `7,271.63` | `13.75` |
62-
| [Hasura] | `2,377.29` | `41.97` |
63-
| [Gqlgen] | `2,089.38` | `49.36` |
64-
| [Apollo GraphQL] | `1,727.46` | `57.71` |
65-
| [Netflix DGS] | `1,599.55` | `65.91` |
66+
<!-- PERFORMANCE_RESULTS_END -->
6667

67-
<!-- PERFORMANCE_RESULTS_END_1 -->
6868

69-
### Throughput (Higher is better)
69+
70+
### 1. `{posts {title body user {name}}}`
71+
#### Throughput (Higher is better)
7072

7173
![Throughput Histogram](assets/req_sec_histogram1.png)
7274

73-
### Latency (Lower is better)
75+
#### Latency (Lower is better)
7476

7577
![Latency Histogram](assets/latency_histogram1.png)
7678

77-
---
78-
79-
### Test Query
80-
```graphql
81-
{
82-
posts {
83-
id
84-
userId
85-
title
86-
user {
87-
id
88-
name
89-
email
90-
}
91-
}
92-
}
93-
```
94-
95-
<!-- PERFORMANCE_RESULTS_START_2 -->
96-
97-
| Server | Requests/sec | Latency (ms) |
98-
|--------:|--------------:|--------------:|
99-
| [Tailcall] | `27,772.00` | `3.59` |
100-
| [Hasura] | `4,227.10` | `23.67` |
101-
| [Caliban] | `1,517.39` | `65.86` |
102-
| [async-graphql] | `1,429.70` | `69.83` |
103-
| [Gqlgen] | `633.51` | `158.42` |
104-
| [Netflix DGS] | `354.80` | `218.91` |
105-
| [Apollo GraphQL] | `278.74` | `356.12` |
106-
107-
<!-- PERFORMANCE_RESULTS_END_2 -->
108-
109-
### Throughput (Higher is better)
79+
### 2. `{posts {title body}}`
80+
#### Throughput (Higher is better)
11081

11182
![Throughput Histogram](assets/req_sec_histogram2.png)
11283

113-
### Latency (Lower is better)
84+
#### Latency (Lower is better)
11485

11586
![Latency Histogram](assets/latency_histogram2.png)
11687

‎analyze.sh

Lines changed: 37 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -101,36 +101,53 @@ done
101101
IFS=$'\n' sortedServers=($(for server in "${!serverRPS[@]}"; do echo "$server ${serverRPS[$server]}"; done | sort -rn -k2 | cut -d' ' -f1))
102102

103103
echo "Sorted servers: ${sortedServers[@]}"
104+
lastServer="${sortedServers[-1]}"
105+
lastServerReqSecs=${avgReqSecs[$lastServer]}
106+
104107
# Start building the resultsTable
105-
resultsTable="<!-- PERFORMANCE_RESULTS_START_${whichBench} -->\n\n| Server | Requests/sec | Latency (ms) |\n|--------:|--------------:|--------------:|"
108+
if [[ $whichBench == 1 ]]; then
109+
resultsTable="<!-- PERFORMANCE_RESULTS_START -->\n\n| Query | Server | Requests/sec | Latency (ms) | Relative |\n|-------:|--------:|--------------:|--------------:|---------:|\n| $whichBench | \`{ posts { id userId title user { id name email }}}\` |"
110+
else
111+
resultsTable="| $whichBench | \`{ posts { title }}\` |"
112+
fi
106113

107114
# Build the resultsTable with sorted servers and formatted numbers
108115
for server in "${sortedServers[@]}"; do
109-
formattedReqSecs=$(printf "%.2f" ${avgReqSecs[$server]} | perl -pe 's/(?<=\d)(?=(\d{3})+(\.\d*)?$)/,/g')
110-
formattedLatencies=$(printf "%.2f" ${avgLatencies[$server]} | perl -pe 's/(?<=\d)(?=(\d{3})+(\.\d*)?$)/,/g')
111-
resultsTable+="\n| [${formattedServerNames[$server]}] | \`${formattedReqSecs}\` | \`${formattedLatencies}\` |"
112-
done
113-
114-
resultsTable+="\n\n<!-- PERFORMANCE_RESULTS_END_${whichBench} -->"
116+
formattedReqSecs=$(printf "%.2f" ${avgReqSecs[$server]} | perl -pe 's/(?<=\d)(?=(\d{3})+(\.\d*)?$)/,/g')
117+
formattedLatencies=$(printf "%.2f" ${avgLatencies[$server]} | perl -pe 's/(?<=\d)(?=(\d{3})+(\.\d*)?$)/,/g')
118+
# Calculate the relative performance
119+
relativePerformance=$(echo "${avgReqSecs[$server]} $lastServerReqSecs" | awk '{printf "%.2f", $1 / $2}')
115120

116-
echo -e $resultsTable
121+
resultsTable+="\n|| [${formattedServerNames[$server]}] | \`${formattedReqSecs}\` | \`${formattedLatencies}\` | \`${relativePerformance}x\` |"
122+
done
117123

118-
# Check if the markers are present
119-
if grep -q "PERFORMANCE_RESULTS_START_${whichBench}" README.md; then
120-
# Replace the old results with the new results
121-
sed -i "/PERFORMANCE_RESULTS_START_${whichBench}/,/PERFORMANCE_RESULTS_END_${whichBench}/c\\$resultsTable" README.md
122-
else
123-
# Append the results at the end of the README.md file
124-
echo -e "\n$resultsTable" >>README.md
124+
if [[ $whichBench == 2 ]]; then
125+
resultsTable+="\n\n<!-- PERFORMANCE_RESULTS_END -->"
125126
fi
126127

128+
echo "resultsTable: $resultsTable"
129+
127130
# Print the results table in a new file
128131
resultsFile="results.md"
129-
echo -e "## Benchmark $whichBench results\n" >>$resultsFile
130-
echo -e $resultsTable >>$resultsFile
131-
132-
# Print the results as a table in the terminal
133-
echo -e $resultsTable | sed "s/<!-- PERFORMANCE_RESULTS_START_${whichBench}-->//;s/<!-- PERFORMANCE_RESULTS_END_${whichBench}-->//"
132+
echo -e $resultsTable >> $resultsFile
133+
134+
135+
if [[ $whichBench == 2 ]]; then
136+
finalResults=$(printf '%s\n' "$(cat $resultsFile)" | sed 's/$/\\n/'| tr -d '\n')
137+
# Remove the last newline character
138+
finalResults=${finalResults::-2}
139+
140+
# Print the results as a table in the terminal
141+
echo -e $finalResults | sed "s/<!-- PERFORMANCE_RESULTS_START-->//;s/<!-- PERFORMANCE_RESULTS_END-->//"
142+
# Check if the markers are present
143+
if grep -q "PERFORMANCE_RESULTS_START" README.md; then
144+
# Replace the old results with the new results
145+
sed -i "/PERFORMANCE_RESULTS_START/,/PERFORMANCE_RESULTS_END/c\\$finalResults" README.md
146+
else
147+
# Append the results at the end of the README.md file
148+
echo -e "\n$finalResults" >> README.md
149+
fi
150+
fi
134151

135152
# Move the generated images to the assets folder
136153
mv $reqSecHistogramFile assets/

‎results.md

Lines changed: 18 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,20 @@
1-
## Benchmark 1 results
1+
<!-- PERFORMANCE_RESULTS_START -->
22

3-
<!-- PERFORMANCE_RESULTS_START_1 -->
3+
| Query | Server | Requests/sec | Latency (ms) | Relative |
4+
|-------:|--------:|--------------:|--------------:|---------:|
5+
| 1 | `{ posts { id userId title user { id name email }}}` |
6+
|| [Tailcall] | `20,459.40` | `4.82` | `38.67x` |
7+
|| [async-graphql] | `2,416.38` | `42.11` | `4.57x` |
8+
|| [Caliban] | `2,112.29` | `51.58` | `3.99x` |
9+
|| [Apollo GraphQL] | `2,013.53` | `49.98` | `3.81x` |
10+
|| [Gqlgen] | `1,427.79` | `76.83` | `2.70x` |
11+
|| [Netflix DGS] | `529.12` | `131.33` | `1.00x` |
12+
| 2 | `{ posts { title }}` |
13+
|| [Tailcall] | `49,734.70` | `2.04` | `18.15x` |
14+
|| [Caliban] | `13,721.40` | `11.63` | `5.01x` |
15+
|| [async-graphql] | `9,333.76` | `11.00` | `3.41x` |
16+
|| [Apollo GraphQL] | `4,299.68` | `25.27` | `1.57x` |
17+
|| [Netflix DGS] | `2,778.82` | `70.96` | `1.01x` |
18+
|| [Gqlgen] | `2,739.60` | `39.57` | `1.00x` |
419

5-
| Server | Requests/sec | Latency (ms) |
6-
|--------:|--------------:|--------------:|
7-
| [Tailcall] | `59,446.30` | `1.67` |
8-
| [Caliban] | `9,322.02` | `11.04` |
9-
| [async-graphql] | `7,271.63` | `13.75` |
10-
| [Hasura] | `2,377.29` | `41.97` |
11-
| [Gqlgen] | `2,089.38` | `49.36` |
12-
| [Apollo GraphQL] | `1,727.46` | `57.71` |
13-
| [Netflix DGS] | `1,599.55` | `65.91` |
14-
15-
<!-- PERFORMANCE_RESULTS_END_1 -->
16-
## Benchmark 2 results
17-
18-
<!-- PERFORMANCE_RESULTS_START_2 -->
19-
20-
| Server | Requests/sec | Latency (ms) |
21-
|--------:|--------------:|--------------:|
22-
| [Tailcall] | `27,772.00` | `3.59` |
23-
| [Hasura] | `4,227.10` | `23.67` |
24-
| [Caliban] | `1,517.39` | `65.86` |
25-
| [async-graphql] | `1,429.70` | `69.83` |
26-
| [Gqlgen] | `633.51` | `158.42` |
27-
| [Netflix DGS] | `354.80` | `218.91` |
28-
| [Apollo GraphQL] | `278.74` | `356.12` |
29-
30-
<!-- PERFORMANCE_RESULTS_END_2 -->
20+
<!-- PERFORMANCE_RESULTS_END -->

‎test_query1.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
graphqlEndpoint="${1:-http://localhost:8000/graphql}"
2-
curl -i -X POST -d '{"query": "{posts{title}}"}' $graphqlEndpoint -H "Content-Type: application/json"
2+
curl -i -X POST -d '{"query": "{posts{id,userId, title, user{id, name, email}}}"}' $graphqlEndpoint -H "Content-Type: application/json"

‎test_query2.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
graphqlEndpoint="${1:-http://localhost:8000/graphql}"
2-
curl -i -X POST -d '{"query": "{posts{id,userId, title, user{id, name, email}}}"}' $graphqlEndpoint -H "Content-Type: application/json"
2+
curl -i -X POST -d '{"query": "{posts{title}}"}' $graphqlEndpoint -H "Content-Type: application/json"

‎wrk/bench1.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
graphqlEndpoint="${1:-http://localhost:8000/graphql}"
2-
wrk -d 10 -t 4 -c 100 -s $(pwd)/wrk/wrk1.lua $graphqlEndpoint
2+
wrk -d 30 -t 4 -c 100 -s $(pwd)/wrk/wrk1.lua $graphqlEndpoint

‎wrk/bench2.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
graphqlEndpoint="${1:-http://localhost:8000/graphql}"
2-
wrk -d 30 -t 4 -c 100 -s $(pwd)/wrk/wrk2.lua $graphqlEndpoint
2+
wrk -d 10 -t 4 -c 100 -s $(pwd)/wrk/wrk2.lua $graphqlEndpoint

‎wrk/wrk1.lua

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
wrk.method = "POST"
2-
wrk.body = '{"operationName":null,"variables":{},"query":"{posts{title}}"}'
2+
wrk.body = '{"operationName":null,"variables":{},"query":"{posts{id,userId,title,user{id,name,email}}}"}'
33
wrk.headers["Connection"] = "keep-alive"
4-
wrk.headers["User-Agent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
4+
wrk.headers["User-Agent"] =
5+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
56
wrk.headers["Content-Type"] = "application/json"

‎wrk/wrk2.lua

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
wrk.method = "POST"
2-
wrk.body = '{"operationName":null,"variables":{},"query":"{posts{id,userId,title,user{id,name,email}}}"}'
2+
wrk.body = '{"operationName":null,"variables":{},"query":"{posts{title}}"}'
33
wrk.headers["Connection"] = "keep-alive"
4-
wrk.headers["User-Agent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
4+
wrk.headers["User-Agent"] =
5+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
56
wrk.headers["Content-Type"] = "application/json"

0 commit comments

Comments
 (0)