Skip to content

gui: compress ws messages, change frankendancer logo #5337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 12, 2025
Merged

gui: compress ws messages, change frankendancer logo #5337

merged 1 commit into from
Sep 12, 2025

Conversation

jherrera-jump
Copy link
Contributor

@jherrera-jump jherrera-jump commented Jun 10, 2025 •
edited
Loading

Allows clients to optionally compress websocket messages larger than 200 bytes (arbitrary threshold).

see firedancer-io/firedancer-frontend#33

@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 3 times, most recently from 10a88e7 to 6e12737 Compare June 10, 2025 20:58
@jherrera-jump jherrera-jump marked this pull request as ready for review June 10, 2025 21:00
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 3 times, most recently from 94460a1 to 3446eea Compare June 10, 2025 21:29
@jherrera-jump
Copy link
Contributor Author

Without compression

(.venv) [jherrera@tsnj2-ossdev-firedancer39 gui]$ python3 bandwidth.py
Incoming bandwidth for ('peers', 'update')                               :  50.68 Mbps
Incoming bandwidth for ('epoch', 'new')                                  :   8.87 Mbps
Incoming bandwidth for ('summary', 'live_tile_timers')                   :   0.15 Mbps
Incoming bandwidth for ('summary', 'live_txn_waterfall')                 :   0.05 Mbps
Incoming bandwidth for ('summary', 'tps_history')                        :   0.03 Mbps
Incoming bandwidth for ('summary', 'live_tile_primary_metric')           :   0.02 Mbps
Incoming bandwidth for ('slot', 'update')                                :   0.02 Mbps
Incoming bandwidth for ('summary', 'startup_progress')                   :   0.01 Mbps
Incoming bandwidth for ('summary', 'tiles')                              :   0.01 Mbps
Incoming bandwidth for ('summary', 'estimated_tps')                      :   0.00 Mbps
Incoming bandwidth for ('summary', 'estimated_slot_duration_nanos')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'optimistically_confirmed_slot')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'completed_slot')                     :   0.00 Mbps
Incoming bandwidth for ('summary', 'root_slot')                          :   0.00 Mbps
Incoming bandwidth for ('summary', 'skip_rate')                          :   0.00 Mbps
Total incoming bandwidth: 59.85 Mbps
Incoming bandwidth for ('summary', 'live_tile_timers')                   :   0.14 Mbps
Incoming bandwidth for ('summary', 'live_txn_waterfall')                 :   0.06 Mbps
Incoming bandwidth for ('slot', 'update')                                :   0.02 Mbps
Incoming bandwidth for ('summary', 'live_tile_primary_metric')           :   0.02 Mbps
Incoming bandwidth for ('summary', 'estimated_tps')                      :   0.00 Mbps
Incoming bandwidth for ('summary', 'optimistically_confirmed_slot')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'estimated_slot_duration_nanos')      :   0.00 Mbps
Total incoming bandwidth: 0.24 Mbps
Incoming bandwidth for ('summary', 'live_tile_timers')                   :   0.14 Mbps
Incoming bandwidth for ('summary', 'live_txn_waterfall')                 :   0.05 Mbps
Incoming bandwidth for ('summary', 'live_tile_primary_metric')           :   0.02 Mbps
Incoming bandwidth for ('slot', 'update')                                :   0.02 Mbps
Incoming bandwidth for ('summary', 'estimated_tps')                      :   0.00 Mbps
Incoming bandwidth for ('summary', 'estimated_slot_duration_nanos')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'optimistically_confirmed_slot')      :   0.00 Mbps
Total incoming bandwidth: 0.24 Mbps

With compression

(.venv) [jherrera@tsnj2-ossdev-firedancer39 fd1]$ python3 src/disco/gui/bandwidth.py
Incoming bandwidth for ('peers', 'update')                               :  11.87 Mbps
Incoming bandwidth for ('epoch', 'new')                                  :   4.48 Mbps
Incoming bandwidth for ('summary', 'live_tile_timers')                   :   0.15 Mbps
Incoming bandwidth for ('summary', 'live_txn_waterfall')                 :   0.03 Mbps
Incoming bandwidth for ('summary', 'live_tile_primary_metric')           :   0.01 Mbps
Incoming bandwidth for ('slot', 'update')                                :   0.01 Mbps
Incoming bandwidth for ('summary', 'tps_history')                        :   0.01 Mbps
Incoming bandwidth for ('summary', 'estimated_tps')                      :   0.00 Mbps
Incoming bandwidth for ('summary', 'startup_progress')                   :   0.00 Mbps
Incoming bandwidth for ('summary', 'estimated_slot_duration_nanos')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'optimistically_confirmed_slot')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'tiles')                              :   0.00 Mbps
Incoming bandwidth for ('summary', 'completed_slot')                     :   0.00 Mbps
Incoming bandwidth for ('summary', 'root_slot')                          :   0.00 Mbps
Incoming bandwidth for ('summary', 'skip_rate')                          :   0.00 Mbps
Total incoming bandwidth: 16.58 Mbps
Incoming bandwidth for ('summary', 'live_tile_timers')                   :   0.14 Mbps
Incoming bandwidth for ('summary', 'live_txn_waterfall')                 :   0.03 Mbps
Incoming bandwidth for ('summary', 'live_tile_primary_metric')           :   0.01 Mbps
Incoming bandwidth for ('slot', 'update')                                :   0.01 Mbps
Incoming bandwidth for ('summary', 'estimated_tps')                      :   0.00 Mbps
Incoming bandwidth for ('summary', 'optimistically_confirmed_slot')      :   0.00 Mbps
Incoming bandwidth for ('summary', 'estimated_slot_duration_nanos')      :   0.00 Mbps
Total incoming bandwidth: 0.20 Mbps

Note that since small messages are not compressed, the bandwidth after startup is barely changed.

@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 8 times, most recently from 18eaf56 to a2bfb2b Compare June 16, 2025 15:25
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from c47d81c to f564feb Compare June 25, 2025 18:34
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 4 times, most recently from 2cfed3c to c25d68e Compare July 14, 2025 23:21
@jherrera-jump jherrera-jump marked this pull request as draft July 14, 2025 23:29
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from 517d5c9 to f89d914 Compare July 14, 2025 23:38
@jherrera-jump jherrera-jump marked this pull request as ready for review July 14, 2025 23:44
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 3 times, most recently from e81889b to 0b4c3ca Compare July 18, 2025 14:19
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from 871b9d2 to b371aed Compare July 30, 2025 20:25
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from 361cc4f to b7a86ea Compare August 6, 2025 20:03
mmcgee-jump
mmcgee-jump reviewed Aug 8, 2025
View reviewed changes
src/waltz/http/fd_http_server.c Outdated
http->stage_len = 0UL;
#if FD_HAS_ZSTD
uchar * _zstd_ctx = FD_SCRATCH_ALLOC_APPEND( l, 16UL, ZSTD_estimateCCtxSize( FD_HTTP_ZSTD_COMPRESSION_LEVEL ) );
http->zstd_ctx = ZSTD_initStaticCCtx(_zstd_ctx, ZSTD_estimateCCtxSize( FD_HTTP_ZSTD_COMPRESSION_LEVEL ));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NULL on error, and formatting, I'd also move the init stuff down below with the other joins()

Suggested change
http->zstd_ctx = ZSTD_initStaticCCtx(_zstd_ctx, ZSTD_estimateCCtxSize( FD_HTTP_ZSTD_COMPRESSION_LEVEL ));
http->zstd_ctx = ZSTD_initStaticCCtx( _zstd_ctx, ZSTD_estimateCCtxSize( FD_HTTP_ZSTD_COMPRESSION_LEVEL ) );

mmcgee-jump
mmcgee-jump reviewed Aug 8, 2025
View reviewed changes
src/waltz/http/fd_http_server.c Outdated
#if FD_HAS_ZSTD
uchar * _zstd_ctx = FD_SCRATCH_ALLOC_APPEND( l, 16UL, ZSTD_estimateCCtxSize( FD_HTTP_ZSTD_COMPRESSION_LEVEL ) );
http->zstd_ctx = ZSTD_initStaticCCtx(_zstd_ctx, ZSTD_estimateCCtxSize( FD_HTTP_ZSTD_COMPRESSION_LEVEL ));
ulong err = ZSTD_CCtx_setParameter(http->zstd_ctx, 100, FD_HTTP_ZSTD_COMPRESSION_LEVEL);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ulong err = ZSTD_CCtx_setParameter(http->zstd_ctx, 100, FD_HTTP_ZSTD_COMPRESSION_LEVEL);
ulong err = ZSTD_CCtx_setParameter( http->zstd_ctx, 100, FD_HTTP_ZSTD_COMPRESSION_LEVEL );

mmcgee-jump
mmcgee-jump reviewed Aug 8, 2025
View reviewed changes
src/waltz/http/fd_http_server_private.h
int stage_err;
ulong stage_off;
ulong stage_len;

/* The server needs to maintain two copies of the data (one
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alignment

mmcgee-jump
mmcgee-jump requested changes Aug 8, 2025
View reviewed changes
Copy link
Contributor

@mmcgee-jump mmcgee-jump left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from a3ed74d to 3111f6a Compare August 11, 2025 16:13
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from 3794d44 to b234e5e Compare August 13, 2025 15:08
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch from b234e5e to 7b57221 Compare August 20, 2025 14:48
mmcgee-jump
mmcgee-jump reviewed Sep 9, 2025
View reviewed changes
mmcgee-jump
mmcgee-jump requested changes Sep 9, 2025
View reviewed changes
Copy link
Contributor

@mmcgee-jump mmcgee-jump left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch 2 times, most recently from 42eca1e to b45ed3c Compare September 11, 2025 19:03
mmcgee-jump
mmcgee-jump previously approved these changes Sep 11, 2025
View reviewed changes
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch from b45ed3c to 9814be0 Compare September 12, 2025 15:43
@jherrera-jump jherrera-jump changed the title gui: compress ws messages gui: compress ws messages, change frankendancer logo Sep 12, 2025
@jherrera-jump jherrera-jump force-pushed the jherrera/websocket-compression branch from 9814be0 to d746026 Compare September 12, 2025 15:43
@jherrera-jump jherrera-jump added this pull request to the merge queue Sep 12, 2025
Merged via the queue into main with commit f5210fc Sep 12, 2025
11 checks passed
@jherrera-jump jherrera-jump deleted the jherrera/websocket-compression branch September 12, 2025 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mmcgee-jump mmcgee-jump mmcgee-jump approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jherrera-jump @mmcgee-jump