Add MCCP4 support - New protocol created using zstd with substantial performance improvements. #98

Coffee-Nerd · 2025-08-19T04:29:46Z

Implements MCCP4 compression using Zstandard library as an alternative to existing MCCP2 (zlib).
Provides better compression ratios and performance while maintaining full backward compatibility.

Also moves the original timing for MCCP2 to not include WriteToLog or DisplayMsg in the timing, since those are i/o related and processing related and not related to compression timing.

Let me know if you have any questions, comments, concerns, or ideas.

Cheers.

Implemented support for MCCP4 (Zstandard) compression in the Phase_WILL function, allowing the client to negotiate this compression method. Updated logic to prevent agreeing to multiple compression methods simultaneously and added handling in the Phase_SUBNEGOTIATION_IAC function.

Refactor the compression logic to differentiate between MCCP1/MCCP2 (zlib) and MCCP4 (Zstandard). Implement direct processing for Zstandard compressed data and ensure proper cleanup of resources. Update statistics and error handling for both compression types.

Implement the core functionality for MCCP4 compression, including initialization, cleanup, and processing of Zstandard compressed data. This addition enhances the compression capabilities of MUSHclient, providing improved performance and better compression ratios compared to previous methods.

…9, to follow the standard progression (mccp = 85, mccp2 = 86, mccp3 = 87, mccp4 = 88)

…(Zstandard) support.

… and update statistics for MCCP4 (Zstandard).

This commit introduces a comprehensive set of new files for Zstandard compression support, including error handling, memory management, and decompression logic. Key components include zstd_errors.h, zstd.h, and various common utilities for efficient data processing. This enhancement lays the groundwork for improved compression capabilities using zstd.

This update improves the handling of decompressed data by ensuring accurate timing and memory management. It introduces a flag to indicate the end of a Zstandard frame, allowing for better processing of remaining raw Telnet bytes.

… function This update moves the timing measurement to occur after decompression but before I/O and display processing, ensuring more accurate tracking of compression time taken.

fiendish · 2025-08-19T13:49:29Z

Not reviewed yet, but exploratory EXEs are available at https://github.com/Coffee-Nerd/mushclient/releases

fiendish · 2025-08-21T01:45:35Z

please include the license file(s) for zstd.

Also include a readme for the zstd directory explaining exactly where it came from, what version it is, and how to update it.

fiendish · 2025-08-20T23:58:28Z

doc.cpp

+    // End timing here - after decompression but before I/O and display processing
+    if (App.m_iCounterFrequency)
+      {
+      QueryPerformanceCounter (&finish);
+      m_iCompressionTimeTaken += finish.QuadPart - start.QuadPart;
+      }
+


What's the point of moving this down from before the preceding error condition?

You're right. This should be resolved with this
aae74b7

An additional timing stop has been added in case of error as well.

fiendish · 2025-08-21T00:21:03Z

mccp4.cpp

+        // Cleanup on allocation failure
+        if (m_zstd_inbuf) free(m_zstd_inbuf);
+        if (m_zstd_outbuf) free(m_zstd_outbuf);
+        ZSTD_freeDStream((ZSTD_DStream*)m_zstd_dstream);
+        m_zstd_dstream = NULL;
+        m_zstd_inbuf = NULL;
+        m_zstd_outbuf = NULL;
+        m_zstd_incap = 0;
+        m_zstd_outcap = 0;


this looks redundant with CleanupZstd below

Very true. A call to CleanupZstd should be fine.

This should be resolved with 92fcfd8

fiendish · 2025-08-21T02:13:54Z

MUSHclient.vcxproj

not without reducing this file down to a minimal change

Wait what? There were only a few line changes when I committed this.. I need to look at what happened.

These are the only real line changes I made. Hmm..

This should be fixed with
6d5d62c

The line endings caused the issue with the major diff. It has also now been alphabetically sorted as requested, and the zstd header file included.

fiendish

vcxproj diff needs to be trimmed way down.

It can be done manually from the original in a text editor.

Add mccp4.cpp source file (alphabetically after mcdatetime.cpp)
Add zstd library source files (after existing zlib files)
Add zstd.h header file (alphabetically with other headers)

I also think you don't need all those zstd files, only some of them.

Fixed CRLF line endings in vcxproj file Removed unnecessary zstd files (pool.c, threading.c) Added only essential zstd decompression files Properly alphabetized file entries as requested

…taken, including stopping timing for early returns.

…Zstd function and improve timing accuracy during error scenarios.

Coffee-Nerd · 2025-08-21T03:57:23Z

please include the license file(s) for zstd.

Also include a readme for the zstd directory explaining exactly where it came from, what version it is, and how to update it.

These are now included here:

53aa15d

0ead7d2

Coffee-Nerd · 2025-08-21T04:05:00Z

vcxproj diff needs to be trimmed way down.

It can be done manually from the original in a text editor.
* Add mccp4.cpp source file (alphabetically after mcdatetime.cpp)

* Add zstd library source files (after existing zlib files)

* Add zstd.h header file (alphabetically with other headers)
I also think you don't need all those zstd files, only some of them.

pool.c/.h and threading.c/.h were also removed, seen here:

2e63f32

…F-8 (with BOM) text, with CRLF line terminators

… Git issue

… preventing this

Coffee-Nerd · 2025-08-28T10:29:33Z

Update here: we're waiting on the gitattributes file to be merged so we can push the vcxproj file without the changed line-endings.

… when compression is disabled and improve comments for clarity.

…rototype

…y payloads, support multiple encodings, and improve error handling for Zstandard and deflate initialization. Update comments for better clarity.

…file as intended on push.

fiendish · 2025-09-13T02:38:19Z

No objections here in principle, but do you have a document describing the protocol and a test mechanism, and is mudlet implementing it?

Coffee-Nerd · 2025-09-13T14:33:55Z

No objections here in principle, but do you have a document describing the protocol and a test mechanism, and is mudlet implementing it?

Here are one of the latest drafts on MCCP4 (named MCCPX in draft, but it's MCCP4)

https://www.last-outpost.com/LO/protocols/mccpx/draft-rahjiii-MCCPX-04.md

Can also be found here:

https://mudstandards.org/mud/mccpX_draft/

MCCP4 Mudlet implementation can be found here, but might be potentially not working as well as our implementation from what I've heard. They may have remedied that by now.

Mudlet MCCP4 PR

Here is a test server I made with someone:

MCCP4 Test Server

You can also test MCCP4 (and many other things) against our client capabilities test server, found at test.mudvault.org 5050

The list of clients who have already implemented MCCP4 now are Lociterm, Mudlet, Crystal, MudForge (and MUSHclient, after this, which covers a HUGE percentage of all client users)

If you want to test the client with a MUD, you can connect via darkwiz.org 6969 and play with the release .exe in the fork.

Coffee-Nerd · 2025-10-02T12:53:11Z

Bump on this.

Coffee-Nerd added 13 commits August 16, 2025 14:09

Add support for MCCP v4 (Zstandard) compression

00f1585

Initialize variables for MCCP v4 support in doc_construct.cpp

8e787d3

Add Zstandard header for MCCP4 support in stdafx.h

6fdc5ee

Add MCCP v4 (Zstandard) label to compression ratio in LoadPrefsP15

e142a97

After consideration I changed the telopt_compress4 to 88 instead of 8…

ad0af77

…9, to follow the standard progression (mccp = 85, mccp2 = 86, mccp3 = 87, mccp4 = 88)

Update MUSHclient project file to include new source files for MCCP4 …

ec2b4e6

…(Zstandard) support.

Enhance MCCP compression logging to include specific compression type…

cd89f30

… and update statistics for MCCP4 (Zstandard).

Improve timing measurement for decompression in ProcessZstdCompressed…

f850648

… function This update moves the timing measurement to occur after decompression but before I/O and display processing, ensuring more accurate tracking of compression time taken.

fiendish reviewed Aug 21, 2025

View reviewed changes

fiendish self-requested a review August 21, 2025 02:15

fiendish requested changes Aug 21, 2025

View reviewed changes

Coffee-Nerd added 5 commits August 21, 2025 13:06

Fix vcxproj line endings and use minimal zstd file set

6d5d62c

Fixed CRLF line endings in vcxproj file Removed unnecessary zstd files (pool.c, threading.c) Added only essential zstd decompression files Properly alphabetized file entries as requested

Adjust the timing logic for accurate measurement of compression time …

aae74b7

…taken, including stopping timing for early returns.

Include Zstd license

0ead7d2

Add an update guide for Zstd

53aa15d

Refactor error handling in Zstandard decompression to utilize Cleanup…

92fcfd8

…Zstd function and improve timing accuracy during error scenarios.

Remove unused zstd files to keep dependency minimal.

2e63f32

Coffee-Nerd added 3 commits August 21, 2025 14:12

Remove original zstd include positions..

8684fa7

Remove UTF-8 BOM from vcxproj to clean up diff

a4c77e2

Fix encoding again

cf3a28c

Coffee-Nerd added 6 commits August 21, 2025 22:17

Delete accidental push of converted file.

f037c42

Fix encoding MUSHclient.vcxproj to XML 1.0 document, Unicode text, UT…

80ddc4e

…F-8 (with BOM) text, with CRLF line terminators

Normalize vcxproj to UTF-8 BOM with CRLF

8a0c32c

Normalize MUSHclient.vcxproj with appropriate CRLF line terminators -…

034792b

… Git issue

Normalize text with appropriate line endings: Git config autocrlf was…

0bf488c

… preventing this

Final attempt to force CRLF line terminators

c3bdff8

fiendish marked this pull request as draft August 31, 2025 04:42

Enhance MCCP4 Zstandard handling: process remaining uncompressed data…

c59e2fc

… when compression is disabled and improve comments for clarity.

Coffee-Nerd marked this pull request as ready for review August 31, 2025 06:54

Coffee-Nerd added 3 commits August 31, 2025 22:34

Add mccp4 subnegotiation constants, negotiation state, and function p…

c21f0d5

…rototype

Enhance MCCP4 compression handling: implement fallback mode for empt…

27dc6d5

…y payloads, support multiple encodings, and improve error handling for Zstandard and deflate initialization. Update comments for better clarity.

Remove fake double mccp4.c to get the new gitattributes to save the …

83b924f

…file as intended on push.

Coffee-Nerd force-pushed the master branch from 162218b to 83b924f Compare August 31, 2025 12:37

Fix line endings for doc.h to CRLF

c7a9ff7

Add MCCP4 support - New protocol created using zstd with substantial performance improvements. #98

Are you sure you want to change the base?

Add MCCP4 support - New protocol created using zstd with substantial performance improvements. #98

Uh oh!

Conversation

Coffee-Nerd commented Aug 19, 2025

Uh oh!

fiendish commented Aug 19, 2025

Uh oh!

fiendish commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fiendish left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Coffee-Nerd commented Aug 21, 2025

Uh oh!

Coffee-Nerd commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Coffee-Nerd commented Aug 28, 2025

Uh oh!

fiendish commented Sep 13, 2025

Uh oh!

Coffee-Nerd commented Sep 13, 2025

Uh oh!

Coffee-Nerd commented Oct 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fiendish commented Aug 21, 2025 •

edited

Loading

fiendish left a comment •

edited

Loading

Coffee-Nerd commented Aug 21, 2025 •

edited

Loading