Messages concatenating on acceptor causes seqnumtoohigh and seqnumtoolow errors which leads huge replays

[parulabrol-osttra]

Describe the bug
Our client (initiator) connects to us via SSL and and sends us a huge influx of messages, during this influx of messages we see that 2 messages are printed concatenated as a single message in messages.log, earlier we thought that this is just a issue where messages are printed as concatenated but we saw that only one message was processed and other was ignored, an instance from message.log:

20250627-07:00:52.28820250627-07:00:52.288: : 8=FIXT.1.1�9=397�35=J�34=18500�49=XXXX�52=20250627-07:00:52.229......8=FIXT.1.1�9=384�35=J�34=18501�49=XXXX�52=20250627-07:00:52.229�56=XXXX�6=13.008�15=XXX�22=2..

I have trimmed messages as these are client specific.

Here we can see two issues:

• Date is concatenated 20250627-07:00:52.28820250627-07:00:52.288
• Messages with sequence id 18500, 18501 are concatenated.

These are the event logs:
20250627-07:00:58.060: MsgSeqNum too high, expecting 18500 but received 18501

because of which huge number of messages are replayed, as messages are pushed from client which are then cached.

The main problem is that we are not able to identify why 2 messages are getting clubbed.

on similar lines issue was reported where a message was duplicated but client sent messages correctly in order:

20250730-09:01:45.449: 8=FIXT.1.1�9=392�35=J�34=90813�49=XXXX10=016� 20250730-09:01:45.449: 8=FIXT.1.1�9=392�35=J�34=90813�49=XXXX1010=016�

This caused a MsgSeqNumTooLow which triggered a logout.

We are suspecting it's either Apache mina or FixMessageDecoder that is causing this issue.

I was trying to find reference on any other ticket mentioning same issue, but couldn't find anything. Any help is deeply appreciated as this has caused huge problems.

To Reproduce
Don't know how to reproduce :(

Expected behaviour
Expected behaviour is that each message should have been processed separately, no message should have been missed.

system information:

• OS: Linux
• Java version JDK8
• QFJ Version1.5.2

[chrjohn]

Hi @parulabrol-osttra , thanks for the detailed bug report. That is a pretty old version that you got there. 14 years old, wow.

I found an issue in the old JIRA which looks like the problem you are seeing: https://www.quickfixj.org/jira/browse/QFJ-544

Apart from that it is hard to help you with such an old version. But maybe you are lucky and QFJ 1.5.3 fixes your problem.
But I can only strongly recommend to upgrade to a recent version since I hope that more bugs were fixed than introduced in 14 years... 😉

[parulabrol-osttra]

Hi @chrjohn ,

Thanks for your reply and the insight.

You are correct; this version is quite old and has been running in its current state for a long time, which makes a simple upgrade to a newer version very challenging.

We actually have a wrapper written around QuickFIXJ in our application where we have already implemented a specific fix to accommodate the issue you mentioned (QFJ-544: https://www.quickfixj.org/jira/browse/QFJ-544).

Despite this internal accommodation, we are still seeing occurrences of the original issue during peak load times. Unfortunately, we have been unable to reliably reproduce this issue in lower environments.

Any thoughts or insights you might have on why this happens under high load would surely help us a lot!

Thanks again,
Parul

[parulabrol-osttra]

Hi @chrjohn ,

Could you please review my previous comment? Addressing this issue is critical as it will resolve a significant production problem and save us substantial effort. Your attention to this would be greatly appreciated.

Regards,
Parul

[chrjohn]

Hi @parulabrol-osttra ,

please understand that is not possible to provide support for such an old version that even has been customized by you ("wrapper written around QFJ...").

I can only suggest the following:

• try to reproduce the problem, maybe it helps to write a stress test for this since it seems the problem is only observed under high load (maybe New jcstress concurrency stress testing module #1054 helps)
• upgrade QFJ (although this is no guarantee that it will solve the problem, however I do not know from others reporting the same problem as you, so there is a good chance that the problem has been fixed in the meantime)
• find someone who will solve this problem for you. But that someone surely has to be paid. ;) I have only been granted limited time for QFJ.

Cheers
Chris