Assertion failed: rd_slice_abs_offset

[marghidanu]

Description

While reading data from a consumer and writing to a sink I get the following error:

Assertion failed: rd_slice_abs_offset(new_slice) <= new_slice->end (rdbuf.c: rd_slice_narrow_copy: 1027)

No additional information is provided. One thing I've noticed is that the message is not always the last one. I've upgraded the client and librdkafka to 1.2.0 and this is still occurring.

Thank you

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
• Apache Kafka broker version:
• Client configuration: {...}
• Operating system:
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[edenhill]

Can you reproduce this with 'debug': protocol,msg,fetch,broker`' and provide the logs?

Please also fill out the issue checklist (consumer config, broker version)

[marghidanu]

Thank you for the quick response! Below are the requested logs, the brokers (3) are all running Kafka 2.3.0.

%7|1570700466.709|PROTOUFLOW|rdkafka#consumer-1| [thrd:10.133.33.153:9092/bootstrap]: 10.133.33.153:9092/1: Protocol read buffer underflow at 16777700/16777736 (rd_kafka_msgset_reader_v2:1028): expected 225 bytes > 36 remaining bytes (truncated response from broker (ok)): my_topic [45] MessageSet at offset 1400062 payload size 225                                                                                                                       
%7|1570700466.798|CONSUME|rdkafka#consumer-1| [thrd:10.133.33.153:9092/bootstrap]: 10.133.33.153:9092/1: Enqueue 8323 message(s) (622215 bytes, 8323 ops) on my_topic [45] fetch queue (qlen 8323, v6, last_offset 14000 61, 0 ctrl msgs)                                                                                                                                                                                                                           │
%7|1570700466.798|FETCH|rdkafka#consumer-1| [thrd:10.133.33.153:9092/bootstrap]: 10.133.33.153:9092/1: Topic my_topic [48] MessageSet size 1048576, error "Success", MaxOffset 948711, LSO 948711, Ver 6/6               
Assertion failed: rd_slice_abs_offset(new_slice) <= new_slice->end (rdbuf.c: rd_slice_narrow_copy: 1027)

[edenhill]

Thank you.

It would be very useful to know where exactly this is happening in the code, could you try the following?

$ ulimit -c unlimited
$ <reproduce the issue>
$ gdb $(/usr/bin/env python) core   # on osx the core file will be in /cores/core.<pid>
gdb) bt
gdb) exit

and provide us with the output?

[marghidanu]

This originally was running in a Docker container in our K8s cluster. I created a VM and ran it in there with the same parameters. The output for gdb is:

(gdb) bt
#0  0x00007f79af2227a1 in abort () from /lib/ld-musl-x86_64.so.1
#1  0x0000558b0b03f7e0 in ?? ()
#2  0x0000000000000020 in ?? ()
#3  0x0000000000000000 in ?? ()

[edenhill]

Okay, that backtrace isn't very helpful unfortunately :(

Another way that might work is this:

$ gdb $(/usr/bin/env python)
gdb) run /path/to/your/script.py and -its arguments
# wait for crash
gdb) bt full

And if this method works please stay in gdb for me to instruct on more commands.

[marghidanu]

Unfortunately, it looks pretty much the same:

(gdb) bt full
#0  0x00007f5fa9f217a1 in abort () from /lib/ld-musl-x86_64.so.1
No symbol table info available.
#1  0x000056447b8607e0 in ?? ()
No symbol table info available.
#2  0x0000000000000020 in ?? ()
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.

[edenhill]

Ouch, okay, too bad :(

Can you provide your full consumer configuration?

Do you know if the topic transactional messages on it? (from e.g., kafka streams, ksql)

[marghidanu]

I think I got it fixed by adding the api.version.request and broker.version.fallback to the Consumer arguments.

args = {'bootstrap.servers': self.brokers,
            'group.id': self.group_id,
            'enable.auto.commit': False,
            'debug': 'protocol,msg,fetch,broker',
            'api.version.request': False,
            'broker.version.fallback': '0.9.0.0'}

There's nobody else doing transactional messages on the topic. What's worth mentioning is that we are actually consuming data from two topics.

[marghidanu]

Well it actually stopped crashing but there's a more visible error now:

KafkaError{code=_BAD_MSG,val=-199,str="Local: Bad message format"}

[edenhill]

What versions of librdkafka have you tried?

[marghidanu]

So far only these two: 1.1.0 and 1.2.0
We are reading messages in batch, 1000 at the time. Could this be related to this? Maybe the buffer is not large enough?

[edenhill]

This is in the FetchResponse parsing code, it is executed prior to the application asking for the messages, so not likely to be related.

Are you building librdkafka separately or using the binary wheels of the confluent python client?

[edenhill]

Do you know if these messages are compressed in the topic?

[marghidanu]

We're using the librdkafka package from Alpine and the messages are not compressed.

[edenhill]

Would you be ok with building your own version of librdkafka, to get some debug symbols?

$ apk del librdkafka librdkafka-dev
$ apk add gcc make openssl-dev zlib-dev git  # and possibly others
$
$ git clone https://github.com/edenhill/librdkafka
$ cd librdkafka
$ ./configure --disable-optimization --enable-devel --prefix=/usr
$ make
$ sudo make install
$
$ gdb $(/usr/bin/env python)
gdb) run /your/py/script.py
# wait for crash
gdb) bt full

at this point keep the terminal open as I might want to extract some object fields.