Skip to content

Fix 32-bit overflow in byteList size calc in LZW_GenerateStream#118

Open
rootvector2 wants to merge 1 commit into
dloebl:mainfrom
rootvector2:fix-bytelist-overflow-raw
Open

Fix 32-bit overflow in byteList size calc in LZW_GenerateStream#118
rootvector2 wants to merge 1 commit into
dloebl:mainfrom
rootvector2:fix-bytelist-overflow-raw

Conversation

@rootvector2
Copy link
Copy Markdown
Contributor

Bug

In LZW_GenerateStream (src/cgif_raw.c:352-353) the two byte-buffer
size calculations evaluate MAX_CODE_LEN * lzwPos in 32-bit
unsigned arithmetic before extending into a uint64_t:

uint64_t MaxByteListLen = MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull;
uint64_t MaxByteListBlockLen = MAX_CODE_LEN * lzwPos * (BLOCK_SIZE + 1ull) / 8ull / BLOCK_SIZE + 2ull + 1ull +1ull;

MAX_CODE_LEN is the int literal 12 and lzwPos is uint32_t, so
the intermediate product is computed in uint32_t. The / 8ull
(and the later * (BLOCK_SIZE + 1ull) on the second line) only
extends the result after the multiplication, so the cast comes too
late. Once lzwPos > UINT32_MAX / 12 (~358M), the multiplication
silently wraps mod 2^32.

The maximum reachable lzwPos is roughly
numPixel + numPixel/(MAX_DICT_LEN-initDictLen-2) + 2, and numPixel
comes from MULU16(width, height) where both are uint16_t. A frame
of 65535 * 65535 pixels (the max the raw API accepts) drives
lzwPos to about 4.295e9.

For that frame:

expression buggy value actual value
MaxByteListLen ~536 MB ~6.4 GB
MaxByteListBlockLen ~539 MB ~6.47 GB

byteList is then malloc'd to the truncated size, while
create_byte_list / create_byte_list_block proceed to write up to
the real (un-truncated) byte count into it - a multi-GB heap buffer
overflow on the byteList allocation. This is the same shape of bug
that #103 fixed for pLZWData, just on the very next two
allocations.

Reproduce / observe

A minimal reproducer of just the arithmetic, built with
-fsanitize=unsigned-integer-overflow:

#include <stdio.h>
#include <stdint.h>
#define MAX_CODE_LEN 12
int main(void) {
    uint32_t lzwPos = 400000000u;
    uint64_t v = MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull;
    printf(\"%llu\n\", (unsigned long long)v);
}
runtime error: unsigned integer overflow: 12 * 400000000 cannot be
represented in type 'uint32_t' (aka 'unsigned int')
63129091

Whereas the correct upper bound for that lzwPos is 600000003.

End-to-end the bug is reachable from cgif_raw_addframe with a
single frame whose width * height is large enough to push lzwPos
above ~358M. The raw API caps width/height at uint16_t but does
not cap width * height, so a single 18000+ x 18000+ frame is
enough to trigger it on a host with sufficient memory for the input
buffers.

Fix

Cast MAX_CODE_LEN to uint64_t so the whole expression is
evaluated in 64-bit and the multiplication can no longer wrap. The
change is local to LZW_GenerateStream and matches the
(size_t)numPixel cast that #103 added to the pLZWData allocation
two lines above.

-  uint64_t MaxByteListLen = MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull;
-  uint64_t MaxByteListBlockLen = MAX_CODE_LEN * lzwPos * (BLOCK_SIZE + 1ull) / 8ull / BLOCK_SIZE + 2ull + 1ull +1ull;
+  uint64_t MaxByteListLen = (uint64_t)MAX_CODE_LEN * lzwPos / 8ull + 2ull + 1ull;
+  uint64_t MaxByteListBlockLen = (uint64_t)MAX_CODE_LEN * lzwPos * (BLOCK_SIZE + 1ull) / 8ull / BLOCK_SIZE + 2ull + 1ull +1ull;

Build / test

Built locally with meson + clang and ran the full existing test
suite (meson test -C builddir): all 60 tests pass. The minimal
reproducer no longer trips
-fsanitize=unsigned-integer-overflow after the patch, and the
computed values match the un-truncated 64-bit results.

MAX_CODE_LEN is an int literal (12) and lzwPos is uint32_t, so the
intermediate product MAX_CODE_LEN * lzwPos was evaluated in 32-bit
unsigned arithmetic before being divided into a uint64_t. For lzwPos
above UINT32_MAX/12 (~358M) the multiplication wraps modulo 2^32, so
MaxByteListLen / MaxByteListBlockLen end up much smaller than the
actual number of bytes create_byte_list / create_byte_list_block can
write into the buffer (heap overflow on the byteList malloc).

For a max-size frame (65535 * 65535 pixels) the buggy value is about
536 MB whereas create_byte_list can write up to ~6.4 GB. Cast
MAX_CODE_LEN to uint64_t so the entire expression is evaluated in
64-bit, matching the existing fix from dloebl#103 for pLZWData.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant