feat: support chat datasets with THD, BSHD + CP and padding fixes#1416

Open

hemildesai wants to merge 29 commits intomainfrom

hemil/data-fixes

Commits on Mar 4, 2026

feat: support chat datasets in packed sequence THD collator
hemildesai
committed
test: add missing unit tests for data and CP changes
hemildesai
committed
fix: correct attention_mask for pre-padded input_ids in _package_tokenized_example

hemildesai
and
claude
committed
refactor: remove division-by-zero guards from this PR

hemildesai
and
claude
committed
docs: add comment explaining padding=False in format_chat_template

hemildesai
and
claude
committed
feat: shuffle HF datasets before slicing in chat dataset loader

hemildesai
and
claude
committed
feat: make shuffle seed configurable in ChatDataset

hemildesai
and
claude
committed
refactor: default shuffle_seed to None, set explicitly in yaml

hemildesai
and
claude
committed
fix: pop padding_mask from batch before passing to model forward

hemildesai
and
claude
committed
fix: accept **kwargs in GPT2LMHeadModel.forward for padding_mask

hemildesai
and
claude
committed
fix: stop stripping padding_mask from batch before model forward

hemildesai
and
claude
committed
fix: do not default split to "train" in _load_openai_messages

hemildesai
and
claude
committed
docs: fix comment about pre-padded input_ids source

hemildesai
and
claude
committed
docs: fix incorrect function name in comment

hemildesai
and
claude
committed
fix: do not append EOS to truncated sequences in format_chat_template

hemildesai
and
claude
committed
feat: resolve chat_template from file path in ChatDataset

hemildesai
and
claude
committed
fix: pass actual padding to apply_chat_template and support longest

hemildesai
and
claude
committed
fix: remove dataset-level longest padding resolution

hemildesai
and
claude
committed
fix: remove attention_mask from THD context parallelism path

hemildesai
and
claude
committed
fix: compute attention_mask content_length before next-token shift

hemildesai
and
claude
committed
fix: update functional tests for new padding behavior

hemildesai
and
claude
committed
fix: skip supervised-token assertion when truncation removes assistant

hemildesai
and
claude
committed
fix: remove eos_token_id assertions from pad-eos overlap tests

hemildesai
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support chat datasets with THD, BSHD + CP and padding fixes#1416

feat: support chat datasets with THD, BSHD + CP and padding fixes#1416
hemildesai wants to merge 29 commits intomainfrom
hemil/data-fixes

Commits on Mar 4, 2026

feat: support chat datasets in packed sequence THD collator

test: add missing unit tests for data and CP changes

fix: correct attention_mask for pre-padded input_ids in _package_tokenized_example

refactor: remove division-by-zero guards from this PR

docs: add comment explaining padding=False in format_chat_template

feat: shuffle HF datasets before slicing in chat dataset loader

feat: make shuffle seed configurable in ChatDataset

refactor: default shuffle_seed to None, set explicitly in yaml

fix: pop padding_mask from batch before passing to model forward

fix: accept **kwargs in GPT2LMHeadModel.forward for padding_mask

fix: stop stripping padding_mask from batch before model forward

fix: do not default split to "train" in _load_openai_messages

docs: fix comment about pre-padded input_ids source

docs: fix incorrect function name in comment

fix: do not append EOS to truncated sequences in format_chat_template

feat: resolve chat_template from file path in ChatDataset

fix: pass actual padding to apply_chat_template and support longest

fix: remove dataset-level longest padding resolution

fix: remove attention_mask from THD context parallelism path

fix: compute attention_mask content_length before next-token shift

fix: update functional tests for new padding behavior

fix: skip supervised-token assertion when truncation removes assistant

fix: remove eos_token_id assertions from pad-eos overlap tests

Commits on Mar 6, 2026

test fixes

fix: move pad_seq_len_divisible to dataloader.collate_fn in DCP tests

fix: register pad_seq_len_divisible overrides in conftest

Commits on Mar 9, 2026

Merge remote-tracking branch 'origin/main' into hemil/data-fixes

fix: avoid unnecessary padding when max_len is already divisible

fix: handle multi-part content in _normalize_messages