[QEff Finetune] Adding dataset padding changes #478
Conversation
quic-swatia
requested review from
quic-rishinr,
ochougul,
quic-hemagnih and
quic-amitraj
as code owners
Signed-off-by: Swati Allabadi <[email protected]>
quic-swatia
force-pushed
the
dataset_padding
branch
from
b8104e1 to
adebe02
Compare
| @@ -64,19 +64,35 @@ def get_dataloader_kwargs(train_config, dataset, dataset_processer, split): | |||
|
|
|||
| def get_dataloader(tokenizer, dataset_config, train_config, split: str = "train"): | |||
| dataset = get_preprocessed_dataset(tokenizer, dataset_config, split, context_length=train_config.context_length) | |||
| dl_kwargs = get_dataloader_kwargs(train_config, dataset, tokenizer, split) | |||
| dataset = dataset.select(range(0, 10)) | |||
There was a problem hiding this comment.
Why slicing dataset to pick first 10 samples?
There was a problem hiding this comment.
This was added for local experiments. Removed in the PR.
| dl_kwargs = get_dataloader_kwargs(train_config, dataset, tokenizer, split) | ||
| dataset = dataset.select(range(0, 10)) | ||
| dataset = dataset.map(lambda x: {"input_length": len(x["input_ids"])}) | ||
| dataset = dataset.sort("input_length") |
There was a problem hiding this comment.
Done the sorting here for the non padded dataset in place of being done in sampler.py for the padded dataset to keep the dummy samples in the end.
| dummy_row["labels"] = [-100] * len(dummy_row["labels"]) | ||
| padding_size = 0 | ||
| num_replicas = dist.get_world_size() | ||
| if len(dataset) % num_replicas > 0: |
There was a problem hiding this comment.
bs>1 is not considered here.
There was a problem hiding this comment.
Had skipped it since we are not supporting bs > 1 as of now. Made this change for sake of completion.
| if len(dataset) % num_replicas > 0: | ||
| padding_size = num_replicas - len(dataset) % num_replicas | ||
|
|
||
| dummy_data = [dummy_row.copy() for _ in range(padding_size)] |
There was a problem hiding this comment.
L78 to L80 can be refactored.
There was a problem hiding this comment.
Found this way cleaner. Please suggest if you have anything better idea.
| @@ -192,6 +192,9 @@ def train( | |||
| ) as verifier: | |||
| model_outputs = model(**batch) | |||
| loss = model_outputs.loss # Forward call | |||
| if (batch["labels"] != -100).sum() == 0: | |||
| loss = loss.nan_to_num(nan=0.0) | |||
There was a problem hiding this comment.
This loss is zeroed for dummy samples. But the total loss is averaged across all samples including dummy samples. Correct it.
| @@ -237,6 +242,9 @@ def train( | |||
| step_metric_val = float(torch.exp(loss.detach().float())) | |||
| train_step_metric.append(step_metric_val) | |||
|
|
|||
| # Accumalate gradients | |||
| loss = loss / train_config.gradient_accumulation_steps | |||
There was a problem hiding this comment.
This should change.
E.g. 100 samples, 30 global bs.
For first 30 samples, loss = loss / 30
For next 30 samples, loss = loss / 30
For next 30 samples, loss = loss / 30
For last 10 samples, loss = loss / 10 not loss = loss / 30
| @@ -439,6 +447,9 @@ def evaluation_helper(model, train_config, eval_dataloader, device): | |||
| outputs = model(**batch) | |||
| loss = outputs.loss | |||
|
|
|||
| if (batch["labels"] != -100).sum() == 0: | |||
| loss = loss.nan_to_num(nan=0.0) | |||
There was a problem hiding this comment.
Same comment as above
|
|
||
| dummy_data = [dummy_row.copy() for _ in range(padding_size)] | ||
| dummy_dataset = datasets.Dataset.from_list(dummy_data) | ||
| combined_dataset = datasets.concatenate_datasets([dataset, dummy_dataset]) |
There was a problem hiding this comment.
try to enclose this padding logic in separate function.
Signed-off-by: Swati Allabadi <[email protected]>
No description provided.