Codalab submission questions

[stijndebeer]

So I submitted two trained models with a test dice of 0.6, and they have gotten a dice of 0.23 and 0.18 on the Codalab server.
Currently, I am quite stumped on how this could happen on the peak performance benchmark.
I calculate the dice as the mean for all the classes, and attached below is the Codalab prediction

One of the models uses the original TA-baseline normalization, and the second model uses imagenet normalization.
Both models are trained using a combined CE+DICE loss

Could someone enlighten me on what I might have missed?

[TimJaspers0801]

Mmmmmh, strange. I have checked your submission, and everything seems fine. However, I do see that the model only works "okay" on a few classes. could you send me the code snipped from your data loading in the training and evaluation loader?

[stijndebeer]

I used the same code as given at the beginning of the course.
This is the result that I get for my classes on the test set
mean_dice_score=[0.90, 0.78, 0.89, 0.42, 0.44, 0.54, 0.57, 0.70, 0.92, 0.65, 0.92, 0.71, 0.37, 0.91, 0.09, 0.27, 0.04, 0.32, 0.50], overall_mean_dice=0.58

# Load datasets
train_dataset = Cityscapes(
    args.data_dir, 
    split="train", 
    mode="fine", 
    target_type="semantic", 
    transforms=final_transform
)
valid_dataset = Cityscapes(
    args.data_dir, 
    split="val", 
    mode="fine", 
    target_type="semantic", 
    transforms=transform  # No augmentation for validation
)

train_dataset = wrap_dataset_for_transforms_v2(train_dataset)
valid_dataset = wrap_dataset_for_transforms_v2(valid_dataset)

train_dataloader = DataLoader(
    train_dataset, 
    batch_size=args.batch_size, 
    shuffle=True,
    num_workers=args.num_workers
)
valid_dataloader = DataLoader(
    valid_dataset, 
    batch_size=args.batch_size, 
    shuffle=False,
    num_workers=args.num_workers
)

[stijndebeer]

train_transform = Compose([
    ToImage(),
    RandomCrop((256, 256), pad_if_needed=True),
    RandomHorizontalFlip(p=0.5),
    RandomRotation(degrees=10),
    RandomApply([
        # ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05),
        RandomPerspective(distortion_scale=0.1, p=0.5),
        GaussianBlur(kernel_size=(3, 3), sigma=(0.1, 2.0)),
    ], p=0.5),
    ToDtype(torch.float32, scale=True),
    Normalize(mean=mean, std=std),
])

# Define transforms for validation (no augmentations)
transform = Compose([
    ToImage(),
    Resize((256, 256)),
    ToDtype(torch.float32, scale=True),
    Normalize(mean=mean,std=std),
])

[TimJaspers0801]

I see in the process_data.py file (preprocess() function) you submitted in your last submission that you still resize the images to 512x512 instead of 256,256 (the resolution you used to train your model. Could you try submitting it once with this adjustment? Don't worry about the maximum number of submissions. If this is the issue, I will just delete your older submissions.

[TimJaspers0801]

still not good I see... Did you manage to train the original Unet model?

[stijndebeer]

yess, i have those weights, but not specifically the test dice

[TimJaspers0801]

Could you share the train.py file with me? just send it on teams