special token "[semantic]"

[GinnyXiao]

For training your newly released checkpoint supporting semantic level segmentation, you added a special token "[semantic]" before the input prompt in the training data. I was wondering what difference this token makes? You have not modified your model architecture, which gives 1-1 mapping between your prediction and your prompt. The training strategy for referring seg datasets and semantics seg datasets remain the same, except for this token. Why would adding a "[semantic]" token improve your model ability? Would training without it yield similar performance?

Thank you very much in advance!

[CoderZhangYx]

When I apply semantic-segmentation datasets like ade20k, I add '[semantic]' because the category name refers to all the instances within the image in those datasets. This is in contrast to original referring datasets where the prompt only refers to a single instance. I apply the special token to balance the conflict, making the model behave better during joint training. By employing semantic segmentation datasets, the model is able to segment multiple objects, and also stuff categories if prompted with '[semantic]'

[GinnyXiao]

Thank you for the quick reply! By "behaving better" I wonder if the authors have conducted an ablation study showing adding such a special token boosts performance. If prompted only the category name (and trained in such way using a semantic segmentation dataset), the model won't perform as well as the current strategy does?

[CoderZhangYx]

In our current arxiv paper, we present comparisons in tab.7, showing that adding semantic data without '[semantic]' token would lead to a decrease in several RefCOCO metrics. In tab.2 we show that all metrics rise with extra data + "[semantic]" token.
the figure below may help better grasp the semantic conflict problem.