Some questions about the paper

[handsomelys]

How do I understand $E_{Sam}$ and the corresponding $E^{T-I}_{Sam}$ in the paper? Is it constructed using the positional embedding in the transformer like the learnable embedding $E_{Pos}$ etc. mentioned above?

[invictus717]

Exactly. They are just random initialized vanilla positional embeddings.

[invictus717]

These paired embeddings share the same weights to label the corresponding text paired datasets

[handsomelys]

These paired embeddings share the same weights to label the corresponding text paired datasets

Thanks Reply!

[handsomelys]

Thank you very much for your enthusiastic reply. But I still have some questions about pretraining objectives:
Are the two terms in Formula 4 not equivalent? Why?

What is the value of predictions $p_v$ in Formula 5? Is the value of $p_v$ obtained directly through an MLP layer or will it go through an activation function similar to sigmoid?

How is the conditional causal masked mentioned in Formula 6 done? Is it to mask the last 60% of all tokens, and then use BERT to reconstruct the masked tokens in an autoregressive manner ?

[invictus717]

Because they' re two matrix of text and multimodal features. Their dot products are transposed matrix. So for the columns and rows, the summations are different, especially dealing with a huge batch size.

[invictus717]

The match process is provided in our released code:

MiCo/model/mico.py

Line 44 in 89c91c9

class Match_head(nn.Module):

[invictus717]

The causal pretraining process is exactly as you say, which is intuitive and simple.

[invictus717]

If you have any further questions, please feel free to reach out.

[handsomelys]

If you have any further questions, please feel free to reach out.

Thanks again for your reply!

[gongbudaizhe]

Hi, may I ask some questions about the paper?

1. Why Z_I = [Z_I^1, Z_I^2, ... Z_I^LI]? what does LI mean? what is sample length? is it the num frames of a video?
2. What does E_C^I mean? Why we need this term?