Not all models files are being written when all nodes have learnable embeddings

[thvasilo]

When training an LP model where all nodes have learnable embeddings (no features) on SageMaker, it seems like model files are not being saved for every epoch, although topk-model-to-save is left at its default value (inf)

Example output uploaded to S3:

    ── epoch-13
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── Movie
    │   ├── Person
    │   └── User
    ├── epoch-14
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── model.bin
    │   ├── Movie
    │   ├── optimizers.bin
    │   ├── Person
    │   └── User
    ├── epoch-15
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── Movie
    │   ├── Person
    │   └── User
    ├── epoch-16
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── Movie
    │   ├── Person
    │   └── User
    ├── epoch-17
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── model.bin
    │   ├── Movie
    │   ├── optimizers.bin
    │   ├── Person
    │   └── User
    ├── epoch-18
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── model.bin
    │   ├── Movie
    │   ├── optimizers.bin
    │   ├── Person
    │   └── User
    ├── epoch-19
    │   ├── Actor
    │   ├── Director
    │   ├── Genre
    │   ├── Movie
    │   ├── Person
    │   └── User

[zhjwy9343]

Hi, could you provide brief steps of reproducing this issue?

[thvasilo]

I've tried to replicate this with a few configs on SageMaker but have not been successful. I'll close this and re-open if I encounter this again with a good way to replicate.