Hi,
I've been implementing the method from your STONE and S-KEY papers and training it from scratch on my own corpus, as a personal project on a single GPU at home. The released code and weights made that possible, and I can confirm the checkpoint reproduces 72.1 MIREX on the canonical 604 GiantSteps tracks exactly through your inference path.
My own runs land well below the published 60k figure, and I've been working through my setup to find what I have wrong. I've ruled out what I can check against the release: the architecture loads with strict=True, my CPSD terms match loss_key numerically on random inputs, the ν pseudo-label and the pitch-class profile compute identically, and the front-end is frozen in both. My corpus is different and not comparable to yours, so some gap is expected, but not this much. Five things I could not determine from either paper or this repo:
1. Where in each track do segments A and B come from? STONE §4.1 says two disjoint 15-second segments per song, which is what I do, but not which part of the track. I take the first 30 seconds and split it in half, which I now think is a poor choice given how often intros are harmonically unrepresentative. Was it the opening, the middle, or a random position? Relatedly, does get_datasets.step_percent = 0.5 describe the spacing between the two segment starts?
2. What was --train-steps for the 60k model? main.py documents it as steps per epoch with a default of 512, and the released checkpoint's optimiser state shows 25,073 steps at epoch 98, so 256. I had been assuming an epoch meant a full pass over the corpus, which for me is 469 steps. I have since tried both 12,663 and 23,450 total steps and got roughly the same result either way, so this may not be my problem, but I would like to know what the published run actually used.
3. Loss weights. §II-F gives λ_BCE = 1.5 and λ_avg = 15 without distinguishing the two training scales, while the released checkpoint's gin_info records weights = [1, 1.7, 1], which becomes 2.55 after the ×1.5 in skey_loss.py, and 20 for the distribution term. Which pair did the 60k model use?
4. How long was the linear warm-up? The paper gives the schedule but not its length. I have been using 5% of total steps.
5. How often does the fifth-shift local minimum occur? §III-B mentions the model sometimes reaching it. Roughly what fraction of runs, and did you train more than once per configuration? I see large seed-to-seed variation and cannot tell whether that is inherent to the objective or something I have introduced.
Any subset of these would help. Happy to report back whatever I work out.
Hi,
I've been implementing the method from your STONE and S-KEY papers and training it from scratch on my own corpus, as a personal project on a single GPU at home. The released code and weights made that possible, and I can confirm the checkpoint reproduces 72.1 MIREX on the canonical 604 GiantSteps tracks exactly through your inference path.
My own runs land well below the published 60k figure, and I've been working through my setup to find what I have wrong. I've ruled out what I can check against the release: the architecture loads with
strict=True, my CPSD terms matchloss_keynumerically on random inputs, the ν pseudo-label and the pitch-class profile compute identically, and the front-end is frozen in both. My corpus is different and not comparable to yours, so some gap is expected, but not this much. Five things I could not determine from either paper or this repo:1. Where in each track do segments A and B come from? STONE §4.1 says two disjoint 15-second segments per song, which is what I do, but not which part of the track. I take the first 30 seconds and split it in half, which I now think is a poor choice given how often intros are harmonically unrepresentative. Was it the opening, the middle, or a random position? Relatedly, does
get_datasets.step_percent = 0.5describe the spacing between the two segment starts?2. What was
--train-stepsfor the 60k model?main.pydocuments it as steps per epoch with a default of 512, and the released checkpoint's optimiser state shows 25,073 steps at epoch 98, so 256. I had been assuming an epoch meant a full pass over the corpus, which for me is 469 steps. I have since tried both 12,663 and 23,450 total steps and got roughly the same result either way, so this may not be my problem, but I would like to know what the published run actually used.3. Loss weights. §II-F gives λ_BCE = 1.5 and λ_avg = 15 without distinguishing the two training scales, while the released checkpoint's
gin_inforecordsweights = [1, 1.7, 1], which becomes 2.55 after the ×1.5 inskey_loss.py, and 20 for the distribution term. Which pair did the 60k model use?4. How long was the linear warm-up? The paper gives the schedule but not its length. I have been using 5% of total steps.
5. How often does the fifth-shift local minimum occur? §III-B mentions the model sometimes reaching it. Roughly what fraction of runs, and did you train more than once per configuration? I see large seed-to-seed variation and cannot tell whether that is inherent to the objective or something I have introduced.
Any subset of these would help. Happy to report back whatever I work out.