Quality/performance of the alignment tools

[pfriesch]

How is the is the quality of the alignements of the different methods? Or how would one even assess the performance of the produced alignments?

[pettarin]

On how to assess the quality of the alignment: the scientific literature is full of articles with different methods. A popular method consists in averaging "expert" judgments over a test corpus. A more scientific approach consists in manually labeling a "gold standard corpus" (or you take an existing, crowdsourced corpus like the manually-crafted closed captions of YouTube video...) and you compute the RMS error of the alignment found by your forced aligner. Another method uses WER scores. But again, entire academic careers have been spent on the subject...

On the usefulness of having a column in the current table for that bit of information, I would say I mostly agree, but I also fear that it will spark heated discussions if we only add "good" or "bad" or "great" possible values to it. On the other hand, setting up an objective measure procedure is not trivial.

[pfriesch]

Thanks for the answer! I am not working on the topic anymore, but at the time, I think I was also interested on the anecdotal performance you might have experienced. You can close this issue then, if you want, since there won't be a solution anytime soon :)

(Just an Idea I just had: Another way to measure the performance could be to train a GMM-HMM on the output since that kind of model should be sensitive to alignment issues. But you are right, that could be a whole paper.)