Bug: YouTube transcript ingestion hits 429 (Too Many Requests)

[juananpe]

Summary

YouTube timedtext (subtitles) requests during transcript ingestion sometimes return 429 Too Many Requests because the ingest plugin performs rapid sequential fetches with no retry/backoff and no configurable inter-request delay.

Steps to reproduce

1. Submit multiple videos for transcript ingestion in rapid succession (e.g., 10–20 videos with ~5–15 seconds between submissions).
2. Observe ingestion jobs failing with 429 Client Error: Too Many Requests for url: https://www.youtube.com/api/timedtext?....
3. Confirm subtitle download path uses yt-dlp to find subtitles/automatic_captions and then requests.get(subtitle_url) to fetch the VTT.

I think that we just need to ingest videos using some kind of queue, one by one, with a backoff mechanism (otherwise we will trigger the 429 error)

[LucasImpulse]

Hey, I've had my go at it, I've changed two blocks in youtube_transcript_ingest.py one around line 183 which used to just request.get, and another at line 332 where I added one line that adds a standard delay between 4 to 11 seconds. I also added imports time and random at the top of the file to facilitate their function.

BACKOFF
At Line 183's block (now 185 in pull request): arbitrary max retries value, can be changed, the ingest will retry that many times. I added a Google Chrome header to make the request seem less bot-y and less likely to be rejected. Then it will try as many times as it's allowed to do what was happening before with requests.get and each time it fails with HTML error 429 like in bug report image above, it will exponentially (again, can be changed) increase the time before the next attempt, so as to backoff if API limits have already been reached.

DELAY
At line 332 (now 354 in pull request): there's a delay that triggers after the first video is ingested. It will be random between 4 and 11 seconds, this can be changed if API limits get stricter in future of course.

[granludo]

What is the status of this one? @juananpe

[LucasImpulse]

Been wondering, fixed up the one line.

[juananpe]

merged, thanks