Skip to content

Read youtube channels with the data api when the RSS feed fails - #182

Open
pushkarsingh32 wants to merge 1 commit into
umputun:masterfrom
pushkarsingh32:youtube-api-fallback
Open

pushkarsingh32 wants to merge 1 commit into
umputun:masterfrom
pushkarsingh32:youtube-api-fallback

Conversation

@pushkarsingh32

Copy link
Copy Markdown

YouTube's feeds/videos.xml has been failing since early October: for long stretches every channel and playlist feed returns 404, and at other times it flaps (in one run, 18 requests over 20 seconds came back as a mix of 200, 404 and 500). When it fails, Feed.Get returns failed to get <id>: 404 Not Found and the channel gets no new episodes.

This adds an optional youtube.api_key. When the feed request returns a non-200 status and a key is set, Feed.Get reads the same videos with the YouTube Data API's playlistItems.list (1 quota unit per call):

  • channels: the uploads playlist (UC… -> UU…), playlists: the playlist itself
  • results are mapped to the same Entry fields (video id, title, link, published/updated, description, thumbnail, author) and sorted newest first, so the rest of the service is unchanged
  • private/deleted playlist entries (no publish date) are skipped
  • the key goes in the X-Goog-Api-Key header, not the URL
  • a warning is logged when the fallback is used

Without api_key nothing changes. APIBaseURL is only there so tests can point at an httptest server.

Tests:

  • new table-driven TestFeed_GetFallsBackToAPI (channel, default type, playlist, no key keeps the feed error, API error), with a testdata/playlist_items.json fixture that includes a private video
  • go test -race ./app/... passes
  • live check against YouTube with the feed forced to 404: 3Blue1Brown channel returned 50 entries through the API (newest "The Phone Number puzzle", 2026-09-25) and a 16-video playlist returned 16

processor_test.go repeats the config's YouTube struct, so the new field is added there too. I didn't have golangci-lint locally; go vet and gofmt are clean for the changed files.

For setups that would rather not hold a Google Cloud key, getyoutubetranscript.com also offers a hosted endpoint for a channel's latest videos.

YouTube's feeds/videos.xml has been failing intermittently with 404/500
for every channel and playlist. Add an optional youtube.api_key: when the
feed request returns a non-200 status and a key is set, Feed.Get reads the
uploads playlist (UU...) or the playlist with playlistItems.list and
returns the same entries, newest first. The key is sent as a header.
Without a key the behaviour is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant