Repository navigation
Read youtube channels with the data api when the RSS feed fails - #182
Open
pushkarsingh32 wants to merge 1 commit into
Open
pushkarsingh32 wants to merge 1 commit into
pushkarsingh32 wants to merge 1 commit into
Conversation
YouTube's feeds/videos.xml has been failing intermittently with 404/500 for every channel and playlist. Add an optional youtube.api_key: when the feed request returns a non-200 status and a key is set, Feed.Get reads the uploads playlist (UU...) or the playlist with playlistItems.list and returns the same entries, newest first. The key is sent as a header. Without a key the behaviour is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
YouTube's
feeds/videos.xmlhas been failing since early October: for long stretches every channel and playlist feed returns 404, and at other times it flaps (in one run, 18 requests over 20 seconds came back as a mix of 200, 404 and 500). When it fails,Feed.Getreturnsfailed to get <id>: 404 Not Foundand the channel gets no new episodes.This adds an optional
youtube.api_key. When the feed request returns a non-200 status and a key is set,Feed.Getreads the same videos with the YouTube Data API'splaylistItems.list(1 quota unit per call):UC…->UU…), playlists: the playlist itselfEntryfields (video id, title, link, published/updated, description, thumbnail, author) and sorted newest first, so the rest of the service is unchangedX-Goog-Api-Keyheader, not the URLWithout
api_keynothing changes.APIBaseURLis only there so tests can point at anhttptestserver.Tests:
TestFeed_GetFallsBackToAPI(channel, default type, playlist, no key keeps the feed error, API error), with atestdata/playlist_items.jsonfixture that includes a private videogo test -race ./app/...passesprocessor_test.gorepeats the config's YouTube struct, so the new field is added there too. I didn't have golangci-lint locally;go vetandgofmtare clean for the changed files.For setups that would rather not hold a Google Cloud key, getyoutubetranscript.com also offers a hosted endpoint for a channel's latest videos.