From 33387c48b6e10701e949b2b996a9dbd5a52552df Mon Sep 17 00:00:00 2001 From: Umputun Date: Sat, 19 Sep 2026 21:42:10 -0500 Subject: [PATCH] docs: correct what the youtube item limits actually control max_per_channel was documented as the per-channel download limit, which it is not. setDefaults gives every channel Keep from system.max_per_feed when the channel sets none, and for a configured channel with a positive effective keep that value overrides max_per_channel for downloads. What max_per_channel does limit is the channel source page, RSS requests for channel ids absent from the config, and files written by POST /yt/rss/generate. Also documents that the upstream feed returns only a window of recent entries, so a limit above that window raises retention rather than fetching more per poll, and adds a backlog item for the generate/serve limit split this turned up. Fixed #125 --- README.md | 13 +++++++++-- ...ted-and-served-rss-use-different-limits.md | 23 +++++++++++++++++++ 2 files changed, 34 insertions(+), 2 deletions(-) create mode 100644 docs/backlog/generated-and-served-rss-use-different-limits.md diff --git a/README.md b/README.md index d6f2a7a..8e43196 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ youtube: # youtube configuration, optional base_playlist_url: "https://www.youtube.com/feeds/videos.xml?playlist_id=" # base url for youtube playlist update: 60s # update interval for youtube feeds skip_shorts: 120s # skip videos (and audios) shorter than this value, optional - max_per_channel: 2 # max number of the latest videos per yt channel to download and process + max_per_channel: 2 # item limit for the channel source page and fallback RSS generation, not a download limit files_location: ./var/yt # location for downloaded youtube files rss_location: ./var/rss # location for generated youtube channel's RSS channels: # list of youtube channels to download and process @@ -73,7 +73,7 @@ youtube: # youtube configuration, optional system: # system configuration update: 1m # update interval for checking source feeds http_response_timeout: 30s # http response timeout - max_per_feed: 10 # max items per feed to be processed and inclueded in the final RSS + max_per_feed: 10 # max items per feed to be processed and included in the final RSS, also the default keep for yt channels max_total: 50 # max total items to be included in the final RSS max_keep: 1000 # max items to be kept in the internal database base_url: http://localhost:8080 # base url for the generated RSS and media files @@ -81,6 +81,15 @@ system: # system configuration _see [examples](https://github.com/umputun/feed-master/tree/master/_example/etc) for more details._ +**YouTube item limits.** Each channel's `keep` is the limit that matters. It caps how many entries are +processed per poll and how many items `/yt/rss/{channel}` returns, and retention cleanup keeps up to +`keep + 1` stored entries. A channel with no explicit `keep` falls back to `system.max_per_feed`, not to +`youtube.max_per_channel`. + +The upstream YouTube feed returns only a window of recent entries per channel, so a limit above that +window does not fetch more videos in a single poll. It raises how many collected entries are retained +and served over time. + ### Single-feed configuration For a very simple configuration, command-line only configuration is available. In this case only a single source feed is allowed and yt processing is disabled. The command-line configuration is the following: diff --git a/docs/backlog/generated-and-served-rss-use-different-limits.md b/docs/backlog/generated-and-served-rss-use-different-limits.md new file mode 100644 index 0000000..2a91aa8 --- /dev/null +++ b/docs/backlog/generated-and-served-rss-use-different-limits.md @@ -0,0 +1,23 @@ +--- +worth: maybe +where: app/api/server.go:313 +added: 2026-09-19 +--- +# Generated and served RSS can apply different item limits to the same channel + +`regenerateRSSCtrl` builds `youtube.FeedInfo{ID: f.ID}` for every configured channel, dropping the +channel's `Keep`. `Service.keep` then falls back to `KeepPerChannel`, which is `youtube.max_per_channel`. +The `GET /yt/rss/{channel}` handler at `app/api/server.go:288` looks the channel up in the config and +passes the whole `FeedInfo`, so it uses the channel's effective `keep`. Automatic RSS writes after a +successful download (`app/youtube/service.go:339`) do the same. + +So the two paths can produce different item counts for one channel, when their limits differ and enough +entries are stored. With `keep: 35`, `max_per_channel: 20` and at least 21 stored entries, `POST +/yt/rss/generate` writes 20 items and a later automatic rewrite writes up to 35. + +The fix is to pass the configured `FeedInfo` in `regenerateRSSCtrl`, as the GET handler does. Whether +that is the right call depends on whether `max_per_channel` should keep any role at all, and nobody has +ruled on that, which is why this is `maybe`. The unconfigured-channel fallback in the GET handler uses +`max_per_channel` too. + +Deferred during the documentation correction for #125, which exposed the split but did not resolve it.