RSS feed preview feature

[wojtekweg]

Some websites provide invalid RSS channels - empty or discontinued. Not to mention that some pages have multiple feeds, indistinguishable with the URL itself or even misconfigures RSS with different than 200 response.

It would be nice to see what's under the found RSS. The bare minimum would be to see the RSS content, good to see the metadata of the cannel (title, description, last update) and the best to see some aggregated stats like posts per month.

In my opinion such feature would strongly elevate the website UX.

[wojtekweg]

Actually, I am on to it :) Will propose PR within next few days

[mratmeyer]

thanks for the suggestion, this is a great idea!

i think it definitely makes sense to add metadata like item count, last updated date, and description. the good news here is this should be relatively low-cost since i already fetch the full feed content to extract the title. parsing the additional metadata would just mean generalizing these methods to scrape more attributes and passing it back in the same request.

for post previews, i'm open to adding them but want to be careful about not cluttering the UI. i think a "Preview" button/icon that displays a popup with the latest X posts could work well.

a few other points:

best to see some aggregated stats like posts per month

this seems doable but perhaps a bit more complex since it requires parsing all item dates. would you consider that a nice-to-have for a follow-up, or essential for the first implementation? it may be worth breaking this out until after the metadata changes are in

Some websites provide invalid RSS channels - empty or discontinued. Not to mention that some pages have multiple feeds, indistinguishable with the URL itself or even misconfigures RSS with different than 200 response

i think there's a bit to unpack here. in scraperUtils.ts i already filter for status codes and content type, but only for URLs found by guessing common paths. scraped tags and rule-generated feeds aren't validated the same way. ideally the validation logic should be generalized and applied to all discovery methods. i can add this as an action item unless you'd like to take it on.

for empty/discontinued feeds do you think there's anything else we should do differently? or would seeing the title, description, last updated date, post count, and statistics be enough and the user can decide if it's relevant?

feel free to make a PR for any of these changes! though these are just things i'm thinking about, not requirements for your PR. if you want to tackle part of it or want feedback along the way let me know and we can coordinate for me to step in and help out.

[wojtekweg]

Hey @mratmeyer, thanks for the reply! Appreciate your thoughtful response!

I have done some research on RSS in the meantime. There are some other lookup pages, but I think yours has the best UI/UX!:) There is one that has preview, but it's not open source, bloated with ads and generally sluggish in my opinion. However, I have found other nice project that exposes the RSS lookup API. As an user I'd recommend you to switch for it and use the FE fetch as a backup when that dependency will be down, or even call the URL your way and supplement the data with that API app asynchronously to make sure the information for user is returned as quick as possible (direct call instead of the proxy call with that API). But I am not owner of yours project, so I won't open the PR with that change and leave the decision up to you.

As an argument for the change, see that the both above mentioned RSS lookup apps provide more information than yours app. See example of http://nytimes.com/ - your app gives 1 result, while the Feedsearch returns multiple channels, probably embedded somewhere within the page (rssfinder.app mentioned above returns more results as well). From architecture point of view it would also maybe better choice to delegate the lookup for the backend rather than frontend.

Either way, I will do the change as you suggested - will include only the basic metadata with simple UI as I have already started preparing the PR. I will open it as soon as I will be sure of my changes. Yet I'd strongly recommend to think about utilising the Feedsearch API power to make the best possible RSS lookup app:)

[wojtekweg]

And to add another argument in favour of the Feedsearch - it gathers all the data that we talked about earlier like the velocity (Mean number of items per day in the feed at the current time), last update, item count etc. Also it validates the feed and returns the favicon URL too to elevate the UI even further.

[mratmeyer]

hi @wojtekweg, thanks for the research and suggestions! i did some digging into RSS Finder and Feedsearch inspired by your comments and it was a really useful exercise.

from what i can tell, the additional results from these tools are mostly coming from stored/manual feed databases rather than a more advanced scraping approach. for example, the NYT feeds like https://rss.nytimes.com/services/xml/rss/nyt/Tennis.xml aren't discoverable by any scraper— NYT just has a manual RSS page listing them all. the good news is this maps perfectly to RSS Lookup's rules feature, and i'll go ahead and add NYT as a rule. this is actually a great example of where we can grow the project over time by adding rules for major sites like this, either via PRs or potentially an AI assisted workflow.

in terms of using external APIs like Feedsearch, i'm going to hold off on that for now. i think a lot of the value of RSS Lookup for people is that it's fully open source, and the open source community is tightly overlapped with the privacy community. i'd rather invest in improving our own detection and rules than add external dependencies. but your suggestion definitely pushed me to think more about coverage gaps, which is really helpful.

for what it's worth, RSS Lookup's scraping already holds up well in cases where these other tools fall short. for example Fly.io and Steam pages where our path-checking and rules system picks up feeds that the others miss entirely.

some comparison screenshots if you're curious

Fly.io

Feedsearch returns nothing found:

RSS Finder returns completely irrelevant suggestions:

RSS Lookup finds it:

Steam (via a community-contributed rule)

Feedsearch returns nothing:

RSS Finder returns more irrelevant suggestions:

RSS Lookup finds it no problem:

regarding the features you mentioned like velocity, item count, and favicons— i'm excited about all of these. they fit really well with the project direction and we can build them out incrementally. looking forward to your metadata PR and happy to help out or give feedback along the way!

[mratmeyer]

fwiw, i just asked claude to add NYT feeds and it did it in a few minutes: #21 https://claude.ai/code/session_01RvTZEXw1h37Gxr9kay54B6 :)

edit: i've added an additional 7 news sources from some of the top news sites based on traffic. this can be an ongoing effort for me to go through the top 50 news sites to get more coverage