A possible (but slow) workaround for context limit

[nonetrix]

I think a good workaround for the context limit could be to have another instance of the LLM summarize the page, then the next page, so on and so on, until you finally ask it to summarize all the summaries into one and remove duplicate information, or present both as possible answers if there is conflicting infomation. Not sure how well this would work, but might work decently somewhat perhaps

[mamei16]

I have experimented with both extractive and abstractive summarization methods to allow a LLM to process whole webpages, but classic extractive summarization algorithms often "summarize away" important parts of the page and rip sentences out of context, while using a language model for abstractive summarization brings us back to the start: Now the language model that is supposed to summarize the web page needs to have a large enough context limit to fit the whole page, and that context needs to fit in memory as well!

This is why I have focused on web searches, where the model mostly searches for specific pieces of information rather than broad concepts, which the model can often recall from its training data. Ideally, the model would iteratively keep on searching until the retrieved information is deemed to be enough to answer the user's question (like Bing®™ is doing), but I think if I tried to do this with the DuckDuckGo API, it would result in rate-limiting very quickly.

Appreciate you sharing the idea though and thank you for coming to my Ted talk.

PS: Your Tokyo Night color theme for KDE is awesome, imma use that