Help getting this to work?

[jdbornem]

This extension seems perfect for what I want, and I'm impressed at the capability; if I can get it to work!
I'm an extreme novice at Python (although several other languages), git, environments, LLMs, etc. Be kind.

Regarding Installation, I'm on Windows, and I installed the extension as normal using the "Install or update and extension" function in text-generation-webui, providing the path to the LLM_Web_search git page.
However, the provided directions to load dependencies by updating the conda environment did not work. "Update the conda environment in which you installed the dependencies" Updating the environment as directly went fine, and it even installed the dependencies successfully. However, I kept receiving numerous errors when I tried to start the webui with LLM_Web_search enabled as a default extension (python server.py --extension LLM_Web_search). Unfortunately, I did not capture all of them at the time, but there are alot of other's talking about the errors with langchain.

Finally, I just opened the command window through "cmd_windows.bat", and I did not activate the conda environment, I just directly installed the dependencies from the root text-generation-webui folder manually using pip:
pip install faiss-cpu=1.7.4
pip install duckduckgo-search==4.1.0
pip install beautifulsoup4==4.12.2
pip install langchain==0.0.352
pip install unstructured==0.6.6
pip install rank_bm25==0.2.2
pip install sentence-transformers==2.2.2

Then the webui finally loaded with the LLM_Web_search extension enabled!

Now, how in the world to get the model to actually use the search function?

So my LLM_Web_search tab now looks like this. I have saved the "Custom System Message" using the default "websearch_gpt4_prompt". I used the recommended "Search command regex string", and despite the instructions talking about how to customize it, I'm still not sure why I would want to.

I usually use "TheBloke_Mixtral-8x7B-Instruct-v0.1-GPTQ" loaded with ExLlama2_HF (or AutoGPTQ). It wasn't using the search function at all, it just "pretended" to, so for the sake of consistency, I tried the "orca-2-13b.Q8_0.gguf" loaded with llama.cpp. Indicated as being "the most consistent" as per here: #12 (reply in thread)

I also created a "Searcher" character, with the description recommended:

A chat between a curious user and artificial intelligence assistant. The assistant is never confident about facts and up-to-date information. The assistant can search the web for facts and up to date information using the following search command format:

Search_web("query")

The search tool will search the web for these keywords and return the results. Finally, the assistant extracts the information from the results of the search tool to guide its response.

I'm not quite sure if I should be changing anything on the "Instruction Template" tab under "Parameters". It shows the saved Custom System Message, but which template should I choose? Right now it's on the Llama-v2.

Now, into the Chat tab, in chat-instruct mode:

It tried to do the search, exactly as in the demonstration, but it's not actually searching, nor showing or using any results.
I've tried several other queries as well, including the one shown about the weather in NY. (The actual DuckDuckGo search for weather in NY is that it is 42 degrees. This response is a halucination.)

What am I doing wrong?
How do I get this to work?

[mamei16]

The likely problem and solution

To get the most probable cause out of the way first: If the screenshot shows your current settings, then you're trying to use the default websearch_gpt4_prompt with a custom regex string. This regex string does not match the the search command defined in "websearch_gpt4_prompt", so that's why it's not actually searching. To use the default regex strings, just leave these text fields blank.

Characters for searching

Regarding the "searcher" character: If you're using a "Custom System Message", you don't need to also define a specific character where you provide similar instructions. You can just create a character with an empty context and use the "Custom System Message" to tell the model what it should do.

Instruction templates

Many LLMs have been trained on data with a specific format (which is defined in the instruction template) and will not work well (or at all) if you're not using the same format when using the LLM to generate text for you. These are generally called instruction-tuned LLMs.
In the easiest case, a message similar to the following will appear in the web UI after you've loaded an LLM:

In the case of this screenshot, you simply choose the ChatML instruction template and click load. If such a message is not displayed, you'll have to read the model description on huggingface to figure out what template you should be using.

Windows installation issues

Lastly, thanks a lot for providing a solution for the installation issues you faced on Windows, I'll be sure to mention it if others run into similar problems!

[jdbornem]

Thank you for the guidance!
Unfortunately, I won't be able to test further for about a week, but once I do, I will be sure to update here.

[jdbornem]

I had a chance to try again, and I

• reverted the regex string to default
• used the websearch_gpt4_prompt
• loaded the model: orca-2-13b.Q8_0.gguf
• loaded the ChatML instruction
• Used an empty character context

And I should have thought of this sooner, but I also have the extension Superboogav2 running for memory, but it obviously intercepts the prompts, and was causing the problems. LLM_Web_search did not work, until I disabled Superboogav2 and reloaded.

Then it worked as described!

I'm not sure if I'll run it long term, since with some testing, the raw results rarely have very much actual information to answer questions. It gave me the correct time in a specific city, but questions like "Who won the AFC Championship game?" only result in the history of the AFC Championship. "Who won the AFC Championship in 2024?" still didn't give any specific information.
Questions on technical topics can sometimes answer specific questions, but I've found that the LLM without internet is just as likely to give a good answer.

None of this is the fault of the extension! It's a great capability. It's more about the limitation of the blurbs included in search results.
But I'll test more.

Thank you!

[mamei16]

Hey, good to hear you got it working!

Just to avoid any misunderstandings: Selecting the search result blurbs is, in fact, the most important (and probably hardest) part of the extension. Most LLM web search programs retrieve search results in one of two ways:

1. Simply return the title of each search result and associated short text that is shown under each search result. This does not visit any website in the search results.
2. Use Retrieval-Augmented Generation (RAG) in an effort to extract the parts of each website that are most relevant to the search query. This involves downloading each website, converting the HTML code to plain text as best as possible, splitting this plain text into chunks and, finally, retrieving the chunks of text that are most similar to the search query.

This extension started out using method 1., but often times the sought after information is buried somewhere in the website and is not available just in the search engine results alone. This is why the switch to method 2. was made, which in general is more effective at retrieving information, but is more reliant on tuning different parameters, such as chunk size, Similarity Score Threshold and the number of websites to process and chunks to return.
For example, with my parameter settings:

• Max. search results to return per query: 8
• Number of search results to process per query: 10
• Langchain Similarity Score Threshold : 0.5
• Chunk size: 500

- asking the LLM "Hi! I have a question: Who won the AFC Championship game?" results in the following results and answer:

AI: Hello! To find out who won the most recent AFC Championship game, let me search the web for that information. Please give me a moment to gather the data. (Search_web("AFC Championship winner"))

Click to reveal raw search results

Search tool:
Result 1:
AFC champion
then advances to face the winner of the
NFC Championship Game
in the
Super Bowl
.
The game was established as part of the
1970 merger
between the NFL and the
American Football League
(AFL), with the merged league realigning into two
conferences
. Since 1984,
[1]
each winner of the AFC Championship Game has also received the
Lamar Hunt Trophy
, named after the founder of both the AFL and the
Kansas City Chiefs
,
Lamar Hunt
.
History
Source URL: https://en.wikipedia.org/wiki/AFC_Championship_Game

Result 2:
and
Denver Broncos
each have eight AFC championships.  The Buffalo Bills won six AFL or AFC championships and the Miami Dolphins and
Oakland/Los Angeles Raiders
have each won five AFL or AFC championships.
[1]
The Raiders have also been the AFL or AFC runner up, as a result of losing the AFL or AFC Championship Game, a record nine times.
[3]
The Steelers have been the runner up eight times, while the
Los Angeles/San Diego Chargers
have been the runner up seven times.
[4]
[5]
Source URL: https://en.wikipedia.org/wiki/List_of_AFC_champions

Result 3:
For the list of AFC Championship Games, see
AFC Championship Game
. For the list of NFC Champions, see
List of NFC champions
.
Tom Brady
won 9 AFC Championship titles with the
New England Patriots
, the most by any player.
The
American Football Conference
(AFC) is one of two conferences within the
National Football League
, the
National Football Conference
(NFC) being the other.  
The AFC has its roots in the
American Football League
Source URL: https://en.wikipedia.org/wiki/List_of_AFC_champions

Result 4:
AFC Championship Games
American Football League Championship Game
American Football League championship seasons
Super Bowl lists
National Football League records and achievements
Lists of sports championships
Hidden categories:
CS1 maint: multiple names: authors list
Articles with short description
Short description is different from Wikidata
Articles with hCards
This page was last edited on 5 February 2024, at 19:36
(UTC)
.
Text is available under the
Source URL: https://en.wikipedia.org/wiki/List_of_AFC_champions

Result 5:
[9]
[10]
List of AFC Championship Games
See also:
List of AFC champions
Numbers in parentheses in the table are AFC Championships
.
Bold
indicates team won Super Bowl that year
.
Numbers in parentheses in the city and stadium column is the number of times that metropolitan area and stadium has hosted an AFC Championship, respectively.
Season
Playoffs
Winning team
Score
Losing team
Score
Location
Stadium
1970
1970–71
Baltimore Colts
(1)
27
Oakland Raiders
17
Baltimore, Maryland
[fn 1]
Source URL: https://en.wikipedia.org/wiki/AFC_Championship_Game

Result 6:
.
History
The first AFC Championship Game was played following the 1970 regular season after the
merger
between the NFL and the AFL. The game is considered the successor to the former
AFL Championship
, and its game results are listed with that of its predecessor in the annual
NFL Record and Fact Book
.
[2]
Source URL: https://en.wikipedia.org/wiki/AFC_Championship_Game

Result 7:
Archived
from the original on 2015-10-01
. Retrieved
2019-01-21
.
^
"2000 AFC Championship"
.
Pro-Football-Reference
. Sports Reference LLC. Archived from
the original
on 2008-10-10
. Retrieved
2008-11-22
.
^
"Super Bowl XXXV"
.
Pro-Football-Reference
. Sports Reference LLC.
Archived
from the original on 2019-02-26
. Retrieved
2008-11-22
.
Retrieved from "
https://en.wikipedia.org/w/index.php?title=List_of_AFC_champions&oldid=1203858240
"
Categories
:
National Football League lists
Source URL: https://en.wikipedia.org/wiki/List_of_AFC_champions

Result 8:
Trophy
2023 season
M&T Bank Stadium
Baltimore, Maryland
January 28, 2024
Kansas City Chiefs
17,
Baltimore Ravens
10
The
AFC Championship Game
is the annual championship game of the
American Football Conference
(AFC) and one of the two semifinal
playoff
games of the
National Football League
(NFL), the largest professional
American football
league in the world. The game is played on the last Sunday in January by the two remaining playoff teams, following the AFC postseason's first two rounds. The
Source URL: https://en.wikipedia.org/wiki/AFC_Championship_Game

AI: Thank you for your question! According to my search results, in the 2023 season AFC Championship Game on January 28, 2024, the Kansas City Chiefs defeated the Baltimore Ravens with a score of 17-10.

As you can see in the example above, the actual answer was only contained in Result 8, meaning a lower "Max. search results to return per query" might not have contained the answer. So it is necessary to strike a balance between including more search results to increase the chance of them containing the necessary info and not including too many results to avoid exceeding the max. context length or overwhelming the model.

TLDR

The limitation of the blurbs included in search results is most definitely the fault of the extension, but it's not an easy problem to solve.

[ill13]

Simplified instructiosn for new users.

• Install the extension
• Leave everything on default
• Open your Character [Parameters | Chat] and append the following text:

You will search the web for facts and up to date information using the following search command format:
Search_web("query")
The search tool will search the web for these keywords and return the results. Finally, you will extract the information from the results of the search tool to guide your response.

• Try asking something like What's the current time in NYC? or What's going on with the HellDivers2 PSN controversy?