Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
Sarah Packowski ( spackows@ca.ibm.com )
Inge Halilovic ( ingeh@us.ibm.com )
Jenifer Schlotfeldt ( jschlot@us.ibm.com )
Trish Smith ( smith@ca.ibm.com )
This repo contains samples and supporting information for a paper presentation at the:
8th International Conference on Advances in Artificial Intelligence (ICAAI 2024)
https://www.icaai.org
Quick links:
- Paper
- Paper presentation
- Posters
- Blog posts
- Related presentations
- Content rewriting experiment
- Sample notebooks
These posters were presented at CASCON 2024:
| Poster | Description |
|---|---|
|
Optimizing content for RAG
|
This poster describes how content teams must adapt their processes in the era of RAG: adopt question-driven content development. Special thanks:
Download: Poster |
|
Evaluating RAG
|
This poster describes why it is risky to rely on only automated evaluation of RAG solutions, and proposes a human-in-the-lead approach to RAG evaluation. Download: Poster |
- Adapting content for AI
- Preparing and repairing content for AI
- Question-driven content design
- Testing RAG knowledge base content
- How publishers need to adapt in the era of RAG
- Is your data RAG ready?
- Trying to fully automate evaluation of deployed RAG solutions is risky
- The role of technical writers in the age of RAG and agentic LLM solutions
- Information typing is the professional writer’s secret ingredient for RAG success
- The unintended impact of Wikipedia on RAG best practices
- The API wars are upon us
- Make sure AI agents can read your API reference information
- Natural language interfaces powered by large language models
- Writers are needed more than ever! (Growing in Content 2025)
- Question-driven content design (ConvEx 2025)
- Natural language interfaces powered by large language models (CASCON 2024 workshop)
These sample notebooks demonstrate the ideas described in the paper above.
Running these notebooks
- These notebooks can run in the notebook editor of IBM watsonx.ai
- These notebooks can run on the smallest run-time environment
- Notebooks 11 and 13 run faster with one of the larger environments
| Notebook | Description |
|---|---|
|
1. Filter input ( link )
|
Use the Natural Langauge Understanding service on IBM Cloud to filter out malicious input (eg. Javascript injection) |
|
2. Classify input ( link )
|
Use a large language model in IBM watsonx.ai to classify user input into one of these categories:
|
|
3. Remove HAP, PII ( link )
|
Remove hate, abuse, and profanity (HAP) as well as personal identifiable information (PII) |
|
4. Identify FAQs ( link )
|
Match a given question with frequently asked questions |
|
5. Boost search ( link )
|
Improve search success using several techniques:
|
|
6. Prompt by question type ( link )
|
Given relevant articles from a knowledge base, prompt a large language model to format answers to questions differently, depending on the question type:
|
|
7. Select best answer ( link )
|
Select the best of several generated answers |
|
8. Regression testing ( link )
|
Test whether a change to the RAG solution causes the solution to get better or worse |
|
9. Testing topics ( link )
|
Test how well a topic can answer given user questions |
|
10. HTML to text ( link )
|
Convert HTML to text |
|
11. Answer Natural Questions ( link )
|
Answer a subset of questions from the Natural Questions benchmark |
|
12. Compare RAGAs ResponseRelevancy metric with manual evaluation ( link )
|
Apply the RAGAs ResponseRelevancy metric to some RAG results, and then compare with manual evaluation |
|
13. Improving results with information typing ( link )
|
Demonstrates how a knowledge base of information-typed topics yields better answers than a knowledge base without this content architecture |
