SHL Conversational Assessment Recommender

Stateless conversational agent that helps hiring managers move from a vague intent (“I am hiring a Java developer”) to a grounded shortlist of SHL Individual Test Solutions from the SHL product catalog.

Aligned with SHL_AI_Intern_Assignment.pdf (SHL Labs AI Intern take-home).

---

Quick start

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

# 1) Build catalog (once, or when SHL site changes)
python scripts/crawl_shl_catalog.py

# 2) Build vector index
python scripts/build_vector_store.py --reset

# 3) API + UI (two terminals)
uvicorn backend.main:app --reload --port 8000
streamlit run frontend/app.py

Optional .env:

GEMINI_API_KEY=...           # better type extraction from queries
CHAT_USE_GEMINI_REPLIES=false
CHROMA_PATH=chroma_db

---

Assignment compliance checklist

Requirement	Implementation
GET /health → {"status": "ok"}	backend/main.py
POST /chat stateless, full messages history	ChatAgent + ChatRequest / ChatResponse
Response schema: reply, recommendations, end_of_conversation	data_models.ChatResponse
Recommendations: name, url, test_type (catalog codes e.g. K, K,P)	ChatRecommendationItem
1–10 recommendations when committed; [] when clarifying/refusing	Orchestrator + analyzer
Clarify vague queries before recommending	conversation_analyzer + clarification_engine
Refine mid-conversation	Re-retrieve on merged user history
Compare assessments (catalog-grounded)	comparison_engine
Refuse off-topic / injection	refusal_guard
Max 8 turns (user + assistant messages)	len(messages) > 8 in chat_agent.py
Catalog-only URLs	validate_catalog_recommendations
Individual Test Solutions only	crawler.py filters pre-packaged job solutions
Target latency < 30s	Rule-based routing; optional Gemini polish off by default
Approach Document	APPROACH.md (Design choices, retrieval, prompts)

Extra (not required by assignment): POST /recommend for batch Recall@10 workflows.

---

How to run the scraper

The scraper lives in src/shl_recommender/crawler.py and is invoked via:

python scripts/crawl_shl_catalog.py

What it does

1. Fetches all catalog listing pages from https://www.shl.com/products/product-catalog/ (Individual Test Solutions table only).
2. For each row, opens the detail page and extracts description, job levels, languages, duration, remote/adaptive flags, and type codes (A, B, C, D, E, K, P, S).
3. Saves progress and raw HTML under data_pages/page_XX.html (cache for re-runs; gitignored).

---

How retrieval works (RAG)

This project uses retrieval-augmented ranking, not a document-QA LLM chain.

Matching (retrieval + ranking)

1. Embed the user query (or merged conversation text).
2. Query Chroma for top candidate_pool_size (default 20) by cosine similarity.
3. Extract assessment types from the query (Gemini if configured, else keyword heuristics in type_extraction.py).
4. Rank: type-matching candidates first, then by embedding similarity within each group.
5. Count: return 1–10 for /chat, 5–10 default for /recommend.

---

Why 8 turns?

The assignment states:

The evaluator caps each conversation at 8 turns including user & assistant and each call at a 30 second timeout.

---

API examples

Health

curl http://localhost:8000/health
# {"status":"ok"}

Chat — recommendations

curl -X POST http://localhost:8000/chat -H "Content-Type: application/json" -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Mid-level Java developer working with stakeholders; need technical and personality screening\"}]}"

Example response shape:

{
  "reply": "Based on your needs...",
  "recommendations": [
    {"name": "Java 8 (New)", "url": "https://www.shl.com/...", "test_type": "K"},
    {"name": "OPQ32r", "url": "https://www.shl.com/...", "test_type": "P"}
  ],
  "end_of_conversation": false
}

---

Tests and evaluation

# Run unit tests
python -m pytest tests/test_agent.py

# Run multi-turn Recall@10 evaluation
python scripts/run_eval.py

---

Deployment (Render)

See render.yaml. Before submitting:

1. Deploy backend; ensure data/shl_individual_assessments.csv and chroma_db/ exist on the instance.
2. Set GEMINI_API_KEY if using Gemini type extraction.
3. Verify GET /health and POST /chat from the public URL.

---

Files safe to delete locally

Item	Reason
data_pages/	HTML scrape cache; regenerated on crawl
__pycache__/, .pytest_cache/	Auto-regenerated
Old assignment PDFs	Keep only SHL_AI_Intern_Assignment.pdf

Do not delete: data/shl_individual_assessments.csv, chroma_db/ (active index), or src/.

---

Remaining improvements

• Create standalone Approach Document (APPROACH.md)
• Fix unit test signature mismatch and improve test coverage
• Implement automated Recall@10 Evaluation script (scripts/run_eval.py)
• Embedding-based name resolution for “compare X and Y”
• Production load test for p95 < 30s on /chat
• Optional CHAT_USE_GEMINI_REPLIES=true after latency check