Pipeline Todos

[nanjiangwill]

Task 1: Update prompt and thesaurus

• update few shot example for each eval_term @ethan-yz-hao
• update thesaurus based on zoning guide @ethan-yz-hao
• find the units

Task 2: Search

• [] merge search results, if two search results are near to each other, merge then
  • for example, page [1,2,3] merge with page [5,6,7] to be [1,2,3,4,5,6,7]
  • LOW Priority since there are 2% of data have this situation
• if not result, try fuzzy and note this is fuzzy search, add this is search output variable

Task 3: District Extraction

• Extract based on match ranking

Task 4: LLM merge with normalization

• [] Mege LLM and normalization stage

Task 5: How to save LLM cost

• [] same prompt for different eval_term

Task 6: Ask model to give which page it found the info