A very Good open-source Project

[ZoengGwanwai]

That is a very good open-source project, and some suggestions.

First, the web-search, somebody like me use SearXNG .json document for my LLM, but your project unsupport. So I added the SearXNG search function with vibe-coding. I do not think I am a good coder, so I want the official support for SearXNG Search.

And the second, maybe an course outline list can help us learn the systematic subject, and a management based on timeline. It always used in the real school. One Lesson is too short to learn the real skill, such as a code language.

As a Chinese, my English is so bad, I am very very very sorry for my poor English. 英語真的好難。

[wyuc]

Thanks! SearXNG support sounds useful. If you have a working implementation, feel free to open a PR.

The course outline / multi-lesson curriculum idea is interesting. Right now each classroom is a standalone lesson. A curriculum system would be a bigger feature but it's on our radar.

Your English is fine btw :)

[ZoengGwanwai]

[ZoengGwanwai]

The main step includes these, i will contribute my code after i have some free time.

1. Receive user input;
2. Transform user input into const query (from LLM);
3. Search for .json file using SearXNG Example;
4. Integrate .json content into the answer variable;

Document: https://docs.searxng.org/dev/search_api.html

[ZoengGwanwai]

I can only read and write Python, the vibe code have a bad structure, and i can not write next.js, i am so sorry that i cannot PR the function finally.

/lib/web-search/searxng.ts

/**
 * SearXNG Web Search
 * Sorry, I only read python, i am so sorry that is so bad, because i can not write the next.js program, the all code is generated by AI, that is so trunk. 
 *
 * Step:
 * 1. Receive user input(query)
 * 2. Transform user input into search_words, LLM is better, but i can only use LLM by Python Language.
 * 3. Search each search_word .json file using SearXNG
 * 4. Integrate .json content into the answer variable
 */

import { proxyFetch } from '@/lib/server/proxy-fetch';
import type { WebSearchResult, WebSearchSource } from '@/lib/types/web-search';

// ==================== Step 2: find the search_word ====================

/**
 * get the keyword (search_word) for NLP method
 */
function extractKeywordsTraditional(query: string): string[] {
  if (!query || typeof query !== 'string') {
    return [];
  }

  // 1. clean the query
  let cleaned = query
    .toLowerCase()
    .replace(/[^\u4e00-\u9fa5a-zA-Z0-9\s]/g, ' ')  // 保留中文、英文、數字
    .replace(/\s+/g, ' ')
    .trim();

  // 2. set keyword
  const stopWords = new Set([
    // 中文停用詞
    '的', '了', '在', '是', '我', '有', '和', '就', '不', '人', '都', '一', '一個', '上', '也', '很', '到', '說', '要', '去', '你', '會', '著', '沒有', '看', '好', '自己', '這', '那', '什麼', '怎麼', '為什麼', '嗎', '吧', '呢', '啊', '哦', '嗯',
    // English words forbidden
    'the', 'be', 'to', 'of', 'and', 'a', 'in', 'that', 'have', 'i', 'it', 'for', 'not', 'on', 'with', 'he', 'as', 'you', 'do', 'at', 'this', 'but', 'his', 'by', 'from', 'they', 'we', 'say', 'her', 'she', 'or', 'an', 'will', 'my', 'one', 'all', 'would', 'there', 'their', 'what', 'so', 'up', 'out', 'if', 'about', 'who', 'get', 'which', 'go', 'me', 'when', 'make', 'can', 'like', 'time', 'no', 'just', 'him', 'know', 'take', 'people', 'into', 'year', 'your', 'good', 'some', 'could', 'them', 'see', 'other', 'than', 'then', 'now', 'look', 'only', 'come', 'its', 'over', 'think', 'also', 'back', 'after', 'use', 'two', 'how', 'our', 'work', 'first', 'well', 'way', 'even', 'new', 'want', 'because', 'any', 'these', 'give', 'day', 'most', 'us', 'is', 'are', 'was', 'were', 'been', 'has', 'had', 'do', 'does', 'did', 'doing'
  ]);

  // 3. split the word
  const words = cleaned.split(' ').filter(word => {
    if (!word || word.length === 0) return false;
    if (stopWords.has(word)) return false;
    if (/^\d+$/.test(word) && word.length < 3) return false;  // 過濾單個數字
    return true;
  });

  // 4. get the phrase
  const phrases: string[] = [];
  const queryLower = query.toLowerCase();
  
  // 4.1 get the quotedMarches
  const quotedMatches = query.match(/"([^"]+)"/g) || [];
  quotedMatches.forEach(match => {
    const phrase = match.replace(/"/g, '').trim();
    if (phrase.length > 0) phrases.push(phrase);
  });

  // 4.2 get the patterns
  const patterns = [
    { regex: /(?:如何|怎麼|怎樣).+?(?=\?|？|$|。)/, weight: 2 },
    { regex: /(?:教授|teaching|教學|learning).+?(?=\?|？|$|。)/, weight: 2 },
    { regex: /(?:theme|unit|主題|專案).+?(?=\?|？|$|。)/i, weight: 2 },
    { regex: /(?:課程|lesson|課時).+?(?=\?|？|$|。)/, weight: 1.5 },
  ];

  patterns.forEach(({ regex }) => {
    const match = queryLower.match(regex);
    if (match && match[0]) {
      phrases.push(match[0].trim());
    }
  });

  // 5. It is very hard for me, LLM maybe better, but i am too lazy to write a new route api for /web_search
  const wordFreq = new Map<string, number>();
  words.forEach(word => {
    const len = word.length;
    let weight = 1;
    if (len > 4) weight = 2;  
    if (/[\u4e00-\u9fa5]/.test(word) && len >= 2) weight *= 1.5;  // for Chinese
    
    wordFreq.set(word, (wordFreq.get(word) || 0) + weight);
  });

  // 6. Select the key words
  const sortedWords = Array.from(wordFreq.entries())
    .sort((a, b) => b[1] - a[1])
    .map(([word]) => word)
    .slice(0, 5);

  const finalKeywords = [...new Set([...phrases, ...sortedWords])];

  // 8. And the second method, the original query...
  if (finalKeywords.length === 0) {
    return [query.trim().substring(0, 100)];
  }

  return finalKeywords.slice(0, 3);  // 最多返回 3 個搜索詞
}

/**
 * 將關鍵詞用空格連接成搜索字符串
 * 並截斷為 const 格式
 */
function buildSearchQuery(keywords: string[]): string {
  // 過濾無效值，防止 undefined
  const validKeywords = keywords.filter(k => 
    k && typeof k === 'string' && k.trim().length > 0
  );

  if (validKeywords.length === 0) {
    return '';
  }

  // 用空格連接，並截斷到合理長度（SearXNG 通常支持較長查詢）
  const joined = validKeywords.join(' ');
  
  // 截斷為 const 格式（這裡指固定長度限制）
  const MAX_LENGTH = 200;
  return joined.length > MAX_LENGTH ? joined.substring(0, MAX_LENGTH) : joined;
}

// ==================== Step 3: SearXNG 搜索 ====================

/**
 * 使用 SearXNG 搜索
 */
async function searchWithSearXNGSingle(
  searchQuery: string,
  baseUrl: string
): Promise<{
  searchQuery: string;
  results: Array<{
    title: string;
    url: string;
    content: string;
    engine: string;
    score?: number;
  }>;
  numberOfResults: number;
  rawJson: any;
}> {
  if (!searchQuery || typeof searchQuery !== 'string') {
    throw new Error('Invalid search query: must be a non-empty string');
  }

  if (!baseUrl || typeof baseUrl !== 'string') {
    throw new Error('Invalid baseUrl: must be a non-empty string');
  }

  // clean the baseUrl
  const cleanBaseUrl = baseUrl.replace(/\/$/, '');

  // SearXNG JSON API endpoint
  const searchUrl = new URL(`${cleanBaseUrl}/search`);
  searchUrl.searchParams.set('q', searchQuery);
  searchUrl.searchParams.set('format', 'json');

  const res = await proxyFetch(searchUrl.toString(), {
    method: 'GET',
    headers: {
      'Accept': 'application/json',
    },
  });

  if (!res.ok) {
    const errorText = await res.text().catch(() => '');
    throw new Error(`SearXNG API error (${res.status}): ${errorText || res.statusText}`);
  }

  const rawJson = await res.json();

  if (!rawJson || typeof rawJson !== 'object') {
    throw new Error('Invalid SearXNG response: not an object');
  }

  const data = rawJson as {
    query?: string;
    number_of_results?: number;
    results?: Array<{
      title?: string;
      url?: string;
      content?: string;
      engine?: string;
      score?: number;
    }>;
  };

  const results = (data.results || []).map(r => ({
    title: r.title || 'Untitled',
    url: r.url || '#',
    content: r.content || '',
    engine: r.engine || 'unknown',
    score: typeof r.score === 'number' ? r.score : 1.0,
  }));

  return {
    searchQuery,
    results,
    numberOfResults: typeof data.number_of_results === 'number' 
      ? data.number_of_results 
      : results.length,
    rawJson,
  };
}

// ==================== The main step ====================

/**
 * 
 * Step 1: Receive user input(query)
 * Step 2: Transform user input into search_words (LLM is better, but that is not)
 * Step 3: Search using SearXNG with space-separated keywords
 * Step 4: Integrate .json content into the answer variable
 */
export async function searchWithSearXNG(params: {
  query: string;
  baseUrl: string;
  maxResults?: number;
}): Promise<WebSearchResult> {
  const { query, baseUrl, maxResults = 5 } = params;

  // ==================== Step 1: Receive user input ====================
  console.log(`[Step 1] Received user query: "${query}"`);

  // 防禦式檢查
  if (!query || typeof query !== 'string') {
    throw new Error('Query must be a non-empty string');
  }

  // ==================== Step 2: Get the keywords ====================
  const keywords = extractKeywordsTraditional(query);
  const searchQuery = buildSearchQuery(keywords);
  
  console.log(`[Step 2] Extracted keywords:`, keywords);
  console.log(`[Step 2] Built search query: "${searchQuery}"`);

  if (!searchQuery) {
    throw new Error('Failed to extract valid search keywords from query');
  }

  // ==================== Step 3: SearXNG Search ====================
  console.log(`[Step 3] Searching with query: "${searchQuery}"`);
  
  const searchResult = await searchWithSearXNGSingle(searchQuery, baseUrl);

  // ==================== Step 4: The result ====================
  
  // 去重並限制結果數量
  const seenUrls = new Set<string>();
  const integratedSources: WebSearchSource[] = [];

  for (const r of searchResult.results) {
    if (!seenUrls.has(r.url)) {
      seenUrls.add(r.url);
      integratedSources.push({
        title: r.title,
        url: r.url,
        content: r.content,
        score: r.score,
      });
    }
  }

  const finalSources = integratedSources.slice(0, maxResults);

  // 構建 answer
  const answer = `
針對問題 "${query}" 的搜索結果：

提取的關鍵詞: ${keywords.join(', ')}
搜索查詢: "${searchQuery}"
找到結果數: ${searchResult.numberOfResults}

相關來源:
${finalSources.map((s, i) => `${i + 1}. ${s.title} (${s.url})`).join('\n')}

摘要:
${finalSources.map(s => `- ${s.content.substring(0, 150)}...`).join('\n')}
`.trim();

  console.log('[Step 4] Integrated search results');

  return {
    answer,
    sources: finalSources,
    query: query,
    responseTime: 0,
  };
}

/**
 * 可選：與 Tavily 共用相同的格式化函數
 */
export { formatSearchResultsAsContext } from './tavily';

[markqvist]

@ZoengGwanwai if you're interested in completely local setups for OpenMAIC, you can check out my Ember port. It's optimized for completely local running and inference. SearXNG is next on my todo-list for it, along with adapters for hooking custom Python code and functionality into the system. So I think you may like it :)

Also huge thanks @wyuc for maintaining and working on this fantastic project! It is a dream come true for local, powerful learning. I created my Ember fork because my own philosophy of what I want it to be probably does not align perfectly with the core OpenMAIC project, but you are of course welcome to take anything you can use from the fork back into OpenMAIC.

Keep up the good work!