[ARCHITECTURE] Replace Ollama with llama.cpp native C++ integration

[mikejmorgan-ai]

Problem

Current architecture uses Ollama (Go-based, 500MB+) as a separate installed application. This makes Cortex feel like a bolted-on tool rather than native OS integration.

Solution (from Ed's feedback)

Integrate llama.cpp directly as a C++ library:

• Native binary integration (no separate process)
• Links directly into system components
• Smaller footprint, faster startup
• Feels like part of the OS, not an app
• Build custom binaries for natural language processing

Why llama.cpp over Ollama

Ollama	llama.cpp
Go binary, 500MB+	C++ library, <50MB
Separate process	Links into daemon
Feels like installed app	Feels like OS component
Network API overhead	Direct function calls

Technical Notes

• Reference: https://github.com/ggerganov/llama.cpp
• Consider static linking for single binary distribution
• Must support ARM64 and x86_64
• GGUF model format

Acceptance Criteria

• llama.cpp integrated as shared library or static link
• Cortex daemon can call inference without spawning external process
• Works offline with no cloud dependency
• Startup time < 100ms for inference readiness
• Memory footprint < 100MB with model loaded
• Documentation for building from source

Related Issues

• [FEATURE] AI Shell with lightweight apt-trained model for natural language command generation #424 (tiny model) - Will use this runtime
• [FEATURE] [CRITICAL] Autonomous Security Vulnerability Management & Patching #422 (security patching) - Will use local inference
• [BUG] Remove hardcoded llama3.2 model, allow configurable Ollama model selection #383 (Ollama model selection) - Will be superseded

Bounty: $150 (+ $150 bonus after funding)

Paid on merge to main.

[Anshgrover23]

@sujay-d07 Kindly update Issue description.

[ipezygj]

llama.cpp Native C++ Integration for Cortex

Architecture Overview

cortex-daemon
├── src/
│   ├── inference/
│   │   ├── llama_engine.h        # C++ engine interface
│   │   ├── llama_engine.cpp      # llama.cpp integration
│   │   ├── inference_manager.h   # High-level manager
│   │   └── inference_manager.cpp
│   ├── bindings/
│   │   ├── llama_c_bridge.h      # C bridge for FFI
│   │   └── llama_c_bridge.cpp
│   └── models/
│       ├── model_registry.h
│       └── model_registry.cpp
├── vendor/
│   └── llama.cpp/               # Git submodule
├── CMakeLists.txt
├── scripts/
│   ├── build.sh
│   ├── download_model.sh
│   └── install.sh
└── docs/
    └── building_from_source.md

Implementation

1. CMakeLists.txt

cmake_minimum_required(VERSION 3.20)
project(cortex-daemon VERSION 1.0.0 LANGUAGES C CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_C_STANDARD 11)

# Build options
option(CORTEX_STATIC_LINK    "Link llama.cpp statically"          ON)
option(CORTEX_METAL           "Enable Apple Metal GPU (macOS)"    OFF)
option(CORTEX_CUDA            "Enable NVIDIA CUDA GPU"            OFF)
option(CORTEX_VULKAN          "Enable Vulkan GPU"                 OFF)
option(CORTEX_AVX2            "Enable AVX2 SIMD (x86_64)"        OFF)

# Detect architecture
if(CMAKE_SYSTEM_PROCESSOR MATCHES "arm64|aarch64")
    set(ARCH_ARM64 TRUE)
    message(STATUS "Target: ARM64")
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64|amd64|AMD64")
    set(ARCH_X86_64 TRUE)
    message(STATUS "Target: x86_64")
else()
    message(FATAL_ERROR "Unsupported architecture: ${CMAKE_SYSTEM_PROCESSOR}")
endif()

# ── llama.cpp submodule ──────────────────────────────────────────────────────
set(LLAMA_CPP_DIR "${CMAKE_SOURCE_DIR}/vendor/llama.cpp")

if(NOT EXISTS "${LLAMA_CPP_DIR}/CMakeLists.txt")
    message(FATAL_ERROR
        "llama.cpp submodule not found. Run:\n"
        "  git submodule update --init --recursive")
endif()

# Forward GPU options into llama.cpp's own build
set(LLAMA_METAL  ${CORTEX_METAL}  CACHE BOOL "" FORCE)
set(LLAMA_CUDA   ${CORTEX_CUDA}   CACHE BOOL "" FORCE)
set(LLAMA_VULKAN ${CORTEX_VULKAN} CACHE BOOL "" FORCE)

# Build llama.cpp as a library only (no CLI executables)
set(LLAMA_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_TESTS    OFF CACHE BOOL "" FORCE)
set(LLAMA_BUILD_SERVER   OFF CACHE BOOL "" FORCE)

add_subdirectory(${LLAMA_CPP_DIR} llama_cpp_build)

# ── Cortex inference library ─────────────────────────────────────────────────
set(INFERENCE_SOURCES
    src/inference/llama_engine.cpp
    src/inference/inference_manager.cpp
    src/bindings/llama_c_bridge.cpp
    src/models/model_registry.cpp
)

if(CORTEX_STATIC_LINK)
    add_library(cortex_inference STATIC ${INFERENCE_SOURCES})
else()
    add_library(cortex_inference SHARED ${INFERENCE_SOURCES})
    set_target_properties(cortex_inference PROPERTIES
        VERSION   ${PROJECT_VERSION}
        SOVERSION 1
    )
endif()

target_include_directories(cortex_inference
    PUBLIC
        $<BUILD_INTERFACE:${CMAKE_SOURCE_DIR}/src>
        $<INSTALL_INTERFACE:include/cortex>
    PRIVATE
        ${LLAMA_CPP_DIR}
        ${LLAMA_CPP_DIR}/common
)

target_link_libraries(cortex_inference
    PUBLIC  llama
    PRIVATE $<$<PLATFORM_ID:Linux>:pthread>
            $<$<PLATFORM_ID:Linux>:dl>
)

# Architecture-specific flags
if(ARCH_ARM64)
    target_compile_options(cortex_inference PRIVATE -march=armv8-a+dotprod+fp16)
elseif(ARCH_X86_64)
    target_compile_options(cortex_inference PRIVATE
        -march=native
        $<$<BOOL:${CORTEX_AVX2}>:-mavx2 -mfma>
    )
endif()

target_compile_options(cortex_inference PRIVATE
    -O3
    -ffast-math
    -fvisibility=hidden
    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wall -Wextra -Wpedantic>
)

# ── Cortex daemon executable ─────────────────────────────────────────────────
add_executable(cortexd src/main.cpp)
target_link_libraries(cortexd PRIVATE cortex_inference)

# ── Tests ────────────────────────────────────────────────────────────────────
enable_testing()
add_subdirectory(tests)

---
**Payment Address:** `0xa32ca744f86a91eaf567e2be4902f64bc33c2813`

[Fred-Zhang83]

我可以来做这个（如果确实是有偿/悬赏任务的话）。为了对齐预期并尽快开始，麻烦确认几件事：

1. 悬赏/报酬金额与结算方式（是 GitHub Sponsors / Bounty 平台 / 直接转账？）
2. 验收标准（通过哪些测试/功能点算完成？）
3. 期望交付形式（PR 合并即算？还是需要 release/文档/示例？）

你补充以上信息后，我可以先给出实现方案与预计工期，然后直接开 PR 推进。