fix: address unresolved review comments from PyPDF File Processor PR#4743 (#5173)

# What does this PR do?

Addresses remaining unresolved review comments from PR #4743 (PyPDF File
Processor integration) to ensure the file processing pipeline is
consistent, correctly typed, and aligned with API expectations.

**Key changes:**

- **Remove legacy chunking fallback:** Eliminate the
`_legacy_chunk_file` method and all fallback paths from
`OpenAIVectorStoreMixin`. The system now raises a clear `RuntimeError`
if `file_processor_api` is not configured, instead of silently degrading
to legacy inline parsing.
- **Wire `file_processor_api` through all vector_io providers:** Add
`Api.file_processors` to `optional_api_dependencies` in the registry,
pass it through all 12 factory functions, and accept/forward it in all 9
adapter constructors.
- **Make `files_api` required in PyPDF constructors:** Remove the
default `None` from both `PyPDFFileProcessorAdapter` and
`PyPDFFileProcessor`, and use `deps[Api.files]` (bracket access) in the
factory to fail fast if somehow missing.
- **Bounded file reads:** Implemented chunked reading for direct uploads
to cap memory usage, and add a size check on the `file_id` retrieval
path against `max_file_size_bytes`.
- **Clear error for missing `file_id`:** Wrap `openai_retrieve_file` in
a `try/except` that surfaces a `ValueError("File with id '...' not
found")`, with a new test covering this case.
- **Conditional text cleaning:** Make the `.strip()` whitespace-only
page filter conditional on the `clean_text` config setting.
- **Dead code cleanup:** Remove unused `file_processor_api` field from
`VectorStoreWithIndex` and the now-unused `make_overlapped_chunks`
import from the mixin.

Closes #4743

## Test Plan

### Automated tests

**1. Unit tests (mixin + vector_io)**

```bash
KMP_DUPLICATE_LIB_OK=TRUE uv run --group unit pytest -sv tests/unit/
```

All `test_contextual_retrieval.py` (16 tests) and
`test_vector_store_config_registration.py` tests — these exercise the
refactored `OpenAIVectorStoreMixin`.

**2. PyPDF file processor tests (20/20 pass)**

```bash
uv run --group test pytest -sv tests/integration/file_processors/test_pypdf_processor.py
```

**3. Full integration suite (replay mode)**

```bash
uv run --group test pytest -sv tests/integration/ --stack-config=starter
```

Result: **4 failed, 54 passed, 639 skipped, 1 xfailed**

All 4 failures are pre-existing and unrelated:

- `test_safety_with_image` — Pydantic schema mismatch (`type: 'image'`
vs `'image_url'`)
- `test_starter_distribution_config_loads_and_resolves` /
`test_postgres_demo_distribution_config_loads` — relative path
`FileNotFoundError`
- `test_mcp_tools_list_with_schemas` — no local MCP server (`Connection
refused`)

No regressions in vector_io, file_search, or ingestion workflows.

### Manual E2E verification (with starter distro)

```bash
export OLLAMA_URL="http://localhost:11434/v1"
llama stack run starter
```

**1. Verify route is registered:**

```bash
curl -sS http://localhost:8321/v1/inspect/routes \
  | jq -r '.data[] | select(.route|test("file-processors";"i"))'
```

Expected:

```json
{
  "route": "/v1alpha/file-processors/process",
  "method": "POST",
  "provider_types": [
    "inline::pypdf"
  ]
}
```

**2. Verify OpenAPI contains the endpoint:**

```bash
curl -sS http://localhost:8321/openapi.json | rg -n "/v1alpha/file-processors/process|file-processors" -i
```

**3. Direct file upload:**

```bash
curl -sS -X POST http://localhost:8321/v1alpha/file-processors/process \
  -F "file=@/path/to/sample.pdf" | jq
```

Expected: chunks response with `metadata.processor = "pypdf"`.

**4. Via file_id:**

```bash
FILE_ID="$(curl -sS -X POST http://localhost:8321/v1/files \
  -F "file=@/path/to/sample.pdf" -F "purpose=assistants" | jq -r .id)"

curl -sS -X POST http://localhost:8321/v1alpha/file-processors/process \
  -F "file_id=${FILE_ID}" | jq
```

Expected: chunks response with `metadata.processor = "pypdf"` and
`file_id` in chunk metadata.

---------

Signed-off-by: roburishabh <roburishabh@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Charlie Doern <cdoern@redhat.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Francisco Javier Arceo <arceofrancisco@gmail.com>

Author: 5 people

.github/workflows/integration-vector-io-tests.yml

@@ -215,7 +215,7 @@ jobs:
           INFINISPAN_PASSWORD: ${{ matrix.vector-io-provider == 'remote::infinispan' && 'password' || '' }}
         run: |
           uv run --no-sync \
-            pytest -sv --stack-config="files=inline::localfs,inference=inline::sentence-transformers?trust_remote_code=true,vector_io=${{ matrix.vector-io-provider }}" \
+            pytest -sv --stack-config="files=inline::localfs,inference=inline::sentence-transformers?trust_remote_code=true,vector_io=${{ matrix.vector-io-provider }},file_processors=inline::pypdf" \
             tests/integration/vector_io
 
       - name: Check Storage and Memory Available After Tests

docs/docs/providers/file_processors/inline_pypdf.mdx

@@ -16,7 +16,6 @@ PyPDF-based file processor for extracting text content from documents.
 |-------|------|----------|---------|-------------|
 | `default_chunk_size_tokens` | `int` | No | 800 | Default chunk size in tokens when chunking_strategy type is 'auto' |
 | `default_chunk_overlap_tokens` | `int` | No | 400 | Default chunk overlap in tokens when chunking_strategy type is 'auto' |
-| `max_file_size_bytes` | `int` | No | 104857600 | Maximum file size in bytes for uploaded files (default 100MB) |
 | `extract_metadata` | `bool` | No | True | Whether to extract PDF metadata (title, author, etc.) |
 | `clean_text` | `bool` | No | True | Whether to clean extracted text (remove extra whitespace, normalize line breaks) |
 

src/llama_stack/providers/inline/file_processor/pypdf/__init__.py

@@ -17,7 +17,7 @@ async def get_provider_impl(config: PyPDFFileProcessorConfig, deps: dict[Api, An
 
     assert isinstance(config, PyPDFFileProcessorConfig), f"Unexpected config type: {type(config)}"
 
-    files_api = deps.get(Api.files)
+    files_api = deps[Api.files]
 
     impl = PyPDFFileProcessorAdapter(config, files_api)
     return impl

src/llama_stack/providers/inline/file_processor/pypdf/adapter.py

@@ -16,7 +16,7 @@
 class PyPDFFileProcessorAdapter:
     """Adapter for PyPDF file processor."""
 
-    def __init__(self, config: PyPDFFileProcessorConfig, files_api=None) -> None:
+    def __init__(self, config: PyPDFFileProcessorConfig, files_api) -> None:
         self.config = config
         self.files_api = files_api
         self.processor = PyPDFFileProcessor(config, files_api)

src/llama_stack/providers/inline/file_processor/pypdf/config.py

@@ -28,12 +28,6 @@ class PyPDFFileProcessorConfig(BaseModel):
         description="Default chunk overlap in tokens when chunking_strategy type is 'auto'",
     )
 
-    max_file_size_bytes: int = Field(
-        default=100 * 1024 * 1024,
-        ge=1,
-        description="Maximum file size in bytes for uploaded files (default 100MB)",
-    )
-
     # PDF-specific options
     extract_metadata: bool = Field(default=True, description="Whether to extract PDF metadata (title, author, etc.)")
 

src/llama_stack/providers/inline/file_processor/pypdf/pypdf.py

@@ -5,10 +5,12 @@
 # the root directory of this source tree.
 
 import io
+import mimetypes
 import time
 import uuid
 from typing import Any
 
+import chardet
 from fastapi import HTTPException, UploadFile
 from pypdf import PdfReader
 
@@ -33,7 +35,7 @@
 class PyPDFFileProcessor:
     """PyPDF-based file processor for PDF documents."""
 
-    def __init__(self, config: PyPDFFileProcessorConfig, files_api=None) -> None:
+    def __init__(self, config: PyPDFFileProcessorConfig, files_api) -> None:
         self.config = config
         self.files_api = files_api
 
@@ -44,72 +46,73 @@ async def process_file(
         options: dict[str, Any] | None = None,
         chunking_strategy: VectorStoreChunkingStrategy | None = None,
     ) -> ProcessFileResponse:
-        """Process a PDF file and return chunks."""
+        """Process a file and return chunks. Supports PDF and text-based files."""
 
-        # Validate input
         if not file and not file_id:
             raise ValueError("Either file or file_id must be provided")
         if file and file_id:
             raise ValueError("Cannot provide both file and file_id")
 
         start_time = time.time()
 
-        # Get PDF content
+        # Upload size limits are enforced by the router layer (upload_safety.py).
+        # The provider trusts that `file` has already been bounded-read and
+        # `file_id` references a file accepted by the Files API.
         if file:
-            # Read from uploaded file
             content = await file.read()
-            if len(content) > self.config.max_file_size_bytes:
-                raise ValueError(
-                    f"File size {len(content)} bytes exceeds maximum of {self.config.max_file_size_bytes} bytes"
-                )
             filename = file.filename or f"{uuid.uuid4()}.pdf"
         elif file_id:
-            # Get file from file storage using Files API
-            if not self.files_api:
-                raise ValueError("Files API not available - cannot process file_id")
-
-            # Get file metadata
             file_info = await self.files_api.openai_retrieve_file(RetrieveFileRequest(file_id=file_id))
             filename = file_info.filename
 
-            # Get file content
             content_response = await self.files_api.openai_retrieve_file_content(
                 RetrieveFileContentRequest(file_id=file_id)
             )
             content = content_response.body
 
+        mime_type, _ = mimetypes.guess_type(filename)
+        mime_category = mime_type.split("/")[0] if (mime_type and "/" in mime_type) else None
+
+        if mime_type == "application/pdf":
+            return self._process_pdf(content, filename, file_id, chunking_strategy, start_time)
+        elif mime_category == "text":
+            return self._process_text(content, filename, file_id, chunking_strategy, start_time)
+        else:
+            # Attempt text decoding as a fallback for unknown types
+            log.warning("Unknown mime type, attempting text extraction", mime_type=mime_type, filename=filename)
+            return self._process_text(content, filename, file_id, chunking_strategy, start_time)
+
+    def _process_pdf(
+        self,
+        content: bytes,
+        filename: str,
+        file_id: str | None,
+        chunking_strategy: VectorStoreChunkingStrategy | None,
+        start_time: float,
+    ) -> ProcessFileResponse:
+        """Process a PDF file."""
         pdf_bytes = io.BytesIO(content)
         reader = PdfReader(pdf_bytes)
 
         if reader.is_encrypted:
             raise HTTPException(status_code=422, detail="Password-protected PDFs are not supported")
 
-        # Extract text from PDF
         text_content, failed_pages = self._extract_pdf_text(reader)
 
-        # Clean text if configured
         if self.config.clean_text:
             text_content = self._clean_text(text_content)
 
-        # Extract metadata if configured
         pdf_metadata = {}
         if self.config.extract_metadata:
             pdf_metadata = self._extract_pdf_metadata(reader)
 
         document_id = str(uuid.uuid4())
-
-        # Prepare document metadata (include filename and file_id)
-        document_metadata = {
-            "filename": filename,
-            **pdf_metadata,
-        }
+        document_metadata: dict[str, Any] = {"filename": filename, **pdf_metadata}
         if file_id:
             document_metadata["file_id"] = file_id
 
         processing_time_ms = int((time.time() - start_time) * 1000)
-
-        # Create response metadata
-        response_metadata = {
+        response_metadata: dict[str, Any] = {
             "processor": "pypdf",
             "processing_time_ms": processing_time_ms,
             "page_count": pdf_metadata.get("page_count", 0),
@@ -120,13 +123,48 @@ async def process_file(
         if failed_pages:
             response_metadata["failed_pages"] = failed_pages
 
-        # Handle empty text - return empty chunks with metadata
         if not text_content or not text_content.strip():
             return ProcessFileResponse(chunks=[], metadata=response_metadata)
 
-        # Create chunks for non-empty text
         chunks = self._create_chunks(text_content, document_id, chunking_strategy, document_metadata)
+        return ProcessFileResponse(chunks=chunks, metadata=response_metadata)
+
+    def _process_text(
+        self,
+        content: bytes,
+        filename: str,
+        file_id: str | None,
+        chunking_strategy: VectorStoreChunkingStrategy | None,
+        start_time: float,
+    ) -> ProcessFileResponse:
+        """Process a text-based file (txt, csv, md, etc.)."""
+        detected = chardet.detect(content)
+        encoding = detected["encoding"] or "utf-8"
+        try:
+            text_content = content.decode(encoding)
+        except UnicodeDecodeError:
+            text_content = content.decode("utf-8", errors="replace")
 
+        if self.config.clean_text:
+            text_content = self._clean_text(text_content)
+
+        document_id = str(uuid.uuid4())
+        document_metadata: dict[str, Any] = {"filename": filename}
+        if file_id:
+            document_metadata["file_id"] = file_id
+
+        processing_time_ms = int((time.time() - start_time) * 1000)
+        response_metadata: dict[str, Any] = {
+            "processor": "text",
+            "processing_time_ms": processing_time_ms,
+            "extraction_method": "text",
+            "file_size_bytes": len(content),
+        }
+
+        if not text_content or not text_content.strip():
+            return ProcessFileResponse(chunks=[], metadata=response_metadata)
+
+        chunks = self._create_chunks(text_content, document_id, chunking_strategy, document_metadata)
         return ProcessFileResponse(chunks=chunks, metadata=response_metadata)
 
     def _extract_pdf_text(self, reader: PdfReader) -> tuple[str, list[str]]:
@@ -140,8 +178,9 @@ def _extract_pdf_text(self, reader: PdfReader) -> tuple[str, list[str]]:
             except Exception as e:
                 failed_pages.append(f"page {page_num + 1}: {e}")
                 continue
-            if page_text and page_text.strip():
-                text_parts.append(page_text)
+            if page_text:
+                if not self.config.clean_text or page_text.strip():
+                    text_parts.append(page_text)
 
         return "\n".join(text_parts), failed_pages
 
@@ -209,6 +248,9 @@ def _create_chunks(
         elif chunking_strategy.type == "static":
             chunk_size = chunking_strategy.static.max_chunk_size_tokens
             overlap_size = chunking_strategy.static.chunk_overlap_tokens
+        elif chunking_strategy.type == "contextual":
+            chunk_size = chunking_strategy.contextual.max_chunk_size_tokens
+            overlap_size = chunking_strategy.contextual.chunk_overlap_tokens
         else:
             chunk_size = self.config.default_chunk_size_tokens
             overlap_size = self.config.default_chunk_overlap_tokens

src/llama_stack/providers/inline/vector_io/faiss/faiss.py

@@ -344,7 +344,7 @@ async def query_hybrid(
 
 
 class FaissVectorIOAdapter(OpenAIVectorStoreMixin, VectorIO, VectorStoresProtocolPrivate):
-    """Vector I/O adapter using FAISS for in-memory vector similarity search."""
+    """VectorIO adapter that uses FAISS for similarity search and vector storage."""
 
     def __init__(
         self,

src/llama_stack/providers/remote/vector_io/weaviate/weaviate.py

@@ -290,7 +290,7 @@ async def query_hybrid(
 
 
 class WeaviateVectorIOAdapter(OpenAIVectorStoreMixin, VectorIO, VectorStoresProtocolPrivate):
-    """Vector I/O adapter for remote Weaviate instances."""
+    """VectorIO adapter that uses Weaviate for similarity search and vector storage."""
 
     def __init__(
         self,