Skip to content

Commit 53740cb

Browse files
committed
perf(converter): lazy-import markitdown to cut startup and enable slim packaging
markitdown is only used for the non-PDF, non-Markdown ingest branch, but it was imported at module top level — so every `import openkb` eagerly pulled in markitdown → magika → onnxruntime (tens of MB, slow). Move the import into the branch that actually uses it. Effects (measured): - `import openkb.cli` cold start 4.64s → 3.53s (~24% faster); magika / onnxruntime / markitdown no longer load unless an Office-format document is converted. - A packaged (PyInstaller) build can now exclude magika/onnxruntime for a slimmer artifact when Office-format ingest isn't needed. Previously the unconditional module-level markitdown import made magika load-bearing at startup, so excluding it crashed the whole app on launch. markitdown appends warning filters on import but leaves the CLI's front "ignore" filter at index 0, so no re-suppression is needed at the new import site. test_converter patches `markitdown.MarkItDown` (the real attribute the local import binds) instead of the removed `openkb.converter.MarkItDown`. Claude-Session: https://claude.ai/code/session_01KZyUSGAzVL9yxpsWWPv6Y2
1 parent 7cde496 commit 53740cb

2 files changed

Lines changed: 10 additions & 3 deletions

File tree

openkb/converter.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@
1111
from pathlib import Path
1212

1313
import pymupdf
14-
from markitdown import MarkItDown
1514

1615
from openkb.config import load_config
1716
from openkb.images import convert_pdf_with_images, copy_relative_images, extract_base64_images
@@ -230,7 +229,15 @@ def convert_document(
230229
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
231230
markdown = convert_pdf_with_images(src, doc_name, images_dir)
232231
else:
233-
# Non-PDF, non-MD: use markitdown (docx, pptx, html, etc.)
232+
# Non-PDF, non-MD: use markitdown (docx, pptx, html, etc.).
233+
# Imported lazily: markitdown pulls in magika → onnxruntime (tens
234+
# of MB, slow to import) that only this branch needs. Keeping it
235+
# out of module import makes `import openkb` cheap and lets a
236+
# packaged build drop those deps when Office-format ingest isn't
237+
# required. markitdown appends warning filters on import but leaves
238+
# the CLI's front "ignore" filter in place, so no re-suppress here.
239+
from markitdown import MarkItDown
240+
234241
mid = MarkItDown()
235242
result = mid.convert(str(src), keep_data_uris=True)
236243
markdown = result.text_content

tests/test_converter.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ def test_docx_conversion_enables_keep_data_uris(self, kb_dir, tmp_path):
145145
mock_result.text_content = "![](data:image/png;base64,abc123)"
146146

147147
with (
148-
patch("openkb.converter.MarkItDown") as mock_markitdown,
148+
patch("markitdown.MarkItDown") as mock_markitdown,
149149
patch(
150150
"openkb.converter.extract_base64_images",
151151
return_value="converted markdown",

0 commit comments

Comments
 (0)