OCR and beyond: Reading data from unstructured sources

[amotl]

Pitch

Is unstructured text input out of scope for omniload? Who defines its features boundaries?

Example: PDF extraction

What about loading a bunch of PDF documents into a spreadsheet, where each row represents a document, and each column cell includes content from a specific section of the document?

References

• https://github.com/baidu/Unlimited-OCR (discuss)

[amotl]

xberg

Document intelligence: text, tables, metadata from 91+ formats with optional OCR.

https://github.com/xberg-io/xberg
https://docs.xberg.io/

[amotl]

text2ics

With text2ics by @jakob1379, and another miniature pipeline element to import iCalendar to an RDBMS database, omniload could provide the whole shebang of extracting events from unstructured text files, and creating a structured tabular feed stored in a database table for easier searching and other purposes.

[jakob1379]

What the, how'd I end up here? Lovely project BTW, feel very similar in spirit to https://steampipe.io/ if I understand it correctly? 😁

[amotl]

Yes omniload is certainly similar, but not Zero-ETL, but real incremental ETL based on dlt. It looks like relevant plugins of Steampipe could benefit its polyglottiness where applicable, thank you!

[amotl]

Any format to Markdown

We knew the venerable Markitdown, but anydoc was new to us. Thank you, @tomsideguide!

• https://github.com/microsoft/markitdown
• https://github.com/firecrawl/anydoc