Skip to content

Add 5 database connectors, fix SQL injection, and standardize agent result models#4

Open
yelinso344 wants to merge 1 commit into
harshitboots:mainfrom
yelinso344:main
Open

Add 5 database connectors, fix SQL injection, and standardize agent result models#4
yelinso344 wants to merge 1 commit into
harshitboots:mainfrom
yelinso344:main

Conversation

@yelinso344

Copy link
Copy Markdown

Summary

  • Add MongoDB, Amazon Redshift, DuckDB, Microsoft Fabric, and Elasticsearch connectors
  • Fix SQL injection vulnerability across all SQL connectors
  • Standardise PDF agent result classes to Pydantic BaseModel (consistent with CSV agents)
  • Fix Elasticsearch body= API breakage (removed in v8+)
  • Install missing databricks-sql-connector package
  • Update requirements.txt, README.md, and .env.example to reflect all changes

Changes

New connectors (src/connectors/)

  • mongodb.py — URI-based, returns collection as DataFrame
  • redshift.py — uses redshift-connector, mirrors PostgreSQL interface
  • duckdb_conn.py — local file-based, uses .df() native return
  • fabric.py — ODBC Driver 18 for SQL Server (OS-level dep, noted in README)
  • elasticsearch_conn.py — optional auth, SSL toggle, fetches index as DataFrame

Security fix — SQL injection (src/connectors/)

Added _validate_identifier() to postgres.py, mysql.py, databricks.py,
snowflake_conn.py, and bigquery.py. Rejects any identifier containing
characters outside A-Za-z0-9_. before SQL is executed.

Pydantic standardisation (src/agents/)

Converted PDFParserResult, EntityExtractorResult, RiskDetectorResult,
and ActionExtractorResult from hand-rolled classes with manual model_dump()
to pydantic.BaseModel — consistent with all CSV pipeline agents.

Bug fixes

  • elasticsearch_conn.py — replaced deprecated body= kwarg with direct params (v8+ API)
  • databricks.pydatabricks-sql-connector was missing from venv and requirements

Docs & config

  • README.md — updated connector status table, architecture tree, code examples,
    roadmap, tech stack, contributing guide, and agent pattern snippet
  • .env.example — added entries for all 10 connectors
  • requirements.txt — regenerated from venv (adds pymongo, redshift-connector,
    duckdb, elasticsearch, pyodbc, databricks-sql-connector)

Test plan

  • Run pytest tests/ -v — all 11 existing tests pass
  • CSV pipeline — upload demo CSV, run all 5 agents
  • PDF pipeline — upload demo PDF, run all 5 agents
  • MongoDB — connect to local instance, fetch a collection, run pipeline
  • Redshift — connect, fetch table, run pipeline
  • DuckDB — point at a .duckdb file, fetch table, run pipeline
  • Elasticsearch — connect to local instance, fetch an index
  • Microsoft Fabric — requires ODBC Driver 18 installed at OS level
  • Verify SQL injection guard rejects a table name like users; DROP TABLE users

@harshitboots

Copy link
Copy Markdown
Owner

This PR was created from the same base commit as PR #2 (DuckDB), and likely PR #3 (Redshift). Since then:

  1. Rebase onto main
  2. Resolve Conflicts
  3. Force Push the Rebased Branch

kindly rebase it seems ok to merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants