Add 5 database connectors, fix SQL injection, and standardize agent result models#4
Open
yelinso344 wants to merge 1 commit into
Open
Add 5 database connectors, fix SQL injection, and standardize agent result models#4yelinso344 wants to merge 1 commit into
yelinso344 wants to merge 1 commit into
Conversation
Owner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
body=API breakage (removed in v8+)databricks-sql-connectorpackagerequirements.txt,README.md, and.env.exampleto reflect all changesChanges
New connectors (
src/connectors/)mongodb.py— URI-based, returns collection as DataFrameredshift.py— usesredshift-connector, mirrors PostgreSQL interfaceduckdb_conn.py— local file-based, uses.df()native returnfabric.py— ODBC Driver 18 for SQL Server (OS-level dep, noted in README)elasticsearch_conn.py— optional auth, SSL toggle, fetches index as DataFrameSecurity fix — SQL injection (
src/connectors/)Added
_validate_identifier()topostgres.py,mysql.py,databricks.py,snowflake_conn.py, andbigquery.py. Rejects any identifier containingcharacters outside
A-Za-z0-9_.before SQL is executed.Pydantic standardisation (
src/agents/)Converted
PDFParserResult,EntityExtractorResult,RiskDetectorResult,and
ActionExtractorResultfrom hand-rolled classes with manualmodel_dump()to
pydantic.BaseModel— consistent with all CSV pipeline agents.Bug fixes
elasticsearch_conn.py— replaced deprecatedbody=kwarg with direct params (v8+ API)databricks.py—databricks-sql-connectorwas missing from venv and requirementsDocs & config
README.md— updated connector status table, architecture tree, code examples,roadmap, tech stack, contributing guide, and agent pattern snippet
.env.example— added entries for all 10 connectorsrequirements.txt— regenerated from venv (adds pymongo, redshift-connector,duckdb, elasticsearch, pyodbc, databricks-sql-connector)
Test plan
pytest tests/ -v— all 11 existing tests pass.duckdbfile, fetch table, run pipelineusers; DROP TABLE users