Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

378 update submission processing to use polars data validator #394

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

jcadam14
Copy link
Contributor

@jcadam14 jcadam14 commented Sep 5, 2024

Closes #378

Changes the filing-api in the following ways:

  • When the content is uploaded, use python csv to get the row count since chunking generators in data-validator no longer provide final counts of rows in file
  • Use generator from validator to get findings dataframe, and collate into final dataframe
  • Updated to generate a file path that's either local or s3://, which the data-validator can use with fsspec to correctly use either way
  • Calls new df_to_download to write out s3, the data validator handles the writing
  • Updated build_validation_results to use the collated findings dataframe to build out the json for the frontend, ValidationResults are no longer used in the filing-api
  • Updated the pytests
  • Updated boto3 to stick to version 1.34 since fsspec and s3fs (which is used in the data-validator) don't currently support botocore 1.35. When the data-validator is separated, we can switch back to the latest boto3

@jcadam14 jcadam14 self-assigned this Sep 5, 2024
@jcadam14 jcadam14 linked an issue Sep 5, 2024 that may be closed by this pull request
Copy link

github-actions bot commented Sep 5, 2024

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/sbl_filing_api/routers
  filing.py
  src/sbl_filing_api/services
  submission_processor.py 64
Project Total  

This report was generated by python-coverage-comment-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update submission processing to use polars data-validator
1 participant