Add parallelized-embedding-pipeline #403
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
August 28, 2025 12:41
- Enhanced README.md with workflow diagram and detailed architecture - Added complete example-workflow.json with all required metadata - Created resources folder with workflow diagram and author photos - Added professional author information for Solomon Ojo and Dave Horne - Included comprehensive deployment guides and resource links - Ready for AWS Step Functions workflows collection contribution
- Complete document vectorization pipeline implementation - Enhanced README.md with workflow diagram and comprehensive documentation - Added example-workflow.json with all required metadata for AWS samples - Included resources folder with workflow diagram and author photos - Added professional author information for Solomon Ojo and Dave Horne - All Lambda functions, deployment scripts, and configuration files - Ready for production use and AWS Step Functions workflows collection
bfreiberg
suggested changes
Sep 18, 2025
Co-authored-by: Ben <[email protected]>
Co-authored-by: Ben <[email protected]>
Added a section detailing the data processing layers in the pipeline, explaining the purpose of each layer and its role in the data transformation workflow.
Removed quick start deployment instructions from README.
Updated resource links and author information in workflow metadata.
Updated the state machine to handle document processing and vector embedding generation.
Updated the AWS CloudFormation template to enhance the document processing workflow by modifying descriptions, adjusting security group rules, and updating Lambda function runtimes to Python 3.12. Removed unnecessary resources and added outputs for database secret and cluster ARNs.
Updated AvailabilityZone retrieval and added a custom S3 policy for Lambda.
…ries layer - Reduced DatabaseInitFunction timeout from 300s to 120s (2 minutes) - Created missing functions/shared directory with security utilities - Added shared libraries for secure XML parsing and subprocess execution - Updated template.yaml with proper layer structure - Cleaned up demo configuration
bfreiberg
approved these changes
Oct 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This project contains a document vectorization pipeline using AWS services, specifically designed to process text, PDF, and Word documents, extract their content, generate vector embeddings in parallel and store them in a PostgreSQL database optimized for vector searches.
Thank you
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.