Optimize parallel file processing in folder scan and fix dead lock problem in Gunicorn multi workers mode

[danielaskdd]

Description

Optimize parallel file processing in folder scan and fix dead lock problem in Gunicorn multi workers mode

Related Issues

n/a

Changes Made

• Enqueue files to pipeline batch by batch to utilize parallel documents processing ability of the index stage
• Changed parallel enqueue to sequential to prevent CPU overload by heavy file extraction jobs
• Resolve dead lock problem which occur in parallel documents processing with Gunicorn mode
• Prioritize OS environment variables over .env file to improve Docker compatibility for the server
• Increased default timeout values for Gunicorn configuration

Checklist

• Changes tested locally
• Code reviewed
• Documentation updated (if necessary)
• Unit tests added (if applicable)