feat(docker): Comprehensive Docker Setup Enhancement with Performance & Security Improvements #890

kzndotsh · 2025-06-06T10:41:40Z

No description provided.

…curity improvements Add comprehensive performance testing and monitoring scripts to track Docker build and runtime efficiency. Introduce a new GitHub Actions workflow for automated Docker performance testing, including nightly runs and PR comments. Implement security best practices in Docker setup, such as non-root user execution, read-only filesystems, and resource limits. Update Dockerfile and docker-compose files to reflect these changes. These enhancements aim to ensure the Docker setup is efficient, secure, and maintainable. By automating performance testing and monitoring, potential regressions can be identified early, and security vulnerabilities can be mitigated. The changes also facilitate a more robust CI/CD pipeline, improving overall development workflow.

…old handling Update the environment setup to distinguish between development and production configurations by introducing separate environment variables for database URLs and bot tokens. This change allows for more flexible testing scenarios by simulating different environments. Refactor the performance threshold checks to use a single failure flag (`THRESHOLD_FAILED`) instead of multiple environment variables. This simplifies the logic and improves readability. The step to fail the job if thresholds are exceeded is now integrated into the threshold checking logic, providing immediate feedback and reducing redundancy. These changes improve the workflow's maintainability and adaptability to different testing environments, ensuring more robust and clear performance testing.

…mance and flexibility - Update `.dockerignore` to include all markdown files except `README.md` and `requirements.md`, ensuring essential documentation is included in the build context. - Modify `Dockerfile` to enable parallel installation with Poetry, initialize a git repository for build processes, and perform aggressive cleanup for optimized production images. This reduces image size and enhances security by removing unnecessary files and metadata. - Add labels to the production image for better traceability and metadata management. - Introduce a non-root user in the production stage for enhanced security. - Update `scripts/test-docker.sh` to support command-line arguments for cache control and aggressive cleanup, improving test flexibility and performance. - Implement a cleanup function to manage Docker resources efficiently, preventing resource leaks and ensuring a clean test environment. - Adjust Docker run commands to use `--entrypoint=""` for more controlled execution during tests. These changes aim to optimize the Docker build process, reduce image size, and enhance security by removing unnecessary files. The test script improvements provide more control over the testing environment, allowing for more accurate performance assessments.

Update the Ruff version in the pre-commit configuration to v0.11.13 to ensure consistency with the formatter version. Change the hook id from 'ruff' to 'ruff-check' to align with the updated configuration. refactor(Dockerfile): update image metadata for clarity Modify the Dockerfile to update the image description and author/vendor fields for better clarity and readability. This change ensures that the metadata accurately reflects the project's branding as "All Things Linux". feat(cli/docker): enhance Docker CLI with additional commands and options Introduce new commands and options to the Docker CLI for improved functionality and user experience. Add commands for checking Docker availability, managing Tux-related resources, and executing various Docker operations. These enhancements provide more control and flexibility in managing Docker services and resources, especially for development and testing purposes.

Remove the `security_opt: no-new-privileges:true` option from the docker-compose development configuration. This change is made to simplify the configuration and avoid potential issues with services that require elevated privileges during development. The removal ensures that the development environment is less restrictive, which can be beneficial for debugging and testing purposes.

…ble thresholds Update the performance benchmarks section to include default performance thresholds that can be configured via environment variables. This change provides flexibility for different environments and hardware capabilities, allowing users to set custom thresholds for build time, startup time, Prisma generation, and memory usage. test(test-docker.sh): add performance threshold checks and memory parsing Enhance the test script to include performance threshold checks based on configurable environment variables. Add detailed logging for each performance metric, and implement memory usage parsing to handle various units (B, KiB, MiB, etc.). This ensures that performance metrics are within acceptable ranges and provides feedback for optimization if thresholds are exceeded.

…sting documentation Improve Docker cleanup scripts to ensure only test-related resources are removed, preserving system images and containers. Introduce detailed documentation for Docker testing strategies, including safety guidelines, comprehensive testing scenarios, and recovery procedures. The changes aim to prevent accidental removal of critical system resources during Docker cleanup processes, ensuring that only test-specific images and containers are affected. This enhances the reliability and safety of the CI/CD pipeline and local development environments. Comprehensive testing documentation provides clear guidance on executing various test scenarios, ensuring robust Docker functionality across all developer workflows.

Remove outdated Docker-related documentation and scripts to streamline the project and reduce maintenance overhead. The removed files include guides on Docker cleanup safety, security, testing strategies, and comprehensive testing scripts. These documents and scripts were replaced by a new unified Docker guide (DOCKER.md) that consolidates all relevant information and provides a more efficient and updated approach to Docker management. The new guide offers a comprehensive overview of the Docker setup, including performance improvements, testing strategies, security measures, and practical usage, making the old documents redundant. This change simplifies the documentation structure and ensures that all Docker-related information is current and easily accessible. feat(scripts): introduce unified docker management script Add `docker-toolkit.sh` to consolidate Docker operations such as testing, monitoring, and management into a single script. This script replaces multiple individual scripts (`monitor-resources.sh`, `quick-docker-test.sh`, `test-docker.sh`) to streamline Docker management tasks. The new script provides a comprehensive set of commands for quick validation, standard performance testing, comprehensive testing, resource monitoring, and safe cleanup of Docker resources. It also includes detailed logging and metrics collection for performance analysis. The change aims to simplify Docker operations by providing a single entry point for all related tasks, improving maintainability and usability for developers. The unified script ensures consistent execution and reporting across different Docker operations.

.github/workflows/docker-test.yml

… script

Replace `exec poe try run tux --dev start` with `exec poetry run tux --dev start` to ensure the correct execution of the development server. The previous command contained a typo or incorrect command sequence, which could lead to runtime errors or unexpected behavior during the development phase. This change ensures that the application starts correctly in development mode using the intended command.

…rinciple

…tch configuration

…cker-toolkit.sh

… logs behavior and healthcheck functionality

…nce - addresses Sourcery AI performance suggestion

…LW1510 linter rule

The PERFORMANCE-MONITORING.md file is deleted to reduce redundancy and streamline the documentation. The information contained in this file may have been moved to a more appropriate location or deemed unnecessary for the current project scope. This change helps in maintaining a cleaner and more focused documentation set, ensuring that only relevant and up-to-date information is available to the developers and users.

tux/cli/docker.py

…e workflow Add `MANUAL_COMMIT_REF` and `MANUAL_BASE_REF` environment variables to the maintenance workflow to allow for more flexible manual triggering of the workflow with specific commit references. This enhancement provides better control over the workflow execution when manual intervention is required. refactor(docker.py): modularize and enhance subprocess command validation Refactor the `_safe_subprocess_run` function to improve readability and maintainability by breaking down the logic into smaller, self-contained functions. This change enhances the security and clarity of the subprocess execution by clearly defining validation and sanitization steps. The refactoring also ensures that resource name validation is more robust and modular, reducing the risk of command injection vulnerabilities.

…nsistency Update documentation and scripts to reflect the change in container naming from 'app' to 'tux'. This change ensures consistency across all documentation and scripts, aligning with the naming convention used in the Docker setup. build(Dockerfile): remove git repo after dynamic versioning for cleaner builds Initialize a minimal git repository for Poetry dynamic versioning and remove it after use to keep the Docker image clean. This change ensures that the build environment remains uncluttered, reducing potential issues with leftover files. perf(scripts): calculate averages dynamically based on iterations Modify the script to calculate average build times dynamically based on the actual number of iterations. This change allows for more accurate performance metrics, especially when the number of iterations varies. fix(cli): remove dangling Docker images using built-in filter Implement a method to remove all dangling Docker images using Docker's built-in filter. This change helps in maintaining a clean Docker environment by automatically removing unused images, thus freeing up space and reducing clutter.

…best practices Revise the Dockerfile to include detailed comments and documentation for each stage of the multi-stage build process. This improves understanding and maintainability by clearly explaining the purpose and impact of each stage, as well as the security and optimization measures implemented. The changes also include comprehensive usage examples and best practices, making it easier for developers to understand the build process and customize it for different environments. This is crucial for ensuring consistency, security, and efficiency across development and production deployments.

Add new commands to the CLI usage documentation to provide users with more comprehensive guidance on available Docker operations. This includes commands for opening a shell, showing running containers, restarting services, checking health status, cleaning up resources, validating configurations, and pulling images. refactor(Dockerfile): move Prisma setup to non-root user section Reorganize the Dockerfile to run Prisma setup commands as the non-root user. This change addresses permission issues and aligns with security best practices by ensuring that all operations are executed with minimal privileges. docs(docker-compose.dev.yml): enhance documentation for development Add extensive comments and documentation to the development Docker Compose file. This includes detailed explanations of development features, workflow optimizations, and best practices to assist developers in setting up and using the development environment effectively. docs(docker-compose.yml): improve production Docker Compose documentation Enhance the production Docker Compose file with comprehensive comments and documentation. This provides clarity on security features, resource management, monitoring, and operational excellence, ensuring that the production environment is configured for stability and efficiency.

Introduce .editorconfig to enforce consistent coding styles across different editors and IDEs. This includes settings for indentation, charset, and line endings for various file types. The .gitattributes file is added to ensure consistent line endings and to handle binary files appropriately. These changes aim to improve code readability and maintainability by standardizing the development environment across the team. chore(pre-commit): update pre-commit config for markdown and prettier Exclude Markdown files from trailing whitespace removal to preserve intentional line breaks. Update prettier configuration to align with .editorconfig settings, ensuring consistent formatting across YAML and JSON files. This enhances the development workflow by maintaining formatting consistency and preventing unnecessary diffs.

… execution Switching from `poetry run` to explicit Python module execution ensures that the Prisma commands are run in the correct virtual environment. This change improves reliability by directly invoking the Python interpreter from the virtual environment, reducing potential issues with environment path resolution.

Convert the Docker toolkit from a bash script to a Python script for better maintainability and integration. This change introduces a new Python-based Docker toolkit that consolidates all Docker operations, including testing, monitoring, and management. The Python script provides improved error handling, logging, and flexibility in executing Docker commands. It also supports additional testing options such as quick and comprehensive tests, enhancing the overall testing strategy for Docker environments. The transition to Python allows for easier maintenance and integration with other Python-based tools and libraries.

…ation Introduce a new test job in the CI workflow to ensure Python code changes are tested. This job checks for changes in Python files and test-related files, installs dependencies using Poetry, and runs tests with coverage reporting. This ensures that any modifications to the codebase are validated through automated testing, improving code quality and reliability. Additionally, enhance Docker validation by supporting both Docker Compose v1 and v2. This change ensures compatibility with different environments where either version might be installed, preventing potential build failures due to version mismatches. The .env file creation for Docker validation is also simplified for consistency.

Include a step to generate the Prisma client in the CI workflow before running tests and format checks. This ensures that the Prisma client is up-to-date and available, preventing potential errors during the CI process due to missing or outdated client files. This change enhances the reliability and consistency of the CI pipeline by ensuring all necessary code is generated before execution.

…roups Change the poetry install command to use the `--with` option for better clarity and to ensure all necessary dependency groups are installed. In the first job, the `dev` and `types` groups are explicitly included, ensuring development and type-checking dependencies are installed. In the second job, the `dev`, `test`, and `types` groups are included to ensure all dependencies required for testing and development are available. This change improves the reliability and consistency of the CI workflow by ensuring all necessary dependencies are installed.

The `fail_ci_if_error` option is removed from the Codecov action configuration. This option was set to false, which means that any errors in the Codecov action would not cause the CI to fail. By removing this option, the default behavior is used, which is to fail the CI if there is an error in the Codecov action. This change ensures that any issues with code coverage reporting are caught and addressed promptly, maintaining the integrity of the CI process.

cursor

✅ BugBot reviewed your changes and found no bugs!

Was this report helpful? Give feedback by reacting with 👍 or 👎

Set a coverage threshold with `--cov-fail-under=0` to ensure that the test suite does not fail due to coverage percentage. This allows the tests to pass regardless of coverage, which is useful during initial setup or when coverage is not a priority. Additionally, set `fail_ci_if_error: false` for the Codecov action to prevent CI failures if there are issues with uploading coverage data, ensuring that CI pipelines are not blocked by external service issues.

codecov · 2025-06-09T07:48:51Z

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

…st coverage Introduce a new Codecov configuration file to manage coverage reporting and status checks for the Tux bot project. This setup enforces different coverage targets and thresholds for various components, ensuring critical areas like the database layer and error handlers maintain higher coverage standards. Enhance the GitHub Actions CI workflow to run unit, database, and integration tests separately, each with dedicated coverage reports and flags. This separation allows for more granular control over coverage reporting and better insights into specific test areas. Uploading test results to Codecov provides a comprehensive view of test performance and coverage, aiding in maintaining code quality.

Introduce a Codecov configuration file to manage code coverage reporting for the Tux bot project. This configuration sets global coverage precision, rounding, and range, and defines status checks for pull requests. It establishes coverage targets and thresholds for different project components, ensuring critical areas like the database and core functionalities have higher standards. The file also configures pull request comments, ignores specific paths from coverage calculations, and manages components and flags. This enhancement aims to improve code quality by enforcing coverage standards and providing detailed insights into code coverage throughout the development process.

Introduce a comprehensive Codecov configuration file to enhance code coverage tracking and reporting for the Tux Discord Bot project. The new .codecov.yml file provides a more detailed and structured approach to coverage management, including tiered coverage standards, component- based tracking, and intelligent CI integration. This change aims to improve the accuracy and granularity of coverage reports, ensuring higher standards for critical components and better feedback for developers. The previous codecov.yml file is removed to avoid duplication and potential conflicts.

kzndotsh added 8 commits June 6, 2025 02:17

This comment was marked as outdated.

Sign in to view

kzndotsh requested review from RainzDev, electron271 and anemoijereja-eden June 6, 2025 10:42

github-advanced-security bot found potential problems Jun 6, 2025

View reviewed changes

.github/workflows/docker-test.yml Fixed Show fixed Hide fixed

This comment was marked as outdated.

Sign in to view

kzndotsh added 9 commits June 6, 2025 06:54

fix(ci): update Docker test workflow to use unified docker-toolkit.sh…

7097201

… script

fix(ci): add explicit permissions to jobs following least privilege p…

414a5e3

…rinciple

docs(DOCKER.md): fix YAML syntax error with duplicate path keys in wa…

c7f30ca

…tch configuration

docs(DOCKER.md): fix inconsistent script references to use unified do…

264d7de

…cker-toolkit.sh

fix(docker): move return statement to else block per TRY300 linter rule

08f071e

fix(docker): address additional Sourcery AI review feedback - improve…

5b82662

… logs behavior and healthcheck functionality

perf(docker): compile regex patterns outside loop for better performa…

388cea5

…nce - addresses Sourcery AI performance suggestion

fix(docker): explicit check parameter for subprocess.run to satisfy P…

8438f7f

…LW1510 linter rule

Uh oh!

feat(docker): Comprehensive Docker Setup Enhancement with Performance & Security Improvements #890

feat(docker): Comprehensive Docker Setup Enhancement with Performance & Security Improvements #890

Uh oh!

Conversation

kzndotsh commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as resolved.

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

✅ BugBot reviewed your changes and found no bugs!

Uh oh!

codecov bot commented Jun 9, 2025

Welcome to Codecov 🎉

Uh oh!

Uh oh!

kzndotsh commented Jun 6, 2025 •

edited

Loading