Skip to content

feat(docker): Comprehensive Docker Setup Enhancement with Performance & Security Improvements #890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 148 commits into
base: main
Choose a base branch
from

Conversation

kzndotsh
Copy link
Contributor

@kzndotsh kzndotsh commented Jun 6, 2025

No description provided.

kzndotsh added 8 commits June 6, 2025 02:17
…curity improvements

Add comprehensive performance testing and monitoring scripts to track
Docker build and runtime efficiency. Introduce a new GitHub Actions
workflow for automated Docker performance testing, including nightly
runs and PR comments. Implement security best practices in Docker
setup, such as non-root user execution, read-only filesystems, and
resource limits. Update Dockerfile and docker-compose files to reflect
these changes.

These enhancements aim to ensure the Docker setup is efficient,
secure, and maintainable. By automating performance testing and
monitoring, potential regressions can be identified early, and
security vulnerabilities can be mitigated. The changes also facilitate
a more robust CI/CD pipeline, improving overall development workflow.
…old handling

Update the environment setup to distinguish between development and production configurations by introducing separate environment variables for database URLs and bot tokens. This change allows for more flexible testing scenarios by simulating different environments.

Refactor the performance threshold checks to use a single failure flag (`THRESHOLD_FAILED`) instead of multiple environment variables. This simplifies the logic and improves readability. The step to fail the job if thresholds are exceeded is now integrated into the threshold checking logic, providing immediate feedback and reducing redundancy.

These changes improve the workflow's maintainability and adaptability to different testing environments, ensuring more robust and clear performance testing.
…mance and flexibility

- Update `.dockerignore` to include all markdown files except `README.md` and `requirements.md`, ensuring essential documentation is included in the build context.
- Modify `Dockerfile` to enable parallel installation with Poetry, initialize a git repository for build processes, and perform aggressive cleanup for optimized production images. This reduces image size and enhances security by removing unnecessary files and metadata.
- Add labels to the production image for better traceability and metadata management.
- Introduce a non-root user in the production stage for enhanced security.
- Update `scripts/test-docker.sh` to support command-line arguments for cache control and aggressive cleanup, improving test flexibility and performance.
- Implement a cleanup function to manage Docker resources efficiently, preventing resource leaks and ensuring a clean test environment.
- Adjust Docker run commands to use `--entrypoint=""` for more controlled execution during tests.

These changes aim to optimize the Docker build process, reduce image size, and enhance security by removing unnecessary files. The test script improvements provide more control over the testing environment, allowing for more accurate performance assessments.
Update the Ruff version in the pre-commit configuration to v0.11.13 to ensure consistency with the formatter version. Change the hook id from 'ruff' to 'ruff-check' to align with the updated configuration.

refactor(Dockerfile): update image metadata for clarity

Modify the Dockerfile to update the image description and author/vendor fields for better clarity and readability. This change ensures that the metadata accurately reflects the project's branding as "All Things Linux".

feat(cli/docker): enhance Docker CLI with additional commands and options

Introduce new commands and options to the Docker CLI for improved functionality and user experience. Add commands for checking Docker availability, managing Tux-related resources, and executing various Docker operations. These enhancements provide more control and flexibility in managing Docker services and resources, especially for development and testing purposes.
Remove the `security_opt: no-new-privileges:true` option from the
docker-compose development configuration. This change is made to
simplify the configuration and avoid potential issues with services
that require elevated privileges during development. The removal
ensures that the development environment is less restrictive, which
can be beneficial for debugging and testing purposes.
…ble thresholds

Update the performance benchmarks section to include default
performance thresholds that can be configured via environment
variables. This change provides flexibility for different
environments and hardware capabilities, allowing users to set
custom thresholds for build time, startup time, Prisma generation,
and memory usage.

test(test-docker.sh): add performance threshold checks and memory parsing

Enhance the test script to include performance threshold checks
based on configurable environment variables. Add detailed logging
for each performance metric, and implement memory usage parsing
to handle various units (B, KiB, MiB, etc.). This ensures that
performance metrics are within acceptable ranges and provides
feedback for optimization if thresholds are exceeded.
…sting documentation

Improve Docker cleanup scripts to ensure only test-related resources are removed, preserving system images and containers. Introduce detailed documentation for Docker testing strategies, including safety guidelines, comprehensive testing scenarios, and recovery procedures.

The changes aim to prevent accidental removal of critical system resources during Docker cleanup processes, ensuring that only test-specific images and containers are affected. This enhances the reliability and safety of the CI/CD pipeline and local development environments. Comprehensive testing documentation provides clear guidance on executing various test scenarios, ensuring robust Docker functionality across all developer workflows.
Remove outdated Docker-related documentation and scripts to streamline
the project and reduce maintenance overhead. The removed files include
guides on Docker cleanup safety, security, testing strategies, and
comprehensive testing scripts. These documents and scripts were
replaced by a new unified Docker guide (DOCKER.md) that consolidates
all relevant information and provides a more efficient and updated
approach to Docker management.

The new guide offers a comprehensive overview of the Docker setup,
including performance improvements, testing strategies, security
measures, and practical usage, making the old documents redundant. This
change simplifies the documentation structure and ensures that all
Docker-related information is current and easily accessible.

feat(scripts): introduce unified docker management script

Add `docker-toolkit.sh` to consolidate Docker operations such as
testing, monitoring, and management into a single script. This script
replaces multiple individual scripts (`monitor-resources.sh`,
`quick-docker-test.sh`, `test-docker.sh`) to streamline Docker
management tasks.

The new script provides a comprehensive set of commands for quick
validation, standard performance testing, comprehensive testing,
resource monitoring, and safe cleanup of Docker resources. It also
includes detailed logging and metrics collection for performance
analysis.

The change aims to simplify Docker operations by providing a single
entry point for all related tasks, improving maintainability and
usability for developers. The unified script ensures consistent
execution and reporting across different Docker operations.

This comment was marked as outdated.

cursor-com[bot]

This comment was marked as outdated.

sourcery-ai[bot]

This comment was marked as outdated.

kzndotsh added 9 commits June 6, 2025 06:54
Replace `exec poe try run tux --dev start` with
`exec poetry run tux --dev start` to ensure the correct
execution of the development server. The previous command
contained a typo or incorrect command sequence, which could
lead to runtime errors or unexpected behavior during the
development phase. This change ensures that the application
starts correctly in development mode using the intended
command.
… logs behavior and healthcheck functionality
…nce - addresses Sourcery AI performance suggestion

This comment was marked as resolved.

cursor-com[bot]

This comment was marked as outdated.

sourcery-ai[bot]

This comment was marked as outdated.

The PERFORMANCE-MONITORING.md file is deleted to reduce redundancy
and streamline the documentation. The information contained in this
file may have been moved to a more appropriate location or deemed
unnecessary for the current project scope. This change helps in
maintaining a cleaner and more focused documentation set, ensuring
that only relevant and up-to-date information is available to the
developers and users.
cursor-com[bot]

This comment was marked as outdated.

@kzndotsh

This comment was marked as resolved.

cursor-com[bot]

This comment was marked as resolved.

sourcery-ai[bot]

This comment was marked as resolved.

…e workflow

Add `MANUAL_COMMIT_REF` and `MANUAL_BASE_REF` environment variables
to the maintenance workflow to allow for more flexible manual
triggering of the workflow with specific commit references. This
enhancement provides better control over the workflow execution
when manual intervention is required.

refactor(docker.py): modularize and enhance subprocess command validation

Refactor the `_safe_subprocess_run` function to improve readability
and maintainability by breaking down the logic into smaller,
self-contained functions. This change enhances the security and
clarity of the subprocess execution by clearly defining validation
and sanitization steps. The refactoring also ensures that resource
name validation is more robust and modular, reducing the risk of
command injection vulnerabilities.
cursor-com[bot]

This comment was marked as resolved.

…nsistency

Update documentation and scripts to reflect the change in container
naming from 'app' to 'tux'. This change ensures consistency across
all documentation and scripts, aligning with the naming convention
used in the Docker setup.

build(Dockerfile): remove git repo after dynamic versioning for cleaner builds

Initialize a minimal git repository for Poetry dynamic versioning
and remove it after use to keep the Docker image clean. This change
ensures that the build environment remains uncluttered, reducing
potential issues with leftover files.

perf(scripts): calculate averages dynamically based on iterations

Modify the script to calculate average build times dynamically
based on the actual number of iterations. This change allows for
more accurate performance metrics, especially when the number of
iterations varies.

fix(cli): remove dangling Docker images using built-in filter

Implement a method to remove all dangling Docker images using
Docker's built-in filter. This change helps in maintaining a clean
Docker environment by automatically removing unused images, thus
freeing up space and reducing clutter.
cursor-com[bot]

This comment was marked as resolved.

…best practices

Revise the Dockerfile to include detailed comments and documentation
for each stage of the multi-stage build process. This improves
understanding and maintainability by clearly explaining the purpose
and impact of each stage, as well as the security and optimization
measures implemented. The changes also include comprehensive usage
examples and best practices, making it easier for developers to
understand the build process and customize it for different
environments. This is crucial for ensuring consistency, security, and
efficiency across development and production deployments.
cursor-com[bot]

This comment was marked as resolved.

Add new commands to the CLI usage documentation to provide users
with more comprehensive guidance on available Docker operations.
This includes commands for opening a shell, showing running
containers, restarting services, checking health status, cleaning
up resources, validating configurations, and pulling images.

refactor(Dockerfile): move Prisma setup to non-root user section

Reorganize the Dockerfile to run Prisma setup commands as the
non-root user. This change addresses permission issues and aligns
with security best practices by ensuring that all operations are
executed with minimal privileges.

docs(docker-compose.dev.yml): enhance documentation for development

Add extensive comments and documentation to the development
Docker Compose file. This includes detailed explanations of
development features, workflow optimizations, and best practices
to assist developers in setting up and using the development
environment effectively.

docs(docker-compose.yml): improve production Docker Compose documentation

Enhance the production Docker Compose file with comprehensive
comments and documentation. This provides clarity on security
features, resource management, monitoring, and operational
excellence, ensuring that the production environment is
configured for stability and efficiency.
cursor-com[bot]

This comment was marked as resolved.

Introduce .editorconfig to enforce consistent coding styles across
different editors and IDEs. This includes settings for indentation,
charset, and line endings for various file types. The .gitattributes
file is added to ensure consistent line endings and to handle binary
files appropriately. These changes aim to improve code readability
and maintainability by standardizing the development environment
across the team.

chore(pre-commit): update pre-commit config for markdown and prettier

Exclude Markdown files from trailing whitespace removal to preserve
intentional line breaks. Update prettier configuration to align with
.editorconfig settings, ensuring consistent formatting across YAML
and JSON files. This enhances the development workflow by maintaining
formatting consistency and preventing unnecessary diffs.
cursor-com[bot]

This comment was marked as resolved.

… execution

Switching from `poetry run` to explicit Python module execution
ensures that the Prisma commands are run in the correct virtual
environment. This change improves reliability by directly invoking
the Python interpreter from the virtual environment, reducing
potential issues with environment path resolution.
cursor-com[bot]

This comment was marked as resolved.

Convert the Docker toolkit from a bash script to a Python script for better maintainability and integration. This change introduces a new Python-based Docker toolkit that consolidates all Docker operations, including testing, monitoring, and management. The Python script provides improved error handling, logging, and flexibility in executing Docker commands. It also supports additional testing options such as quick and comprehensive tests, enhancing the overall testing strategy for Docker environments. The transition to Python allows for easier maintenance and integration with other Python-based tools and libraries.
cursor-com[bot]

This comment was marked as resolved.

…ation

Introduce a new test job in the CI workflow to ensure Python
code changes are tested. This job checks for changes in Python
files and test-related files, installs dependencies using Poetry,
and runs tests with coverage reporting. This ensures that any
modifications to the codebase are validated through automated
testing, improving code quality and reliability.

Additionally, enhance Docker validation by supporting both Docker
Compose v1 and v2. This change ensures compatibility with different
environments where either version might be installed, preventing
potential build failures due to version mismatches. The .env file
creation for Docker validation is also simplified for consistency.
cursor-com[bot]

This comment was marked as resolved.

Include a step to generate the Prisma client in the CI workflow
before running tests and format checks. This ensures that the
Prisma client is up-to-date and available, preventing potential
errors during the CI process due to missing or outdated client
files. This change enhances the reliability and consistency of
the CI pipeline by ensuring all necessary code is generated
before execution.
cursor-com[bot]

This comment was marked as resolved.

…roups

Change the poetry install command to use the `--with` option for better
clarity and to ensure all necessary dependency groups are installed.
In the first job, the `dev` and `types` groups are explicitly included,
ensuring development and type-checking dependencies are installed. In
the second job, the `dev`, `test`, and `types` groups are included to
ensure all dependencies required for testing and development are
available. This change improves the reliability and consistency of the
CI workflow by ensuring all necessary dependencies are installed.
cursor-com[bot]

This comment was marked as resolved.

The `fail_ci_if_error` option is removed from the Codecov action
configuration. This option was set to false, which means that any
errors in the Codecov action would not cause the CI to fail. By
removing this option, the default behavior is used, which is to
fail the CI if there is an error in the Codecov action. This change
ensures that any issues with code coverage reporting are caught
and addressed promptly, maintaining the integrity of the CI process.
Copy link

@cursor-com cursor-com bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ BugBot reviewed your changes and found no bugs!


Was this report helpful? Give feedback by reacting with 👍 or 👎

Set a coverage threshold with `--cov-fail-under=0` to ensure that the
test suite does not fail due to coverage percentage. This allows the
tests to pass regardless of coverage, which is useful during initial
setup or when coverage is not a priority. Additionally, set
`fail_ci_if_error: false` for the Codecov action to prevent CI failures
if there are issues with uploading coverage data, ensuring that CI
pipelines are not blocked by external service issues.
Copy link

codecov bot commented Jun 9, 2025

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

kzndotsh added 3 commits June 9, 2025 13:50
…st coverage

Introduce a new Codecov configuration file to manage coverage
reporting and status checks for the Tux bot project. This setup
enforces different coverage targets and thresholds for various
components, ensuring critical areas like the database layer and
error handlers maintain higher coverage standards.

Enhance the GitHub Actions CI workflow to run unit, database, and
integration tests separately, each with dedicated coverage reports
and flags. This separation allows for more granular control over
coverage reporting and better insights into specific test areas.
Uploading test results to Codecov provides a comprehensive view of
test performance and coverage, aiding in maintaining code quality.
Introduce a Codecov configuration file to manage code coverage
reporting for the Tux bot project. This configuration sets global
coverage precision, rounding, and range, and defines status checks
for pull requests. It establishes coverage targets and thresholds
for different project components, ensuring critical areas like the
database and core functionalities have higher standards. The file
also configures pull request comments, ignores specific paths from
coverage calculations, and manages components and flags. This
enhancement aims to improve code quality by enforcing coverage
standards and providing detailed insights into code coverage
throughout the development process.
Introduce a comprehensive Codecov configuration file to enhance code
coverage tracking and reporting for the Tux Discord Bot project. The
new .codecov.yml file provides a more detailed and structured approach
to coverage management, including tiered coverage standards, component-
based tracking, and intelligent CI integration. This change aims to
improve the accuracy and granularity of coverage reports, ensuring
higher standards for critical components and better feedback for
developers. The previous codecov.yml file is removed to avoid
duplication and potential conflicts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants