-
Notifications
You must be signed in to change notification settings - Fork 5
feat(xqwatcher): migrate from EC2 ASG to Kubernetes Deployment #4287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+1,850
−314
Merged
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
de52643
feat(ol_types): add xqwatcher to Services and Application enums
blarghmatey c8d154a
feat(xqwatcher): update Vault policy to include xqueue credentials path
blarghmatey 3f6c0ca
feat(xqwatcher): replace EC2 ASG deployment with Kubernetes Deployment
blarghmatey fb44413
feat(xqwatcher): update all 9 stack configs for Kubernetes deployment
blarghmatey 5613460
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 1eedc5e
fix(xqwatcher): address PR review feedback and deployment issues
blarghmatey 3b62704
chore: Use xqwatcher image from dockerhub pull-through cache
blarghmatey 0d21de1
fix(xqwatcher): use correct xqueue-watcher CLI arguments
blarghmatey 105f767
fix(xqwatcher): enable automount_service_account_token on Deployment …
blarghmatey 365d19b
fix(xqwatcher): reference container image by digest instead of mutabl…
blarghmatey 738d91a
fix(xqwatcher): invoke entrypoint via uv run
blarghmatey 763968b
fix(xqwatcher): pass --no-sync to uv run to skip dependency installation
blarghmatey 950ebf2
fix(xqwatcher): fix config directory structure for watcher discovery
blarghmatey d2cd7bc
fix(xqwatcher): serialize confd_json to string before writing to Vault
blarghmatey e6a8d65
fix(xqwatcher): require XQWATCHER_DOCKER_DIGEST; remove docker_tag fa…
blarghmatey 708b03e
feat(xqwatcher): rewrite ContainerGrader image through ECR pull-throu…
blarghmatey 37ed37b
ci: suppress CodeQL actions extractor failure
blarghmatey 273bb96
feat(concourse): add reusable grader image build pipelines
blarghmatey 02460a5
fix(concourse): track current working branches for grader image pipel…
blarghmatey f8c786c
fix(concourse): set build context to repo root for grader base image
blarghmatey 1a3cc07
feat(concourse): ensure ECR repositories exist before pushing grader …
blarghmatey 0198734
fix: pass repo-name-only to registry-image ECR resources
blarghmatey 0e0e8d0
fix: track feature branch in meta pipeline pipeline-code resource
blarghmatey 21929ee
feat: use SSH credentials for private grader repository access
blarghmatey 6a8e14e
fix: use odlbot SSH key for private grader repo access
blarghmatey 7c66896
fix: use DockerHub grader base image as build trigger
blarghmatey ff13341
feat: wire XQWATCHER_GRADER_* env vars and fix ECR image rewrite
blarghmatey 0f8f7f5
config: Update MITx CI watcher config for use on K8s
blarghmatey 5e38539
config: Get grader path to strip erroneous prefix
blarghmatey 23298ed
fix: Set proper grader root for dockerized graders
blarghmatey 3392b9f
fix: Don't strip path components either
blarghmatey a4fc006
fix: Update mitx CI watcher password to match xqueue
blarghmatey 94edcd7
feat(xqwatcher): migrate grader config from SOPS/Vault KV to Pulumi c…
blarghmatey 41b0aba
fix: address PR review feedback
blarghmatey 7811043
feat(xqwatcher): add Docker+Pulumi Concourse pipeline
blarghmatey db24a45
fix: address remaining Copilot PR review feedback
blarghmatey f5eb04a
Add graders-mit-686x image build pipeline
blarghmatey d1f4d77
Delete .github/codeql/codeql-config.yml
blarghmatey d001f6e
Add 686x ContainerGrader to mitxonline production stack
blarghmatey ab4683a
Add edxorg xqueue server ref for 686x grader
blarghmatey 0d529f8
feat(xqwatcher): surface new security env vars and add pod seccomp pr…
blarghmatey 12ffdca
fix: address unresolved PR review feedback
blarghmatey a9df683
Apply suggestion from @Copilot
feoh f5ffad6
chore: Update watcher pipeline
blarghmatey 1ac0e76
feat(xqwatcher): add HPA for CPU/memory-based autoscaling
blarghmatey 5148008
feat(xqwatcher): separate edxorg server into its own Deployment
blarghmatey File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
119 changes: 119 additions & 0 deletions
119
src/ol_concourse/pipelines/open_edx/grader_images/base_image_pipeline.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| """ | ||
| Pipeline that builds the xqueue-watcher grader base image and pushes it to | ||
| both DockerHub and ECR. | ||
|
|
||
| The base image (grader_support/Dockerfile.base) is the foundation for all | ||
| course-specific grader images. Publishing it to both registries allows: | ||
| - DockerHub (mitodl/xqueue-watcher-grader-base): public reference usable | ||
| without AWS credentials; used in grader repo Dockerfiles as the default | ||
| GRADER_BASE_IMAGE build arg. The per-grader Concourse build pipelines | ||
| trigger off this DockerHub image so a base image rebuild automatically | ||
| triggers downstream grader image rebuilds. | ||
| - ECR (mitodl/xqueue-watcher-grader-base): private mirror for use inside | ||
| AWS without DockerHub rate-limit concerns. | ||
|
|
||
| Triggers: | ||
| - Push to the xqueue-watcher repo on paths under grader_support/. | ||
| """ | ||
|
|
||
| import sys | ||
|
|
||
| from ol_concourse.lib.containers import container_build_task, ensure_ecr_task | ||
| from ol_concourse.lib.models.fragment import PipelineFragment | ||
| from ol_concourse.lib.models.pipeline import ( | ||
| GetStep, | ||
| Identifier, | ||
| Input, | ||
| Job, | ||
| Pipeline, | ||
| PutStep, | ||
| ) | ||
| from ol_concourse.lib.resources import git_repo, registry_image | ||
|
|
||
| _AWS_ACCOUNT_ID = "610119931565" | ||
| _AWS_REGION = "us-east-1" | ||
| _BASE_IMAGE_REPO = "mitodl/xqueue-watcher-grader-base" | ||
|
|
||
|
|
||
| def grader_base_image_pipeline() -> Pipeline: | ||
| """Return the pipeline that builds and publishes the grader base image.""" | ||
| xqwatcher_repo = git_repo( | ||
| name=Identifier("xqueue-watcher-code"), | ||
| uri="https://github.com/mitodl/xqueue-watcher", | ||
| branch="main", | ||
| paths=["grader_support/"], | ||
| ) | ||
|
|
||
| # DockerHub push target — public, used by grader repo Dockerfiles as default | ||
| # GRADER_BASE_IMAGE build arg and accessible without AWS credentials. | ||
| dockerhub_base_image = registry_image( | ||
| name=Identifier("grader-base-dockerhub"), | ||
| image_repository=_BASE_IMAGE_REPO, | ||
| image_tag="latest", | ||
| username="((dockerhub.username))", | ||
| password="((dockerhub.password))", # noqa: S106 | ||
| ) | ||
|
|
||
| # ECR push target — private mirror for use inside AWS without DockerHub | ||
| # rate-limit concerns. The per-grader Concourse build pipelines trigger | ||
| # off the DockerHub base image (grader_base_dockerhub_repo), not ECR. | ||
| ecr_base_image = registry_image( | ||
| name=Identifier("grader-base-ecr"), | ||
| image_repository=_BASE_IMAGE_REPO, | ||
| image_tag="latest", | ||
| ecr_region=_AWS_REGION, | ||
| ) | ||
|
|
||
| build_job = Job( | ||
| name=Identifier("build-grader-base-image"), | ||
| plan=[ | ||
| GetStep(get=xqwatcher_repo.name, trigger=True), | ||
| container_build_task( | ||
| inputs=[Input(name=xqwatcher_repo.name)], | ||
| build_parameters={ | ||
| "CONTEXT": str(xqwatcher_repo.name), | ||
| "DOCKERFILE": ( | ||
| f"{xqwatcher_repo.name}/grader_support/Dockerfile.base" | ||
| ), | ||
| }, | ||
| ), | ||
| ensure_ecr_task(_BASE_IMAGE_REPO), | ||
| # Push to DockerHub first — fail fast if credentials are wrong | ||
| # before consuming the ECR push quota. | ||
| PutStep( | ||
| put=dockerhub_base_image.name, | ||
| params={ | ||
| "image": "image/image.tar", | ||
| "additional_tags": f"./{xqwatcher_repo.name}/.git/describe_ref", | ||
| }, | ||
| ), | ||
| PutStep( | ||
| put=ecr_base_image.name, | ||
| params={ | ||
| "image": "image/image.tar", | ||
| "additional_tags": f"./{xqwatcher_repo.name}/.git/describe_ref", | ||
| }, | ||
| ), | ||
| ], | ||
| ) | ||
|
|
||
| fragment = PipelineFragment( | ||
| resources=[xqwatcher_repo, dockerhub_base_image, ecr_base_image], | ||
| jobs=[build_job], | ||
| ) | ||
|
|
||
| return Pipeline( | ||
| resource_types=fragment.resource_types, | ||
| resources=fragment.resources, | ||
| jobs=fragment.jobs, | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| pipeline_json = grader_base_image_pipeline().model_dump_json(indent=2) | ||
| with open("definition.json", "w") as definition: # noqa: PTH123 | ||
| definition.write(pipeline_json) | ||
| sys.stdout.write(pipeline_json) | ||
| sys.stdout.write( | ||
| "\nfly -t <target> set-pipeline -p build-grader-base-image -c definition.json\n" | ||
| ) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.