Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ocrd-network: optional deploy mongodb and queue #1048

Merged
merged 39 commits into from
Jun 21, 2023
Merged

ocrd-network: optional deploy mongodb and queue #1048

merged 39 commits into from
Jun 21, 2023

Conversation

joschrew
Copy link
Contributor

@joschrew joschrew commented May 10, 2023 •
edited
Loading

Just a draft for now for easier usage.

This is an extensions to 1030.

With this mongoDB and rabbitMQ can be skipped with the deployer to allow self management of this two parts/containers.
For the deployment on the VM I want to change to self managed queue and database so I need this therefore.

stweil and others added 18 commits March 15, 2023 20:34
@stweil
Dockerfile: Use virtual environment
edcf855 
python3-pip can now be removed, and running pip instead of pip3 is sufficient.

This also avoids the installation of python3-setuptools and python3-wheel
(requirements of python3-pip).

Signed-off-by: Stefan Weil <[email protected]>
@bertsky
give up workaround for shapely-CUDA issue
5c003fa
@bertsky
rehash after pip upgrade
2250550
@bertsky
keep gcc, no autoremove
00a0f6f
@bertsky
docker-cuda: change base image, no multi-CUDA runtimes
410783f
@bertsky
reinstate workaround for shapely, but more robust
de86e0f
@bertsky
core-cuda: use CUDA 11.8, install cuDNN via pip and make available sy…
c1178f9 
…stem-wide via ld.so.conf
@bertsky
core-cuda: install more CUDA libs via pip and ld.so.conf, simplify Do…
d3d54bf 
…ckerfile for that
@bertsky
make install on py36: prefer binary OpenCV/Numpy via pip config inste…
357e729 
…ad of preinstall
@bertsky
make install on py36: fix prefer-binary syntax
2640b71
@bertsky
make install on py36: revert to prefer-binary via install
b3618a9
@kba
📦 v2.50.0
b713030
@bertsky
Merge branch 'master' of https://github.com/OCR-D/core into reduce-cuda
c0c153e
@bertsky
Merge branch 'pr-1008' into reduce-cuda
209fa21
@bertsky
core-cuda: use same CUDA libs as needed for Torch anyway
a6cf5ff
@joschrew
Fix mongodb credentials usage
1da66f4 
Previously mongodb credentials from config were accidentially ignored
and it was always deployed without credentials
@joschrew
Add skip_deployment flag for queue and database
8530bd9
@joschrew
Make deployer optionally skip mongo and rabbitmq
b4e6576
@bertsky
Copy link
Collaborator

bertsky commented May 10, 2023

This is an extensions to 1030.

please rebase onto #1030:

git fetch origin
git rebase origin/processor-server
git push -f

@joschrew
Copy link
Contributor Author

This is an extensions to 1030.

please rebase onto #1030:

git fetch origin
git rebase origin/processor-server
git push -f

-> Current branch ocrd-network-optional-deploy is up to date. I think it is up to date or I'm missing something.

@MehmedGIT
Copy link
Contributor

He means to rebase to #1030, not the master branch. So it's easier to see what has been changed on top of #1030.

@joschrew
Copy link
Contributor Author

joschrew commented May 10, 2023 •
edited
Loading

How to do that?

Edit: found it and changed the base of this PR (via github gui). Hopefully this is what was meant because i didn't rebase anything for that.

@joschrew joschrew changed the base branch from master to processor-server May 10, 2023 11:48
@MehmedGIT MehmedGIT self-requested a review May 15, 2023 09:03
MehmedGIT
MehmedGIT reviewed May 15, 2023
View reviewed changes
Copy link
Contributor

@MehmedGIT MehmedGIT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me and can be merged.

ocrd_network/ocrd_network/deployment_utils.py Outdated Show resolved Hide resolved
ocrd_network/ocrd_network/deployment_utils.py Outdated Show resolved Hide resolved
@kba kba marked this pull request as ready for review May 15, 2023 13:16
@bertsky
docker-cuda: rewrite…
f5e212a 
- extra Dockerfile.cuda instead of guessing from base image name
- re-use non-CUDA ocrd/core instead of nvidia/cuda base image
- install CUDA toolkit and libraries via (micro)mamba
  instead of nvidia-pyindex CUDA libraries made available system-wide
- get cuDNN and CUDA libs from conda-forge and nvidia channels
bertsky and others added 18 commits June 2, 2023 02:36
@bertsky
docker-cuda: improve (reduce size) again…
85a5d16 
- install CUDA libraries via nvidia-pyindex again (but not nvcc)
- ensure they can be compiled/linked against system-wide (with nvcc)
@bertsky
remove out-dated processor resources
47eff22
@bertsky
Revert "Merge remote-tracking branch 'hnesk/no-more-pkg_resources' in…
12e781c 
…to release-2.36.0"

This reverts commit 9a54ef6, reversing
changes made to a78d4c5.
@bertsky
make help: improve description
bac1a45
@kba
📝 changelog
95062b0
@kba
📦 v2.51.0
c636f2c
@bertsky
docker-image: reuse local ghcr.io image instead of docker.io
8c7b761 
(only cosmetic)
@kba
disable logging tests until properly fixed
4bfac5e
@kba
test_workspace_bagger: use ocr-d.de instead of google.com for testing
ca5f342
@kba
test bashlib: /usr/bin/env bash instead of /bin/bash
3f05745
@kba
debug gh actions
af38a2c
@kba
readme: remove dockerhub/travis badge, add GH actions badge
c63ab4c
@kba
ci: disable upterm for gh actions
555dece
@bertsky
docker-cuda: move recipe to reusable makefile target deps-cuda
d76409e
@kba
make deps-cuda: Set MAMBA_ROOT_PREFIX to CONDA_PREFIX
2538979
@kba
📝 changelog
79ad301
@kba
Merge pull request #1055 from bertsky/deps-cuda
6708624 
docker-cuda: move recipe to reusable makefile target deps-cuda
@kba
Merge branch 'master' into ocrd-network-optional-deploy
821f70c
@kba kba merged commit 84ad16e into processor-server Jun 21, 2023
@kba kba deleted the ocrd-network-optional-deploy branch June 21, 2023 11:58
@bertsky bertsky mentioned this pull request Jun 27, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MehmedGIT MehmedGIT MehmedGIT left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@joschrew @bertsky @MehmedGIT @kba @stweil