Skip to content

CU-869aa22g2 Add ElasticSearch bits from working_with_cogstack #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 40 commits into
base: main
Choose a base branch
from
Open

CU-869aa22g2 Add ElasticSearch bits from working_with_cogstack #123

wants to merge 40 commits into from

Conversation

mart-r
Copy link
Collaborator

@mart-r mart-r commented Aug 29, 2025 •
edited
Loading

This PR adds only the ES parts from working_with_cogstack.
This is based on the newer cogstack2.py and relevant search notebook.

Currently:

  • Move code from WWC
  • Refactor code
  • Add a few simple tests for cogstack.py module
  • Add a test that runs through the notebook
  • Add workflow
  • Real world validation
  • Documentation on usage (i.e download scripts)
  • Documentation on usage alongside the MedCAT-specific parts
  • Integrate into cogstack-jupyter-hub

github-advanced-security[bot]
github-advanced-security bot found potential problems Aug 29, 2025
View reviewed changes
.github/workflows/cogstack-es_main.yml
Comment on lines +17 to +42
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [ '3.9', '3.10', '3.11', '3.12' ]
max-parallel: 4

steps:
- uses: actions/checkout@v5
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Check types
run: |
python -m mypy --follow-imports=normal *.py
- name: Lint
run: |
ruff check *.py
- name: Test
run: |
pytest tests

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}

Copilot Autofix

AI about 1 month ago

To fix the issue, add a permissions block specifying the least necessary privileges for the job. In this workflow, the job does not push changes, create issues, comment on pull requests, or modify repository content—it only checks out code and runs tests/linting. Therefore, limiting permissions to contents: read at the job or workflow root is most appropriate. You can add this block either at the top (outside jobs:) so it applies to all jobs, or directly under the types-lint-tests: job section if you only want to affect that job. The best approach is adding to the root so that future jobs inherit safe defaults.

Edit .github/workflows/cogstack-es_main.yml and insert the block:

permissions:
  contents: read

immediately after the workflow name: and before the on: key.

Suggested changeset 1
.github/workflows/cogstack-es_main.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/cogstack-es_main.yml b/.github/workflows/cogstack-es_main.yml
--- a/.github/workflows/cogstack-es_main.yml
+++ b/.github/workflows/cogstack-es_main.yml
@@ -1,4 +1,6 @@
 name: cogstack-es - Test
+permissions:
+  contents: read
 
 on:
   push:
EOF
@@ -1,4 +1,6 @@
name: cogstack-es - Test
permissions:
contents: read

on:
push:
Copilot is powered by AI and may make mistakes. Always verify output.
@mart-r mart-r mentioned this pull request Sep 15, 2025
Open
mart-r added 28 commits September 23, 2025 14:43
@mart-r
CU-869aa22g2: Add ES requirement
8af57d8
@mart-r
CU-869aa22g2: Add initial README
64f95ad
@mart-r
CU-869aa22g2: Add (slightly) converted credentials
fcf4d96
@mart-r
CU-869aa22g2: Add cogstack module ported from WWC
bfdcf82
@mart-r
CU-869aa22g2: Add search template notebook
2332a63
@mart-r
CU-869aa22g2: Add ipython dependency (for cogstack and notebook)
4dc92aa
@mart-r
CU-869aa22g2: Add tqdm dependency
4f2d463
@mart-r
CU-869aa22g2: Add pandas dependency
61b20f0
@mart-r
CU-869aa22g2: Add a few initial tests
ec2a780
@mart-r
CU-869aa22g2: Remove a bunch of extra whitespace
0322164
@mart-r
CU-869aa22g2: Add ruff dependecny
696e669
@mart-r
CU-869aa22g2: Run ruff on cogstack module
90d339b
@mart-r
CU-869aa22g2: Move to primitives collections for type hinting
77b3f3d
@mart-r
CU-869aa22g2: Some further linting changes
fe1656f
@mart-r
CU-869aa22g2: Refactor cogstack module to make a little more sense
780a3fd
@mart-r
CU-869aa22g2: Rename some methods for better descriptions
403a2e8
@mart-r
CU-869aa22g2: Rename a method name for better descriptions
2fd8507
@mart-r
CU-869aa22g2: Minor whitespace fix
7e04913
@mart-r
CU-869aa22g2: Remove path add in notebook
62b60ca
@mart-r
CU-869aa22g2: Remove unused import from notebook
92aaabd
@mart-r
CU-869aa22g2: Import username and password from credentials in case t…
651a55b 
…hey are neded
@mart-r
CU-869aa22g2: Add nbconvert dev-dependency
e98f647
@mart-r
CU-869aa22g2: Add default indices to get fields for
4bd6246
@mart-r
CU-869aa22g2: Improve error handling (avoid hiding stack trace)
ab5dee8
@mart-r
CU-869aa22g2: Add default indices in notebook example
ab8da2c
@mart-r
CU-869aa22g2: Update progress bar handling during exception handling
4b77182
@mart-r
CU-869aa22g2: Add default indices in notebook examples
3a604cf
@mart-r
CU-869aa22g2: Add data folder
921d682
@mart-r mart-r force-pushed the CU-869aa22g2-add-es-from-wwc branch from c4172ee to 329f047 Compare September 23, 2025 14:22
@tomolopolis
Copy link
Member

Task linked: CU-869aa22g2 Add a separate ES-specific branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mart-r @tomolopolis