Skip to content

Use src-py-lib 0.1.6 GraphQL pagination #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
marcleblanc2 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Use src-py-lib 0.1.6 GraphQL pagination #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
03da4d1
Use src-py-lib 0.1.6 GraphQL pagination
marcleblanc2 May 29, 2026
865d7b2
Prepare for memory efficiency analysis
marcleblanc2 May 30, 2026
9b7136a
Optimize full-set permission planning memory
marcleblanc2 May 30, 2026
72eba38
Refactor maps schema, expose Python API, refactor subcommands
marcleblanc2 Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 45 additions & 32 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# src-auth-perms-sync
<!-- HUMAN-MAINTAINED — DO NOT EDIT THIS FILE -->
<!-- HUMAN-MAINTAINED SECTION START — DO NOT EDIT THIS SECTION -->

src-auth-perms-sync automates Sourcegraph's Explicit Permissions GraphQL API,
setting user-to-repo permissions based on mapping rules, for example:
Expand Down Expand Up @@ -92,34 +92,48 @@ Feel free to open issues or PRs, but responses are best effort.
- Requires Python 3.11
- Recommended: Use a Python virtual environment

### Install from a GitHub Release

Use this when the VM can reach GitHub and PyPI during install:
### Install from PyPI

```bash
python3.11 -m venv .venv
. .venv/bin/activate
pip install \
"https://github.com/sourcegraph/src-auth-perms-sync/releases/download/v0.1.0/src_auth_perms_sync-0.1.0-py3-none-any.whl"
```
pip install src-auth-perms-sync

### Restricted/offline install from a GitHub Release
# Run the CLI
src-auth-perms-sync --help
```

Use this when the VM cannot reach package indexes during install
### Restricted / offline install from a GitHub Release

Download the .tar.gz file from the GitHub release
Download the .tar.gz file from [a GitHub release](https://github.com/sourcegraph/src-auth-perms-sync/releases)

```bash
tar -xzf src-auth-perms-sync-linux-x64.tar.gz
python3.11 -m venv .venv
. .venv/bin/activate

pip install --no-index --find-links ./wheelhouse src-auth-perms-sync

# Run the CLI
src-auth-perms-sync --help
```

After either install method, run the CLI from the activated virtual environment:
### Import into your own Python script

```bash
src-auth-perms-sync --help
```python
from pathlib import Path

import src_auth_perms_sync as src

config = src.Config(
src_endpoint="https://sourcegraph.example.com",
src_access_token="sgp_...",
maps_path=Path("/absolute/path/to/maps.yaml"),
apply=False, # Dry run (default), set to True to make changes
)

succeeded = src.Set(config)

# Other command wrappers:
# succeeded = src.Get(config)
# succeeded = src.Restore(config)
# succeeded = src.SyncSamlOrgs(config)
```

## Inputs
Expand All @@ -136,21 +150,20 @@ src-auth-perms-sync --help
- A list of filters for users
- A list of filters for repos
- See [maps-example.yaml](./maps-example.yaml)
- An empty maps.yaml file is created for you on the first --get run
- An empty maps.yaml file is created for you on the first `get` run

## Usage: Permission sync

1. **Get auth providers and code hosts**

```bash
uv run src-auth-perms-sync [--get]
uv run src-auth-perms-sync get
```

- Queries the Sourcegraph instance for auth providers and code host connections
- Writes generated reference files `auth-providers.yaml` and `code-hosts.yaml` under
`src-auth-perms-sync-runs/<endpoint>/`
- Creates an empty `maps.yaml` if it doesn't exist
- Runs by default when no command is selected

2. **Configure mapping rules**

Expand All @@ -161,20 +174,20 @@ src-auth-perms-sync --help
3. **Set: Dry run**

```bash
uv run src-auth-perms-sync --set maps.yaml --full
uv run src-auth-perms-sync set --maps-path maps.yaml --full
```

4. **Set: Apply**

```bash
uv run src-auth-perms-sync --set maps.yaml --full --apply
uv run src-auth-perms-sync set --maps-path maps.yaml --full --apply
```

5. **Restore: Dry run**

```bash
uv run src-auth-perms-sync \
--restore backups/maps.yaml/2026-04-27-08-24-25-set-apply/before.json
uv run src-auth-perms-sync restore \
--restore-path backups/maps.yaml/2026-04-27-08-24-25-set-apply/before.json
```

- Roll back the explicit-permissions state on the
Expand All @@ -183,8 +196,8 @@ src-auth-perms-sync --help
6. **Restore: Apply**

```bash
uv run src-auth-perms-sync \
--restore backups/maps.yaml/2026-04-27-08-24-25-set-apply/before.json \
uv run src-auth-perms-sync restore \
--restore-path backups/maps.yaml/2026-04-27-08-24-25-set-apply/before.json \
--apply
```

Expand All @@ -193,7 +206,7 @@ src-auth-perms-sync --help
1. **Get user and org metadata**

```bash
uv run src-auth-perms-sync --sync-saml-orgs
uv run src-auth-perms-sync sync-saml-orgs
```

- Queries the Sourcegraph instance for auth providers, users, users' SAML groups, and orgs
Expand All @@ -202,11 +215,11 @@ src-auth-perms-sync --help
2. **Apply org sync**

```bash
uv run src-auth-perms-sync --sync-saml-orgs --apply
uv run src-auth-perms-sync sync-saml-orgs --apply
```

- Creates the orgs if they don't exist, and sync the members from the SAML groups to the orgs
- `--sync-saml-orgs` can also be added to a `--set` run, to run both at the same time
- `--sync-saml-orgs` can also be added to a `get` or `set` run, to run both at the same time

## Options

Expand All @@ -231,9 +244,9 @@ src-auth-perms-sync-runs/endpoint/
- The `src-auth-perms-sync-runs` dir is created under your current working directory
- The `endpoint` dir is created with the hostname from `SRC_ENDPOINT`
- If `maps.yaml` doesn't exist already, it'll be created for you
- `auth-providers.yaml` and `code-hosts.yaml` are created / replaced by the `--get` command,
- `auth-providers.yaml` and `code-hosts.yaml` are created / replaced by the `get` command,
for you to copy values from, to use in your `maps.yaml`
- Only one `maps.yaml` file can be used at a time per Sourcegraph instance, as each `--set --apply`
- Only one `maps.yaml` file can be used at a time per Sourcegraph instance, as each `set --apply`
command resets the state on the Sourcegraph instance to the `maps.yaml` file which was used
- Each run of the script creates a new `timestamp-command` dir under the `runs` dir, with:
- A log file
Expand All @@ -243,4 +256,4 @@ src-auth-perms-sync-runs/endpoint/
- An `after.json` file, capturing the new state
- A `diff.json` file, a shorter, reviewable file containing the diffs between before and after

<!-- HUMAN-MAINTAINED — DO NOT EDIT THIS FILE -->
<!-- HUMAN-MAINTAINED SECTION END — DO NOT EDIT ABOVE -->
40 changes: 36 additions & 4 deletions dev/TODO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,34 @@
# TODO

## High priority: Bump src-py-lib after Node ID helper release

- After releasing `src-py-lib` with Sourcegraph Node ID helpers, update
`pyproject.toml` and `uv.lock` to depend on that new version.
## High priority: Customer feedback

- Allow use as either CLI or importable module
- Take maps.yaml as a constructor object?

## High priority: Instrument with OpenTelemetry — in progress

- [ ] Add OTel-native traces, metrics, and wide log events in `src-py-lib`.
- [ ] Add shared OTel bootstrap config/helpers with `--otel` and standard
`OTEL_*` env-var-backed CLI args.
- [ ] Replace custom trace-context propagation with OTel W3C propagation.
- [ ] Instrument shared HTTP and GraphQL clients manually, preserving safe
sanitized attributes and Sourcegraph-specific metadata.
- [ ] Rename Sourcegraph debug tracing from `--trace` to `--fetch-sg-traces`.
- [ ] Wire `src-auth-perms-sync` to the shared OTel bootstrap without doing
import-time logger/provider setup.
- [ ] Verify pyright, tests, and CLI help in both repos.

## High priority: Reduce worst-case full-permission sync load

- Use the stress-run evidence in
[sourcegraph-explicit-permissions-tracing.md](./sourcegraph-explicit-permissions-tracing.md)
to request Sourcegraph bulk explicit-permission read and write APIs.
- Add an explicit destructive/performance-test mode to the e2e runner so giant
stress runs can skip or defer full restore cleanup when the goal is finding
the server-side breaking point.
- Revisit full snapshot capture once Sourcegraph exposes a bulk read path;
replace aliased `User.permissionsInfo.repositories(source: API)` calls before
raising concurrency further.

## Medium priority: Lightweight incremental updates

Expand Down Expand Up @@ -69,6 +94,13 @@ If/when we revisit:
3. Add a CLI flag (e.g. `--cross-check-capture`) gated behind a clear
"this doubles capture cost" warning.

## Low priority: Grouped full-set plan if memory is still too high

Phase 1 now avoids per-repo username sets for non-overlapping full-set maps.
If memory remains too high after re-measuring, implement the Phase 2 grouped
plan in [mapping-efficiency.md](./mapping-efficiency.md): combine map-entry
overlays into final groups of repos that share the same desired username tuple.

## Low priority: Expand group-membership filters beyond SAML

`allowGroups`-style enforcement exists on more than just SAML, but only
Expand Down
Loading