Skip to content

Commit acf2465

Browse files
filipecosta90filipecosta90
authored
self-contained-coordinator kickoff and package reorg (#3)
* self-contained-coordinator kickoff and package reorg
1 parent 014e871 commit acf2465

37 files changed

+2752
-659
lines changed

‎.github/release-drafter-config.yml

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name-template: 'Version $NEXT_PATCH_VERSION'
2+
tag-template: 'v$NEXT_PATCH_VERSION'
3+
categories:
4+
- title: 'Spec changes: topologies'
5+
labels:
6+
- 'spec-topologies'
7+
- title: 'Spec changes: test-suites'
8+
labels:
9+
- 'spec-test-suites'
10+
- title: 'Spec changes: builders'
11+
labels:
12+
- 'spec-builders'
13+
- title: 'Tooling changes: API'
14+
labels:
15+
- 'tool-api'
16+
- title: 'Tooling changes: Artifacts builder'
17+
labels:
18+
- 'tool-builder'
19+
- title: 'Tooling changes: Self contained coordinator'
20+
labels:
21+
- 'tool-self-coordinator'
22+
- title: 'Tooling changes: Large scale coordinator'
23+
labels:
24+
- 'tool-large-coordinator'
25+
- title: 'Maintenance'
26+
label: 'chore'
27+
change-template: '- $TITLE @$AUTHOR (#$NUMBER)'
28+
exclude-labels:
29+
- 'skip-changelog'
30+
template: |
31+
## Redis performance and observability spec change
32+
33+
$CHANGES

‎.github/workflows/poetry-ci-test-lint.yml

+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: Run Tests
2+
on:
3+
pull_request:
4+
push:
5+
branches:
6+
- master
7+
8+
jobs:
9+
pytest:
10+
strategy:
11+
matrix:
12+
python-version: [ 3.6, 3.7, 3.8, 3.9 ]
13+
os: [ ubuntu-latest, macos-latest ]
14+
env:
15+
ACTIONS_ALLOW_UNSECURE_COMMANDS: true
16+
runs-on: ${{ matrix.os }}
17+
name: os ${{ matrix.os }} python ${{ matrix.python-version }} Linting, testing, and compliance
18+
steps:
19+
- uses: actions/checkout@master
20+
- name: Set up Python ${{ matrix.python-version }}
21+
uses: actions/setup-python@v1
22+
with:
23+
python-version: ${{ matrix.python-version }}
24+
25+
- name: Install Poetry
26+
uses: dschep/[email protected]
27+
28+
- name: Cache Poetry virtualenv
29+
uses: actions/cache@v1
30+
id: cache
31+
with:
32+
path: ~/.virtualenvs
33+
key: poetry-${{ hashFiles('**/poetry.lock') }}
34+
restore-keys: |
35+
poetry-${{ hashFiles('**/poetry.lock') }}
36+
37+
- name: Set Poetry config
38+
run: |
39+
poetry config virtualenvs.in-project false
40+
poetry config virtualenvs.path ~/.virtualenvs
41+
42+
- name: Install Dependencies
43+
run: poetry install
44+
if: steps.cache.outputs.cache-hit != 'true'
45+
46+
- name: Check formatting compliance with black
47+
run: poetry run black --check .
48+
49+
- name: Check PEP8 compliance with flake
50+
run: poetry run flake8
51+
52+
- name: Test with pytest
53+
run: poetry run pytest --cov redisbench_admin
54+
55+
- name: Upload coverage
56+
run: poetry run codecov -t ${{ secrets.CODECOV_TOKEN }}

‎.github/workflows/publish-pypi.yml

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: Publish Pypi
2+
on:
3+
release:
4+
types: [ published ]
5+
6+
jobs:
7+
pytest:
8+
name: Publish to PyPi
9+
runs-on: ubuntu-latest
10+
env:
11+
ACTIONS_ALLOW_UNSECURE_COMMANDS: true
12+
steps:
13+
- uses: actions/checkout@master
14+
- name: Set up Python 3.7
15+
uses: actions/setup-python@v1
16+
with:
17+
python-version: 3.7
18+
19+
- name: Install Poetry
20+
uses: dschep/[email protected]
21+
22+
- name: Cache Poetry virtualenv
23+
uses: actions/cache@v1
24+
id: cache
25+
with:
26+
path: ~/.virtualenvs
27+
key: poetry-${{ hashFiles('**/poetry.lock') }}
28+
restore-keys: |
29+
poetry-${{ hashFiles('**/poetry.lock') }}
30+
31+
- name: Set Poetry config
32+
run: |
33+
poetry config virtualenvs.in-project false
34+
poetry config virtualenvs.path ~/.virtualenvs
35+
36+
- name: Install Dependencies
37+
run: poetry install
38+
if: steps.cache.outputs.cache-hit != 'true'
39+
40+
- name: Publish to PyPI
41+
if: github.event_name == 'release'
42+
run: |
43+
poetry publish -u __token__ -p ${{ secrets.PYPI_TOKEN }} --build

‎.github/workflows/release-drafter.yml

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
name: Release Drafter
2+
3+
on:
4+
push:
5+
# branches to consider in the event; optional, defaults to all
6+
branches:
7+
- master
8+
9+
jobs:
10+
update_release_draft:
11+
runs-on: ubuntu-latest
12+
steps:
13+
# Drafts your next Release notes as Pull Requests are merged into "master"
14+
- uses: release-drafter/release-drafter@v5
15+
with:
16+
# (Optional) specify config name to use, relative to .github/. Default: release-drafter.yml
17+
config-name: release-drafter-config.yml
18+
env:
19+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎.gitignore

+4-1
Original file line numberDiff line numberDiff line change
@@ -218,4 +218,7 @@ fabric.properties
218218
.idea/httpRequests
219219

220220
# Android studio 3.1+ serialized cache file
221-
.idea/caches/build_file_checksums.ser
221+
.idea/caches/build_file_checksums.ser
222+
223+
# Json result files
224+
*.json

‎Readme.md

+9-7
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Current supported benchmark tools:
4646
│
4747
▼
4848
┌─────────────────────────────────────────────────┐
49-
│2) api │
49+
│2) redis-benchmarks-spec-api │
5050
│ - Converts the HTTP info into an stream entry │
5151
│ - XADD stream:redis:redis:commit <...> │
5252
└────────────────────────┬────────────────────────┘
@@ -58,7 +58,7 @@ Current supported benchmark tools:
5858
└─────┘ │
5959
6060
│ ┌────────────────────────────────────────────────┐
61-
─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│2.2) build_agent │
61+
─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│2.2) redis-benchmarks-spec-builder │
6262
│ - based on ./setups/platforms │
6363
│ - build different required redis artifacts │
6464
└───────────────────────┬────────────────────────┘
@@ -86,15 +86,17 @@ Current supported benchmark tools:
8686
In a very brief description, gh.com/redis/redis upstream changes trigger an HTTP API call containing the
8787
relevant git information.
8888

89-
The HTTP request is then converted into an event ( tracked within redis ) that will trigger multiple build variants requests based upon the distinct platforms described in [`platforms`](./setups/platforms/).
89+
The HTTP request is then converted into an event ( tracked within redis ) that will trigger multiple build variants requests based upon the distinct platforms described in [`platforms`](redis_benchmarks_specification/setups/platforms/).
9090

9191
As soon as a new build variant request is received, the build agent ([`build_agent`](./build_agent/)) prepares the artifact(s) and proceeds into adding an artifact benchmark event so that the benchmark coordinator ([`benchmark_coordinator`](./benchmark_coordinator/)) can deploy/manage the required infrastructure and DB topologies, run the benchmark, and export the performance results.
9292
## Directory layout
9393

94-
* `setups`
95-
* [`platforms`](./setups/platforms/): contains the standard platforms considered to provide steady stable results, and to represent common deployment targets.
96-
* [`topologies`](./setups/topologies/): contains the standard deployment topologies definition with the associated minimum specs to enable the topology definition.
97-
* [`test_suites`](./test-suites/): contains the benchmark suites definitions, specifying the target redis topology, the tested commands, the benchmark utility to use (the client), and if required the preloading dataset steps.
94+
* `redis_benchmarks_specification/setups`
95+
* [`platforms`](redis_benchmarks_specification/setups/platforms/): contains the standard platforms considered to provide steady stable results, and to represent common deployment targets.
96+
* [`topologies`](redis_benchmarks_specification/setups/topologies/): contains the standard deployment topologies definition with the associated minimum specs to enable the topology definition.
97+
* [`builders`](redis_benchmarks_specification/setups/builders/): contains the build environment variations, that enable to build Redis with different compilers, compiler flags, libraries, etc...
98+
99+
* [`test_suites`](redis_benchmarks_specification/test-suites/): contains the benchmark suites definitions, specifying the target redis topology, the tested commands, the benchmark utility to use (the client), and if required the preloading dataset steps.
98100
* [`validator`](./validator/): contains the benchmark specifications validator utility
99101
* [`build_agent`](./build_agent/): contains the benchmark build agent utility that receives an event indicating a new build variant, generates the required redis binaries to test, and triggers the benchmark run on the listening agents.
100102
* [`benchmark_coordinator`](./benchmark_coordinator/): contains the coordinator utility that listens for benchmark suite run requests and setups the required steps to spin the actual benchmark topologies and to trigger the actual benchmarks.

‎utilities/tests/__init__.py renamed to ‎__init__.py

File renamed without changes.

0 commit comments

Comments
 (0)