Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shard query splitting: Split queries by stream shards #715

Merged
merged 53 commits into from
Sep 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
14d614a
feat(CombineResponses): create module
matyax Aug 16, 2024
02ab877
test(CombineResponses): add unit tests and adjust implementation
matyax Aug 16, 2024
7ebdfd0
chore: install uuid
matyax Aug 16, 2024
001c0d2
feat(shardQuerySplitting): create module
matyax Aug 16, 2024
8c40a09
chore: interpolate queries
matyax Aug 16, 2024
b93b873
feat(shardQuerySplitting): query from high to low shard
matyax Aug 16, 2024
73f141c
fix(combineResponses): improve nanos handling
matyax Aug 16, 2024
258b7f8
feat(shardQuerySplitting): group shards in requests
matyax Aug 19, 2024
ce3402e
feat(mergeResponses): properly calculate dataframe length
matyax Aug 19, 2024
a305dfc
feat(shardQuerySplitting): retry failed requests
matyax Aug 20, 2024
aa29e4e
feat(mergeResponses): compare timestamps using nanoseconds if present
matyax Aug 21, 2024
dd0d805
test(combineResponses): add extra test case
matyax Aug 21, 2024
3e07749
feat(mergeResponses): fix frame length calculation
matyax Aug 21, 2024
ef009ab
feat(shardQuerySplitting): sort shards desc
matyax Aug 22, 2024
e97ae94
fix(shardQuerySplitting): increase retries
matyax Aug 22, 2024
3fc44b2
feat(shardQuerySplitting): adjust shards based on timerange
matyax Aug 22, 2024
1bd685d
chore(ci): simplify
matyax Aug 22, 2024
4c7d95d
feat(shardQuerySplitting): dynamically calculate max requests
matyax Aug 26, 2024
a689d90
fix(shardQuerySplitting): do not rely on max shard as shards length
matyax Aug 26, 2024
6069305
feat(shardQuerySplitting): pass service name to values call
matyax Aug 27, 2024
7e030d1
feat(shardQuerySplitting): do not retry if maximum series reached
matyax Aug 27, 2024
8c64a37
feat(combineResponses): consider id when comparing log timestamps
matyax Aug 28, 2024
86e6af3
chore: print shards and not max shard
matyax Aug 28, 2024
113fe61
chore: remove max shard
matyax Aug 28, 2024
abdc129
fix(combineResponses): fix nanos recombination
matyax Aug 28, 2024
145797c
chore: words
matyax Aug 29, 2024
47b5501
feat(querying): exclude log queries from sharding
matyax Aug 29, 2024
0066811
fix(getShardRequests): debug group generation
matyax Aug 29, 2024
deb533d
feat(shardQuerySplitting): integrate feature flag
matyax Aug 29, 2024
f1bc31b
feat(shardQuerySplitting): fix falling back to regular query
matyax Sep 2, 2024
d786308
chore: rename functions
matyax Sep 2, 2024
9c3977b
feat(jest): add observable matchers
matyax Sep 3, 2024
23a8170
test(shardQuerySplitting): add base test setup
matyax Sep 3, 2024
8b4d8eb
test(shardQuerySplitting): add more tests
matyax Sep 3, 2024
896b64f
Revert "chore(ci): simplify"
matyax Sep 4, 2024
563767a
chore: spelling
matyax Sep 4, 2024
94c3a6e
chore(shardQuerySplitting): add docs
matyax Sep 4, 2024
1d32c98
chore: simplify
matyax Sep 4, 2024
1f51768
chore: fix test title
matyax Sep 4, 2024
37bc0e8
fix(logql): improve selector for single shard
matyax Sep 9, 2024
65cb27e
feat(shardQuerySpitting): try new sharding approach
matyax Sep 9, 2024
f457123
feat(shardQuerySpitting): improve shard volume spreading
matyax Sep 9, 2024
de87626
chore: spelling
matyax Sep 9, 2024
0705076
feat(shardQuerySplitting): start with the highest numbers
matyax Sep 10, 2024
284af9f
fix(combineResponses): small optimization
matyax Sep 10, 2024
b07278c
feat(shardQuerySplitting): implement incremental backoff
matyax Sep 10, 2024
dd4ba97
chore: words
matyax Sep 10, 2024
ef38488
fix(shardQuerySplitting): skip avg from sharding
matyax Sep 11, 2024
bbe917c
fix(shardQuerySplitting): start with loading state Loading and switch…
matyax Sep 11, 2024
d968b23
chore: update e2e
matyax Sep 11, 2024
bcbf11c
chore: update e2e tests
matyax Sep 11, 2024
333f7d5
chore: disable exploreLogsShardSplitting in ci
matyax Sep 11, 2024
9a91c2d
chore: revert test update
matyax Sep 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker-compose.dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ services:
container_name: 'grafana-logsapp'
platform: 'linux/amd64'
environment:
- GF_FEATURE_TOGGLES_ENABLE=accessControlOnCall lokiLogsDataplane exploreLogsShardSplitting
- GF_FEATURE_TOGGLES_ENABLE=accessControlOnCall lokiLogsDataplane
build:
context: ./.config
args:
Expand Down
17 changes: 17 additions & 0 deletions index.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { Observable } from 'rxjs';

type ObservableType<T> = T extends Observable<infer V> ? V : never;

declare global {
namespace jest {
interface Matchers<R, T = {}> {
toEmitValues<E = ObservableType<T>>(expected: E[]): Promise<CustomMatcherResult>;
/**
* Collect all the values emitted by the observables (also errors) and pass them to the expectations functions after
* the observable ended (or emitted error). If Observable does not complete within OBSERVABLE_TEST_TIMEOUT_IN_MS the
* test fails.
*/
toEmitValuesWith<E = ObservableType<T>>(expectations: (received: E[]) => void): Promise<CustomMatcherResult>;
}
}
}
6 changes: 6 additions & 0 deletions jest-setup.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,15 @@ import { TextDecoder, TextEncoder } from 'util';

import './.config/jest-setup';

import { toEmitValuesWith } from './tests/matchers';

global.TextEncoder = TextEncoder;
global.TextDecoder = TextDecoder;

expect.extend({
toEmitValuesWith,
});

// mock the intersection observer and just say everything is in view
const mockIntersectionObserver = jest.fn().mockImplementation((callback) => ({
observe: jest.fn().mockImplementation((elem) => {
Expand Down
4 changes: 3 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
"@types/react-scroll-sync": "^0.9.0",
"@types/react-table": "^7.7.20",
"@types/testing-library__jest-dom": "5.14.8",
"@types/uuid": "^10.0.0",
"copy-webpack-plugin": "^11.0.0",
"cspell": "6.13.3",
"css-loader": "^6.7.3",
Expand Down Expand Up @@ -96,7 +97,8 @@
"react-dom": "18.2.0",
"react-router-dom": "^5.2.0",
"rxjs": "7.8.1",
"tslib": "2.5.3"
"tslib": "2.5.3",
"uuid": "^10.0.0"
},
"resolutions": {
"**/fast-loops": "^1.1.4"
Expand Down
8 changes: 8 additions & 0 deletions project-words.txt
Original file line number Diff line number Diff line change
Expand Up @@ -476,3 +476,11 @@ yass
noparser
whoopsie
myproject
decbytes
frametype
Nanos
nanos
subquery
Subscription
uuidv
without
Loading
Loading