50 shades of storage #30583

adameat · 2025-12-12T07:07:32Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Description for reviewers

...

ydbot · 2025-12-12T07:09:49Z

Run Extra Tests

Run additional tests for this PR. You can customize:

Test Size: small, medium, large (default: all)
Test Targets: any directory path (default: ydb/)
Sanitizers: ASAN, MSAN, TSAN
Coredumps: enable for debugging (default: off)
Additional args: custom ya make arguments

github-actions · 2025-12-12T07:11:17Z

⚪ 2025-12-12 07:11:17 UTC Pre-commit check linux-x86_64-release-asan for dbe65e2 has started.
⚪ 2025-12-12 07:11:33 UTC Artifacts will be uploaded here
⚪ 2025-12-12 07:13:40 UTC ya make is running...
🟡 2025-12-12 08:23:55 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12665	12586	0	59	8	12

🟢 2025-12-12 08:24:05 UTC Build successful.
🟡 2025-12-12 08:24:40 UTC ydbd size 3.9 GiB changed* by +389.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1dfc2ea`	merge: `dbe65e2`	diff	diff %
ydbd size	4 133 996 408 Bytes	4 134 395 136 Bytes	+389.4 KiB	+0.010%
ydbd stripped size	1 534 460 056 Bytes	1 534 654 520 Bytes	+189.9 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-12-12T07:11:38Z

⚪ 2025-12-12 07:11:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for dbe65e2 has started.
⚪ 2025-12-12 07:11:55 UTC Artifacts will be uploaded here
⚪ 2025-12-12 07:14:03 UTC ya make is running...
🟡 2025-12-12 08:52:59 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41082	38146	0	11	2902	23

⚪ 2025-12-12 08:53:15 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-12-12 09:05:29 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
60 (only retried tests)	47	0	0	0	13

🟢 2025-12-12 09:05:36 UTC Build successful.
🟡 2025-12-12 09:06:01 UTC ydbd size 2.3 GiB changed* by +243.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `1dfc2ea`	merge: `dbe65e2`	diff	diff %
ydbd size	2 468 644 104 Bytes	2 468 893 672 Bytes	+243.7 KiB	+0.010%
ydbd stripped size	525 324 544 Bytes	525 398 080 Bytes	+71.8 KiB	+0.014%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-12-12T07:12:26Z

🟢 2025-12-12 07:12:25 UTC The validation of the Pull Request description is successful.

Copilot

Pull request overview

This pull request adds new storage capacity metrics to the viewer API for monitoring storage health and capacity across nodes and storage groups. The PR introduces four new metrics: MaxPDiskUsage, MaxVDiskSlotUsage, MaxVDiskRawUsage, and CapacityAlert, along with support for sorting, grouping, and filtering by these metrics.

Key changes:

Added new storage capacity metrics (MaxPDiskUsage, MaxVDiskSlotUsage, MaxVDiskRawUsage, CapacityAlert) to track storage utilization
Implemented calculation methods for aggregating VDisk and PDisk metrics at node and group levels
Extended API documentation to include new fields for sorting, grouping, and filtering operations

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
ydb/core/viewer/viewer_nodes.h	Added new enum fields, struct members, calculation methods (CalcVDisks, CalcPDisks), and whiteboard request initialization for node-level storage metrics
ydb/core/viewer/viewer_groups.h	Added new enum fields, struct members, CalcCapacityMetrics method, and whiteboard request handling for group-level storage metrics
ydb/core/viewer/tests/test.py	Updated test normalization to include new fields and added new test cases for grouping by CapacityAlert
ydb/core/viewer/tests/canondata/result.json	Updated canonical test output to reflect new fields in API responses
ydb/core/viewer/protos/viewer.proto	Added new optional fields to TStorageGroupInfo and TNodeInfo messages
ydb/core/viewer/pdisk_info.h	Modified whiteboard requests to request all fields instead of defaults
ydb/core/viewer/json_handlers_viewer.cpp	Incremented API version numbers for nodes and groups handlers

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ydb/core/viewer/viewer_nodes.h

ydb/core/viewer/viewer_groups.h

ydb/core/viewer/viewer_nodes.h

github-actions · 2025-12-12T09:24:58Z

⚪ 2025-12-12 09:24:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for 258d4d8 has started.
⚪ 2025-12-12 09:25:19 UTC Artifacts will be uploaded here
⚪ 2025-12-12 09:27:56 UTC ya make is running...
🟡 2025-12-12 11:05:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41084	38149	0	5	2902	28

⚪ 2025-12-12 11:05:53 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-12-12 11:18:43 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
56 (only retried tests)	42	0	0	0	14

🟢 2025-12-12 11:18:50 UTC Build successful.
🟡 2025-12-12 11:19:15 UTC ydbd size 2.3 GiB changed* by +233.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `017c585`	merge: `258d4d8`	diff	diff %
ydbd size	2 468 728 488 Bytes	2 468 967 608 Bytes	+233.5 KiB	+0.010%
ydbd stripped size	525 341 088 Bytes	525 412 512 Bytes	+69.8 KiB	+0.014%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-12-12T09:24:59Z

⚪ 2025-12-12 09:24:59 UTC Pre-commit check linux-x86_64-release-asan for 258d4d8 has started.
⚪ 2025-12-12 09:25:44 UTC Artifacts will be uploaded here
⚪ 2025-12-12 09:27:44 UTC ya make is running...
🟡 2025-12-12 10:36:13 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12665	12544	0	105	7	9

🟢 2025-12-12 10:36:27 UTC Build successful.
🟡 2025-12-12 10:36:59 UTC ydbd size 3.9 GiB changed* by +365.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `017c585`	merge: `258d4d8`	diff	diff %
ydbd size	4 134 132 184 Bytes	4 134 505 920 Bytes	+365.0 KiB	+0.009%
ydbd stripped size	1 534 512 760 Bytes	1 534 692 440 Bytes	+175.5 KiB	+0.012%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

rosik · 2025-12-12T12:40:17Z

ydb/core/viewer/viewer_groups.h

+                case EGroupFields::MaxVDiskSlotUsage:
+                    SortCollection(GroupView, [](const TGroup* group) { return group->MaxVDiskSlotUsage; }, ReverseSort);
+                    break;
+                case EGroupFields::MaxNormalizedOccupancy:


Suggested change

case EGroupFields::MaxNormalizedOccupancy:

case EGroupFields::MaxNormalizedOccupancy:

case EGroupFields::CapacityAlert:

SortCollection(GroupView, [](const TGroup* group) { return group->MaxNormalizedOccupancy; }, ReverseSort);

break;

Поправочка. CapacityAlert надо сортировать по MaxVDiskSlotUsage

rosik · 2025-12-12T12:46:36Z

ydb/core/viewer/viewer_groups.h

+                case EGroupFields::MaxNormalizedOccupancy:
+                    return (float)MaxNormalizedOccupancy;
+                case EGroupFields::CapacityAlert:
+                    return (ui64)CapacityAlert;


Suggested change

case EGroupFields::MaxNormalizedOccupancy:

return (float)MaxNormalizedOccupancy;

case EGroupFields::CapacityAlert:

return (ui64)CapacityAlert;

case EGroupFields::MaxNormalizedOccupancy:

case EGroupFields::CapacityAlert:

return (float)MaxNormalizedOccupancy;

Это нужно чтобы диски с одинаковым алертом (все зеленые) были отсортированы

Я сейчас осознал, что CapacityAlert корректнее сортировать по MaxVDiskSlotUsage, а не по MaxNormalizedOccupancy.

Давай лучше сделаем так:

case EGroupFields::MaxPDiskUsage: return (float)MaxPDiskUsage; case EGroupFields::MaxVDiskSlotUsage: case EGroupFields::CapacityAlert: return (float)MaxVDiskSlotUsage; case EGroupFields::MaxNormalizedOccupancy: return (float)MaxNormalizedOccupancy;

И для узлов и для Group.ApplySort тоже

rosik · 2025-12-12T12:52:35Z

ydb/core/viewer/viewer_groups.h

В void FillVDiskFromVSlotInfo кажется не хватает новых полей по аналогии с https://github.com/ydb-platform/ydb/pull/27792/files#diff-8b1badd7d4809ec27a64cdcb492e027aa5a275f28fe14fa9abf6581e5f4b8838R1391

rosik · 2025-12-12T12:58:54Z

ydb/core/viewer/viewer_groups.h

И в ProcessResponses не хватает инициализации полей https://github.com/ydb-platform/ydb/pull/27792/files#diff-8b1badd7d4809ec27a64cdcb492e027aa5a275f28fe14fa9abf6581e5f4b8838R1455

В канонизированных результатах group.MaxPDiskUsage=0, а должен быть non-zero-number

rosik · 2025-12-12T15:30:57Z

ydb/core/viewer/viewer_nodes.h

    PileName,
+    MaxPDiskUsage,
+    MaxVDiskSlotUsage,
+    MaxVDiskRawUsage,


Suggested change

MaxVDiskRawUsage,

Предлагаю убрать MaxVDiskRawUsage для узлов. Для групп эта метрика нужна для глубокого траблшутинга (см https://github.com/ydb-platform/ydb-rfc/blob/main/164_capacity_metrics.md#рекомендации), а для узлов только будет сбивать с толку. Пользователю мы хотим рассказывать только про PDiskUsage и VDiskSlotUsage (я рассчитываю что они просто заменят собой group.DiskUsage и group.Usage)

rosik · 2025-12-12T15:39:40Z

ydb/core/viewer/viewer_groups.h

+                case EGroupFields::MaxNormalizedOccupancy:
+                    return (float)MaxNormalizedOccupancy;
+                case EGroupFields::CapacityAlert:
+                    return (ui64)CapacityAlert;


Я сейчас осознал, что CapacityAlert корректнее сортировать по MaxVDiskSlotUsage, а не по MaxNormalizedOccupancy.

Давай лучше сделаем так:

case EGroupFields::MaxPDiskUsage: return (float)MaxPDiskUsage; case EGroupFields::MaxVDiskSlotUsage: case EGroupFields::CapacityAlert: return (float)MaxVDiskSlotUsage; case EGroupFields::MaxNormalizedOccupancy: return (float)MaxNormalizedOccupancy;

И для узлов и для Group.ApplySort тоже

rosik · 2025-12-12T15:40:33Z

ydb/core/viewer/viewer_groups.h

+                case EGroupFields::MaxVDiskSlotUsage:
+                    SortCollection(GroupView, [](const TGroup* group) { return group->MaxVDiskSlotUsage; }, ReverseSort);
+                    break;
+                case EGroupFields::MaxNormalizedOccupancy:


Поправочка. CapacityAlert надо сортировать по MaxVDiskSlotUsage

rosik · 2025-12-12T15:42:52Z

ydb/core/viewer/viewer_nodes.h

+        float MaxVDiskRawUsage = 0;
+        float MaxNormalizedOccupancy = 0;


Suggested change

float MaxVDiskRawUsage = 0;

float MaxNormalizedOccupancy = 0;

Как я уже сказал, MaxVDiskRawUsage для узлов я считаю не нужен. MaxNormalizedOccupancy тоже. В моей версии патча он нужен был для сортировки по CapacityAlert, но сейчас я понял что лучше для этого использовать MaxVDiskSlotUsage

rosik · 2025-12-12T15:45:45Z

ydb/core/viewer/viewer_nodes.h

+                case ENodeFields::CapacityAlert:
+                    return static_cast<int>(CapacityAlert);


Suggested change

case ENodeFields::CapacityAlert:

return static_cast<int>(CapacityAlert);

case ENodeFields::CapacityAlert:

return MaxVDiskSlotUsage;

rosik · 2025-12-12T15:47:19Z

ydb/core/viewer/viewer_nodes.h

+                case ENodeFields::MaxVDiskRawUsage:
+                    SortCollection(NodeView, [](const TNode* node) { return node->MaxVDiskRawUsage; }, ReverseSort);
+                    NeedSort = false;
+                    break;
+                case ENodeFields::MaxVDiskSlotUsage:
+                    SortCollection(NodeView, [](const TNode* node) { return node->MaxVDiskSlotUsage; }, ReverseSort);
+                    NeedSort = false;
+                    break;
+                case ENodeFields::CapacityAlert:
+                    SortCollection(NodeView, [](const TNode* node) { return node->MaxNormalizedOccupancy; }, ReverseSort);
+                    NeedSort = false;
+                    break;


Suggested change

case ENodeFields::MaxVDiskRawUsage:

SortCollection(NodeView, [](const TNode* node) { return node->MaxVDiskRawUsage; }, ReverseSort);

NeedSort = false;

break;

case ENodeFields::MaxVDiskSlotUsage:

SortCollection(NodeView, [](const TNode* node) { return node->MaxVDiskSlotUsage; }, ReverseSort);

NeedSort = false;

break;

case ENodeFields::CapacityAlert:

SortCollection(NodeView, [](const TNode* node) { return node->MaxNormalizedOccupancy; }, ReverseSort);

NeedSort = false;

break;

case ENodeFields::MaxVDiskSlotUsage:

case ENodeFields::CapacityAlert:

SortCollection(NodeView, [](const TNode* node) { return node->MaxVDiskSlotUsage; }, ReverseSort);

NeedSort = false;

break;

adameat self-assigned this Dec 12, 2025

adameat requested a review from a team as a code owner December 12, 2025 07:07

Copilot AI review requested due to automatic review settings December 12, 2025 07:07

github-actions bot added the not-for-changelog label Dec 12, 2025

Copilot started reviewing on behalf of adameat December 12, 2025 07:09 View session

Copilot AI reviewed Dec 12, 2025

View reviewed changes

50 shades of storage

54c8947

adameat force-pushed the 50-shades-of-storage branch from 79f1c20 to 54c8947 Compare December 12, 2025 09:23

rosik requested changes Dec 12, 2025

View reviewed changes

rosik reviewed Dec 12, 2025

View reviewed changes

		float MaxVDiskRawUsage = 0;
		float MaxNormalizedOccupancy = 0;

		case ENodeFields::CapacityAlert:
		return static_cast<int>(CapacityAlert);

50 shades of storage #30583

Are you sure you want to change the base?

50 shades of storage #30583

Conversation

adameat commented Dec 12, 2025

Changelog entry

Changelog category

Description for reviewers

Uh oh!

ydbot commented Dec 12, 2025

Run Extra Tests

Uh oh!

github-actions bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rosik Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Dec 12, 2025 •

edited

Loading

github-actions bot commented Dec 12, 2025 •

edited

Loading

github-actions bot commented Dec 12, 2025 •

edited

Loading

github-actions bot commented Dec 12, 2025 •

edited

Loading

rosik Dec 12, 2025 •

edited

Loading