Skip to content

Commit 6772a15

Browse files
starpitstarpit
committed
feat: add node and pod stats to dashboard
1 parent e293ef6 commit 6772a15

File tree

4 files changed

+272
-18
lines changed

4 files changed

+272
-18
lines changed

‎plugins/plugin-client-default/notebooks/dashboard-live.md

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ layout:
1010
---
1111

1212
--8<-- "./dashboard-summary.md"
13+
--8<-- "./dashboard-source.md"
14+
1315
---
1416

1517
=== "Application Logs"
@@ -34,15 +36,37 @@ layout:
3436
tail -n 500 -f $LOGDIR/resources/gpu.txt
3537
```
3638

37-
=== "Kubernetes Events"
39+
=== "Advanced"
40+
=== "Node Utilization"
3841

39-
```shell
40-
---
41-
execute: now
42-
maximize: true
43-
outputOnly: true
44-
---
45-
tail -n 500 -f $LOGDIR/events/kubernetes.txt
46-
```
42+
```shell
43+
---
44+
execute: now
45+
maximize: true
46+
outputOnly: true
47+
---
48+
tail -n 500 -f $LOGDIR/resources/node-stats.txt
49+
```
50+
51+
=== "Pod Utilization"
52+
53+
```shell
54+
---
55+
execute: now
56+
maximize: true
57+
outputOnly: true
58+
---
59+
tail -n 500 -f $LOGDIR/resources/pod-stats.txt
60+
```
61+
62+
=== "Kubernetes Events"
63+
64+
```shell
65+
---
66+
execute: now
67+
maximize: true
68+
outputOnly: true
69+
---
70+
tail -n 500 -f $LOGDIR/events/kubernetes.txt
71+
```
4772

48-
--8<-- "./dashboard-source.md"

‎plugins/plugin-client-default/notebooks/dashboard.md

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ layout:
1010
---
1111

1212
--8<-- "./dashboard-summary.md"
13+
--8<-- "./dashboard-source.md"
1314

1415
---
1516

@@ -25,13 +26,25 @@ layout:
2526
--8<-- "$LOGDIR/resources/gpu.txt"
2627
```
2728

28-
=== "Kubernetes Events"
29+
=== "Advanced"
30+
=== "Node Utilization"
2931

30-
```json
31-
---
32-
language: shell
33-
---
34-
--8<-- "$LOGDIR/events/kubernetes.txt"
35-
```
32+
```ansi
33+
--8<-- "$LOGDIR/resources/node-stats.txt"
34+
```
35+
36+
=== "Pod Utilization"
37+
38+
```ansi
39+
--8<-- "$LOGDIR/resources/pod-stats.txt"
40+
```
41+
42+
=== "Kubernetes Events"
43+
44+
```json
45+
---
46+
language: shell
47+
---
48+
--8<-- "$LOGDIR/events/kubernetes.txt"
49+
```
3650

37-
--8<-- "./dashboard-source.md"

‎tests/plugin-codeflare/dashboard/inputs/1/resources/node-stats.txt

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
Sample 2022-06-21T09:01:50-04:00
2+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
3+
4+
Sample 2022-06-21T09:02:10-04:00
5+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
6+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
7+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
8+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
9+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
10+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
11+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
12+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
13+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
14+
15+
Sample 2022-06-21T09:02:26-04:00
16+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
17+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
18+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
19+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
20+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
21+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
22+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
23+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
24+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
25+
26+
Sample 2022-06-21T09:02:41-04:00
27+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
28+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
29+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
30+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
31+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
32+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
33+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
34+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
35+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
36+
37+
Sample 2022-06-21T09:02:57-04:00
38+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
39+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
40+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
41+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
42+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
43+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
44+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
45+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
46+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
47+
48+
Sample 2022-06-21T09:03:12-04:00
49+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
50+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
51+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
52+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
53+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
54+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
55+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
56+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
57+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
58+
59+
Sample 2022-06-21T09:03:28-04:00
60+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
61+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
62+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
63+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
64+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
65+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
66+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
67+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
68+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
69+
70+
Sample 2022-06-21T09:03:43-04:00
71+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
72+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
73+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
74+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
75+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
76+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
77+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
78+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
79+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
80+
81+
Sample 2022-06-21T09:03:59-04:00
82+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
83+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
84+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
85+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
86+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
87+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
88+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
89+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
90+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
91+
92+
Sample 2022-06-21T09:04:14-04:00
93+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
94+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
95+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
96+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
97+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
98+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
99+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
100+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
101+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
102+
103+
Sample 2022-06-21T09:04:29-04:00
104+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
105+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
106+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
107+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
108+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
109+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
110+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
111+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
112+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
113+
114+
Sample 2022-06-21T09:04:45-04:00
115+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
116+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
117+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
118+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
119+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
120+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
121+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
122+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
123+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
124+
125+
Sample 2022-06-21T09:05:00-04:00
126+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
127+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
128+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
129+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
130+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
131+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
132+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
133+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
134+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
135+
136+
Sample 2022-06-21T09:05:16-04:00
137+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
138+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
139+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
140+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
141+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
142+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
143+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
144+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
145+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
146+
147+
Sample 2022-06-21T09:05:31-04:00
148+
NAME GPUCap GPUFree CPUCap CPUFree MemCap MemFree DiskCap DiskFree Type
149+
ip-10-0-131-199.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
150+
ip-10-0-131-59.ec2.internal <none> <none> 4 3500m 16407104Ki 15256128Ki 125293548Ki 115470533646 m4.xlarge
151+
ip-10-0-136-165.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
152+
ip-10-0-138-45.ec2.internal 1 1 8 7500m 62855724Ki 61704748Ki 125293548Ki 115470533646 p3.2xlarge
153+
ip-10-0-154-249.ec2.internal <none> <none> 2 1500m 8149576Ki 6998600Ki 125293548Ki 115470533646 m4.large
154+
ip-10-0-158-5.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
155+
ip-10-0-167-175.ec2.internal <none> <none> 2 1500m 8149572Ki 6998596Ki 125293548Ki 115470533646 m4.large
156+
ip-10-0-169-105.ec2.internal <none> <none> 4 3500m 16407108Ki 15256132Ki 125293548Ki 115470533646 m4.xlarge
157+

0 commit comments

Comments
 (0)