Add larger memory flavors (#938)

Signed-off-by: Marc Schöchlin <[email protected]>
Signed-off-by: Matthias Büchse <[email protected]>
Signed-off-by: Kurt Garloff <[email protected]>
Co-authored-by: Marc Schöchlin <[email protected]>
Co-authored-by: Matthias Büchse <[email protected]>
Co-authored-by: Kurt Garloff <[email protected]>

Author: 4 people

.github/workflows/create-flavors-spec.yml

@@ -5,8 +5,8 @@ on:
     branches:
     - main
     paths:
-      - 'Tests/iaas/standard-flavors/flavor-manager-input.py'
-      - 'Tests/iaas/scs-0103-v1-flavors.yaml'
+      - 'Tests/iaas/scs_0103_standard_flavors/standard_flavors.py'
+      - 'Tests/iaas/generate_flavors_yaml.py'
   workflow_dispatch:
 
 jobs:
@@ -27,7 +27,7 @@ jobs:
         pip install PyYAML==6.0
     - name: Run flavor-manager-input.py
       run: |
-        python Tests/iaas/standard-flavors/flavor-manager-input.py < Tests/iaas/scs-0103-v1-flavors.yaml > Tests/iaas/SCS-Spec.MandatoryFlavors.verbose.yaml
+        python Tests/iaas/generate_flavors_yaml.py > Tests/iaas/SCS-Spec.MandatoryFlavors.verbose.yaml
     - name: Commit changes
       uses: stefanzweifel/git-auto-commit-action@v4
       with:

Standards/scs-0103-v1-standard-flavors.md

@@ -7,21 +7,18 @@ track: IaaS
 description: |
   The SCS-0103 standard outlines mandatory and recommended specifications for flavors and properties in OpenStack
   environments to ensure uniformity across SCS clouds. Mandatory and recommended flavors are defined with specific
-  configurations of vCPUs, vCPU types, RAM, and root disk sizes, alongside extra specs like scs:name-vN, scs:cpu-type,
+  configurations of vCPUs, vCPU types, RAM, and root disk sizes, alongside extra_specs like scs:name-vN, scs:cpu-type,
   and scs:diskN-type to detail the flavor's specifications. This standard facilitates guaranteed availability and
   consistency of flavors, simplifying the deployment process for DevOps teams.
 ---
 
 ## Introduction
 
-This is v1.1 of the standard, which lifts the following restriction regarding the property `scs:name-vN`:
-this property may now be used on any flavor, rather than standard flavors only. In addition, the "vN" is
-now interpreted as "name variant N" instead of "version N of the naming standard". Note that this change
-indeed preserves compliance, i.e., compliance with v1.0 implies compliance with v1.1.
+Note that this is v1.2 of this standard. See the closing section for more details.
 
 ## Terminology
 
-extra_specs
+extra\_specs:
   Additional properties on an OpenStack flavor, see
   [OpenStack Nova user documentation](https://docs.openstack.org/nova/2024.1/user/flavors.html#extra-specs)
   and
@@ -36,9 +33,9 @@ OpenStack providers thus typically offer a large selection of flavors.
 While flavors can be discovered (`openstack flavor list`), it is helpful for users (DevOps teams),
 to have a guaranteed set of flavors available on all SCS clouds, so these need not be discovered.
 
-## Properties (extra_specs)
+## Properties (extra\_specs)
 
-The following extra_specs are recognized, together with the respective semantics:
+The following extra\_specs are recognized, together with the respective semantics:
 
 - `scs:name-vN=NAME` (where `N` is a positive integer, and `NAME` is some string) means that
   `NAME` is a valid name for this flavor according to any major version of the [SCS standard on
@@ -56,22 +53,22 @@ The following extra_specs are recognized, together with the respective semantics
 
 Whenever ANY of these are present on ANY flavor, the corresponding semantics must be satisfied.
 
-The extra_spec `scs:name-vN` is to be interpreted as "name variant N". This name scheme is designed to be
+The extra\_spec `scs:name-vN` is to be interpreted as "name variant N". This name scheme is designed to be
 backwards compatible with v1.0 of this standard, where `scs:name-vN` is interpreted as
 "name according to naming standard vN". We abandon this former interpretation for two reasons:
 
 1. the naming standards admit multiple (even many) names for the same flavor, and we want to provide a means
    of advertising more than one of them (said standards recommend using two: a short one and a long one),
 2. the same flavor name may be valid according to multiple versions at the same time, which would lead to
-   a pollution of the extra_specs with redundant properties; for instance, the name
+   a pollution of the extra\_specs with redundant properties; for instance, the name
    `SCS-4V-16` is valid for both [scs-0100-v2](scs-0100-v2-flavor-naming.md) and
    [scs-0100-v3](scs-0100-v3-flavor-naming.md), and, since it does not use any extension, it will be valid
    for any future version that only changes the extensions, such as the GPU vendor and architecture.
 
 Note that it is not required to use consecutive numbers to number the name variants.
 This way, it becomes easier to remove a single variant (no "closing the gap" required).
 
-If extra_specs of the form `scs:name-vN` are used to specify SCS flavor names, it is RECOMMENDED to include
+If extra\_specs of the form `scs:name-vN` are used to specify SCS flavor names, it is RECOMMENDED to include
 names for the latest stable major version of the standard on flavor naming.
 
 ## Standard SCS flavors
@@ -99,7 +96,7 @@ Note that this statement does not preclude the existence of additional flavors.
 | SCS-4V-32        |      4 | shared-core   |         32 |                 |            |
 | SCS-1L-1         |      1 | crowded-core  |          1 |                 |            |
 
-### Recommended
+### Recommended, part 1
 
 | Recommended name | vCPUs  | vCPU type     | RAM [GiB]  | Root disk [GB]  | Disk type  |
 | ---------------- | ------ | ------------- | ---------- | --------------- | ---------- |
@@ -117,6 +114,26 @@ Note that this statement does not preclude the existence of additional flavors.
 | SCS-4V-32-100    |      4 | shared-core   |         32 |             100 | (any)      |
 | SCS-1L-1-5       |      1 | crowded-core  |          1 |               5 | (any)      |
 
+### Recommended, part 2
+
+The following flavors were added with v1.2 of this standard. If a CSP wants to offer
+flavors with more RAM than the ones above, they should try to use these.
+
+| Recommended name | vCPUs  | vCPU type     | RAM [GiB]  | Root disk [GB]  | Disk type  |
+| ---------------- | ------ | ------------- | ---------- | --------------- | ---------- |
+| SCS-16V-64       |     16 | shared-core   |         64 |                 |            |
+| SCS-8V-64        |      8 | shared-core   |         64 |                 |            |
+| SCS-16V-128      |     16 | shared-core   |        128 |                 |            |
+
+Note that no flavors with disks have been added here; providers are of course welcome to
+also add variants with unspecified (e.g. `-200`) or ssd+ (e.g. `-200s`) disk types.
+Sticking to the 5, 10, 20, 50, 100, 200, 500, 1000 schedule for disk sizes is recommended
+in that case to avoid unnecessary fragmentation.
+
+Likewise, flavors with more vCPUs (e.g. `-32V`, `-64V`) may be added and we recommend
+sticking to powers of two and to keep the vCPU to GiB RAM ratios 1:2, 1:4 and 1:8,
+unless customers have very specific demands.
+
 ### Guarantees and properties
 
 The figures given in the table (number of CPUs, amount of RAM, root disk size) must match
@@ -158,6 +175,13 @@ instance life cycle.)
 
 ## Previous standard versions
 
+This is v1.2 of the standard, which adds recommended flavors with more RAM.
+
+Version 1.1 lifted a restriction regarding the property `scs:name-vN` as follows:
+this property may now be used on any flavor, rather than standard flavors only. In addition, the "vN" is
+now interpreted as "name variant N" instead of "version N of the naming standard". Note that this change
+indeed preserves compliance, i.e., compliance with v1.0 implies compliance with v1.1.
+
 The list of standard flavors used to be part of the flavor naming standard up until
 [version 3](scs-0100-v3-flavor-naming.md). The following changes have been made to
 the list in comparison with said standard:

Standards/scs-0103-w1-standard-flavors-implementation.md

@@ -9,7 +9,10 @@ supplements:
 ## Operational tooling
 
 The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
-create all standard, mandatory SCS flavors for you. It takes input that can be generated by
+create all standard, mandatory as well as recommended SCS flavors for you. It now has a `--limit-memory`
+(defaulting to 32 GiB) to skip the creation of recommended flavors above this memory limit.
+
+You can generate input for it using the tool
 [`flavor-manager-input.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/scs_0100_flavor_naming/flavor-manager-input.py).
 
 ## Automated tests

Tests/iaas/SCS-Spec.MandatoryFlavors.verbose.yaml

@@ -109,90 +109,120 @@ recommended:
   ram: 4096
   disk: 10
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-1V:4:10
   scs:name-v2: SCS-1V-4-10
 - name: SCS-2V-8-20
   cpus: 2
   ram: 8192
   disk: 20
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-2V:8:20
   scs:name-v2: SCS-2V-8-20
 - name: SCS-4V-16-50
   cpus: 4
   ram: 16384
   disk: 50
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-4V:16:50
   scs:name-v2: SCS-4V-16-50
 - name: SCS-8V-32-100
   cpus: 8
   ram: 32768
   disk: 100
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-8V:32:100
   scs:name-v2: SCS-8V-32-100
 - name: SCS-1V-2-5
   cpus: 1
   ram: 2048
   disk: 5
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-1V:2:5
   scs:name-v2: SCS-1V-2-5
 - name: SCS-2V-4-10
   cpus: 2
   ram: 4096
   disk: 10
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-2V:4:10
   scs:name-v2: SCS-2V-4-10
 - name: SCS-4V-8-20
   cpus: 4
   ram: 8192
   disk: 20
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-4V:8:20
   scs:name-v2: SCS-4V-8-20
 - name: SCS-8V-16-50
   cpus: 8
   ram: 16384
   disk: 50
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-8V:16:50
   scs:name-v2: SCS-8V-16-50
 - name: SCS-16V-32-100
   cpus: 16
   ram: 32768
   disk: 100
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-16V:32:100
   scs:name-v2: SCS-16V-32-100
 - name: SCS-1V-8-20
   cpus: 1
   ram: 8192
   disk: 20
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-1V:8:20
   scs:name-v2: SCS-1V-8-20
 - name: SCS-2V-16-50
   cpus: 2
   ram: 16384
   disk: 50
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-2V:16:50
   scs:name-v2: SCS-2V-16-50
 - name: SCS-4V-32-100
   cpus: 4
   ram: 32768
   disk: 100
   scs:cpu-type: shared-core
+  scs:disk0-type: network
   scs:name-v1: SCS-4V:32:100
   scs:name-v2: SCS-4V-32-100
 - name: SCS-1L-1-5
   cpus: 1
   ram: 1024
   disk: 5
   scs:cpu-type: crowded-core
+  scs:disk0-type: network
   scs:name-v1: SCS-1L:1:5
   scs:name-v2: SCS-1L-1-5
-
+- name: SCS-16V-64
+  cpus: 16
+  ram: 65536
+  scs:cpu-type: shared-core
+  scs:name-v1: SCS-16V:64
+  scs:name-v2: SCS-16V-64
+- name: SCS-8V-64
+  cpus: 8
+  ram: 65536
+  scs:cpu-type: shared-core
+  scs:name-v1: SCS-8V:64
+  scs:name-v2: SCS-8V-64
+- name: SCS-16V-128
+  cpus: 16
+  ram: 131072
+  scs:cpu-type: shared-core
+  scs:name-v1: SCS-16V:128
+  scs:name-v2: SCS-16V-128

Tests/iaas/SCS-Spec.MandatoryFlavors.yaml

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""
+Generate flavor specification file for osism's flavor manager,
+cf. https://github.com/osism/openstack-flavor-manager .
+
+The spec file is output to stdout.
+"""
+import logging
+import sys
+
+import yaml
+
+from scs_0100_flavor_naming.flavor_names import compute_flavor_spec
+from scs_0103_standard_flavors.standard_flavors import \
+    SCS_0103_V1_MANDATORY, SCS_0103_V1_RECOMMENDED
+
+logger = logging.getLogger(__name__)
+
+
+def convert_flavor_spec(canonical_name, spec):
+    """convert `spec` into format for openstack-flavor-manager"""
+    converted = {
+        'name': canonical_name,
+        'cpus': spec['cpus'],
+        'ram': int(1024 * spec['ram']),
+    }
+    if 'disk' in spec:
+        converted['disk'] = spec['disk']
+    for k, v in spec.items():
+        if not k.startswith('scs:'):
+            continue
+        converted[k] = v
+    return converted
+
+
+def compute_spec_list(canonical_names):
+    return [
+        convert_flavor_spec(
+            canonical_name,
+            compute_flavor_spec(canonical_name),
+        )
+        for canonical_name in canonical_names
+    ]
+
+
+def main(argv):
+    logging.basicConfig(format='%(levelname)s: %(message)s', level=logging.INFO)
+
+    # boilerplate / scaffolding
+    result = yaml.safe_load("""
+reference:
+  - field: name
+    mandatory_prefix: SCS-
+  - field: public
+    default: true
+  - field: disabled
+    default: false
+  - field: cpus
+  - field: ram
+  - field: disk
+mandatory: []
+recommended: []
+""")
+
+    result['mandatory'] = compute_spec_list(SCS_0103_V1_MANDATORY)
+    result['recommended'] = compute_spec_list(SCS_0103_V1_RECOMMENDED)
+
+    print(yaml.dump(result, sort_keys=False))
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

Tests/iaas/generate_flavors_yaml.py

@@ -14,9 +14,9 @@
 
 import openstack
 
+from scs_0100_flavor_naming.flavor_names import compute_flavor_spec
 from scs_0100_flavor_naming.flavor_names_check import \
-    compute_scs_flavors, compute_scs_0100_syntax_check, compute_scs_0100_semantics_check, \
-    compute_flavor_spec
+    compute_scs_flavors, compute_scs_0100_syntax_check, compute_scs_0100_semantics_check
 from scs_0101_entropy.entropy_check import \
     compute_scs_0101_image_property, compute_scs_0101_flavor_property, compute_canonical_image, \
     compute_collected_vm_output, compute_scs_0101_entropy_avail, compute_scs_0101_rngd, \