Skip to content

Commit

Permalink
Merge pull request #114 from Green-Software-Foundation/add-append-docs
Browse files Browse the repository at this point in the history 
add docs for append feature
  • Loading branch information
@manushak
manushak authored Oct 3, 2024
2 parents 0b8dd04 + b05716a commit d53472b
Show file tree
Hide file tree
Showing 3 changed files with 357 additions and 24 deletions.
1 change: 1 addition & 0 deletions docs/major-concepts/if.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ The available options and their shortcuts are:
- `--no-output` or `-n` (optional): suppress the output to console
- `--help` or `-h`: prints out help instruction
- `--debug`: enables IF execution logs
- `--append`: allows you to rerun an already-computed manifest and append new values to the existing data.

The only required command is `--manifest`. Without a valid path to a manifest file, `if-run` has nothing to execute.

Expand Down
337 changes: 337 additions & 0 deletions docs/reference/cli.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -341,3 +341,340 @@ if-csv -m ./my-manifest.yml -p carbon
```sh
if-run -m ./my-manifest.yml | if-csv -p carbon -o ./my-outdata
```


## `--append`

You can re-use a manifest file to make multiple batches of observations, appending the results to the existing outputs. The command that makes this possible is `--append`. To use `--append` you have to pass a manifest files that has **already been computed** - i.e.it already has outputs. If you do, then the newly generated outputs will be appended to the existing output data.

The use case for this is when you want to repeatedly monitor the same resource or set of resources without changign the manifest config - you just want to grab new observations. The `--append` command allows you to do this without havign to generate lots of individual manifest files.

### example

With a computed manifest:

```yaml
name: append
description: >-
a complete pipeline that starts with mocked CPU utilization data and outputs
operational carbon in gCO2eq
initialize:
plugins:
mock-observations:
path: builtin
method: MockObservations
config:
timestamp-from: '2024-03-05T00:00:04.000Z'
timestamp-to: '2024-03-05T00:00:07.000Z'
duration: 1
components:
- name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
generators:
common:
cloud/vendor: azure
randint:
cpu/energy:
min: 1
max: 99
mem/energy:
min: 1
max: 99
sum:
path: builtin
method: Sum
config:
input-parameters:
- cpu/energy
- mem/energy
output-parameter: energy
execution:
command: >-
/home/user/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
/home/user/Code/if/src/index.ts -m
manifests/examples/mock-cpu-util-to-carbon.yml -s
environment:
if-version: 0.4.0
os: linux
os-version: 5.15.0-107-generic
node-version: 21.4.0
date-time: 2024-06-18T14:18:44.864Z (UTC)
dependencies:
- '@babel/[email protected]'
- '@babel/[email protected]'
- '@commitlint/[email protected]'
- '@commitlint/[email protected]'
- '@grnsft/[email protected]'
- '@jest/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
status: success
tree:
pipeline:
compute:
- mock-observations
- sum
regroup:
- cloud/region
- name
defaults: null
inputs:
- timestamp: '2024-03-05T00:00:00.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 5
mem/energy: 10
- timestamp: '2024-03-05T00:00:01.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 71
mem/energy: 5
- timestamp: '2024-03-05T00:00:02.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 36
mem/energy: 74
outputs:
- timestamp: '2024-03-05T00:00:00.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 5
mem/energy: 10
energy: 15
- timestamp: '2024-03-05T00:00:01.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 71
mem/energy: 5
energy: 76
- timestamp: '2024-03-05T00:00:02.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 36
mem/energy: 74
energy: 110
```
run
```sh
npm run if-run -- -m manifests/outputs/features/append.yaml -o manifests/outputs/features/re-append --append
```

And see the following output (with new observations appended to old observations):

```yaml
name: append
description: >-
a complete pipeline that starts with mocked CPU utilization data and outputs
operational carbon in gCO2eq
initialize:
plugins:
mock-observations:
path: builtin
method: MockObservations
config:
timestamp-from: '2024-03-05T00:00:04.000Z'
timestamp-to: '2024-03-05T00:00:07.000Z'
duration: 1
components:
- name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
generators:
common:
cloud/vendor: azure
randint:
cpu/energy:
min: 1
max: 99
mem/energy:
min: 1
max: 99
sum:
path: builtin
method: Sum
config:
input-parameters:
- cpu/energy
- mem/energy
output-parameter: energy
execution:
command: >-
/Users/jcrowley/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
/Users/jcrowley/Development/gsf/if/src/if-run/index.ts -m
manifests/outputs/features/append.yaml -o
manifests/outputs/features/re-append --append
environment:
if-version: 0.6.0
os: macOS
os-version: 14.6.1
node-version: 20.16.0
date-time: 2024-09-04T01:05:58.758Z (UTC)
dependencies:
- '@babel/[email protected]'
- '@babel/[email protected]'
- '@commitlint/[email protected]'
- '@commitlint/[email protected]'
- '@grnsft/[email protected]'
- '@jest/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- '@types/[email protected]'
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
- [email protected]
status: success
tree:
pipeline:
compute:
- mock-observations
- sum
regroup:
- cloud/region
- name
defaults: null
children:
westus3:
children:
server-1:
inputs:
- timestamp: '2024-03-05T00:00:00.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 5
mem/energy: 10
- timestamp: '2024-03-05T00:00:01.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 71
mem/energy: 5
- timestamp: '2024-03-05T00:00:02.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 36
mem/energy: 74
outputs:
- timestamp: '2024-03-05T00:00:00.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 5
mem/energy: 10
energy: 15
- timestamp: '2024-03-05T00:00:01.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 71
mem/energy: 5
energy: 76
- timestamp: '2024-03-05T00:00:02.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 36
mem/energy: 74
energy: 110
- timestamp: '2024-03-05T00:00:04.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 2
mem/energy: 26
energy: 28
- timestamp: '2024-03-05T00:00:05.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 67
mem/energy: 27
energy: 94
- timestamp: '2024-03-05T00:00:06.000Z'
duration: 1
name: server-1
cloud/instance-type: Standard_E64_v3
cloud/region: westus3
cloud/vendor: azure
cpu/energy: 88
mem/energy: 6
energy: 94
```
Loading

0 comments on commit d53472b

Please sign in to comment.