Skip to content

Commit 9df3896

Browse files
jeremiaswernerjeremiaswerner
committed
migrate the simulation tutorial to walk milestone
1 parent 1a8c5ee commit 9df3896

File tree

13 files changed

+434
-0
lines changed

13 files changed

+434
-0
lines changed

‎beta/serverless-fleets/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,10 @@ data/taskstore/*
66
data/output/docling/*
77
data/output/wordcount/*
88
data/output/inferencing/*
9+
data/output/simulation/*
910
!data/output/docling/.keep
1011
!data/output/wordcount/.keep
1112
!data/output/inferencing/.keep
13+
!data/output/simulation/.keep
1214
*/.DS_Store
1315
.DS_Store

‎beta/serverless-fleets/data/output/simulation/.keep‎

Whitespace-only changes.

‎beta/serverless-fleets/images/examples_simulation_flow.png‎

4.41 KB
Loading

‎beta/serverless-fleets/tutorials/simulation/.ceignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
venv

‎beta/serverless-fleets/tutorials/simulation/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
venv

‎beta/serverless-fleets/tutorials/simulation/Dockerfile‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
FROM python:3.11-slim-bookworm
2+
3+
ENV GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no"
4+
5+
RUN apt-get update \
6+
&& apt-get clean
7+
8+
WORKDIR /app
9+
10+
COPY simulate.py /app/simulate.py
11+
COPY requirements.txt /app/requirements.txt
12+
13+
RUN python3 -m ensurepip && pip install --no-cache-dir -r /app/requirements.txt
14+
15+
# Reset the entrypoint
16+
ENTRYPOINT []
17+
18+
CMD ["python3", "simulate.py"]
19+

‎beta/serverless-fleets/tutorials/simulation/README.md‎

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
# Tutorial: Monte Carlo Simulation
2+
3+
This tutorial provides a comprehensive guide of running a simple stock price simulation from local source code using serverless fleets. The example, performs a Monte Carlo Simulation to calculate the Value-at-Risk of 24 stocks. As an input the user provides the source code and the list of 24 commands for each stock ticker. Code Engine *builds* are used to build the container image. The *serverless fleet* will autonomously spawns workers to run the simulation. The result is written to a Cloud Object Storage bucket.
4+
5+
Key steps covered in the Tutorial:
6+
1. Containerization with Code Engine: Build the simulation container and push it to a registry for deployment.
7+
3. Run a fleet of workers that automatically runs the container, ensuring scalability and efficiency.
8+
4. Download the results from COS
9+
10+
11+
![](../../images/examples_simulation_flow.png)
12+
13+
> Note: The tutorial assumes that you have created the fleet sandbox using the fully automated approach which creates the rclone environment as well as the upload/download scripts. If that's not the case, you would need to upload the PDFs and download the results using the COS CLI or other means.
14+
15+
## Steps
16+
17+
18+
### Step 1 - Build and Push the container registry
19+
20+
Build the container image using Code Engine's build capabilities by running the following command in the `tutorials/simulation` directory.
21+
22+
If you're interested review the code, by looking at the [simulate.py](./simulate.py), which contains a simple method that downloads the last year stock prices of a stock ticker, performs 100k simulations and writes the VaR at a csv file. It receives the stock ticker and output directory as input arguments.
23+
```
24+
cat simulate.py
25+
```
26+
27+
Now, run the build script to run a Code Engine build to build a container image using and push it to the container registry
28+
29+
```
30+
cd tutorials/simulation
31+
./build
32+
```
33+
34+
### Step 2 - Prepare the tasks
35+
36+
In this tutorial we use the `--tasks-from-file commands.jsonl` option to submit the tasks. Therefore we have prepared a file in [jsonfiles](https://jsonlines.org/) format which contains 1 task per line. Each line specifies command and arguments for this task. Let's review the file in the `tutorials/simulation` directory:
37+
38+
```
39+
cat commands.jsonl
40+
```
41+
42+
<a name="Output"></a>
43+
<details>
44+
<summary>Output</summary>
45+
46+
```
47+
➜ simulation cat commands.jsonl
48+
{ "cmds": ["python3"], "args": ["simulate.py", "AKAM", "/output"]}
49+
{ "cmds": ["python3"], "args": ["simulate.py", "AA", "/output"]}
50+
{ "cmds": ["python3"], "args": ["simulate.py", "MO", "/output"]}
51+
{ "cmds": ["python3"], "args": ["simulate.py", "AMZN", "/output"]}
52+
{ "cmds": ["python3"], "args": ["simulate.py", "AMGN", "/output"]}
53+
{ "cmds": ["python3"], "args": ["simulate.py", "AAPL", "/output"]}
54+
{ "cmds": ["python3"], "args": ["simulate.py", "T", "/output"]}
55+
{ "cmds": ["python3"], "args": ["simulate.py", "BA", "/output"]}
56+
{ "cmds": ["python3"], "args": ["simulate.py", "CAT", "/output"]}
57+
{ "cmds": ["python3"], "args": ["simulate.py", "CVX", "/output"]}
58+
{ "cmds": ["python3"], "args": ["simulate.py", "DIS", "/output"]}
59+
{ "cmds": ["python3"], "args": ["simulate.py", "KO", "/output"]}
60+
{ "cmds": ["python3"], "args": ["simulate.py", "DELL", "/output"]}
61+
{ "cmds": ["python3"], "args": ["simulate.py", "F", "/output"]}
62+
{ "cmds": ["python3"], "args": ["simulate.py", "INTC", "/output"]}
63+
{ "cmds": ["python3"], "args": ["simulate.py", "IBM", "/output"]}
64+
{ "cmds": ["python3"], "args": ["simulate.py", "MSFT", "/output"]}
65+
{ "cmds": ["python3"], "args": ["simulate.py", "NFLX", "/output"]}
66+
{ "cmds": ["python3"], "args": ["simulate.py", "NVDA", "/output"]}
67+
{ "cmds": ["python3"], "args": ["simulate.py", "ORCL", "/output"]}
68+
{ "cmds": ["python3"], "args": ["simulate.py", "QCOM", "/output"]}
69+
{ "cmds": ["python3"], "args": ["simulate.py", "X", "/output"]}
70+
{ "cmds": ["python3"], "args": ["simulate.py", "VZ", "/output"]}
71+
{ "cmds": ["python3"], "args": ["simulate.py", "V", "/output"]}
72+
```
73+
74+
</details>
75+
<br/>
76+
77+
78+
### Step 3 - Run the Fleet
79+
80+
Now run the fleet to process the 24 stock tickers. In this tutorial we use the `--tasks-from-file commands.jsonl` to specify the tasks. Each task will get 1 CPU and 4 GB memory. We specify a specific machine profile `--worker-profile mx2-4x32` and configure `--max-scale 24` to run all 24 simulations in parallel. Therefore, the system is deploying 6 workers, each running 4 instances concurrently, e.g. each worker is running 4 simulations at a point in time.
81+
```
82+
./run
83+
```
84+
85+
<a name="Output"></a>
86+
<details>
87+
<summary>Output</summary>
88+
89+
```
90+
➜ simulation ./run
91+
ibmcloud code-engine beta fleet create --name fleet-c042e88d-1
92+
--image private.br.icr.io/ce--fleet-simulation/simulation
93+
--registry-secret ce-auto-icr-private-br-sao
94+
--worker-profile mx2-4x32
95+
--tasks-from-local-file commands.jsonl
96+
--cpu 1
97+
--memory 8G
98+
--max-scale 24
99+
--mount-data-store /output=fleet-output-store:/simulation
100+
Successfully created fleet with name 'fleet-c042e88d-1' and ID '630569df-c6c0-44dc-a376-e2f5d8d7aad8'
101+
Run 'ibmcloud ce beta fleet get --id 630569df-c6c0-44dc-a376-e2f5d8d7aad8' to check the fleet status.
102+
Run 'ibmcloud ce beta fleet worker list --fleet-id 630569df-c6c0-44dc-a376-e2f5d8d7aad8' to retrieve a list of provisioned workers.
103+
Run 'ibmcloud ce beta fleet task list --fleet-id 630569df-c6c0-44dc-a376-e2f5d8d7aad8' to retrieve a list of tasks.
104+
OK
105+
```
106+
</details>
107+
<br/>
108+
109+
Review the fleet
110+
```
111+
ibmcloud code-engine beta fleet get --id <fleet-id>
112+
```
113+
<a name="Output"></a>
114+
<details>
115+
<summary>Output</summary>
116+
117+
```
118+
➜ simulation ibmcloud ce beta fleet get --id 630569df-c6c0-44dc-a376-e2f5d8d7aad8
119+
Getting fleet '630569df-c6c0-44dc-a376-e2f5d8d7aad8'...
120+
OK
121+
122+
Name: fleet-c042e88d-1
123+
ID: 630569df-c6c0-44dc-a376-e2f5d8d7aad8
124+
Status: pending
125+
Created: 5s
126+
Project region: br-sao
127+
Project name: fleetlab-dev--ce-project
128+
129+
Tasks status:
130+
Failed: 0
131+
Canceled: 0
132+
Successful: 0
133+
Running: 0
134+
Pending: 24
135+
Total: 24
136+
137+
Code:
138+
Container image reference: private.br.icr.io/ce--fleet-simulation/simulation
139+
Registry access secret: ce-auto-icr-private-br-sao
140+
141+
Tasks specification:
142+
Task state store: fleet-task-store
143+
Data store JSON reference: fleet-task-store
144+
Data store object path: /ce/2c76a9f0-507e-472b-84be-81efe50403f8/fleet-input/80d4f857-62f9-4292-9672-364109ae4aa6.jsonl
145+
146+
Resources and scaling:
147+
CPU per instance: 1
148+
Memory per instance: 8G
149+
Preferred worker profile: mx2-4x32
150+
Max number of instances: 24
151+
Max retries per task: 3
152+
153+
Network placement:
154+
Subnet CRN 0: crn:v1:bluemix:public:is:br-sao-1:a/327016f62a9544c18e7efdd4213297dd::subnet:02t7-61ad2d36-695c-41b2-8bd1-38ee926cb94a
155+
```
156+
</details>
157+
<br/>
158+
159+
160+
161+
Verify that the machines are starting
162+
```
163+
ibmcloud code-engine beta fleet worker list --fleet-id <fleet-id>
164+
```
165+
<a name="Output"></a>
166+
<details>
167+
<summary>Output</summary>
168+
169+
```
170+
➜ ibmcloud ce beta fleet worker list --fleet-id 709f5fee-59ca-41b3-b518-9e5f665e4d78
171+
Listing serverless fleet workers...
172+
OK
173+
174+
Name ID Status Profile IP Zone Version
175+
fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-0 885a775f-9918-4da5-b9ee-40bb3dc3f02e initializing mx2-4x32 10.250.0.24 br-sao-1 v0.0.78
176+
fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-1 554c2232-f7fe-4081-babc-c7490e90742a initializing mx2-4x32 10.250.0.23 br-sao-1 v0.0.78
177+
fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-2 9b7f8195-10c1-43d5-854d-09043608f4d9 initializing mx2-4x32 10.250.0.25 br-sao-1 v0.0.78
178+
fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-3 033c0bbf-f601-4df5-a53e-76b9fe7ef255 initializing mx2-4x32 10.250.0.22 br-sao-1 v0.0.78
179+
fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-4 c21b4a8b-7468-415c-905e-7cafa8646222 initializing mx2-4x32 10.250.0.21 br-sao-1 v0.0.78
180+
fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-5 1113a857-9880-4193-9e6e-d62fc9c66a0f initializing mx2-4x32 10.250.0.26 br-sao-1 v0.0.78
181+
```
182+
</details>
183+
<br/>
184+
185+
Observe the tasks:
186+
187+
```
188+
ibmcloud code-engine beta fleet task list --fleet-id <fleet-id>
189+
```
190+
<a name="Output"></a>
191+
<details>
192+
<summary>Output</summary>
193+
194+
```
195+
➜ serverless-fleets ibmcloud ce beta fleet task list --fleet-id 709f5fee-59ca-41b3-b518-9e5f665e4d78
196+
Listing serverless fleet tasks...
197+
OK
198+
199+
Index ID Status Result code Worker name
200+
000-00000-00000000000000000000 767658c4-a311-5cdb-86ee-1f81d5a0546d running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-4
201+
000-00000-00000000000000000001 8b6cf1f7-839e-56e0-b5ba-5cf76b950774 running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-3
202+
000-00000-00000000000000000002 53d21f36-a46a-5caa-a3ec-ad37c33bf9ac running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-3
203+
000-00000-00000000000000000003 5b4426d6-bb15-5c39-b38a-bd42bbc402ca running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-4
204+
000-00000-00000000000000000004 9bc3ab9b-36a9-5bf1-aba4-6f594ca520ed running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-0
205+
000-00000-00000000000000000005 b21f26d8-bda3-5950-b109-f9eac16381ec running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-5
206+
000-00000-00000000000000000006 78d47365-3ced-59bb-b895-955c52fb4c28 running fleet-709f5fee-59ca-41b3-b518-9e5f665e4d78-1
207+
...
208+
```
209+
</details>
210+
<br/>
211+
212+
### Step 4 - Download results
213+
214+
Download the results from the COS by running the following command in the root directory:
215+
```
216+
./download
217+
```
218+
219+
You can find the results under
220+
```
221+
ls -l data/output/simulation/ticker_*
222+
```
223+
224+
<a name="Output"></a>
225+
<details>
226+
<summary>Output</summary>
227+
228+
```
229+
➜ serverless-fleets ls -l data/output/simulation/ticker_*
230+
-rw-r--r-- 1 jeremiaswerner staff 31 Sep 10 15:16 ticker_AA.result
231+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_AAPL.result
232+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_AKAM.result
233+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_AMGN.result
234+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_AMZN.result
235+
-rw-r--r-- 1 jeremiaswerner staff 31 Sep 10 15:16 ticker_BA.result
236+
-rw-r--r-- 1 jeremiaswerner staff 32 Sep 10 15:16 ticker_CAT.result
237+
-rw-r--r-- 1 jeremiaswerner staff 32 Sep 10 15:16 ticker_CVX.result
238+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_DELL.result
239+
-rw-r--r-- 1 jeremiaswerner staff 32 Sep 10 15:16 ticker_DIS.result
240+
-rw-r--r-- 1 jeremiaswerner staff 30 Sep 10 15:16 ticker_F.result
241+
-rw-r--r-- 1 jeremiaswerner staff 32 Sep 10 15:16 ticker_IBM.result
242+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_INTC.result
243+
-rw-r--r-- 1 jeremiaswerner staff 31 Sep 10 15:16 ticker_KO.result
244+
-rw-r--r-- 1 jeremiaswerner staff 31 Sep 10 15:16 ticker_MO.result
245+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_MSFT.result
246+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_NFLX.result
247+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_NVDA.result
248+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_ORCL.result
249+
-rw-r--r-- 1 jeremiaswerner staff 33 Sep 10 15:16 ticker_QCOM.result
250+
-rw-r--r-- 1 jeremiaswerner staff 30 Sep 10 15:16 ticker_T.result
251+
-rw-r--r-- 1 jeremiaswerner staff 30 Sep 10 15:16 ticker_V.result
252+
-rw-r--r-- 1 jeremiaswerner staff 31 Sep 10 15:16 ticker_VZ.result
253+
-rw-r--r-- 1 jeremiaswerner staff 27 Sep 10 15:16 ticker_X.result
254+
```
255+
</details>
256+
<br/>
257+

‎beta/serverless-fleets/tutorials/simulation/build‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/bin/sh
2+
3+
#!/bin/bash
4+
5+
echo "Cleaning up previous images ..."
6+
IMAGE=$(ibmcloud cr images | grep "ce--fleet-simulation" | awk '{print $1}')
7+
if [[ "$IMAGE" != "" ]]; then
8+
ibmcloud cr image-rm $(ibmcloud cr images | grep "ce--fleet-simulation" | awk '{print $1}')
9+
fi
10+
11+
# getting the automatically created registry secret, e.g. ce-auto-icr-private-<region>
12+
REGION=$(ibmcloud target -o json | jq -r '.region.name')
13+
REGISTRY_SECRET_NAME="ce-auto-icr-private-$REGION"
14+
REGISTRY_HOST=$(ibmcloud ce secret get -n $REGISTRY_SECRET_NAME -o json | jq -r ".data.server")
15+
16+
echo "Building new image $REGISTRY_HOST/ce--fleet-simulation-${uuid}/simulation:latest from ./src "
17+
echo "This takes about 1-2 minutes ..."
18+
19+
uuid=$(uuidgen | tr '[:upper:]' '[:lower:]' | awk -F- '{print $1}')
20+
21+
ibmcloud ce buildrun submit --source . --strategy dockerfile --image $REGISTRY_HOST/ce--fleet-simulation/simulation:latest --registry-secret $REGISTRY_SECRET_NAME --name ce--fleet-simulation-build-${uuid} --size medium --timeout 300
22+
23+
ibmcloud ce buildrun logs -f -n ce--fleet-simulation-build-${uuid}
24+

‎beta/serverless-fleets/tutorials/simulation/commands.jsonl‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{ "cmds": ["python3"], "args": ["simulate.py", "AKAM", "/output"]}
2+
{ "cmds": ["python3"], "args": ["simulate.py", "AA", "/output"]}
3+
{ "cmds": ["python3"], "args": ["simulate.py", "MO", "/output"]}
4+
{ "cmds": ["python3"], "args": ["simulate.py", "AMZN", "/output"]}
5+
{ "cmds": ["python3"], "args": ["simulate.py", "AMGN", "/output"]}
6+
{ "cmds": ["python3"], "args": ["simulate.py", "AAPL", "/output"]}
7+
{ "cmds": ["python3"], "args": ["simulate.py", "T", "/output"]}
8+
{ "cmds": ["python3"], "args": ["simulate.py", "BA", "/output"]}
9+
{ "cmds": ["python3"], "args": ["simulate.py", "CAT", "/output"]}
10+
{ "cmds": ["python3"], "args": ["simulate.py", "CVX", "/output"]}
11+
{ "cmds": ["python3"], "args": ["simulate.py", "DIS", "/output"]}
12+
{ "cmds": ["python3"], "args": ["simulate.py", "KO", "/output"]}
13+
{ "cmds": ["python3"], "args": ["simulate.py", "DELL", "/output"]}
14+
{ "cmds": ["python3"], "args": ["simulate.py", "F", "/output"]}
15+
{ "cmds": ["python3"], "args": ["simulate.py", "INTC", "/output"]}
16+
{ "cmds": ["python3"], "args": ["simulate.py", "IBM", "/output"]}
17+
{ "cmds": ["python3"], "args": ["simulate.py", "MSFT", "/output"]}
18+
{ "cmds": ["python3"], "args": ["simulate.py", "NFLX", "/output"]}
19+
{ "cmds": ["python3"], "args": ["simulate.py", "NVDA", "/output"]}
20+
{ "cmds": ["python3"], "args": ["simulate.py", "ORCL", "/output"]}
21+
{ "cmds": ["python3"], "args": ["simulate.py", "QCOM", "/output"]}
22+
{ "cmds": ["python3"], "args": ["simulate.py", "X", "/output"]}
23+
{ "cmds": ["python3"], "args": ["simulate.py", "VZ", "/output"]}
24+
{ "cmds": ["python3"], "args": ["simulate.py", "V", "/output"]}

‎beta/serverless-fleets/tutorials/simulation/local‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
python3 -m venv venv
2+
source ./venv/bin/activate
3+
python3 -m pip install --upgrade pip
4+
pip3 install -r requirements.txt

0 commit comments

Comments
 (0)