Skip to content

Commit 3864d37

Browse files
authored
fix: address Apps SDK alpha lifecycle, storage, and packaging reports (#611)
* fix: CON-1755 handle Python 3.10 future timeouts * test: cover CON-1755 sync bridge timeout propagation * fix: protect CON-1753 deployment artifacts from secret files * fix: complete CON-1751 task cleanup after cancellation * fix: use storage-capable placement for CON-1752 volumes * fix: enforce CON-1751 runtime deadlines with focused regressions * fix: reject global volumes on cpu compute * fix: enforce task timeouts remotely and refine artifact exclusions
1 parent abc4ee1 commit 3864d37

19 files changed

Lines changed: 910 additions & 197 deletions

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,14 @@
66

77
* Apps expose `NetworkVolume` and `GlobalVolume` resources with explicit path-keyed mounts, lazy create-if-missing provisioning, and execution-bound filesystem access. Global volumes use GraphQL for lookup and creation.
88

9+
### Bug Fixes
10+
11+
* Task calls support indefinite execution with bounded interruption cleanup. Failed deletion retains the pod ID for recovery.
12+
* Network volume creation intersects catalog storage support with every sharing resource's hardware stock and datacenter constraints. Unsupported explicit pins fail before creation; existing volumes retain their authoritative IDs and datacenters.
13+
* Global volume attachments require GPU compute for tasks and endpoints.
14+
* Deployment artifacts apply project ignore rules and mandatory secret-file exclusions while preserving vendored dependency assets.
15+
* Synchronous calls wait across polling timeouts on Python 3.10 and propagate operation timeouts.
16+
917
## [1.12.0](https://github.com/runpod/runpod-python/compare/v1.11.0...v1.12.0) (2026-08-10)
1018

1119

‎docs/cli/references/projects.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,27 @@ You may need to update the default configuration within `runpod.toml` to match y
1111
## Ignore Files and Folders
1212

1313
Create a `.runpodignore` file in the root of your project to ignore files and folders from being uploaded to the Runpod platform, the same file will also be used to ignore files that should not trigger an API server reload.
14+
15+
### Flash deployment artifacts
16+
17+
`rp flash deploy` and `rp flash deploy --build-only` apply Git-style ignore
18+
patterns in this order, with later rules taking precedence:
19+
20+
1. Default exclusions for local environments, caches, tests, and build archives.
21+
2. Project `.gitignore` files, with deeper files overriding parents in their subtree.
22+
3. The project-root `.runpodignore`.
23+
24+
Ancestor and global Git ignores are not read. Negation (`!`) can re-include
25+
ordinary files, but excluded parent directories must also be re-included.
26+
27+
Negation cannot include `.git`, `.runpod`, `.flash`, the root `env/` or
28+
`runpod_manifest.json`, or credential-like source paths such as `.env` variants,
29+
PEM/key files, private SSH keys, cloud credential directories, and credentials,
30+
secrets, or service-account files. These are filename safeguards, not secret
31+
scanning; review the build-only artifact and supply credentials through worker
32+
environment variables or a secret store.
33+
34+
The manifest and vendored `env/` are added separately. Source ignore rules do
35+
not strip dependency CA bundles. Symlinks are omitted from source and
36+
dependencies; the output artifact and dependency build directory are never
37+
copied back into source.

‎examples/apps/README.md‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,40 @@ workers pick up the new code automatically. the exhaustive
2020
feature-by-feature suite lives in [`tests/e2e/examples`](../../tests/e2e/examples).
2121

2222
`await task.spawn.aio(...)` returns a task job that owns its pod.
23-
`await job.wait(timeout=...)` terminates that pod when waiting finishes, including
24-
timeout, task failure, or cancellation. use `await job.cancel()` to abandon a
25-
spawned task explicitly. a failed termination retains `job.pod_id` for cleanup.
23+
`await job.wait(timeout=...)` attempts to terminate that pod when waiting finishes,
24+
including timeout, task failure, or cancellation. use `await job.cancel()` to
25+
abandon a spawned task explicitly. leaving the client without waiting or cancelling
26+
does not cancel intentional detached work.
27+
28+
cancellation waits for bounded cleanup and retries transient deletion failures.
29+
failed deletion retains `job.pod_id`; check the console after a cleanup warning.
30+
active tasks can run indefinitely, including detached work after client exit.
31+
`remote()` and `job.wait()` impose no execution timeout by default. pod deletion
32+
requires a working control-plane API and pod-scoped credentials.
33+
34+
## Storage
35+
36+
`NetworkVolume(name_or_id, size=50, datacenter=None, create=True)` references
37+
datacenter-local storage. New volumes use one catalog capability snapshot per
38+
provisioning run to select a datacenter with network storage support and hardware
39+
stock for every resource sharing the volume, respecting their datacenter pins.
40+
An unsupported explicit volume pin fails before creation instead of relocating;
41+
catalog lookup failures also stop creation. Existing volumes retain their IDs and
42+
datacenters regardless of new-volume eligibility. Their consumers must still be
43+
schedulable in that datacenter. `GlobalVolume(name_or_id, create=True)`
44+
references global storage without a datacenter constraint. Both inherit from
45+
the abstract `Volume` base, resolve by name or ID, and create missing storage
46+
when provisioning remote compute. Set `create=False` to require existing storage.
47+
Global-volume lookup and creation use GraphQL; network volumes use REST.
48+
49+
Declare attachments with `mounts={"/path": volume}`. Tasks support one network
50+
and one global volume at distinct, non-overlapping paths. Queue and API resources
51+
support one volume at `/runpod-volume`. Global volumes require GPU compute for
52+
both tasks and endpoints.
53+
54+
Inside worker code, `volume.path` returns the configured mount path. The runtime
55+
binds declared references and resolved IDs before importing user code. Access
56+
raises if the volume is unmounted or has multiple bindings. Evaluate `.path`
57+
inside remote functions, not during local module discovery. Calling `.local()`
58+
does not mount remote storage on the client machine.
59+

‎requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ colorama >= 0.4.6, < 0.4.7
99
cryptography >= 50.0.1
1010
fastapi[all] >= 0.141.1
1111
paramiko >= 5.0.0
12+
pathspec >= 0.12.1
1213
prettytable >= 3.18.0
1314
psutil >= 7.2.2
1415
py-cpuinfo >= 9.0.0

‎runpod/apps/api.py‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
absent from rest. management verbs for the wider sdk stay in runpod.api.ctl_commands.
66
"""
77

8-
from typing import Any, Dict, List, Optional
8+
from typing import Any, Dict, List, Optional, Set
99

1010
import aiohttp
1111

@@ -387,6 +387,31 @@ async def cpu_stock_status(
387387
)
388388
return _stock_in_datacenter(data, data_center_id)
389389

390+
async def network_volume_datacenters(self) -> Set[str]:
391+
"""datacenters supporting any network volume tier chosen by the backend."""
392+
data = await run_rest_request_async(
393+
"GET", "/v2/catalog/datacenters", api_key=self._api_key
394+
)
395+
datacenters = data.get("dataCenters") if isinstance(data, dict) else None
396+
if not isinstance(datacenters, list):
397+
raise QueryError("datacenter catalog is missing dataCenters")
398+
supported = set()
399+
for dc in datacenters:
400+
if (
401+
not isinstance(dc, dict)
402+
or not isinstance(dc.get("id"), str)
403+
or not dc["id"]
404+
or not isinstance(dc.get("networkVolumeTypes"), list)
405+
or any(
406+
not isinstance(tier, str) or not tier
407+
for tier in dc["networkVolumeTypes"]
408+
)
409+
):
410+
raise QueryError("datacenter catalog has invalid networkVolumeTypes")
411+
if dc["networkVolumeTypes"]:
412+
supported.add(dc["id"])
413+
return supported
414+
390415
async def list_global_volumes(self) -> List[Dict[str, Any]]:
391416
data = await self._execute(app_queries.QUERY_GLOBAL_VOLUMES, retry=True)
392417
return data["myself"]["globalStoreBuckets"]

‎runpod/apps/context.py‎

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,18 @@
11
"""execution context detection and the sync/async bridge."""
22

33
import asyncio
4+
import concurrent.futures
5+
import logging
46
import os
57
import threading
8+
import time
69
from enum import Enum
710
from typing import Any, Coroutine
811

12+
log = logging.getLogger(__name__)
13+
CLEANUP_TIMEOUT = 30.0
14+
BRIDGE_CLEANUP_TIMEOUT = CLEANUP_TIMEOUT + 5.0
15+
916

1017
class Context(Enum):
1118
"""where the current process is running."""
@@ -68,16 +75,45 @@ def _ensure_loop(cls) -> asyncio.AbstractEventLoop:
6875
@classmethod
6976
def run(cls, coro: Coroutine[Any, Any, Any]) -> Any:
7077
loop = cls._ensure_loop()
71-
future = asyncio.run_coroutine_threadsafe(coro, loop)
78+
completed = threading.Event()
79+
interrupted = threading.Event()
80+
running = []
81+
82+
async def run():
83+
running.append(asyncio.current_task())
84+
try:
85+
if interrupted.is_set():
86+
coro.close()
87+
raise asyncio.CancelledError()
88+
return await coro
89+
finally:
90+
completed.set()
91+
92+
def cancel():
93+
interrupted.set()
94+
if running:
95+
running[0].cancel()
96+
97+
future = asyncio.run_coroutine_threadsafe(run(), loop)
7298
try:
7399
while True:
74100
try:
75101
return future.result(timeout=0.2)
76-
except TimeoutError:
102+
except concurrent.futures.TimeoutError:
77103
if future.done():
78104
raise
79105
except BaseException:
80-
future.cancel()
106+
loop.call_soon_threadsafe(cancel)
107+
deadline = time.monotonic() + BRIDGE_CLEANUP_TIMEOUT
108+
while not completed.is_set():
109+
try:
110+
remaining = deadline - time.monotonic()
111+
if remaining <= 0:
112+
log.warning("interrupted operation cleanup is still pending")
113+
break
114+
completed.wait(min(remaining, 0.2))
115+
except BaseException:
116+
continue
81117
raise
82118

83119

‎runpod/apps/datacenter.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
"""datacenter selection for app resources.
22
3-
only datacenters with storage support and S3 API support are listed"""
3+
network volume creation support is discovered from the datacenter catalog."""
44

55
from enum import Enum
66
from typing import List
@@ -41,8 +41,8 @@ def all(cls) -> List["DataCenter"]:
4141
return list(cls)
4242

4343

44-
# datacenters with high cpu serverless stock, restricted to the
45-
# storage+S3 set above. cpu5c/cpu5g are only stocked in EU-RO-1.
44+
# datacenters with high cpu serverless stock. storage support is checked
45+
# separately; cpu5c/cpu5g are only stocked in EU-RO-1.
4646
CPU3_DATACENTERS: List[DataCenter] = [
4747
DataCenter.EU_CZ_1,
4848
DataCenter.EU_RO_1,

0 commit comments

Comments
 (0)