fix: enforce CON-1751 runtime deadlines with focused regressions

Author: KAJdev

CHANGELOG.md

@@ -8,7 +8,7 @@
 
 ### Bug Fixes
 
-* Wait for task cleanup after interruption, preserve cancellation and task errors, recover in-flight pod creation, and retry transient task deletion failures without cancelling detached work.
+* Task calls complete bounded interruption cleanup and carry absolute runtime-enforced lifetime deadlines. Failed deletion retains the pod ID for recovery.
 * Network volume creation intersects catalog storage support with every sharing resource's hardware stock and datacenter constraints. Unsupported explicit pins fail before creation; existing volumes retain their authoritative IDs and datacenters.
 * Deployment artifacts apply project ignore rules and mandatory secret-file exclusions while preserving vendored dependency assets.
 * Synchronous calls wait across polling timeouts on Python 3.10 and propagate operation timeouts.

docs/cli/references/projects.md

@@ -14,46 +14,24 @@ Create a `.runpodignore` file in the root of your project to ignore files and fo
 
 ### Flash deployment artifacts
 
-`rp flash deploy` and `rp flash deploy --build-only` apply git-style patterns to
-source files, using `pathspec`. Precedence, from lowest to highest, is:
+`rp flash deploy` and `rp flash deploy --build-only` apply Git-style ignore
+patterns in this order, with later rules taking precedence:
 
-1. Default local-file exclusions: virtual environments, Python caches, test
-   directories and test modules, `node_modules`, `.DS_Store`, and `*.tar.gz`.
-2. `.gitignore` files within the project, with deeper files overriding parent
-   rules for their subtree.
+1. Default exclusions for local environments, caches, tests, and build archives.
+2. Project `.gitignore` files, with deeper files overriding parents in their subtree.
 3. The project-root `.runpodignore`.
 
-Ancestor and global Git ignore files are not read. Within each ignore file, the
-last matching rule wins. `/` anchors a pattern to that ignore file's directory;
-trailing `/` matches directories; `**`, comments, escaped characters and `!`
-negation follow Git ignore syntax. Excluded directories are not traversed, so
-re-include the parent before its files. For example, to deploy a fixture from
-the otherwise excluded `tests` directory:
-
-```gitignore
-!tests/
-tests/*
-!tests/fixture.json
-```
-
-Some safeguards cannot be overridden by negation: `.git`, `.runpod`, `.flash`,
-the root `env/` and `runpod_manifest.json` paths, and credential-like source
-paths. Credential safeguards include `.env` and its variants, `*.env` and its
-variants, `*.pem`, `*.key`, SSH private-key names, `.ssh`, `.aws`, `.azure`,
-`.kube`, `.docker/config.json`, `.netrc`, `.npmrc`, `.pypirc`, `.git-credentials`,
-`.boto`, `credentials`/`credentials.*`, `secrets`/`secrets.*`, and
-`service-account*.json`/`service_account*.json`. Use worker environment variables
-or a secret store for credentials instead of packaging them with source.
-
-These are filename safeguards, not secret scanning: credentials embedded in
-ordinary source or differently named files can still be uploaded. Review the
-build-only artifact before deployment.
-
-The generated manifest and vendored `env/` are added separately and cannot be
-replaced by source files. Source ignore rules, including the PEM/key safeguards,
-do not apply to vendored dependencies, so dependency CA bundles are preserved.
-The output artifact itself and the dependency build directory are never copied
-back into source. Source and dependency symlinks are omitted, including
-project-contained links and linked ignore files; directory links are not
-traversed. Hardlinked regular files are stored as independent regular members,
-so the archive requires no link extraction support.
+Ancestor and global Git ignores are not read. Negation (`!`) can re-include
+ordinary files, but excluded parent directories must also be re-included.
+
+Negation cannot include `.git`, `.runpod`, `.flash`, the root `env/` or
+`runpod_manifest.json`, or credential-like source paths such as `.env` variants,
+PEM/key files, private SSH keys, cloud credential directories, and credentials,
+secrets, or service-account files. These are filename safeguards, not secret
+scanning; review the build-only artifact and supply credentials through worker
+environment variables or a secret store.
+
+The manifest and vendored `env/` are added separately. Source ignore rules do
+not strip dependency CA bundles. Symlinks are omitted from source and
+dependencies; the output artifact and dependency build directory are never
+copied back into source.

examples/apps/README.md

@@ -25,11 +25,8 @@ including timeout, task failure, or cancellation. use `await job.cancel()` to
 abandon a spawned task explicitly. leaving the client without waiting or cancelling
 does not cancel intentional detached work.
 
-cancellation gives cleanup up to 30 seconds to recover an in-flight create response
-and delete the pod; synchronous calls allow up to 35 seconds for the coroutine to
-acknowledge cleanup after ctrl-c. repeated interrupts do not interrupt deletion.
-transient delete failures are retried up to three times. cleanup errors do not
-replace an existing task error or cancellation, and failed deletion retains
-`job.pod_id` for recovery. cleanup that exceeds the grace period continues only
-while the client's event loop remains alive. check the pod in the console after a
-cleanup warning; client process exit alone does not stop billing.
+cancellation waits for bounded cleanup and retries transient deletion failures.
+failed deletion retains `job.pod_id`; check the console after a cleanup warning.
+each task carries an absolute one-hour deadline that the runtime checks even during
+active work. polling and container restarts do not extend it. pod deletion still
+requires a working control-plane API and pod-scoped credentials.

runpod/apps/targets.py

@@ -806,7 +806,6 @@ class PodTarget(InvocationTarget):
     runner on the pod (see runpod.apps.tasks).
     """
 
-    # tasks default to a long window; the pod's terminateAfter is the backstop
     TASK_TIMEOUT_SECONDS = 3600.0
 
     def __init__(

runpod/apps/tasks.py

@@ -7,8 +7,8 @@
   3. POST the FunctionRequest to /execute (remote) or /submit (spawn)
   4. collect the response, terminate the pod
 
-`terminateAfter` is set at deploy time as a server-side safety net so a
-crashed client cannot leak a running pod indefinitely.
+the runtime watchdog requests pod deletion at the absolute task deadline,
+including when the client has exited or the function is still running.
 """
 
 import asyncio
@@ -17,7 +17,7 @@
 import secrets
 import sys
 import time
-from datetime import datetime, timedelta, timezone
+from datetime import timedelta
 from typing import Any, Dict, List, Optional
 
 import aiohttp
@@ -31,7 +31,6 @@
 
 TASK_PORT = 8080
 
-# requested server-side deadline for abandoned tasks
 DEFAULT_MAX_LIFETIME = timedelta(hours=1)
 
 READY_POLL_INTERVAL = 2.0
@@ -133,14 +132,14 @@ def _pod_input(spec: ResourceSpec, token: str, task_name: str) -> Dict[str, Any]
     start the runtime package via dockerArgs.
     """
     spec.validate()
-    terminate_after = (datetime.now(timezone.utc) + DEFAULT_MAX_LIFETIME).isoformat()
 
     from .secret import render_env
 
     env = {
         "RUNPOD_TASK_TOKEN": token,
         "RUNPOD_TASK_PORT": str(TASK_PORT),
         **render_env(spec.env),
+        "RUNPOD_TASK_DEADLINE": str(time.time() + DEFAULT_MAX_LIFETIME.total_seconds()),
     }
 
     from .images import image_for_spec, local_python_version
@@ -150,7 +149,6 @@ def _pod_input(spec: ResourceSpec, token: str, task_name: str) -> Dict[str, Any]
         "imageName": image_for_spec(spec, python_version=local_python_version()),
         "ports": f"{TASK_PORT}/http",
         "containerDiskInGb": spec.container_disk_gb or (10 if spec.is_cpu else 30),
-        "terminateAfter": terminate_after,
         "supportPublicIp": True,
     }
 

tests/test_apps/test_api_client.py

@@ -514,37 +514,14 @@ async def test_any_supported_tier_is_eligible(self):
                 {"id": "US-IL-1", "networkVolumeTypes": []},
                 {"id": "EU-RO-1", "networkVolumeTypes": ["STANDARD"]},
                 {"id": "US-KS-2", "networkVolumeTypes": ["PREMIUM"]},
-                {"id": "US-MO-2", "networkVolumeTypes": ["STANDARD", "PREMIUM"]},
             ]
         }
-        with patch(
-            "runpod.apps.api.run_rest_request_async",
-            AsyncMock(return_value=catalog),
-        ) as rest:
-            supported = await AppsApiClient(
-                api_key="test-key"
-            ).network_volume_datacenters()
-        assert supported == {"EU-RO-1", "US-KS-2", "US-MO-2"}
-        rest.assert_awaited_once_with(
-            "GET", "/v2/catalog/datacenters", api_key="test-key"
-        )
-
-    @pytest.mark.parametrize(
-        "catalog",
-        [
-            {},
-            {"dataCenters": None},
-            {"dataCenters": [{"id": "EU-RO-1"}]},
-            {"dataCenters": [{"id": "EU-RO-1", "networkVolumeTypes": "STANDARD"}]},
-        ],
-    )
-    async def test_missing_capability_is_not_assumed_available(self, catalog):
         with patch(
             "runpod.apps.api.run_rest_request_async",
             AsyncMock(return_value=catalog),
         ):
-            with pytest.raises(QueryError):
-                await AppsApiClient().network_volume_datacenters()
+            supported = await AppsApiClient().network_volume_datacenters()
+        assert supported == {"EU-RO-1", "US-KS-2"}
 
 
 class TestVolumesRegistrySecrets: