Add nemo-skills-core subpackage for lightweight installs

[gwarmstrong]

Adds a lightweight nemo-skills-core subpackage (core/ subdirectory)
with only inference, evaluation, and tool calling deps. Default
pip install nemo-skills is unchanged (installs everything).

Changes

• core/pyproject.toml + core/requirements.txt: New subpackage installable via pip install ./core or git URL with #subdirectory=core. Single source of truth for core deps, referenced by both core and root pyproject.toml.
• nemo_skills/pipeline/__init__.py: Import guard using importlib.metadata -- importing pipeline modules with only core installed raises a clear ImportError instead of a cryptic ModuleNotFoundError.
• nemo_skills/_cli_stub.py: Stub ns CLI entry point for core-only installs that prints a helpful message.
• nemo_skills/evaluation/evaluator/__init__.py: Lazy evaluator registry using string paths instead of eager imports, so core-only installs don't fail on benchmark-specific deps (faiss, func_timeout, etc.).
• nemo_skills/dataset/utils.py + nemo_skills/pipeline/dataset.py: Moved cluster-dependent dataset logic into pipeline module to keep core free of nemo_run imports.
• requirements/pipeline.txt: New requirements file for pipeline-only deps (nemo_run, typer, etc.).
• .github/workflows/tests.yml: Install uv in CI for use with testing installation.
• Docs: Added installation guide, updated CONTRIBUTING.md with dependency placement guidance.

Summary by CodeRabbit

• New Features

  • Lightweight core package with a minimal CLI message and local-first dataset loading (cluster workflows remain available via the pipeline).
  • Evaluators now load lazily on demand to reduce startup cost.

• Documentation

  • Expanded installation & dependency guide explaining full vs core installs, extras, and the Core/Pipeline boundary.

• Chores

  • Packaging split into core/pipeline/dev dependency groups; added pipeline dependencies and core requirements.
  • CI workflow adds a UV setup step.

[greptile-apps]

_{13 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Setting _EVALUATOR_MAP_PATHS[eval_type] = None creates a fragile state. If _resolved_evaluator_map is ever cleared or doesn't contain the eval_type, _get_evaluator_fn will call _resolve(None) and crash.

Suggested change

	_EVALUATOR_MAP_PATHS[eval_type] = None
	_resolved_evaluator_map[eval_type] = eval_fn
	# Store function directly, bypassing the lazy resolution path
	_resolved_evaluator_map[eval_type] = eval_fn

[gwarmstrong]

Good catch, switched to a "<dynamically-registered>" sentinel to be safe.

[gwarmstrong]

Actually, reverting this back to None. The _resolved_evaluator_map cache is internal and never cleared, so this scenario cannot happen in practice. Per our project guidelines: "Don't add error handling, fallbacks, or validation for scenarios that can't happen." If the cache were somehow corrupted, crashing is the correct signal.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a lightweight core package and requirements, documents installation and the Core/Pipeline dependency boundary, reorganizes optional extras in packaging, adds a CI step for uv, implements lazy evaluator resolution, refactors dataset loading to prefer local modules and delegates cluster handling to a new pipeline dataset module, and adds a runtime guard for pipeline imports.

Changes

Cohort / File(s)	Summary
CI Workflow \.github/workflows/tests.yml	Adds an "Install uv" step (astral-sh/setup-uv@v4) before dependency installation.
Docs & Contribution Guidance CONTRIBUTING.md, docs/basics/installation.md, mkdocs.yml	Adds installation docs and a new "Respect the Core / Pipeline dependency boundary" section; registers new docs nav entry.
Core package metadata & deps core/pyproject.toml, core/requirements.txt	Adds pyproject for nemo-skills-core and a core requirements file listing core runtime dependencies.
Root packaging & extras pyproject.toml, requirements/pipeline.txt, requirements/main.txt	Splits optional-dependencies into core, pipeline, dev; adds pipeline deps (nemo_run, wandb, typer, constrained click) and adds rich to main requirements.
CLI stub nemo_skills/_cli_stub.py	Adds minimal CLI stub main() that prints a lightweight-mode message and exits.
Dataset loading (core) nemo_skills/dataset/utils.py	Reworks get_dataset_module to prefer local imports, remove prior cluster helpers, add extra_datasets handling, and emit DeprecationWarning + delegate to pipeline loader when cluster_config is provided.
Dataset loading (pipeline) nemo_skills/pipeline/dataset.py	Adds pipeline-aware dataset loader supporting local, local-unmounted, and remote cluster downloads; exposes get_dataset_module for cluster-enabled resolution.
Evaluator lazy loading nemo_skills/evaluation/evaluator/__init__.py	Replaces eager evaluator imports with path-based maps and runtime import resolution (lazy loading) for evaluator functions/classes; adds resolver utilities and caches.
Pipeline import guard nemo_skills/pipeline/__init__.py	Adds runtime guard using package metadata to raise ImportError when pipeline is imported but only core is installed.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Core as nemo_skills.dataset.utils
    participant Pipeline as nemo_skills.pipeline.dataset
    participant Cluster

    rect rgba(100, 150, 200, 0.5)
    Note over User,Core: Local-only flow (default)
    User->>Core: get_dataset_module(dataset, data_dir=None)
    Core->>Core: import from nemo_skills.dataset or local path
    Core-->>User: return dataset module
    end

    rect rgba(200, 100, 150, 0.5)
    Note over User,Cluster: Cluster flow (deprecated in Core)
    User->>Core: get_dataset_module(dataset, cluster_config=...)
    Core->>Core: emit DeprecationWarning
    Core->>Pipeline: delegate get_dataset_module(...)
    Pipeline->>Cluster: fetch / download cluster module (remote)
    Cluster-->>Pipeline: module content / init.py
    Pipeline->>Core: imported module
    Core-->>User: return dataset module
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• Evaluation on OJBench #848 — touches the evaluator registry; directly related to the lazy/path-based evaluator changes.
• Fix run.Script refactor #1133 — large pipeline scripting/orchestration refactor; related to pipeline dataset and orchestration changes.
• Revert "Use run.Script for generate pipeline (#1052)" #1125 — pipeline orchestration and script/command abstraction changes; related to pipeline-layer refactors in this PR.

Suggested reviewers

• Kipok
• activatedgeek

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 64.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title clearly and concisely summarizes the main change: adding a lightweight nemo-skills-core subpackage for installations that don't require the full pipeline dependencies.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into main

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch georgea/refactor-separable-pipeline

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 5

🤖 Fix all issues with AI agents

In `@nemo_skills/evaluation/evaluator/__init__.py`:
- Around line 113-117: The error message incorrectly labels
`_EVALUATOR_MAP_PATHS.keys()` as "All supported types" when it only contains
function-based evaluators; update the ValueError text in the raise block (the
code that references eval_type) to clearly distinguish class-based vs
function-based types by either listing both maps together (combine
`_EVALUATOR_CLASS_MAP_PATHS.keys()` and `_EVALUATOR_MAP_PATHS.keys()`) or
renaming the second label to "Function-based evaluator types" so users see
accurate descriptions of `_EVALUATOR_CLASS_MAP_PATHS` and
`_EVALUATOR_MAP_PATHS`.
- Around line 106-107: register_evaluator currently stores None into
_EVALUATOR_MAP_PATHS[eval_type], which will cause AttributeError when code later
iterates or calls _resolve expecting a path string; change register_evaluator so
it stores a sentinel string (e.g. "<dynamic>") into
_EVALUATOR_MAP_PATHS[eval_type] instead of None, and ensure any
resolution/display logic in _resolve/_get_evaluator_fn treats that sentinel as a
dynamic entry (or filters it out) so rsplit is only called on real path strings;
update references to _EVALUATOR_MAP_PATHS, register_evaluator,
_resolved_evaluator_map, _get_evaluator_fn, and _resolve accordingly.
- Line 137: Remove the leftover debug print statement print(f"evaluator:
{evaluator}") from the module (it should not be in production code); either
delete that line or replace it with an appropriate logger.debug call using the
module logger (e.g., logger.debug("evaluator: %s", evaluator)) so diagnostics
use the configured logging system and not stdout—locate the print by searching
for the exact string and update in the __init__ module where the evaluator
variable is in scope.
- Around line 93-94: Remove the debug print statement print(f"evaluator:
{evaluator}") from the module so it no longer emits debug output; locate the
temporary print in the evaluator initialization block near where EVALUATOR_MAP
and EVALUATOR_CLASS_MAP are set and delete that single line, leaving the maps
and the helper functions (_get_evaluator_fn, _get_evaluator_cls, evaluate,
get_evaluator_class) intact so iteration via EVALUATOR_MAP/EVALUATOR_CLASS_MAP
still works per the documented design.

In `@nemo_skills/pipeline/dataset.py`:
- Around line 60-62: The check uses cluster_config.get("executor") which masks a
missing-key error; change it to access the key directly
(cluster_config["executor"]) so missing executor raises immediately, and keep
the logic that if cluster_config is None or cluster_config["executor"] in (None,
"none") then return _get_local_dataset_module(dataset, data_dir); update any
related code paths that assume executor exists (e.g., the code around
get_unmounted_path in nemo_skills/pipeline/utils/mounts.py) to rely on the same
direct-access semantics to fail fast on misconfiguration.

🧹 Nitpick comments (6)

CONTRIBUTING.md (1)

56-59: Fenced code block missing language specifier.

Minor nit from markdownlint — adding a language (e.g., text) would silence MD040.

Proposed fix

-```
+```text
 Pipeline can import from Core.
 Core CANNOT import from Pipeline.
-```
+```

core/requirements.txt (1)

17-27: Section label "math evaluation" is misleading — several packages below it aren't math-specific.

mcp, numpy, openai, requests, rich, tqdm, and transformers are general-purpose dependencies, not math evaluation specific. Consider either reorganizing sections or using a broader label like # --- general / shared ---.

nemo_skills/pipeline/dataset.py (3)

39-51: Imported module outlives its backing file.

import_from_path is called inside a TemporaryDirectory context manager. Once the with block exits, the downloaded init.py is deleted, but the module object (and its __file__ attribute) still references the now-removed path. This works at runtime because CPython caches the compiled bytecode in memory, but it can cause confusing errors if any downstream code inspects module.__file__ or attempts a reload.

Consider moving the temp directory lifecycle to the caller or keeping it alive longer if module introspection is needed.

---

44-50: Chain the re-raised exception for clearer tracebacks.

Per the static analysis hint (B904), raise ... from err preserves the original traceback context.

Proposed fix

         try:
             cluster_download_file(cluster_config, cluster_dataset_path, tmp_path)
-        except FileNotFoundError:
-            raise RuntimeError(
+        except FileNotFoundError as err:
+            raise RuntimeError(
                 f"Init file {mounted_path} not found on the cluster. "
                 f"Please check the dataset name you're using. Did you forget to run prepare data commands?"
-            )
+            ) from err

---

109-113: Chain the re-raised RuntimeError for clearer tracebacks.

Same B904 pattern — add from err to preserve the original ModuleNotFoundError context.

Proposed fix

-        except ModuleNotFoundError:
-            raise RuntimeError(
+        except ModuleNotFoundError as err:
+            raise RuntimeError(
                 f"Dataset {dataset} not found in any of the searched locations: "
                 f"{data_dir if data_dir else 'nemo_skills.dataset'}, {extra_datasets}"
-            )
+            ) from err

nemo_skills/dataset/utils.py (1)

116-135: Chain re-raised exceptions for clearer tracebacks.

Same pattern as flagged in pipeline/dataset.py — the raise RuntimeError(...) statements at Lines 120 and 126 inside except clauses should use from to preserve the original exception context.

Proposed fix

     except ModuleNotFoundError:
         dataset = dataset.replace(".", "/")
         extra_datasets = extra_datasets or os.environ.get("NEMO_SKILLS_EXTRA_DATASETS")
         if extra_datasets is None:
-            raise RuntimeError(f"Dataset {dataset} not found in {data_dir if data_dir else 'nemo_skills.dataset'}")
+            raise RuntimeError(
+                f"Dataset {dataset} not found in {data_dir if data_dir else 'nemo_skills.dataset'}"
+            ) from None
         if extra_datasets_type == ExtraDatasetType.local or extra_datasets_type is None:
             with add_to_path(extra_datasets):
                 try:
                     dataset_module = importlib.import_module(dataset)
-                except ModuleNotFoundError:
-                    raise RuntimeError(
+                except ModuleNotFoundError as err:
+                    raise RuntimeError(
                         f"Dataset {dataset} not found in any of the searched locations: "
                         f"{data_dir if data_dir else 'nemo_skills.dataset'}, {extra_datasets}"
-                    )
+                    ) from err

[greptile-apps]

_{13 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Setting _EVALUATOR_MAP_PATHS[eval_type] = None is fragile. If _resolved_evaluator_map gets cleared or doesn't contain eval_type, _get_evaluator_fn will call _resolve(None) and crash with ValueError: not enough values to unpack.

The current implementation works only because the function is immediately added to _resolved_evaluator_map, but this implicit dependency is error-prone. Consider either:

1. Not setting _EVALUATOR_MAP_PATHS[eval_type] at all (just use _resolved_evaluator_map)
2. Setting it to a sentinel string that provides a better error message if accidentally resolved

[gwarmstrong]

Good catch, switched to a "<dynamically-registered>" sentinel to be safe.

[gwarmstrong]

Actually, reverting this back to None. The _resolved_evaluator_map cache is internal and never cleared, so this scenario cannot happen in practice. Per our project guidelines: "Don't add error handling, fallbacks, or validation for scenarios that can't happen." If the cache were somehow corrupted, crashing is the correct signal.

[greptile-apps]

Additional Comments (1)

requirements/main.txt
rich is in core/requirements.txt but missing from requirements/main.txt, violating the rule that "all core and pipeline deps must also appear in requirements/main.txt"

rich

[greptile-apps]

_{14 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@nemo_skills/pipeline/dataset.py`:
- Around line 54-62: The function _get_default_dataset_module currently drops
extra_datasets and extra_datasets_type when cluster_config is None by delegating
to _get_local_dataset_module(dataset, data_dir); update
_get_default_dataset_module to forward extra_datasets and extra_datasets_type
into the local call (e.g., call _get_local_dataset_module(dataset, data_dir,
extra_datasets=..., extra_datasets_type=...)) so get_dataset_module's outer
ModuleNotFoundError path remains reachable and callers' extra_datasets are
honored; ensure the function signature for _get_default_dataset_module accepts
the extra_* params and that _get_local_dataset_module is invoked with those
parameters.

🧹 Nitpick comments (3)

nemo_skills/pipeline/dataset.py (2)

39-51: Chain exception context with from when re-raising.

Static analysis (B904) correctly flags that re-raising inside except without from loses the original traceback context. This applies here and at lines 109-113.

Proposed fix

-        except FileNotFoundError:
-            raise RuntimeError(
+        except FileNotFoundError as exc:
+            raise RuntimeError(
                 f"Init file {mounted_path} not found on the cluster. "
                 f"Please check the dataset name you're using. Did you forget to run prepare data commands?"
-            )
+            ) from exc

---

91-113: Chain the inner RuntimeError re-raise with from.

Same B904 issue as above — preserve context for debugging.

Proposed fix

-        except ModuleNotFoundError:
-            raise RuntimeError(
+        except ModuleNotFoundError as exc:
+            raise RuntimeError(
                 f"Dataset {dataset} not found in any of the searched locations: "
                 f"{data_dir if data_dir else 'nemo_skills.dataset'}, {extra_datasets}"
-            )
+            ) from exc

nemo_skills/evaluation/evaluator/__init__.py (1)

93-94: Semantic change in EVALUATOR_MAP / EVALUATOR_CLASS_MAP values.

These aliases now expose dotted-path strings instead of resolved callables/classes. Any downstream code (external plugins, scripts) that iterates .values() expecting callables will break silently. The comment on lines 90-92 documents the intent, and the repo itself only uses these for key enumeration, so this is safe internally. Just worth noting for external consumers if this is a public API.

[gwarmstrong]

Added rich to requirements/main.txt, thanks.

[Kipok]

are you sure this covers all benchmarks? Generally, we should move to keeping this reqs really simple and move most benchmark-specific requirements to install at runtime, but for now probably we might need some more packages here? E.g. datasets is almost certainly needed and then other benchmark specific things, like sacrebleu, etc.

[gwarmstrong]

I revisited the separation. This should contain all the reqs not needed for cluster orchestration now.

[Kipok]

this is actually a core dependency, it's being used in summarize-results, which is required for core functionality. Currently summarize-results is kind of in a weird half-pipeline state, but we should fix it to cleanly separate it into pipeline and non-pipeline components via #779 (comment)

[gwarmstrong]

fixed

[Kipok]

this part I don't fully understand - I think benchmark-specific packages should go to core for now as otherwise the code will fail when those benchmarks are used e.g. in evaluator. Eventually we should migrate to jit install, but it's not done yet, so I'd put those into core

[gwarmstrong]

Fair. My original scope was pretty PythonTool specific, but I think we can come up with something that makes a little more sense in terms of aligning the core code with core dependencies.

[gwarmstrong]

yeah it's now in core and there is a clearer description of what's in pipeline vs core

[Kipok]

can we not link multiple requirements listed in pyproject.toml? We duplicate?

[gwarmstrong]

we should be able to do that. It should be implemented that way now--I updated this at one point so it links against the file rather than duplicating. Will fix.

[gwarmstrong]

done

[Kipok]

I actually think that if we have a case like this, it means we need to redesign something. Ideally separation should be clean, and we shouldn't need to duplicate functionality. E.g. the dataset module part is a bit messy and there is probably a way to do it better, such that there is a pipeline level that only manages pulling from cluster and then there is a local level that always assumes things are present locally and is being called inside pipeline directly

[gwarmstrong]

Makes sense, updated the docs here to reflect that and made the implementation more consistent with the guidance here/there.

[Kipok]

can we keep this inside requirements/core.txt? Would be simpler for people to only look in a single folder for all reqs

[gwarmstrong]

It has to be in the same directory as the pyproject.toml for the install of nemo-skills-core to work. But I can put a symlink in the requirements/ directory so it is at least clear that this file exists if someone goes to look in requirements.txt for it?

[Kipok]

comment should be updated or we should move core/requirement.txt into core.txt in this folder

[Kipok]

can we dynamically find everything inside nemo_skills except pipeline subfolder?

[gwarmstrong]

done

[Kipok]

can we maybe keep the part in here brief, just summarize the logic in a few sentences / bullet points. And then the full description we move to another .md file? I think the full description is quite helpful, but it's a bit too detailed for the guidelines section, which I hope we can keep relatively short

[gwarmstrong]

sure. made it brief and moved the bulk of the guidelines to core/README.md, which is referenced here now

[Kipok]

do we use it anywhere?

[gwarmstrong]

good catch, it was from an old testing strategy, removed

[Kipok]

is this true? I don't think we create fresh envs in tests?

[gwarmstrong]

yeah this is outdated from old tests, good cattch

[Kipok]

I'd maybe move this part somewhere else (e.g. only keep in contributing.md or better in a new .md where extra details from contributing can go as well). It's helpful, but probably a bit too dense for the "basics" part of the docs. More oriented towards people who'd need to modify our code

[gwarmstrong]

okay, put it in the core/README.md