WIP: Port the canonical R vignette curriculum to Python

README.md

@@ -7,15 +7,21 @@
 [![Docs](https://img.shields.io/badge/docs-ovvo--financial.github.io-blue)](https://ovvo-financial.github.io/NNS-python/)
 [![License](https://img.shields.io/badge/license-GPL--3.0--only-blue)](https://github.com/OVVO-Financial/NNS-python/blob/main/LICENSE)
 
-`ovvo-nns` brings Nonlinear Nonparametric Statistics to Python as the `nns` import package. It is a parity-focused port of the R `NNS` 13.0+ package, designed for real-world data that violate symmetry, linearity, or distributional assumptions.
-
-NNS is built around partial moments, the lower and upper components of variance, and uses them across nonlinear dependence, correlation, causation, regression, classification, forecasting, stochastic dominance, stochastic superiority, Monte Carlo simulation, and numerical differentiation workflows.
-
-
-> NNS was created by Fred Viole as the companion R package to Viole, F. and Nawrocki, D. (2013), *Nonlinear Nonparametric Statistics: Using Partial Moments*. **Book (2nd Edition):** https://ovvo-financial.github.io/NNS/book/
->   
->  **Implementation:** For a direct quantitative finance implementation of NNS, see [OVVO Labs](https://www.ovvolabs.com)
-
+`ovvo-nns` brings Nonlinear Nonparametric Statistics to Python as the `nns`
+import package. It is a parity-focused port of the R `NNS` 13.0+ package,
+designed for real-world data that violate symmetry, linearity, or
+distributional assumptions.
+
+The R package is the reference implementation and the source of truth for the
+statistical behavior, terminology, and canonical example curriculum. Python is
+native and does not call R at runtime.
+
+> NNS was created by Fred Viole as the companion R package to Viole, F. and
+> Nawrocki, D. (2013), *Nonlinear Nonparametric Statistics: Using Partial
+> Moments*. **Book (2nd Edition):** https://ovvo-financial.github.io/NNS/book/
+>
+> **Implementation:** For a direct quantitative finance implementation of NNS,
+> see [OVVO Labs](https://www.ovvolabs.com).
 
 ## Package at a glance
 
@@ -25,24 +31,18 @@ NNS is built around partial moments, the lower and upper components of variance,
 | Import package | `nns` |
 | Current version | `1.4.0` |
 | Python | `>=3.11` |
-| Required runtime dependencies | NumPy, SciPy |
+| Required runtime dependencies | NumPy, SciPy, Matplotlib |
 | R required at runtime | No |
 | Native acceleration | Private, optional `nns._nnscore` kernels where available |
 | Public API status | Stable, parity-focused |
 | License | GPL-3.0-only |
 
-The public package is Python-native and does not call R at runtime. Some core kernels can use the private `_nnscore` extension when it is present, while public functions keep Python implementations and explicit fallback behavior.
-
 ## Install
 
 ```bash
 pip install ovvo-nns
 ```
 
-This includes the matplotlib plotting API (`nns.plotting`); matplotlib is a
-regular dependency and is imported lazily, so `import nns` stays light. See
-the [plot parity policy](https://ovvo-financial.github.io/NNS-python/plot_parity_policy/).
-
 Use the package as `nns`:
 
 ```python
@@ -51,66 +51,24 @@ import nns
 print(nns.__version__)
 ```
 
-Source builds use `scikit-build-core` and `nanobind` for the optional native extension. Published wheels should be preferred when available.
+Published wheels are preferred. Source builds use `scikit-build-core` and
+`nanobind` for the optional native extension.
 
 ## Quick start
 
 ```python
 import numpy as np
 from nns import lpm, nns_dep, nns_reg, upm
 
-x = np.array([-2.0, -1.0, 0.5, 3.0], dtype=np.float64)
-
-lower = lpm(degree=2, target=0.0, x=x)
-upper = upm(degree=2, target=0.0, x=x)
-
-print("lower partial moment:", lower)
-print("upper partial moment:", upper)
-```
-
-Measure nonlinear dependence:
-
-```python
-import numpy as np
-from nns import nns_cor, nns_dep
-
-grid = np.linspace(-2.0, 2.0, 80, dtype=np.float64)
-y = grid**2
-
-print("NNS dependence:", nns_dep(grid, y))
-print("NNS correlation:", nns_cor(grid, y))
-```
-
-Fit a nonlinear regression and estimate new points:
-
-```python
-import numpy as np
-from nns import nns_reg
-
-x = np.linspace(-3.0, 3.0, 80, dtype=np.float64)
-y = np.sin(x) + 0.2 * x
-points = np.array([-1.5, 0.0, 1.5], dtype=np.float64)
+x = np.array([-2.0, -1.0, 0.5, 3.0])
+print("LPM2:", lpm(2, 0.0, x))
+print("UPM2:", upm(2, 0.0, x))
 
-fit = nns_reg(x, y, point_est=points, confidence_interval=None)
+grid = np.linspace(-2.0, 2.0, 80)
+print("nonlinear dependence:", nns_dep(grid, grid**2))
 
-print("R2:", fit["R2"])
-print(np.column_stack((points, fit["Point.est"])))
-```
-
-Forecast a univariate series:
-
-```python
-import numpy as np
-from nns import nns_arma, nns_seas
-
-t = np.arange(1, 60, dtype=np.float64)
-series = 10.0 + np.sin(t / 3.0) + 0.05 * t
-
-seasonality = nns_seas(series, modulo=[3, 4, 6], mod_only=True)
-forecast = nns_arma(series, h=3, seasonal_factor=4, method="lin")
-
-print("best seasonal period:", seasonality["best.period"])
-print("forecast:", forecast)
+fit = nns_reg(grid, np.sin(grid), point_est=np.array([-1.0, 0.0, 1.0]))
+print("point estimates:", fit["Point.est"])
 ```
 
 ## Main API areas
@@ -127,67 +85,69 @@ print("forecast:", forecast)
 | Stochastic dominance | `fsd`, `ssd`, `tsd`, `nns_sd_cluster`, `sd_efficient_set` |
 | Stochastic superiority and simulation | `nns_ss`, `nns_mc`, `nns_meboot` |
 | Differentiation | `nns_diff`, `dy_dx`, `dy_d` |
-| Categorical helpers | `encode_factor_codes`, `factor_2_dummy`, `factor_2_dummy_fr`, `prepare_factor_predictors` |
-
-See [API status](https://ovvo-financial.github.io/NNS-python/api_status/) for implemented, partial, guarded, and known-gap paths.
-
-## Design boundaries
 
-NNS Python prioritizes stable public behavior from installed R NNS 13.0+, not private helper parity. The package returns NumPy arrays and plain dictionaries rather than R `data.table` objects, uses explicit Python errors for several unsafe R coercions, and generally ignores plotting side effects.
+See [API status](https://ovvo-financial.github.io/NNS-python/api_status/) for
+implemented, partial, guarded, and known-gap paths.
 
-Important boundaries:
+## Canonical examples
 
-- R is used only for parity tests and local cache regeneration, not normal runtime use.
-- Stochastic exact stream parity is not expected because Python paths use NumPy random generation.
-- Factor and class ordering should be passed explicitly when ordering matters.
-- Direct raw-factor `nns_m_reg(..., factor_2_dummy=True)` is intentionally guarded. Use `prepare_factor_predictors(...)` before `nns_m_reg(...)`.
-- Compute functions still return values, not figures; passing `plot=True` (where R has it) additionally renders a Matplotlib figure as a side effect via the `nns.plotting` layer, which is color/element-faithful to R but not pixel-diffed. The plot functions can also be called directly on a computed result.
+The R package's nine numbered vignettes define the canonical NNS curriculum.
+Python follows the same numbering, topic names, and statistical intent:
 
-See [behavior conventions](https://ovvo-financial.github.io/NNS-python/conventions/) for detailed compatibility notes.
+| # | Topic | Python entry point |
+|---|---|---|
+| 01 | Overview | [`01_overview.py`](examples/vignettes/01_overview.py) |
+| 02 | Partial Moments | [`02_partial_moments.py`](examples/vignettes/02_partial_moments.py) |
+| 03 | Correlation and Dependence | [`03_correlation_and_dependence.py`](examples/vignettes/03_correlation_and_dependence.py) |
+| 04 | Normalization and Rescaling | [`04_normalization_and_rescaling.py`](examples/vignettes/04_normalization_and_rescaling.py) |
+| 05 | Sampling and Simulation | [`05_sampling_and_simulation.py`](examples/vignettes/05_sampling_and_simulation.py) |
+| 06 | Comparing Distributions | [`06_comparing_distributions.py`](examples/vignettes/06_comparing_distributions.py) |
+| 07 | Clustering and Regression | [`07_clustering_and_regression.py`](examples/vignettes/07_clustering_and_regression.py) |
+| 08 | Classification | [`08_classification.py`](examples/vignettes/08_classification.py) |
+| 09 | Forecasting | [`09_forecasting.py`](examples/vignettes/09_forecasting.py) |
 
-## Examples
+The mapping is recorded in
+[`examples/vignettes/manifest.yml`](examples/vignettes/manifest.yml) and
+validated in CI. The older unnumbered scripts remain as focused examples and
+backward-compatible entry points.
 
-Runnable, self-checking example scripts live in
-[`examples/vignettes`](https://github.com/OVVO-Financial/NNS-python/tree/main/examples/vignettes), mirroring the R NNS vignettes. They
-are exercised in CI by `tests/docs/test_vignette_examples.py`, so they stay in
-sync with the package.
-
-| Topic | Script |
-|---|---|
-| Overview | [`overview.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/overview.py) |
-| Partial moments | [`partial_moments.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/partial_moments.py) |
-| Descriptive and distributional tools | [`descriptive_distributional_tools.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/descriptive_distributional_tools.py) |
-| Dependence and nonlinear association | [`dependence_nonlinear_association.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/dependence_nonlinear_association.py) |
-| Normalization and rescaling | [`normalization_rescaling.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/normalization_rescaling.py) |
-| Hypothesis, ANOVA and stochastic superiority | [`hypothesis_anova_stochastic_superiority.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/hypothesis_anova_stochastic_superiority.py) |
-| Regression, boosting, stacking and causality | [`regression_boosting_stacking_causality.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/regression_boosting_stacking_causality.py) |
-| Time series forecasting | [`time_series_forecasting.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/time_series_forecasting.py) |
-| Simulation, bootstrap and risk-neutral | [`simulation_bootstrap_riskneutral.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/simulation_bootstrap_riskneutral.py) |
-| Portfolio and stochastic dominance | [`portfolio_stochastic_dominance.py`](https://github.com/OVVO-Financial/NNS-python/blob/main/examples/vignettes/portfolio_stochastic_dominance.py) |
-
-Run one example:
+Run one canonical example:
 
 ```bash
-uv run python examples/vignettes/partial_moments.py
+uv run python examples/vignettes/02_partial_moments.py
 ```
 
-Run all of them with a PASS/FAIL summary:
+Run all nine in R curriculum order:
 
 ```bash
 uv run python examples/run_all_vignettes.py
 ```
 
-## Documentation
+## Design boundaries
+
+NNS Python prioritizes stable public behavior from installed R NNS 13.0+, not
+private helper parity. The package returns NumPy arrays and plain dictionaries
+rather than R `data.table` objects and uses explicit Python errors for unsafe R
+coercions.
 
-The full documentation site is hosted at
-**<https://ovvo-financial.github.io/NNS-python/>**.
+- R is used for parity tests and local cache regeneration, not normal runtime.
+- Exact stochastic stream parity is not expected for every randomized path.
+- Factor and class ordering should be supplied explicitly when it matters.
+- Classification codes follow the R contract and start at 1.
+- Compute functions return values; `plot=True` adds Matplotlib rendering as a
+  side effect without changing the statistical result.
 
-- [API reference manual](https://ovvo-financial.github.io/NNS-python/api_reference/)
+See [behavior conventions](https://ovvo-financial.github.io/NNS-python/conventions/)
+for detailed compatibility notes.
+
+## Documentation
+
+- [API reference](https://ovvo-financial.github.io/NNS-python/api_reference/)
 - [API status and known gaps](https://ovvo-financial.github.io/NNS-python/api_status/)
-- [Behavior conventions and intentional divergences](https://ovvo-financial.github.io/NNS-python/conventions/)
-- [Parity target, cache regeneration, and automation](https://ovvo-financial.github.io/NNS-python/parity/)
+- [Behavior conventions](https://ovvo-financial.github.io/NNS-python/conventions/)
+- [Parity policy and cache regeneration](https://ovvo-financial.github.io/NNS-python/parity/)
 - [Benchmarks](https://ovvo-financial.github.io/NNS-python/benchmarks/)
-- [Examples](https://github.com/OVVO-Financial/NNS-python/tree/main/examples/vignettes)
+- [Canonical examples](examples/vignettes/README.md)
 
 ## Development
 
@@ -204,11 +164,9 @@ Run benchmark tests explicitly:
 uv run pytest -n0 -m benchmark --benchmark-enable tests/benchmarks/
 ```
 
-The default parity suite is cache-backed and does not require `Rscript`. `Rscript` and the R `NNS` package are needed only when regenerating parity caches or running live R comparison scripts.
-
-## Benchmarks
-
-Benchmarks compare selected Python paths with installed R NNS 13.0+ baselines. Many core operations are faster in Python, while some large stochastic-dominance workloads remain faster in R because the R package uses compiled kernels for those paths. See [benchmarks](https://ovvo-financial.github.io/NNS-python/benchmarks/) for current measurements and commands.
+The default parity suite is cache-backed and does not require `Rscript`.
+`Rscript` and the R `NNS` package are needed only when regenerating parity
+caches or running live R comparison scripts.
 
 ## Authors and contributors
 
@@ -218,4 +176,5 @@ Benchmarks compare selected Python paths with installed R NNS 13.0+ baselines. M
 
 ## Attribution
 
-Upstream R package and reference implementation: [OVVO-Financial/NNS](https://github.com/OVVO-Financial/NNS)
+Upstream R package and reference implementation:
+[OVVO-Financial/NNS](https://github.com/OVVO-Financial/NNS)

examples/run_all_vignettes.py

@@ -1,54 +1,35 @@
-"""Run every NNS Python vignette end to end and print its output.
+"""Run the nine canonical NNS Python vignettes in R curriculum order.
 
-This is a single, IDLE-friendly driver for the vignette example scripts in
-``examples/vignettes/``. Open it in IDLE and press **F5** (or run
-``python examples/run_all_vignettes.py`` from a terminal) to execute all
-vignettes in the documented order and print each one's output, so you can
-compare it against the R NNS vignettes PDF.
-
-Each vignette also self-checks with assertions, so this driver reports PASS/FAIL
-per vignette and a final summary, and exits non-zero if any vignette fails.
-
-Requires the package to be importable (``pip install -e .`` from the repo root).
+The R package is the source of truth. The numbered Python entry points under
+``examples/vignettes`` follow the same 01-09 topic sequence and self-check with
+assertions. This driver reports PASS/FAIL and exits non-zero on failure.
 """
-
 from __future__ import annotations
 
 import importlib.util
 import os
+import sys
 import time
 import traceback
 from pathlib import Path
 
 REPO_ROOT = Path(__file__).resolve().parents[1]
 VIGNETTE_DIR = REPO_ROOT / "examples" / "vignettes"
 
-# Ordered to match the R NNS vignettes PDF.
-# (number, title, example-script stem)
 VIGNETTES = [
-    ("00", "Overview", "overview"),
-    ("01", "Partial moments", "partial_moments"),
-    ("02", "Descriptive & distributional tools",
-     "descriptive_distributional_tools"),
-    ("03", "Dependence & nonlinear association",
-     "dependence_nonlinear_association"),
-    ("04", "Normalization & rescaling",
-     "normalization_rescaling"),
-    ("05", "Hypothesis, ANOVA & stochastic superiority",
-     "hypothesis_anova_stochastic_superiority"),
-    ("06", "Regression, boosting, stacking & causality",
-     "regression_boosting_stacking_causality"),
-    ("07", "Time series forecasting",
-     "time_series_forecasting"),
-    ("08", "Simulation, bootstrap & risk-neutral",
-     "simulation_bootstrap_riskneutral"),
-    ("09", "Portfolio & stochastic dominance",
-     "portfolio_stochastic_dominance"),
+    ("01", "Overview", "01_overview"),
+    ("02", "Partial Moments", "02_partial_moments"),
+    ("03", "Correlation and Dependence", "03_correlation_and_dependence"),
+    ("04", "Normalization and Rescaling", "04_normalization_and_rescaling"),
+    ("05", "Sampling and Simulation", "05_sampling_and_simulation"),
+    ("06", "Comparing Distributions", "06_comparing_distributions"),
+    ("07", "Clustering and Regression", "07_clustering_and_regression"),
+    ("08", "Classification", "08_classification"),
+    ("09", "Forecasting", "09_forecasting"),
 ]
 
 
-def _load_vignette_main(stem):
-    """Import an example script by path and return its ``main`` callable."""
+def _load_vignette_main(stem: str):
     path = VIGNETTE_DIR / f"{stem}.py"
     spec = importlib.util.spec_from_file_location(f"nns_vignette_{stem}", path)
     if spec is None or spec.loader is None:
@@ -59,10 +40,12 @@ def _load_vignette_main(stem):
 
 
 def run() -> int:
-    # Match the cwd the test suite uses so any relative paths resolve.
     os.chdir(REPO_ROOT)
+    # Numbered compatibility entry points import maintained sibling scripts.
+    if str(VIGNETTE_DIR) not in sys.path:
+        sys.path.insert(0, str(VIGNETTE_DIR))
 
-    results = []
+    results: list[tuple[str, str, bool, float]] = []
     for number, title, stem in VIGNETTES:
         banner = f"  Vignette {number}: {title}  "
         rule = "=" * max(len(banner), 60)
@@ -78,11 +61,10 @@ def run() -> int:
         except Exception:
             ok = False
             print(traceback.format_exc())
-        elapsed = time.perf_counter() - start
-        results.append((number, title, ok, elapsed))
+        results.append((number, title, ok, time.perf_counter() - start))
 
     print("\n" + "=" * 60)
-    print("  Vignette verification summary")
+    print("  Canonical vignette verification summary")
     print("=" * 60)
     passed = 0
     for number, title, ok, elapsed in results:
@@ -91,8 +73,6 @@ def run() -> int:
         print(f"  [{status}] {number}  {title}  ({elapsed:.2f}s)")
     print("-" * 60)
     print(f"  {passed}/{len(results)} vignettes passed")
-    if passed != len(results):
-        print("  Some vignettes FAILED — see tracebacks above.")
     return 0 if passed == len(results) else 1
 
 

examples/vignettes/01_overview.py

@@ -0,0 +1,12 @@
+"""Canonical vignette 01: Overview.
+
+Source of truth:
+    NNS/vignettes/NNSvignette_01_Overview.Rmd
+
+The maintained implementation lives in ``overview.py``. This numbered entry
+point keeps the Python example catalog aligned with the canonical R sequence.
+"""
+from overview import main
+
+if __name__ == "__main__":
+    main()