Add time slicing benchmark

.github/workflows/testing.yml

@@ -65,6 +65,14 @@ jobs:
         working-directory: nwb_benchmarks
         run: python -m nwb_benchmarks.scripts.check_dandi_urls
 
+      # `--group` installs a dependency group from `pyproject.toml` and needs pip 25.1 or newer.
+      - name: Run unit tests
+        working-directory: nwb_benchmarks
+        run: |
+          python -m pip install --upgrade "pip>=25.1"
+          python -m pip install --group test
+          python -m pytest tests
+
       # This run is a smoke test of the benchmark machinery, not a measurement, so each benchmark is restricted to a
       # single parameter set. ASV matches the `--bench` pattern against `module.Class.method({parameters})`, which
       # lets the pattern pick one parametrization. Every parameter set opens the full remote NWB file in `setup`,

docs/conf.py

@@ -14,6 +14,7 @@
     "sphinx_search.extension",  # Allows for auto search function the documentation
     "sphinx.ext.viewcode",  # Shows source code in the documentation
     "sphinx.ext.extlinks",  # Allows to use shorter external links defined in the extlinks variable.
+    "sphinx_tabs.tabs",  # Groups platform-specific code blocks into synchronized tabs
 ]
 
 templates_path = ["_templates"]

docs/development.rst

@@ -29,6 +29,20 @@ Writing thorough docstrings is encouraged; please follow the Numpy style.
 Import and submodule structure follows ``scikit-learn`` standard.
 
 
+Unit Tests
+----------
+
+The ``tests`` directory holds unit tests for the code around the benchmarks, such as the reduction of ASV results and
+the helpers of the incremental slicing benchmarks. They run offline and do not run any benchmark. To run them...
+
+.. code-block::
+
+    pip install -e . --group test
+    pytest tests
+
+The CI runs them on every pull request, before the benchmark smoke test.
+
+
 Customized Machine Header
 -------------------------
 

docs/requirements-rtd.txt

@@ -3,3 +3,4 @@ sphinx==5.1.1
 pydata-sphinx-theme==0.14.1
 readthedocs-sphinx-search==0.3.2
 sphinx-copybutton==0.5.0
+sphinx-tabs==3.4.5

docs/running_benchmarks.rst

@@ -52,6 +52,157 @@ Or drop the ``sudo`` if on Windows.
 Many of the current tests can take several minutes to complete; the entire suite will take many times that. Grab some coffee, read a book, or better yet (when the suite becomes larger) just leave it to run overnight.
 
 
+Environment Variables
+---------------------
+
+The benchmark suite uses a few optional environment variables to configure network tracking and to opt in to
+long-running benchmark families that are disabled by default.
+
+``NWB_BENCHMARKS_NETWORK_INTERFACE``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Network tracking benchmarks monitor traffic on a specific network interface. Set ``NWB_BENCHMARKS_NETWORK_INTERFACE``
+to the interface that should be monitored, for example ``en0`` on macOS or ``eth0`` on many Linux systems:
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export NWB_BENCHMARKS_NETWORK_INTERFACE=en0
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:NWB_BENCHMARKS_NETWORK_INTERFACE="Ethernet"
+
+If this variable is not set, the package will warn when ``tshark`` is available. Network tracking results may be
+missing or invalid until a suitable interface is configured.
+
+``TSHARK_PATH``
+~~~~~~~~~~~~~~~
+
+Network tracking benchmarks require the ``tshark`` executable. If ``tshark`` is installed on your ``PATH``, no extra
+configuration is needed. Otherwise, set ``TSHARK_PATH`` to the absolute path to the executable:
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export TSHARK_PATH=/path/to/tshark
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:TSHARK_PATH="C:\Program Files\Wireshark\tshark.exe"
+
+If ``tshark`` cannot be found, network tracking benchmarks are skipped.
+
+``RUN_DOWNLOAD_BENCHMARKS``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Benchmarks that download entire remote test files are disabled by default because they can take a long time and consume
+substantial bandwidth and disk space. Set ``RUN_DOWNLOAD_BENCHMARKS`` to any non-empty value to include them:
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export RUN_DOWNLOAD_BENCHMARKS=true
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:RUN_DOWNLOAD_BENCHMARKS="true"
+
+``RUN_INCREMENTAL_SLICING_BENCHMARKS``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Incremental slicing benchmarks are disabled by default because they can be long-running. Set
+``RUN_INCREMENTAL_SLICING_BENCHMARKS`` to opt in. The value can either enable all incremental slicing benchmarks or
+select specific data modalities.
+
+Accepted disabled values are unset, empty, ``false``, ``0``, ``no``, or ``off``.
+
+Accepted values to run all incremental slicing modalities are ``true``, ``1``, ``yes``, ``on``, or ``all``:
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export RUN_INCREMENTAL_SLICING_BENCHMARKS=true
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:RUN_INCREMENTAL_SLICING_BENCHMARKS="true"
+
+Accepted modality names are ``ecephys``, ``ophys``, and ``icephys``. To run only one modality:
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export RUN_INCREMENTAL_SLICING_BENCHMARKS=icephys
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:RUN_INCREMENTAL_SLICING_BENCHMARKS="icephys"
+
+To run multiple modalities, provide a comma-separated list:
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export RUN_INCREMENTAL_SLICING_BENCHMARKS=icephys,ophys
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:RUN_INCREMENTAL_SLICING_BENCHMARKS="icephys,ophys"
+
+Invalid values raise an error when the benchmark suite is imported so that misspellings do not accidentally run the
+wrong set of benchmarks.
+
+Example: running only icephys incremental slicing benchmarks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. tabs::
+
+    .. group-tab:: macOS / Linux
+
+        .. code-block:: bash
+
+            export RUN_INCREMENTAL_SLICING_BENCHMARKS=icephys
+            nwb_benchmarks run --bench track_incremental_slicing --debug
+
+    .. group-tab:: Windows (PowerShell)
+
+        .. code-block:: powershell
+
+            $env:RUN_INCREMENTAL_SLICING_BENCHMARKS="icephys"
+            nwb_benchmarks run --bench track_incremental_slicing --debug
+
+
 Additional Flags
 ----------------
 

pyproject.toml

@@ -12,7 +12,7 @@ packages = ["src/nwb_benchmarks"]
 
 [project]
 name = "nwb_benchmarks"
-version="0.1.2"
+version="0.2.0"
 authors = [
   { name="Cody Baker", email="cody.baker@catalystneuro.com" },
 ]
@@ -59,10 +59,12 @@ nwb_benchmarks = "nwb_benchmarks.command_line_interface:main"
 database = ["polars"]
 dev = ["ipython", "pre-commit"]
 figures = ["seaborn", "matplotlib", "pyarrow"]
+test = ["pytest", {include-group = "database"}, {include-group = "figures"}]
 all = [
     {include-group = "database"},
     {include-group = "dev"},
-    {include-group = "figures"}
+    {include-group = "figures"},
+    {include-group = "test"}
 ]
 
 

src/nwb_benchmarks/benchmarks/params.py

@@ -1,3 +1,6 @@
+import os
+import warnings
+
 # The DANDI API download URLs in this module (`https_url_no_redirect`) are literals, resolved once with
 # `nwb_benchmarks.core.get_https_url(..., follow_redirects=False)`, so that importing the benchmark suite, which ASV does
 # in every process it starts, does not look every asset up through the DANDI API. The S3 locations those URLs redirect
@@ -198,6 +201,126 @@
 # icephys data has shape (N,) and chunk shape (8192,)
 icephys_slices = [(slice(0, 8192 * i),) for i in range(1, 6)]
 
+#################################### INCREMENTAL SLICE PARAMETERS ###################################
+
+
+def _parse_incremental_slicing_benchmarks_config(value: str | None) -> tuple[bool, set[str] | None]:
+    """Parse RUN_INCREMENTAL_SLICING_BENCHMARKS into enabled state and selected modalities."""
+    if value is None:
+        return False, None
+
+    normalized_value = value.strip().lower()
+    if normalized_value in {"", "false", "0", "no", "off"}:
+        return False, None
+
+    if normalized_value in {"true", "1", "yes", "on", "all"}:
+        return True, None
+
+    selected_modalities = {modality.strip().lower() for modality in normalized_value.split(",") if modality.strip()}
+    valid_modalities = {"ecephys", "ophys", "icephys"}
+    invalid_modalities = selected_modalities - valid_modalities
+    if invalid_modalities:
+        raise ValueError(
+            f"Invalid RUN_INCREMENTAL_SLICING_BENCHMARKS value {value!r}. "
+            "Expected true, all, ecephys, ophys, icephys, or a comma-separated list of modalities."
+        )
+
+    return True, selected_modalities
+
+
+(
+    RUN_INCREMENTAL_SLICING_BENCHMARKS,
+    RUN_INCREMENTAL_SLICING_BENCHMARK_MODALITIES,
+) = _parse_incremental_slicing_benchmarks_config(os.environ.get("RUN_INCREMENTAL_SLICING_BENCHMARKS", None))
+
+if RUN_INCREMENTAL_SLICING_BENCHMARKS:
+    if RUN_INCREMENTAL_SLICING_BENCHMARK_MODALITIES is None:
+        incremental_slicing_benchmark_description = "all incremental slicing benchmarks"
+    else:
+        incremental_slicing_benchmark_description = (
+            "incremental slicing benchmarks for modalities "
+            f"{', '.join(sorted(RUN_INCREMENTAL_SLICING_BENCHMARK_MODALITIES))}"
+        )
+    warnings.warn(
+        "RUN_INCREMENTAL_SLICING_BENCHMARKS is set. "
+        f"{incremental_slicing_benchmark_description} will be run, which may take a long time."
+    )
+
+
+def filter_incremental_slice_params_by_modality(params: tuple[dict, ...]) -> tuple[dict, ...]:
+    """Filter incremental slicing benchmark params by RUN_INCREMENTAL_SLICING_BENCHMARKS modality selection."""
+    if RUN_INCREMENTAL_SLICING_BENCHMARK_MODALITIES is None:
+        return params
+
+    return tuple(
+        parameter_set
+        for parameter_set in params
+        if parameter_set.get("modality") in RUN_INCREMENTAL_SLICING_BENCHMARK_MODALITIES
+    )
+
+
+incremental_ecephys_params = dict(
+    name="EcephysIncrementalSliceTestCase",
+    modality="ecephys",
+    object_name="ElectricalSeries",
+    slice_template=ecephys_slices[0],
+    slice_strategy="iterate_time_axis",
+)
+incremental_ophys_params = dict(
+    name="OphysIncrementalSliceTestCase",
+    modality="ophys",
+    object_name="TwoPhotonSeries",
+    slice_template=ophys_slices[0],
+    slice_strategy="iterate_time_axis",
+)
+incremental_icephys_params = dict(
+    name="IcephysIncrementalSliceTestCase",
+    modality="icephys",
+    slice_strategy="iterate_icephys_timeseries",
+)
+
+hdf5_redirected_read_incremental_slice_params = filter_incremental_slice_params_by_modality(
+    (
+        dict(**incremental_ecephys_params, https_url=hdf5_ecephys_params["https_url_redirected"]),
+        dict(**incremental_ophys_params, https_url=hdf5_ophys_params["https_url_redirected"]),
+        dict(**incremental_icephys_params, https_url=hdf5_icephys_params["https_url_redirected"]),
+    )
+)
+
+hdf5_no_redirect_download_incremental_slice_params = filter_incremental_slice_params_by_modality(
+    (
+        dict(**incremental_ecephys_params, https_url=hdf5_ecephys_params["https_url_no_redirect"]),
+        dict(**incremental_ophys_params, https_url=hdf5_ophys_params["https_url_no_redirect"]),
+        dict(**incremental_icephys_params, https_url=hdf5_icephys_params["https_url_no_redirect"]),
+    )
+)
+
+zarr_direct_read_incremental_slice_params = filter_incremental_slice_params_by_modality(
+    (
+        dict(**incremental_ecephys_params, https_url=zarr_ecephys_params["https_url_direct"]),
+        dict(**incremental_ophys_params, https_url=zarr_ophys_params["https_url_direct"]),
+        dict(**incremental_icephys_params, https_url=zarr_icephys_params["https_url_direct"]),
+    )
+)
+
+zarr_no_redirect_download_incremental_slice_params = filter_incremental_slice_params_by_modality(
+    (
+        dict(**incremental_ecephys_params, https_url=zarr_ecephys_params["https_url_no_redirect"]),
+        dict(**incremental_ophys_params, https_url=zarr_ophys_params["https_url_no_redirect"]),
+        dict(**incremental_icephys_params, https_url=zarr_icephys_params["https_url_no_redirect"]),
+    )
+)
+
+lindi_no_redirect_download_incremental_slice_params = filter_incremental_slice_params_by_modality(
+    (
+        dict(**incremental_ecephys_params, https_url=lindi_ecephys_params["https_url_no_redirect"]),
+        dict(**incremental_ophys_params, https_url=lindi_ophys_params["https_url_no_redirect"]),
+        dict(**incremental_icephys_params, https_url=lindi_icephys_params["https_url_no_redirect"]),
+    )
+)
+
+#################################### REMOTE SLICE PARAMETERS ###################################
+
 hdf5_redirected_read_slice_params = []
 for index, slice_range in enumerate(ecephys_slices):
     hdf5_redirected_read_slice_params.append(