Add time intervals

.github/workflows/lint.yml

@@ -20,5 +20,7 @@ jobs:
     steps:
         - name: Checkout repository
           uses: actions/checkout@v4
+        - name: Install clang-format
+          run: pipx install clang-format
         - name: Lint
           run: cmake -D FORMAT_COMMAND=clang-format -P cmake/lint.cmake

.github/workflows/tests.yml

@@ -144,7 +144,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         python -m pip install nwbinspector
-        nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_subject_exists --json-file-path out.json
+        nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_electrodes_location_allen_ccf --json-file-path out.json
         if ! grep -q '"messages": \[\]' out.json; then
           echo "NWBInspector found issues in the NWB files"
           exit 1

.github/workflows/upgrade_schema.yml

@@ -58,7 +58,7 @@ jobs:
         run: |
           mkdir -p nwb_files
           cp build/tests/data/*.nwb nwb_files/
-          nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_subject_exists --json-file-path out.json
+          nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_electrodes_location_allen_ccf --json-file-path out.json
           if ! grep -q '"messages": \[\]' out.json; then
             echo "NWBInspector found issues in the NWB files"
             exit 1

CMakeLists.txt

@@ -36,18 +36,21 @@ add_library(
     src/nwb/device/Device.cpp
     src/nwb/ecephys/ElectricalSeries.cpp
     src/nwb/ecephys/SpikeEventSeries.cpp
+    src/nwb/epoch/TimeIntervals.cpp
     src/nwb/event/EventsTable.cpp
     src/nwb/event/TimestampVectorData.cpp
     src/nwb/event/DurationVectorData.cpp
     src/nwb/file/ElectrodeGroup.cpp
     src/nwb/file/ElectrodesTable.cpp
+    src/nwb/file/Subject.cpp
     src/nwb/misc/AnnotationSeries.cpp
     src/nwb/hdmf/base/Container.cpp
     src/nwb/hdmf/base/Data.cpp
     src/nwb/hdmf/table/DynamicTable.cpp
+    src/nwb/hdmf/table/ElementIdentifiers.cpp
     src/nwb/hdmf/table/MeaningsTable.cpp
     src/nwb/hdmf/table/VectorData.cpp
-    src/nwb/hdmf/table/ElementIdentifiers.cpp
+    src/nwb/hdmf/table/VectorIndex.cpp
     src/spec/NamespaceRegistry.cpp
 )
 

docs/Doxyfile.in

@@ -81,3 +81,4 @@ PROJECT_LOGO           = "@PROJECT_SOURCE_DIR@/resources/logo/aqnwb-logo.png"
 
 # Fail on warnings
 WARN_AS_ERROR = FAIL_ON_WARNINGS_PRINT
+DOT_GRAPH_MAX_NODES = 100

docs/pages/1_userdocs.dox

@@ -7,6 +7,7 @@
  * - \subpage user_install_page
  * - \subpage workflow
  * - \subpage events
+ * - \subpage time_intervals
  * - \subpage read_page
  * - \subpage reads3_page
  * - \subpage hdf5io

docs/pages/devdocs/read_design.dox

@@ -291,6 +291,12 @@
  *    - The \ref AQNWB::NWB::RegisteredType::create "RegisteredType::create" method is used to create
  *      an instance of a registered subclass by name. This method looks up the subclass name in the 
  *      registry and calls the corresponding factory function to create an instance.
+ *    - Each I/O object caches one canonical registered object per file path.
+ *      The path-only \ref AQNWB::NWB::RegisteredType::create "create" overload
+ *      resolves the object's on-disk ``namespace`` and ``neurodata_type`` before
+ *      reusing a cache entry. A request whose registered type conflicts with an
+ *      existing cache entry fails rather than returning a different type or
+ *      creating a second object for the same path.
  *
  * 5. **Class Name and Namespace Retrieval**:
  *    - The \ref AQNWB::NWB::RegisteredType::getTypeName "getTypeName" and
@@ -339,6 +345,13 @@
  * with the type already set. The same approach is also applied in the case of \ref AQNWB::NWB::Data "Data" and its derived
  * class \ref AQNWB::NWB::DataTyped "DataTyped".
  *
+ * ``DataTyped<DTYPE>``, ``VectorDataTyped<DTYPE>``, and
+ * ``NWBDataTyped<DTYPE>`` are typed read facades, not canonical registered
+ * objects. They are intentionally not stored in
+ * \ref AQNWB::IO::RecordingObjects "RecordingObjects"; use their specialized
+ * ``create`` or ``from...`` methods to obtain a typed view of a dataset without
+ * changing the canonical object cached for its path.
+ *
  * For further details and alternative approaches for implementing templated \ref AQNWB::NWB::RegisteredType "RegisteredType"
  * classes, see \ref implement_templated_registered_type.  
  *

docs/pages/devdocs/record_design.dox

@@ -18,7 +18,7 @@
  * AqNWB records datasets efficiently via \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects. The main components involved in
  * writing data to an NWB file via AqNWB are:
  *
- * 1. \ref DEFINE_DATASET_FIELD Macro
+ * 1. \ref DEFINE_DATASET_FIELD Macro  and \ref AQNWB::NWB::RegisteredType "RegisteredType"
  *   - The \ref DEFINE_DATASET_FIELD macro not only defines methods for reading datasets for a particular neurodata_type (as described in \ref read_design_page),
  *     but also defines methods for retrieving \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects that are used for recording to individual datasets.
  *     For each dataset field defined with this macro, a corresponding method is generated that returns a \ref AQNWB::IO::BaseRecordingData "BaseRecordingData"
@@ -31,13 +31,14 @@
  *   - It provides methods for writing data blocks to the dataset, such as \ref AQNWB::IO::BaseRecordingData::writeDataBlock "writeDataBlock",
  *      which can handle different data types and dimensions.
  *
- * 3. \ref AQNWB::NWB::RegisteredType "RegisteredType"
- *   - \ref AQNWB::NWB::RegisteredType "RegisteredType" maintains a cache of \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects via the `m_recordingDataCache` member.
+ * 3. \ref AQNWB::IO::BaseIO::m_recordingDataCache "BaseIO::m_recordingDataCache"
+ *   - \ref AQNWB::IO::BaseIO "BaseIO" maintains a cache of \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects that belong to the I/O object/file.
  *     This cache allows reusing the same \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" object when it is requested multiple times,
  *     improving performance and retaining the recording position. The cache is essential for writing data to the dataset in a streaming fashion,
  *     as it ensures that each write continues from where the previous write left off. The cache also avoids the need for manually maintaining
  *     the objects and allows caching of an arbitrary number of \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" object such that the 
- *     individual neurodata_type classes do not need to worry about maintaining their recording state.
+ *     individual RegisteredType classes (i.e., neurodata_types) do not need to worry about maintaining their recording state, but the state is maintained 
+ *     centrally in the I/O object they belong to.
  *
  * 4. \ref AQNWB::IO::BaseIO "BaseIO"
  *   - \ref AQNWB::IO::BaseIO "BaseIO" and its implementations (e.g., \ref AQNWB::IO::HDF5::HDF5IO "HDF5IO") are responsible for
@@ -64,6 +65,7 @@
  *                 <tr><td align="left">+ getDataSet(): BaseRecordingData</td></tr>
  *                 <tr><td colspan="2" bgcolor="lightgray"><b>Attributes</b></td></tr>
  *                 <tr><td align="left">- m_recording_objects:  std::shared_ptr&lt;RecordingObjects&gt;</td></tr>
+ *                 <tr><td align="left">- m_recordingDataCache: std::unordered_map&lt;std::string, std::shared_ptr&lt;BaseRecordingData&gt;&gt;</td></tr>
  *             </table>
  *         >
  *     ];
@@ -91,11 +93,8 @@
  *         label=<
  *             <table border="0" cellborder="1" cellspacing="0">
  *                 <tr><td colspan="2" bgcolor="lightgray"><b>RegisteredType</b></td></tr>
- *                 <tr><td colspan="2" bgcolor="lightgray"><b>Functions</b></td></tr>
- *                 <tr><td align="left">+ clearRecordingDataCache(): void</td></tr>
  *                 <tr><td colspan="2" bgcolor="lightgray"><b>Attributes</b></td></tr>
- *                 <tr><td align="left">+ m_recordingDataCache: std::unordered_map</td></tr>
- *                 <tr><td align="left">+ m_io: std::shared_ptr&lt;BaseIO&gt;</td></tr>
+ *                 <tr><td align="left">+ m_io: std::weak_ptr&lt;BaseIO&gt;</td></tr>
  *                 <tr><td align="left">+ m_path: std::string</td></tr>
  *             </table>
  *         >
@@ -119,7 +118,8 @@
  *
  *     RegisteredType -> Container [arrowhead=empty, style=dashed];
  *     Container -> BaseRecordingData [label="created by functions defined via \nDEFINE_DATASET_FIELD"];
- *     RegisteredType -> BaseRecordingData [label="caches"];
+ *     HDF5IO -> BaseRecordingData [label="caches"];
+ *     HDF5IO -> RegisteredType [label="caches"];
  *     BaseRecordingData -> HDF5IO [label="uses for\nwriting data"];
  *     HDF5IO -> NWBFile [label="write data"];
  * }
@@ -197,7 +197,6 @@
  *   \ref AQNWB::IO::RecordingObjects::addRecordingObject "addRecordingObject" and 
  *   \ref AQNWB::IO::RecordingObjects::addRecordingObject "getRecordingObject" methods.
  * - **Finalization of recording objects:** via \ref AQNWB::IO::RecordingObjects::finalize "finalize"
- * - **Clearing of cached BaseRecordingData objects** via \ref AQNWB::IO::RecordingObjects::clearRecordingDataCache "clearRecordingDataCache";
  *
  * \section record_design_nwbio_utils NWB I/O convenience utilities
  *
@@ -233,9 +232,20 @@
  * facilitate reliable memory management. 
  *
  * \note
- * Registration with the \ref AQNWB::IO::RecordingObjects "RecordingObjects"  of the I/O occurs when a 
- * \ref AQNWB::NWB::RegisteredType "RegisteredType" is being created via the static
- * \ref AQNWB::NWB::RegisteredType::create "RegisteredType::create" factory methods. 
+ * Canonical \ref AQNWB::NWB::RegisteredType "RegisteredType" objects created
+ * through \ref AQNWB::NWB::RegisteredType::create "RegisteredType::create"
+ * are registered with the I/O's \ref AQNWB::IO::RecordingObjects
+ * "RecordingObjects". The cache has one canonical registered object per I/O
+ * and path; repeated compatible factory calls return that same instance. A
+ * request for a conflicting registered type fails rather than creating a
+ * second object for the path.
+ *
+ * Typed dataset facades such as \ref AQNWB::NWB::DataTyped "DataTyped",
+ * \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped", and
+ * \ref AQNWB::NWB::NWBDataTyped "NWBDataTyped" are intentionally transient
+ * and are not registered with RecordingObjects. They share the I/O's
+ * BaseRecordingData cache with the canonical object when accessing the same
+ * dataset.
  *
  * \section recording_design_further_reading Further Reading 
  *

docs/pages/devdocs/registered_types.dox

@@ -84,7 +84,10 @@
  *    DEFINE_ATTRIBUTE_FIELD(getDescription, std::string, "description", Description of the data)
  *    @endcode
  *    and in for datasets also a method for retrieving a \ref AQNWB::IO::BaseRecordingData "BaseRecordingData"
- *    wrapper used for recording data to a dataset.
+ *    wrapper used for recording data to a dataset. The \ref AQNWB::IO::BaseRecordingData "BaseRecordingData"
+ *    objects are cached by the \ref AQNWB::IO::BaseIO "BaseIO" object to ensure that the current
+ *    state of the recording process (e.g., the \ref AQNWB::IO::BaseRecordingData::getPosition "position")
+ *    are stored.
  *    @code
  *    DEFINE_DATASET_FIELD(getData, recordData, float, "data", The main data)
  *    @endcode
@@ -389,19 +392,25 @@
  * the \ref DEFINE_REGISTERED_FIELD macro, we can set the data type for read at compile time, simplifying read. 
  *
  *  \note
- *     Since the templacted  \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped<DTYPE>" does not call
- *     \ref REGISTER_SUBCLASS we must manually declare: 1) `friend class AQNWB::NWB::RegisteredType;` 
- *     and 2) `using VectorData::VectorData;` (as protected) to make sure we have access to the 
- *     constructors and define `static std::shared_ptr<VectorDataTyped> create( const std::string& path, std::shared_ptr<AQNWB::IO::BaseIO> io) {  return RegisteredType::create<VectorDataTyped>(path, io); }`
- *     to make sure we have the factory method available.
- *
+ *     Since the templated \ref AQNWB::NWB::VectorDataTyped
+ *     "VectorDataTyped<DTYPE>" does not call \ref REGISTER_SUBCLASS, it must
+ *     declare `friend class AQNWB::NWB::RegisteredType;` and inherit the
+ *     protected `VectorData` constructors with `using VectorData::VectorData;`
+ *     so its specialized factory can construct it. That factory must create a
+ *     transient typed view rather than delegate to
+ *     \ref AQNWB::NWB::RegisteredType::create
+ *     "RegisteredType::create<T>", because that factory registers canonical
+ *     objects in \ref AQNWB::IO::RecordingObjects "RecordingObjects".
+ *     `DataTyped<DTYPE>` and `NWBDataTyped<DTYPE>` follow the same transient
+ *     typed-view pattern; only their non-templated base classes are canonical
+ *     registered types.
+ *
+ *  \par
  *  \note
- *     The `std::unique_ptr<..>` template type is not covariant, i.e.,  `std::unique_ptr<DerivedClass>` does 
- *     not automatically convert to `std::unique_ptr<BaseClass>`.  I.e.,  while \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped<DTYPE>" 
- *     can be used anywhere \ref AQNWB::NWB::VectorData "VectorData" is being used, when using `std::unique_ptr<..>`
- *     we cannot rely on the compiler to automatically upcast for us, but we will need to explicitly release
- *     and upcast `std::unique_ptr<VectorDataTyped<DTYPE>` if `std::unique_ptr<VectorData>` is required. Since 
- *     defining the `DTYPE` is primarily useful for read, we therefore typically use 
+ *     AqNWB uses `std::shared_ptr` for registered and typed-view objects, so a
+ *     `std::shared_ptr<VectorDataTyped<DTYPE>>` converts to
+ *     `std::shared_ptr<VectorData>` when needed. Since defining `DTYPE` is
+ *     primarily useful for read, we therefore typically use
  *     \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped<DTYPE>" on read while
  *     using \ref AQNWB::NWB::VectorData "VectorData" otherwise. 
  *
@@ -529,6 +538,7 @@
  * than the requested version. In practice, this is mainly relevant if you are generating classes for the
  * NWB `core` and `hdmf-common` namespaces.
  *
+ * \par
  * \note
  * When generating the test app via the `--generate-test-app` option we also
  * need to generate the schema headers files and save them in the `/spec` folder.

docs/pages/userdocs/events.dox

@@ -121,6 +121,7 @@
  *
  * After `startRecording`, retrieve each column object via the corresponding
  * `read*Column()` accessor on the table, then call
+ * \ref AQNWB::NWB::Data::recordData "recordData" on the column and
  * \ref AQNWB::IO::BaseRecordingData::writeDataBlock "writeDataBlock" to write the
  * full column vector in one call.  Finally, set the row IDs with
  * \ref AQNWB::NWB::DynamicTable::setRowIDs "setRowIDs".

docs/pages/userdocs/read.dox

@@ -184,6 +184,67 @@
  *
  * \snippet tests/examples/test_ecephys_data_read.cpp  example_use_std_variant_to_compute_on_data
  * 
+ * \subsection read_example_dynamic_table Reading data from DynamicTable objects
+ *
+ * First, let's get the table we want to read from. In this example, we'll use the `InvalidTimes` table.
+ * We can also see the standard approach of reading a single column's data, which is efficient when you 
+ * only need data from one or a few columns.
+ *
+ * \snippet tests/examples/test_ecephys_data_read.cpp example_read_dynamic_table_rows_setup_snippet
+ *
+ * In addition to reading individual columns, \ref AQNWB::NWB::DynamicTable::readRows "DynamicTable::readRows(start, count, colNames)" 
+ * provides a convenient method for reading a consecutive range of rows from the table. Note, `start`
+ * as the starting row index is *inclusive*, while `count` is the number of rows to read.
+ * `colNames` is optional and allows reading of a subset of columns.
+ *
+ * This method returns a `std::vector<RowData>`, where each \ref AQNWB::Types::RowData "RowData" is a map 
+ * of column names to their corresponding \ref AQNWB::Types::CellValue "CellValue"  values for that row.
+ *
+ * Now, let's use the `readRows` approach to read a slice of rows (e.g., rows 0 to 2). This is convenient 
+ * when you need to process data row-by-row across multiple columns.
+ *
+ * \snippet tests/examples/test_ecephys_data_read.cpp example_read_dynamic_table_rows_all_cols_snippet
+ *
+ * \note
+ * The \ref AQNWB::Types::CellValue "CellValue" struct holds an `std::variant`.  \ref AQNWB::Types::CellValue "CellValue"
+ * provides an implicit conversion operator for convenient shorthand syntax when extracting 
+ * values (e.g., `float val = row.at("col");`). However, because it relies on implicit conversion, it 
+ * could lead to ambiguous function calls or unexpected behavior if the compiler infers the wrong type.
+ * For a safer and more explicit approach, it is recommended to use the `get<T>()` method 
+ * (e.g., `float val = row.at("col").get<float>();`). However, in either approach, a mismatch 
+ * between the underlying type and the target type will result in a `std::bad_variant_access`
+ * runtime error. For additional safety:
+ * - <b>Check before you get via \ref AQNWB::Types::CellValue.holds_alternative "CellValue.holds_alternative":</b>
+ *   You can check if the variant holds a specific type before trying to extract it. If it returns true,
+ *   `std::get` is guaranteed to succeed. 
+ * - **Get a pointer via `std::get_if`:** Instead of `std::get` (which throws on failure), you can use 
+ *   `std::get_if`. It returns a pointer to the value if the type matches, or a `nullptr` if it doesn't
+ *   so you then check if the pointer is null.
+ * - **Handle all types via `std::visit`:** Using `std::visit` you provide a function 
+ *   (or a generic lambda with `auto`) that can accept any of the possible types, and `std::visit`
+ *   will automatically call your function with the correct type that is currently stored. This
+ *   eliminates the need to guess the type and prevents the exception entirely. 
+ *
+ * You can also specify a subset of columns to read by passing a vector of column names as the third 
+ * argument to `readRows`. This can improve performance if you only need data from specific columns.
+ *
+ * \snippet tests/examples/test_ecephys_data_read.cpp example_read_dynamic_table_rows_specific_cols_snippet
+ *
+ * Finally, you can easily print the contents of a `DynamicTable` using the 
+ * \ref AQNWB::NWB::DynamicTable::toString "DynamicTable::toString" method. This is useful for debugging 
+ * and inspecting the table's contents.
+ *
+ * \snippet tests/examples/test_ecephys_data_read.cpp example_read_dynamic_table_to_string_snippet
+ *
+ * This will output the table contents in a readable CSV format:
+ *
+ * \code
+ * InvalidTimes table contents:
+ * id,start_time,stop_time,tags
+ * 0,1.000000,2.000000,"[device_error, user_error]"
+ * 1,3.000000,4.000000,"[external_interference]"
+ * 2,5.000000,6.000000,"[lost_connection, external_interference, user_error]"
+ * \endcode
  *
  * \section read_further_reading Further reading
  * - \ref read_design_page discusses the underlying software design of AqNWB for data read