Update schema to NWB 2.11

.github/workflows/lint.yml

@@ -20,5 +20,7 @@ jobs:
     steps:
         - name: Checkout repository
           uses: actions/checkout@v4
+        - name: Install clang-format
+          run: pipx install clang-format
         - name: Lint
           run: cmake -D FORMAT_COMMAND=clang-format -P cmake/lint.cmake

.github/workflows/tests.yml

@@ -144,7 +144,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         python -m pip install nwbinspector
-        nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_subject_exists --json-file-path out.json
+        nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_electrodes_location_allen_ccf --json-file-path out.json
         if ! grep -q '"messages": \[\]' out.json; then
           echo "NWBInspector found issues in the NWB files"
           exit 1

.github/workflows/upgrade_schema.yml

@@ -58,7 +58,7 @@ jobs:
         run: |
           mkdir -p nwb_files
           cp build/tests/data/*.nwb nwb_files/
-          nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_subject_exists --json-file-path out.json
+          nwbinspector nwb_files --threshold BEST_PRACTICE_VIOLATION --ignore=check_electrodes_location_allen_ccf --json-file-path out.json
           if ! grep -q '"messages": \[\]' out.json; then
             echo "NWBInspector found issues in the NWB files"
             exit 1

CMakeLists.txt

@@ -36,14 +36,21 @@ add_library(
     src/nwb/device/Device.cpp
     src/nwb/ecephys/ElectricalSeries.cpp
     src/nwb/ecephys/SpikeEventSeries.cpp
+    src/nwb/epoch/TimeIntervals.cpp
+    src/nwb/event/EventsTable.cpp
+    src/nwb/event/TimestampVectorData.cpp
+    src/nwb/event/DurationVectorData.cpp
     src/nwb/file/ElectrodeGroup.cpp
     src/nwb/file/ElectrodesTable.cpp
+    src/nwb/file/Subject.cpp
     src/nwb/misc/AnnotationSeries.cpp
     src/nwb/hdmf/base/Container.cpp
     src/nwb/hdmf/base/Data.cpp
     src/nwb/hdmf/table/DynamicTable.cpp
-    src/nwb/hdmf/table/VectorData.cpp
     src/nwb/hdmf/table/ElementIdentifiers.cpp
+    src/nwb/hdmf/table/MeaningsTable.cpp
+    src/nwb/hdmf/table/VectorData.cpp
+    src/nwb/hdmf/table/VectorIndex.cpp
     src/spec/NamespaceRegistry.cpp
 )
 

docs/Doxyfile.in

@@ -81,3 +81,4 @@ PROJECT_LOGO           = "@PROJECT_SOURCE_DIR@/resources/logo/aqnwb-logo.png"
 
 # Fail on warnings
 WARN_AS_ERROR = FAIL_ON_WARNINGS_PRINT
+DOT_GRAPH_MAX_NODES = 100

docs/pages/1_userdocs.dox

@@ -6,6 +6,8 @@
  *
  * - \subpage user_install_page
  * - \subpage workflow
+ * - \subpage events
+ * - \subpage time_intervals
  * - \subpage read_page
  * - \subpage reads3_page
  * - \subpage hdf5io

docs/pages/devdocs/read_design.dox

@@ -291,6 +291,12 @@
  *    - The \ref AQNWB::NWB::RegisteredType::create "RegisteredType::create" method is used to create
  *      an instance of a registered subclass by name. This method looks up the subclass name in the 
  *      registry and calls the corresponding factory function to create an instance.
+ *    - Each I/O object caches one canonical registered object per file path.
+ *      The path-only \ref AQNWB::NWB::RegisteredType::create "create" overload
+ *      resolves the object's on-disk ``namespace`` and ``neurodata_type`` before
+ *      reusing a cache entry. A request whose registered type conflicts with an
+ *      existing cache entry fails rather than returning a different type or
+ *      creating a second object for the same path.
  *
  * 5. **Class Name and Namespace Retrieval**:
  *    - The \ref AQNWB::NWB::RegisteredType::getTypeName "getTypeName" and
@@ -339,6 +345,13 @@
  * with the type already set. The same approach is also applied in the case of \ref AQNWB::NWB::Data "Data" and its derived
  * class \ref AQNWB::NWB::DataTyped "DataTyped".
  *
+ * ``DataTyped<DTYPE>``, ``VectorDataTyped<DTYPE>``, and
+ * ``NWBDataTyped<DTYPE>`` are typed read facades, not canonical registered
+ * objects. They are intentionally not stored in
+ * \ref AQNWB::IO::RecordingObjects "RecordingObjects"; use their specialized
+ * ``create`` or ``from...`` methods to obtain a typed view of a dataset without
+ * changing the canonical object cached for its path.
+ *
  * For further details and alternative approaches for implementing templated \ref AQNWB::NWB::RegisteredType "RegisteredType"
  * classes, see \ref implement_templated_registered_type.  
  *

docs/pages/devdocs/record_design.dox

@@ -18,7 +18,7 @@
  * AqNWB records datasets efficiently via \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects. The main components involved in
  * writing data to an NWB file via AqNWB are:
  *
- * 1. \ref DEFINE_DATASET_FIELD Macro
+ * 1. \ref DEFINE_DATASET_FIELD Macro  and \ref AQNWB::NWB::RegisteredType "RegisteredType"
  *   - The \ref DEFINE_DATASET_FIELD macro not only defines methods for reading datasets for a particular neurodata_type (as described in \ref read_design_page),
  *     but also defines methods for retrieving \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects that are used for recording to individual datasets.
  *     For each dataset field defined with this macro, a corresponding method is generated that returns a \ref AQNWB::IO::BaseRecordingData "BaseRecordingData"
@@ -31,13 +31,14 @@
  *   - It provides methods for writing data blocks to the dataset, such as \ref AQNWB::IO::BaseRecordingData::writeDataBlock "writeDataBlock",
  *      which can handle different data types and dimensions.
  *
- * 3. \ref AQNWB::NWB::RegisteredType "RegisteredType"
- *   - \ref AQNWB::NWB::RegisteredType "RegisteredType" maintains a cache of \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects via the `m_recordingDataCache` member.
+ * 3. \ref AQNWB::IO::BaseIO::m_recordingDataCache "BaseIO::m_recordingDataCache"
+ *   - \ref AQNWB::IO::BaseIO "BaseIO" maintains a cache of \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" objects that belong to the I/O object/file.
  *     This cache allows reusing the same \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" object when it is requested multiple times,
  *     improving performance and retaining the recording position. The cache is essential for writing data to the dataset in a streaming fashion,
  *     as it ensures that each write continues from where the previous write left off. The cache also avoids the need for manually maintaining
  *     the objects and allows caching of an arbitrary number of \ref AQNWB::IO::BaseRecordingData "BaseRecordingData" object such that the 
- *     individual neurodata_type classes do not need to worry about maintaining their recording state.
+ *     individual RegisteredType classes (i.e., neurodata_types) do not need to worry about maintaining their recording state, but the state is maintained 
+ *     centrally in the I/O object they belong to.
  *
  * 4. \ref AQNWB::IO::BaseIO "BaseIO"
  *   - \ref AQNWB::IO::BaseIO "BaseIO" and its implementations (e.g., \ref AQNWB::IO::HDF5::HDF5IO "HDF5IO") are responsible for
@@ -64,6 +65,7 @@
  *                 <tr><td align="left">+ getDataSet(): BaseRecordingData</td></tr>
  *                 <tr><td colspan="2" bgcolor="lightgray"><b>Attributes</b></td></tr>
  *                 <tr><td align="left">- m_recording_objects:  std::shared_ptr&lt;RecordingObjects&gt;</td></tr>
+ *                 <tr><td align="left">- m_recordingDataCache: std::unordered_map&lt;std::string, std::shared_ptr&lt;BaseRecordingData&gt;&gt;</td></tr>
  *             </table>
  *         >
  *     ];
@@ -91,11 +93,8 @@
  *         label=<
  *             <table border="0" cellborder="1" cellspacing="0">
  *                 <tr><td colspan="2" bgcolor="lightgray"><b>RegisteredType</b></td></tr>
- *                 <tr><td colspan="2" bgcolor="lightgray"><b>Functions</b></td></tr>
- *                 <tr><td align="left">+ clearRecordingDataCache(): void</td></tr>
  *                 <tr><td colspan="2" bgcolor="lightgray"><b>Attributes</b></td></tr>
- *                 <tr><td align="left">+ m_recordingDataCache: std::unordered_map</td></tr>
- *                 <tr><td align="left">+ m_io: std::shared_ptr&lt;BaseIO&gt;</td></tr>
+ *                 <tr><td align="left">+ m_io: std::weak_ptr&lt;BaseIO&gt;</td></tr>
  *                 <tr><td align="left">+ m_path: std::string</td></tr>
  *             </table>
  *         >
@@ -119,7 +118,8 @@
  *
  *     RegisteredType -> Container [arrowhead=empty, style=dashed];
  *     Container -> BaseRecordingData [label="created by functions defined via \nDEFINE_DATASET_FIELD"];
- *     RegisteredType -> BaseRecordingData [label="caches"];
+ *     HDF5IO -> BaseRecordingData [label="caches"];
+ *     HDF5IO -> RegisteredType [label="caches"];
  *     BaseRecordingData -> HDF5IO [label="uses for\nwriting data"];
  *     HDF5IO -> NWBFile [label="write data"];
  * }
@@ -197,7 +197,6 @@
  *   \ref AQNWB::IO::RecordingObjects::addRecordingObject "addRecordingObject" and 
  *   \ref AQNWB::IO::RecordingObjects::addRecordingObject "getRecordingObject" methods.
  * - **Finalization of recording objects:** via \ref AQNWB::IO::RecordingObjects::finalize "finalize"
- * - **Clearing of cached BaseRecordingData objects** via \ref AQNWB::IO::RecordingObjects::clearRecordingDataCache "clearRecordingDataCache";
  *
  * \section record_design_nwbio_utils NWB I/O convenience utilities
  *
@@ -233,9 +232,20 @@
  * facilitate reliable memory management. 
  *
  * \note
- * Registration with the \ref AQNWB::IO::RecordingObjects "RecordingObjects"  of the I/O occurs when a 
- * \ref AQNWB::NWB::RegisteredType "RegisteredType" is being created via the static
- * \ref AQNWB::NWB::RegisteredType::create "RegisteredType::create" factory methods. 
+ * Canonical \ref AQNWB::NWB::RegisteredType "RegisteredType" objects created
+ * through \ref AQNWB::NWB::RegisteredType::create "RegisteredType::create"
+ * are registered with the I/O's \ref AQNWB::IO::RecordingObjects
+ * "RecordingObjects". The cache has one canonical registered object per I/O
+ * and path; repeated compatible factory calls return that same instance. A
+ * request for a conflicting registered type fails rather than creating a
+ * second object for the path.
+ *
+ * Typed dataset facades such as \ref AQNWB::NWB::DataTyped "DataTyped",
+ * \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped", and
+ * \ref AQNWB::NWB::NWBDataTyped "NWBDataTyped" are intentionally transient
+ * and are not registered with RecordingObjects. They share the I/O's
+ * BaseRecordingData cache with the canonical object when accessing the same
+ * dataset.
  *
  * \section recording_design_further_reading Further Reading 
  *

docs/pages/devdocs/registered_types.dox

@@ -84,7 +84,10 @@
  *    DEFINE_ATTRIBUTE_FIELD(getDescription, std::string, "description", Description of the data)
  *    @endcode
  *    and in for datasets also a method for retrieving a \ref AQNWB::IO::BaseRecordingData "BaseRecordingData"
- *    wrapper used for recording data to a dataset.
+ *    wrapper used for recording data to a dataset. The \ref AQNWB::IO::BaseRecordingData "BaseRecordingData"
+ *    objects are cached by the \ref AQNWB::IO::BaseIO "BaseIO" object to ensure that the current
+ *    state of the recording process (e.g., the \ref AQNWB::IO::BaseRecordingData::getPosition "position")
+ *    are stored.
  *    @code
  *    DEFINE_DATASET_FIELD(getData, recordData, float, "data", The main data)
  *    @endcode
@@ -389,19 +392,25 @@
  * the \ref DEFINE_REGISTERED_FIELD macro, we can set the data type for read at compile time, simplifying read. 
  *
  *  \note
- *     Since the templacted  \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped<DTYPE>" does not call
- *     \ref REGISTER_SUBCLASS we must manually declare: 1) `friend class AQNWB::NWB::RegisteredType;` 
- *     and 2) `using VectorData::VectorData;` (as protected) to make sure we have access to the 
- *     constructors and define `static std::shared_ptr<VectorDataTyped> create( const std::string& path, std::shared_ptr<AQNWB::IO::BaseIO> io) {  return RegisteredType::create<VectorDataTyped>(path, io); }`
- *     to make sure we have the factory method available.
- *
+ *     Since the templated \ref AQNWB::NWB::VectorDataTyped
+ *     "VectorDataTyped<DTYPE>" does not call \ref REGISTER_SUBCLASS, it must
+ *     declare `friend class AQNWB::NWB::RegisteredType;` and inherit the
+ *     protected `VectorData` constructors with `using VectorData::VectorData;`
+ *     so its specialized factory can construct it. That factory must create a
+ *     transient typed view rather than delegate to
+ *     \ref AQNWB::NWB::RegisteredType::create
+ *     "RegisteredType::create<T>", because that factory registers canonical
+ *     objects in \ref AQNWB::IO::RecordingObjects "RecordingObjects".
+ *     `DataTyped<DTYPE>` and `NWBDataTyped<DTYPE>` follow the same transient
+ *     typed-view pattern; only their non-templated base classes are canonical
+ *     registered types.
+ *
+ *  \par
  *  \note
- *     The `std::unique_ptr<..>` template type is not covariant, i.e.,  `std::unique_ptr<DerivedClass>` does 
- *     not automatically convert to `std::unique_ptr<BaseClass>`.  I.e.,  while \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped<DTYPE>" 
- *     can be used anywhere \ref AQNWB::NWB::VectorData "VectorData" is being used, when using `std::unique_ptr<..>`
- *     we cannot rely on the compiler to automatically upcast for us, but we will need to explicitly release
- *     and upcast `std::unique_ptr<VectorDataTyped<DTYPE>` if `std::unique_ptr<VectorData>` is required. Since 
- *     defining the `DTYPE` is primarily useful for read, we therefore typically use 
+ *     AqNWB uses `std::shared_ptr` for registered and typed-view objects, so a
+ *     `std::shared_ptr<VectorDataTyped<DTYPE>>` converts to
+ *     `std::shared_ptr<VectorData>` when needed. Since defining `DTYPE` is
+ *     primarily useful for read, we therefore typically use
  *     \ref AQNWB::NWB::VectorDataTyped "VectorDataTyped<DTYPE>" on read while
  *     using \ref AQNWB::NWB::VectorData "VectorData" otherwise. 
  *
@@ -529,6 +538,7 @@
  * than the requested version. In practice, this is mainly relevant if you are generating classes for the
  * NWB `core` and `hdmf-common` namespaces.
  *
+ * \par
  * \note
  * When generating the test app via the `--generate-test-app` option we also
  * need to generate the schema headers files and save them in the `/spec` folder.