Update schema to NWB 2.11

[oruebel]

Fix #298
Fix #306

Update schema version to the latest releases

• Updated spec/core.hpp to NWB schema 2.10 2.11
• Updated spec/hdmf_common.hpp to schema version 1.9 1.10
  • MeaningsTable.target is being changed to an object reference attribute in hdmf-common 1.10 and should be changed here accordingly before merge. See Change MeaningsTable.target from a link to an object-reference attribute hdmf-dev/hdmf-common-schema#97. Fix via 79b6fbf here and also updated on Add EventTable type #304 to fix the actual MeaningsTable class.
• Updated spec/hdmf_experimental.hpp to schema version 0.6

From #301 : Add event VectorData types

Fix #299

• Update schematype_to_aqnwb.py to facilitate generation the new types
  • Fixed indent of REGISTER_SUBCLASS call and use the proper core and hdmf-common namespace when used as included schema
  • Updated minimum Python version for resources/utils to ensure use of the latest versions of PyNWB with uv
  • Updated cpp code generation to use new std::weak_ptr via getIO instead of the previous m_io shared pointer
• Implement the new TimestampVectorData and DurationVectorData types:
  • Implemented nwb/event/TimestampVectorData.hpp/cpp
  • Implemented nwb/event/DurationVectorData.hpp/cpp
  • Added unit tests for TimestampVectorData
  • Added unit tests for DurationVectorData
• Update and test that DynamicTable properly supports the new column types:
  • Updated DynamicTable.readColumn using type traits to to support reading of both: i) VectorData columns with a specific data type and ii) columns that are a subtype of VectorData (e.g., TimestampVectorData and DurationVectorData)
  • Added override for DyanmicTable.addColumn to allow adding of fully configured VectorData columns and derived column types
  • Added unit test to ensure TimestampVectorData and DurationVectorData can be added viaDynamicTable.addColumn and read back via DynamicTable.readColumn
• Other changes:
  • Updated the CHANGELOG

From #302 : Add MeaningsTable

Fix #284

Implement new MeaningsTable type:

• Implement the new src/nwb/hdmf/table/MeaningsTable.* source for MeaningsTable
• Implement comprehensive unit tests for MeaningsTable

Update DynamicTable to support adding MeaningsTables:

• Added "DEFINE_UNNAMED_REGISTERED_FIELD(" to create readMeaningsTable for reading MeaningsTables associated with a DynamicTable. This also creates createMeaningsTableInstance methods for creatin a new MeaningsTables that is not yet initialized.
• Added DynamicTable.createMeaningsTable methods for creating a fully initialized MeaningsTable that is associated with a particular column of the DynamicTable
• Add unit tests to test the new DynamicTable.createMeaningsTable and DynamicTable.readMeaningsTable methods.
  - [ ] Update the docs/tutorials as necessary to describe the use of MeaningsTable alongside DynamicTable This will be part of follow-up once the full stack for EventsTable has been added

Other Changes

• Updated ElectrodesTable.initialize to add a new optional parameter rowChunkSize. Similar to the MeaningsTable the existing ElectrodesTable also creates pre-defined columns in initialize. Currently the chunking was set in ElectrodesTable to 1, which is too small. Adding rowChunkSize increases the default to 100 and allows the caller to set the chunk size.
• Updated the CHANGELOG

From #304 : Add EventsTable type

Fix #285

Add support for EventsTable

• Added new EventTable type
• Added unit tests for the new EventTable type
• Updated NWBFile to create events/ group and added NWBFile::createEventsTable convenience function

Enhance DynamicTable to improve support for use during acquisition

• Refactored DynamicTable so it creates and initializes its ElementIdentifiers id column internally during initialize(). This is to ensure the ElementIdentifiers are always create before we start the recording. Calling DynamicTable::setRowIDs() previously required creation of ElementIdentifiers but may be called later during recording. As part of this change, setRowIDs() now takes only the row ID values and writes them directly to the table's built-in id column instead of accepting a separate ElementIdentifiers object. In practice this aligns the API with the intended usage, since callers should have always used the table's own id dataset. (@oruebel, #302)
• Updated DynamicTable column-name management to be immediate and table-driven. addColumnName() now flushes new column names to the file as columns are added, setColNames() is now restricted to reordering the existing set of columns (it no longer acts as a general overwrite mechanism), and finalize() no longer writes the colnames attribute because column-name updates are persisted when they occur. This ensures that the colnames attribute is always consistent with the actual columns in the table and helps avoid creation of invalid files. (@oruebel, #304)

Other

• Fix bugs in TimestampVectorData and DurationVectorData; e.g, bad namespace and inconsistent ordering of parameters in initialize
• Update Changelog

From #305 : Row-based Recordings

Fix #303

Problem: The current implementation of DynamicTable is mainly designed for the purpose of metadata tables, e.g., ElectrodeTable, where we can usually fill in the whole table before the actual acquisition. With the new EventTable, however, we need enhanced support for acquiring the data in the table as part of the acquisition workflow.

Goal: Update DynamicTable and EventTable to simplify row-based recording and configuration of the table

Changes:

• Support configuration of the columns of a DynamicTable directly via initialize, rather than just incremental viaaddColumn
  • Added Data::DataSpecBase and Data::DataSpec<T> to support runtime configuration of DynamicTable column layouts and to decouple table schema definition from object initialization. Each Data subclass (e.g., VectorData, ElementIdentifiers, TimestampVectorData, DurationVectorData) now exposes a nested DataSpec struct that bundles the dataset configuration, description, and any type-specific parameters needed to create and initialize the column.
  • Added DynamicTable::createDefaultDataSpecs static factory to return the default ordered list of DataSpec objects for a table (currently just the id column). Subclasses (ElectrodesTable, EventsTable, MeaningsTable) override this to append their own required columns, allowing callers to retrieve, inspect, and customize the default column configuration before passing it to initialize().
  • Updated DynamicTable::initialize, ElectrodesTable::initialize, EventsTable::initialize, and MeaningsTable::initialize to accept a std::vector<DataSpecPtr> column-spec list instead of individual rowChunkSize / resolution / flag parameters.
  • Added DynamicTable::addColumn(DataSpecPtr) overload consistent with the new DataSpec-based API used by initialize() and createDefaultDataSpecs().
  • Added DyanmicTable::validateDataSpecs (and DynamicTable::checkRequiredColumnNames helper function) to validate column specifications as part of DyanmicTable::initialize before initialization. Also updated subclassed of DynamicTable (e.g., ElectrodesTable, EventsTable, MeaningsTable) to override validateDataSpecs to provide their specific validation logic.
• Support row-based write of DynamicTable across all columns of the table
  • Added DynamicTable::addRow and DynamicTable::addRows methods to support row-based data insertion into a DynamicTable. Rows are specified as DynamicTable::RowData (an unordered_map from column name to CellValue variant), enabling type-safe, column-keyed writes without requiring callers to manage per-column buffers directly. Row IDs are auto-generated if not provided.
  • Added BaseDataType::BaseDataVariant scalar variant type and BaseDataType::createEmptyVectorVariant helper to support runtime-typed single-cell and buffer operations needed by addRow/addRows. (@cline, @oruebel, #305)
• Update Documentation
  • Added tutorial to demonstrate acquisition workflow for DynamicTable. The tutorial uses EventsTable as an example and shows: 1) row-based appending of data and 2)column-based write and addition of full columns to a table, and 3) adding of a MeaningsTable to a columns
  • Updated the CHANGELOG
• Created Follow-up issues
  • Appending rows for reference columns is not yet working. See Support appending rows and DataSpec configuration for reference columns in DynamicTable #313
  • Update schema_to_aqnwb.py utility to follow the new structure for DynamicTable. See Update schema_to_aqnwb.py utility to follow the new structure for DynamicTable #314

From #325 : Add TimeIntervals

Fix #323
Fix #324

• Added new src/nwb/epoch/TimeInterval for annotating time intervals
• Added new src/nwb/hdmf/table/VectorIndex for storing index columns
• Added VectorData::appendData and VectorIndex::appendData to simplify appending values to columns
• Moved implementation in VectorData.hpp to the source VectorData.cpp file
• Moved definition of CellValue and RowDatas to `Types to avoid circular dependency
• Enhance CellValue to support both scalar and vector data for support of ragged array columns
• Updated DynamicTable::addRows etc to support ragged array column write
• Added DynamicTable::toString to support converting rows to a string representation
• Added DynamicTable::getNumberOfRows to support retrieving the number of rows in a DynamicTable.
• Added unit tests for VectorIndex
• Added unit tests for Timeintervals
• Added DynamicTable::readRows to support reading of full rows. This also moved CellValue and RowData to Types.hpp. It also added VectorData::readCellValues and VectorIndex::readTargetCellValues.
• Updated the user tutorial on reading data to specifically discuss reading from DynamicTable
• Updated NWBFile to support the standard intervals tables, e.g., trials, invalid_time, epochs
• Updated CHANGELOG
• Added tutorials for the new TimeIntervals type in docs/pages/userdocs/time_intervals.dox and corresponding new tests/examples/testTimeIntervalsWorkflowExamples.cpp
• Updated lint.yml workflow to install clang-format via pipx to ensure use of the latest version of clang-format
• Updated Types.hpp to use a namespace instead of class to group types. For more users this change should not require any code changes since the syntax for accessing the types is the same (e.g., AQNWB::Types::Status), but for users who were using using declarations to import the type can now use using namespace AQNWB::Types.

Additional changes from #328

Fix #327.

• Updated RegisteredType::create to retrieve objects from the RecordingObjects cache of the I/O if they already exist, rather than creating a new instance. This ensures that multiple requests for the same object path return the exact same object instance.
• Moved BaseRecordingData cache from RegisteredType to BaseIO, which also ensures that there is only one BaseRecordingObject per Dataset path+io. This makes the caching consistent between objects and datasets by moving all the caches to the I/O and provides stronger consistency in management of objects for users.
• These changes to caching also allowed us to remove VectorIndex::setTargetColumn from Add time intervals #325. This logic was problematic as it: i) exposed internal logic to avoid duplicate objects to the user and ii) required DynamicTable to updated caches in VectorIndex. By centralizing all the object caches this is no longer necessary.

Future tasks

• Add support for multi-ragged table columns #326 : follow-up issue to add support for multi-ragged columns in DynamicTable

From #320 : Add Subject

Fix #290

nwb-inspector upgraded check_subject_exists to a warning/error for DANDI compliance. AqNWB did not yet support Subject and was silently ignoring this check in CI via --ignore=check_subject_exists.

Changes

• Added Subject class to represent the /general/subject group in NWB files.
• Added Subject::SubjectSpec to configure a Subject and facilitate initialization also via NWBFile.
• Updated nwbinspector tests to remove --ignore=check_subject_exists option to require subject.
• Updated NWBFile::initialize to accept a SubjectSpec argument for subject metadata initialization

[Copilot]

Pull request overview

This PR updates the embedded, auto-generated NWB/HDMF schema specifications used by AqNWB to align with the latest upstream releases referenced in Issue #298.

Changes:

• Updated the NWB core schema spec to version 2.10.0, including new/updated schema components (e.g., nwb.event and updates under nwb.file, nwb.misc, nwb.image, nwb.behavior).
• Updated the HDMF common schema spec to version 1.9.0, including adding the resources spec (HERD external resources tables) under hdmf-common.
• Updated the HDMF experimental schema spec to version 0.6.0, adjusting the namespace contents accordingly.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
src/spec/core.hpp	Bumps core schema to NWB 2.10.0 and incorporates new/updated core spec content (including nwb.event and updated NWBFile structure).
src/spec/hdmf_common.hpp	Bumps HDMF common schema to 1.9.0 and adds the resources specification (HERD) to the namespace.
src/spec/hdmf_experimental.hpp	Bumps HDMF experimental schema to 0.6.0 and updates namespace schema sources accordingly.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.52484% with 299 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.66%. Comparing base (c092a77) to head (7f71882).

Files with missing lines	Patch %	Lines
src/nwb/hdmf/table/DynamicTable.cpp	83.78%	71 Missing ⚠️
src/nwb/NWBFile.cpp	60.24%	66 Missing ⚠️
src/nwb/hdmf/table/VectorIndex.cpp	84.32%	42 Missing ⚠️
src/nwb/hdmf/base/Data.cpp	68.83%	24 Missing ⚠️
src/nwb/file/Subject.cpp	75.00%	21 Missing ⚠️
src/nwb/hdmf/table/VectorData.cpp	82.64%	21 Missing ⚠️
src/nwb/RegisteredType.cpp	55.55%	8 Missing ⚠️
src/io/BaseIO.hpp	82.05%	7 Missing ⚠️
src/nwb/RegisteredType.hpp	53.84%	6 Missing ⚠️
src/nwb/event/DurationVectorData.cpp	80.64%	6 Missing ⚠️
... and 9 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #300      +/-   ##
==========================================
- Coverage   84.69%   83.66%   -1.03%     
==========================================
  Files          57       71      +14     
  Lines        2699     4280    +1581     
  Branches      347      624     +277     
==========================================
+ Hits         2286     3581    +1295     
- Misses        413      699     +286     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

🟡 Changes recommended

Ragged-table reopening and ID generation can corrupt tables, while DataSpec validation permits schema-invalid datasets.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 87/88 changed files
• Comments generated: 17
• Review effort level: Balanced