Adopt nextgen-kernels-api for kernel client management

[Zsailer]

Overview

This PR replaces the custom kernel client implementation in jupyter-server-documents with nextgen-kernels-api, a more extensive and reusable implementation of the Jupyter kernel protocol.

Key Changes

• Removed ~1,700 lines of custom kernel code (kernels/ directory and associated tests)
• Added dependency on nextgen-kernels-api >= 0.6.0
• Simplified configuration using traitlets instead of custom subclasses
• Improved message routing with filtering to exclude output messages from websocket clients while routing them to the output processor

Why nextgen-kernels-api?

The nextgen-kernels-api library aims to provide a robust, well-tested kernel client implementation that:

1. Decouples kernel API improvements from server-side documents - The long-needed improvements to Jupyter Server's kernel API can evolve independently while jupyter-server-documents depends on a stable interface
2. Enables code reuse - Other projects beyond jupyter-server-documents can leverage the same kernel client implementation
3. Reduces maintenance burden - By using upstream code, we eliminate duplicate implementations and testing

There's an open discussion about moving nextgen-kernels-api into the jupyter-ai-contrib org: https://github.com/orgs/jupyter-ai-contrib/discussions/5

Technical Details

The PR:

• Configures MultiKernelManager and KernelManager via traitlets to use DocumentAwareKernelClient
• Uses message type filtering (msg_types, exclude_msg_types) to route messages efficiently
• Maintains all existing functionality for real-time collaboration, output processing, and awareness updates
• Preserves cell execution state tracking and document synchronization

Testing

All existing tests pass. The custom kernel tests were removed as they tested implementation details now handled by nextgen-kernels-api.

---

Diff stats: 24 files changed, 294 insertions(+), 1981 deletions(-)

[Zsailer]

The integration test failure is unrelated. I'm seeing this in other places as well. It looks like an error with transitive dependencies in other places in the JupyterLab stack.

[dlqqq]

@Zsailer See #169. I was able to find a workaround to the integration test failures that are affecting our projects.