refactor(streaming)!: drop streaming field from config

[Pouyanpi]

Description

streaming is enalbed per request:
via stream_async method
via chat CLI (--streaming)
(we still support streaming when StremingHandler is used with generate_async and generate_events_async for backward compatibility)

the error message:

nemoguardrails.exceptions.StreamingNotSupportedError: Cannot use --streaming with config `./examples/bots/abc` because output rails are configured but streaming is not enabled for them.

To fix this, either:
  1. Enable streaming for output rails by adding to your config.yml:
     rails:
       output:
         streaming:
           enabled: True

  2. Or run without the --streaming flag:
     nemoguardrails chat ./examples/bots/abc

Documentation:

#1542

[codecov]

Codecov Report

❌ Patch coverage is 57.69231% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
nemoguardrails/cli/chat.py	43.75%	9 Missing ⚠️
nemoguardrails/rails/llm/llmrails.py	83.33%	1 Missing ⚠️
nemoguardrails/server/api.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Overview

Greptile Summary

This PR refactors streaming functionality from a static configuration field to a per-request capability, removing the deprecated streaming field from RailsConfig.

Key Changes:

• Removed RailsConfig.streaming field and streaming_supported property from nemoguardrails/rails/llm/config.py:1376
• Removed main_llm_supports_streaming flag from LLMRails class
• Streaming is now configured at call-site via stream_async() method, which calls _configure_main_llm_streaming() on-demand at nemoguardrails/rails/llm/llmrails.py:1226
• Improved error handling in CLI to catch InvalidRailsConfigurationError and provide helpful setup instructions at nemoguardrails/cli/chat.py:95
• Simplified server API streaming logic to directly use stream_async() instead of generate_async() with streaming handler at nemoguardrails/server/api.py:429
• Changed stream_usage to always be enabled (was conditional on config.streaming) at nemoguardrails/rails/llm/llmrails.py:379
• Updated exception types from ValueError to InvalidRailsConfigurationError for streaming validation errors
• Comprehensive test updates removing all references to the deprecated streaming field

Impact:

• Breaking change: Users with streaming: True in config.yml will need to remove this field
• Streaming is now enabled by passing --streaming flag to CLI or stream: true in API requests
• More flexible: Each request can independently choose streaming without config changes

Confidence Score: 4/5

• This PR is safe to merge with careful attention to the breaking change and error message handling
• The refactoring is well-implemented with comprehensive test coverage and improved error messages. The score is 4 (not 5) due to one minor concern: the error handling in nemoguardrails/cli/chat.py:95 catches exceptions generically and then re-checks the message string, which could be fragile if error messages change
• Pay attention to nemoguardrails/cli/chat.py for the string-based error message matching logic

Important Files Changed

File Analysis

Filename	Score	Overview
nemoguardrails/cli/chat.py	4/5	Improved error handling for streaming with better user-facing error messages; removed early streaming validation check
nemoguardrails/rails/llm/config.py	5/5	Removed deprecated streaming field and streaming_supported property from RailsConfig
nemoguardrails/rails/llm/llmrails.py	4/5	Moved streaming configuration to stream_async() call site; removed main_llm_supports_streaming flag; stream_usage now always enabled
nemoguardrails/server/api.py	4/5	Simplified streaming API to use stream_async() directly instead of generate_async() with streaming_handler

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI/API
    participant LLMRails
    participant Config
    participant LLM

    Note over User,LLM: Before: Streaming configured in config.yml
    User->>CLI/API: Request with streaming flag
    CLI/API->>Config: Check streaming_supported
    Config-->>CLI/API: Return boolean
    CLI/API->>LLMRails: Check main_llm_supports_streaming
    alt streaming not supported
        CLI/API->>User: WARNING: fallback to normal mode
    else streaming supported
        CLI/API->>LLMRails: generate_async(streaming_handler)
        LLMRails->>LLM: Generate with streaming
        LLM-->>CLI/API: Stream chunks
    end

    Note over User,LLM: After: Streaming enabled per request
    User->>CLI/API: Request with streaming flag
    CLI/API->>LLMRails: stream_async(messages)
    LLMRails->>LLMRails: _validate_streaming_with_output_rails()
    alt output rails without streaming config
        LLMRails-->>CLI/API: Raise InvalidRailsConfigurationError
        CLI/API-->>User: Helpful error message with fix instructions
    else valid configuration
        LLMRails->>LLMRails: _configure_main_llm_streaming(llm)
        LLMRails->>LLM: Set streaming=True attribute
        LLMRails->>LLMRails: generate_async(streaming_handler)
        LLM-->>CLI/API: Stream chunks
        CLI/API-->>User: Display chunks
    end

Loading

[greptile-apps]

Additional Comments (7)

1. examples/configs/streaming/config.yml, line 17 (link)

   syntax: The streaming: True field has been removed from RailsConfig in this PR. This configuration file needs to be updated to remove this line.

2. examples/configs/llm/hf_pipeline_dolly/config.yml, line 6 (link)

   syntax: The streaming: True field has been removed from RailsConfig. This configuration file needs to be updated.

3. examples/configs/gs_content_safety/config/config.yml, line 22 (link)

   syntax: The streaming: True field has been removed from RailsConfig. This configuration file needs to be updated.

4. examples/scripts/demo_streaming.py, line 38 (link)

   syntax: The streaming: True field has been removed from RailsConfig. This example code needs to be updated to remove this line from the YAML configuration.

5. examples/scripts/demo_streaming.py, line 103 (link)

   syntax: The streaming: True field has been removed from RailsConfig. Remove this line from the YAML configuration.

6. docs/configure-rails/yaml-schema/streaming/global-streaming.md, line 1-162 (link)

   syntax: This entire documentation page describes the streaming: True field which has been removed from RailsConfig. The page needs to be either removed or completely rewritten to reflect the new streaming behavior where streaming is enabled per-request via the stream_async() method rather than through global configuration.

7. examples/notebooks/generate_events_and_streaming.ipynb, line 71 (link)

   syntax: The streaming: True field has been removed from RailsConfig. This notebook needs to be updated to remove this configuration line.

_{15 files reviewed, 7 comments}

_{Edit Code Review Agent Settings | Greptile}

[Pouyanpi]

@Pouyanpi : we can move this later to llm_call when we drop the streaming callback handler feature.

[greptile-apps]

_{15 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}