You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feature: #1538
Revise streaming docs to clarify usage of stream_async(), remove
outdated global streaming config, and add CLI usage instructions.
Explain output rails streaming requirements and deprecation of
StreamingHandler. Improve examples and guidance for token usage
tracking.
Copy file name to clipboardExpand all lines: docs/configure-rails/yaml-schema/streaming/global-streaming.md
+31-50Lines changed: 31 additions & 50 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,50 +1,33 @@
1
1
---
2
-
title: Global Streaming
3
-
description: Enable streaming mode for LLM token generation in config.yml.
2
+
title: Streaming
3
+
description: Using streaming mode for LLM token generation in NeMo Guardrails.
4
4
---
5
5
6
-
# Global Streaming
6
+
# Streaming
7
7
8
-
Enable streaming mode for the main LLM generation at the top level of `config.yml`.
8
+
NeMo Guardrails supports streaming LLM responses via the `stream_async()` method. No configuration is required to enable streaming—simply use `stream_async()` instead of `generate_async()`.
9
9
10
-
## Configuration
10
+
## Basic Usage
11
11
12
-
```yaml
13
-
streaming: True
14
-
```
15
-
16
-
## What It Does
17
-
18
-
When enabled, global streaming:
12
+
```python
13
+
from nemoguardrails import LLMRails, RailsConfig
19
14
20
-
- Sets `streaming = True` on the underlying LLM model
21
-
- Enables `stream_usage = True` for token usage tracking
22
-
- Allows using the `stream_async()` method on `LLMRails`
23
-
- Makes the LLM produce tokens incrementally instead of all at once
asyncfor chunk in rails.stream_async(messages=messages):
21
+
print(chunk, end="", flush=True)
22
+
```
28
23
29
24
---
30
25
31
-
## When to Use
32
-
33
-
### Streaming Without Output Rails
34
-
35
-
If you do not have output rails configured, only global streaming is needed:
36
-
37
-
```yaml
38
-
streaming: True
39
-
```
40
-
41
-
### Streaming With Output Rails
26
+
## Streaming With Output Rails
42
27
43
-
When using output rails with streaming, you must also configure [output rail streaming](output-rail-streaming.md):
28
+
When using output rails with streaming, you must configure [output rail streaming](output-rail-streaming.md):
44
29
45
30
```yaml
46
-
streaming: True
47
-
48
31
rails:
49
32
output:
50
33
flows:
@@ -53,27 +36,15 @@ rails:
53
36
enabled: True
54
37
```
55
38
56
-
---
39
+
If output rails are configured but `rails.output.streaming.enabled` is not set to `True`, calling `stream_async()` will raise an `StreamingNotSupportedError`.
57
40
58
-
## Python API Usage
41
+
---
59
42
60
-
### Simple Streaming
43
+
##Streaming With Handler (Deprecated)
61
44
62
-
```python
63
-
from nemoguardrails import LLMRails, RailsConfig
64
-
65
-
config = RailsConfig.from_path("./config")
66
-
rails = LLMRails(config)
45
+
> **Warning:** Using `StreamingHandler` directly is deprecated and will be removed in a future release. Use `stream_async()` instead.
0 commit comments