Summary
Add support for processing NDJSON (newline-delimited JSON) streams and multi-document YAML files.
Proposed Behavior
NDJSON / JSON Lines (--slurp / -s)
# Process each line independently (default streaming mode)
echo '{"name":"alice"}
{"name":"bob"}' | gq '{ name }'
# Slurp all lines into an array, then query
echo '{"name":"alice"}
{"name":"bob"}' | gq -s '{ name }'
# Output: [{"name":"alice"}, {"name":"bob"}]
Multi-document YAML
# Process multi-document YAML (--- separated)
kubectl get pods -o yaml | gq '{ items { metadata { name, namespace } } }'
Streaming mode
# Process a continuous stream without buffering everything into memory
tail -f log.json | gq '{ level, message }'
Motivation
NDJSON is ubiquitous in real-world pipelines — application logs, API pagination, kubectl output, database exports, and event streams all commonly use this format. This is table-stakes for CLI data tools. Without it, users can't use gq in many of the most common data processing scenarios.
Considerations
- Should streaming be the default for NDJSON, with
--slurp to collect into an array?
- How does multi-document YAML interact with the output format? (one doc per result vs. merged)
- Memory implications: streaming mode should process one document at a time without buffering
- Error handling: should one malformed line/document abort the whole stream or skip and continue?
Summary
Add support for processing NDJSON (newline-delimited JSON) streams and multi-document YAML files.
Proposed Behavior
NDJSON / JSON Lines (
--slurp/-s)Multi-document YAML
Streaming mode
Motivation
NDJSON is ubiquitous in real-world pipelines — application logs, API pagination,
kubectloutput, database exports, and event streams all commonly use this format. This is table-stakes for CLI data tools. Without it, users can't use gq in many of the most common data processing scenarios.Considerations
--slurpto collect into an array?