Skip to content

feat: Slurp mode and multi-document support #72

Description

@daavidrgz

Summary

Add support for processing NDJSON (newline-delimited JSON) streams and multi-document YAML files.

Proposed Behavior

NDJSON / JSON Lines (--slurp / -s)

# Process each line independently (default streaming mode)
echo '{"name":"alice"}
{"name":"bob"}' | gq '{ name }'

# Slurp all lines into an array, then query
echo '{"name":"alice"}
{"name":"bob"}' | gq -s '{ name }'
# Output: [{"name":"alice"}, {"name":"bob"}]

Multi-document YAML

# Process multi-document YAML (--- separated)
kubectl get pods -o yaml | gq '{ items { metadata { name, namespace } } }'

Streaming mode

# Process a continuous stream without buffering everything into memory
tail -f log.json | gq '{ level, message }'

Motivation

NDJSON is ubiquitous in real-world pipelines — application logs, API pagination, kubectl output, database exports, and event streams all commonly use this format. This is table-stakes for CLI data tools. Without it, users can't use gq in many of the most common data processing scenarios.

Considerations

  • Should streaming be the default for NDJSON, with --slurp to collect into an array?
  • How does multi-document YAML interact with the output format? (one doc per result vs. merged)
  • Memory implications: streaming mode should process one document at a time without buffering
  • Error handling: should one malformed line/document abort the whole stream or skip and continue?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions