query: potential candidate for query format: jsonpath #2924

reschke · 2024-10-20T11:26:37Z

Spec: https://www.rfc-editor.org/rfc/rfc9535.html

how to augment the existing media type definition for QUERY (do we need to?)
what response format to use? (application/json)?

timbray · 2024-10-20T15:04:11Z

If I'm building an API around the new QUERY method, surely I'd have to specify the media types for both query and response, right? So I'm not sure what's special about JSONPath.

reschke · 2024-10-20T16:54:45Z

The theory is that for a media type to be applicable for QUERY the semantics for QUERY should be specified somewhere. That idea dates back to the time when people just used application/json as payload format for PATCH, with different ideas what that actually means.

But I agree that the semantics for JsonPath with QUERY are pretty clear, maybe except for the question what the JSON document being queried actually is. But that can remain an implementation detail of the server. Any recommendations about what format we should use in the response? Maybe you even could suggest an example?

timbray · 2024-10-20T17:35:52Z

I disagree at a basic level. A JSONPath selects the parts of some JSON text that it matches. What the JSON text is, and what the effect of selecting parts of it is, seem entirely application-dependent. Maybe it’s applied to the user’s order history and forwards the selected portions to Accounting, and returns status code 200 but no body. Maybe it applies the JSONPath to that same order history, creates an analysis, and returns application/csv. These things are designed to be general-purpose expressions and I really can’t think of anything useful that could be said. -T

…

On Oct 20, 2024 at 9:55:06 AM, Julian Reschke ***@***.***> wrote: The theory is that for a media type to be applicable for QUERY the semantics for QUERY should be specified somewhere. That idea dates back to the time when people just used application/json as payload format for PATCH, with different ideas what that actually means. But I agree that the semantics for JsonPath with QUERY are pretty clear, maybe except for the question what the JSON document being queried actually is. But that can remain an implementation detail of the server. Any recommendations about what format we should use in the response? Maybe you even could suggest an example? — Reply to this email directly, view it on GitHub <#2924 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEJEZHO73S3LG3L5MQYVLZ4PN6VAVCNFSM6AAAAABQINSZYOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRVGEYTOMJWHE> . You are receiving this because you commented.Message ID: ***@***.***>

reschke · 2024-10-20T19:34:09Z

Ok, just checking: we don't want to define a standard response format, because we want to leave it specific to the server? I agree with that.

But for an example, would you recommend a specific response format (iff that this is just an example)?

timbray · 2024-10-20T21:52:03Z

That example of sending a JSONPath which is applied to the user's saved data and returns an analysis in a CSV isn't terrible? Or it might return a nice graph as a JPEG or PDF.

…

On Sun, Oct 20, 2024, 12:34 PM Julian Reschke ***@***.***> wrote: Ok, just checking: we don't want to define a standard response format, because we want to leave it specific to the server? I agree with that. But for an example, would you recommend a specific response format (iff that this is just an example)? — Reply to this email directly, view it on GitHub <#2924 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEJEZ3XJOF74XEEAEY7ADZ4QAULAVCNFSM6AAAAABQINSZYOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRVGE4DSMZXGU> . You are receiving this because you commented.Message ID: ***@***.***>

mnot · 2024-10-20T23:06:50Z

I think the difference here is that PATCH has side effects on the resource, whereas QUERY should not (generally). PATCH needed definition to describe those side effects, so it doesn't become another POST escape hatch.

reschke · 2024-10-21T03:17:13Z

Ok, so we'll just add an example using jsonpath.

reschke · 2024-11-01T20:34:23Z

Proposal:

QUERY /errata.json HTTP/1.1
Host: example.org
Content-Type: application/jsonpath
Accept: application/json

$..[?@doc-id=="RFC9110" && @errata_status_code!="Rejected"].[errata_id, submit_date]

from a fictious RFC errata database, selects all errata for RFC9110 which have not been rejected, and returns id, submission_date for each

HTTP/1.1 200 OK
Content-Type: application/json

[
  "7105",
  "2022-08-26",
  "7109",
  "2022-08-31",
  "7138",
  "2022-09-23",
  "7306",
  "2023-01-13",
  "7419",
  "2023-04-11"
]

timbray · 2024-11-01T21:06:22Z

On Nov 1, 2024 at 1:34:45 PM, Julian Reschke ***@***.***> wrote: Content-Type: example/query

Accept: application/jsonpath

That should be Content-Type: application/jsonpath, Accept: application/json, no? Proposal:

…

QUERY /errata.json HTTP/1.1 Host: example.org Content-Type: example/query Accept: application/jsonpath ***@***.***=="RFC9110" && @errata_status_code!="Rejected"].[errata_id, submit_date] from a fictious RFC errata database, selects all errata for RFC9110 which have not been rejected, and returns id, submission_date for each HTTP/1.1 200 OK Content-Type: application/json [ "7105", "2022-08-26", "7109", "2022-08-31", "7138", "2022-09-23", "7306", "2023-01-13", "7419", "2023-04-11" ] — Reply to this email directly, view it on GitHub <#2924 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEJE5DR5WYV4B73HCY3PDZ6PQWLAVCNFSM6AAAAABQINSZYOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJSGU2TCMZQGE> . You are receiving this because you commented.Message ID: ***@***.***>

reschke · 2024-11-01T23:10:42Z

Oh, indeed. Thanks for catching this. Will edit in place.

reschke · 2024-11-03T15:41:21Z

We could also use this as example of conneg:

HTTP/1.1 200 OK
Content-Type: text/csv

errata_id, submit_date
7105, 2022-08-26
7109, 2022-08-31
7138, 2022-09-23
7306, 2023-01-13
7419, 2023-04-11

mnot · 2024-11-17T11:00:25Z

Part of #1906

cabo · 2024-11-20T22:42:24Z

I agree with using Accept: to specify the desired response format (we could even give application/json a specific meaning in this context); application/jsonpath is fine as a query content type.

reschke · 2025-02-19T20:05:16Z

@timbray @cabo - I need help with getting the syntax correct. What tool should I use? Is https://jsonpath.com/ ok? FWIW, the sample data is https://www.rfc-editor.org/errata.json

reschke · 2025-02-19T20:43:16Z

-> #1906

cabo · 2025-02-19T20:56:58Z

@timbray @cabo - I need help with getting the syntax correct. What tool should I use? Is https://jsonpath.com/ ok? FWIW, the sample data is https://www.rfc-editor.org/errata.json

I use a simple tool called jpt (gem install jpt). @glyn probably has more recommendations...
(See also his blog article https://underlap.org/jsonpath-rfc-nearing-publication .)

reschke · 2025-02-20T08:29:58Z

jpt seems to use the same parser as https://jsonpath.com/...

I should have been more precise on what problems I encountered:

Escaping "-" in names. The query

$..[[email protected]=='RFC4954']

results in parse error:

Expected one of [A-Z], [a-z], "_", [�-퟿], [-�], [0-9], " ", "\t", "\n", "\r", "[", ".", "==", "!=", "<=", ">=", "<", ">", "..", "&&", "||", ",", "]" at line 1, column 11 (byte 11):

The other question was how to specify multiple members for output; in the original example I had:

$..[?@doc-id=="RFC9110" && @errata_status_code!="Rejected"].[errata_id, submit_date]

Is this even possible in regular JSONPath?

martinthomson · 2025-02-20T08:40:16Z

Here's what I found works if I altered their example: $["doc-id"] or $["doc\u002did"]

That is, if you want "-", you have to not use the .<label> syntax, but instead ["<label>"], using JSON-style quoting in that string.

cabo · 2025-02-20T10:19:28Z

jpt seems to use the same parser as https://jsonpath.com/...

I should have been more precise on what problems I encountered:

Escaping "-" in names. The query

$..[[email protected]=='RFC4954']

@["doc-id"] is a way to include other characters in a name.

The other question was how to specify multiple members for output; in the original example I had:

$..[?@doc-id=="RFC9110" && @errata_status_code!="Rejected"].[errata_id, submit_date]

Is this even possible in regular JSONPath?

We didn't include projection (the SELECT... part of SQL) in JSONPath.
You get pointers into an instance, or the values at these points.
No construction of new values.

reschke · 2025-02-20T10:34:10Z

I now have

http-extensions/draft-ietf-httpbis-safe-method-w-body.xml

Line 868 in 7383116

<t>

Do you have a good idea how to produce JSONPath results for the response body? Is there a notation I could use? A list of normalized paths perhaps?

cabo · 2025-02-20T23:45:37Z

JSONPath does not specify an output format, just saying that this could be a list of normalized paths and/or the values at these paths. The reply at line 896 looks good to me as an example for representing the latter list as a JSON array.

Please do fix line 884 ["doc\u002did"]; I have no idea what made Martin escape the hyphen-minus, which is perfectly valid ASCII at this point.
Also, the line breaking and indentation of the query is quite weird.

reschke · 2025-02-21T05:15:55Z

Please try. "doc-id" is not accepted by the parser: #2924 (comment)

For the query, I wanted to indent it in order to avoid overflowing the rfc txt limits. If you can suggest a better variant, please go ahead.

cabo · 2025-02-21T05:57:22Z

   name-segment        = ("[" name-selector "]") /
                         ("." member-name-shorthand)
   member-name-shorthand = name-first *name-char
   name-selector       = string-literal
   string-literal      = %x22 *double-quoted %x22 /     ; "string"
                         %x27 *single-quoted %x27       ; 'string'

hyphen-minus is not a name-char, so a member-name-shorthand like

@.doc-id

is a syntax error, as is

@.doc\u002did

Instead,

@["doc-id"]

or

@['doc-id']

is the correct way to use this member name in a name-segment.
The string-literals inside a name-selector also allow escapes like \u002d, but there is no point for hyphen-minus.

reschke · 2025-02-21T06:00:52Z

That's why I was so confused. Let me try again.

Indeed:

$..[[email protected]_status_code=="Rejected" && @["doc-id"]=="RFC9110"]

does work. I believe I tried

$..[[email protected]_status_code=="Rejected" && @.["doc-id"]=="RFC9110"]

Sorry for the confusion

glyn · 2025-02-24T15:12:13Z

@timbray @cabo - I need help with getting the syntax correct. What tool should I use? Is https://jsonpath.com/ ok? FWIW, the sample data is https://www.rfc-editor.org/errata.json

I use a simple tool called jpt (gem install jpt). @glyn probably has more recommendations... (See also his blog article https://underlap.org/jsonpath-rfc-nearing-publication .)

Sorry for the slow reply. Here are two "live" implementations of RFC 9535 which you may already have gotten from my blog post:

reschke · 2025-02-24T20:33:45Z

See #3002

reschke · 2025-02-24T20:34:29Z

@glyn - thanks; I just merged the new examples section. Maybe you want to check?

glyn · 2025-02-25T12:10:26Z

@reschke:

@glyn - thanks; I just merged the new examples section. Maybe you want to check?

I skimmed through #3002, but wasn't really sure what which JSONPath query expressions you wanted me to check. If you can drop them here, I'll take a look.

reschke · 2025-02-25T23:20:26Z

---> https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-08.html#name-more-query-formats

glyn · 2025-02-26T04:47:06Z

$..[
     [email protected]_status_code=="Rejected"
     && @.submit_date>"2024"
   ]
   ["doc-id"]

Confirmed that this is valid JSONPath syntax.

reschke added the query-method label Oct 20, 2024

mnot added the editorial label Nov 17, 2024

reschke closed this as completed Feb 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

query: potential candidate for query format: jsonpath #2924

query: potential candidate for query format: jsonpath #2924

reschke commented Oct 20, 2024 •

edited

Loading

timbray commented Oct 20, 2024

reschke commented Oct 20, 2024

timbray commented Oct 20, 2024 via email

reschke commented Oct 20, 2024

timbray commented Oct 20, 2024 via email

mnot commented Oct 20, 2024

reschke commented Oct 21, 2024

reschke commented Nov 1, 2024 •

edited

Loading

timbray commented Nov 1, 2024 via email

reschke commented Nov 1, 2024

reschke commented Nov 3, 2024 •

edited

Loading

mnot commented Nov 17, 2024

cabo commented Nov 20, 2024

reschke commented Feb 19, 2025

reschke commented Feb 19, 2025

cabo commented Feb 19, 2025

reschke commented Feb 20, 2025

martinthomson commented Feb 20, 2025

cabo commented Feb 20, 2025

reschke commented Feb 20, 2025 •

edited

Loading

cabo commented Feb 20, 2025

reschke commented Feb 21, 2025

cabo commented Feb 21, 2025

reschke commented Feb 21, 2025 •

edited

Loading

glyn commented Feb 24, 2025

reschke commented Feb 24, 2025

reschke commented Feb 24, 2025

glyn commented Feb 25, 2025

reschke commented Feb 25, 2025

glyn commented Feb 26, 2025

query: potential candidate for query format: jsonpath #2924

query: potential candidate for query format: jsonpath #2924

Comments

reschke commented Oct 20, 2024 • edited Loading

timbray commented Oct 20, 2024

reschke commented Oct 20, 2024

timbray commented Oct 20, 2024 via email

reschke commented Oct 20, 2024

timbray commented Oct 20, 2024 via email

mnot commented Oct 20, 2024

reschke commented Oct 21, 2024

reschke commented Nov 1, 2024 • edited Loading

timbray commented Nov 1, 2024 via email

reschke commented Nov 1, 2024

reschke commented Nov 3, 2024 • edited Loading

mnot commented Nov 17, 2024

cabo commented Nov 20, 2024

reschke commented Feb 19, 2025

reschke commented Feb 19, 2025

cabo commented Feb 19, 2025

reschke commented Feb 20, 2025

martinthomson commented Feb 20, 2025

cabo commented Feb 20, 2025

reschke commented Feb 20, 2025 • edited Loading

cabo commented Feb 20, 2025

reschke commented Feb 21, 2025

cabo commented Feb 21, 2025

reschke commented Feb 21, 2025 • edited Loading

glyn commented Feb 24, 2025

reschke commented Feb 24, 2025

reschke commented Feb 24, 2025

glyn commented Feb 25, 2025

reschke commented Feb 25, 2025

glyn commented Feb 26, 2025

reschke commented Oct 20, 2024 •

edited

Loading

reschke commented Nov 1, 2024 •

edited

Loading

reschke commented Nov 3, 2024 •

edited

Loading

reschke commented Feb 20, 2025 •

edited

Loading

reschke commented Feb 21, 2025 •

edited

Loading