Define fragment identifier specifications for OAS / Schema

[ioggstream]

I expect

• fragment identifier spec:
• for openapi+json / yaml
• schema+json
• +yaml, yaml moved to Fragment identifier for yaml and +yaml #21

[jdesrosiers]

This might be tricky for OAS 3.1. Prior to 3.1, it's pretty straightforward, fragments are JSON Pointers (RFC 6901). But, 3.1 adopts all of JSON Schema which means plain name fragments are allowed as well. But, how plain name fragments are defined depends on which JSON Schema draft is used.

[darrelmiller]

We might want to limit fragment identifiers in an OpenAPI document to be only OpenAPI objects. So you could reference #/components/schemas/sample but not be able to point inside the sample JSON schema.

[jdesrosiers]

References need to be able to point to sub-schemas because they can in JSON Schema and there are JSON Schemas in OpenAPI documents.

I think this document just needs to define how to identify a location. It can defer to the specific OAS/JSONSchema version to determine whether a reference to that particular location makes sense or is allowed.

[cjaccino]

This may be a bad assumption:
"fragment identifier spec for xxx+json and xxx+yaml are defined"

The JSON media type registration does not provide for fragments. It's more of an opportunity than a problem; simply stipulate that rfc6901: JSON Pointer be applied the same for YAML as it would be for JSON.

The inclusion of support for plain names could be considered as well. When +yaml is used for an object of type application/schema+yaml or similar (A JSON Schema written in yaml?), a type with anchors, the plain names could follow the customary handling. OTOH, maybe that is an issue to sort out in the registration of the type that uses the +yaml suffix.

[cjaccino]

Per @darrelmiller

We might want to limit fragment identifiers in an OpenAPI document to be only OpenAPI objects. So you could reference #/components/schemas/sample but not be able to point inside the sample JSON schema.

And per @jdesrosiers

References need to be able to point to sub-schemas because they can in JSON Schema and there are JSON Schemas in OpenAPI documents.
I think this document just needs to define how to identify a location. It can defer to the specific OAS/JSONSchema version to determine whether a reference to that particular location makes sense or is allowed.

For Openapi 3.1 and forward, we ought to be able to reference independently managed JSON Schemas, not just OpenAPI specs. And if the JSON Schema rules apply to OpenAPI documents, shouldn't they be able to reference anywhere in that document? Good practice might dictate a narrower set of patterns, but I'd hope to not create new one-off rules for OpenAPI; the move to JSON Schema was to take on its advantages, right?

[ioggstream]

@cjaccino as of now yaml and +yaml won't define a fragment and delegate this operation to a "materialized" media type.

that is an issue to sort out in the registration of the type that uses the +yaml suffix

Sure: this should be addressed in application/openapi+yaml

[ioggstream]

@jdesrosiers @darrelmiller WRT Fragment identifiers, IIUC

1. OAS (json/yaml) uses JSON Pointers
2. Schema uses JSON Pointers | Plain names
3. an OAS 3.1 cannot use Plain names since it's not a json schema
4. an OAS 3.1 can reference a Schema via JSON Pointer. The Schema can then reference another Schema via Plain names

Is that correct?

[jdesrosiers]

That's not correct. Consider this example located at file:///path/to/openapi.yaml.

components:
  schemas:
    foo:
      $anchor: foo
      type: string
    bar:
      $ref: '#foo'
      minLength: 3

In this example, #foo resolves to file:///path/to/openapi.yaml#foo. The document (file:///path/to/openapi.yaml) has the OPAS media type and has a plain name fragment. Therefore, the OAS media type needs to support plain name fragments in order to support the JSON Schema embedded within it. The fragment section for the OAS media type can simply reference the fragment section I wrote up for the JSON Schema media type. They should be identical.

[ioggstream]

I think that "Plain names" cannot be easily supported unless the media type is explicitly declared as "application/schema+" or "application/openapi+".

This means that an application producing an application/json or application/yaml document cannot be expected to parse that kind of fragment.

[darrelmiller]

That's not correct. Consider this example located at file:///path/to/openapi.yaml.

components:
  schemas:
    foo:
      $anchor: foo
      type: string
    bar:
      $ref: '#foo'
      minLength: 3

In this example, #foo resolves to file:///path/to/openapi.yaml#foo. The document (file:///path/to/openapi.yaml) has the OPAS media type and has a plain name fragment. Therefore, the OAS media type needs to support plain name fragments in order to support the JSON Schema embedded within it. The fragment section for the OAS media type can simply reference the fragment section I wrote up for the JSON Schema media type. They should be identical.

@jdesrosiers This example you show worries me. Promoting JSON Schema anchors to root level names in an OpenAPI document seems like a big risk to OpenAPI being able to introduce new keywords in the future.

[handrews]

@darrelmiller how are URI fragment and keyword namespaces related?

[darrelmiller]

@handrews OAS Reference objects can use OAS keywords as part of their URI fragment
e.g.

../someOasDocument.json#/components/responses/someResponse

Having read a bit more about the allowed values in $anchor members, it does appear that we could disambiguate between these anchors and JSON Pointers into OAS documents. However, it is concerning that JSON schema is effectively taking ownership of a big chunk of the OAS URI fragment space.

[handrews]

@darrelmiller A fragment is always either unambiguously a JSON Pointer fragment (which MUST either be the empty string or start with a /) or unambiguously a plain name fragment (which MUST be non-empty and MUST start with a character from a set of characters that excludes /).

JSON Schema's plain name fragments are derived from the WC3's Best Practices for Fragment Identifiers and Media Type Definitions, specifically:

Best Practice 3: Reserve Plain Name Fragids

If the media type includes structures that can be given local names or identifiers, plain name fragids should be reserved for addressing those structures.

and

A plain name fragid is a fragid that is used to identify a named structure within a document, such as one identified by an @id attribute in HTML, a @xml:id attribute in XML or the name of a function within a Python program. These fragids are opaque to processors and as such they do not normally include punctuation characters, though this depends on the language: in XML, for example, they usually match the NCName production from XML Namespaces [XML-NAMES11] which means they can contain hyphens (-) and periods (.).

JSON Schema uses the NCName production as it seemed like a reasonable one, particularly back when it was common for APIs to offer both JSON and XML representations (which I'm sure some still do).

So completely aside from JSON Schema, it would make sense for OpenAPI to support plain name fragments.

I guess my question is, what else would you want to do with fragments that would match the plain name / NCName syntax? There's no reason that plain name fragments couldn't be created by some part of OAS other than JSON Schema as well. There is always the possibility of defining the same fragid multiple times within the same resource, which would be a little more likely if some aspect of OAS automatically creates fragids. But it's entirely possible already (and also possible in HTML and any number of other media types).

Plus, if you really need a 3rd type of fragid, there are many ways to design a syntax that wouldn't conflict with either JSON Pointer fragments or plain name fragments.

[ioggstream]

JSON Schema's plain name fragments are derived from the WC3 BCP

Ok. I think that in this case though, this approach can be problematic since JSON Schema can be referenced inside documents of other formats. I think that OAS / JSON Schema folks should deep dive on that, if you haven't already planned it.

Moreover, if one day you may decide to dismiss plain names (e.g. maybe you decide to use something like #anchor=foo) like @cabo suggested before, I strongly suggest that you should not be hindered by current implementation since you are working on a I-D, not a published RFC. The same goes for dialects.

So completely aside from JSON Schema, it would make sense for OpenAPI to support plain name fragments.

But... with which format? For example, OAS could decide to use operationId ;)

FYI for YAML the current setup:

• will not define plain name fragments;
• will use YAML anchors if the frag starts with *
• will use JSON Pointer if the frag starts with /

My proposal for now is just to delegate the fragment processing to the relevant specs: this means that the implementation needs to identify the specific media type (e.g. OAS version, JSchema version) before processing. I think that it's a quite high price for supporting plain names, but that's up to you.

I just hope this wouldn't delay the registration too much :)
Thanks everyone for this interesting discussion, and have a nice day,
R:

[handrews]

@ioggstream

Ok. I think that in this case though, this approach can be problematic since JSON Schema can be referenced inside documents of other formats. I think that OAS / JSON Schema folks should deep dive on that, if you haven't already planned it.

JSON Schema allows embedding multiple resources in a document (JSON Schema or otherwise) based on the presence of $id, so anyone who wants to namespace the fragids can just give their embedded schema objects a $id (which is an absolute URI) and reference the fragments via that.

This is possible in OpenAPI 3.1 (the first OAS version that supports JSON Schema's $id and $anchor). People aren't used to using it, but they're not used to using $anchor either. By default, people just write JSON Schema objects, which are not embedded resources, but any JSON Schema object can be made a JSON Schema resource with $id, which becomes the resource's base URI.

If your schema objects don't set a $id, then the base URI is that of the entire document (the OAS resource), in which case you're working with a global namespace.

[handrews]

It would be entirely reasonable for OAS to warn that using $anchor in a JSON Schema object that is not the child of a JSON Schema resource risks a name collision with future OAS fragment rules, or something to that effect. It can't be turned off entirely, but the problem of conflicting user-defined fragments is not limited to JSON Schema (embedded or standalone).

[Relequestual]

That's not correct. Consider this example located at file:///path/to/openapi.yaml.

components:
  schemas:
    foo:
      $anchor: foo
      type: string
    bar:
      $ref: '#foo'
      minLength: 3

In this example, #foo resolves to file:///path/to/openapi.yaml#foo. The document (file:///path/to/openapi.yaml) has the OPAS media type and has a plain name fragment. Therefore, the OAS media type needs to support plain name fragments in order to support the JSON Schema embedded within it. The fragment section for the OAS media type can simply reference the fragment section I wrote up for the JSON Schema media type. They should be identical.

I had assumed that each JSON Schema is treated as its own "schema resource", and therefore cannot be referenced as you suggest.

Can you point to anything in the spec which supports your understanding?

[Relequestual]

My issue here is, if you did allow such a thing, you would be making any schema bundling process potentially create different validation results. What happens if bar schema had a subschema which defined a foo anchor? Which anchor would the reference then resolve to?

[cabo]

How is this discussion relevant for the YAML fragment identifier syntax?

[handrews]

@Relequestual it's in the explanation of base URIs.

The OAS document isn't a bundle, unless schema objects use $id, in which case it's the same bundling behavior as JSON Schema in general.

What happens if bar schema had a subschema which defined a foo anchor? Which anchor would the reference then resolve to?

It's exactly the same as if OAS's components was JSON Schema's $defs.

In any media type that allows document authors to define anchors, it's possible to define the same anchor twice. People do it in HTML all the time (to the continual irritation of QA departments everywhere trying to write automation using anchors as stable references).

@cabo

How is this discussion relevant for the YAML fragment identifier syntax?

OAS can be written in JSON or YAML and essentially imports JSON Schema's fragment behavior. Although plain name fragments an only be defined within schema objects (using $anchor, or also $dynamicAnchor but let's ignore that because $dynamicAnchor has issues that will most likely be fixed by making it not interact with URI fragments).

None of that imposes constraints on application/yaml but the +yaml for OAS needs to figure it out, as the fragment syntax and semantics ought to be consistent across the +json and +yaml representations.

[jdesrosiers]

@darrelmiller, were your concerns get adequately addressed?

it is concerning that JSON schema is effectively taking ownership of a big chunk of the OAS URI fragment space.

The way I see it, the OAS media-type should just define that plain-name fragments identify a named location in the document. They don't identify a schema, just a location. What is expected at that location depends on the context. JSON Schema defines the $anchor keyword to identify a named location. OAS can define their own ways to define a named location in an OpenAPI document without any conflict in the fragment space. Of course there could be name conflicts, but that's a document authoring issue, not a media-type issue.

If OAS wants to introduce additional fragment types, there are many characters that aren't allowed in plain-name fragments that can be used for disambiguation such as using "$" for JSON Path.

[handrews]

OK to summarize here for the OpenAPI media types only only (all applies to both +json and +yaml as it should for any fragment):

• OAS 3.0 and earlier only use JSON Pointer fragments, including for schema objects
• OAS 3.1 (and likely any 3.2+, but not 4.0) use JSON Pointer fragments and inherit JSON Schema's fragment requirements for Schema Objects
  • Schema Objects in OpenAPI documents that do not have an $id are part of the the main OpenAPI resource, (as described in the JSON Schema spec on embedding the format) using its base URI, therefore any JSON Schema keywords that create plain name fragments create them on the OpenAPI resource - all that needs to happen here is to delegate that to the appropriate JSON Schema rules, as determined by the value of jsonSchemaDialect in the root OpenAPI Object
  • Schema Objects with an $id are embedded schema resources per JSON Schema, and fragment-creating keywords work as they would in any JSON Schema resource
  • OpenAPI also inherits JSON Schema's caveats regarding JSON Pointer fragments that cross an $id
  • JSON Schema dialects that are also different versions (e.g. draft-07, which is widely used outside of OAS, including in AsyncAPI) can have different URI and fragment-producing keywords, although there are really only two: $id can define fragments in draft-06/07, and $id was called id in drafts older than 06)
  • OAS implementations are only required to support draft 2020-12; presumably implementations that support earlier drafts will support their URI and fragment-producing keywords

Objections that the above is not a good idea in some way are, at this point, fairly irrelevant. OAS 3.1 has been out for more than three years now, and this is how it works. OAS 3.1.1 will come out this year and clarify the referencing and identification behavior in the 3.1 line.

[handrews]

It's also worth noting that OAS 4 (a.k.a. moonwalk) will likely have a different approach to fragments, or at least have component-name-based plain name fragments or other JSON Pointer-alternative syntax.

[handrews]

See also:

• operationId as a plain name fragment #111

[handrews]

@ioggstream

I think that "Plain names" cannot be easily supported unless the media type is explicitly declared as "application/schema+" or "application/openapi+".

This means that an application producing an application/json or application/yaml document cannot be expected to parse that kind of fragment.

I don't see the problem here for application/openapi+json or application/openapi+yaml. For one thing, application/json doesn't support any fragment syntax at all, and RFC6901 goes out of its way to state that JSON Pointer is not a fragment syntax for application/json, just in case its lack of presence in the JSON RFCs since then didn't make it clear enough.

So if we're bound by application/json and application/yaml, we can't do anything with fragments. Clearly that's not workable, so we're defining fragment syntaxes for application/openapi+whatever. There's no reason plain names would be any more problematic than JSON Pointers in that sense.

• JSON, +json, and +yaml don't support any fragments, so we won't be colliding there
• YAML supports JSON Pointers (compatible!) and alias nodes (which do not syntactically collide with either JSON Pointer or plain name fragments)

So I don't see the problem. I don't expect parsers of application/json to support any kind of fragment, so parsers of application/yaml not supporting one of the two fragment syntaxes seems irrelevant. What am I missing?

[ioggstream]

Briefly:

• application/yaml uses JSON pointers and alias names
• +yaml says that it does not inherit from yaml

This means that:

With openapi+* and schema+* we can decide whatever we like

[handrews]

@ioggstream thanks! OK so I just misread where the difficulties are, and since we are defining openapi+* (and schema+*, but I'll leave JSON Schema to others) we're all clear. 🎉