Expose request serializer and response deserializer code to support pagination APIs

[edorphy]

Motivation

Like many APIs, App Store Connect API supports pagination. The ASC API in particular provides a links construct which includes 'self', 'first', and 'next' page links.

However, when using tools like Swift OpenAPI Generator, because the 'cursor' element is not provided in the spec, one cannot take the 'next' url and construct a new request.

I want to be able to use the 'next' link url and create a request, pass it to the client, and reuse the serializer and deserializer generated code. At present because the serialize and deserialize are contained within the client functions there is no way to invoke it and reuse the code.

While it would change the generated code significantly, one could expose a static function serialize and another deserialize on the operation that would enable construction and invocation of a 'next' pagination style API call.

Proposed solution

I will use one of the most basic ASC APIs, Apps Get Collection.

When calling this with a sort order by name, and a limit by 2, the API produces a 'next' link for me.

po json.links.next
https://api.appstoreconnect.apple.com/v1/apps?sort=name&cursor=<REDACTED-OPAQUE-STRING>%3D%3D&limit=2

Great, so now I need to traverse the next page, but there is no way to hit the API and benefit from all of the type safe, status-code safe, content-type and deserialization logic.

I removed a bunch of the query parameter code to reduce the sample.

/// - Remark: HTTP `GET /v1/apps`.
    /// - Remark: Generated from `#/paths//v1/apps/get(apps_getCollection)`.
    public func appsGetCollection(_ input: Operations.AppsGetCollection.Input) async throws -> Operations.AppsGetCollection.Output {
        try await client.send(
            input: input,
            forOperation: Operations.AppsGetCollection.id,
            serializer: { input in
                let path = try converter.renderedPath(
                    template: "/v1/apps",
                    parameters: []
                )
                var request: HTTPTypes.HTTPRequest = .init(
                    soar_path: path,
                    method: .get
                )
                suppressMutabilityWarning(&request)
                try converter.setQueryItemAsURI(
                    in: &request,
                    style: .form,
                    explode: false,
                    name: "filter[name]",
                    value: input.query.filterName
                )
                
                // Lots of removed code to simplify listing ...

                try converter.setQueryItemAsURI(
                    in: &request,
                    style: .form,
                    explode: true,
                    name: "limit[subscriptionGroups]",
                    value: input.query.limitSubscriptionGroups
                )
                converter.setAcceptHeader(
                    in: &request.headerFields,
                    contentTypes: input.headers.accept
                )
                return (request, nil)
            },
            deserializer: { response, responseBody in
                switch response.status.code {
                case 200:
                    let contentType = converter.extractContentTypeIfPresent(in: response.headerFields)
                    let body: Operations.AppsGetCollection.Output.Ok.Body
                    let chosenContentType = try converter.bestContentType(
                        received: contentType,
                        options: [
                            "application/json"
                        ]
                    )
                    switch chosenContentType {
                    case "application/json":
                        body = try await converter.getResponseBodyAsJSON(
                            Components.Schemas.AppsResponse.self,
                            from: responseBody,
                            transforming: { value in
                                .json(value)
                            }
                        )
                    default:
                        preconditionFailure("bestContentType chose an invalid content type.")
                    }
                    return .ok(.init(body: body))
                case 400:
                    let contentType = converter.extractContentTypeIfPresent(in: response.headerFields)
                    let body: Operations.AppsGetCollection.Output.BadRequest.Body
                    let chosenContentType = try converter.bestContentType(
                        received: contentType,
                        options: [
                            "application/json"
                        ]
                    )
                    switch chosenContentType {
                    case "application/json":
                        body = try await converter.getResponseBodyAsJSON(
                            Components.Schemas.ErrorResponse.self,
                            from: responseBody,
                            transforming: { value in
                                .json(value)
                            }
                        )
                    default:
                        preconditionFailure("bestContentType chose an invalid content type.")
                    }
                    return .badRequest(.init(body: body))
                case 401:
                    let contentType = converter.extractContentTypeIfPresent(in: response.headerFields)
                    let body: Operations.AppsGetCollection.Output.Unauthorized.Body
                    let chosenContentType = try converter.bestContentType(
                        received: contentType,
                        options: [
                            "application/json"
                        ]
                    )
                    switch chosenContentType {
                    case "application/json":
                        body = try await converter.getResponseBodyAsJSON(
                            Components.Schemas.ErrorResponse.self,
                            from: responseBody,
                            transforming: { value in
                                .json(value)
                            }
                        )
                    default:
                        preconditionFailure("bestContentType chose an invalid content type.")
                    }
                    return .unauthorized(.init(body: body))
                case 403:
                    let contentType = converter.extractContentTypeIfPresent(in: response.headerFields)
                    let body: Operations.AppsGetCollection.Output.Forbidden.Body
                    let chosenContentType = try converter.bestContentType(
                        received: contentType,
                        options: [
                            "application/json"
                        ]
                    )
                    switch chosenContentType {
                    case "application/json":
                        body = try await converter.getResponseBodyAsJSON(
                            Components.Schemas.ErrorResponse.self,
                            from: responseBody,
                            transforming: { value in
                                .json(value)
                            }
                        )
                    default:
                        preconditionFailure("bestContentType chose an invalid content type.")
                    }
                    return .forbidden(.init(body: body))
                case 429:
                    let contentType = converter.extractContentTypeIfPresent(in: response.headerFields)
                    let body: Operations.AppsGetCollection.Output.TooManyRequests.Body
                    let chosenContentType = try converter.bestContentType(
                        received: contentType,
                        options: [
                            "application/json"
                        ]
                    )
                    switch chosenContentType {
                    case "application/json":
                        body = try await converter.getResponseBodyAsJSON(
                            Components.Schemas.ErrorResponse.self,
                            from: responseBody,
                            transforming: { value in
                                .json(value)
                            }
                        )
                    default:
                        preconditionFailure("bestContentType chose an invalid content type.")
                    }
                    return .tooManyRequests(.init(body: body))
                default:
                    return .undocumented(
                        statusCode: response.status.code,
                        .init(
                            headerFields: response.headerFields,
                            body: responseBody
                        )
                    )
                }
            }
        )
    }

So in order to reuse either the serialization or deserialization code, it needs to be ripped out of this generated code. My initial thought would be to add an extension on the operation. This would be done in the generated client code, not the types code.

For brevity, I am just providing a screenshot of the model code for this API.

Excerpt from the generated Types.swift file

Proposed extensions in the Client.swift file

extension Operations.AppsGetCollection {
    static func serializeInput(_ input: Operations.AppsGetCollection.Input) throws -> (HTTPRequest, HTTPBody?) {
        // generated code goes here
    }

    static func deserializeOutput(response: HTTPResponse, body: HTTPBody) async throws -> Operations.AppGetCollection.Output {
        // generated code goes here
    }
}

An alternative would be to extend the input and output types with serialize and deserialize functions.

Alternatives considered

1. Modify the OpenAPI Spec file to add a 'cursor' parameter, however, it isn't documented, and doesn't solve the problem at scale.
2. Copy, paste, modify code. That defeats the whole purpose of using generated code
3. Modify the generated code to expose the UniversalClient, make a new request, assume happy path status code response, deserialize the type, reducing the copy, paste, modify code.

These are not good options. Not enabling pagination is also a non-starter.

Additional information

As a frame of reference, other generators structure the generated output such that one can easily implement pagination.

Ref: https://github.com/AvdLee/appstoreconnect-swift-sdk/blob/master/Sources/PagedRequest.swift

[czechboy0]

Hi @edorphy,

thanks for filing this detailed issue 🙏 You're right, this isn't easy to do today.

The main limitation is that the OpenAPI specification itself doesn't provide any conventions that we could rely on to generate additional code. Maybe it should, and it would be a good idea to check out the open development of 3.2 and 4.0 going on right now here: https://github.com/OAI/OpenAPI-Specification - if it's not covered yet, you could file a new issue and link it from here. Once OpenAPI contains paging support, we'll have a clearer path forward for implementing that support in Swift OpenAPI Generator.

More generally, we don't implement custom conventions that aren't part of the OpenAPI specification in Swift OpenAPI Generator, more on that here. That said, we do sometimes open up customization points to allow adopters to implement custom logic, from custom transports, middlewares, date coders, and so on.

In this particular case of App Store Connect, it seems that the next link is documented in the OpenAPI doc, but the cursor query parameter isn't. So one option would be to file a ticket using feedbackassistant.apple.com asking the App Store Connect team to add cursor to the OpenAPI doc, and in the meantime, add it locally (you could write a script that adds it to all operations that have a paging response). Once that's in place, you could write an extension on Components.Schemas.PagedDocumentLinks that extracts just the cursor from the next link, and passes it to the next request, and build a paging convenience function on top of it.

Going further, you could define a protocol that unifies all operations that have a paging response and a cursor parameter in the request, and write a generic function that automates the parsing. That would involve adding a protocol conformance to all the operations you're interested in.

An alternative solution would be to implement a ClientMiddleware that amends the request/response (similar to the solution you mention above). You could use a task-local to propagate data to/from it.

The exact shape you proposed wouldn't work, as the conversion logic is heavily informed by the Configuration struct passed into the Client and registerHandlers APIs for client and server, respectively. And even if the customization point contained that, what would an example implementation look like? Do you have an example in mind?

Thanks again for filing this, I'm also interested in us finding a better solution here, so let's try to figure it out together here. 🙂

[edorphy]

@czechboy0

Thanks for the equally detailed reply.

I will look to making an issue against the spec itself for a standardized pagination mechanism.

Regarding ASC API, for what it's worth I already created that feedback.

FB17754031 (AppStoreConnect: OpenAPI specification doesn't include cursor query parameter in requests making paging from generated code impossible) - May 31, 2025