Tricking oapi-codegen into working with OpenAPI 3.1 specs

As has been noted before, oapi-codegen doesn't (yet) support OpenAPI 3.1. So how do you end up working with an API that's producing an OpenAPI 3.1 spec?

This weekend, I've been migrating dependency-management-data to use the new V1 API for endoflife.date, which now requires OpenAPI 3.1.

I started setting myself about to downgrade the spec from OpenAPI 3.1 to OpenAPI 3.0, through a convoluted process which would require careful handling when updating the spec, when I remembered that in the last big release of oapi-codegen, v2.4.0, we introduced support for the OpenAPI Overlay specification, which allows modifying the spec without directly modifying the spec file itself.

With this in mind, I started working out how I'd make the relevant changes to do this - chances are it was probably only a few OpenAPI 3.1 features that were being taken advantage of, so I may be able to use Overlays to forcibly downgrade those usages, instead of the full spec being downgraded unnecessarily.

The below has been tested with oapi-codegen v2.4.1 and github.com/speakeasy-api/openapi-overlay v0.10.1, both of which the latest tagged release at time of writing. Using a lower version of github.com/speakeasy-api/openapi-overlay may not work, due to JSON Path implementation changes.

Before

Let's look at what happens when you try and use an OpenAPI 3.1 spec with oapi-codegen.

Trying to re-generate our client code boilerplate for this new spec led to an error:

# in the DMD repo, get the V1 spec, which uses OpenAPI 3.1
curl https://endoflife.date/docs/api/v1/openapi.yml -Lo internal/endoflifedate/client/spec.yaml
# then try and rebuild the client boilerplate via `oapi-codegen`
$ go generate ./internal/endoflifedate/client
WARNING: You are using an OpenAPI 3.1.x specification, which is not yet supported by oapi-codegen (https://github.com/oapi-codegen/oapi-codegen/issues/373) and so some functionality may not be available. Until oapi-codegen supports OpenAPI 3.1, it is recommended to downgrade your spec to 3.0.x
error generating code: error generating type definitions: error generating Go types for component schemas: error converting Schema ProductDetails to Go type: error generating Go schema for property 'labels': error generating Go schema for property 'discontinued': error resolving primitive type: unhandled Schema type: &[string null]
exit status 1
internal/endoflifedate/client/generate.go:3: running "go": exit status 1

In this case - as I'm aware of the fact that an OpenAPI 3.1 spec is allowed to have multiple types, I know that this error is because of our spec version mismatch.

(Aside: we should improve this error message)

We can see these in the source OpenAPI spec:

    ProductDetails:
      type: object
      # ...
      properties:
        # ...
        versionCommand:
          # This is where we see the issue
          type:
            - string
            - "null"
          examples:
            - "lsb_release --release"
        labels:
          description: Product labels.
          type: object
          required:
            - eol
          properties:
            # ...
            discontinued:
              # This is where we see the issue
              type:
                - string
                - "null"
              examples:
                - "Discontinued"

As we can see here, oapi-codegen is struggling to reconcile the duplicated types for these fields, which indicate these are nullable strings.

In OpenAPI 3.1 this is the correct way to note this, but if we were using OpenAPI 3.0, we'd write:

# ...
discontinued:
  type: string
  nullable: true
  examples:
  - "Discontinued"

There were a few other places in the spec that needed this hand-migrating, so I set about looking at how I'd get around to doing it.

So how do we do it?

The hard way

I started by trying to do it "the hard way", by manually constructing an OpenAPI Overlay document.

I say this is the "the hard way" given it involves hand-crafting JSON Path expressions, which can sometimes be a bit gnarly.

Additionally, trying to remember where I was in the large YAML document was a bit awkward, and so I spent some time trying to get my Neovim setup to display the JSON Path / YAML Path / some level of information on what path I was at inside the YAML document, but unfortunately I've still not found a good option - got a recommendation? Please let me know!

I started by using the JSON Path online playground (in RFC 9335 mode), given it was a fully public spec and I'm very wary of online tools for anything remotely non-public, and was making some reasonable, but slow, progress.

The easy way

At this point, I had a half-memory, and was sure I remembered that Speakeasy's OpenAPI Overlay library - which we use in oapi-codegen - had a CLI inside it which may have this functionality.

It indeed did have that functionality, and meant that I could take the original spec, make some changes to it, and then use openapi-overlay compare to generate an Overlay from those changes 🚀

For instance:

# in the DMD repo, get the V1 spec, which uses OpenAPI 3.1
curl https://endoflife.date/docs/api/v1/openapi.yml -Lo  internal/endoflifedate/client/spec.yaml
# create a copy to show what we want
cp internal/endoflifedate/client/spec.yaml internal/endoflifedate/client/spec-want.yaml
# make edits to the "what we want" spec
nvim internal/endoflifedate/client/spec-want.yaml

# then generate an Overlay
openapi-overlay compare internal/endoflifedate/client/spec.yaml internal/endoflifedate/client/spec-want.yaml > internal/endoflifedate/client/overlay.yaml

From here, we then get an overlay.yaml that looks like:

overlay: 1.0.0
x-speakeasy-jsonpath: rfc9535
info:
  title: Overlay internal/endoflifedate/client/spec.yaml => internal/endoflifedate/client/spec-want.yaml
  version: 0.0.0
actions:
  - target: $["components"]["schemas"]["ProductVersion"]["properties"]["date"]["type"]
    update: string
  - target: $["components"]["schemas"]["ProductVersion"]["properties"]["date"]
    update:
      nullable: true
  # ... 31 other actions

The easiest way

In retrospect, it may have been even easier to use a tool that downgrades OpenAPI 3.1 to OpenAPI 3.0, and then wiring the diff between them into openapi-overlay, providing a completely hands-off approach, as well as making sure that any more subtle behaviour changes between OpenAPI 3.1 and OpenAPI 3.0 are correctly handled.

I've since tested it with npx @apiture/openapi-down-convert@latest and found I'd missed some tweaks, where the spec is using the new OpenAPI 3.1 examples which would need to be downgraded to OpenAPI 3.0's example.

In my case, I don't mind about that, but there may be other things to watch out for.

After

Some code-golfing later

The generated Overlay from openapi-overlay was absolutely great, but involved a fair bit of duplication.

This is absolutely the correct thing to do - make the actions explicit, as there's not always a "general case" that can be used.

In my case, however, I knew that any place there was a nullable string or a nullable boolean, I wanted to replace it with an OpenAPI 3.0 nullable: true, so I could work a little more to create a generic action to reduce the number of actions to maintain.

I started off by (or maybe succumbed to) asking Copilot (via GPT-4o) how to do it, and it didn't quite get it right. After several failed iterations, I ended up reading the RFC 9335 spec to determine which functions were available to use, and this ended up resulting in actions such as:

  # ...

  # avoid `oapi-codegen` warning us that we're using an OpenAPI 3.1 spec
  - target: "$.openapi"
    update: "3.0.3"

  - target: "$.components.schemas.*.properties[?(@.type && length(@.type) == 2 && (@.type[0] == 'string' && @.type[1] == 'null'))]"
    update:
      type: string
      nullable: true
  # ...

This significantly reduced the duplication across the Overlay, but I do wonder how much I'll enjoy maintaining those more complex JSON Paths in the future.

With this Overlay in place, I can now wire it into oapi-codegen:

 # yaml-language-server: $schema=https://raw.githubusercontent.com/oapi-codegen/oapi-codegen/v2.4.1/configuration-schema.json
 package: client
 generate:
   client: true
   models: true
 output: client.gen.go
+output-options:
+  overlay:
+    path: overlay.yaml

With this in place, I can now re-generate the client code with no concerns from oapi-codegen:

$ go generate ./internal/endoflifedate/client
# no output, therefore success 👏🏼

And now we can use an upstream OpenAPI 3.1 spec as if it were an OpenAPI 3.0 one with oapi-codegen 🚀

Further thoughts

I'm sure some slightly cynical folks will be thinking "why don't you just add support for OpenAPI 3.1?" To this, I'd say have a read of the thread as well as the conversation in kin-openapi, which oapi-codegen currently relies upon for OpenAPI 3.x support.

I very much hope that we will be able to add support for this in the future, but it's almost certainly not possible without financial support to prioritise the work, so might be worth seeing if you can put your money where your mouth is.

In the meantime, this has been a fairly ergonomic solution to the problem, and I'm loving Overlays. I went from 0 to solution pretty quickly, which is surprising given I've forgotten everything I learned about them when I implemented the feature last year 😹