Automating the syncing of files between repos with Renovate and Venidr

For a few years, I've been trying to get a perfect workflow for keeping vendored files in sync between repos.

Especially with oapi-codegen, where it's our recommendation to vendor the OpenAPI specification and your generated code, there can be a bit of awkwardness in place to try and keep your copy of some other repo's OpenAPI spec in sync with yours.

I've written before about how I've used GitHub Actions to check if the files are out-of-sync and using GitHub Actions to actually sync the files, but as ever I'm looking for better workflows.

My view is that if I can use Renovate for a task, I'll try to, as it's the best at dependency management, in my now more biased opinion.

Over the years, before I joined Mend, I've worked upstream with Renovate to see if we can add a "file sync manager", but I'd never really managed to get the time from my employers to do so, despite it providing a tonne of value.

Now I'm in a privileged position of being able to lead the roadmap of the project, and having much more context and understanding for Renovate, I sat down to look again at the "file sync manager" proposal, and see if I could chip away at it.

I noticed a comment from past me about how Vendir may do what we want, but at the time didn't support the datasources we'd want (HTTP and Git/GitHub).

When I looked into it this week, I was very happy to see that, actually, Vendir now does have support for HTTP and Git sources 👀

Setting it up

I've updated the example repo from my GitHub Actions syncing workflow to use Renovate to sync OpenAPI specs from a private repo, which you can see in action in this PR.

As with my previous examples, let's say that we have a couple of OpenAPI specs in jamietanna/example-github-actions-sync-files-private that we want to sync with our repository.

We can set up a vendir.yml like so, and run a vendir sync to initialise everything:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  - path: internal
    contents:
      - path: private-apis
        git:
          url: https://github.com/jamietanna/example-github-actions-sync-files-private
          ref: origin/main
        includePaths:
          - api/**/*.yml
          - api/**/*.yaml

Next, we want to make sure that Renovate periodically updates the files, using vendir sync.

Within the Vendir manager in Renovate, the terminology for this functionality is the lockFileMaintenance update type.

We can enable this with the following configuration:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "lockFileMaintenance": {
    "enabled": true,
    "schedule": [
      "at any time"
    ]
  }
}

The at any time scheduling means that whenever Renovate runs, and it has the ability to create a new PR for updates (i.e. you're under the limit of PRs that will be open concurrently), it'll try and see if any updates come from vendir sync. This allows you to get updates sooner than later, rather than the default of receiving updates once a week.

(There are some tweaks to this configuration that are out-of-scope for this post, like scoping lockFileMaintenance to only the Vendir manager, or to create a specific name for the PRs, or i.e. running go generate ./... after updating those OpenAPI specs)

For example, you'll now receive a PR like this example which is the result of running vendir sync, and shows updates to the OpenAPI specs in the repository, coming from the private repository.

Also of note is that the lockfile introduces commit message metadata so it knows where Vendir last synced from.

This works pretty nicely, and because it's now managed through Renovate, you can configure the PR description, grouping, post-update actions, and scheduling with much more control than having to write that logic yourself!

Other sources available

You'll notice that we're specifying a ref for the git option, but if we pin to a tag version, Renovate can update those tags (and then re-run vendir sync), so you can keep files in sync between releases.

It's also possible to use some of the other source types that Vendir supports - the HTTP backend, for instance, is what we'll be using with rootly-go in the future.

Caveats

Folder structure

One way with how Vendir works is that it must vendor files into a specific directory, as specified by the path fields.

This is a little awkward, if you already have a structure for where you want files to be, so you may want to symlink from the Vendir'd file to the location you want the file to be in.

Authentication needed

As long as Renovate has a Host Rule that allows it to authenticate to the private repository, you'll now see a PR like this example which is the result of running vendir sync.

(Note that on the Mend-hosted, the way we configure these means that you need to - for the short-term - set a Host Rule for the repository you're trying to access, with a GitHub Personal Access Token)