Building dynamic jobs with BuildKite

If you're using Buildkite for your builds, you may want to reduce duplication in your job's configuration by looping over certain variables, for instance "for each of these 5 environments, deploy the application".

Although you could use a shell script / loop through them in the actual pipeline's configuration, BuildKite's got some great functionality to dynamically define a pipeline.

However, while reading through this, I didn't find it clear as to whether you could only define the dynamic logic when you're defining the job's steps in the UI/via the API, for instance:

- command: ".buildkite/ | buildkite-agent pipeline upload"

Fortunately that's not the case, and we can see below that - as long as buildkite-agent is available in the executing environment - you can call buildkite-agent pipeline upload to dynamically insert whatever pipeline you'd like into the build!

For instance, if we have our job's steps defined as:

- command: buildkite-agent pipeline upload .buildkite/pipeline.yml

Then we can define .buildkite/pipeline.yml as:

  - label: ":go: Build"
    command: "echo go build -o list-environments"
  - wait
  - label: ":aws: Determine environments to deploy to"
      # if this was a built binary, not a shell script, we'd use `buildkite-agent artifact download` here
      - ./list-environments | .buildkite/ | buildkite-agent pipeline upload

In this case, we would see the following resulting result:

A screenshot of the BuildKite pipeline, which shows that the pipeline includes the steps already defined in the YAML, and then has two additional steps for environment deployment that have been dynamically inserted

This can be found as an example project on

Written by Jamie Tanna's profile image Jamie Tanna on , and last updated on .

Content for this article is shared under the terms of the Creative Commons Attribution Non Commercial Share Alike 4.0 International, and code is shared under the Apache License 2.0.

#blogumentation #buildkite.

This post was filed under articles.

Interactions with this post

Interactions with this post

Below you can find the interactions that this page has had using WebMention.

Have you written a response to this post? Let me know the URL:

Do you not have a website set up with WebMention capabilities? You can use Comment Parade.