Overriding a Gradle project property which has . and _ characters, without modifying the buildscript

Yesterday I was working with a codebase which uses a gradle.properties, and for ✨ reasons ✨ I need to overwrite a given property, without modifying the codebase.

This is something that the docs indicate is a common need:

This feature is useful when you don’t have admin rights to a continuous integration server, and you need to set property values that should not be easily visible. Since you cannot use the -P option in that scenario, nor change the system-level configuration files, the correct strategy is to change the configuration of your continuous integration build job, adding an environment variable setting that matches an expected pattern. This won’t be visible to normal users on the system.

But unfortunately I found it to be particularly painful, as I have a property that includes both a . and _.

(The below is tested with Gradle 7.6.4)

Why is this painful? Let's start with looking at how the project is set up. We've got a gradle.properties like so:

me.jvt.example=something
me.jvt.example_underscore=another_value

In our Gradle Kotlin buildscript, we've got:

var prop = providers.gradleProperty("me.jvt.example").orNull
println("The value of me.jvt.example is: ${prop ?: "<null>"}")

prop = providers.gradleProperty("me.jvt.example_underscore").orNull
println("The value of me.jvt.example_underscore is: ${prop ?: "<null>"}")

Then, we can check that this is loaded correctly:

% ./gradlew

> Configure project :
The value of me.jvt.example is: something
The value of me.jvt.example_underscore is: another_value

Do environment variables work?

I started by using an environment variable, which was recommended in the docs as a way to do it when you can't modify the properties used to execute Gradle, where you can use i.e.:

ORG_GRADLE_PROJECT_foo=bar

However, it wasn't explained in the docs as to how this would work with a property with both a . and _ in it.

I wondered if this would work how Spring Boot's environment variable naming works, but after playing around with different options, the below didn't seem to work:

env ORG_GRADLE_PROJECT_ME_JVT_EXAMPLE_UNDERSCORE=overridden ORG_GRADLE_PROJECT_ME_JVT_EXAMPLE=over_ridden ./gradlew

> Configure project :
The value of me.jvt.example is: something
The value of me.jvt.example_underscore is: another_value

However, it does seem to work when using the plain property names in the environment variables, such as:

env ORG_GRADLE_PROJECT_me.jvt.example=overridden ORG_GRADLE_PROJECT_me.jvt.example_underscore=over_ridden ./gradlew

> Configure project :
The value of me.jvt.example is: overridden
The value of me.jvt.example_underscore is: over_ridden

That works 🚀 Or does it..?

Environment variables with unexpected characters may not work

Interestingly this seemed to work when running interactively in bash, but doesn't work if you try and export the variables:

$ export ORG_GRADLE_PROJECT_me.jvt.example=overridden ORG_GRADLE_PROJECT_me.jvt.example_underscore=over_ridden
bash: export: `ORG_GRADLE_PROJECT_me.jvt.example=overridden': not a valid identifier
bash: export: `ORG_GRADLE_PROJECT_me.jvt.example_underscore=over_ridden': not a valid identifier

As noted on the StackOverflow post "Exporting a variable with dot (.) in it", the shell doesn't deem that to be a valid variable, but using it with env seems to be safe (which is my preference anyway).

I'd also read some docs about how the _ will be implicitly remapped to a ., so I'd have to modify the buildscript to reference me.jvt.example.underscore which was not intended.

However, when I started using that with Renovate, which was the key problem I was trying to solve, it didn't seem to work:

(Note that the below command was intentionally set up to fail with an error code - the important thing here is that we need to get the output of the properties)

       "err": {
         "cmd": "/bin/sh -c ./gradlew exit",
         "stderr": "..."
         "stdout": "...> Configure project :
The value of me.jvt.example is: something
The value of me.jvt.example_underscore is: another_value
",
         "options": {
           "cwd": "/tmp/renovate/repos/github/jamietanna-testing/renovate-testing-gradle-opts",
           "encoding": "utf-8",
           "env": {
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "ORG_GRADLE_PROJECT_me.jvt.example": "overridden",
             "ORG_GRADLE_PROJECT_me.jvt.example_underscore": "over_ridden",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },

Notice that we've wired in the variables to override the properties which seems to be correct, but they are not working.

Falling back to an GRADLE_OPTS

As it didn't end up working, and had some caveats around usage, I looked for an alternate version.

As this is a Gradle project property, it could be set with i.e.:

./gradlew properties -Pme.jvt.example_underscore=new_value_here

We could try and use JAVA_OPTS, but find that this doesn't work as that's for the underlying JVM being started up, not for the Gradle process itself:

$ env JAVA_OPTS="-Pme.jvt.example_underscore=new_value_here" ./gradlew properties
Unrecognized option: -Pme.jvt.example_underscore=new_value_here
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Notice that this does not work, as we need to set the org.gradle.project prefix to indicate to Gradle that we're trying to override a project property:

% env GRADLE_OPTS="-Dorg.gradle.project.me.jvt.example_underscore=new_value_here" ./gradlew properties

> Configure project :
The value of me.jvt.example is: something
The value of me.jvt.example_underscore is: new_value_here

Omitting the prefix will result in the property not working:

# NOTE doesn't work
% env GRADLE_OPTS="-Dme.jvt.example_underscore=new_value_here" ./gradlew properties

> Configure project :
The value of me.jvt.example is: something
The value of me.jvt.example_underscore is: another_value

As we can see, using GRADLE_OPTS to specify these project properties are probably the most resilient, and don't require any code changes.