Spring Boot Externalized Config on Command Line

Spring Boot applications do not always have to serve as a web service located on the Internet. You can also use Spring Boot (or Spring without the Boot) for a command-line utility. I was recently faced with this task, and one requirement for the tool was to support setting a profile-specific configuration on the command line. This isn’t earth-shattering per se since that is a regular Spring feature. The goal was to provide a profile-specific configuration file on the command line that is not bundled in the application.

Imagine developing a Cloud service and running different environments for the different phases of your project – one for development tests, a staging environment, and, finally, the production environment. Connecting to the different environments may require secrets you do not want to be bundled in the application – and, thus, the source tree.

Now, you could roll your own configuration file reader. But wouldn’t it be nice to make full support of Spring’s @Value annotation or @ConfigurationProperties classes?

Setup

In typical Spring fashion, there are multiple ways to do it. First, a bit of setup, namely the configuration class and the configuration itself. You can find the complete source code on GitHub.

I prefer @ConfigurationProperties classes over @Value, so here we go.

@Data
@NoArgsConstructor
@AllArgsConstructor
@ConfigurationProperties("external-config")
public class ExternalConfigProperties {

    private Location input;
    private Location output;

    @Data
    @NoArgsConstructor
    @AllArgsConstructor
    public static class Location {
        private String path;
    }
}

The @Configuration class tells Spring Boot to consider using the config-properties class. Nothing special. The critical bit is the @EnableConfigurationProperties(ExternalConfigProperties.class) annotation.

@Configuration
@RequiredArgsConstructor
@EnableConfigurationProperties(ExternalConfigProperties.class)
public class AppConfig {

    private final ExternalConfigProperties externalConfigProperties;

    @Bean
    public AppRunner appRunner() {
        return new AppRunner(externalConfigProperties);
    }
}

And finally, the bundled application.yml configuration file.

logging:
  level:
    root: INFO
    com.thecodeslinger.externalconfig.ExternalConfigApplication: WARN

external-config:
  input:
    path: /home/bundled/thecode
  output:
    path: /home/bundled/slinger

Since it is just a demo, the values are merely dummies. I tried to add a bit of spice by creating a nested structure.

Usage

How would you specify a configuration file on the command line?

Example #1: Use a “config” folder

In the current working directory, create a folder named “config” and add a file named “application.yml” to it. You do not have to specify anything on the command line using this solution. Spring Boot will automatically pick up the additional configuration.

src % ls
config	main

src % ls config 
application.yml

src % java -jar ../target/external-config-1.0.0.jar param1
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.2)

-> AppRunner.run() Command Line Arguments
Argument: param1
-> ExternalConfigProperties
Input path: /Users/bundled/thecode
Output path: /Users/bundled/slinger

The “config” folder can be in a completely different location from the executed JAR file. If you require more than one configuration file, e.g., to support multiple profiles, you can typically append the profile name to the application.yml.

src % ls config 
application-mac.yml

src % java -jar ../target/external-config-1.0.0.jar param1 --spring.profiles.active=mac
-> AppRunner.run() Command Line Arguments
Argument: param1
Argument: --spring.profiles.active=mac
-> ExternalConfigProperties
Input path: /Users/mac/thecode
Output path: /Users/mac/slinger

Note the differences in the config:

  • It starts with “/Users” in the “mac” profile, whereas the bundled default is “/home”.
  • The external config disables the Spring banner.

There is a second option for setting the profile configuration: a JVM parameter instead of an application parameter. The Spring option will not appear as a command-line argument for your application when you employ this method.

src % java -Dspring.profiles.active=mac -jar ../target/external-config-1.0.0.jar param1
-> AppRunner.run() Command Line Arguments
Argument: param1
-> ExternalConfigProperties
Input path: /Users/mac/thecode
Output path: /Users/mac/slinger

Notice that you now use “-D” instead of “--” to precede the configuration setting spring.profiles.active=mac.

Example #2: Set a config folder

Instead of relying on an implicit “config” folder in the current working directory, you can set any arbitrarily named folder on the command line.

Important: The folder must end with a “/”.

Spring supports two configuration options to achieve this:

  • spring.config.location
  • spring.config.additional-location

The first option replaces the complete bundled file with a file in the folder you set on the command line. The second option augments the bundled file and only overwrites the bundled values with those found in the folder’s file.

Therefore, the first option must be a complete file that cannot rely on defaults. With the second option, you only set the values you want to change.

The file’s name must follow the same pattern as in Example #1application.yml

 or application-<profile>.yml.

Because Spring is complicated, you can also set either of these options using JVM or application parameters.

Assuming the filename is application.yml and using a JVM parameter:

% java -Dspring.config.additional-location=src/config/ -jar target/external-config-1.0.0.jar param1 param2

-> AppRunner.run() Command Line Arguments
Argument: param1
Argument: param2
-> ExternalConfigProperties
Input path: /Users/mac/thecode
Output path: /Users/mac/slinger

Assuming the filename is application-mac.yml and using application parameters:

% java -jar target/external-config-1.0.0.jar param1 param2 --spring.config.additional-location=src/config/ --spring.profiles.active=mac

-> AppRunner.run() Command Line Arguments
Argument: param1
Argument: param2
Argument: --spring.config.location=src/config/
Argument: --spring.profiles.active=mac
-> ExternalConfigProperties
Input path: /Users/mac/thecode
Output path: /Users/mac/slinger

The same applies to the spring.config.location configuration option.

Example #3: Specify a file explicitly

As before, you can use spring.config.location or spring.config.additional-location, and the same rules regarding replacing and augmenting apply. Only this time, you do not set a folder but a file. A profile is not required anymore.

% java -Dspring.config.location=src/config/application-mac.yml -jar target/external-config-1.0.0.jar param1 param2

Or

% java -jar target/external-config-1.0.0.jar param1 param2 \
       --spring.config.location=src/config/application-mac.yml

Famous Last Words

As you can see, Spring allows for a lot of flexibility, and I have not even shown all possible permutations on how you can combine these options. The most compatible way is to use JVM arguments since the Spring configuration options are not passed to your application’s list of arguments. For a simple demo utility, this is a non-issue. However, if you rely on a 3rd party command line parser, like Apache Commons CLI, or build your own, you might run into issues. The parser of your choice may not recognize the additional parameters and complain or trip during parsing. I plan to tackle this problem with Apache Commons CLI in another blog post, where I noticed this behavior.

Thank you very much for reading.

Create Native Java Executable using jpackage – Sort of

I have always been the kind of developer who prefers to use native code and write native code. My background is in C++, and I have worked with Microsoft’s WinAPI early in my career. That is to say: I like it fast, and I do not mind going to lower levels.

I am not stuck in the past, though, and as such, I, too, have evolved with the times. I still like C++, but I also see how languages like Java and its great tooling can boost productivity in comparison. As a result, I write code fast. Java is the tool of the trade at my current job, and performance usually is not a problem anymore. The JVM has improved, and computer hardware has, so performance is usually not an issue anymore.

There is one little problem, however: Usage. Let me explain.

Read More »

Apple Silicon M1 for Software Development: Java, C++ with Qt

Apple’s laptops have been making quite the splash since the end of 2020 and have made a massive comeback as a professional tool one year later with the M1 Pro and Max designs. Most of the reviews I have seen focus on the editing and rendering capabilities of these new MacBooks. A few reviewers throw a compile test in the mix, but compiling Chromium or any other huge project is only a part of the equation. Developers don’t just compile code; they also use tools and IDEs to develop their software.

Being new to the M1 world, I wanted to recap my experiences so far briefly. I use Java professionally, and I also have a C++ application based on the Qt framework that I wrote an eon ago and still use productively. Being a former C++ professional, I am about native performance, and I like native software. Therefore, I intended to utilize as many Apple Silicon-native tools as possible. Luckily, one year after its release to the desktop world, the most popular applications have caught up. Let me go through my tool suite one by one.

Read More »

Emulate Java Enums With Values in C# (Pt. 2, Improved With Conversion Operator Overload)

In a previous blog post, I demonstrated how Java’ enums that contain one or more values/objects can be emulated with C#. One thing bothered me, though: the switch statement and how inconvenient it was to determine the proper type. Worst of all, it was not type-safe. In my simple example, it was easy because I was using strings. Imagine your fake-enum does not contain a string to quickly identify the instance.

Well, there is a prettier workaround – and it involves an actual enum. I was thinking about how the same could be done in C++ and in C++, you can have type conversion operators. Then I searched if such a feature also exists in C#, and sure enough, it does.

Read More »

Emulate Java Enums With Values in C#

Update August 28, 2021

I have written a follow-up that improves on the following solution using a type conversion operator overload.

When I started dabbling in C#, I wondered if it supports values in enums. In Java, an enum instance can have properties (called fields in Java lingo) associated with the enum’s literals. By taking advantage of this feature, you can encode more information in an enum, like a string, for example, or a constant number. You can even embed instantiated class objects, maybe to associate an object factory with a literal.

In my use case, I wanted to achieve a form of a key-value-pair mapping. I require certain illegal characters in the NTFS file or directory names to be replaced with a given code. I use HTML encoding for my needs because I can simply look up the values online if I need to.

Here is the Java reference example. First, let me start with the basic enum definition (I use Lombok to auto-generate boilerplate code like the constructor and accessors).

@Getter
@RequiredArgsConstructor
enum CharacterReplacementCode  {
    COLON(":", "&58;"),
    POUND("#", "&35;"),
    QUESTION_MARK("?", "&63;"); 

    private final String character;
    private final String replacement;
    
    @Override
    public String toString() {
        return String.format("Character '%s' substituted by code '%s'", character, replacement); 
    }
}
Read More »

Convert Java POJO With Protobuf field to JSON Using Jackson

In this blog post, I will explain how to convert a regular Java class that contains a Protobuf message field to JSON using the Jackson library, for example, in a Spring Boot application as a return value of an @Controller method.

You might wonder, how this is such a big deal? After all, you can create complex POJO hierarchies, and Jackson will pick them up just fine. Well, maybe this error message will convince you.

o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [
    Request processing failed; nested exception is org.springframework.http.converter.HttpMessageConversionException: 
    Type definition error: [simple type, class com.google.protobuf.UnknownFieldSet]; 
    nested exception is com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Direct self-reference leading to cycle 
    (through reference chain: com.thecodeslinger.ppjs.web.dto.AwesomeDto["awesomePowerUp"]
        ->com.thecodeslinger.ppjs.proto.AwesomePowerUpOuterClass$AwesomePowerUp["unknownFields"]
        ->com.google.protobuf.UnknownFieldSet["defaultInstanceForType"])] with root cause

Java classes that you create with the Protobuf compiler require their JSON converter JsonFormat.Printer. So, how can we get Jackson and JsonFormat.Printer love each other and have a wedding together?

Simple: we create a custom JsonSerializer.

public class ProtobufSerializer extends JsonSerializer<Message> {

    private final JsonFormat.Printer protobufJsonPrinter = JsonFormat.printer();

    @Override
    public void serialize(Message anyProtobufMessage, JsonGenerator jsonGenerator, SerializerProvider serializerProvider)
            throws IOException {
        // The magic sauce: use the Protobuf JSON converter to write a raw JSON
        // string of the Protobuf message instance.
        jsonGenerator.writeRawValue(protobufJsonPrinter.print(anyProtobufMessage));
    }
}

This class is very simple and can work with any Protobuf Message class instance. This way, it is universal and only needs to be written once. The main ingredient is the jsonGenerator.writeRawValue method that takes the input without modification. Since we already ensure a proper JSON format using Protobuf’s converter, this is no problem in this case. Otherwise, be careful with this method.

The last step is to annotate the Message field in the POJO, so Jackson knows what to do.

@JsonSerialize(using = ProtobufSerializer.class)
private final AwesomePowerUpOuterClass.AwesomePowerUp awesomePowerUp;

You can find a complete working example that uses Spring Boot and a REST endpoint on my Github account.

Maven Failsafe Plugin environmentVariables Not Working (org.postgresql.Driver claims to not accept jdbcUrl)

A few months ago I had written about how one can setup a PostgreSQL database in a docker container during the Maven testing phase (part 1 and part 2). Today, I wanted to iterate on this topic using Testcontainers. Unfortunately, before I could get to that I ran into issues with the original project. For some reason I was now getting the following error:

Caused by: java.lang.RuntimeException: Driver org.postgresql.Driver claims to not accept jdbcUrl, jdbc:postgresql://localhost:${it-database.port}/docker_db_testing_tests

Maven did not replace the variable it-database in the integration test application.properties file. The question is: "why now"? It has worked before. Now, one thing that I changed is that this time I was using Linux instead of Windows. Either way, the fix was simple, although not obvious. It seems to be an issue with the Maven Failsafe plugin.

Not working:

<configuration>
    <environmentVariables>
        <it-database.port>${it-database.port}</it-database.port>
    </environmentVariables>
</configuration>

Working:

<configuration>
    <systemPropertyVariables>
        <it-database.port>${it-database.port}</it-database.port>
    </systemPropertyVariables>
</configuration>

Here is the link to the commit.

Spring Multipart File – Can I Read InputStream Multiple Times?

The short answer is Yes.

Here is the long version and why I even asked myself this question.

If you are familiar with ServletRequest then you probably know that calling its getInputStream method only works once. If you need to read the body data multiple times then it is up to you to cache it in a buffer or employ workarounds such as a "caching servlet request". Unfortunately, this fact is not stated in the Javadoc of ServletRequest#getInputStream so it is no wonder this question gets asked.

Spring’s MultipartFile is a bit different here. It, too, has a getInputStream method, but this one can be called multiple times. Again, it is not obvious from the documentation which is why I am making this mental note for myself and others who are researching this question because they know about the behavior of ServletRequest and assume – as I did – it is the same for MultipartFile. Fortunately, it is not.

In my case I needed to compute a hash of an uploaded file and then move the file to Azure’s Blob Storage. The Azure API used an InputStream and I assumed, once I had consumed the multipart InputStream that I could not use that API anymore.

As a side note: Using DigestInputStream it is possible to do this in one go, move the data to storage and while doing that compute the hash. In my case, I needed the hash first to compare it with a value that was provided on upload. Only when they match can the data be transferred to storage.

Mockito “when” vs. “verify”

Recently, during a code review at work there was a discussion about whether Mockito’s verify is necessary when when (or given, which is a synonym) is parameterized. The quick answer is "no".

Imagine this contrived example. I have two classes, Calculator and ComplexOperationService, where the latter requires the former. The goal is to write a unit test for ComplexOperationService and mock all usages of Calculator.

Here are the two classes.

public class Calculator {

    public int sum(final int a, final int b) {
        return a + b;
    }
}

@RequiredArgsConstructor
public class ComplexOperationService {
    
    private final Calculator calculator;

    public void doComplexOperation(final int a, final int b) {
        System.out.println(calculator.sum(a, b));
    }
}
Read More »

Integration Testing With Docker Maven Plugin, PostgreSQL, Flyway

Some things in software development require more than mocks and unit testing. If your application uses a database it makes sense to also hit that database in automated testing to ensure custom SQL queries work correctly, Hibernate relations are set up properly and also that database migrations are successful.

This blog post was written with a focus on the latter. I will be using Spring Boot talking to a PostgreSQL database. The database structure is managed via Flyway and, basically customary for Java applications, Maven serves as the build and dependency management tool. Docker will also play a role because we’ll be creating and running a PostgreSQL docker image for testing. From Maven. Every time the test is executed. And to spice things up, we’ll also create a custom database and user in that dockerized PSQL image.

I have created a working sample on Github and you can follow every single step by taking a look at the commit history. There you can see individual changes, starting from an empty Spring Boot application with no database to the final solution with Spring Data JPA and Flyway.

In the following sections and snippets, I will highlight the important parts of each step.

Read More »

Comparing Java Optional vs C++ STL optional

Optionals in Java have been around for some time now, basically since the release of version 8. My other language of choice, C++, has received this feature in version C++17. Since I am currently in the process of writing some C++ code, I was curious how they were implemented there. Optionals are trying to solve a problem that is likely to plague any language. What shall a method or function return if there is no value? Or shall it not return anything but instead start crying like a petulant child and throw an exception?

As an introduction, let me dive a little bit into why we need optionals (or do we?) and compare two different implementations of this concept, one being java.util.Optional and the other C++ std::optional. I chose to compare these two language for several reasons:

  1. I work with Java in my day job, so I have a good idea of how it works there.
  2. As mentioned, C++ is one of the languages I know quite well too.
  3. The main reason: both optional implementations are add-on classes rather than language features. More on that later.
Read More »

Spring @ConfigurationProperty a Bean or not?

Semi-recently (“semi” because procrastination kept me from writing, so it’s more like two months ago, but blog posts have to start with “recently” when you try to explain yourself why you are writing what you are writing – but I’m getting sidetracked here, so let’s move on) I was wondering whether Java classes annotated with Spring’s @ConfigurationProperty should be declared as a bean, e.g. with @Component. I didn’t find a definitive answer, but I found three ways on how to do it – typical Spring, I guess.

Here’s a quick setup:

My configuration class.

package com.thecodeslinger.configpropsdemo;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties; 

@Data 
@ConfigurationProperties(prefix = "demo")
public class Configuration {

    private String elegy;
}

My main application:

package com.thecodeslinger.configpropsdemo;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import javax.annotation.PostConstruct;

@SpringBootApplication
public class ConfigPropsDemoApplication {

   @Autowired
   private Configuration configuration;

   @PostConstruct
   public void postConstruct() {
      System.out.println(configuration.getElegy());
   }

   public static void main(String[] args) {
      SpringApplication.run(ConfigPropsDemoApplication.class, args);
   }
}

And finally, my properties file:

demo.elegy=R.I.P. Kobe

It’s not an elegant setup, but that’s not the point. It does the job for now.

If you run the application in this state, Spring will greet you with an error message.

APPLICATION FAILED TO START

Description:
Field configuration in com.thecodeslinger.configpropsdemo.ConfigPropsDemoApplication required a bean of type 'com.thecodeslinger.configpropsdemo.Configuration' that could not be found.
The injection point has the following annotations:
- @org.springframework.beans.factory.annotation.Autowired(required=true)
Action:
Consider defining a bean of type 'com.thecodeslinger.configpropsdemo.Configuration' in your configuration.

It obviously cannot find the configuration bean.

Option #1: Slap @Component to it.

package com.thecodeslinger.configpropsdemo;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Data
@Component
@ConfigurationProperties(prefix = "demo")
public class Configuration {

    private String elegy;
}

Option #2: Use the @EnableConfigurationProperties annotation.

import org.springframework.boot.context.properties.EnableConfigurationProperties;

@SpringBootApplication
@EnableConfigurationProperties(Configuration.class)
public class ConfigPropsDemoApplication {

Option #3: Use @ConfigurationPropertiesScan to explicitly name the packages to scan for.

import org.springframework.boot.context.properties.ConfigurationPropertiesScan;

@SpringBootApplication
@ConfigurationPropertiesScan({"com.thecodeslinger.configpropsdemo"})
public class ConfigPropsDemoApplication {

All three options achieve what you’re aiming for, a running application.

:: Spring Boot :: (v2.2.4.RELEASE)
2020-02-03 19:57:43.562 INFO 4612 --- [ main] c.t.c.ConfigPropsDemoApplication : Starting ConfigPropsDemoApplication on DESKTOP-C0O3OKC with PID 4612 (D:\OneDrive\Code\Java\config-props-demo\target\classes started by lober in D:\OneDrive\Code\Java\config-props-demo)
2020-02-03 19:57:43.562 INFO 4612 --- [ main] c.t.c.ConfigPropsDemoApplication : No active profile set, falling back to default profiles: default
R.I.P. Kobe
2020-02-03 19:57:43.921 INFO 4612 --- [ main] c.t.c.ConfigPropsDemoApplication : Started ConfigPropsDemoApplication in 0.573 seconds (JVM running for 1.083)

So, is there any benefit of one over the other? The Spring documentation has the following to say:

Sometimes, classes annotated with @ConfigurationProperties might not be suitable for scanning, for example, if you’re developing your own auto-configuration or you want to enable them conditionally. In these cases, specify the list of types to process using the @EnableConfigurationProperties annotation. This can be done on any @Configuration class, as shown in the following example:

I’m not using a @Configuration class in my example, but if you were, you could leverage that to load your configuration classes based on @Profile annotations. Although @Component works too, it’s not mentioned in that part of the Spring documentation (“Type-safe Configuration Properties”).

For myself, I might go with @EnableConfigurationProperties and if it makes sense, even have dedicated @Configuration classes linked to @Profile. For little samples like this one it’s obviously overkill. In a remotely useful application, the additional overhead may be worth it for structural and documentational reasons.

Kotlin Object Expression – What more can object do?

In a previous post I explained what Kotlin Object Declarations are. This time around it’s about the declaration’s sibling, the Object Expression.

An object is not just a glorified static replacement or a singleton. object can be used where Java usually utilizes anonymous inner classes. Let’s look at a more realistic scenario: a JButton and an ActionListener or a MouseListener.

Read More »

Kotlin Object Declarations – The fake-static

Instead of implementing my own backup application as I had planned a long time ago, I’m wandering off (re)learning Kotlin after a long absence from that language. In my defense though, I’m doing it in the context of the backup app which will not be Java as originally intended (or maybe later for comparison, who knows, I obviously can’t be trusted with my plans). Putting that aside, the most confusing concept of Kotlin for a Java developer is the object. What is that thing doing that a class can’t do and how do we declare static fields and methods? I know it’s nothing new, but that part seems to have changed a bit since I used Kotlin about two (?) years ago. So, for me this is a refresh of old information and also something new and by writing about it I will engrave it in my brain once and for all. And your confusion will hopefully turn into some productive… fusion… of some sort… or so.

Read More »

Unwanted JUnit 4 Dependency with Kotlin and JUnit 5

I ran across this issue only by accident because I was investigating a completely different problem. I wrote a quick test to debug my issue and was wondering why custom serializers and deserializers are not registered with the Jackson ObjectMapper. I had a nice init() function that was annotated with @Before. So, what the hell?

Let’s back up a bit for some context.

  • Kotlin Project
  • Runs on Java 12
  • JUnit 5 as test engine
  • AssertK for assertions (just for the sake of completeness)

I’m used to JUnit 4, so in my test I used @Before to annotate a setup method. It was one of the many options IntelliJ presented to me.

@Before
fun init() {
    val module = SimpleModule()
    module.addDeserializer(Instant::class.java, InstantDeserializer())
    module.addSerializer(Instant::class.java, InstantSerializer())
    mapper.registerModule(module)
}

The method wasn’t called, however. But it’s annotated! Well, it’s the wrong annotation if you’re using JUnit 5. The correct one is @BeforeEach. This one and @BeforeClass (new name @BeforeAll) have been changed from version 4 to 5 to make their meaning more obvious.

But that’s besides the point. The question is: where does this @Before come from then?

A look at the dependency tree quickly reveals the culprit.

It’s the official JetBrains Kotlin JUnit test artifact. Although it doesn’t hurt me to have it in my project, it certainly caused some confusion and I’d like to avoid that in the future. Hence, I excluded the old version of JUnit in my POM file for this dependency.

<dependency>
    <groupId>org.jetbrains.kotlin</groupId>
    <artifactId>kotlin-test-junit</artifactId>
    <version>${kotlin.version}</version>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Problem solved.