In two previous blog posts, I explained how to create a Kafka consumer and producer with the Spring Cloud Stream framework. In the Famous Last Words section of the producer, I already hinted at the notion of utilizing this technology for connecting to Azure Event Hub. While doing so, I discovered an error in one of Microsoft’s examples that has cost me about two days of work. I show you how to avoid the dreaded “Node -1 disconnected” error.

In this tutorial, I explain how to use the exact same code to connect to Azure Event Hub using a Shared Access Signature Token (connection string) and a Service Principal.

I have good news and bad news. Which one first? The bad? Okay, here we go:

There will not be any code in this tutorial, only YAML configuration.

Now to the good part:

There will not be any code in this tutorial, only YAML configuration.

This is the beauty of Spring Cloud Stream. Granted, I am not even swapping the binder for an Azure-native variant. So why would there be any code changes? But let me say this: I briefly plugged in the Event Hub Binder without changing the code in my research on getting this to work. Even the updates to the config were minimal. A few Event Hub-specific settings, especially the Storage Account for checkpoints, and that was it.

Enough foreplay; let me explain what you likely came here for.

Connected By a String

Connection strings are not the ideal way of managing access to Azure resources. If you can, avoid them. However, I’ve been a developer long enough to know this is not always an option. Sometimes, it is even the easiest solution. I thought so, too, but it cost me about two days of research. I just wanted to debug locally to see if the application and settings work before committing to Git.

The root of all evil is hidden in one of Microsoft’s tutorials. If you try to do something as I did

(Why else would you be here?)

you likely came across this guide from Microsoft. It looks easy enough, but it contains one fatal flaw.

spring:
  cloud:
    azure:
      eventhubs:
        connection-string: ${AZ_EVENTHUBS_CONNECTION_STRING}

This configuration does not work. All it gives you are obscure errors like these.

org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Node -1 disconnected.
org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Cancelled in-flight API_VERSIONS request with correlation id 0 due to node -1 being disconnected (elapsed time since creation: 244ms, elapsed time since send: 244ms, request timeout: 3600000ms)

VPNs, Network Security Groups, and Private Links secured my working environment. Reading the error, the first that came to mind steered me toward networking issues. But that was not it. Luckily, I could cross-check in a testing environment without all that security. The error stayed the same.

(Jerk)

The correct configuration for this scenario actually looks like this. It appears wrong, I know, but it really works.

stream:
  kafka:
    binder:
      configuration:
        security.protocol: SASL_SSL
        sasl.mechanism: PLAIN
        sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://{EVENT-HUB-NAMESPACE}.servicebus.windows.net/;SharedAccessKeyName={SAS-NAME};SharedAccessKey={SAS-KEY};EntityPath={EVENT-HUB-NAME}";

This extra set of configuration options fixed the issue. The connection string must be part of the sasl.jaas.config property, and it also requires the values SASL_SSL for security.protocol and PLAIN for sasl.mechanism. Microsoft’s documentation contains this information. Unfortunately, it is not in the same location as the tutorial I linked earlier.

Note: $ConnectionString is not a variable for you to replace. This is part of the value. You only replace the connection string that is in password=””.

My Service is a Principal

Good for your service! What does that mean for the Spring Cloud Stream config? Let me phrase it this way: had I used this from the start, I would never have had material for this blog post.

Using a service principal is very simple. When you have the Azure Identity library on the classpath (I assume Spring Boot here), it automatically tries several authentication methods. It also works with a Managed Identity, something I strongly recommend if you have the option. It is not an option for local debugging; this is where a Service Principal can come in handy. But it will work the same way when deployed to a Kubernetes cluster.

For a Service Principal, you must define three environment variables:

• AZURE_TENANT_ID
• AZURE_CLIENT_ID
• AZURE_CLIENT_SECRET

You mustn’t set security.protocol: SASL_SSL and sasl.mechanism: PLAIN! Otherwise, you will be haunted by the “Node -1 disconnected” error.

You can find all available options in the documentation. However, I recommend looking at the .NET documentation containing actual explanations.

(Why the second-class treatment for Java developers, Microsoft?)

The Ultimate Most Optimal Perfect Event Hub Settings for Kafka

(All the clickbaity video titles in my YouTube feed may have finally gotten to me.)

I have one more thing for you. Microsoft has published a list of recommended configuration options for Kafka clients. If you want to include them in your Spring Cloud Stream application, you can do it the following way.

spring:
  cloud:
    stream:
      kafka:
        binder:
          configuration:
            metadata.max.age.ms: 180000
            connections.max.idle.ms: 180000
          consumerProperties:
            heartbeat.interval.ms: 3000
            session.timeout.ms: 30000
            max.poll.interval.ms: 300000
          producerProperties:
            max.request.size: 1000000
            retries: 3
            request.timeout.ms: 30000
            metadata.max.idle.ms: 180000
            linger.ms: 50
            delivery.timeout.ms: 90150
            compression.type: none

Big Picture Mode

To put it all together, here is a complete sample configuration.

spring:
  cloud:
    stream:
      kafka:
        binder:
          brokers: [EVENT-HUB-NAMESPACE].servicebus.windows.net:9093
          autoCreateTopics: false
          configuration:
            metadata.max.age.ms: 180000
            connections.max.idle.ms: 180000
#            security.protocol: SASL_SSL
#            sasl.mechanism: PLAIN
#            sasl.jaas.config: org.apache.<snip-the-long-string>;EntityPath={EVENT-HUB-NAME}";
          consumerProperties:
            heartbeat.interval.ms: 3000
            session.timeout.ms: 30000
            max.poll.interval.ms: 300000
          producerProperties:
            max.request.size: 1000000
            retries: 3
            request.timeout.ms: 30000
            metadata.max.idle.ms: 180000
            linger.ms: 50
            delivery.timeout.ms: 90150
            compression.type: none
      function:
        definition: requestMessageProducer;requestMessageConsumer;fakeNewsError
      bindings:
        requestMessageConsumer-in-0:
          destination: fake-news
          group: impartial-outlet
          error-handler-definition: fakeNewsError
          consumer:
            maxAttempts: 1
        requestMessageProducer-out-0:
          destination: fake-news
          producer:
            partitionKeyExpression: payload.type

Famous Last Words

The moral of the story is that Microsoft apparently prefers to provide for .NET developers more than for the Java world. Secondly, and I know this is not easy, it is terrible that the tutorials are inaccurate. Microsoft has a ton of excellent documentation on Azure and all of its complex workings. I am sure keeping it all up-to-date is a full-time job for a sizeable team. It does not change the fact that I have lost countless hours, nerves, and hair (joking; there is nothing left to lose) figuring this out.

Anyway, I hope I could save you some time so you can actually enjoy developing with Spring for Azure.

Thank you for reading.