The Trail That Went Cold

The message came late Wednesday afternoon: “No data is arriving.”

Production. Customers paying hundreds per month for real-time data, data they use to make business decisions, were staring at stale dashboards. Our peer-to-peer RSocket mesh had gone silent.

Our customer success lead was already fielding calls. The CEO had pinged Slack twice. A Friday evening about to disappear into the void.

The Silence

Monitoring dashboards showed healthy connections with services up, heartbeats flowing, and every metric green. But customers weren’t getting their data.

Logs showed nothing: no errors, no warnings. Just successful health checks and the absence of actual payload messages. The connection was alive. The channel was open. Data was being sent. It vanished somewhere in between.

The Documentation Trail

I checked the RSocket protocol specification. Our implementation matched the examples. Request streams, bidirectional communication, peer-to-peer topology. All correct.

Hours of documentation, cross-referencing, staring at code that looked exactly like the examples. Nothing worked.

Then I found it, buried in DEBUG output, hidden among thousands of routine log lines:

CanceledException (0x203): The payload is too big to be send as a single frame
with a max frame length 16777215. Consider enabling fragmentation

16,777,215 bytes — about 16 megabytes. The RSocket protocol’s hard frame limit.

Our batch aggregation normally ran around 2MB, but after a busy day it had grown past the limit. Finally, a clue.

The Configuration Maze

RSocket uses a 24-bit field for frame length, a hard protocol limit. For payloads over 16MB, you need fragmentation.

The Spring Boot docs showed the fix:

spring:
  rsocket:
    server:
      fragment-size: 1MB

I deployed the fix and went home thinking the problem was solved.

The next morning it was still broken, with the same error and the same silent failure. The only new clue:

io.rsocket.exceptions.ConnectionErrorException: Connection closed unexpectedly

The config was in place, the documentation followed to the letter, but it was still failing. The YAML looked correct, the IDE autocompleted it, and the docs said it should work.

Into the Source

There’s a moment in debugging when you realize the documentation isn’t going to save you. When the official answer doesn’t work and you have to dig deeper. I opened Spring Boot’s RSocket autoconfiguration source.

The entire server configuration class, the one that would read my fragment-size property, was wrapped in a conditional:

@ConditionalOnProperty("spring.rsocket.server.port")
@Configuration(proxyBeanMethods = false)
static class EmbeddedServerConfiguration {
    // ...
    map.from(properties.getServer().getFragmentSize())
       .to(factory::setFragmentSize);
    // ...
}

And there it was.

We were running RSocket over WebSocket, sharing the HTTP port. spring.rsocket.server.port was never set. The entire configuration class was never loaded. My YAML property was being parsed, validated, stored but never applied.

The fix required bypassing the autoconfiguration entirely and configuring fragmentation programmatically on the server side:

@Configuration
class RSocketConfig {

    @Bean
    fun rSocketServerCustomizer() = RSocketServerCustomizer { server ->
        server.fragment(1024 * 1024)  // 1MB server-side fragments
    }
    
}

Direct configuration on both sides of the connection, bypassing the YAML and conditional annotations that never applied.

The Fix

I deployed, and data flowed. The batch that had been failing for days went through cleanly. A handful of lines of code, after two days of hunting. The customer success lead could finally stop apologizing.

You can reproduce this yourself: the companion repository includes tests that prove each stage — large batches failing without config, still failing with YAML config, and succeeding only with the programmatic fix.

The Lesson

This bug illustrates a systemic risk in modern frameworks: validated configuration that silently doesn’t apply.

The @ConditionalOnProperty pattern is everywhere in Spring Boot. Your YAML parses without error. Your IDE autocompletes the properties. The mapping code exists. But a conditional annotation somewhere in the autoconfiguration stack decides your deployment topology doesn’t qualify, and your configuration is silently ignored.

This is the particular hell of framework abstraction layers: each layer makes assumptions about how you’re deploying. RSocket assumes you might use WebSocket transport. Spring Boot assumes you might run on a dedicated port. When those assumptions intersect in unexpected ways, you get configuration that looks correct, validates correctly, and does nothing.

In system architecture reviews, those deployment assumptions deserve the same scrutiny as service boundaries. Follow the trail from configuration to runtime behavior, and verify that a healthy connection is actually delivering the data customers depend on.

---

Key Takeaways

The business problem: Real-time data customers paying hundreds per month saw stale dashboards for days. Monitoring showed all green. Logs showed no errors.

The technical cause: A framework configuration property was silently ignored due to deployment topology assumptions baked into Spring Boot’s autoconfiguration.

Why this matters to your organization: This pattern (validated config that silently doesn’t apply) exists in every Spring Boot application using @ConditionalOnProperty. It’s almost certainly present in yours. The question isn’t whether you have silent configuration failures. The question is whether you’ll discover them before or after they cost you customers.

The capability question for engineering leaders: How many of your team can debug past the documentation layer? When the framework lies to you, who on your team knows how to find the truth?

Reproduce it yourself: Clone the companion repository and run ./gradlew test to see all three scenarios (broken, YAML-fix-that-isn’t, programmatic fix) pass or fail as described.

---

Dealing with mysterious microservices failures? Let’s talk.