Lost in a Forest You Built

When NovaPay hit 50 engineers, their standup meetings started running 45 minutes.

It crept up slowly. At 20 people, standups were fifteen minutes. At 35, they hit half an hour, more cross-team coordination, more “I’m blocked on…” updates, more “let’s take this offline” deferrals that never actually happened. By 50, the morning ritual had become a performance review of the previous day’s merge conflicts.

Maria, their VP of Engineering, didn’t notice at first. She was too busy hiring. NovaPay had just closed their Series B. The board wanted to see headcount translate into velocity. Instead, velocity was flat. And getting flatter.

This is the story of how NovaPay’s architecture started working against them. I’ve worked on this pattern enough times that I can tell the story in aggregate. NovaPay is fictionalized. The arc isn’t.

The First Crack: Invisible Boundaries

The first sign appeared in code review.

A junior engineer on the payments team submitted what should have been a simple PR: add a new payment method. Twenty-three files changed. Seven across “payments,” four in “fraud detection,” five in “ledger,” seven in “notifications.”

“Why does adding Klarna touch fraud detection?” Maria asked.

“The fraud scoring module has a helper function we need,” the engineer explained. “And the ledger needs to know about the payment type for settlement. And notifications needs the payment method name for receipts.”

The monolith had internal structure: modules, packages, folders that different teams “owned.” But those boundaries existed only in the minds of engineers. The runtime didn’t know them, the database didn’t enforce them, and the ORM happily let anyone import anything.

Over three years of growth, these small violations had accumulated. Payments reached into fraud, fraud reached into users, and users reached into ledger. The modules were supposed to be separate, but in practice they were one intertwined mass.

The benchmark that should have been a warning: Deploys required coordination between 4+ teams. More than 30% of imports crossed module boundaries. A “small change” routinely touched files across the codebase.

The Slow Burn: Database as Integration Layer

Six months later, the mobile team shipped a new feature: real-time balance updates.

The obvious solution: read directly from the ledger database. Fast, simple, and no API to build, so it shipped in two weeks. The PM was thrilled.

What nobody documented: the mobile app was now coupled to the ledger schema. When the ledger team needed to optimize their balance calculation, they discovered the mobile app would break. Then they discovered the fraud team was also reading from that table. And the settlement batch job. And a compliance reporting script someone had written in 2021 and forgotten about.

Four services, one table, zero coordination. The database had become the integration layer, a pattern that works until it doesn’t.

The breaking point came during a routine schema migration. The ledger team added an index to improve query performance. The migration ran in staging and looked fine, so they deployed to production.

The mobile app’s balance queries started timing out. The fraud team’s risk scoring slowed to a crawl. Customer support tickets spiked. The incident took six hours to resolve. Not because the fix was hard, but because nobody knew the full blast radius.

The benchmark that should have been a warning: 4 services wrote to shared tables. Schema changes required coordinating multiple teams. Shared tables had no clear owner.

The Tipping Point: Deploy Pipeline from Hell

The engineering team had a saying: “Never deploy on Friday.” Then it became “Never deploy after 2 PM.” Then it became “Never deploy unless you can babysit it for three hours.”

When NovaPay started, deploys took ten minutes. By the time Maria noticed the standups problem, deploys took four hours. The test suite alone ran ninety minutes, most of it waiting for sandbox API rate limits on payment gateway integrations. Staging didn’t mirror production card networks, so nobody trusted the results anyway. Then there were the approval gates: security sign-off, compliance review, product owner blessing.

Engineers adapted. They batched changes and feature branches lived for weeks. “I’ll ship this with my next PR” became common vocabulary. Each merge became a bigger event, with more risk, requiring more coordination.

One Thursday, a critical payment bug was discovered. Money was being double-charged, and the fix was three lines. Getting those three lines to production took eleven hours, through the full deploy gauntlet. Customers were charged incorrectly for a full business day. For a company processing millions in monthly volume, every hour of double-charging erodes trust that takes months to rebuild, and the regulatory exposure doesn’t help either.

The benchmark that should have been a warning: CI took 90+ minutes. Deploys took 4+ hours end-to-end. Feature branches averaged 8 days before merge.

The Human Cost: On-Call Burnout

The settlement batch job failed at 2:47 AM on a Tuesday.

The on-call engineer — two months at the company — opened the runbook. Step one: “Check the reconciliation service logs.” She didn’t know where reconciliation service logs lived. Step two: “If the error is XYZ, run recovery script.” The error wasn’t XYZ. Step six: “Escalate to James.” James had changed teams eight months ago. His replacement had quit.

She woke up two other engineers. Neither knew the settlement system, so they found someone who used to work on it, now a manager who hadn’t written code in a year. Together, they pieced together enough to run a manual recovery. The partner bank’s batch window closed in ninety minutes. They made it with twelve minutes to spare.

The next morning, two engineers asked about switching to teams with lighter on-call rotations. Within a month, one had left the company entirely.

The benchmark that should have been a warning: Engineers were paged 4+ times per week. Over 50% of incidents required escalation beyond on-call. Runbooks were missing or wrong for most critical systems.

The Knowledge Crisis

The engineer who built the core banking integration left eighteen months ago. The one who understood SWIFT message formats was interviewing at competitors. The PCI compliance audit was in three months.

“Why is encryption implemented this way?” the auditor would ask. The answer existed somewhere: in a Slack thread from 2021, in a design doc that nobody could find, in the head of someone who no longer worked there.

New hires took four to six months to become productive. Projects stalled when specific engineers went on vacation, and “ask Sarah, she knows how that works” was heard a dozen times a day. Sarah was exhausted and quietly updating her LinkedIn.

The benchmark that should have been a warning: New engineers couldn’t ship meaningful features for 4+ months. 5 critical systems had single-point-of-knowledge dependency. Documentation coverage was below 20%.

The Turning Point

Maria finally called an all-hands.

“We need to rewrite the platform,” suggested the new VP of Product. He’d seen this movie before at his last company.

“We can’t stop shipping features for eighteen months,” Maria replied. “Our competitors won’t wait.”

This is the moment where companies usually plateau. The big rewrite is too risky, but doing nothing means slowly sinking. Most teams choose a third option: denial. They hire more engineers, hoping bodies will compensate for friction. It never works.

Maria chose a different path.

The Evolution

Instead of a rewrite, NovaPay drew boundaries, starting with the most painful bottleneck.

The question was: where to start? Maria’s team ranked their modules by three criteria: incident frequency, number of cross-team dependencies, and deployment coupling. The settlement system scored highest on all three. It was causing the worst on-call incidents, touched the most teams, and was the scariest thing to change. That made it the right first target: high pain meant high visibility for the approach.

Where to start: Choose a boundary where incident pressure, cross-team dependencies, and deployment coupling overlap. Give the extraction a named owner and protected capacity; adding it to everyone’s spare-time backlog leaves the same risks in place.

That is the trail to follow in legacy modernization: reduce a specific operational risk while keeping the rest of the business moving, rather than making every team wait for a replacement platform.

They staffed a dedicated extraction team of four engineers: two who knew the settlement domain deeply and two strong generalists. This wasn’t 20% time or a side project. It was their full-time job, protected from feature work by Maria personally. When product managers pushed back, she had a ready answer: “Every month without clear service boundaries increases the blast radius of the next incident.”

The extraction followed the strangler fig pattern. The new settlement service went up alongside the monolith. An API gateway routed traffic: new requests went to the new service, legacy paths stayed on the monolith. They ran dual writes during the transition: every transaction written to both the old shared tables and the new service’s private database, with automated reconciliation catching discrepancies. Feature flags controlled the cutover, module by module.

The old database tables became that service’s private domain. Other systems that used to read directly from shared tables now had to call the API. This was the hardest sell. Teams that had built features on direct database access had to rewrite their queries as API calls. But every team that made the switch stopped getting paged for settlement incidents.

The architecture went from this:

Before: App → Shared DB ← Mobile, Fraud, Settlement, Compliance (all reading and writing the same tables)

To this:

After: App → Settlement API → Settlement DB (private). Mobile, Fraud, Compliance → Settlement API (contract-based access only)

Three months of careful migration with zero downtime. When it was done, the settlement team could deploy independently. On-call incidents dropped 60%. The runbook finally made sense because there was a clear owner. The team celebrated by deploying twice in one day, something that would have been unthinkable three months earlier.

Then they tackled the next bottleneck: the fraud scoring module. Same pattern: rank the pain, staff a dedicated team, strangle the old code behind an API, let the team own their deploys. Months four through six. By month seven, the pattern was well-understood enough that product squads started extracting their own modules without the dedicated team’s help.

One by one, the monolith shrank. The boundaries that existed only in engineers’ minds became real, enforced by network calls and API contracts. Deploys got faster, feature branches got shorter, and standups got shorter.

Keeping motivation up over eighteen months of extraction work was its own challenge. Maria made each successful extraction visible: the team that could now deploy independently got recognized at the company all-hands. Progress was tracked on a public dashboard: percentage of traffic routed through clean APIs versus the monolith. Engineers could watch the monolith shrink in real time.

It took eighteen months of steady, unglamorous extraction with no big bang and no platform freeze. Features kept shipping throughout. The strangler fig pattern meant the old code still worked while the new code proved itself. Work that doesn’t make for conference talks, but makes everything else possible.

By the time they finished, NovaPay had 80 engineers. Standups were fifteen minutes again.

Is This Your Company?

The specifics will differ, but the arc is consistent.

If you’re recognizing these patterns (the coordination overhead, the shared databases, the deploy dread, the on-call burnout, the knowledge silos), your architecture is working against your growth. And every engineer who leaves takes four to six months of ramp-up time with them, replacing it with another four to six months of onboarding cost for their successor. Technical debt compounds through attrition.

The answer is incremental evolution, not a rewrite: identify the most painful bottleneck, draw a boundary around it, extract it, make it better. Then move to the next one.

The hardest part is usually the first decision, not the extraction itself, but choosing where to start. Rank your pain points by incident frequency, cross-team coupling, and deployment friction. The module that scores highest on all three is your first target. High pain means high visibility, which means organizational buy-in for the approach.

If you need to make the case to leadership: frame it as risk reduction. Every month without clear service boundaries increases the blast radius of the next incident, and in fintech, incidents have regulatory consequences.

Seeing these patterns? The Pointer Assessment is the diagnostic phase: a prioritized view of bottlenecks and a starting point for extraction. The Chase is the execution phase, working through that extraction alongside your team. Let’s figure out where to start.