🐞 Production Bug: 100% CPU Spike from Empty Catch Block — Fixed with Spring Retry! 😮🔥

  🌪️ The Chaos Begins

It was a normal day in production... until suddenly, one server started heating up 🔥. And I don’t mean metaphorically — its CPU usage shot up to 100% and stayed there, like it had chugged 10 cups of espresso ☕💥.

No OutOfMemoryError.
No Exceptions.
No logs screaming for help.
Just... silence and a fan spinning like a helicopter 🌀.

🧩 The Initial Clues

We SSH'd into the server and ran:

top -H -p <java_process_id>

We noticed one thread hogging the CPU — continuously.

Then we used VisualVM to peek inside.
Stack trace of that CPU-hogging thread looked like this:

at com.example.MyProcessor.process(MyProcessor.java:42)
at com.example.MyProcessor.run(MyProcessor.java:30)
...

Hmm… That class wasn’t doing anything fancy. Just a retry mechanism inside a loop.

---

🔍 The Suspect Code

Here’s what the code looked like:

while (true) {
    try {
        doSomeCriticalOperation();
        break; // Exit on success
    } catch (Exception e) {
        // 🙈 Empty catch — suppresses all errors silently
    }
}

Looks innocent, right? But this was the silent killer. 😬

---

🧠 What's Actually Happening?

Let’s break it down:

• doSomeCriticalOperation() was failing due to a misconfiguration in a service dependency.

• The exception was caught and silently ignored.

• The loop kept retrying immediately — without delay, logging, or even sleeping.

• This meant the thread entered a tight loop, running full-speed without yielding the CPU.

💡 Result: One thread spins at 100%, wasting CPU cycles, doing absolutely nothing productive.

Now imagine this inside a microservice deployed across 10 pods. If every pod had even one such spinning thread… 💥 chaos.

---

🧯 How We Fixed It (Spring Retry to the Rescue!)

Instead of writing manual retry loops (which can go wrong as we saw), we went with a production-grade solution: Spring Retry — and it was a game changer.

---

✅ Spring Retry — Simple Yet Powerful!

Step 1: Add dependencies (Maven):

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-aspects</artifactId>
</dependency>

Step 2: Enable Retry in Spring Boot App

@EnableRetry
@SpringBootApplication
public class MyApp { }

Step 3: Annotate the method you want to retry

@Service
public class MyService {

    @Retryable(
        value = { RemoteServiceException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 2000, multiplier = 2))
    public void doSomeCriticalOperation() {
        // Your code that may fail temporarily
        System.out.println("Trying critical operation...");
        throw new RemoteServiceException("Oops! Remote system failed");
    }

    @Recover
    public void recover(RemoteServiceException e) {
        // This is called if all retries fail
        System.out.println("Recovering gracefully after retries failed");
    }
}

---

🤔 What’s Happening Here? — Let Me Explain

🔁 @Retryable — Automatic Retry Logic

When you annotate a method with @Retryable, Spring will automatically retry that method if it throws the specified exception.

Here's what this means in plain English:

@Retryable(
  value = { RemoteServiceException.class },
  maxAttempts = 3,
  backoff = @Backoff(delay = 2000, multiplier = 2)
)

💡 Breakdown:

• value: What exception(s) to retry on. Here, RemoteServiceException.

• maxAttempts = 3: Try a total of 3 times (initial + 2 retries).

• backoff: Adds a delay between attempts.

  • delay = 2000: Start with 2 seconds delay.

  • multiplier = 2: Delay grows exponentially (2s → 4s → 8s...).

So, if the method fails:

• First retry after 2 seconds,

• Second retry after 4 seconds,

• Third retry after 8 seconds,

• ... then give up.

🛑 @Recover — Graceful Fallback After All Retries Fail

If even after all retries the method still keeps failing, the method annotated with @Recover will be called.

This is like your “Plan B” — for logging, alerts, or triggering a failover.

⚠️ Very Important:
The method signature of @Recover must match the original method + the exception type as the first parameter.

---

❓ Common Doubt: Will @Recover Be Called If Retry Succeeds?

Great question — and it’s one that confuses many people!

🧠 Question:

If my @Retryable method fails once, but then succeeds on the second retry, will @Recover still be called?

🟢 Answer:

No. @Recover is only called if all retry attempts fail.

Here’s what happens:

• ✅ If the method succeeds in any retry → Spring considers it successful → @Recover is not called.

• ❌ If it fails in all attempts → @Recover will be triggered with the last thrown exception.

📊 Quick Example:

With this config:

@Retryable(
    value = RemoteServiceException.class,
    maxAttempts = 3,
    backoff = @Backoff(delay = 1000)
)

• 1st attempt fails

• 2nd attempt succeeds ✅
  → @Recover will NOT be called

But if:

• 1st attempt fails

• 2nd attempt fails

• 3rd attempt fails
  → Now Spring gives up and calls the method annotated with @Recover 🛑

---

✅ Always remember:

Spring Retry calls @Recover only as a final fallback — not during intermediate retries.

💡 Why It’s Better Than Manual Retry

✅ No tight loops
✅ No wasted CPU
✅ Controlled retry with backoff
✅ Easy configuration
✅ Better observability
✅ Cleaner code = fewer bugs

---

🎁 Final Thought

Replacing our naive while(true) with Spring Retry instantly improved the system’s resilience and clarity. And the best part? Even junior developers could now read and understand what’s going on — thanks to annotations and structured retries.

---

🔚 Wrapping Up

This bug taught me one thing:
Even something as small as an empty catch block can trigger chaos. And with tools like Spring Retry, we don’t just patch things — we make our systems smarter and safer. 💪

So next time something fails — don’t spin in circles 🔄.
Retry smart, sleep well. 😴

Have you used Spring Retry before? Or hit similar retry nightmares? Drop your stories in the comments — I’d love to hear them!

🪄 Written by: Anand @ JavaBeanBag