Skip to main content

🐞 Production Bug: 100% CPU Spike from Empty Catch Block — Fixed with Spring Retry! 😮🔥

  🌪️ The Chaos Begins

It was a normal day in production... until suddenly, one server started heating up 🔥. And I don’t mean metaphorically — its CPU usage shot up to 100% and stayed there, like it had chugged 10 cups of espresso ☕💥.

No OutOfMemoryError.
No Exceptions.
No logs screaming for help.
Just... silence and a fan spinning like a helicopter 🌀.

🧩 The Initial Clues

We SSH'd into the server and ran:


top -H -p <java_process_id>

We noticed one thread hogging the CPU — continuously.

Then we used VisualVM to peek inside.
Stack trace of that CPU-hogging thread looked like this:


at com.example.MyProcessor.process(MyProcessor.java:42)
at com.example.MyProcessor.run(MyProcessor.java:30)
...

Hmm… That class wasn’t doing anything fancy. Just a retry mechanism inside a loop.

🔍 The Suspect Code

Here’s what the code looked like:


while (true) {
    try {
        doSomeCriticalOperation();
        break; // Exit on success
    } catch (Exception e) {
        // 🙈 Empty catch — suppresses all errors silently
    }
}

Looks innocent, right? But this was the silent killer. 😬

🧠 What's Actually Happening?

Let’s break it down:

  • doSomeCriticalOperation() was failing due to a misconfiguration in a service dependency.

  • The exception was caught and silently ignored.

  • The loop kept retrying immediately — without delay, logging, or even sleeping.

  • This meant the thread entered a tight loop, running full-speed without yielding the CPU.

💡 Result: One thread spins at 100%, wasting CPU cycles, doing absolutely nothing productive.

Now imagine this inside a microservice deployed across 10 pods. If every pod had even one such spinning thread… 💥 chaos.

🧯 How We Fixed It (Spring Retry to the Rescue!)

Instead of writing manual retry loops (which can go wrong as we saw), we went with a production-grade solution: Spring Retry — and it was a game changer.

✅ Spring Retry — Simple Yet Powerful!

Step 1: Add dependencies (Maven):


<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-aspects</artifactId>
</dependency>

Step 2: Enable Retry in Spring Boot App


@EnableRetry
@SpringBootApplication
public class MyApp { }

Step 3: Annotate the method you want to retry


@Service
public class MyService {

    @Retryable(
        value = { RemoteServiceException.class },
        maxAttempts = 3,
        backoff = @Backoff(delay = 2000, multiplier = 2))
    public void doSomeCriticalOperation() {
        // Your code that may fail temporarily
        System.out.println("Trying critical operation...");
        throw new RemoteServiceException("Oops! Remote system failed");
    }

    @Recover
    public void recover(RemoteServiceException e) {
        // This is called if all retries fail
        System.out.println("Recovering gracefully after retries failed");
    }
}

🤔 What’s Happening Here? — Let Me Explain

🔁 @Retryable — Automatic Retry Logic

When you annotate a method with @Retryable, Spring will automatically retry that method if it throws the specified exception.

Here's what this means in plain English:


@Retryable(
  value = { RemoteServiceException.class },
  maxAttempts = 3,
  backoff = @Backoff(delay = 2000, multiplier = 2)
)

💡 Breakdown:

  • value: What exception(s) to retry on. Here, RemoteServiceException.

  • maxAttempts = 3: Try a total of 3 times (initial + 2 retries).

  • backoff: Adds a delay between attempts.

    • delay = 2000: Start with 2 seconds delay.

    • multiplier = 2: Delay grows exponentially (2s → 4s → 8s...).

So, if the method fails:

  • First retry after 2 seconds,

  • Second retry after 4 seconds,

  • Third retry after 8 seconds,

  • ... then give up.

🛑 @Recover — Graceful Fallback After All Retries Fail

If even after all retries the method still keeps failing, the method annotated with @Recover will be called.

This is like your “Plan B” — for logging, alerts, or triggering a failover.

⚠️ Very Important:
The method signature of @Recover must match the original method + the exception type as the first parameter.

❓ Common Doubt: Will @Recover Be Called If Retry Succeeds?

Great question — and it’s one that confuses many people!

🧠 Question:

If my @Retryable method fails once, but then succeeds on the second retry, will @Recover still be called?

🟢 Answer:

No. @Recover is only called if all retry attempts fail.

Here’s what happens:

  • ✅ If the method succeeds in any retry → Spring considers it successful → @Recover is not called.

  • ❌ If it fails in all attempts → @Recover will be triggered with the last thrown exception.

📊 Quick Example:

With this config:


@Retryable(
    value = RemoteServiceException.class,
    maxAttempts = 3,
    backoff = @Backoff(delay = 1000)
)

  • 1st attempt fails

  • 2nd attempt succeeds ✅
    → @Recover will NOT be called

But if:

  • 1st attempt fails

  • 2nd attempt fails

  • 3rd attempt fails
    → Now Spring gives up and calls the method annotated with @Recover 🛑

✅ Always remember:

Spring Retry calls @Recover only as a final fallback — not during intermediate retries.

💡 Why It’s Better Than Manual Retry

✅ No tight loops
✅ No wasted CPU
✅ Controlled retry with backoff
✅ Easy configuration
✅ Better observability
✅ Cleaner code = fewer bugs

🎁 Final Thought

Replacing our naive while(true) with Spring Retry instantly improved the system’s resilience and clarity. And the best part? Even junior developers could now read and understand what’s going on — thanks to annotations and structured retries.

🔚 Wrapping Up

This bug taught me one thing:
Even something as small as an empty catch block can trigger chaos. And with tools like Spring Retry, we don’t just patch things — we make our systems smarter and safer. 💪

So next time something fails — don’t spin in circles 🔄.
Retry smart, sleep well. 😴

Have you used Spring Retry before? Or hit similar retry nightmares? Drop your stories in the comments — I’d love to hear them!

🪄 Written by: Anand @ JavaBeanBag

Labels:

Comments

Post a Comment

Popular posts from this blog

🔐 Is final Really Final in Java? The Truth May Surprise You 😲

💬 “When I was exploring what to do and what not to do in Java, one small keyword caught my eye — final . I thought it meant: locked, sealed, frozen — like my fridge when I forget to defrost it.”   But guess what? Java has its own meaning of final… and it’s not always what you expect! 😅 Let’s break it down together — with code, questions, confusion, jokes, and everything in between. 🎯 The Confusing Case: You Said It's Final... Then It Changed?! 🫠 final List<String> names = new ArrayList <>(); names.add( "Anand" ); names.add( "Rahul" ); System.out.println(names); // [Anand, Rahul] 🤯 Hold on... that’s final , right?! So how on earth is it still changing ? Time to dive deeper... 🧠 Why Is It Designed Like This? Here’s the key secret: In Java, final applies to the reference , not the object it points to . Let’s decode this like a spy mission 🕵️‍♂️: Imagine This: final List<String> names = new ArrayList <>(); Be...
Post a Comment
Read more

🎢 Java Loops: Fun, Fear, and ForEach() Fails

🌀 Oops, I Looped It Again! — The Ultimate Java Loop Guide You Won't Forget “I remember this question from one of my early interviews — I was just 2 years into Java and the interviewer asked, ‘Which loop do you prefer and why?’” At first, I thought, “Duh! for-each is cleaner.” But then he grilled me with cases where it fails. 😵 That led me to explore all loop types, their powers, and their pitfalls. Let’s deep-dive into every major Java loop with examples &  real-world guidance so you'll never forget again. 🔁 Loop Type #1: Classic For Loop — “The Old Reliable” ✅ When to Use: You need an index You want to iterate in reverse You want full control over loop mechanics ✅ Good Example: List<String> names = List.of("A", "B", "C"); for (int i = 0; i < names.size(); i++) { System.out.println(i + ": " + names.get(i)); } 🔥 Reverse + Removal Example: List<String> item...
Post a Comment
Read more

🧵 Virtual Threads in Java — The Ultimate Guide with Diagrams, Code & Interview Qs!

🚀 “How are Virtual Threads different from Thread Pools?” 😵 “Are they OS threads or JVM threads?” 🙃 “Should I still use CompletableFuture?” 🤯 “How do I even use them in real-time microservices?” 🧠 What are Virtual Threads? Virtual Threads (introduced in Java 21 as stable 🎉) are lightweight threads managed by the JVM instead of the OS kernel. 👉 They look like normal threads, but don’t hog OS resources like traditional threads. 🧠 What is the OS Kernel? 🏛️ OS Kernel = The Brain of the Operating System It’s the core part of your OS (Windows, Linux, Mac) that: Manages memory 🧠 Schedules threads 🕒 Talks to hardware 💻 Handles I/O operations 📨 When you create a traditional thread in Java, the JVM asks the OS Kernel to create a real OS-level thread. 🖼️ Imagine This... ┌───────────────────────────┐ │ Your Java Application │ └────────────┬──────────────┘ │ ...
Post a Comment
Read more