Basic Hystrix Circuit Breaker Implementation

Circuit Breakers are one of the best tools in our toolbox to use while creating a backend structure with dependent modules. In this post, I will try to set up a basic Hystrix Circuit Breaker example and demonstrate some different configurations.

For demonstrating this concept I’ve created two Spring Boot applications called producer and consumer. The producer returns an integer counter and increases the value after each request but when the counter is between 5 and 20 it will wait for 5000 ms before returning the counter.

@RestController
public class ProducerController {

    AtomicInteger counter = new AtomicInteger(0);

    @RequestMapping(value = "/getCounter", method = RequestMethod.GET)
    public Integer getCounter() throws InterruptedException {
        int k = counter.incrementAndGet();
        if (k > 5 && k < 20) {
            Thread.sleep(5000);
        }
        return counter.intValue();
    }
}
@Service
public class ConsumerService {

  private final RestTemplate restTemplate;

  public ConsumerService(RestTemplate rest) {
   this.restTemplate = rest;
  }

  public String consume() {
   URI uri = URI.create("http://localhost:8080/getCounter");
   return this.restTemplate.getForObject(uri, String.class);
  }
}
@RestController
@SpringBootApplication
public class ConsumerApplication {

  @Autowired
  private ConsumerService consumerService;

  @Bean
  public RestTemplate rest(RestTemplateBuilder builder) {
   return builder.build();
  }

  @RequestMapping("/getCurrentCounter")
  public String toRead() {
   return consumerService.consume();
  }

  public static void main(String[] args) {
   SpringApplication.run(ConsumerApplication.class, args);
  }
}

Consumer Microservice is receiving the counter and returning it to the caller. Currently, there isn’t any circuit breaker or fallback method. So if the Producer Microservice goes down, Consumer Microservice will return an error to the client.

GET http://localhost:8080/getCounter
 org.apache.http.conn.HttpHostConnectException: 
Connect to localhost:8080 [localhost/127.0.0.1] 
failed: Connection refused (Connection refused)

When we start the producer MS it will work just fine. Retrieving the counter and returning it back to the client until it gets to my intentionally slow request. Then it will take approximately 5000 ms to get a response back from the producer. This is an issue because it is keeping our precious threads busy. You can see the results below. It is evident that we need to do something about it.

GET http://localhost:8080/getCounter
 HTTP/1.1 200 
 Content-Type: application/json
 Transfer-Encoding: chunked
 Date: Sun, 04 Apr 2021 10:13:59 GMT
 Keep-Alive: timeout=60
 Connection: keep-alive
 1
 Response code: 200; Time: 96ms; Content length: 1 bytes
GET http://localhost:8080/getCounter
 HTTP/1.1 200 
 Content-Type: application/json
 Transfer-Encoding: chunked
 Date: Sun, 04 Apr 2021 11:03:36 GMT
 Keep-Alive: timeout=60
 Connection: keep-alive
 15
 Response code: 200; Time: 5007ms; Content length: 1 bytes

What is a Circuit Breaker?

The circuit breaker is a design pattern that allows developers to handle connections with unreliable services. So if you have a system that keeps getting down for no reason that you have to connect, you can set up a circuit breaker while making this connection. By doing this you will have the chance to stop making requests to a service that won’t answer.

Circuit breaker is a design pattern used in software development. It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure, or unexpected system difficulties.

Wikipedia – Circuit breaker design pattern

Implementing the Circuit Breaker Pattern

So, how can we implement this into our sample project? First, we need to add the dependency for Hystrix Circuit Breaker to our build.gradle file.

implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix'

After adding the dependency we need to add the @EnableCircuitBreaker annotation to our application class.

Now we’re ready to use the Hystrix Circuit Breaker annotation on our methods. We will modify the Consumer Service by adding @HystrixCommand annotation to our method and creating a fallback method. After these modifications when our producer microservice is down consumer microservice won’t try to call it. Instead, the fallback method will be called. These are the modified versions of our two classes, ConsumerService, and ConsumerApplication.

@Service
public class ConsumerService {

    private final RestTemplate restTemplate;

    public ConsumerService(RestTemplate rest) {
        this.restTemplate = rest;
    }

    @HystrixCommand(fallbackMethod = "fallback")
    public String consume() {
        URI uri = URI.create("http://localhost:8080/getCounter");
        return this.restTemplate.getForObject(uri, String.class);
    }

    @SuppressWarnings("unused")
    public String fallback() {
        return "Can't retrieve the real value, returning the default: 1000";
    }
}
@EnableCircuitBreaker
@RestController
@SpringBootApplication
public class ConsumerApplication {

  @Autowired
  private ConsumerService consumerService;

  @Bean
  public RestTemplate rest(RestTemplateBuilder builder) {
   return builder.build();
  }

  @RequestMapping("/getCurrentCounter")
  public String toRead() {
   return consumerService.consume();
  }

  public static void main(String[] args) {
   SpringApplication.run(ConsumerApplication.class, args);
  }
}

After building and running Consumer Microservice now, even if Producer is down, it will give a default response by calling the fallback method.

GET http://localhost:8090/getCurrentCounter
 HTTP/1.1 200 
 Content-Type: text/plain;charset=UTF-8
 Content-Length: 58
 Date: Sun, 04 Apr 2021 17:01:47 GMT
 Keep-Alive: timeout=60
 Connection: keep-alive
 Can't retrieve the real value, returning the default: 1000
 Response code: 200; Time: 216ms; Content length: 58 bytes

Open the circuit after a certain threshold

So we’ve overcome the issue of failing if the produces is down but this is not enough. Remember our producer microservice responds really slow to our requests from 5th to 20th. What we need to do is to configure our circuit breaker to detect this outage and stop making requests and keeping threads busy. To do this we need to add a couple of configurations to our @HystrixCommand annotation. So the finished annotation would look like this.

@HystrixCommand(fallbackMethod = "fallback", commandProperties = {
        @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "500"),
        @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"),
        @HystrixProperty(name = "fallback.isolation.semaphore.maxConcurrentRequests", value = "1"),
        @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "50")
})
public String consume() {
    URI uri = URI.create("http://localhost:8080/getCounter");
    return this.restTemplate.getForObject(uri, String.class);
}

Of course, I exaggerated the parameters a little bit to test my case. You can see what are the parameters and how to set them from here. After this configuration, I wrote a simple bash script to keep sending requests to my consumer microservice.

!/bin/bash
 for i in {1..95}
 do
     echo -n "$i -->"
     curl http://localhost:8090/getCurrentCounter
     echo " "
     sleep 0.1
 done

Normally if there weren’t any additional options this script should show that only requests from 5th to 20th get a return value from fallback but because we’ve put up some threshold limits after detecting the outage circuit breaker will stay open until the end of the window. After the failed attempt count in the active window went down to a certain percentage it will try again. This is the output of the bash script above.

1 -->Can't retrieve the real value, returning the default: 1000 
 2 -->2 
 3 -->3 
 4 -->4 
 5 -->5 
 6 -->Can't retrieve the real value, returning the default: 1000 
 7 -->Can't retrieve the real value, returning the default: 1000 
 8 -->Can't retrieve the real value, returning the default: 1000 
 9 -->Can't retrieve the real value, returning the default: 1000 
 10 -->Can't retrieve the real value, returning the default: 1000 
 11 -->Can't retrieve the real value, returning the default: 1000 
 12 -->Can't retrieve the real value, returning the default: 1000 
 13 -->Can't retrieve the real value, returning the default: 1000 
 14 -->Can't retrieve the real value, returning the default: 1000 
 15 -->Can't retrieve the real value, returning the default: 1000 
 16 -->Can't retrieve the real value, returning the default: 1000 
 17 -->Can't retrieve the real value, returning the default: 1000 
 18 -->Can't retrieve the real value, returning the default: 1000 
 19 -->Can't retrieve the real value, returning the default: 1000 
 20 -->Can't retrieve the real value, returning the default: 1000 
 21 -->Can't retrieve the real value, returning the default: 1000 
 22 -->Can't retrieve the real value, returning the default: 1000 
 23 -->Can't retrieve the real value, returning the default: 1000 
 24 -->Can't retrieve the real value, returning the default: 1000 
 25 -->Can't retrieve the real value, returning the default: 1000 
 26 -->Can't retrieve the real value, returning the default: 1000 
 27 -->Can't retrieve the real value, returning the default: 1000 
 28 -->Can't retrieve the real value, returning the default: 1000 
 29 -->Can't retrieve the real value, returning the default: 1000 
 30 -->Can't retrieve the real value, returning the default: 1000 
 31 -->Can't retrieve the real value, returning the default: 1000 
 32 -->Can't retrieve the real value, returning the default: 1000 
 33 -->Can't retrieve the real value, returning the default: 1000 
 34 -->Can't retrieve the real value, returning the default: 1000 
 35 -->Can't retrieve the real value, returning the default: 1000 
 36 -->Can't retrieve the real value, returning the default: 1000 
 37 -->Can't retrieve the real value, returning the default: 1000 
 38 -->Can't retrieve the real value, returning the default: 1000 
 39 -->Can't retrieve the real value, returning the default: 1000 
 40 -->Can't retrieve the real value, returning the default: 1000 
 41 -->Can't retrieve the real value, returning the default: 1000 
 42 -->Can't retrieve the real value, returning the default: 1000 
 43 -->Can't retrieve the real value, returning the default: 1000 
 44 -->Can't retrieve the real value, returning the default: 1000 
 45 -->Can't retrieve the real value, returning the default: 1000 
 46 -->Can't retrieve the real value, returning the default: 1000 
 47 -->Can't retrieve the real value, returning the default: 1000 
 48 -->Can't retrieve the real value, returning the default: 1000 
 49 -->Can't retrieve the real value, returning the default: 1000 
 50 -->Can't retrieve the real value, returning the default: 1000 
 51 -->Can't retrieve the real value, returning the default: 1000 
 52 -->Can't retrieve the real value, returning the default: 1000 
 53 -->Can't retrieve the real value, returning the default: 1000 
 54 -->Can't retrieve the real value, returning the default: 1000 
 55 -->Can't retrieve the real value, returning the default: 1000 
 56 -->Can't retrieve the real value, returning the default: 1000 
 57 -->Can't retrieve the real value, returning the default: 1000 
 58 -->Can't retrieve the real value, returning the default: 1000 
 59 -->Can't retrieve the real value, returning the default: 1000 
 60 -->Can't retrieve the real value, returning the default: 1000 
 61 -->Can't retrieve the real value, returning the default: 1000 
 62 -->Can't retrieve the real value, returning the default: 1000 
 63 -->Can't retrieve the real value, returning the default: 1000 
 64 -->Can't retrieve the real value, returning the default: 1000 
 65 -->Can't retrieve the real value, returning the default: 1000 
 66 -->Can't retrieve the real value, returning the default: 1000 
 67 -->Can't retrieve the real value, returning the default: 1000 
 68 -->Can't retrieve the real value, returning the default: 1000 
 69 -->Can't retrieve the real value, returning the default: 1000 
 70 -->Can't retrieve the real value, returning the default: 1000 
 71 -->Can't retrieve the real value, returning the default: 1000 
 72 -->Can't retrieve the real value, returning the default: 1000 
 73 -->Can't retrieve the real value, returning the default: 1000 
 74 -->Can't retrieve the real value, returning the default: 1000 
 75 -->Can't retrieve the real value, returning the default: 1000 
 76 -->Can't retrieve the real value, returning the default: 1000 
 77 -->Can't retrieve the real value, returning the default: 1000 
 78 -->Can't retrieve the real value, returning the default: 1000 
 79 -->Can't retrieve the real value, returning the default: 1000 
 80 -->Can't retrieve the real value, returning the default: 1000 
 81 -->Can't retrieve the real value, returning the default: 1000 
 82 -->Can't retrieve the real value, returning the default: 1000 
 83 -->Can't retrieve the real value, returning the default: 1000 
 84 -->Can't retrieve the real value, returning the default: 1000 
 85 -->Can't retrieve the real value, returning the default: 1000 
 86 -->Can't retrieve the real value, returning the default: 1000 
 87 -->Can't retrieve the real value, returning the default: 1000 
 88 -->Can't retrieve the real value, returning the default: 1000 
 89 -->Can't retrieve the real value, returning the default: 1000 
 90 -->Can't retrieve the real value, returning the default: 1000 
 91 -->20 
 92 -->21 
 93 -->22 
 94 -->23 
 95 -->24 

As you can see from the output normally we would expect it only to return the default response for requests from 5 to 20 but it did not try to connect until the 91st request. Because the circuit was opened and did not allow requests to go through. After enough time passed it is reopened and tried to connect again.

This is the view from the Hystrix Monitor while the application is running.

Hystrix Monitor

Thanks for reading! You can see all my posts about Java by clicking here!

Leave a Reply