Circuit Breakers are one of the best tools in our toolbox to use while creating a backend structure with dependent modules. In this post, I will try to set up a basic Hystrix Circuit Breaker example and demonstrate some different configurations.
For demonstrating this concept I’ve created two Spring Boot applications called producer and consumer. The producer returns an integer counter and increases the value after each request but when the counter is between 5 and 20 it will wait for 5000 ms before returning the counter.
@RestController
public class ProducerController {
AtomicInteger counter = new AtomicInteger(0);
@RequestMapping(value = "/getCounter", method = RequestMethod.GET)
public Integer getCounter() throws InterruptedException {
int k = counter.incrementAndGet();
if (k > 5 && k < 20) {
Thread.sleep(5000);
}
return counter.intValue();
}
}@Service
public class ConsumerService {
private final RestTemplate restTemplate;
public ConsumerService(RestTemplate rest) {
this.restTemplate = rest;
}
public String consume() {
URI uri = URI.create("http://localhost:8080/getCounter");
return this.restTemplate.getForObject(uri, String.class);
}
}@RestController
@SpringBootApplication
public class ConsumerApplication {
@Autowired
private ConsumerService consumerService;
@Bean
public RestTemplate rest(RestTemplateBuilder builder) {
return builder.build();
}
@RequestMapping("/getCurrentCounter")
public String toRead() {
return consumerService.consume();
}
public static void main(String[] args) {
SpringApplication.run(ConsumerApplication.class, args);
}
}Consumer Microservice is receiving the counter and returning it to the caller. Currently, there isn’t any circuit breaker or fallback method. So if the Producer Microservice goes down, Consumer Microservice will return an error to the client.
GET http://localhost:8080/getCounter
org.apache.http.conn.HttpHostConnectException:
Connect to localhost:8080 [localhost/127.0.0.1]
failed: Connection refused (Connection refused)When we start the producer MS it will work just fine. Retrieving the counter and returning it back to the client until it gets to my intentionally slow request. Then it will take approximately 5000 ms to get a response back from the producer. This is an issue because it is keeping our precious threads busy. You can see the results below. It is evident that we need to do something about it.
GET http://localhost:8080/getCounter
HTTP/1.1 200
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 04 Apr 2021 10:13:59 GMT
Keep-Alive: timeout=60
Connection: keep-alive
1
Response code: 200; Time: 96ms; Content length: 1 bytesGET http://localhost:8080/getCounter
HTTP/1.1 200
Content-Type: application/json
Transfer-Encoding: chunked
Date: Sun, 04 Apr 2021 11:03:36 GMT
Keep-Alive: timeout=60
Connection: keep-alive
15
Response code: 200; Time: 5007ms; Content length: 1 bytesWhat is a Circuit Breaker?
The circuit breaker is a design pattern that allows developers to handle connections with unreliable services. So if you have a system that keeps getting down for no reason that you have to connect, you can set up a circuit breaker while making this connection. By doing this you will have the chance to stop making requests to a service that won’t answer.
Circuit breaker is a design pattern used in software development. It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure, or unexpected system difficulties.
Wikipedia – Circuit breaker design pattern
Implementing the Circuit Breaker Pattern
So, how can we implement this into our sample project? First, we need to add the dependency for Hystrix Circuit Breaker to our build.gradle file.
implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix'After adding the dependency we need to add the @EnableCircuitBreaker annotation to our application class.
Now we’re ready to use the Hystrix Circuit Breaker annotation on our methods. We will modify the Consumer Service by adding @HystrixCommand annotation to our method and creating a fallback method. After these modifications when our producer microservice is down consumer microservice won’t try to call it. Instead, the fallback method will be called. These are the modified versions of our two classes, ConsumerService, and ConsumerApplication.
@Service
public class ConsumerService {
private final RestTemplate restTemplate;
public ConsumerService(RestTemplate rest) {
this.restTemplate = rest;
}
@HystrixCommand(fallbackMethod = "fallback")
public String consume() {
URI uri = URI.create("http://localhost:8080/getCounter");
return this.restTemplate.getForObject(uri, String.class);
}
@SuppressWarnings("unused")
public String fallback() {
return "Can't retrieve the real value, returning the default: 1000";
}
}@EnableCircuitBreaker
@RestController
@SpringBootApplication
public class ConsumerApplication {
@Autowired
private ConsumerService consumerService;
@Bean
public RestTemplate rest(RestTemplateBuilder builder) {
return builder.build();
}
@RequestMapping("/getCurrentCounter")
public String toRead() {
return consumerService.consume();
}
public static void main(String[] args) {
SpringApplication.run(ConsumerApplication.class, args);
}
}After building and running Consumer Microservice now, even if Producer is down, it will give a default response by calling the fallback method.
GET http://localhost:8090/getCurrentCounter
HTTP/1.1 200
Content-Type: text/plain;charset=UTF-8
Content-Length: 58
Date: Sun, 04 Apr 2021 17:01:47 GMT
Keep-Alive: timeout=60
Connection: keep-alive
Can't retrieve the real value, returning the default: 1000
Response code: 200; Time: 216ms; Content length: 58 bytesOpen the circuit after a certain threshold
So we’ve overcome the issue of failing if the produces is down but this is not enough. Remember our producer microservice responds really slow to our requests from 5th to 20th. What we need to do is to configure our circuit breaker to detect this outage and stop making requests and keeping threads busy. To do this we need to add a couple of configurations to our @HystrixCommand annotation. So the finished annotation would look like this.
@HystrixCommand(fallbackMethod = "fallback", commandProperties = {
@HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "500"),
@HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "1"),
@HystrixProperty(name = "fallback.isolation.semaphore.maxConcurrentRequests", value = "1"),
@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "50")
})
public String consume() {
URI uri = URI.create("http://localhost:8080/getCounter");
return this.restTemplate.getForObject(uri, String.class);
}Of course, I exaggerated the parameters a little bit to test my case. You can see what are the parameters and how to set them from here. After this configuration, I wrote a simple bash script to keep sending requests to my consumer microservice.
!/bin/bash
for i in {1..95}
do
echo -n "$i -->"
curl http://localhost:8090/getCurrentCounter
echo " "
sleep 0.1
doneNormally if there weren’t any additional options this script should show that only requests from 5th to 20th get a return value from fallback but because we’ve put up some threshold limits after detecting the outage circuit breaker will stay open until the end of the window. After the failed attempt count in the active window went down to a certain percentage it will try again. This is the output of the bash script above.
1 -->Can't retrieve the real value, returning the default: 1000
2 -->2
3 -->3
4 -->4
5 -->5
6 -->Can't retrieve the real value, returning the default: 1000
7 -->Can't retrieve the real value, returning the default: 1000
8 -->Can't retrieve the real value, returning the default: 1000
9 -->Can't retrieve the real value, returning the default: 1000
10 -->Can't retrieve the real value, returning the default: 1000
11 -->Can't retrieve the real value, returning the default: 1000
12 -->Can't retrieve the real value, returning the default: 1000
13 -->Can't retrieve the real value, returning the default: 1000
14 -->Can't retrieve the real value, returning the default: 1000
15 -->Can't retrieve the real value, returning the default: 1000
16 -->Can't retrieve the real value, returning the default: 1000
17 -->Can't retrieve the real value, returning the default: 1000
18 -->Can't retrieve the real value, returning the default: 1000
19 -->Can't retrieve the real value, returning the default: 1000
20 -->Can't retrieve the real value, returning the default: 1000
21 -->Can't retrieve the real value, returning the default: 1000
22 -->Can't retrieve the real value, returning the default: 1000
23 -->Can't retrieve the real value, returning the default: 1000
24 -->Can't retrieve the real value, returning the default: 1000
25 -->Can't retrieve the real value, returning the default: 1000
26 -->Can't retrieve the real value, returning the default: 1000
27 -->Can't retrieve the real value, returning the default: 1000
28 -->Can't retrieve the real value, returning the default: 1000
29 -->Can't retrieve the real value, returning the default: 1000
30 -->Can't retrieve the real value, returning the default: 1000
31 -->Can't retrieve the real value, returning the default: 1000
32 -->Can't retrieve the real value, returning the default: 1000
33 -->Can't retrieve the real value, returning the default: 1000
34 -->Can't retrieve the real value, returning the default: 1000
35 -->Can't retrieve the real value, returning the default: 1000
36 -->Can't retrieve the real value, returning the default: 1000
37 -->Can't retrieve the real value, returning the default: 1000
38 -->Can't retrieve the real value, returning the default: 1000
39 -->Can't retrieve the real value, returning the default: 1000
40 -->Can't retrieve the real value, returning the default: 1000
41 -->Can't retrieve the real value, returning the default: 1000
42 -->Can't retrieve the real value, returning the default: 1000
43 -->Can't retrieve the real value, returning the default: 1000
44 -->Can't retrieve the real value, returning the default: 1000
45 -->Can't retrieve the real value, returning the default: 1000
46 -->Can't retrieve the real value, returning the default: 1000
47 -->Can't retrieve the real value, returning the default: 1000
48 -->Can't retrieve the real value, returning the default: 1000
49 -->Can't retrieve the real value, returning the default: 1000
50 -->Can't retrieve the real value, returning the default: 1000
51 -->Can't retrieve the real value, returning the default: 1000
52 -->Can't retrieve the real value, returning the default: 1000
53 -->Can't retrieve the real value, returning the default: 1000
54 -->Can't retrieve the real value, returning the default: 1000
55 -->Can't retrieve the real value, returning the default: 1000
56 -->Can't retrieve the real value, returning the default: 1000
57 -->Can't retrieve the real value, returning the default: 1000
58 -->Can't retrieve the real value, returning the default: 1000
59 -->Can't retrieve the real value, returning the default: 1000
60 -->Can't retrieve the real value, returning the default: 1000
61 -->Can't retrieve the real value, returning the default: 1000
62 -->Can't retrieve the real value, returning the default: 1000
63 -->Can't retrieve the real value, returning the default: 1000
64 -->Can't retrieve the real value, returning the default: 1000
65 -->Can't retrieve the real value, returning the default: 1000
66 -->Can't retrieve the real value, returning the default: 1000
67 -->Can't retrieve the real value, returning the default: 1000
68 -->Can't retrieve the real value, returning the default: 1000
69 -->Can't retrieve the real value, returning the default: 1000
70 -->Can't retrieve the real value, returning the default: 1000
71 -->Can't retrieve the real value, returning the default: 1000
72 -->Can't retrieve the real value, returning the default: 1000
73 -->Can't retrieve the real value, returning the default: 1000
74 -->Can't retrieve the real value, returning the default: 1000
75 -->Can't retrieve the real value, returning the default: 1000
76 -->Can't retrieve the real value, returning the default: 1000
77 -->Can't retrieve the real value, returning the default: 1000
78 -->Can't retrieve the real value, returning the default: 1000
79 -->Can't retrieve the real value, returning the default: 1000
80 -->Can't retrieve the real value, returning the default: 1000
81 -->Can't retrieve the real value, returning the default: 1000
82 -->Can't retrieve the real value, returning the default: 1000
83 -->Can't retrieve the real value, returning the default: 1000
84 -->Can't retrieve the real value, returning the default: 1000
85 -->Can't retrieve the real value, returning the default: 1000
86 -->Can't retrieve the real value, returning the default: 1000
87 -->Can't retrieve the real value, returning the default: 1000
88 -->Can't retrieve the real value, returning the default: 1000
89 -->Can't retrieve the real value, returning the default: 1000
90 -->Can't retrieve the real value, returning the default: 1000
91 -->20
92 -->21
93 -->22
94 -->23
95 -->24 As you can see from the output normally we would expect it only to return the default response for requests from 5 to 20 but it did not try to connect until the 91st request. Because the circuit was opened and did not allow requests to go through. After enough time passed it is reopened and tried to connect again.
This is the view from the Hystrix Monitor while the application is running.
Thanks for reading! You can see all my posts about Java by clicking here!