Hi Friends,
In this section, I thought to discuss about one potential issue, which one can easily overlook and when your service really becomes busy, you will get this “Socket Exhaustion” exception. One thing to keep in mind that distributed systems are prone to error. Therefore, its always good to anticipate it before and take precautionary measures.
One minor communication issue between services, can bring the entire system down.
Let’s say, we are hosting our services hosted on individual containers, off-course containers are the de-facto choice when it comes for distributed system. There in container, assume that service becomes briefly unavailable, then it can lead to catastrophic issue, if this scenario is not handled. These are the exact situation, which we need to handle gracefully. Between Microservices, one small communication error can create havoc to entire application.
Common Issues:
- Container crashed and services become briefly unavailable till the time new container is getting created.
- Service overloaded with request and it becomes very slow or unresponsive.
- HttpClient can lead to Socket Exhaustion
What should we do, whenever it happens?
- Using HttpClient
- Async communication, can drastically improve system availability.
- Retry with exponential backoff
- Circuit breaker
HttpClient:- If you are using HttpClient directly for making Microservice communications, and its not gracefully handled, then this will lead to too many open sockets, which in turn will lead to Socket Exhaustion exception. Although, we are disposing HttpClient, underlying socket is not disposed. Generally, sockets gets timed out after a while. Therefore, in many scenarios, this won’t be a problem. But, when service really becomes busy during high load time, then this will be a problem.
In order to get rid of this issue, IHttpClientFactory is introduced in .net core. This will create managed instances of HttpClient instances without the risk of Socket Exhaustion issue. Through, dependency injection, we register, where we want HttpClient instances. And, since these instances are managed, .Net will take care of cleaning up these under the hood. It also manages the lifetime of Http Message handler. Therefore, instead of creating HttpClient, we can use IHttpClientFactory in startup class. By introducing Service Client Factory, it will create service communication, which is itself very resilient.
This Http client method will add the HTTPClientFactory, all the related services to the services collection. And in the type we pass it, its also going to make sure that http client going to instantiate it. This is also known as typed client approach. And the way, we use this under the hood, is like
Next thing to make it more resilient, we can think of using Polly library. Via Polly, we can implement certian policies, which can make service more resilient in the event of any intermittent failure. This means, we can implement retry policy, to make it more resilient. It provides couple of options under the hood like
- Retry:- Retrying the service after certain time
- Timeout:- To ensure the caller never has to wait beyond the configured timeout.
- CircuitBreaker:- After certain no of attempts, open the circuit to avoid load on service.
- etc
And this is fully integrated with HTTPClientFactory. Below, I have pasted the snippet, explaining the same.
And, its implementation looks like
First method as part of polly, allows to retry this service for certain no of times. You can read more about these policies here. I am trying 5 attempts here, with said interval. Obviously, its not a good idea to keep on retrying for that particular service, it will completely block I/O as well. Hence, its always a good idea to club this offering with Circuit breaker policy. 2nd snippet explaining the same. This will break the circuit after 3 attempts after receiving the same error message and then this will pause the service or open the circuit for 10 seconds.
Therefore, these are couple of ways of making microservices more resilient.
I hope you guys have liked this discussion. Will meet again with some other topic. Till then, stay tuned and Happy Coding.
Thanks,
Rahul Sahay
Happy Coding
