API Monitoring Dashboard using Graylog.

APIs is a language for a microservice. They are essentially the only elements that are accessible to other teams. API response time and the consistency of the API are two major SLA defined by each team. API monitoring is an essential part of microservice monitoring. API monitoring has different forms such as monitoring frequency of API, response time of API, API calling pattern, API success rate. All of these are essential attributes to monitor. Monitoring these attributes and acting on the problems identified in them is the only thing that keeps microservice healthy.

We have also faced issues in the past where particularly one client (context) was facing a latency issue and we have spent quite a large time identifying an API that had a latency issue. To identify API which had latency issues, We had to dig in through the audits logs. With this incident, we have identified that there is a scope of improvement in our API monitoring.

So, We have decided to build a dashboard where we can track the performance of API at a granular level. We have identified some areas where we can improve the monitoring. Below are the attributes that we wanted to monitor in our new dashboard.

What we have used to create our new dashboard.

We use Graylog as a log aggregator for our service. While our service serves any API request it has the all details that we need to create a new dashboard. To monitor our service for the above-mentioned point we would need few details of a request that is being served. Below are the key information we would need to monitor the performance of an API

We have added just one simple log line in our service with these details. This simple log line looks like below.

This log line will result in below simple log.

With this log line, we just have all the details we wanted to be logged to generate a new dashboard but all of these details were part of the message filed. But to query on each attribute of API request we would need each of these attributes to be added as separate fields in elastic search.

After Extracting all of these fields we have used Grafana for visualisation and elastic search as DataSource for Grafana. After all of these changes we were able to build very good dashboard where we can track performance of our API at granular level as stated above.

Below are the some screen shot of our new dashboard. Please note that response time is in millisecond.

Below are some screen shots of newly Grafana Dashboard.

You can also build similar dashboard for your service using Graylog, Grafana and Elastic search. Follow below steps to build Dashboard.

1. Create grok Extractor on message filed to extract the response time and other fields that you want to query. In our case we have extracted practice id, response time, username, request path, Http status code so that we can query on each filed.

2. Steps to create grok pattern

Related posts:

Life is hard for everyone. Even in the sunny valleys of Cutieland, a simple Cutie farmer must work hard to make its way in the world. After reviewing raid and adventure champions, we decided to look…

March has been a very turbulent period for the Steem blockchain with a HardFork that created a new chain Hive. This has affected many things and the Steem Engine tribes are affected as well. Here we…

I was speaking with Shan today, and a concept came up of transmitting what you are feeling or understanding to another person, to have them feel or get it in the same way. Is it altered when it is…