Day 74 — Connecting EC2 with Grafana

5 min readOct 17, 2023

On this 74th day of our 90-day journey, we delve into the intricacies of connecting Amazon EC2 instances with Grafana, exploring the seamless integration of these powerful tools and unlocking their combined potential.

Task 1

Connect an Linux and one Windows EC2 instance with Grafana and monitor the different components of the server.

Prerequisites:

You should have two EC2 instances: one running Linux (e.g., Amazon Linux or Ubuntu) and another running Windows Server.
Both instances should have internet connectivity.
Basic knowledge of AWS services, SSH for Linux, and RDP for Windows.

Configure Loki and Promtail

Create a directory “grafana-configs” and download the yaml file that will contain the required configuration for running loki.

mkdir grafana-configs

#Download Loki Config
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

2. Download the yaml file that will contain the required configuration for running promtail.

#Download Promtail Config
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Run Loki Docker container using the below command.

docker run --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

In the above step, 3100 port is mapped to the loki container. Therefore let’s open the port in the security group to make the loki container running.

Navigate to the loki URL and wait for sometime to make the loki live.

IpofEC2:3100/ready

Wait for some time

You can also see the metrics which means the logs which is the sole purpose of loki to collect.

Run the Promtail Docker container using the below command and check with docker ps to see the container status.

docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki:loki grafana/promtail:2.8.0 -config.file=/mnt/config/promtail-config.yaml

Data Source in Grafana

Now, navigate to the Grafana webapp and on the homepage choose the add data source option.

2. Provide the HTTP URL as below to connect the loki data source to Grafana so that loki will send the logs to grafana.

localhost:3100

Click on save & test to connect the data source.

Checking logs in Loki

Click on explore in the above screenshot after adding the data source. In the label filters, we can choose job and varlogs which is generally the path /var/log/*log in the backend to show all the system logs.

Click on the run query in the above screenshot to execute and show all the system logs as below.

Creating Dashboard

Let’s add the log to the dashboard by choosing the option from the above screenshot location.

Now the system logs are added to the grafana dashboard. Let’s add some more by clicking on visualization to add some graphs as shown below screenshot.

In Label filters choose job and varlogs and line contains to error to show all the lines with error and select the duration to show all the lines with error in the logs.

Similarly, let’s check the error lines in grafana log that is placed in /var/log/grafana/grafana.log

To achieve the aim of displaying the grafana log, we need to provide the grafana log path to the promtail-config.yaml file in the target section as shown below.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  - targets:
      - localhost
    labels:
      job: grafanalogs
      __path__: /var/log/grafana/*log

We can now choose the label filters to set the job and grafanalogs with the line contains and visualization option to view in a graphical manner. We can add this to our dashboard.

Now, install nginx in the EC2 instance.

sudo apt-get install nginx

Use the proper label filters to show an aggregate sum of words repeating nginx while installing. This can be achieved by setting the varlogs as label filters.