Setting up Prometheus, Grafana and Spring Boot

The fastest way to get Prometheus and Grafana up and running is via Docker. It's a huge timesaver regarding configuration and security, since Docker offers a solid base. The provided Docker-images are well preconfigured, making the easy getting in.
If Docker-Compose is not yet installed, feel free to check out the official documentation.

In order to get started, create a new docker-compose.yml file on the server where Docker and Docker-Compose have been installed:

WATCH OUT: The default password "admin" should be changed at all costs if the server is publicly accessible, as this unsecure password might leave the door open for script kiddies and other undesirables.

Another file prometheus.yml needs to be created within the folder containing the docker-compose.yml. Add the following code:

We've now defined a scrape job that accesses the service localhost:9090, the Prometheus server itself. This configuration will load various expressions into Prometheus that you can use to read metric data related to the Prometheus server.

After setting up the configuration, the Docker-container needs to be started using the following command:

If no Firewall or anything similar is standing in the way, Prometheus and Grafana should now be accessible.

Prometheus can be accessed using the following link:

<server-ip>:9090

As we defined a Scraper in the configuration file that reads the metrics out of the Prometheus-ontainer, you may use the following link to show the number of HTTP-requests:
<server-ip>:9090/graph?g0.expr=prometheus_http_requests_total&g0.tab=0&g0.display_mode=lines&g0.show_exemplars=0&g0.range_input=1h

Grafana can be accessed with the following link:
<server-ip>:3000

The required password is the one defined in the docker-compose.yml-file.

As to be expected, Grafana ist quite empty. Before defining your own dashboard, Prometheus needs to be integrated using the following link:
<server-ip>:3000/connections/datasources/new

Select Promotheus and enter the following IP:
<server-ip>:9090

Save afterwards.

In Grafana, metric data is defined in dashboards and displayed visually depending on the configuration. To create a dashboard, open the following link:
<server-ip>:3000/dashboards

Create a new dashboard by clicking on "New" in the top right corner, then selected Add visualization and as previously defined, selected Prometheus as data source.

Let's approach the monitoring with applications in mind.

Imagine having a Spring Boot application with loads of traffic and a hefty chunk of log entries stored in different areas. In this section, we'll set the whole thing up - using the Prometheus-dependency Promtail to send the log entries to Loki. Loki acts as an interface for Grafana in order to display the logs visually.

As build-tool, we'll use Maven, however Gradle works too - the approach is the same

Maven:

We'll add the following dependency the pom.xml-file:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Gradle:

As for gradle, the following code needs to be added to the build.gradle:

implementation 'io.micrometer:micrometer-registry-prometheus'

The project needs to be rebuilt in order to apply the changes.

In order to be able to read the metrics, either the application.properties or application.yml need to be updated as follows:

management.endpoints.web.exposure.include=prometheus
management.endpoint.prometheus.enabled=true

This change makes the Prometheus-metrics accessible via the respective endpoint.

Watch out: If Spring Boot Security is active, the endpoint needs to be allowed accordingly in order to access the metrics.

In the following step, we'll update the scraper in the Prometheus configuration.

Let's add the following content to the prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
      
  - job_name: 'spring-boot'
    static_configs:
      - metrics_path: "/actuator/prometheus"
      - targets: ['<springboot-server-ip>:8080']

Restart the container after applying these changes.

All done? There should be new expressions available in Prometheus or Grafana, such as jvm_info for example.

Reading Spring Boot logs within Grafana is sensible, as it allows you to monitor the performance and behaviour of your application in real time. The centralized visualisation of logs is a great tool to identify and analyze problems faster, which leads to more efficient debugging- and optimization processes. Grafana offers the possibility to search through logs interactively and correlate different metrics, thus greatly simplifying the error diagnosis.  Not only does this give you an overview of the logs, but it also allows you to make informed decision regarding the improvement of your application.

In this process, a Docker installation is  required in order to install the docker-image Promtail, which has Spring Boot running. Promtail is used to send Logs to Loki, which can also be installed via Docker-image, and is read by Grafana as it alreads offers a Loki datasource to display logs visually. This has no longer anything to do with Prometheus.

Create a docker-compose.yaml file on the Spring Boot server with the following content:

version: '3.8'
services:
  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    volumes:
      - ../tomcat/logs:/var/log
      - ./promtail-config.yml:/etc/promtail/promtail-config.yml
    command: -config.file=/etc/promtail/promtail-config.yml
    restart: always

Watch out: The path ../tomcat/logs is merely an example. It's important to adjust the path of your logs, as it might be different on your server.

Create a new file named promtail-config.yml and use the following template as content:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url:<grafana-server-ip>/loki/api/v1/push

scrape_configs:
  - job_name: "tomcat-logs"
    static_configs:
      - targets:
          - localhost
        labels:
          job: "tomcat"
          __path__: /var/log/*.log

Make sure that the url-entry points to your Grafana-server.
Go ahead and start the container afterwards.

Loki can be installed at the same location as Grafana by updaten the docker-compose.yml on your server as follows:

version: '3.7'

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

  # loki hinzufügen
  loki:
    image: grafana/loki:2.9.10
    container_name: loki
    command: -config.file=/etc/loki/local-config.yaml
    restart: always
    ports:
      - "3100:3100"

Start the Loki-container after applying these changes.

In order to include Loki into Grafana, head back here:

<server-ip>:3000/connections/datasources/new

and select Loki as data source.und wählst als Data-Source diesmal Loki aus. Enter the following URL:

<server-ip>:3100

This allows you to create a new Dashboard and use Loki as data source.

And there we go! The configuration is finally done. You can now enjoy and visualize the logs and metrics data of your Spring Boot application in Grafana. The possibility to have it all in one place makes it much easier to track the performance of your application and to react quickly to potential issues.

Grafana is a powerful tool that displays the collected data in an efficient way. You can create interactive dashboards that show the state of your application at a glance. Whether you're monitoring performance or analyzing errors, the combination of Prometheus, Grafana and Spring Boot gives you the control you need.

Have fun experimenting with the metrics and logs!