Monitoring Go Applications: Advanced Metrics and Alerts

Advanced Monitoring of Go Applications: From Metrics to Alerts

In modern Go (Golang) development, basic logging and Health Checks are not enough. Production systems require deep observability: collecting custom metrics, exporting them to Prometheus, visualizing in Grafana, and automatic alerts. In this article, we will cover the advanced level of monitoring Go applications — from metrics architecture to configuring alerts based on business logic.

Monitoring Architecture: Prometheus + Go

The standard monitoring stack for Go includes:

• Prometheus — a system for collecting and storing metrics (pull model).
• Prometheus Go client (prometheus/client_golang) — a library for exporting metrics.
• Grafana — dashboard visualization.
• Alertmanager — alert management (email, Slack, Telegram).

The Go application must provide an HTTP endpoint (usually /metrics) from which Prometheus scrapes data. Let's look at the basic setup.

package main

import (    "net/http"    "github.com/prometheus/client_golang/prometheus"    "github.com/prometheus/client_golang/prometheus/promhttp")

func main() {    // Register standard Go metrics (runtime, GC, goroutines)    prometheus.MustRegister(prometheus.NewProcessCollector(prometheus.ProcessCollectorOpts{}))    prometheus.MustRegister(prometheus.NewGoCollector())

    http.Handle("/metrics", promhttp.Handler())    http.ListenAndServe(":8080", nil)}

By running the application, you will get standard metrics: go_goroutines, process_cpu_seconds_total, go_memstats_alloc_bytes, and others. But for advanced monitoring, this is not enough — you need custom metrics.

Creating Custom Metrics: Gauge, Counter, Histogram

Prometheus supports four main metric types. Let's look at them using the example of a Go application processing HTTP requests.

Counter — number of events

Used for counting requests, errors, task executions.

var requestCounter = prometheus.NewCounterVec(    prometheus.CounterOpts{        Name: "http_requests_total",        Help: "Total number of HTTP requests",    },    []string{"method", "endpoint", "status"},)

func init() {    prometheus.MustRegister(requestCounter)}

func handler(w http.ResponseWriter, r *http.Request) {    // Request processing    requestCounter.WithLabelValues(r.Method, r.URL.Path, "200").Inc()}

Gauge — current values

Ideal for monitoring queues, connection pools, temperature.

var queueSize = prometheus.NewGauge(    prometheus.GaugeOpts{        Name: "task_queue_size",        Help: "Current task queue size",    },)

func addToQueue() {    queueSize.Inc()}

func processFromQueue() {    queueSize.Dec()}

Histogram — value distribution

Used for measuring response time or data size.

var requestDuration = prometheus.NewHistogramVec(    prometheus.HistogramOpts{        Name:    "http_request_duration_seconds",        Help:    "Histogram of request processing time",        Buckets: prometheus.DefBuckets, // 0.005, 0.01, 0.025, 0.05, ...    },    []string{"endpoint"},)

func measureRequest(endpoint string, duration float64) {    requestDuration.WithLabelValues(endpoint).Observe(duration)}

Summary — quantiles

Similar to histogram, but calculates exact quantiles (p50, p95, p99).

var requestLatency = prometheus.NewSummaryVec(    prometheus.SummaryOpts{        Name:       "http_request_latency_seconds",        Help:       "Request latency (quantiles)",