Monitoring Go Applications: Advanced Metrics and Alerts

Monitoring Go Applications: Advanced Metrics and Alerts

Modern Go applications require not only high performance but also reliable monitoring. Basic metric collection (CPU, memory, requests) is just the first step. At an advanced level, it is necessary to implement custom metrics, configure alerts based on business logic, and ensure fault tolerance. In this article, we will look at how to implement comprehensive monitoring of Go applications using Prometheus, Grafana, and custom alerts.

1. Advanced Metric Collection in Go with Prometheus

The prometheus/client_golang library allows you to collect not only standard metrics but also custom ones. Let's look at an example of setting up a custom metric collector for an HTTP server.

package main\n\nimport (\n    "net/http"\n    "github.com/prometheus/client_golang/prometheus"\n    "github.com/prometheus/client_golang/prometheus/promhttp"\n)\n\nvar (\n    requestDuration = prometheus.NewHistogramVec(\n        prometheus.HistogramOpts{\n            Name:    "http_request_duration_seconds",\n            Help:    "Duration of HTTP requests.",\n            Buckets: prometheus.DefBuckets,\n        },\n        []string{"method", "path", "status"},\n    )\n    activeRequests = prometheus.NewGauge(\n        prometheus.GaugeOpts{\n            Name: "http_active_requests_total",\n            Help: "Current number of active requests.",\n        },\n    )\n)\n\nfunc init() {\n    prometheus.MustRegister(requestDuration)\n    prometheus.MustRegister(activeRequests)\n}\n\nfunc metricsMiddleware(next http.Handler) http.Handler {\n    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n        activeRequests.Inc()\n        defer activeRequests.Dec()\n\n        timer := prometheus.NewTimer(prometheus.ObserverFunc(func(v float64) {\n            requestDuration.WithLabelValues(r.Method, r.URL.Path, "200").Observe(v)\n        }))\n        defer timer.ObserveDuration()\n\n        next.ServeHTTP(w, r)\n    })\n}\n\nfunc main() {\n    mux := http.NewServeMux()\n    mux.Handle("/metrics", promhttp.Handler())\n    mux.Handle("/api", metricsMiddleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n        w.Write([]byte("Hello, Go!"))\n    })))\n    http.ListenAndServe(":8080", mux)\n}

In this example, we create a histogram for request duration and a gauge for active requests. Using labels allows you to detail metrics by methods, paths, and statuses.

2. Configuring Alerts Based on Business Logic

Alerts should be not only technical (high CPU load) but also business-related. For example, exceeding the response time for a critical endpoint or a drop in the number of successful transactions. Alert configuration in Prometheus (file alerts.yml):

groups:\n  - name: go_app_alerts\n    rules:\n      - alert: HighRequestLatency\n        expr: http_request_duration_seconds{job="go-app"} > 1.0\n        for: 5m\n        labels:\n          severity: critical\n        annotations:\n          summary: "High request latency ({{ $value }}s)"\n          description: "Endpoint {{ $labels.path }} has had latency over 1 second for 5 minutes."\n\n      - alert: ActiveRequestsSurge\n        expr: http_active_requests_total{job="go-app"} > 100\n        for: 2m\n        labels:\n          severity: warning\n        annotations:\n          summary: "Active requests surge ({{ $value }})"\n          description: "The number of active requests has exceeded 100 for 2 minutes."\n\n      - alert: ErrorRateHigh\n        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1\n        for: 10m\n        labels:\n          severity: critical\n        annotations:\n          summary: "High 5xx error rate"\n          description: "Error rate exceeds 10% over the last 10 minutes."

Note the use