Monitoring Rust Applications: Advanced Metrics and Alerts

Introduction: Why Monitoring Rust Applications is a Challenge and an Opportunity

Rust is renowned for its performance and memory safety. However, when your application operates at the edge of hardware capabilities (for example, a high-load WebSocket server or a data parser), standard CPU and RAM metrics are often insufficient. At an advanced level, monitoring Rust applications requires not just data collection, but creating a precise alerting system that can predict a failure before it occurs.

In this article, we will explore how to build a production-ready monitoring system for Rust: from custom metrics using the prometheus crate to setting up alerts via Alertmanager and Grafana.

1. Advanced Metrics: Beyond Just HTTP Requests

Basic monitoring (request count, latency) is the minimum. At an advanced level, we collect metrics that reflect the internal state of the application: queue sizes, number of active tasks in Tokio, connection pool status.

1.1. Custom Metrics with Histogram and Gauge Types

We use the prometheus crate. Consider an example of collecting metrics for a database connection pool:

use prometheus::{HistogramOpts, HistogramVec, Gauge, register_histogram_vec, register_gauge};use lazy_static::lazy_static;

lazy_static! {    static ref DB_CONNECTION_POOL: Gauge = register_gauge!(        "db_connection_pool_size",        "Current number of active connections in the pool"    ).unwrap();

    static ref DB_QUERY_DURATION: HistogramVec = register_histogram_vec!(        HistogramOpts::new("db_query_duration_seconds", "Database query execution time"),        &["query_type", "status"]    ).unwrap();}

// Usage in codefn execute_query(query_type: &str) -> Result<(), String> {    DB_CONNECTION_POOL.inc();    let start = std::time::Instant::now();    // ... execute query ...    let duration = start.elapsed().as_secs_f64();    DB_QUERY_DURATION.with_label_values(&[query_type, "success"]).observe(duration);    DB_CONNECTION_POOL.dec();    Ok(())}

Important: Use HistogramVec to categorize queries by type (SELECT, INSERT) and status (success, error). This will allow you to build heatmaps in Grafana.

1.2. Metrics from Inside the Tokio Runtime

For asynchronous applications, it is critical to know the scheduler load. We use tokio-metrics:

use tokio_metrics::RuntimeMetrics;use std::sync::Arc;

async fn monitor_runtime(metrics: Arc<RuntimeMetrics>) {    loop {        tokio::time::sleep(std::time::Duration::from_secs(5)).await;        let num_alive_tasks = metrics.num_alive_tasks();        let blocking_threads = metrics.num_blocking_threads();        // Send these values to Prometheus via Gauge        println!("Active tasks: {}, Blocking threads: {}", num_alive_tasks, blocking_threads);    }}

These metrics help identify task leaks (when num_alive_tasks is constantly growing) or excessive use of block_in_place.

2. Metrics Collection Architecture: Push vs Pull

In the Rust ecosystem, the Pull model (Prometheus scraper) is more commonly used, but for short-lived processes (CLI utilities, parsers), the Push approach via Pushgateway is convenient.

2.1. Pull Model: HTTP Endpoint

The standard way is to run a separate HTTP server on port 9090 (or another) inside your application:

use warp::Filter;use prometheus::{Encoder, TextEncoder};

async fn metrics_handler() -> Result<impl warp::Reply, warp::Rejection> {    let metric_families = prometheus::gather();    let mut buffer = Vec::new();    let encoder = TextEncoder::new();    encoder.encode(&metric_families, &mut buffer).unwrap();    Ok(warp::reply::with_header(bu