Observability in Rust: Tracing, Metrics, and Logs for Reliable Systems

Why Observability is a Superpower for Rust Developers

Imagine: your Rust service is running in production, but suddenly latency skyrockets. Logs are silent, metrics show 200 OK, and users are complaining. Sound familiar? This is the classic "black box." Without observability in Rust, you are blind.

Observability is not just "logs + metrics." It is the ability to ask any questions of your system without needing to write new code. In Rust, with its emphasis on performance and reliability, observability becomes a mandatory ingredient for production-grade applications.

In this article, we will break down the three pillars of observability in Rust: tracing, metrics, and logs. You will get ready-to-use code examples with tracing, OpenTelemetry, and Prometheus that you can immediately implement in your project.

Tracing in Rust: A Request's Journey Through the System

Tracing is the life story of a single request. Instead of scattered logs, you get a connected path: from the API entry point to the database query and the client response. For Rust, the best tool is the tracing crate from the Tokio community.

Setting Up tracing in a Rust Application

Let's start with basic setup. Add to your Cargo.toml:

[dependencies]tracing = "0.1"tracing-subscriber = "0.3"tracing-opentelemetry = "0.23"opentelemetry = "0.22"opentelemetry-otlp = "0.15"

Now initialize the tracer in main.rs:

use tracing_subscriber::{fmt, prelude::*, EnvFilter};use opentelemetry::{global, KeyValue};use opentelemetry_otlp::WithExportConfig;

fn init_tracer() -> Result<(), Box<dyn std::error::Error>> {    global::set_text_map_propagator(opentelemetry_jaeger::Propagator::new());        let tracer = opentelemetry_otlp::new_pipeline()        .tracing()        .with_exporter(            opentelemetry_otlp::new_exporter()                .tonic()                .with_endpoint("http://localhost:4317")        )        .with_trace_config(            opentelemetry::sdk::trace::config()                .with_resource(opentelemetry::sdk::Resource::new(vec![                    KeyValue::new("service.name", "rust-observability-demo")                ]))        )        .install_simple()?;

    // Connect tracing with OpenTelemetry    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);    let subscriber = fmt::Subscriber::builder()        .with_env_filter(EnvFilter::from_default_env())        .finish()        .with(telemetry);

    tracing::subscriber::set_global_default(subscriber)?;    Ok(())}

Creating Spans and Traces

Now let's add tracing to the business logic:

use tracing::{span, Span, Level, instrument};

#[instrument]async fn handle_request(user_id: u64) -> Result<String, Box<dyn std::error::Error>> {    let user_data = get_user_data(user_id).await?;    let processed = process_data(&user_data).await?;    Ok(processed)}

#[instrument(skip(db_pool))]async fn get_user_data(user_id: u64) -> Result<User, Box<dyn std::error::Error>> {    // Simulate a database query    let span = Span::current();    span.record("db.query", "SELECT * FROM users WHERE id = $1");        tokio::time::sleep(std::time::Duration::from_millis(50)).await;    Ok(User { id: user_id, name: "Alice".into() })}

#[instrument]async fn process_data(user: &User) -> Result<String, Box<dyn std::error::Error>> {    let result = format!("Processed user: {}", user.name);    tracing::info!(user_id = user.id, "Data processed successfully");    Ok(result)}

The #[instrument] attribute automatically creates a span for the function and records all arguments. In Jaeger