HTTP client in Go: advanced techniques for working with requests

HTTP client in Go: making requests correctly (advanced level)

The built-in net/http package in Go provides powerful but often underestimated capabilities for creating reliable HTTP clients. At the beginner level, developers use http.Get or http.Post, which is convenient for quick prototypes but completely unsuitable for production. In this article, we will cover advanced techniques for configuring an HTTP client in Go: fine-tuning timeouts, managing the connection pool, creating middleware chains, implementing retries, and proper error handling. You will learn to write client code that withstands high loads and is resilient to network failures.

1. Configuring timeouts and connection pool

A basic mistake is using http.DefaultClient. It has no timeouts and can hang indefinitely. For production, always create your own http.Client instance with fine-tuned http.Transport.

package main

import (    "net/http"    "time")

func NewClient() *http.Client {    transport := &http.Transport{        MaxIdleConns:        100,               // Maximum idle connections in the entire pool        MaxConnsPerHost:     10,                // Maximum connections to a single host        MaxIdleConnsPerHost: 5,                 // Maximum idle connections to a single host        IdleConnTimeout:     90 * time.Second,  // Idle connection lifetime        DisableCompression:  false,             // Enable compression (gzip)    }

    client := &http.Client{        Transport: transport,        Timeout:   30 * time.Second, // Total timeout for the entire request    }    return client}

Let's break down the key parameters:

• Timeout — total time limit for executing a request (including reading the response body). Set it based on the expected API speed (usually 10-30 seconds).
• MaxIdleConns — the number of keep-alive connections stored in the pool. For high-load systems, increase it to 200-500.
• MaxConnsPerHost — limits the number of simultaneous connections to a single host. Prevents server overload.

It is also recommended to set DialContext with a custom connection timeout:

transport.DialContext = (&net.Dialer{    Timeout:   5 * time.Second,  // Timeout for TCP handshake    KeepAlive: 30 * time.Second, // Keep-alive to maintain the connection}).DialContext

2. Creating middleware chains for the HTTP client

Go does not have built-in middleware for the client, but we can easily implement the "decorator" pattern by wrapping http.RoundTripper. This allows adding logging, metrics, authentication, and other cross-cutting functions.

type LoggingRoundTripper struct {    Proxied http.RoundTripper}

func (l *LoggingRoundTripper) RoundTrip(r *http.Request) (*http.Response, error) {    start := time.Now()    resp, err := l.Proxied.RoundTrip(r)    duration := time.Since(start)

    log.Printf("[%s] %s %s - %d (%v)",        r.Method, r.URL.String(),        r.Host, resp.StatusCode, duration,    )    return resp, err}

// Usage:transport := &LoggingRoundTripper{    Proxied: http.DefaultTransport,}client := &http.Client{Transport: transport}

You can create a middleware chain, for example, to add authentication headers:

type AuthRoundTripper struct {    Proxied http.RoundTripper    Token   string}

func (a *AuthRoundTripper) RoundTrip(r *http.Request) (*http.Response, error) {    r.Header.Set("Authorization", "Bearer "+a.Token)    return a.Proxied.RoundTrip(r)}

// Chain: Auth → Logging → Transportclient := &http.Client{    Transport: &LoggingRoundTripper{