HTTP client in JavaScript: advanced request techniques in Node.js

HTTP client in JavaScript: making requests correctly — advanced level

Basic HTTP requests using fetch or axios are just the tip of the iceberg. In real production Node.js applications, you encounter dozens of nuances: connection pools, retry attempts on failures, interceptors for logging, streaming large data, and fine-tuning timeouts. In this article, we will cover advanced techniques for working with HTTP clients in JavaScript. You will learn how to manage keep-alive, implement a reliable retry mechanism, use interceptors, and create custom agents. The material is aimed at developers who are already familiar with the basics of HTTP and want to write more efficient and fault-tolerant code.

1. Connection management: Keep-Alive and custom agents

By default, Node.js uses a global pool of HTTP agents. However, for high-load systems, it is important to control the number of simultaneous connections and their reuse. Let's look at configuring keep-alive using the http module and the undici library.

Example with a custom agent (Node.js built-in):

const http = require('http');const https = require('https');

// Create a custom agent with a connection poolconst agent = new http.Agent({  keepAlive: true,  maxSockets: 50,        // Maximum 50 simultaneous sockets  maxFreeSockets: 10,    // No more than 10 free in the pool  timeout: 60000,        // Socket idle timeout (ms)});

const options = {  hostname: 'api.example.com',  port: 80,  path: '/data',  agent: agent,};

const req = http.get(options, (res) => {  let data = '';  res.on('data', (chunk) => { data += chunk; });  res.on('end', () => console.log('Response received:', data));});

req.on('error', (e) => console.error('Error:', e.message));

Modern approach with undici (faster and lighter):

const { Client, Pool } = require('undici');

// Create a pool with keep-aliveconst client = new Pool('https://api.example.com', {  connections: 100,  pipelining: 10,  keepAliveTimeout: 30000,});

// Execute requestclient.request({  method: 'GET',  path: '/data',}).then(({ statusCode, body }) => {  console.log('Status:', statusCode);  body.setEncoding('utf8');  body.on('data', console.log);}).catch(err => console.error(err));

Using custom agents helps avoid socket leak issues and improves performance through connection reuse.

2. Reliable retry mechanism

Networks are unstable. Even temporary failures can lead to data loss. A proper retry strategy should consider the error type, number of attempts, and exponential backoff. Let's implement a universal retry decorator for any HTTP client.

const axios = require('axios');const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

async function withRetry(requestFn, retries = 3, baseDelay = 1000) {  for (let attempt = 1; attempt <= retries; attempt++) {    try {      return await requestFn();    } catch (error) {      // Do not retry on client errors (4xx)      if (error.response && error.response.status >= 400 && error.response.status < 500) {        throw error;      }      if (attempt === retries) {        throw error; // All attempts exhausted      }      const delay = baseDelay * Math.pow(2, attempt - 1) + Math.random() * 1000;      console.log(`Attempt ${attempt} failed. Waiting ${delay}ms...`);      await sleep(delay);    }  }}

// Usageasync function fetchData() {  const response = await withRetry(() => axios.get('https://api.example.com/data'), 5, 500);  return response.data;}

fetchData().then(console.log).catch(console.error);

Key points: excluding 4xx-