Creating a retry mechanism for failed requests

When making API calls, failures due to network issues, server errors, or timeouts are common. A retry mechanism helps improve reliability by automatically retrying failed requests.

---

1. Understanding Retry Mechanisms

Retrying failed requests helps handle temporary failures, but it must be done correctly to avoid overloading the server.

Key Considerations:

• Idempotency: Ensure that retried requests do not cause duplicate actions (e.g., avoid multiple payments).
• Retry Limits: Prevent infinite loops by setting a maximum retry count.
• Exponential Backoff: Wait longer between retries to reduce server load.
• Jitter: Add randomness to avoid multiple clients retrying at the same time.

---

2. Implementing a Retry Mechanism in JavaScript (Axios)

Using Axios, we can create a retry mechanism with exponential backoff.

Installation

npm install axios

Implementation

const axios = require('axios');

const fetchDataWithRetry = async (url, retries = 3, delay = 1000) => {
    for (let attempt = 1; attempt <= retries; attempt++) {
        try {
            const response = await axios.get(url);
            return response.data;
        } catch (error) {
            console.log(`Attempt ${attempt} failed: ${error.message}`);

            if (attempt === retries) {
                throw new Error("All retry attempts failed");
            }

            await new Promise(resolve => setTimeout(resolve, delay * attempt)); // Exponential backoff
        }
    }
};

// Example Usage
fetchDataWithRetry('https://jsonplaceholder.typicode.com/posts/1')
    .then(data => console.log("Data received:", data))
    .catch(err => console.error("Request failed:", err.message));

Features

• Retries 3 times before failing.
• Implements exponential backoff (delay * attempt).
• Logs each failed attempt.

---

3. Retry Mechanism in Python (Requests)

For Python, use the requests library with urllib3.Retry.

Installation

pip install requests

Implementation

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def fetch_data_with_retry(url, retries=3, backoff_factor=1):
    session = requests.Session()
    retry = Retry(
        total=retries,
        backoff_factor=backoff_factor,
        status_forcelist=[500, 502, 503, 504]  # Retry only for server errors
    )
    session.mount("https://", HTTPAdapter(max_retries=retry))

    try:
        response = session.get(url, timeout=5)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as e:
        print(f"Request failed: {e}")
        return None

# Example Usage
data = fetch_data_with_retry("https://jsonplaceholder.typicode.com/posts/1")
print("Data received:", data)

Features

• Retries 3 times before failing.
• Uses backoff_factor to increase wait time.
• Handles specific HTTP errors (500, 502, 503, 504).

---

4. Retry Mechanism in Node.js (Fetch API)

For the native Fetch API, a retry mechanism can be implemented manually.

const fetchWithRetry = async (url, retries = 3, delay = 1000) => {
    for (let attempt = 1; attempt <= retries; attempt++) {
        try {
            const response = await fetch(url);
            if (!response.ok) throw new Error(`HTTP Error: ${response.status}`);
            return await response.json();
        } catch (error) {
            console.log(`Attempt ${attempt} failed: ${error.message}`);

            if (attempt === retries) {
                throw new Error("All retry attempts failed");
            }

            await new Promise(resolve => setTimeout(resolve, delay * attempt)); // Exponential backoff
        }
    }
};

// Example Usage
fetchWithRetry('https://jsonplaceholder.typicode.com/posts/1')
    .then(data => console.log("Data received:", data))
    .catch(err => console.error("Request failed:", err.message));

Features

• Retries 3 times before failing.
• Uses exponential backoff.

---

5. Best Practices for Retry Mechanisms

1. Use Exponential Backoff

• Instead of retrying immediately, increase wait time with each attempt.
• Example formula: delay = baseDelay * 2^attempt

2. Limit Retries

• Set a max retry count (e.g., 3-5) to avoid infinite loops.

3. Handle Specific Errors

• Retry only for transient errors (e.g., 500, 502, 503, 504, timeouts).
• Do NOT retry on 400 errors (e.g., bad request, unauthorized).

4. Add Jitter

• Prevent many clients retrying at the same time by adding randomness.
• Formula: delay = baseDelay * (2^attempt) + random(0, jitter)

Example:

const jitter = Math.random() * 500; // Add randomness up to 500ms

5. Use Idempotency

• Ensure repeated requests do not cause unintended side effects (e.g., multiple payments).
• Example: Use unique request IDs to detect duplicate requests.