Handling large API responses inefficiently

When dealing with large API responses in React applications, inefficient handling can lead to performance bottlenecks, memory issues, and poor user experience. Here are optimized approaches to manage substantial data payloads effectively.

Common Performance Pitfalls

// ❌ Problematic patterns
// 1. Loading entire large dataset at once
const [data, setData] = useState([]);
useEffect(() => {
  fetch('/api/large-dataset')
    .then(res => res.json())
    .then(allData => setData(allData)); // Potentially huge array
}, []);

// 2. Processing large data in the main thread
const processed = bigData.map(item => heavyTransformation(item));

Optimization Strategies

1. Pagination and Chunked Loading

// ✅ Implement server-side pagination
const [data, setData] = useState([]);
const [page, setPage] = useState(1);

const loadMore = useCallback(async () => {
  const response = await fetch(`/api/data?page=${page}&limit=20`);
  const newData = await response.json();
  setData(prev => [...prev, ...newData]);
  setPage(p => p + 1);
}, [page]);

// Use with infinite scroll or "Load More" button

2. Lazy Loading with Virtualization

// ✅ Only render visible items
import { FixedSizeList } from 'react-window';

function LargeList({ data }) {
  const Row = ({ index, style }) => (
    <div style={style}>
      {data[index].name}
    </div>
  );

  return (
    <FixedSizeList
      height={600}
      width={300}
      itemSize={50}
      itemCount={data.length}
    >
      {Row}
    </FixedSizeList>
  );
}

3. Web Workers for Heavy Processing

// ✅ Offload processing to a web worker
// worker.js
self.onmessage = function(e) {
  const result = e.data.map(item => complexCalculation(item));
  postMessage(result);
};

// React component
const worker = useMemo(() => new Worker('./worker.js'), []);

useEffect(() => {
  worker.onmessage = (e) => setProcessedData(e.data);
  return () => worker.terminate();
}, [worker]);

const processData = (rawData) => {
  worker.postMessage(rawData);
};

4. Streaming Responses

// ✅ Process data as it streams
async function processStream(response) {
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let result = '';

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    result += decoder.decode(value, { stream: true });
    // Process partial results as they arrive
    updatePartialResults(JSON.parse(result));
  }
}

fetch('/api/large-stream')
  .then(processStream);

5. Memoization and Selective Rendering

// ✅ Only recompute when necessary
const processedData = useMemo(() => {
  return largeDataset.map(item => ({
    ...item,
    summary: createSummary(item.content)
  }));
}, [largeDataset]); // Only recomputes when dataset changes

// ✅ Use React.memo for child components
const DataRow = React.memo(({ item }) => {
  return <div>{item.name}</div>;
});

Advanced Techniques

1. Progressive Loading with Suspense

// ✅ Load data progressively
const resource = fetchData('/api/large-data'); // Returns a promise

function DataComponent() {
  const data = resource.read(); // Suspense integration
  return <DataView data={data} />;
}

function App() {
  return (
    <Suspense fallback={<Loader />}>
      <DataComponent />
    </Suspense>
  );
}

2. IndexedDB for Client-Side Caching

// ✅ Store large data in IndexedDB
async function cacheLargeData(data) {
  const db = await openDB('MyDB', 1, {
    upgrade(db) {
      db.createObjectStore('largeData');
    }
  });
  await db.put('largeData', data, 'dataset');
}

// Later retrieve only needed portions
async function getCachedData(range) {
  const db = await openDB('MyDB');
  const allData = await db.get('largeData', 'dataset');
  return allData.slice(range.start, range.end);
}

3. Server-Side Filtering and Projections

// ✅ Only request needed fields
fetch('/api/data?fields=id,name,date&filter=active')

// ✅ Use GraphQL for precise data requirements
const query = `
  query {
    largeDataset(limit: 100) {
      id
      name
      stats {
        views
      }
    }
  }
`;

Best Practices

1. Don’t load everything upfront – Implement pagination or lazy loading
2. Avoid main thread blocking – Use web workers for heavy computations
3. Minimize re-renders – Memoize processed data and components
4. Leverage browser capabilities – Use streaming, IndexedDB, virtualization
5. Push processing to the server – Filter/sort/transform data before sending
6. Implement caching strategies – Store data intelligently client-side
7. Use compression – Enable gzip/brotli on server responses

Performance Monitoring

// Track loading performance
const start = performance.now();
fetchLargeData().then(() => {
  const duration = performance.now() - start;
  trackMetrics('data_load', duration);
});

// Monitor memory usage
if (window.performance?.memory) {
  console.log(
    `Used JS heap: ${(performance.memory.usedJSHeapSize / 1048576).toFixed(2)} MB`
  );
}

By implementing these strategies, you can ensure your React application handles large datasets efficiently while maintaining smooth performance and responsive user interfaces.