The error MemoryError: cannot allocate memory occurs when Python runs out of memory while trying to allocate a large amount of RAM for a variable, object, or operation. This usually happens when handling large datasets, inefficient loops, or memory-intensive computations.
1. Common Causes and Solutions
Cause 1: Processing Large Data in Memory
If you try to load an extremely large file, list, or dataset into memory, Python might not have enough RAM to handle it.
Example (Reading a Large File into Memory at Once)
with open("large_file.txt", "r") as file:
data = file.read() # This loads the entire file into memory
If the file is several gigabytes, this can cause a MemoryError.
Solution: Read the file line by line instead:
with open("large_file.txt", "r") as file:
for line in file:
process(line) # Process each line without loading the entire file
Cause 2: Creating Large Lists or Arrays
If you create a huge list or NumPy array, it may exceed available memory.
Example (Creating a Huge List in Memory)
big_list = [i for i in range(10**9)] # List with 1 billion elements
Solution: Use generators instead of lists to save memory:
big_generator = (i for i in range(10**9)) # Uses less memory
Or use NumPy arrays with efficient data types:
import numpy as np
big_array = np.arange(10**9, dtype=np.int32) # Uses less memory than default int64
Cause 3: Inefficient Looping and Copying
Unnecessary copies of data structures waste memory.
Example (Copying a Large List)
data = [1] * 10**8 # Large list
copy_data = data[:] # Creates a duplicate, doubling memory usage
Solution: Use the original reference instead of copying:
copy_data = data # No extra memory usage
If modification is needed, use slicing wisely:
copy_data = data[:len(data)//2] # Keep only half the data
Cause 4: Large Pandas DataFrames
Pandas loads entire datasets into memory, which can be inefficient.
Example (Reading a Large CSV Directly into Pandas)
import pandas as pd
df = pd.read_csv("huge_dataset.csv") # May exceed available RAM
Solution: Read the file in chunks:
chunk_size = 100000 # Process 100,000 rows at a time
for chunk in pd.read_csv("huge_dataset.csv", chunksize=chunk_size):
process(chunk) # Process smaller chunks instead of loading all at once
Cause 5: Running Out of Swap Space
If your system has low RAM and insufficient swap space, it may trigger a MemoryError.
Solution: Increase swap memory (Linux/macOS):
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
For Windows, increase virtual memory in System Properties → Advanced → Performance Settings.
Cause 6: Recursive Functions Without Limits
Deep recursive functions can cause excessive memory usage.
Example (Uncontrolled Recursion)
def infinite_recursion(n):
return infinite_recursion(n + 1) # Never stops
infinite_recursion(0)
Solution: Use iterative loops instead of recursion or set a recursion limit:
import sys
sys.setrecursionlimit(1000) # Reduce memory usage
Cause 7: Keeping Unused Variables in Memory
Python keeps variables in memory until they are garbage collected.
Solution: Use del and gc.collect() to free memory:
import gc
large_data = [0] * 10**8 # Large list
del large_data # Remove reference
gc.collect() # Force garbage collection
Cause 8: Running on a 32-bit Python Version
A 32-bit Python installation has a memory limit of ~2GB, which may cause MemoryError even if you have more RAM.
Solution: Upgrade to 64-bit Python to utilize more memory.
2. Summary of Fixes
| Issue | Fix |
|---|---|
| Reading large files into memory | Use readline() or read_csv(chunksize=100000) |
| Creating large lists/arrays | Use generators or numpy.int32 |
| Copying large data structures | Use references instead of deep copies |
| Large Pandas DataFrames | Process in chunks instead of loading at once |
| Running out of RAM | Increase swap space or upgrade hardware |
| Uncontrolled recursion | Use iterative loops or sys.setrecursionlimit() |
| Unused variables taking memory | Use del and gc.collect() |
| Running 32-bit Python | Upgrade to 64-bit Python |