The error MemoryError: out of memory while creating large objects occurs when Python runs out of RAM while trying to allocate a large amount of memory for an object. This typically happens when handling large lists, NumPy arrays, Pandas DataFrames, or complex recursive functions.
1. Common Causes and Solutions
Cause 1: Creating Large Lists or Tuples in Memory
Python stores lists and tuples in RAM, so creating huge collections can lead to memory exhaustion.
Example (Creating a Huge List in Memory)
big_list = [i for i in range(10**9)] # List with 1 billion elements
This consumes several gigabytes of RAM.
Solution: Use generators instead of lists:
big_generator = (i for i in range(10**9)) # Uses less memory
Or use NumPy arrays with smaller data types:
import numpy as np
big_array = np.arange(10**9, dtype=np.int32) # Uses less memory than default int64
Cause 2: Loading Large Files into Memory
Reading large files all at once can cause MemoryError.
Example (Loading a Huge File)
with open("large_file.txt", "r") as file:
data = file.read() # Loads entire file into memory
Solution: Read line by line:
with open("large_file.txt", "r") as file:
for line in file:
process(line) # Process without loading everything
For Pandas CSV files:
import pandas as pd
chunk_size = 100000 # Process 100,000 rows at a time
for chunk in pd.read_csv("huge_dataset.csv", chunksize=chunk_size):
process(chunk)
Cause 3: Copying Large Objects in Memory
Python duplicates objects when assigned incorrectly, consuming double the memory.
Example (Unnecessary Copying)
large_data = [1] * 10**8 # Large list
copy_data = large_data[:] # Creates a duplicate in memory
Solution: Use references instead of copies:
copy_data = large_data # No extra memory usage
For NumPy:
import numpy as np
big_array = np.array([1] * 10**8)
copy_array = np.copy(big_array) # Avoid unnecessary copies
Cause 4: Keeping Unused Objects in Memory
Python holds objects in memory until garbage collection.
Solution: Use del and gc.collect() to free memory:
import gc
large_data = [0] * 10**8
del large_data # Remove reference
gc.collect() # Force garbage collection
Cause 5: NumPy or Pandas Using Too Much RAM
Operations on large NumPy arrays or Pandas DataFrames can exceed memory limits.
Example (Creating a Large NumPy Array)
import numpy as np
big_array = np.ones((100000, 100000)) # Extremely large matrix
Solution: Use memory-efficient data types:
big_array = np.ones((100000, 100000), dtype=np.float32) # Uses less memory
For Pandas:
df = pd.read_csv("huge_dataset.csv", dtype={"col1": "int32", "col2": "float32"})
Cause 6: Inefficient Recursive Functions
Uncontrolled recursion can cause memory overflows.
Example (Infinite Recursion)
def infinite_recursion(n):
return infinite_recursion(n + 1)
infinite_recursion(0) # Never stops
Solution: Use iterative loops or limit recursion:
import sys
sys.setrecursionlimit(1000) # Reduce memory usage
Cause 7: Running Python on 32-bit Version
A 32-bit Python version has a memory limit of ~2GB.
Solution: Upgrade to 64-bit Python to use more RAM.
Cause 8: Running Out of Swap Space
If your system has low RAM and no swap space, Python crashes with MemoryError.
Solution: Increase swap memory (Linux/macOS):
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
For Windows, increase virtual memory in System Properties → Advanced → Performance Settings.
2. Summary of Fixes
| Issue | Fix |
|---|---|
| Creating large lists/arrays | Use generators or numpy.int32 |
| Loading large files | Use chunksize or readline() |
| Copying large data structures | Use references instead of deep copies |
| Keeping unused objects | Use del and gc.collect() |
| Running out of swap space | Increase swap memory |
| Recursive functions | Use iterative loops or limit recursion |
| Running 32-bit Python | Upgrade to 64-bit Python |