Too many open file handles

When your Python program opens too many files without closing them properly, it exceeds the system limit, leading to the error:

OSError: [Errno 24] Too many open files

or

IOError: [Errno 24] Too many open files

---

1. Causes of the Error

• Files Not Closed Properly → Forgetting to close files manually.
• Loop Opening Files Without Closing → Opening multiple files in a loop without closing.
• Multiple Processes/Threads → Each thread/process keeps files open, exceeding the system limit.
• Sockets, Pipes, or Database Connections → Open network sockets or database connections count toward the file limit.
• Loggers or File-Based Caches → Keeping log files open without closing them.

---

2. How to Fix It?

Solution 1: Use with open() to Auto-Close Files

Instead of manually closing files, use Python’s with open() to automatically close them after use.

Bad Example (Not Closing Files)

for i in range(10000):  
    f = open(f'file_{i}.txt', 'w')  # Opens files without closing
    f.write("Some text")  

Good Example (Using with open())

for i in range(10000):
    with open(f'file_{i}.txt', 'w') as f:
        f.write("Some text")  # File automatically closes after exiting 'with' block

---

Solution 2: Manually Close Files in Loops

If you must use open(), make sure to close the file explicitly.

Bad Example (Not Closing Files)

files = []
for i in range(10000):
    f = open(f'file_{i}.txt', 'w')  # Opens files but never closes them
    files.append(f)  # Memory keeps growing

Good Example (Closing Files Manually)

files = []
for i in range(10000):
    f = open(f'file_{i}.txt', 'w')
    f.write("Some text")
    f.close()  # Closes file to prevent reaching the limit

---

Solution 3: Increase File Handle Limit (Linux/macOS Only)

Check the current limit using:

ulimit -n

Increase it temporarily (for the session):

ulimit -n 100000

Increase it permanently by editing /etc/security/limits.conf:

* soft nofile 100000  
* hard nofile 200000  

---

Solution 4: Close Unused File Handles in Logging

If using logging, make sure log files close properly.

Bad Example (Logging Keeps Files Open)

import logging

logging.basicConfig(filename='app.log', level=logging.INFO)

for i in range(10000):
    logging.info(f'Iteration {i}')

Good Example (Rotating Log Files)

from logging.handlers import RotatingFileHandler

handler = RotatingFileHandler("app.log", maxBytes=1024 * 1024, backupCount=5)
logging.getLogger().addHandler(handler)

---

Solution 5: Close Database and Socket Connections

For databases and network connections, always close them when done.

Close Database Connections

import sqlite3

conn = sqlite3.connect("database.db")
cursor = conn.cursor()
cursor.execute("SELECT * FROM users")

conn.close()  # Ensure database connection is closed

Close Socket Connections

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("example.com", 80))
s.close()  # Prevents too many open sockets

---

3. Summary of Fixes

Cause	Fix
Forgetting to close files	Use with open() or f.close()
Opening files in a loop without closing	Close files inside the loop
Too many open sockets/connections	Close them after use (s.close(), conn.close())
Logging keeping files open	Use RotatingFileHandler
System file limit too low	Increase using ulimit -n 100000