How to Read and Write Binary Files in Python: A Complete Guide

1. Introduction

1.1 What Is a Binary File?

• Image files (PNG, JPEG, BMP, etc.)
• Audio files (WAV, MP3, AAC, etc.)
• Video files (MP4, AVI, MOV, etc.)
• Compressed files (ZIP, RAR, GZ, etc.)
• Executable files (EXE, DLL, BIN, etc.)

1.2 Difference Between Text and Binary Files

Main Differences

1. Data Structure

• Text files contain only data interpretable as characters.
• Binary files can contain any type of data (images, audio, executable code, etc.).

1. Size

• Text files are usually smaller when storing small amounts of data.
• Binary files may be larger due to encoding overhead.

1. Editing Method

• Text files can be opened and edited directly with editors like Notepad or VS Code.
• Binary files require special tools (e.g., binary editors) for editing.

1.3 Why Handle Binary Files in Python?

① Processing Images and Audio Data

# Example: Read a PNG image as binary
with open("image.png", "rb") as file:
    binary_data = file.read()
    print(binary_data[:20])  # Display the first 20 bytes

② Analyzing Compressed Data

import gzip

# Example: Open a GZ compressed file
with gzip.open("example.gz", "rb") as file:
    content = file.read()
    print(content)

③ Parsing Binary Protocols

import struct

# Example: Convert binary data to integer
binary_data = b'
   '  # 4 bytes of data
integer_value = struct.unpack('<I', binary_data)[0]
print(integer_value)  # Output: 1

1.4 Summary

• Binary files store information such as images, audio, and compressed data.
• Unlike text files, data is stored as sequences of 0s and 1s.
• Python enables parsing, processing, and converting binary data.
• Using Python’s open() function and struct module, binary files can be handled efficiently.

2. How to Read Binary Files in Python and Basic Operations

2.1 Reading Binary Files with the open() Function

Basic Syntax

file = open("example.bin", "rb")  # 'rb' means "read in binary mode"
binary_data = file.read()  # Read the file content
file.close()  # Close the file

2.2 Safe Binary Reading with the with Statement

Example: Safe Reading of Binary Data

with open("example.bin", "rb") as file:
    binary_data = file.read()

# Once the with-block ends, the file is automatically closed

Benefits of Using the with Statement

1. No need to call file.close() (it closes automatically)
2. No resource leaks even if an error occurs
3. Code becomes simpler and more readable

2.3 Variations of Reading Methods

① Read the Entire File at Once (read())

with open("example.bin", "rb") as file:
    binary_data = file.read()  # Read the entire file at once

• Simple and easy to understand
• Efficient for small files

• For large files (hundreds of MB to GB), memory usage may become excessive

② Read in Fixed-Size Chunks (read(n))

with open("example.bin", "rb") as file:
    chunk = file.read(1024)  # Read 1024 bytes (1KB) at a time
    while chunk:
        print(chunk)  # Process the chunk
        chunk = file.read(1024)  # Read the next 1KB

• Reduces memory usage
• Efficient for handling large files

• Not suitable if you need the entire file loaded at once

③ Read Binary Data Line by Line (readline())

with open("example.bin", "rb") as file:
    line = file.readline()  # Read one line
    while line:
        print(line)
        line = file.readline()  # Read the next line

• Binary log files or binary data that includes newlines

• If no newline exists, the entire content is treated as one line, so this method is valid only for specific files

2.4 File Positioning with seek()

Basic Syntax of seek()

file.seek(offset, whence)

Example: Reading Data from the Middle of a File

with open("example.bin", "rb") as file:
    file.seek(10)  # Move to the 10th byte from the beginning
    data = file.read(5)  # Read 5 bytes
    print(data)

• Extract header information
• Analyze specific sections of data

2.5 Getting the Current File Position with tell()

Example: Checking File Position

with open("example.bin", "rb") as file:
    file.read(10)  # Read 10 bytes
    position = file.tell()  # Get the current position
    print(f"Current file position: {position} bytes")

• Check how much of the file has been read
• Debugging when processing files mid-way

2.6 Summary

• Use 'rb' mode when opening binary files in Python
• The with statement ensures files are closed safely
• read() loads the entire file into memory, which may be inefficient for large files — use read(n) instead
• seek() allows moving to any file position, and tell() retrieves the current position

3. Efficient Methods for Reading Binary Files in Python

3.1 Reading the Entire Binary File at Once (read())

Basic Syntax

with open("example.bin", "rb") as file:
    binary_data = file.read()

• Simple and intuitive
• Suitable for small files (a few MB or less)

• Consumes a lot of memory if the file is large (hundreds of MB or more)
• May cause crashes if the file size exceeds available memory

Example

with open("sample.bin", "rb") as file:
    binary_data = file.read()
    print(len(binary_data))  # Display file size in bytes

3.2 Reading in Fixed-Size Chunks (read(n))

Basic Syntax

with open("example.bin", "rb") as file:
    chunk = file.read(1024)  # Read 1024 bytes (1KB) at a time
    while chunk:
        print(chunk)
        chunk = file.read(1024)  # Read the next 1KB

• Reduces memory usage when processing large files
• Supports streaming-like processing

• Additional processing required for real-time handling

Example

with open("large_file.bin", "rb") as file:
    while True:
        chunk = file.read(4096)  # Read 4KB at a time
        if not chunk:
            break  # End when no more data
        print(f"Read {len(chunk)} bytes")

3.3 Reading Binary Data Line by Line (readline())

Basic Syntax

with open("example.bin", "rb") as file:
    line = file.readline()
    while line:
        print(line)
        line = file.readline()

• Binary log files or binary data that includes newline characters

• If no newlines exist, the entire file will be treated as one line
• Not commonly used in general binary processing

3.4 Reading Data from Specific File Positions (seek())

Basic Syntax

file.seek(offset, whence)

Example: Reading Data from a Specific Position

with open("example.bin", "rb") as file:
    file.seek(10)  # Move to the 10th byte from the beginning
    data = file.read(5)  # Read 5 bytes
    print(data)

3.5 Getting the Current File Position (tell())

Example

with open("example.bin", "rb") as file:
    file.read(20)  # Read 20 bytes
    position = file.tell()  # Get the current position
    print(f"Current position: {position} bytes")

• Check how much of the file has been read
• Debug processing within the file

3.6 Using Memory-Mapped Files for Faster Reading (mmap)

Basic Syntax

import mmap

with open("example.bin", "rb") as file:
    with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mmapped_file:
        print(mmapped_file[:100])  # Get the first 100 bytes

• Faster access since the file is handled in memory
• Allows direct access to specific ranges

• More complex than standard file handling
• May cause memory issues if the file is extremely large

3.7 Summary

• Efficient reading of binary files requires choosing methods based on file size and purpose
• Use read() for small files
• Use read(n) for large files to process them in chunks
• Use seek() to retrieve data from specific positions
• Use mmap for fast access, but apply carefully depending on the case

4. How to Analyze Read Binary Data

4.1 What Is the struct Module?

Main Features

• Convert binary data into integers, floats, and strings (unpacking)
• Convert values into binary format (packing)
• Specify endianness (byte order)

Basic Syntax of the struct Module

import struct

# Convert binary data into specific types (unpack)
data = struct.unpack(format, binary_data)

# Convert values into binary data (pack)
binary_data = struct.pack(format, value1, value2, ...)

4.2 Unpacking (Decoding) Binary Data

Example: Decode a 4-Byte Integer (32-bit)

import struct

# Binary data (4 bytes)
binary_data = b'
   '  # Represents 1 (Little Endian)

# Unpack as an unsigned 32-bit integer in Little Endian
value = struct.unpack('<I', binary_data)[0]
print(value)  # Output: 1

4.3 Parsing String Data

Example: Decode a 10-Byte String

import struct

# 10-byte binary data
binary_data = b'HelloWorld'

# Interpret as string
decoded_string = struct.unpack('10s', binary_data)[0].decode('utf-8')
print(decoded_string)  # Output: HelloWorld

Notes

• By specifying the byte size (e.g., 10s), you can extract fixed-length strings.
• Use .decode('utf-8') to convert into the correct character encoding.

4.4 Parsing Floating-Point Numbers

Example: Decode a 4-Byte Float

import struct

# 4-byte float (IEEE 754 format)
binary_data = b'  €?'  # Represents 1.0

# Unpack
value = struct.unpack('<f', binary_data)[0]
print(value)  # Output: 1.0

4.5 Specifying Endianness (Byte Order)

Example: Difference Between Little and Big Endian

import struct

binary_data = b'
   '  # Represents 1 (Little Endian)

# Little Endian (least significant byte first)
little_endian = struct.unpack('<I', binary_data)[0]
print(f"Little Endian: {little_endian}")  # Output: 1

# Big Endian (most significant byte first)
big_endian = struct.unpack('>I', binary_data)[0]
print(f"Big Endian: {big_endian}")  # Output: 16777216

4.6 Summary

• The struct module allows parsing binary data into numbers and strings
• Understanding endianness is essential for accurate decoding
• You can efficiently parse file headers and structured data by unpacking multiple data types together

5. How to Create and Write Binary Files in Python

5.1 Writing Files in Binary Mode

Basic Syntax

with open("example.bin", "wb") as file:
    file.write(binary_data)

• 'wb' means “write in binary mode”.
• Use the write() method to write binary data into a file.

Simple Example: Writing Binary Data

with open("output.bin", "wb") as file:
    file.write(b'
')  # Write 4 bytes of data

• b'
' represents 4 bytes of binary data (in hexadecimal format).
• The actual file size will be 4 bytes.

5.2 Creating Binary Data with struct.pack()

Basic Syntax

import struct
binary_data = struct.pack(format, value1, value2, ...)

• format specifies the binary data type (integer, float, string, etc.).
• value1, value2, ... are the values to be packed.

5.3 Writing Numeric Data to a Binary File

Example: Convert Integers to Binary and Write

import struct

# Unsigned 16-bit integer (H) and unsigned 32-bit integer (I)
binary_data = struct.pack('<HI', 512, 123456789)

# Write to file
with open("numbers.bin", "wb") as file:
    file.write(binary_data)

• <HI → Little Endian: Unsigned 16-bit integer (2 bytes) + Unsigned 32-bit integer (4 bytes)
• 512 →  (hex: 0x0200)
• 123456789 → Í[ (hex: 0x075BCD15)

5.4 Writing String Data to a Binary File

Example: Writing a 10-Byte String

import struct

text = "Hello"
binary_data = struct.pack('10s', text.encode('utf-8'))  # Fixed length: 10 bytes

with open("text.bin", "wb") as file:
    file.write(binary_data)

• '10s' means a fixed-length string of 10 bytes.
• Convert to binary with encode('utf-8').
• If the string is shorter than 10 bytes, padding (null bytes) is added automatically.

5.5 Writing Floating-Point Numbers to a Binary File

Example: Write a Float

import struct

float_value = 3.14
binary_data = struct.pack('<f', float_value)  # 4-byte float

with open("float.bin", "wb") as file:
    file.write(binary_data)

• <f means a Little Endian 4-byte floating-point number.
• The IEEE 754 representation of 3.14 is 0xC3F54840.

5.6 Appending to a Binary File ('ab' Mode)

Example: Append Data to a Binary File

with open("output.bin", "ab") as file:
    file.write(b'ÿÿ')  # Write additional data

• 'ab' means “append in binary mode”.
• Data is added at the end without overwriting existing content.

5.7 Writing Data to Specific Positions with seek()

Example: Write Data at the 10th Byte from the Beginning

with open("output.bin", "r+b") as file:
    file.seek(10)  # Move to the 10th byte
    file.write(b'ª»')  # Write 2 bytes

• 'r+b' mode means “read/write in binary mode”.
• seek(10) moves to the 10th byte from the beginning before writing.

5.8 Summary

• Use 'wb' mode in Python to write binary files
• struct.pack() converts numbers, strings, and floats into binary
• 'ab' (append mode) allows adding data to existing binary files
• seek() lets you write data at specific positions in a file

6. Practical Examples of Handling Binary Files in Python

6.1 Reading and Writing Image Files

Example: Copy an Image File

# Read binary image data
with open("input.png", "rb") as infile:
    image_data = infile.read()

# Write binary image data to another file
with open("copy.png", "wb") as outfile:
    outfile.write(image_data)

• The image data is read in binary and written as-is to another file.
• This ensures that the image is copied without corruption.

6.2 Reading and Writing Audio Files

Example: Read the Header of a WAV File

with open("sample.wav", "rb") as file:
    header = file.read(44)  # The header of a WAV file is 44 bytes
    print(header)

• By reading the first 44 bytes, you can obtain WAV file metadata such as sample rate and channel count.

6.3 Handling Compressed Files (ZIP/GZ)

Example: Extract a ZIP File

import zipfile

with zipfile.ZipFile("archive.zip", "r") as zip_ref:
    zip_ref.extractall("extracted")  # Extract contents into the "extracted" folder

Example: Read a GZ Compressed File

import gzip

with gzip.open("compressed.gz", "rb") as file:
    content = file.read()
    print(content[:50])  # Display the first 50 bytes

6.4 Reading and Writing Executable Files

Example: Copy an Executable File

with open("program.exe", "rb") as infile:
    exe_data = infile.read()

with open("copy.exe", "wb") as outfile:
    outfile.write(exe_data)

• When handling executable files, never attempt to edit them without specialized knowledge.
• Improper modifications may cause the file to become corrupted or insecure.

6.5 Using struct to Parse Binary Data Structures

Example: Parse a Custom Binary Format

• 2-byte unsigned integer
• 4-byte float
• 10-byte string

import struct

binary_data = struct.pack("<Hf10s", 1000, 3.14, b"HelloWorld")

# Decode (unpack)
value1, value2, value3 = struct.unpack("<Hf10s", binary_data)
print(value1, value2, value3.decode("utf-8"))

1000 3.14 HelloWorld

6.6 Summary

• Binary files are widely used for images, audio, video, compressed data, and executables.
• Python’s standard library supports binary file processing with modules such as zipfile, gzip, and struct.
• By combining reading, writing, and parsing, you can handle various binary file formats.

7. Common Pitfalls and Best Practices When Handling Binary Files

7.1 Common Pitfalls

1. Opening Files Without Specifying Binary Mode

• If you open a binary file without using 'rb' or 'wb', Python may attempt to interpret it as text, potentially causing corruption or errors.

2. Memory Overuse When Reading Large Files

• Using read() loads the entire file into memory at once. For files of several hundred MB or GB, this may cause MemoryError.

3. Ignoring Endianness (Byte Order)

• If you decode binary data without specifying the correct endianness, values may be misinterpreted.

4. Editing Executable Files Without Knowledge

• Improper editing of binary executables (EXE, DLL, etc.) may corrupt the file or cause security issues.

5. Incorrect String Encoding

• Binary strings must be decoded with the correct .decode('utf-8') or other encoding. Using the wrong encoding may cause garbled text or errors.

7.2 Best Practices

1. Always Use the with Statement

• The with statement ensures files are automatically closed, preventing resource leaks.

2. Use read(n) for Large Files

• For large files, read them in chunks to reduce memory usage.

3. Specify Endianness Explicitly

• When using struct.unpack(), always specify < (Little Endian) or > (Big Endian).

4. Validate File Format Before Processing

• Check headers or magic numbers to ensure the file type is correct before reading.

5. Use Appropriate Libraries

• For images: PIL/Pillow
• For audio: wave, pydub
• For compressed files: zipfile, gzip

6. Use Memory-Mapped Files (mmap) for Performance

• Efficiently process large files without loading the entire file into memory.

7.3 Summary

• Specify binary mode ('rb', 'wb') when opening binary files.
• Use with for safe file handling.
• Read in chunks for large files.
• Be aware of endianness when decoding.
• Use specialized libraries for each file format.

8. Conclusion

Key Takeaways

• Binary files store data as sequences of bytes rather than human-readable text.
• Use 'rb', 'wb', 'ab' modes to properly read, write, or append binary data.
• The struct module allows converting binary data into integers, floating-point numbers, and strings.
• Endianness (byte order) must be correctly specified to ensure accurate data interpretation.
• Large binary files should be processed in chunks or via mmap to optimize memory usage.
• Best practices: always use the with statement, handle exceptions, and use specialized libraries depending on the file type.

Final Thoughts