Python Built-in Binary Data Types

Python's built-in binary data types provide sequence-oriented representations of byte values. These types are essential for storing, inspecting, transforming, transmitting, and exchanging binary information in memory and across systems. The two primary binary sequence types are the immutable bytes and the mutable bytearray. They represent collections of byte values distinct from Unicode text, which is handled by the str type.

Foundations of Binary Data in Python

A byte is an integer value in the range 0 through 255 inclusive. Binary sequences in Python are ordered collections of such byte values, not sequences of Unicode characters. This distinction is crucial because binary data represents raw byte content, which might correspond to encoded text, images, protocol frames, or arbitrary data, whereas Unicode text is a sequence of abstract characters.

The principal relationship among the three key built-in sequence types is:

• bytes and bytearray contain byte values (integers from 0 to 255).
• str contains Unicode text (abstract characters).

Both bytes and bytearray share many sequence operations such as indexing, slicing, iteration, and concatenation. However, they differ critically in mutability, hashability, supported modification operations, and typical use cases:

• bytes is immutable and hashable.
• bytearray is mutable and not hashable.

Example constructions and inspections:

b = b'Hello\x20World'
ba = bytearray([72, 101, 108, 108, 111])
print(type(b), len(b), list(b), b)
print(type(ba), len(ba), list(ba), ba)

Output might be:

<class 'bytes'> 11 [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100] b'Hello World'
<class 'bytearray'> 5 [72, 101, 108, 108, 111] bytearray(b'Hello')

Note that the displayed representation of binary data may contain printable ASCII characters and escape notation (e.g., \x20 for space), but this does not imply the object is Unicode text. It is simply a representation showing printable bytes in ASCII and escaping others for clarity.

Python Bytes Type

The bytes type represents an immutable sequence of integers in the range 0 through 255. It is suitable for fixed binary content such as encoded text, protocol data, file contents, and other byte-oriented representations.

Bytes literals use the b prefix before a string of ASCII characters and escape sequences. Supported literal characters include ASCII printable characters and common escapes like \n, \t, and hexadecimal byte escapes like \xHH. Direct literal content is ASCII-oriented, so non-ASCII characters must be escaped or constructed differently.

The bytes constructor supports multiple forms:

• bytes(integer) creates a zero-initialized bytes object of the specified length.
• bytes(iterable_of_int) creates bytes from an iterable of integers in 0–255.
• bytes(binary_obj) copies from another binary object (bytes or bytearray).
• bytes(str_obj, encoding) encodes a Unicode string into bytes using the specified encoding.

Examples:

b_literal = b'ABC'
b_len = bytes(4)
b_list = bytes([65, 66, 67])
b_encoded = bytes("ABC", "utf-8")

print(b_literal)
print(b_len)
print(b_list)
print(b_encoded)

Output:

b'ABC'
b'\x00\x00\x00\x00'
b'ABC'
b'ABC'

Indexing and slicing:

• Indexing a bytes object yields an integer byte value.
• Slicing a bytes object produces another bytes object.

Examples:

b = b'Python'

print(b[0])       # 80 (integer byte for 'P')
print(b[-1])      # 110 (integer byte for 'n')
print(b[1:4])     # b'yth' (bytes)
print(b[::2])     # b'Pto' (bytes)

# Iteration
for byte in b:
    print(byte, end=' ')
print()

# Membership
print(111 in b)   # True, 111 is byte for 'o'

# Construct new bytes from a slice
new_b = b[1:4] + b'123'
print(new_b)

Bytes are immutable: item assignment, slice assignment, or in-place modification is not supported. Transformations like concatenation and replacement produce new bytes objects.

b1 = b'abc'
b2 = b'def'
b3 = b1 + b2
print(b3)  # b'abcdef'

b4 = b1 * 3
print(b4)  # b'abcabcabc'

# The following will raise an error:
# b1[0] = 100  # TypeError: 'bytes' object does not support item assignment

bytes methods include searching (find), counting (count), prefix/suffix testing (startswith, endswith), splitting (split, partition), joining (join), replacing (replace), and stripping (strip). These operate on byte values and sequences, not Unicode text.

Example:

b = b'abcabcabc'

print(b.find(b'bc'))       # 1
print(b.count(b'a'))       # 3
print(b.startswith(b'ab')) # True
print(b.split(b'c'))       # [b'ab', b'ab', b'ab', b'']
print(b.partition(b'c'))   # (b'ab', b'c', b'abcabc')
print(b.join([b'x', b'y']))# b'xbxy'
print(b.replace(b'a', b'z')) # b'zbczbczbc'
print(b.strip(b'abc'))     # b'' (all removed)

Python Bytearray Type

bytearray is a mutable sequence of integers in the range 0 through 255. It supports many of the same operations as bytes but allows in-place modification.

Construction forms mirror those of bytes:

• bytearray(integer) creates a zero-initialized mutable sequence.
• bytearray(iterable_of_int) creates from iterable of bytes.
• bytearray(binary_obj) copies from binary data.
• bytearray(str_obj, encoding) encodes from Unicode text.

Examples:

ba_len = bytearray(4)
ba_list = bytearray([65, 66, 67])
ba_copy = bytearray(ba_list)
ba_encoded = bytearray("ABC", "utf-8")

print(ba_len)      # bytearray(b'\x00\x00\x00\x00')
print(ba_list)     # bytearray(b'ABC')
print(ba_copy)     # bytearray(b'ABC')
print(ba_encoded)  # bytearray(b'ABC')

Item and slice assignment allow replacement, insertion, and deletion (via slice assignment with an empty sequence). Assigned values must be integers 0–255.

Example mutation:

ba = bytearray(b'Hello')
ba[0] = 74                  # Change 'H' (72) to 'J' (74)
print(ba)                   # bytearray(b'Jello')

ba[1:3] = b'ey'             # Replace 'el' with 'ey'
print(ba)                   # bytearray(b'Jeylo')

ba[3:3] = b'!!!'            # Insert '!!!' at index 3
print(ba)                   # bytearray(b'Jey!!!lo')

del ba[6:8]                 # Delete 'lo'
print(ba)                   # bytearray(b'Jey!!!')

Mutation methods include:

• append(byte) – add a single byte at the end.
• extend(iterable) – append multiple bytes.
• insert(index, byte) – insert a byte.
• pop([index]) – remove and return a byte.
• remove(byte) – remove first occurrence.
• reverse() – reverse in place.
• clear() – empty the bytearray.

Example:

ba = bytearray(b'abc')
ba.append(100)        # add 'd'
print(ba)             # bytearray(b'abcd')

ba.extend(b'efg')
print(ba)             # bytearray(b'abcdefg')

ba.insert(2, 120)     # insert 'x' at index 2
print(ba)             # bytearray(b'abxcdefg')

ba.pop()
print(ba)             # bytearray(b'abxcdef')

ba.remove(120)        # remove 'x'
print(ba)             # bytearray(b'abcdef')

ba.reverse()
print(ba)             # bytearray(b'fedcba')

ba.clear()
print(ba)             # bytearray(b'')

Aliasing with mutable bytearray objects means two names referring to the same bytearray see all mutations. Creating a copy produces an independent object.

Example:

ba1 = bytearray(b'abc')
ba2 = ba1       # alias
ba3 = ba1.copy()  # independent copy

ba2[0] = 120    # modify through alias
print(ba1)      # bytearray(b'xbc')
print(ba2)      # bytearray(b'xbc')

ba3[1] = 121
print(ba3)      # bytearray(b'xyc')
print(ba1)      # bytearray(b'xbc'), unaffected by ba3 modifications

Binary Sequence Operations

Common operations supported by both bytes and bytearray include:

• Length: len(obj) returns the number of bytes.
• Iteration: iterates over integer byte values.
• Membership: tests for integer byte values or subsequences.
• Concatenation: + joins sequences of the same type.
• Repetition: * repeats the sequence.
• Comparison: lexicographic ordering by byte values.
• Indexing: yields an integer byte value.
• Slicing: yields a new binary sequence of the same type.

Membership testing is nuanced:

• x in obj can test for an integer byte value (0–255).
• Searching for a multi-byte subsequence requires a bytes or bytearray argument.

Lexicographic comparison compares byte values element-wise. This ordering is not the same as numeric interpretation of entire sequences nor natural-language text ordering.

Examples:

b = b'abc'
ba = bytearray(b'abc')

print(len(b), len(ba))
print(list(b), list(ba))

print(98 in b)        # True (byte for 'b')
print(b'bc' in b)     # True (subsequence)
print(99 in ba)       # True

print(b + b'def')     # b'abcdef'
ba.extend(b'def')
print(ba)             # bytearray(b'abcdef')

print(b[1])           # 98
print(ba[1])          # 98

print(b[1:3])         # b'bc'
print(ba[1:3])        # bytearray(b'bc')

print(b == ba)        # True (value equality)
print(b < b'def')     # True (lexicographic)

Construction, Conversion, and Numeric Byte Values

Conversion between bytes and bytearray creates a new object of the target type:

• bytes(bytearray_obj) creates an immutable copy.
• bytearray(bytes_obj) creates a mutable copy.

Simple assignment only creates a new reference.

Individual byte values are integers 0 through 255. Indexing bytes or bytearray returns an integer byte value. Constructing new binary sequences from iterables requires all elements to be valid bytes.

Example conversions:

b = b'abc'
ba = bytearray(b)

print(type(b), b)
print(type(ba), ba)

# Extract integer byte
print(b[0])       # 97

# Reconstruct from integers
new_b = bytes([97, 98, 99])
print(new_b)      # b'abc'

# Attempt invalid construction: raises ValueError
try:
    invalid_b = bytes([256])
except ValueError as e:
    print("Error:", e)

Integer-to-bytes and bytes-to-integer conversions use int.to_bytes and int.from_bytes:

• Specify byte length (number of bytes).
• Specify byte order: 'big' or 'little'.
• Specify signedness: signed=True or False.

These methods encode numeric values to fixed-length byte sequences and decode them accordingly. This is different from encoding text strings.

Examples:

x = 1025
b_big = x.to_bytes(2, byteorder='big')
b_little = x.to_bytes(2, byteorder='little')

print(b_big)        # b'\x04\x01'
print(b_little)     # b'\x01\x04'

y = int.from_bytes(b_big, byteorder='big')
z = int.from_bytes(b_little, byteorder='little')

print(y)            # 1025
print(z)            # 1025

# Signed examples
neg = (-123).to_bytes(2, byteorder='big', signed=True)
print(neg)          # b'\xff\x85'

print(int.from_bytes(neg, byteorder='big', signed=True))   # -123
print(int.from_bytes(neg, byteorder='big', signed=False))  # 65413 (unsigned interpretation)

Hexadecimal Representation of Binary Data

Hexadecimal text is a human-readable representation of byte values, not binary data itself. Each byte corresponds to two hexadecimal digits (0–9, a–f). Hex text differs from the underlying binary sequence.

bytes and bytearray provide methods for hex conversion:

• hex() returns hexadecimal text representation.
• fromhex() class method reconstructs binary from hex text.

Separators (spaces) are supported in fromhex(). Malformed hex input raises ValueError.

Examples:

b = b'\x01\x02\xAA\xFF'
print(b.hex())      # '0102aaff'

b2 = bytes.fromhex('01 02 aa ff')
print(b2)           # b'\x01\x02\xaa\xff'

ba = bytearray.fromhex('deadbeef')
print(ba)           # bytearray(b'\xde\xad\xbe\xef')

# Malformed input
try:
    bytes.fromhex('01 02 aa f')
except ValueError as e:
    print("Error:", e)

Binary Data and Text Boundaries

Encoding converts a Unicode str into a bytes object according to a specified encoding (e.g., UTF-8). Decoding interprets bytes as Unicode text. Byte values themselves carry no intrinsic universal character meaning; meaning depends on the encoding context.

Example:

text = "café 漢字"
b_utf8 = text.encode('utf-8')
print(list(b_utf8))   # integer bytes of UTF-8 encoding

# Modification example — only safe where meaningful
ba = bytearray(b_utf8)
ba[3] = 0x20         # replace 4th byte with space

# Decoding valid bytes
print(ba.decode('utf-8', errors='ignore'))

Arbitrary binary data should not be decoded assuming text, since many byte sequences do not represent valid or meaningful text under any encoding. Binary protocols, compressed files, images, encryption, and encoded text require different interpretation.

Binary Buffers and Interoperability

The buffer concept allows compatible Python objects and external interfaces to expose binary memory efficiently without copying. This is important for performance and interoperability.

memoryview provides a view into binary data:

• Can be sliced to obtain subviews.
• Shares data with the original object if mutable (bytearray).
• Is distinct from a copy.

Example:

ba = bytearray(b'hello')
mv = memoryview(ba)

print(mv[0])          # 104 (byte for 'h')
mv[0] = 72            # change to 'H'
print(ba)             # bytearray(b'Hello')

# Copying bytes creates a distinct object
b = bytes(ba)
mv2 = memoryview(b)
# mv2[0] = 72  # TypeError: cannot modify read-only memory

Solved Binary Data Exercises in Python

Exercise 1: Parsing a Binary Record

Suppose a binary record consists of:

• A 2-byte header field indicating a message type (big-endian unsigned integer).
• A 4-byte payload length (big-endian unsigned integer).
• A payload of variable length.

Write a function to extract these fields and display the payload as hexadecimal.

def parse_record(record: bytes):
    if len(record) < 6:
        raise ValueError("Record too short")

    # Extract header (2 bytes)
    header_bytes = record[0:2]
    msg_type = int.from_bytes(header_bytes, byteorder='big')

    # Extract payload length (4 bytes)
    length_bytes = record[2:6]
    payload_length = int.from_bytes(length_bytes, byteorder='big')

    # Validate payload length
    if len(record) < 6 + payload_length:
        raise ValueError("Incomplete payload")

    # Extract payload
    payload = record[6:6+payload_length]

    print(f"Message type: {msg_type}")
    print(f"Payload length: {payload_length}")
    print(f"Payload (hex): {payload.hex()}")

# Example usage
record = b'\x00\x01\x00\x00\x00\x05hello'
parse_record(record)

Explanation:

• Bytes 0:2 are sliced to get the header field.
• Bytes 2:6 are sliced for the payload length.
• int.from_bytes(..., 'big') interprets these multi-byte integers.
• Payload is sliced based on the length.
• Hexadecimal representation is used for readable output.
• Length validations ensure input completeness.

Exercise 2: Modifying a Mutable Bytearray Record

Given a bytearray record with fields:

• Byte 0: status (0–255)
• Bytes 1–2: counter (big-endian)
• Bytes 3–: data payload

Write a function to validate and update status and counter, create an immutable snapshot, and demonstrate independence.

def update_record(record: bytearray, new_status: int, new_counter: int):
    if not (0 <= new_status <= 255):
        raise ValueError("Invalid status byte")
    if not (0 <= new_counter <= 0xFFFF):
        raise ValueError("Counter out of range")

    # Modify in place
    record[0] = new_status
    record[1:3] = new_counter.to_bytes(2, 'big')

    # Create immutable snapshot
    snapshot = bytes(record)

    return snapshot

# Example usage
rec = bytearray(b'\x01\x00\x01payload')
print("Before:", rec)

snap = update_record(rec, 2, 513)
print("After:", rec)
print("Snapshot:", snap)

# Modify original again
rec[0] = 3
print("Modified again:", rec)
print("Snapshot unchanged:", snap)

Explanation:

• Validates the input ranges.
• Mutates record in place using item and slice assignment.
• Converts to immutable bytes snapshot.
• Demonstrates that snapshot remains unchanged despite further mutations.