Binary Sequence Processing in Python

Binary sequence processing in Python involves the inspection, searching, transformation, decomposition, combination, ASCII-oriented interpretation, formatting, and hexadecimal representation of byte-oriented sequences such as bytes and bytearray. These operations manipulate sequences of byte values (integers from 0 to 255) directly, focusing on the binary content itself rather than Unicode character semantics. This approach emphasizes handling raw byte data for tasks like protocol parsing, binary file manipulation, and low-level data processing where the meaning of each byte is central.

Foundations of Binary Sequence Processing in Python

Binary sequence processing is computation performed over ordered collections of byte values, where each element is an integer between 0 and 255. The binary content is treated independently of any higher-level interpretation such as text encoding, integer decoding, protocol field extraction, compressed data, or serialized objects. The processing focuses on the raw byte values as the fundamental unit.

Python provides two closely related built-in binary sequence types: bytes and bytearray. Both types share sequence-oriented behavior, supporting indexing, slicing, iteration, membership tests, comparison, searching, and many transformation methods. The key distinction is that bytes objects are immutable, meaning their contents cannot be changed after creation, while bytearray objects are mutable and support in-place modification.

Indexing a binary sequence yields an integer representing the byte value at that position. In contrast, slicing a binary sequence produces another binary sequence of the same type (or a related type), representing a contiguous subsequence of bytes. This distinction enables algorithms to either process individual bytes as integers or work with byte sub-blocks as new sequences.

Examples using both bytes and bytearray demonstrate their shared binary processing model:

b = b"example"
ba = bytearray(b)

print(len(b))          # 7
print(b[0])            # 101 (ASCII 'e')
print(b[1:4])          # b"xam"
print(list(b))         # [101, 120, 97, 109, 112, 108, 101]
print(120 in b)        # True (integer membership)
print(b == ba)         # True (value equality)

Python Binary Sequence Operations

Basic operations on binary sequences include:

• Length: len(b) returns the number of bytes.
• Indexing: b[i] returns the byte at position i as an integer (0–255).
• Negative Indexing: b[-1] accesses bytes counting from the end.
• Slicing: b[i:j] returns a binary subsequence of bytes from index i up to but not including j.
• Stepped Slicing: b[i:j:k] returns bytes from i to j stepping by k.
• Iteration: for byte in b: iterates over byte integers.
• Containment (Membership): x in b tests if an integer byte value or binary subsequence appears in b.
• Concatenation: b1 + b2 produces a new binary sequence combining the two.
• Repetition: b * n repeats the sequence n times.
• Equality: b1 == b2 compares content bytewise.
• Lexicographic Comparison: Binary sequences are compared byte by byte in ascending index order.

Membership testing distinguishes two cases:

• Testing for an integer byte value in the range 0 to 255: x in b where x is an integer.
• Testing for a binary subsequence: subseq in b where subseq is a bytes or bytearray object.

Lexicographic ordering compares each byte integer in sequence until a difference is found or one sequence ends. This ordering is bytewise and does not interpret the entire sequence as a number or as human-readable text.

Example operations:

b = b"python"
ba = bytearray(b)

print(b[0])           # 112 ('p')
print(b[-1])          # 110 ('n')
print(b[1:4])         # b"yth"
print(b[::2])         # b"pto"

print(111 in b)       # True (integer membership, 'o')
print(b"th" in b)     # True (binary subsequence membership)
print(b"xy" in b)     # False

print(b + b"!")       # b"python!"
print(b * 2)          # b"pythonpython"

print(b == ba)        # True
print(b < b"pytho")   # False (lexicographic comparison)

The immutability of bytes means that any operation that modifies content returns a new bytes object. In contrast, bytearray supports in-place mutation via item assignment and other methods, allowing modification without creating a new object.

Example demonstrating immutability and mutability:

b = b"data"
ba = bytearray(b)

# bytes are immutable; this creates a new object
b2 = b.replace(b"a", b"A")
print(b2)             # b'dAtA'
print(b)              # b'data'

# bytearray can be mutated in place
ba[1] = ord('A')
print(ba)             # bytearray(b'dAta')

Binary Sequence Searching in Python

Searching within binary sequences involves locating occurrences of integer bytes or binary subsequences. Common methods include:

• Membership test: subseq in b returns True if subseq occurs anywhere in b.
• find(subseq[, start[, end]]): returns the lowest index of subseq or -1 if not found.
• rfind(subseq[, start[, end]]): returns the highest index or -1 if not found.
• index(subseq[, start[, end]]): like find but raises ValueError if no match.
• rindex(subseq[, start[, end]]): like rfind but raises ValueError if no match.
• count(subseq[, start[, end]]): counts non-overlapping occurrences.

Prefix and suffix tests:

• startswith(prefix[, start[, end]]) returns whether the sequence starts with the given prefix or any of the prefixes if a tuple is provided.
• endswith(suffix[, start[, end]]) similarly tests for suffixes.

All searches can specify optional start and end indices to constrain where in the sequence the search occurs. The returned offsets are relative to the binary sequence indexing and do not inherently correspond to external data offsets like file positions or protocol offsets.

Example search usage:

data = b"abcabcabc"
print(b"abc" in data)             # True
print(data.find(b"abc"))          # 0
print(data.rfind(b"abc"))         # 6
print(data.index(b"abc", 1))      # 3
print(data.count(b"abc"))         # 3

print(data.startswith(b"abc"))   # True
print(data.endswith(b"abc"))     # True

print(data.find(b"abc", 3, 7))   # 3 (bounded search)
print(data.find(b"xyz"))          # -1 (not found)

Repeated searching should consider overlapping patterns carefully. For example, counting occurrences via count or repeatedly using find with updated start indices may miss matches that overlap unless the search advances by one byte instead of by the pattern length.

Example to find all occurrences of a marker, including overlapping:

marker = b"aba"
text = b"ababa"
offset = 0
while True:
    pos = text.find(marker, offset)
    if pos == -1:
        break
    print(f"Found at {pos}")
    offset = pos + 1  # advance by 1 to find overlapping matches

Output:

Found at 0
Found at 2

Binary Sequence Transformation in Python

Binary transformation creates new byte sequences or mutates existing ones through operations like replacement, translation, deletion, trimming, padding, and ASCII case modifications.

The replace method substitutes occurrences of a binary subsequence with another. An optional count limits replacements. Since bytes are immutable, replace always returns a new bytes object. For bytearray, replace also returns a new object; in-place mutation must use other methods.

The translate method remaps bytes using a translation table created by bytes.maketrans(from_bytes, to_bytes, delete_bytes). It performs one-to-one byte remapping and can delete specified bytes. Translation differs from variable-length replacements like replace because it operates on individual bytes.

Examples:

b = b"hello world"
# Replace "world" with "there"
print(b.replace(b"world", b"there"))  # b'hello there'

# Create translation table: h->H, delete ' '
trans_table = bytes.maketrans(b"h", b"H", b" ")
print(b.translate(trans_table))       # b'Helloworld'

Trimming and padding methods include:

• strip([bytes]), lstrip([bytes]), rstrip([bytes]): remove specified bytes from ends.
• center(width[, fillbyte]), ljust(width[, fillbyte]), rjust(width[, fillbyte]): pad sequences to a desired width.
• zfill(width): pad with ASCII zero bytes on the left.

These operate on byte values, not Unicode whitespace or characters.

Examples:

b = b"  spam  "
print(b.strip(b" "))        # b'spam'
print(b.lstrip(b" sp"))    # b'am  '
print(b.rstrip(b" am"))    # b"  spa"

print(b.center(12, b'*'))  # b'**  spam  **'
print(b.ljust(10, b'-'))   # b'  spam  --'
print(b.zfill(10))         # b'000  spam  '

bytearray supports direct in-place modification through item assignment, slice assignment, deletion, appending, extending, inserting, and reversing:

ba = bytearray(b"example")
ba[0] = ord('E')
ba[1:3] = b"XY"
del ba[4]
ba.append(ord('Z'))
ba.extend(b"12")
ba.insert(3, ord('Q'))
ba.reverse()
print(ba)

This produces an in-place mutated binary sequence, contrasting with bytes where similar transformations require creation of new objects.

Binary Sequence Splitting and Joining in Python

Splitting binary sequences divides them into lists of subsequences based on separators:

• split(sep=None, maxsplit=-1): splits on a specified byte separator or ASCII whitespace by default.
• rsplit(sep=None, maxsplit=-1): splits from the right.
• partition(sep): splits into exactly three parts (before, separator, after), preserving the separator.
• rpartition(sep): same as partition but from the right.
• splitlines(keepends=False): splits by recognized binary line boundaries (\n, \r, \r\n).

Joining concatenates an iterable of compatible binary sequences using a binary separator:

• sep.join(iterable)

Joining requires all elements to be binary sequences of compatible type (bytes or bytearray), not integers or Unicode strings.

Examples:

b = b"one,two,,three,"
print(b.split(b","))        # [b'one', b'two', b'', b'three', b'']
print(b.rsplit(b",", 2))    # [b'one,two,,three', b'', b'']

print(b.partition(b","))    # (b'one', b',', b'two,,three,')
print(b.rpartition(b","))   # (b'one,two,,three', b',', b'')

lines = b"line1\nline2\r\nline3\rline4"
print(lines.splitlines())   # [b'line1', b'line2', b'line3', b'line4']
print(lines.splitlines(True))  # includes line endings

fields = [b"field1", b"field2", b"field3"]
joined = b"|".join(fields)
print(joined)               # b'field1|field2|field3'

Example: split a delimiter-separated record, validate field count, transform a field, and rejoin:

record = b"ID|Name|Age"
parts = record.split(b"|")
if len(parts) == 3:
    parts[1] = parts[1].upper()
    normalized = b"|".join(parts)
    print(normalized)  # b'ID|NAME|Age'

ASCII-Oriented Binary Processing in Python

ASCII-oriented binary processing interprets byte values according to ASCII character classes or case mappings without converting to Unicode text. This is useful for protocols or formats restricted to ASCII bytes.

Classification methods include:

• isalnum(): all bytes are ASCII letters or digits
• isalpha(): all bytes are ASCII letters
• isdigit(): all bytes are ASCII digits
• isspace(): all bytes are ASCII whitespace (\t, \n, \r, \f, \v, space)
• islower(), isupper(): ASCII letter case
• istitle(): ASCII titlecase pattern

These methods return False if the sequence is empty or contains bytes outside the ASCII range or the tested class.

Examples:

print(b"abc123".isalnum())       # True
print(b"abc!".isalnum())          # False ('!' is punctuation)
print(b"abc".isalpha())            # True
print(b"123".isdigit())            # True
print(b" \t\n".isspace())          # True
print(b"abc".islower())            # True
print(b"ABC".isupper())            # True
print(b"Abc".istitle())            # True
print(b"".isalpha())               # False (empty)
print(b"\x80".isalpha())           # False (non-ASCII byte)

ASCII case transformations:

• lower(), upper(): map ASCII letter bytes to lower/upper case
• capitalize(): first ASCII letter uppercase, rest lowercase
• title(): ASCII titlecase pattern
• swapcase(): swap ASCII letter case

Non-ASCII bytes remain unchanged.

Example:

b = b"Hello World! \x80"
print(b.lower())       # b'hello world! \x80'
print(b.upper())       # b'HELLO WORLD! \x80'
print(b.capitalize())  # b'Hello world! \x80'
print(b.swapcase())    # b'hELLO wORLD! \x80'

ASCII-oriented processing is appropriate for binary protocols, identifiers, delimiters, or restricted ASCII fields. When data represents general human language or requires full Unicode semantics, explicit decoding to Unicode strings is necessary.

Binary Printf-Style Formatting in Python

The % operator on binary sequences formats byte sequences using a binary format template and compatible values. This is distinct from Unicode string interpolation and produces bytes rather than text.

Basic formatting conversions include:

• %b: byte sequence (bytes or bytearray)
• %d, %i, %u: signed/unsigned decimal integers
• %x, %X: hexadecimal integers
• %f, %F, %e, %E, %g, %G: floating-point values
• %r: repr() of the object encoded as bytes
• Width, precision, flags like -, +, 0, space allowed

Mapping keys can be used for dictionary-based substitution with % operator.

Examples:

fmt = b"Name: %b, Age: %d, Score: %.2f%%"
result = fmt % (b"Alice", 30, 95.678)
print(result)  # b'Name: Alice, Age: 30, Score: 95.68%'

# Mapping example
fmt2 = b"%(name)s scored %(score).1f points"
data = {b"name": b"Bob", b"score": 87.5}
print(fmt2 % data)  # b'Bob scored 87.5 points'

Formatting fails if incompatible types are provided or if Unicode strings are used instead of bytes where bytes are expected.

Invalid attempts:

b"Value: %b" % "text"       # TypeError: not bytes
b"Value: %d" % "123"        # TypeError: invalid type for %d
b"%(key)s" % {b"key": "x"}  # TypeError: value must be bytes for %s

Binary printf-style formatting is practical for producing formatted byte-oriented output such as protocol messages or binary logs. It differs from general serialization or Unicode text encoding.

Hexadecimal Binary Representation in Python

Hexadecimal representation encodes each byte as two hexadecimal digits (0–9, a–f), producing a textual string that describes the binary data but is not itself the original binary sequence.

Conversion from binary to hex text is done with hex():

• b.hex() returns a string of lowercase hexadecimal digits representing each byte.
• Some methods support optional separators or grouping for readability (not in all Python versions).

Reconstruction is done with:

• bytes.fromhex(s)
• bytearray.fromhex(s)

These methods accept whitespace in the input string but fail on malformed hex strings or incomplete byte pairs.

Examples:

b = b"\x01\x02\xab\xcd"
hex_str = b.hex()
print(hex_str)  # '0102abcd'

grouped = ":".join(hex_str[i:i+2] for i in range(0, len(hex_str), 2))
print(grouped)  # '01:02:ab:cd'

reconstructed = bytes.fromhex(hex_str)
print(reconstructed == b)  # True

Malformed hex examples:

try:
    bytes.fromhex("01 02 ab cd zz")
except ValueError as e:
    print(e)  # non-hexadecimal number found in fromhex() arg at position 11

try:
    bytes.fromhex("0102a")
except ValueError as e:
    print(e)  # non-hexadecimal number found in fromhex() arg at position 5

# Corrected input
correct = bytes.fromhex("01 02 ab cd")
print(correct)

Hexadecimal representation is widely used for debugging, diagnostics, test fixtures, protocol inspection, binary identifiers, and documentation. It remains a textual display format distinct from the semantic parsing of the binary data it describes.

Solved Binary Sequence Processing Exercises in Python

Exercise 1: Marker Search and Field Normalization

Write a function that receives a binary message containing a fixed marker and delimiter-separated fields. It searches for the marker, validates its position, splits the remaining data, transforms one ASCII-oriented field by uppercasing it, and rejoins the fields into a normalized binary message.

def normalize_message(msg, marker=b"###", delimiter=b"|"):
    pos = msg.find(marker)
    if pos == -1:
        raise ValueError("Marker not found")
    if pos != 0:
        raise ValueError("Marker not at start")
    payload = msg[len(marker):]
    fields = payload.split(delimiter)
    if len(fields) != 3:
        raise ValueError("Unexpected field count")
    # Uppercase second field (ASCII-oriented)
    fields[1] = fields[1].upper()
    normalized = marker + delimiter.join(fields)
    return normalized

# Example usage
msg = b"###id|name|age"
print(normalize_message(msg))  # b'###id|NAME|age'

Step by step:

• Search for marker and confirm it is at the start.
• Extract payload after marker.
• Split by delimiter and check field count.
• Apply ASCII uppercase to the second field.
• Rejoin fields with delimiter and prepend marker.

If marker or fields are missing, appropriate exceptions are raised.

Exercise 2: Pattern Replacement and Hexadecimal Round Trip

Process arbitrary binary data by searching for a byte pattern, replacing selected occurrences, producing grouped hexadecimal output, reconstructing the bytes, and verifying round-trip equality.

def replace_and_hex(data, pattern=b"\x00", replacement=b"\xff", group_size=4):
    replaced = data.replace(pattern, replacement)
    hex_str = replaced.hex()
    grouped_hex = " ".join(hex_str[i:i+group_size*2] for i in range(0, len(hex_str), group_size*2))
    reconstructed = bytes.fromhex(hex_str)
    assert reconstructed == replaced, "Round-trip failed"
    return replaced, grouped_hex, reconstructed

# Example
data = b"\x00\x01\x02\x00\x03\x00\x04\x05"
replaced, grouped_hex, reconstructed = replace_and_hex(data)
print(replaced)       # b'\xff\x01\x02\xff\x03\xff\x04\x05'
print(grouped_hex)    # e.g. 'ff0102ff 03ff0405'
print(reconstructed)  # b'\xff\x01\x02\xff\x03\xff\x04\x05'

Explanation:

• Replace all b"\x00" bytes with b"\xff".
• Convert replaced bytes to hex string.
• Group hex output for readability.
• Reconstruct bytes from hex.
• Check that reconstruction equals the replaced bytes (round-trip).

Malformed hex input would raise on bytes.fromhex.

Exercise 3: Binary Report Construction with Binary Formatting

Build a compact binary report using printf-style bytes formatting from validated numeric and byte inputs, then inspect parts of the result with searching or splitting.

def build_report(name, version, score):
    fmt = b"Report: Name=%b, Version=%d, Score=%.1f\n"
    report = fmt % (name, version, score)
    return report

report = build_report(b"Test", 3, 99.5)
print(report)  # b'Report: Name=Test, Version=3, Score=99.5\n'

# Inspect fields
fields = report.split(b", ")
print(fields)
# [b'Report: Name=Test', b'Version=3', b'Score=99.5\n']

# Search for score field
start = report.find(b"Score=")
print(report[start:])  # b'Score=99.5\n'

Explanation:

• Input values are bytes for name, int for version, float for score.
• Format string uses %b for binary name, %d for int, %.1f for one decimal float.
• Output is a bytes object formatted with binary content.
• Splitting and searching operate directly on the binary report without decoding.

This formatting is distinct from serialization; it produces formatted byte sequences suitable for byte-oriented protocols or logs.