Text Data in Python

Text data in Python is primarily represented as Unicode text through str objects. These str objects allow manipulation of textual characters using operations such as indexing, searching, classification, transformation, splitting, joining, formatting, encoding, and decoding. It is essential to distinguish textual characters, which are abstract Unicode values, from their encoded byte representations stored in bytes objects.

Foundations of Text Data in Python

Python represents textual data mainly using str objects, which hold Unicode text. This text consists of abstract characters known as Unicode code points. In contrast, bytes objects represent sequences of raw byte values, typically used for encoded text or binary data.

Text must be encoded to become bytes under a particular character encoding (e.g., UTF-8), and bytes must be decoded using the same or compatible encoding to produce valid Unicode text.

Strings are immutable sequences that support operations such as:

• Determining length (len)
• Indexing and slicing to access substrings
• Iteration over characters
• Membership testing using in
• Comparisons (==, <, etc.)
• Concatenation (+) and repetition (*)
• Numerous methods for classification, transformation, searching, splitting, joining, formatting, encoding, and decoding

Despite sequence-like behavior, strings represent ordered text rather than arbitrary collections of unrelated elements.

Examples demonstrating basic string operations:

s = "Python"
print(type(s))            # <class 'str'>
print(len(s))             # 6
print(s[0])               # 'P'
print(s[-1])              # 'n'
print(s[1:4])             # 'yth'

for ch in s:
    print(ch, end=' ')    # P y t h o n

print('y' in s)           # True
print(s + "3.10")         # 'Python3.10'
print(s * 2)              # 'PythonPython'

print(str(123))           # '123'
print(str(3.14))          # '3.14'

Python String Type

Python string literals can be written using single quotes '...', double quotes "...", or triple-quoted forms '''...''' or """...""" for multiline text. Escape sequences like \n (newline), \\ (backslash), and \' or \" allow insertion of special characters. Raw-string notation with an r or R prefix disables escape sequence processing, useful for regular expressions or Windows paths.

Adjacent string literals automatically concatenate at compile time:

text = "Hello, " "world!"
print(text)  # Hello, world!

String indexing and slicing support positive indices starting at 0 and negative indices counting backward from -1. The slice syntax [start:stop:step] allows extraction of substrings, with omitted bounds defaulting to start or end of the string. Empty slices return the empty string "". Indexing returns a one-character string; Python does not have a separate character type.

Strings are immutable, so operations like concatenation, replacement, and case conversion return new strings without modifying the original.

Comparison operators (==, <, >, etc.) and membership tests (in) operate on Unicode code points, not locale-aware or natural-language ordering by default.

Example demonstrating various string features:

s1 = 'single quoted'
s2 = "double quoted"
s3 = '''triple
quoted'''
s4 = r'raw\nstring'

print(s1[0])        # 's'
print(s2[-1])       # 'd'
print(s1[2:7])      # 'ngle '
print(s1 + " text") # 'single quoted text'
print(s1 * 2)       # 'single quotedsingle quoted'
print('q' in s1)    # True

# Attempt to change a character (raises TypeError)
try:
    s1[0] = 'S'
except TypeError as e:
    print(e)  # 'str' object does not support item assignment

Unicode Text in Python

Python strings represent Unicode text as sequences of code points, abstract integer values for characters. The ord function converts a single-character string to its Unicode code point integer, and chr converts an integer code point back to a one-character string.

String length and indexing operate on these code points, not on user-perceived grapheme clusters, which may consist of multiple combined code points (e.g., letters with accents).

Unicode normalization provides a way to convert canonically equivalent but differently encoded texts into a consistent form. This is important for comparison, searching, identifiers, and interoperability when semantically identical text must be treated equivalently.

Example demonstrating Unicode and normalization:

import unicodedata

s = "café"                  # 'é' as single code point
s_combined = "cafe\u0301"   # 'e' + combining acute accent

print(s == s_combined)      # False, different code points

print(len(s))               # 4
print(len(s_combined))      # 5

print([ord(c) for c in s])           # [99, 97, 102, 233]
print([ord(c) for c in s_combined])  # [99, 97, 102, 101, 769]

nfc = unicodedata.normalize('NFC', s_combined)
print(nfc == s)             # True, normalized form equals original

String Searching in Python

Substring searching can be performed using:

• Membership operators: 'sub' in text and 'sub' not in text return True or False.
• find(sub): returns the lowest index of substring sub or -1 if not found.
• rfind(sub): returns the highest index of substring sub or -1 if not found.
• index(sub): like find but raises ValueError if not found.
• rindex(sub): like rfind but raises ValueError if not found.
• count(sub): returns the number of non-overlapping occurrences of sub.

Prefix and suffix tests use:

• startswith(prefix) and endswith(suffix).
• Both accept either a single string or a tuple of strings.
• Optional start and end index parameters specify substring bounds.

Example usage:

text = "Hello, world!"

print("world" in text)           # True
print(text.find("lo"))           # 3
print(text.rfind("l"))           # 10
print(text.count("l"))           # 3

print(text.startswith("Hello"))  # True
print(text.endswith("!"))         # True

print(text.startswith(("Hi", "Hello")))  # True

# Case-insensitive search using lowercasing
search = "WORLD"
print(search.lower() in text.lower())   # True

# Using find with case-insensitive search
pos = text.lower().find(search.lower())
print(pos)  # 7 (index of 'world')

String Classification in Python

Python provides Unicode-aware methods to classify string content:

• isalpha(): all characters are alphabetic letters.
• isalnum(): all characters are alphanumeric (letters or digits).
• isdecimal(): all characters are Unicode decimal digits (0–9).
• isdigit(): all characters are digits (includes decimal digits and some others).
• isnumeric(): all characters are numeric (includes digits, fractions, subscripts, Roman numerals, etc.).
• isspace(): all characters are whitespace.
• islower(): all cased characters are lowercase and at least one cased character present.
• isupper(): all cased characters are uppercase and at least one cased character present.
• istitle(): follows titlecase rules (words start with uppercase letters).
• isidentifier(): valid Python identifier syntax.
• isprintable(): all characters are printable or the string is empty.
• isascii(): all characters have code points < 128 (ASCII).

Examples:

print("abc".isalpha())       # True
print("abc123".isalnum())    # True
print("123".isdecimal())     # True
print("²".isdigit())         # True (superscript 2)
print("Ⅷ".isnumeric())       # True (Roman numeral 8)
print(" \t\n".isspace())     # True
print("abc".islower())       # True
print("ABC".isupper())       # True
print("Hello World".istitle()) # True
print("_foo123".isidentifier()) # True
print("abc\n".isprintable()) # False (contains newline)
print("ascii".isascii())     # True
print("á".isascii())         # False
print("".isalpha())          # False (empty string)

String Transformation in Python

Case transformations include:

• lower(): all characters to lowercase.
• upper(): all characters to uppercase.
• casefold(): aggressive lowercasing for caseless matching.
• capitalize(): first character uppercase, rest lowercase.
• title(): uppercase first letter of each word.
• swapcase(): swap case of all characters.

Trimming and replacement:

• strip(): removes leading and trailing whitespace characters.
• lstrip(): removes leading whitespace.
• rstrip(): removes trailing whitespace.
• removeprefix(prefix): removes exact prefix if present.
• removesuffix(suffix): removes exact suffix if present.
• replace(old, new, count=-1): replaces occurrences of substring.

Character translation:

• str.maketrans(mapping): creates translation table.
• translate(table): applies translation, replacing or deleting characters.

Example:

s = "  Hello World  "
print(s.lower())                   # '  hello world  '
print(s.casefold())                # '  hello world  '
print(s.strip())                   # 'Hello World'
print(s.lstrip())                  # 'Hello World  '
print(s.rstrip())                  # '  Hello World'

s2 = "unittest.py"
print(s2.removeprefix("unit"))     # 'test.py'
print(s2.removesuffix(".py"))      # 'unittest'

s3 = "banana"
print(s3.replace("a", "o"))        # 'bonono'

table = str.maketrans("aeiou", "12345", "n")  # map vowels, delete 'n'
print("banana".translate(table))  # 'b1 1 2' (spaces where 'n' deleted)

String Splitting and Joining in Python

Splitting methods:

• split(sep=None, maxsplit=-1): splits on separator or whitespace by default; consecutive separators treated as one if splitting on whitespace.
• rsplit(sep=None, maxsplit=-1): like split but from the right.
• splitlines(keepends=False): splits on line boundaries, optionally keeping end-of-line characters.
• partition(sep): splits into tuple (before, sep, after) at first occurrence.
• rpartition(sep): like partition but from last occurrence.

Joining:

• str.join(iterable): concatenates strings in iterable, separated by the string on which it is called.

Examples:

s = "a,b,c,d"
print(s.split(","))          # ['a', 'b', 'c', 'd']
print(s.rsplit(",", 2))      # ['a,b', 'c', 'd']

lines = "Line1\nLine2\r\nLine3"
print(lines.splitlines())    # ['Line1', 'Line2', 'Line3']
print(lines.splitlines(True)) # ['Line1\n', 'Line2\r\n', 'Line3']

p = "key=value=other"
print(p.partition("="))      # ('key', '=', 'value=other')
print(p.rpartition("="))     # ('key=value', '=', 'other')

fields = ["apple", "banana", "cherry"]
print(", ".join(fields))     # 'apple, banana, cherry'

# Pipeline example: split, strip, join
data = "  apple , banana , cherry  "
cleaned = ", ".join(part.strip() for part in data.split(","))
print(cleaned)               # 'apple, banana, cherry'

String Formatting in Python

String formatting converts values into textual representations arranged according to a format specification.

Python supports several formatting interfaces:

• F-strings: string literals prefixed with f or F, embedding expressions inside {}.
• str.format(): method with replacement fields inside {}.
• Built-in format(value, spec): returns formatted string of value.
• Legacy % operator formatting: C-style formatting with % placeholders.

Formatting controls textual presentation such as alignment, padding, numeric precision, and more, distinct from simple concatenation.

Examples:

name = "Alice"
age = 30
pi = 3.14159

# f-string
print(f"Name: {name}, Age: {age}, Pi: {pi:.2f}")

# str.format
print("Name: {}, Age: {}, Pi: {:.2f}".format(name, age, pi))

# format function
print(format(pi, ".3f"))

# Percent formatting
print("Name: %s, Age: %d" % (name, age))

# Tabular alignment
print(f"{'Name':<10} {'Age':>3}")
print(f"{name:<10} {age:>3}")

Formatted output controls display without altering underlying data values. Representations for users differ from diagnostic or debug representations.

Replacement Field Formatting in Python

Replacement fields inside formatted strings or str.format calls contain:

• Field expressions or names to select values.
• Optional conversion flags (!s, !r, !a) to apply str(), repr(), or ascii().
• Optional format specifications following : that control output formatting.
• Literal text surrounding replacement fields.

F-strings evaluate arbitrary Python expressions inside the field, while str.format supports attribute and item access but not arbitrary expressions.

Positional and named fields:

# f-string with expression and conversion
value = 3.14159
print(f"Pi rounded: {value:.2f}")
print(f"Repr: {value!r}")

# str.format with positional and named fields
print("{0} {1}".format("hello", "world"))
print("{greeting}, {name}!".format(greeting="Hello", name="Alice"))

# Attribute and item access
class Person:
    def __init__(self, name):
        self.name = name

p = Person("Bob")
print(f"Name: {p.name}")
print("{0[name]}".format({"name": "Carol"}))

Escaped braces {{ and }} produce literal { and } characters.

Readable replacement fields avoid complex or repeated expressions, side effects, or mixing textual interpolation with executable code.

Python Format Specification Mini-Language

Format specifications control presentation details such as:

• Fill character and alignment (<, >, ^ for left, right, center).
• Sign handling for numeric values (+, -, space).
• Width as minimum field width.
• Digit grouping options (,, _).
• Precision for floating-point numbers.
• Presentation type (s for string, d for decimal integer, f for fixed-point float, % for percentages, e for scientific notation, x for hex).

Examples:

print(f"{'apple':>10}")      # right aligned in width 10
print(f"{42:+06d}")          # sign with zero-padding width 6: +00042
print(f"{1234567:,}")        # digit grouping with commas: 1,234,567
print(f"{3.14159:.2f}")      # 2 decimal places: 3.14
print(f"{0.25:.0%}")         # percentage: 25%
print(f"{255:#x}")           # hex with 0x prefix: 0xff

Printf-Style String Formatting in Python

The % operator on strings supports C-style formatting with conversion types:

• %s: string (calls str()).
• %r: string (calls repr()).
• %d, %i: decimal integer.
• %f: floating-point number.
• %e, %E: scientific notation.
• %x, %X: hexadecimal integer.
• %o: octal integer.
• Width, precision, and flags (such as 0 for zero-padding, - for left-align) can be specified.

Values are passed as a tuple or mapping for named substitutions.

Examples:

print("Name: %s, Age: %d" % ("Alice", 30))
print("Pi approx: %.2f" % 3.14159)
print("Hex: %#x" % 255)      # '0xff'
print("Escaped %% sign")     # 'Escaped % sign'

mapping = {"name": "Bob", "age": 25}
print("Name: %(name)s, Age: %(age)d" % mapping)

Printf-style formatting is less flexible and readable than modern replacement-field formatting but remains valid and present in legacy code.

Text Encoding and Decoding in Python

Encoding converts Unicode text (str) into a bytes sequence under a named character encoding (e.g., UTF-8), and decoding interprets bytes back into Unicode text.

Common encodings:

• UTF-8: variable-length, byte-oriented, backward compatible with ASCII, widely used.
• UTF-16: 2 or 4 byte units, requires byte order considerations.
• ASCII: 7-bit encoding for basic English characters only.

Encoder and decoder must agree on the encoding; otherwise, data corruption or errors occur.

Encoding and decoding can raise errors or handle them via strategies:

• 'strict': raise an exception.
• 'replace': substitute invalid bytes or characters.
• 'ignore': skip invalid data.
• 'backslashreplace': use escape sequences.

Example:

text = "café 普通话"

# Encode to UTF-8 bytes
b = text.encode("utf-8")
print(b)                      # b'caf\xc3\xa9 \xe6\x99\xae\xe9\x80\x9a\xe8\xaf\x9d'

# Decode back to string
text2 = b.decode("utf-8")
print(text2 == text)          # True

# Attempt ASCII encoding (fails)
try:
    text.encode("ascii")
except UnicodeEncodeError as e:
    print(e)

# Decode invalid bytes with error handling
bad_bytes = b'\xff\xfe\xfa'
print(bad_bytes.decode("utf-8", errors="replace"))

Incorrect decoding leads to mojibake — garbled text resulting from wrong assumptions about encoding. Truncated byte sequences cause decoding failures. Always preserve or explicitly specify encoding when handling external text data.

Solved Text Data Exercises in Python

Below is a complete Python exercise demonstrating Unicode-aware normalization, cleaning, searching, classification, splitting, joining, and formatting.

import unicodedata

records = [
    "  Café, 42, YES  ",
    "naïve, 37, no",
    "résumé, 29, YES",
    "  jalapeño, 21, no "
]

def normalize_and_clean(text):
    # Normalize to NFC
    text = unicodedata.normalize("NFC", text)
    # Strip whitespace and convert to lowercase
    return text.strip().lower()

def classify_response(resp):
    # Classify response as boolean
    return resp in ("yes", "y", "true")

report_lines = []
for rec in records:
    # Normalize and clean
    norm = normalize_and_clean(rec)
    # Split fields by comma
    parts = [p.strip() for p in norm.split(",")]
    if len(parts) != 3:
        continue
    name, age_str, response = parts
    # Case-insensitive search for accented 'é'
    has_accent = "é" in name
    # Classify age field
    if age_str.isdecimal():
        age = int(age_str)
    else:
        age = None
    accepted = classify_response(response)

    # Format report line
    line = f"{name.title():<12} | Age: {age or 'N/A':>3} | Accepted: {accepted} | Accented é: {has_accent}"
    report_lines.append(line)

print("\n".join(report_lines))

Step-by-step explanation:

• Unicode normalization ensures canonical text form.
• Stripping whitespace and lowercasing prepares for consistent searching and classification.
• Splitting extracts structured fields.
• Case-insensitive checks identify special characters.
• Numeric classification distinguishes valid ages.
• Boolean classification recognizes affirmative responses.
• Title casing and formatting produce readable output.
• Edge cases such as missing or malformed data are handled gracefully.

A second exercise demonstrates formatting multilingual text, encoding, decoding, and round-trip validation.

text = "Привет, мир! こんにちは世界 🌍"

formatted = f"Message: {text:^40}"

# Encode as UTF-8
encoded = formatted.encode("utf-8")
print(encoded)  # byte representation

# Decode back
decoded = encoded.decode("utf-8")
print(decoded == formatted)  # True

# Attempt decoding with incompatible encoding (ASCII)
try:
    bad_decoded = encoded.decode("ascii")
except UnicodeDecodeError as e:
    print("Decoding error:", e)

Explanation:

• Text includes Cyrillic, Japanese, and emoji characters.
• Formatting centers the message in a 40-character field.
• Encoding to UTF-8 transforms text into bytes.
• Decoding under matching UTF-8 recovers the original string.
• Decoding bytes with ASCII fails due to invalid byte sequences.
• Bytes must not be treated as ordinary text without decoding.

When selecting Python text operations, choose methods that preserve Unicode semantics without unnecessary conversions between text and bytes. Use formatting mechanisms appropriate to context: f-strings for inline expressions, str.format for dynamic templates, or the format function for isolated formatting.

Always use explicit encoding and decoding boundaries when text crosses external interfaces such as files, networks, or APIs. Avoid conflating text with bytes, and prefer Unicode-aware methods to maintain correctness and interoperability.