Programming for Data Processing

Lecture 10
Part 1:
String Manipulation

Why strings matter

• Clean & canonicalize messy text
• Extract structure (emails, URLs, phones)
• Inputs for joins, features, and QA checks

Bytes vs text (encodings)

• Files: bytes
• Python: Unicode str
• Specify encoding="utf-8"
• Use errors="replace" at ingestion

s = "Café ☕"
b = s.encode("utf-8")
print(type(s).__name__, type(b).__name__)
print(b.decode("utf-8"))
print(b[:-1].decode("utf-8", errors="replace"))
    

str bytes
Café ☕
Café �
    

Unicode normalization & casefold

• NFKC unifies full-width/compat forms
• casefold() > lower() for matching
• Diacritics: strip only when appropriate

import unicodedata
samples = ["Café", "Cafe\u0301", "ＦＯＯ", "Straße"]
for x in samples:
    n = unicodedata.normalize("NFKC", x)
    print(x, "->", n, "->", n.casefold())
    

Café -> Café -> café
Café -> Café -> café
ＦＯＯ -> FOO -> foo
Straße -> Straße -> strasse
    

Idempotent cleaner

• Order: normalize → casefold → trim → collapse
• Idempotent: re-running changes nothing
• Avoid lossy steps too early

import re, unicodedata
def normalize_text(s: str) -> str:
    s = unicodedata.normalize("NFKC", s).casefold()
    return re.sub(r"\s+", " ", s).strip()
print(normalize_text("  ＦＯＯ   Bar\t"))
    

foo bar
    

Regex essentials

• Named groups, flags, non-greedy .*?
• Pre-compile for reuse
• Prefer libraries for complex URLs/HTML

import re
email_re = re.compile(r"(?P<user>[\\w.+-]+)@(?P<domain>[A-Za-z0-9.-]+\\.[A-Za-z]{2,})")
m = email_re.search("contact: first.last+tag@uni.edu")
print(m.group("user"), m.group("domain"))
    

first.last+tag uni.edu
    

Vectorized Pandas string ops

• .str.extract, .str.contains, .str.replace
• regex=False for literals; set na= explicitly
• Reuse compiled patterns

import pandas as pd, re
df = pd.DataFrame({"email":["alice@acme.io","bob.smith@uni.edu"]})
pat = re.compile(r".+@(?P<domain>[^@]+)$")
print(df["email"].str.extract(pat, expand=True).head())
    

        domain
0     acme.io
1     uni.edu
    

Parsing semi-structured

• JSONL: stream line-by-line
• Logs: small regex with groups
• Use parsers for HTML

import json, io
jsonl = '{"u":"a","r":5}\\n{"u":"b","r":3}'
records = [json.loads(line) for line in io.StringIO(jsonl)]
print(records)
    

[{'u': 'a', 'r': 5}, {'u': 'b', 'r': 3}]
    

PII redaction & ethics

• Redact emails/phones early
• Log changes; keep an audit trail
• Preserve meaning; document assumptions

text = "Email: a@b.com, Phone: +1 202 555 0199"
import re
text = re.sub(r"[\\w.+-]+@[\\w.-]+", "<EMAIL>", text)
text = re.sub(r"(?:\\+?\\d[\\d\\- ]{7,}\\d)", "<PHONE>", text)
print(text)
    

Email: <EMAIL>, Phone: <PHONE>