Programming for Data Processing

Introduction

Why This Course?

Your Digital Footprint

• In the last 24 hours, you generated:
  • Dozens of social media interactions
  • Hundreds of GPS/location updates
  • Streaming/video activity logs
• Multiply by billions of people → massive global data

From Data to Decisions

✅ Processing turns raw data into usable information.

Zoom In: Data Processing

• Main tasks of data processing:
  • Cleaning → fix errors, handle missing values
  • Transforming → reformat, normalize, standardize
  • Enriching → add metadata, join data
  • Aggregating → group, summarize, compute statistics
• Good processing makes data reliable for analysis.

Example: Streaming Services

Raw data (messy logs)

User123, 2025-09-01 09:14:32, SongID_981, action=skip
User123, 2025-09-01 09:15:05, SongID_472, action=play
User456, 2025-09-01 09:16:20, SongID_981, action=play
User789, 2025-09-01 09:17:45, SongID_981, action=play
  

Example: Streaming Services

Raw data (messy logs)

User123, 2025-09-01 09:14:32, SongID_981, action=skip
User123, 2025-09-01 09:15:05, SongID_472, action=play
User456, 2025-09-01 09:16:20, SongID_981, action=play
User789, 2025-09-01 09:17:45, SongID_981, action=play
  

Usable data (features)

• User123 prefers SongID_472
• SongID_981 is popular → “trending”
• Result: recommend similar songs + trending chart

Why Python?

• Simple syntax → easy to read & learn
• Popular → huge community & support
• Powerful libraries → analysis, visualization, ML
• Versatile → from quick scripts to large applications
• Career skill → widely used in industry & research

Course Overview

Weekly Breakdown

Resources

• All course materials will be available on the LMS
• Textbook: Python for Data Analysis (Wes McKinney)
  • Available at: wesmckinney.com/book
• Supplementary: Official Python Tutorial
  • Available at: docs.python.org/3/tutorial/

Textbook

Python Tutorial

Python Kickstart

Example Python Program

print("Hello, world!")
x = 5
y = 2 * x
print("y = ", y)

• What does this program do?
  1. Print Hello, world! to the console
  2. Bind the value 5 to the variable x
  3. Compute 2 * x and bind the result to y
  4. Print y =  followed by the value of y

PYTHON PROGRAMS

• A program = definitions + commands
• Definitions → create values & variable names

  x = 5  # definition

• Commands → tell Python what to do

  print(x)  # command

• Run code in a shell or a file (script/Notebook)

Objects

• Programs manipulate data objects
• Objects have a type that defines its actions
  • A human can walk, speak, etc.
  • A dog can bark, fetch, etc.
• Two categories of objects:
  • Scalar – cannot be subdivided
  • Non-scalar – have internal structure that is accessible
    • A string can be concatenated, split
    • A list can be appended, sorted

Scalar Objects

• int – represent integers, e.g. 5
• float – represent real numbers, e.g. 3.27
• bool – represent Boolean values True / False
• NoneType – special, has one value None

>>> type(5)
int
>>> type(3.0)
float

Type Conversions (Cast)

Converts object of one type to another

• float(3) → converts integer 3 to float 3.0

  >>> float(3)
  3.0

• int(3.9) → truncates float 3.9 to integer 3

  >>> int(3.9)
  3

Printing to Console

• Use print() to show output
• In IPython/Notebook, results appear below the code cell:
  • "Out" = result of expression (not printed)

    In [11]: 3+2
    Out[11]: 5

  • print() = explicit program output

    In [12]: print(3+2)
    5

Expressions

• Combine objects & operators to form expressions
• An expression has a value, which has a type
• Syntax for a simple expression:
  <object> <operator> <object>
• Example:
  • The expression 3 + 5 has value 8 of type int

  >>> type(3 + 5)
  int

Operators on int, float

SIMPLE OPERATIONS

• Parentheses tell Python what to do first
• Operator precedence (without parentheses):
  • **
  • *
  • /
  • + and - executed left to right

>>> 2 + 3 * 4
14
>>> (2 + 3) * 4
20

BINDING VARIABLES AND VALUES

The equal sign (=) is an assignment of a value to a variable name.

pi = 3.14159  # pi is a name, 3.14159 is a value
pi_approx = 22/7  # pi_approx is a name, 22/7 is a value

1. Value is stored in memory
2. Assignment binds name to value
3. Retrieve a value by typing its name

ABSTRACTING EXPRESSIONS

Why give names to values of expressions?

pi = 3.14159
radius = 2.2
length = 2 * pi * radius
area = pi * (radius ** 2)

• To reuse names instead of values
• Easier to change code later

PROGRAMMING vs MATH

• In programming, you do not “solve for x”

pi = 3.14159
radius = 2.2
area = pi * (radius ** 2)  # area of circle
radius = radius + 1

• Assignment = store a value in a variable
  • Right side → expression evaluated to a value
  • Left side → variable name
  • radius = radius + 1 is equivalent to radius += 1

CHANGING BINDINGS

• We can re-bind a name with a new assignment
• Old value may still exist in memory; the name just points elsewhere
• Computed values don't update automatically

pi = 3.14
radius = 2.2
area = pi * (radius ** 2)
radius = radius + 1

STRINGS

Output to console

Use print() to output stuff to console

>>> x = 1
>>> print(x)
1
>>> x_str = str(x)
>>> print("my fav num is", x, ".", "x =", x)
my fav num is 1 . x = 1
>>> print("my fav num is " + x_str + ". " + "x = " + x_str)
my fav num is 1. x = 1
  

Input from console

Comparison operators

• i and j are variable names (int, float, or str)
• Comparisons below evaluate to a Boolean
• i > j
• i >= j
• i < j
• i <= j
• i == j → equality test, True if i is equal j
• i != j → inequality test True if i is not equal j

Logic operators on bools

• a and b are variable names (with Boolean values)
• not a
  • True if a is False
  • False if a is True
• a and b
  • True if both are True
• a or b
  • True if either or both are True

Logic operators on bools

Comparison example

>>> pset_time = 15
>>> sleep_time = 8
>>> print(sleep_time > pset_time)
False
>>> drive = True
>>> drink = False
>>> both = drink and drive
>>> print(both)
False