Clean and summarize labelled tables using Python Series and DataFrame objects.

Before starting: Python, basic NumPy arrays, and tabular data.

Practice project: Clean a synthetic study log and verify grouped totals by hand.

1. Introduction, Environment & Core Architecture

  1. Pandas Introduction, History & The Tabular Data Revolution
  2. Environment Setup, Installation & Acceleration Dependencies
    Read the example and try it

    Check which Python executable imported pandas. An installation in a second environment cannot fix imports in the first one.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    import sys
    print(sys.executable, pd.__version__)

    Print the kernel interpreter and the installed pandas version.

    Run this in the kernel that will run your exercise.

    Expected result
    The path identifies your Python interpreter and the second value identifies its pandas version. Exact values vary.
  3. Core Data Structures: Series, DataFrame & Index Anatomy
    Read the example and try it

    A labelled column connects each count to a room. Build a second column to compare each room with a target.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    rooms = pd.DataFrame({"seats": [8, 12]}, index=["Lab", "Hall"])
    print(rooms.assign(spare=rooms.seats - 10))

    Build a Series of seat counts labeled by room.

    Label counts 8 and 12 with Lab and Hall.

    Expected result
    {'Lab': 8, 'Hall': 12}
  4. Mutability, Data Types & Arrow-Backed Dtypes in Pandas 2.x
    Read the example and try it

    A nullable integer column distinguishes an unknown count from zero. Inspect the missing mask before filling anything.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    count = pd.Series([0, None, 4], dtype="Int64")
    print(count.isna().tolist(), count.dtype)

    Create a column with zero, an unknown count and three.

    Use nullable Int64 and inspect its missing mask.

    Expected result
    [False, True, False] Int64
  5. DataFrame Inspection: info(), describe(), head(), tail() & shape
    Read the example and try it

    Count rows and missing entries before interpreting an average. A column can have fewer observed values than the table has rows.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    roster = pd.DataFrame({"mark": [7, None, 5]})
    print(roster.shape, roster["mark"].count())

    Compare the row count and the count of observed scores.

    Use three scores: 7, missing, 5.

    Expected result
    3 2

2. Pandas Series: 1D Labeled Arrays

  1. Creating Series: Sequences, NumPy Arrays, Dictionaries & Scalars
    Read the example and try it

    Building a Series from a dictionary keeps the keys as labels. A later lookup can use a name directly.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    minutes = pd.Series({"east": 12, "west": 8})
    print(minutes.loc["west"])

    Build a room-count Series and retrieve a value by room.

    Look up Hall in a dictionary Series with Lab=8 and Hall=12.

    Expected result
    12
  2. Series Indexing & Slicing: Positional vs Labeled Access
    Read the example and try it

    Labels and positions answer different questions. A zero-based position works even when the index starts at 100.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series(["bus", "rail"], index=[100, 200])
    print(s.loc[200], s.iloc[0])

    Select the first value from a Series whose labels are numbers other than zero.

    Use values 8,12 with index labels 10,20.

    Expected result
    8 — position 0 is label 10.
  3. Vectorized Operations & Automatic Label Alignment in Series
    Read the example and try it

    Before adding two Series, pandas lines up matching keys. Reversing the order of the second Series leaves each shop's sum correct.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    a = pd.Series({"A": 5, "B": 8})
    b = pd.Series({"B": 1, "A": 2})
    print((a+b).to_dict())

    Add two Series with reversed labels and check each sum.

    Use Lab=2, Hall=3 and Hall=10, Lab=20.

    Expected result
    22 13 — Lab first, Hall second.
  4. Series Attributes & Methods: values, index, dtype, nbytes & size
    Read the example and try it

    An index identifies each reading. The size counts entries, including an entry whose value is missing.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([8, None], index=["first", "last"])
    print(s.index.tolist(), s.size, s.count())

    Print the index and dtype and explain the difference.

    Use a labeled int64 Series.

    Expected result
    ['Lab', 'Hall'] int64 — the index names the rooms and the dtype describes the counts.
  5. Value Counts, Unique Elements, Mode & Frequency Analysis
    Read the example and try it

    Frequency and distinct values answer different questions. A label can appear three times while contributing only one distinct value.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    route = pd.Series(["B","A","B","B"])
    print(route.value_counts().to_dict(), route.nunique())

    Check counts for two repeats and one missing observation.

    Count two Lab entries and one missing entry.

    Expected result
    2 1 — observed Lab entries, then missing entries.

3. Pandas DataFrame: 2D Tabular Mechanics

  1. DataFrame Construction: Dict of Lists, List of Dicts & NumPy Arrays
    Read the example and try it

    Each dictionary describes one row. Omitting a field from one row leaves a missing cell in the combined table.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame([{"room":"A","seats":5},{"room":"B"}])
    print(df.isna().sum().to_dict())

    Build two columns for room and capacity.

    Use a Lab with 8 seats and a Hall with 12.

    Expected result
    (2, 2) ['room', 'capacity']
  2. Column Selection, Addition, Renaming & Deletion
    Read the example and try it

    Calculate a column from two existing columns. Selecting just the new column makes the calculation easy to check.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    orders = pd.DataFrame({"qty":[2,3], "price":[4,5]})
    orders["bill"] = orders.qty * orders.price
    print(orders.bill.tolist())

    Rename one column and inspect the original and returned column names.

    Rename capacity to seats in the returned table.

    Expected result
    ['capacity'] ['seats']
  3. Row Selection & Label-Based Indexing with .loc[]
    Read the example and try it

    A label slice includes both ends. With room names as labels, you can request the middle two rooms without their positions.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"seats":[4,6,8]}, index=["A","B","C"])
    print(df.loc["B":"C","seats"].tolist())

    Slice three ordered date labels using loc.

    Index values 1,2,3 by the dates January 1–3, 2025. Slice through January 2.

    Expected result
    [1, 2]
  4. Integer Positional Indexing with .iloc[] & Fast Accessors (.at[], .iat[])
    Read the example and try it

    A positional slice stops before its end. Compare the first two entries with the single entry at position two.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"seat":[3,5,9]})
    print(df.iloc[:2,0].tolist(), df.iat[2,0])

    Read one cell by position and then by its labels.

    Use one row labeled Lab and a seats column containing 8.

    Expected result
    8 8
  5. Boolean Indexing, isin() & Query Expressions (df.query())
    Read the example and try it

    One membership test can cover several allowed routes. A route absent from the list receives a false mask entry.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series(["A","B","C","B"])
    print(s[s.isin(["A","C"])].tolist())

    Keep visits for selected rooms above a threshold.

    Use Lab=4, Hall=8, Gym=12. Permit Lab or Hall and require more than 5 visits.

    Expected result
    ['Hall']

4. Essential Operations, Sorting & MultiIndex

  1. Reindexing, Setting & Resetting Indices (set_index, reset_index, reindex)
    Read the example and try it

    Setting a key as the index makes it available for label lookup. Reindexing can introduce a missing row for an unrecorded day.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"day":["Mon","Wed"],"n":[3,5]}).set_index("day")
    print(df.reindex(["Mon","Tue","Wed"]).isna().sum().to_dict())

    Reset a room index with and without drop=True.

    Set an index named room before resetting.

    Expected result
    ['room', 'seats'] ['seats']
  2. Sorting Data: sort_values(), sort_index() & Multi-Column Sorting
    Read the example and try it

    Break a time tie with the runner's name. Give both columns explicitly so the ordering rule is visible.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"name":["Zed","Ana","Tim"],"sec":[8,8,6]})
    print(df.sort_values(["sec","name"]).name.tolist())

    Sort room records by capacity and then by name for ties.

    Rooms B, A and C have 8, 8 and 4 seats. Sort ascending by seats then name.

    Expected result
    ['C', 'A', 'B']
  3. Ranking Methods & Tie-Breaking: rank(method=average|min|max|dense)
    Read the example and try it

    Tied scores receive the same dense rank. The next distinct score gets the next number.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([12,12,8,4])
    print(s.rank(method="dense", ascending=False).astype(int).tolist())

    Rank scores 9, 9 and 7 in descending order.

    Use dense ranking.

    Expected result
    [1.0, 1.0, 2.0]
  4. Hierarchical Indexing: Creating, Slicing & Cross-Sections (xs())
    Read the example and try it

    An index can record a town and a day together. Choosing a day at its named level compares towns on that day.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    idx = pd.MultiIndex.from_tuples([("A","Mon"),("B","Mon")], names=["town","day"])
    print(pd.Series([4,8], index=idx).xs("Mon", level="day").to_dict())

    Select all rooms from one building in a two-level index.

    Use North/Lab=2, North/Hall=4, South/Lab=8. Select North.

    Expected result
    {'Lab': 2, 'Hall': 4}
  5. MultiIndex Stacking, Level Swapping & Unstacking Operations
    Read the example and try it

    Unstack moves one label level into columns. A two-day series now has one row for the room.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    idx = pd.MultiIndex.from_tuples([("Lab","Mon"),("Lab","Tue")])
    print(pd.Series([3,5], index=idx).unstack())

    Unstack a building-room table and label the new columns.

    Record North/Lab=2 and North/Hall=4, then unstack room.

    Expected result
    ['Hall', 'Lab'] [4, 2]

5. Data Cleaning & Missing Data Handling

  1. Detecting Missing Values: isna(), isnull(), notna() & IEEE NaN vs pd.NA
    Read the example and try it

    A score of zero has been measured. Only the unknown score should produce a true missing flag.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([0, pd.NA, 4], dtype="Int64")
    print(s.isna().tolist())

    Distinguish an unknown reading from the numeric value zero.

    Check missingness for zero and a missing entry.

    Expected result
    [False, True]
  2. Dropping Missing Values: dropna(axis, how, thresh)
    Read the example and try it

    Discard a row only when a field your analysis needs is missing. A blank optional comment can safely stay.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"id":[1,None,3],"note":[None,"ok","ok"]})
    print(df.dropna(subset=["id"]).index.tolist())

    Keep rows with known scores even when a comment is missing.

    Rows 0 and 1 have scores 7 and missing. Both lack comments.

    Expected result
    [0]
  3. Imputing Missing Values: fillna(), ffill(), bfill() & Interpolation
    Read the example and try it

    Carrying a status forward assumes it lasts until another status arrives. The first entry has no earlier reading to use.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([None,"open",None,"closed"])
    print(s.ffill().tolist())

    Sort time observations before comparing forward-filled values.

    Use days 2,1,3 with readings missing,5,8 in one room. Sort by day first.

    Expected result
    [5.0, 5.0, 8.0]
  4. Deduplication: duplicated() Flags & drop_duplicates()
    Read the example and try it

    Keep the latest correction for each ticket. Sort by revision first so the last row really is the most recent.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"id":[7,7],"rev":[2,1],"fare":[5,8]})
    print(df.sort_values("rev").drop_duplicates("id", keep="last").fare.tolist())

    Mark duplicates once by all columns and once by room ID.

    Use room IDs 1,1,1 and scores 4,4,5.

    Expected result
    [False, True, False] [False, True, True]
  5. Vectorized String Methods: str.contains(), str.extract(), str.split() & RegEx
    Read the example and try it

    Split a compound label into parts before comparing room numbers. The result still consists of text until you convert it.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    labels = pd.Series(["Lab-03","Hall-12"])
    print(labels.str.split("-").str[-1].astype(int).tolist())

    Remove surrounding spaces before checking a room prefix.

    Test the labels " LAB-1 " and "Hall".

    Expected result
    [True, False]

6. Data Transformations & Mapping

  1. Mapping & Function Application: map(), apply() & Element-Wise Transforms
    Read the example and try it

    Look up each code in a mapping. A code missing from the dictionary becomes missing and can be checked separately.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    codes = pd.Series(["A","B","X"])
    print(codes.map({"A":"arrived","B":"booked"}).isna().tolist())

    Map three codes with only two entries in the mapping.

    Map L to Lab and H to Hall. Include the unknown code X.

    Expected result
    ['Lab', 'Hall', 'unknown'] — the last label replaces a missing map result.
  2. Discretization & Binning: Continuous to Categorical (cut & qcut)
    Read the example and try it

    Intervals can share a boundary without sharing a value. Ten goes in the interval ending at ten with the default right-closed rule.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    print(pd.cut([2,10,11], [0,10,20], labels=["small","large"]).tolist())

    Compare two quantile groups with two equal-width intervals.

    Use values 1, 2, 3 and 100 with two bins. Print bin numbers starting at zero.

    Expected result
    [0, 0, 1, 1] [0, 0, 0, 1] — quantile groups then equal widths.
  3. Replacing Values: replace() with Scalars, Lists, RegEx & Dictionaries
    Read the example and try it

    Replace a documented sentinel before measuring availability. Zero can be a valid temperature, so keep it as a reading.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([-99,0,12], dtype="Int64")
    print(s.replace(-99,pd.NA).count())

    Replace two legacy labels and preserve a third.

    Replace L and H in [L,H,Gym].

    Expected result
    ['Lab', 'Hall', 'Gym']
  4. Outlier Detection, Truncation & Winsorization with df.clip()
    Read the example and try it

    Capping values affects both extremes. Inspect the raw series alongside the capped one to see which rows changed.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    raw = pd.Series([-1,4,22])
    limited = raw.clip(0,20)
    print(raw.ne(limited).tolist())

    Clip [-2,5,20] to [0,10] and compare row counts.

    Clip the three values to the inclusive range 0–10.

    Expected result
    [0, 5, 10] 3
  5. Categorical Data Type: Memory Compression & Ordered Enums
    Read the example and try it

    Declare size order once so sorting follows the order customers choose. Alphabetical order would put large first.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    sizes = pd.Categorical(["L","S","M"], categories=["S","M","L"], ordered=True)
    print(sizes.sort_values().tolist())

    Compare memory usage before and after categorizing repeated names.

    Use 2,000 alternating room names stored as object. Compare deep byte counts.

    Expected result
    True — category uses fewer bytes for this repetitive sample. Byte totals vary.

7. Combining, Merging & Concatenating

  1. Concatenating Data: pd.concat() along Rows and Columns with Keys
    Read the example and try it

    When concatenating weekly batches, keys record which batch supplied each value. This is useful when row numbers repeat.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    series = pd.concat([pd.Series([7]),pd.Series([9])],keys=["Mon","Tue"])
    print(series.loc["Tue",0])

    Stack two logs and retain their source using keys.

    Stack one entry for each of Mon and Tue and inspect the outer index.

    Expected result
    ['Mon', 'Tue']
  2. Database-Style Merges: pd.merge() Inner, Left, Right & Full Outer Joins
    Read the example and try it

    Use the pupil list as the left input to preserve everyone. The unmatched pupil has an unknown mark after the merge.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    pupils = pd.DataFrame({"id":[4,7,9]})
    marks = pd.DataFrame({"id":[4,9],"mark":[82,61]})
    print(pupils.merge(marks,how="left",on="id").mark.isna().tolist())

    Merge bookings with room details and inspect an unmatched booking.

    Book rooms 1 and 2. Supply details only for room 1. Use a left join.

    Expected result
    ['Lab', 'unknown']
  3. Advanced Merge Keys: Multiple Columns, Dissimilar Names & Index Joins
    Read the example and try it

    Both train and seat identify a reservation. Joining on seat alone could attach another train's passenger.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    left = pd.DataFrame({"train":["A","B"],"seat":[5,5]})
    right = pd.DataFrame({"train":["B"],"seat":[5],"name":["Ira"]})
    print(left.merge(right,on=["train","seat"]).name.tolist())

    Match a room_code column to a code column.

    Join room_code L to code L with the name Lab.

    Expected result
    Lab
  4. DataFrame join() Method & Suffix Collision Disambiguation
    Read the example and try it

    A suffix keeps two same-named columns identifiable after an index join. Decide which one to use in your calculation.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    a = pd.DataFrame({"n":[5]},index=["Lab"])
    b = pd.DataFrame({"n":[7]},index=["Lab"])
    print(a.join(b,lsuffix="_booked",rsuffix="_seats").columns.tolist())

    Join two tables that both have a status column.

    Join status=open and status=ready on the same index.

    Expected result
    ['status_a', 'status_b']
  5. Overlapping Data & Patching: combine_first() and update()
    Read the example and try it

    Use the backup only where today's value is missing. Both Series align by their keys.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    now = pd.Series({"A":4,"B":None})
    old = pd.Series({"A":2,"B":7})
    print(now.combine_first(old).to_dict())

    Fill missing readings from a backup while retaining measured zeros.

    Use current=[0,missing] and backup=[8,5].

    Expected result
    [0.0, 5.0]

8. Reshaping, Pivoting & Cross-Tabulations

  1. Pivoting Tables: df.pivot() for Long-to-Wide Reshaping
    Read the example and try it

    One value per room and day fits in a pivot cell. Repeating a pair would require an aggregation decision.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"room":["A","A"],"day":["Mon","Tue"],"n":[3,7]})
    print(df.pivot(index="room", columns="day", values="n"))

    Try reshaping with a duplicated room-day pair.

    Try two observations for the same room and day.

    Expected result
    ValueError — pivot has two candidates for one cell. Aggregate explicitly to continue.
  2. General Pivot Tables: pd.pivot_table() with Multi-Aggregations & Margins
    Read the example and try it

    When several sales share a day and shop, sum them before laying out the table. A pivot table performs both steps.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"shop":["A","A"],"day":["Mon","Mon"],"n":[3,7]})
    print(df.pivot_table(index="shop",columns="day",values="n",aggfunc="sum").iat[0,0])

    Choose sum or mean for repeated daily readings.

    For Lab/Mon use two disjoint session counts 2 and 4. Compare total and typical session.

    Expected result
    6 3.0
  3. Unpivoting Data: pd.melt() for Wide-to-Long Tidy Data Restructuring
    Read the example and try it

    Melt keeps the room label beside each day's number. This turns two measurement columns into two observations.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"room":["A"],"Mon":[3],"Tue":[7]})
    print(df.melt(id_vars="room",var_name="day",value_name="visits").to_dict("records"))

    Unpivot two weekdays while retaining the room identifier.

    Use Lab with Mon=2 and Tue=4.

    Expected result
    [('Lab', 'Mon', 2), ('Lab', 'Tue', 4)]
  4. Two-Way Frequency Cross-Tabulation: pd.crosstab()
    Read the example and try it

    Count observed pairs of labels. An empty combination in the table means no matching input rows.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    print(pd.crosstab(["A","A","B"],["bus","rail","bus"]))

    Count bookings by building and day.

    Use North/Mon twice and North/Tue once.

    Expected result
    2 1
  5. Advanced Reshaping Pipelines: Combining melt, pivot_table & stack
    Read the example and try it

    Reshape the dates into one field before adding visits by room. This lets the same operation handle additional day columns.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"room":["A","B"],"Mon":[2,3],"Tue":[4,1]})
    long = df.melt(id_vars="room")
    print(long.groupby("room").value.sum().to_dict())

    Check for duplicate keys before pivoting a melted table.

    Try two identical (Lab,Mon) keys before pivoting.

    Expected result
    True — resolve the repeated key first.

9. GroupBy Mechanics & Window Functions

  1. The Split-Apply-Combine Paradigm: df.groupby() Fundamentals
    Read the example and try it

    Each parcel goes into the group named by its route. Summing inside each group gives one weight per route.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"route":["A","B","A"],"kg":[2,7,5]})
    print(df.groupby("route").kg.sum().to_dict())

    Recalculate one room average with and without filling the blank.

    Use the West room from the downloadable visits log: 7, blank, 11.

    Expected result
    9.0 6.0 — excluding the blank divides by two. Filling it with zero divides by three.
  2. Group Aggregations: agg(), Named Aggregations & Multi-Function Metrics
    Read the example and try it

    Named aggregations give each statistic a readable column label. Size counts parcels and max picks the heaviest one.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"route":["A","A"],"kg":[2,7]})
    print(df.groupby("route").agg(parcels=("kg","size"), heaviest=("kg","max")))

    Produce separate total and average columns per room.

    Use Lab visits 2 and 4.

    Expected result
    6 3.0
  3. Group Transformations: transform() for Standardization & Broadcast Metrics
    Read the example and try it

    Transform returns a result aligned with every pupil. Subtract the class mean while keeping the original row order.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"class":["A","B","A"],"n":[2,8,6]})
    print((df.n - df.groupby("class").n.transform("mean")).tolist())

    Attach each room's mean beside each daily visit count.

    Use rooms Lab, Lab, Hall with counts 2,4,8.

    Expected result
    [3.0, 3.0, 8.0]
  4. Group Filtering: filter() for Discarding Small or Non-Compliant Cohorts
    Read the example and try it

    Group filtering keeps or removes whole groups. Here the solitary B observation disappears.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    df = pd.DataFrame({"sensor":["A","B","A"],"n":[2,8,6]})
    print(df.groupby("sensor").filter(lambda batch: len(batch)>=2).n.tolist())

    Keep rooms with at least three observations.

    Use three Lab rows and one Hall row, each with a recorded count.

    Expected result
    ['Lab', 'Lab', 'Lab']
  5. Rolling & Expanding Windows: Moving Averages, EWM & Cumulative Metrics
    Read the example and try it

    An expanding mean uses all readings so far. A two-reading rolling mean forgets the oldest reading on day three.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([2,4,12])
    print(s.expanding().mean().iloc[-1], s.rolling(2).mean().iloc[-1])

    Compare a rolling mean of two days with an expanding mean.

    Use [2,4,9] and compare the last rolling and expanding means.

    Expected result
    6.5 5.0

10. Time Series, I/O Tools & Performance Optimization

  1. Date & Time Fundamentals: pd.to_datetime(), date_range() & DatetimeIndex
    Read the example and try it

    An explicit format gives an ambiguous date one interpretation. Pull out the month afterwards to check the choice.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    day = pd.to_datetime("03/04/2025", format="%d/%m/%Y")
    print(day.month, day.day)

    Flag invalid dates after parsing a mixed column.

    Use 2025-01-01 and an impossible 2025-02-30.

    Expected result
    [False, True]
  2. Time Series Resampling, Shifting & Frequency Conversion
    Read the example and try it

    Combine readings within the same calendar day before comparing days. Resampling needs a time-based index.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series([3,6,1],index=pd.to_datetime(["2025-01-01 09:00","2025-01-01 16:00","2025-01-02 09:00"]))
    print(s.resample("D").sum().tolist())

    Compare yesterday's value with today's on an ordered table.

    Use three consecutive days with values 2,5,4. Compare changes after day one.

    Expected result
    [3.0, -1.0]
  3. Flat File & Tabular I/O: Fast CSV, TSV & Excel Reading/Writing
    Read the example and try it

    Treat identifiers as text on import. A leading zero in a ticket ID is meaningful even though it adds no numeric value.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    from io import StringIO
    df = pd.read_csv(StringIO("id,seats\n007,4"),dtype={"id":str})
    print(df.id.iloc[0])

    Read a tab-separated header as two fields.

    Use one tab between room and visits in the header and one between Lab and 2.

    Expected result
    ['room', 'visits']
  4. High-Speed Columnar Storage: Parquet, Feather & Arrow Integration
    Read the example and try it

    A reader can select just the needed Parquet column. This round trip uses an in-memory file and requires pyarrow or fastparquet.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    from io import BytesIO
    buf = BytesIO()
    pd.DataFrame({"room":["A"],"seats":[4]}).to_parquet(buf,index=False)
    buf.seek(0)
    print(pd.read_parquet(buf,columns=["seats"]).iat[0,0])

    Compare reading two selected columns with the whole table.

    Write three columns room, visits, day to Parquet and read only the first two. Requires pyarrow.

    Expected result
    (1, 3) (1, 2)
  5. High-Performance Pandas: PyArrow Engine, eval(), query() & Memory Tuning
    Read the example and try it

    Measure storage on the actual repeated labels. Categories store a vocabulary and integer codes, which can help when the vocabulary is small.

    Python 3 with pandas. Run the entire example in one cell or script.

    import pandas as pd
    
    s = pd.Series(["north","south"]*500)
    small = s.astype("category")
    print(s.memory_usage(deep=True) > small.memory_usage(deep=True))

    Measure memory before and after changing a repeated text column's dtype.

    Compare deep memory for 5,000 identical object strings before and after categorizing.

    Expected result
    True — the category representation uses fewer bytes here.

Test a Python function with unittest

A test turns an expected behavior into a repeatable check. Keep the calculation separate from printing so a test can inspect its return value. An average of an empty list has no value in this example, so the function raises ValueError instead of silently inventing a result.

Python 3, standard library only

import unittest

def average(scores):
    if not scores:
        raise ValueError("scores must not be empty")
    return sum(scores) / len(scores)

class AverageTests(unittest.TestCase):
    def test_two_scores(self):
        self.assertEqual(average([60, 80]), 70)

    def test_empty_scores(self):
        with self.assertRaises(ValueError):
            average([])
Check the output
Both tests pass. unittest reports 2 tests and OK; elapsed time varies.

Add tests for a single score and decimal scores. Use assertAlmostEqual for a floating-point comparison.

Avoid data leakage when preparing a dataset

A model evaluation is useful only when the held-out rows have not influenced training. Split the rows before learning a mean, scaler, vocabulary, or feature-selection rule. Fit those rules using training data, then reuse them unchanged on validation and test data. Time-ordered data often needs a chronological split instead of a random split.

Python 3; example uses no external packages

training = [10, None, 30]
testing = [None, 90]

observed = [x for x in training if x is not None]
mean = sum(observed) / len(observed)

def fill(values):
    return [mean if x is None else x for x in values]

print(fill(training))
print(fill(testing))
Check the output
[10, 20.0, 30]
[20.0, 90]

Explain why including 90 when computing the replacement mean leaks information. Decide what to do if every training value is missing.

Make a small learning project reproducible

A project is reproducible when another learner can obtain the same inputs, install the required tools, and repeat your checks. Record the tool versions and exact run commands. Use synthetic or permission-cleared sample data. Keep credentials outside the repository. A successful run on your laptop is a starting point; a fresh environment is a better check.

Any language; choose the package and build tools appropriate to that course

study-project/
  README.md          # setup, run, test, expected result
  src/               # your implementation
  tests/             # behavior checks
  data/              # small synthetic example data
  .gitignore         # caches, build output, local secrets
Check the output
This is a suggested directory layout, not a command to execute. Adapt conventions to the chosen language.

Build a score summarizer. Include an empty-input test, a malformed-input test, and one example whose answer can be calculated by hand.

Check pandas keys and totals before reporting

Before trusting a summary, check that required columns exist and that identifiers obey the intended uniqueness rule. A duplicate may be a valid repeated observation or an accidental repeated import; do not drop it without deciding which interpretation applies. Use an explicit error for data validation rather than relying on assertions that can be disabled.

Python 3 with pandas

import pandas as pd

rows = pd.DataFrame({"id": [1, 2], "minutes": [20, 30]})
required = {"id", "minutes"}
if not required.issubset(rows.columns):
    raise ValueError("missing required columns")
if rows["id"].isna().any() or not rows["id"].is_unique:
    raise ValueError("id must be present and unique")
if rows["minutes"].isna().any() or not pd.api.types.is_numeric_dtype(rows["minutes"]):
    raise ValueError("minutes must be numeric and present")
print(rows["minutes"].sum())
Check the output
50

Add a negative duration and decide whether it is valid. Add the corresponding explicit check before grouping or plotting.

Open the pandas learning path on DevStudyHub · Play coding games on this blog

logoblog

No comments:

Post a Comment

Please do not enter any spam link in the comment box.