Clean and summarize labelled tables using Python Series and DataFrame objects.
Before starting: Python, basic NumPy arrays, and tabular data.
Practice project: Clean a synthetic study log and verify grouped totals by hand.
1. Introduction, Environment & Core Architecture
- Pandas Introduction, History & The Tabular Data Revolution
- Environment Setup, Installation & Acceleration Dependencies
Read the example and try it
Check which Python executable imported pandas. An installation in a second environment cannot fix imports in the first one.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd import sys print(sys.executable, pd.__version__)Print the kernel interpreter and the installed pandas version.
Run this in the kernel that will run your exercise.
Expected result
The path identifies your Python interpreter and the second value identifies its pandas version. Exact values vary. - Core Data Structures: Series, DataFrame & Index Anatomy
Read the example and try it
A labelled column connects each count to a room. Build a second column to compare each room with a target.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd rooms = pd.DataFrame({"seats": [8, 12]}, index=["Lab", "Hall"]) print(rooms.assign(spare=rooms.seats - 10))Build a Series of seat counts labeled by room.
Label counts 8 and 12 with Lab and Hall.
Expected result
{'Lab': 8, 'Hall': 12} - Mutability, Data Types & Arrow-Backed Dtypes in Pandas 2.x
Read the example and try it
A nullable integer column distinguishes an unknown count from zero. Inspect the missing mask before filling anything.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd count = pd.Series([0, None, 4], dtype="Int64") print(count.isna().tolist(), count.dtype)Create a column with zero, an unknown count and three.
Use nullable Int64 and inspect its missing mask.
Expected result
[False, True, False] Int64 - DataFrame Inspection: info(), describe(), head(), tail() & shape
Read the example and try it
Count rows and missing entries before interpreting an average. A column can have fewer observed values than the table has rows.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd roster = pd.DataFrame({"mark": [7, None, 5]}) print(roster.shape, roster["mark"].count())Compare the row count and the count of observed scores.
Use three scores: 7, missing, 5.
Expected result
3 2
2. Pandas Series: 1D Labeled Arrays
- Creating Series: Sequences, NumPy Arrays, Dictionaries & Scalars
Read the example and try it
Building a Series from a dictionary keeps the keys as labels. A later lookup can use a name directly.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd minutes = pd.Series({"east": 12, "west": 8}) print(minutes.loc["west"])Build a room-count Series and retrieve a value by room.
Look up Hall in a dictionary Series with Lab=8 and Hall=12.
Expected result
12 - Series Indexing & Slicing: Positional vs Labeled Access
Read the example and try it
Labels and positions answer different questions. A zero-based position works even when the index starts at 100.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series(["bus", "rail"], index=[100, 200]) print(s.loc[200], s.iloc[0])Select the first value from a Series whose labels are numbers other than zero.
Use values 8,12 with index labels 10,20.
Expected result
8 — position 0 is label 10. - Vectorized Operations & Automatic Label Alignment in Series
Read the example and try it
Before adding two Series, pandas lines up matching keys. Reversing the order of the second Series leaves each shop's sum correct.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd a = pd.Series({"A": 5, "B": 8}) b = pd.Series({"B": 1, "A": 2}) print((a+b).to_dict())Add two Series with reversed labels and check each sum.
Use Lab=2, Hall=3 and Hall=10, Lab=20.
Expected result
22 13 — Lab first, Hall second. - Series Attributes & Methods: values, index, dtype, nbytes & size
Read the example and try it
An index identifies each reading. The size counts entries, including an entry whose value is missing.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([8, None], index=["first", "last"]) print(s.index.tolist(), s.size, s.count())Print the index and dtype and explain the difference.
Use a labeled int64 Series.
Expected result
['Lab', 'Hall'] int64 — the index names the rooms and the dtype describes the counts. - Value Counts, Unique Elements, Mode & Frequency Analysis
Read the example and try it
Frequency and distinct values answer different questions. A label can appear three times while contributing only one distinct value.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd route = pd.Series(["B","A","B","B"]) print(route.value_counts().to_dict(), route.nunique())Check counts for two repeats and one missing observation.
Count two Lab entries and one missing entry.
Expected result
2 1 — observed Lab entries, then missing entries.
3. Pandas DataFrame: 2D Tabular Mechanics
- DataFrame Construction: Dict of Lists, List of Dicts & NumPy Arrays
Read the example and try it
Each dictionary describes one row. Omitting a field from one row leaves a missing cell in the combined table.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame([{"room":"A","seats":5},{"room":"B"}]) print(df.isna().sum().to_dict())Build two columns for room and capacity.
Use a Lab with 8 seats and a Hall with 12.
Expected result
(2, 2) ['room', 'capacity'] - Column Selection, Addition, Renaming & Deletion
Read the example and try it
Calculate a column from two existing columns. Selecting just the new column makes the calculation easy to check.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd orders = pd.DataFrame({"qty":[2,3], "price":[4,5]}) orders["bill"] = orders.qty * orders.price print(orders.bill.tolist())Rename one column and inspect the original and returned column names.
Rename capacity to seats in the returned table.
Expected result
['capacity'] ['seats'] - Row Selection & Label-Based Indexing with .loc[]
Read the example and try it
A label slice includes both ends. With room names as labels, you can request the middle two rooms without their positions.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"seats":[4,6,8]}, index=["A","B","C"]) print(df.loc["B":"C","seats"].tolist())Slice three ordered date labels using loc.
Index values 1,2,3 by the dates January 1–3, 2025. Slice through January 2.
Expected result
[1, 2] - Integer Positional Indexing with .iloc[] & Fast Accessors (.at[], .iat[])
Read the example and try it
A positional slice stops before its end. Compare the first two entries with the single entry at position two.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"seat":[3,5,9]}) print(df.iloc[:2,0].tolist(), df.iat[2,0])Read one cell by position and then by its labels.
Use one row labeled Lab and a seats column containing 8.
Expected result
8 8 - Boolean Indexing, isin() & Query Expressions (df.query())
Read the example and try it
One membership test can cover several allowed routes. A route absent from the list receives a false mask entry.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series(["A","B","C","B"]) print(s[s.isin(["A","C"])].tolist())Keep visits for selected rooms above a threshold.
Use Lab=4, Hall=8, Gym=12. Permit Lab or Hall and require more than 5 visits.
Expected result
['Hall']
4. Essential Operations, Sorting & MultiIndex
- Reindexing, Setting & Resetting Indices (set_index, reset_index, reindex)
Read the example and try it
Setting a key as the index makes it available for label lookup. Reindexing can introduce a missing row for an unrecorded day.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"day":["Mon","Wed"],"n":[3,5]}).set_index("day") print(df.reindex(["Mon","Tue","Wed"]).isna().sum().to_dict())Reset a room index with and without drop=True.
Set an index named room before resetting.
Expected result
['room', 'seats'] ['seats'] - Sorting Data: sort_values(), sort_index() & Multi-Column Sorting
Read the example and try it
Break a time tie with the runner's name. Give both columns explicitly so the ordering rule is visible.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"name":["Zed","Ana","Tim"],"sec":[8,8,6]}) print(df.sort_values(["sec","name"]).name.tolist())Sort room records by capacity and then by name for ties.
Rooms B, A and C have 8, 8 and 4 seats. Sort ascending by seats then name.
Expected result
['C', 'A', 'B'] - Ranking Methods & Tie-Breaking: rank(method=average|min|max|dense)
Read the example and try it
Tied scores receive the same dense rank. The next distinct score gets the next number.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([12,12,8,4]) print(s.rank(method="dense", ascending=False).astype(int).tolist())Rank scores 9, 9 and 7 in descending order.
Use dense ranking.
Expected result
[1.0, 1.0, 2.0] - Hierarchical Indexing: Creating, Slicing & Cross-Sections (xs())
Read the example and try it
An index can record a town and a day together. Choosing a day at its named level compares towns on that day.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd idx = pd.MultiIndex.from_tuples([("A","Mon"),("B","Mon")], names=["town","day"]) print(pd.Series([4,8], index=idx).xs("Mon", level="day").to_dict())Select all rooms from one building in a two-level index.
Use North/Lab=2, North/Hall=4, South/Lab=8. Select North.
Expected result
{'Lab': 2, 'Hall': 4} - MultiIndex Stacking, Level Swapping & Unstacking Operations
Read the example and try it
Unstack moves one label level into columns. A two-day series now has one row for the room.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd idx = pd.MultiIndex.from_tuples([("Lab","Mon"),("Lab","Tue")]) print(pd.Series([3,5], index=idx).unstack())Unstack a building-room table and label the new columns.
Record North/Lab=2 and North/Hall=4, then unstack room.
Expected result
['Hall', 'Lab'] [4, 2]
5. Data Cleaning & Missing Data Handling
- Detecting Missing Values: isna(), isnull(), notna() & IEEE NaN vs pd.NA
Read the example and try it
A score of zero has been measured. Only the unknown score should produce a true missing flag.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([0, pd.NA, 4], dtype="Int64") print(s.isna().tolist())Distinguish an unknown reading from the numeric value zero.
Check missingness for zero and a missing entry.
Expected result
[False, True] - Dropping Missing Values: dropna(axis, how, thresh)
Read the example and try it
Discard a row only when a field your analysis needs is missing. A blank optional comment can safely stay.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"id":[1,None,3],"note":[None,"ok","ok"]}) print(df.dropna(subset=["id"]).index.tolist())Keep rows with known scores even when a comment is missing.
Rows 0 and 1 have scores 7 and missing. Both lack comments.
Expected result
[0] - Imputing Missing Values: fillna(), ffill(), bfill() & Interpolation
Read the example and try it
Carrying a status forward assumes it lasts until another status arrives. The first entry has no earlier reading to use.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([None,"open",None,"closed"]) print(s.ffill().tolist())Sort time observations before comparing forward-filled values.
Use days 2,1,3 with readings missing,5,8 in one room. Sort by day first.
Expected result
[5.0, 5.0, 8.0] - Deduplication: duplicated() Flags & drop_duplicates()
Read the example and try it
Keep the latest correction for each ticket. Sort by revision first so the last row really is the most recent.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"id":[7,7],"rev":[2,1],"fare":[5,8]}) print(df.sort_values("rev").drop_duplicates("id", keep="last").fare.tolist())Mark duplicates once by all columns and once by room ID.
Use room IDs 1,1,1 and scores 4,4,5.
Expected result
[False, True, False] [False, True, True] - Vectorized String Methods: str.contains(), str.extract(), str.split() & RegEx
Read the example and try it
Split a compound label into parts before comparing room numbers. The result still consists of text until you convert it.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd labels = pd.Series(["Lab-03","Hall-12"]) print(labels.str.split("-").str[-1].astype(int).tolist())Remove surrounding spaces before checking a room prefix.
Test the labels " LAB-1 " and "Hall".
Expected result
[True, False]
6. Data Transformations & Mapping
- Mapping & Function Application: map(), apply() & Element-Wise Transforms
Read the example and try it
Look up each code in a mapping. A code missing from the dictionary becomes missing and can be checked separately.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd codes = pd.Series(["A","B","X"]) print(codes.map({"A":"arrived","B":"booked"}).isna().tolist())Map three codes with only two entries in the mapping.
Map L to Lab and H to Hall. Include the unknown code X.
Expected result
['Lab', 'Hall', 'unknown'] — the last label replaces a missing map result. - Discretization & Binning: Continuous to Categorical (cut & qcut)
Read the example and try it
Intervals can share a boundary without sharing a value. Ten goes in the interval ending at ten with the default right-closed rule.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd print(pd.cut([2,10,11], [0,10,20], labels=["small","large"]).tolist())Compare two quantile groups with two equal-width intervals.
Use values 1, 2, 3 and 100 with two bins. Print bin numbers starting at zero.
Expected result
[0, 0, 1, 1] [0, 0, 0, 1] — quantile groups then equal widths. - Replacing Values: replace() with Scalars, Lists, RegEx & Dictionaries
Read the example and try it
Replace a documented sentinel before measuring availability. Zero can be a valid temperature, so keep it as a reading.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([-99,0,12], dtype="Int64") print(s.replace(-99,pd.NA).count())Replace two legacy labels and preserve a third.
Replace L and H in [L,H,Gym].
Expected result
['Lab', 'Hall', 'Gym'] - Outlier Detection, Truncation & Winsorization with df.clip()
Read the example and try it
Capping values affects both extremes. Inspect the raw series alongside the capped one to see which rows changed.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd raw = pd.Series([-1,4,22]) limited = raw.clip(0,20) print(raw.ne(limited).tolist())Clip [-2,5,20] to [0,10] and compare row counts.
Clip the three values to the inclusive range 0–10.
Expected result
[0, 5, 10] 3 - Categorical Data Type: Memory Compression & Ordered Enums
Read the example and try it
Declare size order once so sorting follows the order customers choose. Alphabetical order would put large first.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd sizes = pd.Categorical(["L","S","M"], categories=["S","M","L"], ordered=True) print(sizes.sort_values().tolist())Compare memory usage before and after categorizing repeated names.
Use 2,000 alternating room names stored as object. Compare deep byte counts.
Expected result
True — category uses fewer bytes for this repetitive sample. Byte totals vary.
7. Combining, Merging & Concatenating
- Concatenating Data: pd.concat() along Rows and Columns with Keys
Read the example and try it
When concatenating weekly batches, keys record which batch supplied each value. This is useful when row numbers repeat.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd series = pd.concat([pd.Series([7]),pd.Series([9])],keys=["Mon","Tue"]) print(series.loc["Tue",0])Stack two logs and retain their source using keys.
Stack one entry for each of Mon and Tue and inspect the outer index.
Expected result
['Mon', 'Tue'] - Database-Style Merges: pd.merge() Inner, Left, Right & Full Outer Joins
Read the example and try it
Use the pupil list as the left input to preserve everyone. The unmatched pupil has an unknown mark after the merge.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd pupils = pd.DataFrame({"id":[4,7,9]}) marks = pd.DataFrame({"id":[4,9],"mark":[82,61]}) print(pupils.merge(marks,how="left",on="id").mark.isna().tolist())Merge bookings with room details and inspect an unmatched booking.
Book rooms 1 and 2. Supply details only for room 1. Use a left join.
Expected result
['Lab', 'unknown'] - Advanced Merge Keys: Multiple Columns, Dissimilar Names & Index Joins
Read the example and try it
Both train and seat identify a reservation. Joining on seat alone could attach another train's passenger.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd left = pd.DataFrame({"train":["A","B"],"seat":[5,5]}) right = pd.DataFrame({"train":["B"],"seat":[5],"name":["Ira"]}) print(left.merge(right,on=["train","seat"]).name.tolist())Match a room_code column to a code column.
Join room_code L to code L with the name Lab.
Expected result
Lab - DataFrame join() Method & Suffix Collision Disambiguation
Read the example and try it
A suffix keeps two same-named columns identifiable after an index join. Decide which one to use in your calculation.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd a = pd.DataFrame({"n":[5]},index=["Lab"]) b = pd.DataFrame({"n":[7]},index=["Lab"]) print(a.join(b,lsuffix="_booked",rsuffix="_seats").columns.tolist())Join two tables that both have a status column.
Join status=open and status=ready on the same index.
Expected result
['status_a', 'status_b'] - Overlapping Data & Patching: combine_first() and update()
Read the example and try it
Use the backup only where today's value is missing. Both Series align by their keys.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd now = pd.Series({"A":4,"B":None}) old = pd.Series({"A":2,"B":7}) print(now.combine_first(old).to_dict())Fill missing readings from a backup while retaining measured zeros.
Use current=[0,missing] and backup=[8,5].
Expected result
[0.0, 5.0]
8. Reshaping, Pivoting & Cross-Tabulations
- Pivoting Tables: df.pivot() for Long-to-Wide Reshaping
Read the example and try it
One value per room and day fits in a pivot cell. Repeating a pair would require an aggregation decision.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"room":["A","A"],"day":["Mon","Tue"],"n":[3,7]}) print(df.pivot(index="room", columns="day", values="n"))Try reshaping with a duplicated room-day pair.
Try two observations for the same room and day.
Expected result
ValueError — pivot has two candidates for one cell. Aggregate explicitly to continue. - General Pivot Tables: pd.pivot_table() with Multi-Aggregations & Margins
Read the example and try it
When several sales share a day and shop, sum them before laying out the table. A pivot table performs both steps.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"shop":["A","A"],"day":["Mon","Mon"],"n":[3,7]}) print(df.pivot_table(index="shop",columns="day",values="n",aggfunc="sum").iat[0,0])Choose sum or mean for repeated daily readings.
For Lab/Mon use two disjoint session counts 2 and 4. Compare total and typical session.
Expected result
6 3.0 - Unpivoting Data: pd.melt() for Wide-to-Long Tidy Data Restructuring
Read the example and try it
Melt keeps the room label beside each day's number. This turns two measurement columns into two observations.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"room":["A"],"Mon":[3],"Tue":[7]}) print(df.melt(id_vars="room",var_name="day",value_name="visits").to_dict("records"))Unpivot two weekdays while retaining the room identifier.
Use Lab with Mon=2 and Tue=4.
Expected result
[('Lab', 'Mon', 2), ('Lab', 'Tue', 4)] - Two-Way Frequency Cross-Tabulation: pd.crosstab()
Read the example and try it
Count observed pairs of labels. An empty combination in the table means no matching input rows.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd print(pd.crosstab(["A","A","B"],["bus","rail","bus"]))Count bookings by building and day.
Use North/Mon twice and North/Tue once.
Expected result
2 1 - Advanced Reshaping Pipelines: Combining melt, pivot_table & stack
Read the example and try it
Reshape the dates into one field before adding visits by room. This lets the same operation handle additional day columns.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"room":["A","B"],"Mon":[2,3],"Tue":[4,1]}) long = df.melt(id_vars="room") print(long.groupby("room").value.sum().to_dict())Check for duplicate keys before pivoting a melted table.
Try two identical (Lab,Mon) keys before pivoting.
Expected result
True — resolve the repeated key first.
9. GroupBy Mechanics & Window Functions
- The Split-Apply-Combine Paradigm: df.groupby() Fundamentals
Read the example and try it
Each parcel goes into the group named by its route. Summing inside each group gives one weight per route.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"route":["A","B","A"],"kg":[2,7,5]}) print(df.groupby("route").kg.sum().to_dict())Recalculate one room average with and without filling the blank.
Use the West room from the downloadable visits log: 7, blank, 11.
Expected result
9.0 6.0 — excluding the blank divides by two. Filling it with zero divides by three. - Group Aggregations: agg(), Named Aggregations & Multi-Function Metrics
Read the example and try it
Named aggregations give each statistic a readable column label. Size counts parcels and max picks the heaviest one.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"route":["A","A"],"kg":[2,7]}) print(df.groupby("route").agg(parcels=("kg","size"), heaviest=("kg","max")))Produce separate total and average columns per room.
Use Lab visits 2 and 4.
Expected result
6 3.0 - Group Transformations: transform() for Standardization & Broadcast Metrics
Read the example and try it
Transform returns a result aligned with every pupil. Subtract the class mean while keeping the original row order.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"class":["A","B","A"],"n":[2,8,6]}) print((df.n - df.groupby("class").n.transform("mean")).tolist())Attach each room's mean beside each daily visit count.
Use rooms Lab, Lab, Hall with counts 2,4,8.
Expected result
[3.0, 3.0, 8.0] - Group Filtering: filter() for Discarding Small or Non-Compliant Cohorts
Read the example and try it
Group filtering keeps or removes whole groups. Here the solitary B observation disappears.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd df = pd.DataFrame({"sensor":["A","B","A"],"n":[2,8,6]}) print(df.groupby("sensor").filter(lambda batch: len(batch)>=2).n.tolist())Keep rooms with at least three observations.
Use three Lab rows and one Hall row, each with a recorded count.
Expected result
['Lab', 'Lab', 'Lab'] - Rolling & Expanding Windows: Moving Averages, EWM & Cumulative Metrics
Read the example and try it
An expanding mean uses all readings so far. A two-reading rolling mean forgets the oldest reading on day three.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([2,4,12]) print(s.expanding().mean().iloc[-1], s.rolling(2).mean().iloc[-1])Compare a rolling mean of two days with an expanding mean.
Use [2,4,9] and compare the last rolling and expanding means.
Expected result
6.5 5.0
10. Time Series, I/O Tools & Performance Optimization
- Date & Time Fundamentals: pd.to_datetime(), date_range() & DatetimeIndex
Read the example and try it
An explicit format gives an ambiguous date one interpretation. Pull out the month afterwards to check the choice.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd day = pd.to_datetime("03/04/2025", format="%d/%m/%Y") print(day.month, day.day)Flag invalid dates after parsing a mixed column.
Use 2025-01-01 and an impossible 2025-02-30.
Expected result
[False, True] - Time Series Resampling, Shifting & Frequency Conversion
Read the example and try it
Combine readings within the same calendar day before comparing days. Resampling needs a time-based index.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series([3,6,1],index=pd.to_datetime(["2025-01-01 09:00","2025-01-01 16:00","2025-01-02 09:00"])) print(s.resample("D").sum().tolist())Compare yesterday's value with today's on an ordered table.
Use three consecutive days with values 2,5,4. Compare changes after day one.
Expected result
[3.0, -1.0] - Flat File & Tabular I/O: Fast CSV, TSV & Excel Reading/Writing
Read the example and try it
Treat identifiers as text on import. A leading zero in a ticket ID is meaningful even though it adds no numeric value.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd from io import StringIO df = pd.read_csv(StringIO("id,seats\n007,4"),dtype={"id":str}) print(df.id.iloc[0])Read a tab-separated header as two fields.
Use one tab between room and visits in the header and one between Lab and 2.
Expected result
['room', 'visits'] - High-Speed Columnar Storage: Parquet, Feather & Arrow Integration
Read the example and try it
A reader can select just the needed Parquet column. This round trip uses an in-memory file and requires pyarrow or fastparquet.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd from io import BytesIO buf = BytesIO() pd.DataFrame({"room":["A"],"seats":[4]}).to_parquet(buf,index=False) buf.seek(0) print(pd.read_parquet(buf,columns=["seats"]).iat[0,0])Compare reading two selected columns with the whole table.
Write three columns room, visits, day to Parquet and read only the first two. Requires pyarrow.
Expected result
(1, 3) (1, 2) - High-Performance Pandas: PyArrow Engine, eval(), query() & Memory Tuning
Read the example and try it
Measure storage on the actual repeated labels. Categories store a vocabulary and integer codes, which can help when the vocabulary is small.
Python 3 with pandas. Run the entire example in one cell or script.
import pandas as pd s = pd.Series(["north","south"]*500) small = s.astype("category") print(s.memory_usage(deep=True) > small.memory_usage(deep=True))Measure memory before and after changing a repeated text column's dtype.
Compare deep memory for 5,000 identical object strings before and after categorizing.
Expected result
True — the category representation uses fewer bytes here.
Test a Python function with unittest
A test turns an expected behavior into a repeatable check. Keep the calculation separate from printing so a test can inspect its return value. An average of an empty list has no value in this example, so the function raises ValueError instead of silently inventing a result.
Python 3, standard library only
import unittest
def average(scores):
if not scores:
raise ValueError("scores must not be empty")
return sum(scores) / len(scores)
class AverageTests(unittest.TestCase):
def test_two_scores(self):
self.assertEqual(average([60, 80]), 70)
def test_empty_scores(self):
with self.assertRaises(ValueError):
average([])
Check the output
Both tests pass. unittest reports 2 tests and OK; elapsed time varies.Add tests for a single score and decimal scores. Use assertAlmostEqual for a floating-point comparison.
Avoid data leakage when preparing a dataset
A model evaluation is useful only when the held-out rows have not influenced training. Split the rows before learning a mean, scaler, vocabulary, or feature-selection rule. Fit those rules using training data, then reuse them unchanged on validation and test data. Time-ordered data often needs a chronological split instead of a random split.
Python 3; example uses no external packages
training = [10, None, 30]
testing = [None, 90]
observed = [x for x in training if x is not None]
mean = sum(observed) / len(observed)
def fill(values):
return [mean if x is None else x for x in values]
print(fill(training))
print(fill(testing))Check the output
[10, 20.0, 30]
[20.0, 90]Explain why including 90 when computing the replacement mean leaks information. Decide what to do if every training value is missing.
Make a small learning project reproducible
A project is reproducible when another learner can obtain the same inputs, install the required tools, and repeat your checks. Record the tool versions and exact run commands. Use synthetic or permission-cleared sample data. Keep credentials outside the repository. A successful run on your laptop is a starting point; a fresh environment is a better check.
Any language; choose the package and build tools appropriate to that course
study-project/
README.md # setup, run, test, expected result
src/ # your implementation
tests/ # behavior checks
data/ # small synthetic example data
.gitignore # caches, build output, local secretsCheck the output
This is a suggested directory layout, not a command to execute. Adapt conventions to the chosen language.Build a score summarizer. Include an empty-input test, a malformed-input test, and one example whose answer can be calculated by hand.
Check pandas keys and totals before reporting
Before trusting a summary, check that required columns exist and that identifiers obey the intended uniqueness rule. A duplicate may be a valid repeated observation or an accidental repeated import; do not drop it without deciding which interpretation applies. Use an explicit error for data validation rather than relying on assertions that can be disabled.
Python 3 with pandas
import pandas as pd
rows = pd.DataFrame({"id": [1, 2], "minutes": [20, 30]})
required = {"id", "minutes"}
if not required.issubset(rows.columns):
raise ValueError("missing required columns")
if rows["id"].isna().any() or not rows["id"].is_unique:
raise ValueError("id must be present and unique")
if rows["minutes"].isna().any() or not pd.api.types.is_numeric_dtype(rows["minutes"]):
raise ValueError("minutes must be numeric and present")
print(rows["minutes"].sum())Check the output
50Add a negative duration and decide whether it is valid. Add the corresponding explicit check before grouping or plotting.
Open the pandas learning path on DevStudyHub · Play coding games on this blog
No comments:
Post a Comment
Please do not enter any spam link in the comment box.