Data analysis with pandas

DataFrames and Series

A DataFrame is a table with named columns and an index labelling the rows; each column is a Series. pd.read_csv(name) loads a CSV file, using its first line as column names.

• df.shape (rows, columns), df.head(n) and df.tail(n) (first and last rows), df.info() (column types and non-missing counts), df.describe() (count, mean, std, min, quartiles, max for number columns).
• df["score"] is one column. Series methods: mean(), median(), mode(), std(), min(), max(), sum(), count(), value_counts().

This file is used below:

name,house,score
Ana,Red,71
Ben,Blue,64
Cy,Red,
Dee,Green,88
Ben,Blue,64
Eve,Blue,59

import pandas as pd
df = pd.read_csv("results.csv")
print(df.shape)
print(df.head(3))
print(df["score"].mean(), df["score"].median(), df["house"].mode()[0])
print(df["house"].value_counts().to_dict())

Output:

(6, 3)
  name house  score
0  Ana   Red   71.0
1  Ben  Blue   64.0
2   Cy   Red    NaN
69.2 64.0 Blue
{'Blue': 3, 'Red': 2, 'Green': 1}

Missing values (Cy's empty score) are read as NaN and skipped by mean() and median().

Cleaning data

• df.isna() marks missing values; df.dropna() removes rows that contain any.
• df.duplicated() marks rows that repeat an earlier row, so df[~df.duplicated()] keeps first copies only (~ means NOT).
• df.drop(columns=[...]) removes columns.
• Each of these returns a new DataFrame; assign the result.

Sorting, selecting and saving

• df.sort_values("score", ascending=False) sorts by a column.
• df.iloc[i] selects by position; df.loc[condition, column] selects by label or condition.
• Filtering: df[df["score"] > 60].
• df.to_csv(name, index=False) saves without the index column.

with open("results.csv", "w") as f:
    f.write('name,house,score\nAna,Red,71\nBen,Blue,64\nCy,Red,\nDee,Green,88\nBen,Blue,64\nEve,Blue,59\n')

import pandas as pd
df = pd.read_csv("results.csv")
print(df.duplicated().sum(), df["score"].isna().sum())
clean = df[~df.duplicated()].dropna()
print(clean.shape)
top = clean.sort_values("score", ascending=False)
print(top.iloc[0]["name"], top.loc[top["house"] == "Blue", "name"].tolist())
clean.drop(columns=["house"]).to_csv("clean.csv", index=False)
print(open("clean.csv").read().splitlines()[:2])

Output:

1 1
(4, 3)
Dee ['Ben', 'Eve']
['name,score', 'Ana,71.0']

Dates, other separators and what the PLS lists

• df.drop_duplicates() removes repeated rows in one step (the same result as df[~df.duplicated()]).
• read_csv(name, parse_dates=["date"], dayfirst=True) reads a column as dates, treating 03/11/2026 as 3 November; pd.to_datetime(df["date"], dayfirst=True) converts a column after loading. .dt.strftime(format) turns dates back into text.
• to_csv takes sep for another separator (such as a tab, "\t") and date_format; read_csv takes sep, header and skiprows.

import pandas as pd
with open("visits.csv", "w") as f:
    f.write("date,visitors\n03/11/2026,120\n04/11/2026,95\n04/11/2026,95\n05/11/2026,140\n")
df = pd.read_csv("visits.csv", parse_dates=["date"], dayfirst=True)
df = df.drop_duplicates()
print(len(df), df["date"].dt.strftime("%Y-%m-%d").tolist())
df.to_csv("visits.tsv", sep="\t", index=False, date_format="%Y-%m-%d")
print(open("visits.tsv").read().splitlines())

Output:

3 ['2026-11-03', '2026-11-04', '2026-11-05']
['date\tvisitors', '2026-11-03\t120', '2026-11-04\t95', '2026-11-05\t140']

The PLS and real pandas. The PLS spells the option dayFirst, but pandas only accepts lower-case dayfirst (dayFirst raises TypeError). The PLS also shows a df.to_datetime() method, which does not exist: use pd.to_datetime(column). isna(), unique() and idxmax(), used in this lesson, are standard pandas but not in the PLS.

Worked example

Using results.csv, find the mean score of each house after removing duplicates and missing values.

1. ```python with open("results.csv", "w") as f: f.write('name,house,score\nAna,Red,71\nBen,Blue,64\nCy,Red,\nDee,Green,88\nBen,Blue,64\nEve,Blue,59\n') import pandas as pd df = pd.read_csv("results.csv") clean = df[~df.duplicated()].dropna() for house in sorted(clean["house"].unique()): scores = clean.loc[clean["house"] == house, "score"] print(house, scores.mean()) ```
2. Output: ``` Blue 61.5 Green 88.0 Red 71.0 ```

Answer: Clean first so the repeated Ben row does not count twice, then filter by house with loc and take each mean.

Check yourself

1. This program uses the results.csv file from the lesson. What does it print? ```python with open("results.csv", "w") as f: f.write('name,house,score\nAna,Red,71\nBen,Blue,64\nCy,Red,\nDee,Green,88\nBen,Blue,64\nEve,Blue,59\n') import pandas as pd df = pd.read_csv("results.csv") print(df.shape[0]) ```
   Show answer

   6. The file has a header and `6` data rows.

2. Using results.csv from the lesson, what does this print? ```python with open("results.csv", "w") as f: f.write('name,house,score\nAna,Red,71\nBen,Blue,64\nCy,Red,\nDee,Green,88\nBen,Blue,64\nEve,Blue,59\n') import pandas as pd df = pd.read_csv("results.csv") print(df["score"].mean()) ```
   Show answer

   69.2. The missing score is skipped: (71 + 64 + 88 + 64 + 59) / 5 = `69.2`.

3. Using results.csv from the lesson, how many rows remain? ```python with open("results.csv", "w") as f: f.write('name,house,score\nAna,Red,71\nBen,Blue,64\nCy,Red,\nDee,Green,88\nBen,Blue,64\nEve,Blue,59\n') import pandas as pd df = pd.read_csv("results.csv") print(len(df[~df.duplicated()].dropna())) ```
   Show answer

   4. Removing the repeated Ben row leaves 5 rows; removing Cy's row with a missing score leaves `4`.

4. Using results.csv from the lesson, what does this print? ```python with open("results.csv", "w") as f: f.write('name,house,score\nAna,Red,71\nBen,Blue,64\nCy,Red,\nDee,Green,88\nBen,Blue,64\nEve,Blue,59\n') import pandas as pd df = pd.read_csv("results.csv") print(df["house"].value_counts().idxmax()) ```
   Show answer

   Blue. value_counts counts each house: Blue appears 3 times, the most. Output: `Blue`

5. What does this program print? ```python import pandas as pd df = pd.DataFrame({"name": ["Ana", "Ben", "Ana", "Ana"], "score": [71, 64, 71, 80]}) print(len(df.drop_duplicates())) ```
   Show answer

   3. Only the third row repeats an earlier row exactly (Ana, 71). Ana with 80 is a different row, so `3` rows remain.

6. What is the difference between `df.iloc[2]` and `df.loc[2]`?
   • They are always identical
   • iloc selects by position; loc selects by index label (which may differ after sorting or filtering)
   • loc selects columns and iloc selects rows
   • iloc selects the first 2 rows
   Show answer

   iloc selects by position; loc selects by index label (which may differ after sorting or filtering). After rows are dropped or sorted, the label 2 may no longer be at position 2.

7. Which method gives the count, mean, standard deviation, minimum, quartiles and maximum of each numeric column?
   • `df.info()`
   • `df.head()`
   • `df.value_counts()`
   • `df.describe()`
   Show answer

   `df.describe()`. describe() produces summary statistics; info() shows column types and non-missing counts.

8. Why is `index=False` passed to `to_csv`?
   • So the file is not overwritten
   • So missing values are removed
   • So the row index is not written as an extra unnamed first column
   • So the column names are left out
   Show answer

   So the row index is not written as an extra unnamed first column. Without it, pandas writes the index as a column, which appears when the file is read back.

Practise with instant feedback Print this lesson

Sources

• Pearson Edexcel International Advanced Subsidiary/Advanced Level in Computer Science specification, Issue 1 (March 2026)
• Pearson Edexcel IAL Computer Science Unit 4 Programming Language Subset (PLS)
• Pearson Edexcel IAL Computer Science sample assessment materials (Unit 3 and Unit 4)

Last updated 8 November 2026. Spotted a mistake? Tell us.