pandas · Lesson 1.4

Inspecting data

The first five minutes with any new dataset: types, missing values, distinct values and counts.

Practice data: Netflix files

Made up for teaching, not real Netflix data. Read a file with pd.read_csv("plans.csv"). Disclaimer

plans.csv (3 rows): plan_id, plan_name, monthly_price

ratings.csv (1,981 rows): user_id, title_id, rating, rated_at

subscriptions.csv (621 rows): subscription_id, user_id, plan_id, start_date, end_date, cancel_reason

titles.csv (220 rows): title_id, title, type, genres, release_year, duration_min, seasons, maturity_rating, country, date_added, description

users.csv (600 rows): user_id, signup_date, country, age_group, referral_source

watch_sessions.csv (6,421 rows): session_id, user_id, title_id, device, start_ts, minutes_watched

1Column types

.dtypes shows each column's type: int64 and float64 for numbers, str for text, bool for True/False. A number column read as str is a sign of messy data.

import pandas as pd

users = pd.read_csv("users.csv")
users.dtypes

Netflix data

Practice

Read titles.csv and store its column types (.dtypes) in result.

import pandas as pd

titles = pd.read_csv("titles.csv")
result = ...

Netflix data

2Missing values per column

.isna().sum() counts the missing values in every column at once. Run it on every new dataset.

import pandas as pd

users = pd.read_csv("users.csv")
users.isna().sum()

Netflix data

Practice

Store the number of missing values in each column of titles.csv in result (a Series).

import pandas as pd

titles = pd.read_csv("titles.csv")
result = ...

Netflix data

3Counting values

.value_counts() counts how often each value appears, most common first. Add normalize=True to get shares instead of counts.

import pandas as pd

users = pd.read_csv("users.csv")
users["age_group"].value_counts()

Netflix data

Practice

Store the share of users in each country in result, rounded to 3 decimals.

import pandas as pd

users = pd.read_csv("users.csv")
result = ...

Netflix data

4Distinct values

.unique() lists the distinct values and .nunique() counts them. Missing values aren't counted by .nunique() unless you ask.

import pandas as pd

users = pd.read_csv("users.csv")
print(users["referral_source"].unique())
print(users["referral_source"].nunique(), users["referral_source"].nunique(dropna=False))

Netflix data

Practice

Store the number of distinct values in every column of watch_sessions.csv in result.

import pandas as pd

sessions = pd.read_csv("watch_sessions.csv")
result = ...

Netflix data

Quick check

A column of prices shows the type str after read_csv. What's the most likely reason?