import pandas as pd8 pandas Fundamentals
8.1 Purpose
pandas is one of the main Python libraries for working with tabular data.
The two most important objects are:
Series: one-dimensional labeled data.DataFrame: two-dimensional tabular data organized in rows and columns.
Import pandas with the conventional alias:
8.2 Create a DataFrame
A DataFrame can be created from a dictionary where each key represents a column.
df = pd.DataFrame({
"city": ["Puebla", "Oaxaca", "Mérida", "Puebla", "Cancún"],
"year": [2024, 2024, 2025, 2025, 2024],
"value": [18.2, 15.7, 21.4, 19.8, 17.5],
})
df8.3 Inspect data
Before analyzing a dataset, inspect its structure and contents.
8.3.1 First rows
df.head()8.3.2 Last rows
df.tail()8.3.3 Dimensions
df.shape8.3.4 Column names
df.columns8.3.5 Data types
df.dtypes8.3.6 General information
df.info()8.3.7 Descriptive statistics
df.describe()For categorical columns:
df.describe(include="object")9 Selecting Data
9.1 Select one column
Selecting one column with a single pair of brackets returns a Series.
df["city"]Equivalent:
city = df["city"]9.2 Select multiple columns
Use a list of column names.
selected = df[["city", "value"]]
selectedThis returns a DataFrame.
9.3 Select rows and columns with .loc
.loc selects data using labels and Boolean conditions.
df.loc[:, ["city", "value"]]The syntax is:
df.loc[rows, columns]For example:
df.loc[df["value"] >= 18, ["city", "value"]]10 Filtering Rows
Filtering normally uses a Boolean condition.
df["value"] >= 18This produces a Boolean Series:
True
False
True
True
FalseThe Boolean result can then be used to select rows.
filtered = df[df["value"] >= 18]
filteredA common alternative is .loc:
filtered = df.loc[df["value"] >= 18]
filtered10.1 Filter numeric values
10.1.1 Greater than
df.loc[df["value"] > 18]10.1.2 Greater than or equal to
df.loc[df["value"] >= 18]10.1.3 Less than
df.loc[df["value"] < 18]10.1.4 Equal to
df.loc[df["year"] == 2024]10.1.5 Not equal to
df.loc[df["year"] != 2024]10.2 Filter between two values
Use .between().
df.loc[df["value"].between(17, 20)]By default, both limits are included.
This is similar to:
df.loc[
(df["value"] >= 17) &
(df["value"] <= 20)
]11 Filter with isin()
.isin() checks whether each value belongs to a collection of allowed values.
For example, select Puebla and Mérida:
df.loc[df["city"].isin(["Puebla", "Mérida"])]Conceptually:
city isin(...)
-------------------
Puebla True
Oaxaca False
Mérida True
Puebla True
Cancún FalseThis is especially useful when filtering several categories.
Instead of writing:
df.loc[
(df["city"] == "Puebla") |
(df["city"] == "Mérida")
]you can write:
df.loc[df["city"].isin(["Puebla", "Mérida"])]11.1 Exclude values with ~ and isin()
The ~ operator means NOT for Boolean conditions.
df.loc[~df["city"].isin(["Puebla", "Mérida"])]This means:
Keep rows where
cityis not Puebla or Mérida.
12 Multiple Conditions
12.1 AND condition
Use &.
For example:
df.loc[
(df["year"] == 2024) &
(df["value"] >= 18)
]Both conditions must be True.
12.2 OR condition
Use |.
df.loc[
(df["city"] == "Puebla") |
(df["city"] == "Mérida")
]At least one condition must be True.
12.3 NOT condition
Use ~.
df.loc[~(df["year"] == 2024)]A simpler version in this case is:
df.loc[df["year"] != 2024]12.4 Important: use parentheses
When combining conditions, put each condition inside parentheses.
Correct:
df.loc[
(df["year"] == 2024) &
(df["value"] >= 18)
]Avoid:
df.loc[df["year"] == 2024 & df["value"] >= 18]13 String Filters
pandas provides .str methods for filtering text columns.
13.1 Contains text
df.loc[df["city"].str.contains("a")]For case-insensitive matching:
df.loc[df["city"].str.contains("a", case=False)]13.2 Starts with
df.loc[df["city"].str.startswith("P")]13.3 Ends with
df.loc[df["city"].str.endswith("a")]14 Missing Values
Missing values are usually represented as NaN.
14.1 Find missing values
df["value"].isna()Filter rows with missing values:
df.loc[df["value"].isna()]14.2 Find non-missing values
df.loc[df["value"].notna()]15 Filter and Select Columns Together
One of the most common pandas patterns is:
df.loc[condition, columns]For example:
df.loc[
df["city"].isin(["Puebla", "Mérida"]),
["city", "year", "value"]
]Or with multiple conditions:
df.loc[
(df["year"] == 2025) &
(df["value"] >= 19),
["city", "value"]
]16 Useful Filter Patterns
16.1 One category
df.loc[df["city"] == "Puebla"]16.2 Several categories
df.loc[df["city"].isin(["Puebla", "Oaxaca"])]16.3 Exclude categories
df.loc[~df["city"].isin(["Puebla", "Oaxaca"])]16.4 Numeric threshold
df.loc[df["value"] >= 18]16.5 Numeric range
df.loc[df["value"].between(17, 20)]16.6 Two conditions
df.loc[
(df["year"] == 2024) &
(df["value"] >= 18)
]16.7 Text contains
df.loc[df["city"].str.contains("a", case=False)]16.8 Missing values
df.loc[df["value"].isna()]17 Things to Remember
df["column"]usually returns aSeries.df[["column1", "column2"]]returns aDataFrame..loc[rows, columns]is a clear way to filter rows and select columns.==,!=,>,<,>=, and<=create Boolean conditions.- Use
&for AND. - Use
|for OR. - Use
~for NOT. - Put individual conditions inside parentheses.
.isin()is useful when matching several possible values..between()is useful for numeric ranges..str.contains()is useful for text searches..isna()and.notna()identify missing and non-missing values.- Inspect column types with
df.dtypesordf.info()before performing calculations.