Pandas (software)

Data model

Pandas is built around data structures called Series and DataFrames. Data for these collections can be imported from various file formats such as comma-separated values, JSON, Parquet, SQL database tables or queries, and Microsoft Excel.

Series

A Series is a one-dimensional array-like object that stores a sequence of values together with an associated set of labels, called an index. It is built on top of NumPy's array and affords many similar functionalities, but instead of using implicit integer positions, a Series allows explicit index labels of many data types.

A Series can be created from Python lists, dictionaries, or NumPy arrays. If no index is provided, pandas automatically assigns a default integer index ranging from 0 to n-1, where n is the number of items in the Series. A simple example with customized labels is:

<span id="mwdQ"></span><span class="kn" id="mwdg">import</span><span class="w" id="mwdw"> </span><span class="nn" id="mweA">pandas</span><span class="w" id="mweQ"> </span><span class="k" id="mweg">as</span><span class="w" id="mwew"> </span><span class="nn" id="mwfA">pd</span> <span class="n" id="mwfQ">ser</span> <span class="o" id="mwfg">=</span> <span class="n" id="mwfw">pd</span><span class="o" id="mwgA">.</span><span class="n" id="mwgQ">Series</span><span class="p" id="mwgg">([</span><span class="s1" id="mwgw">'a'</span><span class="p" id="mwhA">,</span> <span class="s1" id="mwhQ">'b'</span><span class="p" id="mwhg">,</span> <span class="s1" id="mwhw">'c'</span><span class="p" id="mwiA">],</span> <span class="n" id="mwiQ">index</span><span class="o" id="mwig">=</span><span class="p" id="mwiw">[</span><span class="s2" id="mwjA">"x"</span><span class="p" id="mwjQ">,</span> <span class="s2" id="mwjg">"y"</span><span class="p" id="mwjw">,</span> <span class="s2" id="mwkA">"z"</span><span class="p" id="mwkQ">])</span>

To access a value or list of values from a Series, use its index or list of indices:

<span id="mwmQ"></span><span class="n" id="mwmg">ser</span><span class="p" id="mwmw">[</span><span class="s1" id="mwnA">'x'</span><span class="p" id="mwnQ">]</span> <span class="n" id="mwng">ser</span><span class="p" id="mwnw">[[</span><span class="s1" id="mwoA">'x'</span><span class="p" id="mwoQ">,</span> <span class="s1" id="mwog">'z'</span><span class="p" id="mwow">]]</span>

Series can be used arithmetically, as in the statement series_3 = series_1 + series_2. This will align data points with corresponding index values in series_1 and series_2 (similar to a join in relational algebra), then add them together to produce new values in series_3.

A Series has various attributes, such as name (Series name), dtype (data type of values), shape (number of rows), values, and index. They can be used in many of the same operations as NumPy arrays, with additional methods for reindexing, label-based selection, and handling missing data.

DataFrame

A DataFrame is a two-dimensional, tabular data structure with labeled rows and columns. Each column is stored internally as a Series and may hold a different data type (numeric, string, boolean, etc.). DataFrames can be created by a variety of means, including dictionaries of lists, NumPy arrays, and external files such as CSV or Excel spreadsheets:

<span id="mwzw"></span><span class="n" id="mw0A">df1</span> <span class="o" id="mw0Q">=</span> <span class="n" id="mw0g">pd</span><span class="o" id="mw0w">.</span><span class="n" id="mw1A">Series</span><span class="p" id="mw1Q">([</span><span class="s1" id="mw1g">'A'</span><span class="p" id="mw1w">,</span> <span class="s1" id="mw2A">'B'</span><span class="p" id="mw2Q">,</span> <span class="s1" id="mw2g">'C'</span><span class="p" id="mw2w">])</span><span class="o" id="mw3A">.</span><span class="n" id="mw3Q">to_frame</span><span class="p" id="mw3g">()</span> <span class="n" id="mw3w">df2</span> <span class="o" id="mw4A">=</span> <span class="n" id="mw4Q">pd</span><span class="o" id="mw4g">.</span><span class="n" id="mw4w">DataFrame</span><span class="p" id="mw5A">({</span><span class="s2" id="mw5Q">"grade"</span><span class="p" id="mw5g">:</span> <span class="p" id="mw5w">[</span><span class="s2" id="mw6A">"A"</span><span class="p" id="mw6Q">,</span> <span class="s2" id="mw6g">"B"</span><span class="p" id="mw6w">,</span> <span class="s2" id="mw7A">"C"</span><span class="p" id="mw7Q">],</span> <span class="s2" id="mw7g">"score"</span><span class="p" id="mw7w">:</span> <span class="p" id="mw8A">[</span><span class="mi" id="mw8Q">100</span><span class="p" id="mw8g">,</span> <span class="mi" id="mw8w">80</span><span class="p" id="mw9A">,</span> <span class="mi" id="mw9Q">60</span><span class="p" id="mw9g">]})</span> <span class="n" id="mw9w">df3</span> <span class="o" id="mw-A">=</span> <span class="n" id="mw-Q">pd</span><span class="o" id="mw-g">.</span><span class="n" id="mw-w">read_csv</span><span class="p" id="mw_A">(</span><span class="s1" id="mw_Q">'path/classgrades.csv'</span><span class="p" id="mw_g">)</span>

To retrieve a DataFrame column as a Series, use either 1) the index (dict-like notation) or 2) the name of column if the name is a valid Python identifier (attribute-like access). DataFrames support operations such as column assignment, row and column deletion, label-based indexing with loc, position-based indexing with iloc, reshaping, grouping, and joining. Merge operations implement a subset of relational algebra and allow one-to-one, many-to-one, and many-to-many joins.

Some common attributes of a DataFrame include dtypes (data type of each column), shape (dimensions of the DataFrame returned as a tuple with form (number of rows, number of columns)), index/columns (labels of the DataFrame's rows/columns, respectively, returned as an Index object), values (data in the DataFrame returned as a 2D array), and empty (returns True if the DataFrame is empty).

Index

Index objects hold metadata for Series and Dataframe objects, such as axis labels and names, and are automatically created from input data. By default, a pandas index is a series of integers ascending from 0, similar to the indices of Python arrays. However, indices can also use any NumPy data type, including floating point, timestamps, or strings. Indices are also immutable, which allows them to be safely shared across multiple objects.

pandas' syntax for mapping index values to relevant data is the same syntax Python uses to map dictionary keys to values. For example, if s is a Series, s['a'] will return the data point at index a. Unlike dictionary keys, index values are not guaranteed to be unique. If a Series uses the index value a for multiple data points, then s['a'] will instead return a new Series containing all matching values. A DataFrame's column names are stored and implemented identically to an index. As such, a DataFrame can be thought of as having two indices: one column-based and one row-based. Because column names are stored as an index, these are not required to be unique.

If data is a Series, then data['a'] returns all values with the index value of a. However, if data is a DataFrame, then data['a'] returns all values in the column(s) named a. To avoid this ambiguity, Pandas supports the syntax data.loc['a'] as an alternative way to filter using the index. Pandas also supports the syntax data.iloc[n], which always takes an integer n and returns the nth value, counting from 0. This allows a user to act as though the index is an array-like sequence of integers, regardless of how it is actually defined.

pandas also supports hierarchical indices with multiple values per data point through the "MultiIndex" class. MultiIndex objects allow a single DataFrame to represent multiple dimensions, similar to a pivot table in Microsoft Excel, where each level can optionally carry its own unique name. In practice, data with more than 2 dimensions is often represented using DataFrames with hierarchical indices, instead of the higher-dimension Panel and Panel4D data structures.

Functionality

pandas supports a variety of indexing and subsetting techniques, allowing data to be selected by label, index, or Boolean conditions. For example, df[df['col1'] > 5] will return all rows in the DataFrame df for which the value of the column col1 exceeds 5. The library also implements grouping operations based on the split-apply-combine approach, enabling users to aggregate, transform, or restructure data according to column values or functions applied to index labels. For example, df['col1'].groupby(df['col2']) groups the data in 'col1' by their values in 'col2', while df.groupby(lambda i: i % 2) groups all data in the whole DataFrame by whether their index is even.

The library also provides extensive tools for transforming, filtering and summarizing data. Users may apply arbitrary functions to Series and DataFrames, and because the library is built on top of Numpy, most NumPy functions can be applied directly to pandas objects as well. The library also includes built-in operations for arithmetic operations, string processing, and descriptive statistics such as mean, median, and standard deviation. These built-in functions are designed to handle missing data, usually represented by the floating-point value NaN.

In addition, pandas includes tools for reorganizing data into different structural formats, with methods that can reshape tabular data between "wide" and "long" formats and pivot values based on column labels. pandas also implements a flexible set of relational operations for combining datasets. For instance, merge() links row in DataFrames based on one or more shared keys or indices, supporting one-to-one, one-to-many, and many-to-many relationships in a manner analogous to join operations in relational databases like SQL. DataFrames can also be concatenated or stacked together along an axis through the concat() method, and overlapping data can be further spliced together using combine_first() to fill in missing values.

Furthermore, the library includes specialized support for working with time-series data. Features include the ability to interpolate values and filter using a range of timestamps, such as data['1/1/2023':'2/2/2023'] , which will return all dates between January 1 and February 2. Missing values in time-series data are represented by a dedicated NaT (Not a Timestamp) object, instead of the NaN value it uses elsewhere.

Criticisms

Pandas has been criticized for its inefficiency. The entire dataset must be loaded in RAM, and the library does not optimize query plans or support parallel computing across multiple cores. Wes McKinney, the creator of Pandas, has recommended Apache Arrow as an alternative to address these performance concerns and other limitations. Otherwise, he says, "my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset".

Examples

Pandas is customarily imported as pd.

<span id="mwAcU"></span><span class="kn" id="mwAcY">import</span><span class="w" id="mwAcc"> </span><span class="nn" id="mwAcg">numpy</span><span class="w" id="mwAck"> </span><span class="k" id="mwAco">as</span><span class="w" id="mwAcs"> </span><span class="nn" id="mwAcw">np</span> <span class="kn" id="mwAc0">import</span><span class="w" id="mwAc4"> </span><span class="nn" id="mwAc8">pandas</span><span class="w" id="mwAdA"> </span><span class="k" id="mwAdE">as</span><span class="w" id="mwAdI"> </span><span class="nn" id="mwAdM">pd</span>

Example 1: Food & Nutrition

Here's a fake dataset on the nutritional value of various food items:

<span id="mwAdg"></span><span class="n" id="mwAdk">df</span> <span class="o" id="mwAdo">=</span> <span class="n" id="mwAds">pd</span><span class="o" id="mwAdw">.</span><span class="n" id="mwAd0">DataFrame</span><span class="p" id="mwAd4">({</span> <span class="s2" id="mwAd8">"food"</span><span class="p" id="mwAeA">:</span> <span class="p" id="mwAeE">[</span><span class="s2" id="mwAeI">"Apple"</span><span class="p" id="mwAeM">,</span> <span class="s2" id="mwAeQ">"Banana"</span><span class="p" id="mwAeU">,</span> <span class="s2" id="mwAeY">"Almonds"</span><span class="p" id="mwAec">,</span> <span class="s2" id="mwAeg">"Broccoli"</span><span class="p" id="mwAek">,</span> <span class="s2" id="mwAeo">"Salmon"</span><span class="p" id="mwAes">,</span> <span class="s2" id="mwAew">"Oatmeal"</span><span class="p" id="mwAe0">],</span> <span class="s2" id="mwAe4">"calories"</span><span class="p" id="mwAe8">:</span> <span class="p" id="mwAfA">[</span><span class="mi" id="mwAfE">95</span><span class="p" id="mwAfI">,</span> <span class="mi" id="mwAfM">105</span><span class="p" id="mwAfQ">,</span> <span class="mi" id="mwAfU">164</span><span class="p" id="mwAfY">,</span> <span class="mi" id="mwAfc">55</span><span class="p" id="mwAfg">,</span> <span class="mi" id="mwAfk">208</span><span class="p" id="mwAfo">,</span> <span class="mi" id="mwAfs">158</span><span class="p" id="mwAfw">],</span> <span class="s2" id="mwAf0">"protein_g"</span><span class="p" id="mwAf4">:</span> <span class="p" id="mwAf8">[</span><span class="mf" id="mwAgA">0.5</span><span class="p" id="mwAgE">,</span> <span class="mf" id="mwAgI">1.3</span><span class="p" id="mwAgM">,</span> <span class="mf" id="mwAgQ">6.0</span><span class="p" id="mwAgU">,</span> <span class="mf" id="mwAgY">3.7</span><span class="p" id="mwAgc">,</span> <span class="mf" id="mwAgg">22.0</span><span class="p" id="mwAgk">,</span> <span class="mf" id="mwAgo">6.0</span><span class="p" id="mwAgs">],</span> <span class="s2" id="mwAgw">"carb_g"</span><span class="p" id="mwAg0">:</span> <span class="p" id="mwAg4">[</span><span class="mf" id="mwAg8">25.1</span><span class="p" id="mwAhA">,</span> <span class="mf" id="mwAhE">27.0</span><span class="p" id="mwAhI">,</span> <span class="mf" id="mwAhM">6.4</span><span class="p" id="mwAhQ">,</span> <span class="mf" id="mwAhU">11.2</span><span class="p" id="mwAhY">,</span> <span class="mf" id="mwAhc">0.5</span><span class="p" id="mwAhg">,</span> <span class="mf" id="mwAhk">29.3</span><span class="p" id="mwAho">],</span> <span class="s2" id="mwAhs">"fat_g"</span><span class="p" id="mwAhw">:</span> <span class="p" id="mwAh0">[</span><span class="mf" id="mwAh4">0.3</span><span class="p" id="mwAh8">,</span> <span class="mf" id="mwAiA">0.4</span><span class="p" id="mwAiE">,</span> <span class="mf" id="mwAiI">14.0</span><span class="p" id="mwAiM">,</span> <span class="mf" id="mwAiQ">0.6</span><span class="p" id="mwAiU">,</span> <span class="mf" id="mwAiY">13.0</span><span class="p" id="mwAic">,</span> <span class="mf" id="mwAig">3.2</span><span class="p" id="mwAik">],</span> <span class="s2" id="mwAio">"fiber_g"</span><span class="p" id="mwAis">:</span> <span class="p" id="mwAiw">[</span><span class="mf" id="mwAi0">4.4</span><span class="p" id="mwAi4">,</span> <span class="mf" id="mwAi8">3.1</span><span class="p" id="mwAjA">,</span> <span class="mf" id="mwAjE">3.5</span><span class="p" id="mwAjI">,</span> <span class="mf" id="mwAjM">2.4</span><span class="p" id="mwAjQ">,</span> <span class="mf" id="mwAjU">0.0</span><span class="p" id="mwAjY">,</span> <span class="mf" id="mwAjc">4.0</span><span class="p" id="mwAjg">],</span> <span class="s2" id="mwAjk">"category"</span><span class="p" id="mwAjo">:</span> <span class="p" id="mwAjs">[</span><span class="s2" id="mwAjw">"Fruit"</span><span class="p" id="mwAj0">,</span> <span class="s2" id="mwAj4">"Fruit"</span><span class="p" id="mwAj8">,</span> <span class="s2" id="mwAkA">"Nuts"</span><span class="p" id="mwAkE">,</span> <span class="s2" id="mwAkI">"Vegetable"</span><span class="p" id="mwAkM">,</span> <span class="s2" id="mwAkQ">"Meat"</span><span class="p" id="mwAkU">,</span> <span class="s2" id="mwAkY">"Grain"</span><span class="p" id="mwAkc">]</span> <span class="p" id="mwAkg">})</span> <span class="n" id="mwAkk">df</span>

<span id="mwAkw"></span> <span class="n" id="mwAk0">food</span> <span class="n" id="mwAk4">calories</span> <span class="n" id="mwAk8">protein_g</span> <span class="n" id="mwAlA">carb_g</span> <span class="n" id="mwAlE">fat_g</span> <span class="n" id="mwAlI">fiber_g</span> <span class="n" id="mwAlM">category</span> <span class="mi" id="mwAlQ">0</span> <span class="n" id="mwAlU">Apple</span> <span class="mi" id="mwAlY">95</span> <span class="mf" id="mwAlc">0.5</span> <span class="mf" id="mwAlg">25.1</span> <span class="mf" id="mwAlk">0.3</span> <span class="mf" id="mwAlo">4.4</span> <span class="n" id="mwAls">Fruit</span> <span class="mi" id="mwAlw">1</span> <span class="n" id="mwAl0">Banana</span> <span class="mi" id="mwAl4">105</span> <span class="mf" id="mwAl8">1.3</span> <span class="mf" id="mwAmA">27.0</span> <span class="mf" id="mwAmE">0.4</span> <span class="mf" id="mwAmI">3.1</span> <span class="n" id="mwAmM">Fruit</span> <span class="mi" id="mwAmQ">2</span> <span class="n" id="mwAmU">Almonds</span> <span class="mi" id="mwAmY">164</span> <span class="mf" id="mwAmc">6.0</span> <span class="mf" id="mwAmg">6.4</span> <span class="mf" id="mwAmk">14.0</span> <span class="mf" id="mwAmo">3.5</span> <span class="n" id="mwAms">Nuts</span> <span class="mi" id="mwAmw">3</span> <span class="n" id="mwAm0">Broccoli</span> <span class="mi" id="mwAm4">55</span> <span class="mf" id="mwAm8">3.7</span> <span class="mf" id="mwAnA">11.2</span> <span class="mf" id="mwAnE">0.6</span> <span class="mf" id="mwAnI">2.4</span> <span class="n" id="mwAnM">Vegetable</span> <span class="mi" id="mwAnQ">4</span> <span class="n" id="mwAnU">Salmon</span> <span class="mi" id="mwAnY">208</span> <span class="mf" id="mwAnc">22.0</span> <span class="mf" id="mwAng">0.5</span> <span class="mf" id="mwAnk">13.0</span> <span class="mf" id="mwAno">0.0</span> <span class="n" id="mwAns">Meat</span> <span class="mi" id="mwAnw">5</span> <span class="n" id="mwAn0">Oatmeal</span> <span class="mi" id="mwAn4">158</span> <span class="mf" id="mwAn8">6.0</span> <span class="mf" id="mwAoA">29.3</span> <span class="mf" id="mwAoE">3.2</span> <span class="mf" id="mwAoI">4.0</span> <span class="n" id="mwAoM">Grain</span>

Some possible manipulations and analyses that can be performed:

1. Compute descriptive statistics:

   <span id="mwAok"></span><span class="n" id="mwAoo">df</span><span class="o" id="mwAos">.</span><span class="n" id="mwAow">describe</span><span class="p" id="mwAo0">()</span>

   This provides summaries across numeric columns, including count, mean, standard deviation, and min/max values.
2. Select specific columns:

   <span id="mwApk"></span><span class="n" id="mwApo">df</span><span class="p" id="mwAps">[[</span><span class="s2" id="mwApw">"food"</span><span class="p" id="mwAp0">,</span> <span class="s2" id="mwAp4">"calories"</span><span class="p" id="mwAp8">]]</span>

3. Sort foods by calories (descending order):

   <span id="mwAqM"></span><span class="n" id="mwAqQ">df</span><span class="o" id="mwAqU">.</span><span class="n" id="mwAqY">sort_values</span><span class="p" id="mwAqc">(</span><span class="s2" id="mwAqg">"calories"</span><span class="p" id="mwAqk">,</span> <span class="n" id="mwAqo">ascending</span><span class="o" id="mwAqs">=</span><span class="kc" id="mwAqw">False</span><span class="p" id="mwAq0">)</span>

   Note: Any missing values are automatically sorted to the end of the Series.
4. Find nutrient-dense foods (> 3g protein + > 3g fat):

   <span id="mwArU"></span><span class="n" id="mwArY">df</span><span class="p" id="mwArc">[(</span><span class="n" id="mwArg">df</span><span class="p" id="mwArk">[</span><span class="s2" id="mwAro">"protein_g"</span><span class="p" id="mwArs">]</span> <span class="o" id="mwArw">></span> <span class="mi" id="mwAr0">3</span><span class="p" id="mwAr4">)</span> <span class="o" id="mwAr8">&</span> <span class="p" id="mwAsA">(</span><span class="n" id="mwAsE">df</span><span class="p" id="mwAsI">[</span><span class="s2" id="mwAsM">"fat_g"</span><span class="p" id="mwAsQ">]</span> <span class="o" id="mwAsU">></span> <span class="mi" id="mwAsY">3</span><span class="p" id="mwAsc">)]</span>

5. Group items by category and calculate average macros:

   <span id="mwAss"></span><span class="n" id="mwAsw">df</span><span class="o" id="mwAs0">.</span><span class="n" id="mwAs4">groupby</span><span class="p" id="mwAs8">(</span><span class="s2" id="mwAtA">"category"</span><span class="p" id="mwAtE">)[[</span><span class="s2" id="mwAtI">"carb_g"</span><span class="p" id="mwAtM">,</span> <span class="s2" id="mwAtQ">"fat_g"</span><span class="p" id="mwAtU">,</span> <span class="s2" id="mwAtY">"protein_g"</span><span class="p" id="mwAtc">]]</span><span class="o" id="mwAtg">.</span><span class="n" id="mwAtk">mean</span><span class="p" id="mwAto">()</span>

   This returns a GroupBy object that is analogous to a collection of DataFrames.
6. Find the highest-calorie food per category:

   <span id="mwAuI"></span><span class="n" id="mwAuM">df</span><span class="o" id="mwAuQ">.</span><span class="n" id="mwAuU">loc</span><span class="p" id="mwAuY">[</span><span class="n" id="mwAuc">df</span><span class="o" id="mwAug">.</span><span class="n" id="mwAuk">groupby</span><span class="p" id="mwAuo">(</span><span class="s2" id="mwAus">"category"</span><span class="p" id="mwAuw">)[</span><span class="s2" id="mwAu0">"protein_g"</span><span class="p" id="mwAu4">]</span><span class="o" id="mwAu8">.</span><span class="n" id="mwAvA">idxmax</span><span class="p" id="mwAvE">()]</span>

   idxmax() returns the index value of the maximum value. idxmin() performs an analogous function with the minimum value.
7. Create new columns (macronutrient ratio as a percentage of total calories):

   <span id="mwAvs"></span><span class="n" id="mwAvw">df</span><span class="p" id="mwAv0">[</span><span class="s2" id="mwAv4">"carb_pct"</span><span class="p" id="mwAv8">]</span> <span class="o" id="mwAwA">=</span> <span class="n" id="mwAwE">df</span><span class="p" id="mwAwI">[</span><span class="s2" id="mwAwM">"carb_g"</span><span class="p" id="mwAwQ">]</span> <span class="o" id="mwAwU">*</span> <span class="mi" id="mwAwY">4</span> <span class="o" id="mwAwc">/</span> <span class="n" id="mwAwg">df</span><span class="p" id="mwAwk">[</span><span class="s2" id="mwAwo">"calories"</span><span class="p" id="mwAws">]</span> <span class="o" id="mwAww">*</span> <span class="mi" id="mwAw0">100</span> <span class="n" id="mwAw4">df</span><span class="p" id="mwAw8">[</span><span class="s2" id="mwAxA">"fat_pct"</span><span class="p" id="mwAxE">]</span> <span class="o" id="mwAxI">=</span> <span class="n" id="mwAxM">df</span><span class="p" id="mwAxQ">[</span><span class="s2" id="mwAxU">"fat_g"</span><span class="p" id="mwAxY">]</span> <span class="o" id="mwAxc">*</span> <span class="mi" id="mwAxg">9</span> <span class="o" id="mwAxk">/</span> <span class="n" id="mwAxo">df</span><span class="p" id="mwAxs">[</span><span class="s2" id="mwAxw">"calories"</span><span class="p" id="mwAx0">]</span> <span class="o" id="mwAx4">*</span> <span class="mi" id="mwAx8">100</span> <span class="n" id="mwAyA">df</span><span class="p" id="mwAyE">[</span><span class="s2" id="mwAyI">"protein_pct"</span><span class="p" id="mwAyM">]</span> <span class="o" id="mwAyQ">=</span> <span class="n" id="mwAyU">df</span><span class="p" id="mwAyY">[</span><span class="s2" id="mwAyc">"protein_g"</span><span class="p" id="mwAyg">]</span> <span class="o" id="mwAyk">*</span> <span class="mi" id="mwAyo">4</span> <span class="o" id="mwAys">/</span> <span class="n" id="mwAyw">df</span><span class="p" id="mwAy0">[</span><span class="s2" id="mwAy4">"calories"</span><span class="p" id="mwAy8">]</span> <span class="o" id="mwAzA">*</span> <span class="mi" id="mwAzE">100</span>

8. Handle missing data:

   <span id="mwAzU"></span><span class="n" id="mwAzY">df</span><span class="o" id="mwAzc">.</span><span class="n" id="mwAzg">loc</span><span class="p" id="mwAzk">[</span><span class="mi" id="mwAzo">2</span><span class="p" id="mwAzs">,</span> <span class="s2" id="mwAzw">"fiber_g"</span><span class="p" id="mwAz0">]</span> <span class="o" id="mwAz4">=</span> <span class="kc" id="mwAz8">None</span> <span class="c1" id="mwA0A"># Simulate a missing value</span> <span class="n" id="mwA0E">df</span><span class="p" id="mwA0I">[</span><span class="s2" id="mwA0M">"fiber_g"</span><span class="p" id="mwA0Q">]</span><span class="o" id="mwA0U">.</span><span class="n" id="mwA0Y">fillna</span><span class="p" id="mwA0c">(</span><span class="n" id="mwA0g">df</span><span class="p" id="mwA0k">[</span><span class="s2" id="mwA0o">"fiber_g"</span><span class="p" id="mwA0s">]</span><span class="o" id="mwA0w">.</span><span class="n" id="mwA00">mean</span><span class="p" id="mwA04">())</span>

   This replaces the None value with NaN. Alternatively, we can also drop rows containing missing values by calling df.dropna().
9. Merge DataFrames (combine with grocery prices)

   <span id="mwA1k"></span><span class="n" id="mwA1o">prices</span> <span class="o" id="mwA1s">=</span> <span class="n" id="mwA1w">pd</span><span class="o" id="mwA10">.</span><span class="n" id="mwA14">DataFrame</span><span class="p" id="mwA18">({</span> <span class="s2" id="mwA2A">"food"</span><span class="p" id="mwA2E">:</span> <span class="p" id="mwA2I">[</span><span class="s2" id="mwA2M">"Apple"</span><span class="p" id="mwA2Q">,</span> <span class="s2" id="mwA2U">"Banana"</span><span class="p" id="mwA2Y">,</span> <span class="s2" id="mwA2c">"Almonds"</span><span class="p" id="mwA2g">,</span> <span class="s2" id="mwA2k">"Broccoli"</span><span class="p" id="mwA2o">,</span> <span class="s2" id="mwA2s">"Salmon"</span><span class="p" id="mwA2w">,</span> <span class="s2" id="mwA20">"Oatmeal"</span><span class="p" id="mwA24">],</span> <span class="s2" id="mwA28">"price"</span><span class="p" id="mwA3A">:</span> <span class="p" id="mwA3E">[</span><span class="mf" id="mwA3I">0.7</span><span class="p" id="mwA3M">,</span> <span class="mf" id="mwA3Q">0.5</span><span class="p" id="mwA3U">,</span> <span class="mf" id="mwA3Y">4.2</span><span class="p" id="mwA3c">,</span> <span class="mf" id="mwA3g">2.0</span><span class="p" id="mwA3k">,</span> <span class="mf" id="mwA3o">10.5</span><span class="p" id="mwA3s">,</span> <span class="mf" id="mwA3w">3.4</span><span class="p" id="mwA30">]</span> <span class="p" id="mwA34">})</span> <span class="n" id="mwA38">merged</span> <span class="o" id="mwA4A">=</span> <span class="n" id="mwA4E">df</span><span class="o" id="mwA4I">.</span><span class="n" id="mwA4M">merge</span><span class="p" id="mwA4Q">(</span><span class="n" id="mwA4U">prices</span><span class="p" id="mwA4Y">,</span> <span class="n" id="mwA4c">on</span><span class="o" id="mwA4g">=</span><span class="s2" id="mwA4k">"food"</span><span class="p" id="mwA4o">)</span>

   The on keyword specifies the name of the key column by which the two DataFrames will be merged. To merge datasets with different key column names, use the parameters left_on and right_on instead.
10. Categorize food by calories:

    <span id="mwA5U"></span><span class="n" id="mwA5Y">df</span><span class="p" id="mwA5c">[</span><span class="s2" id="mwA5g">"calorie_level"</span><span class="p" id="mwA5k">]</span> <span class="o" id="mwA5o">=</span> <span class="n" id="mwA5s">pd</span><span class="o" id="mwA5w">.</span><span class="n" id="mwA50">cut</span><span class="p" id="mwA54">(</span> <span class="n" id="mwA58">df</span><span class="p" id="mwA6A">[</span><span class="s2" id="mwA6E">"calories"</span><span class="p" id="mwA6I">],</span> <span class="n" id="mwA6M">bins</span><span class="o" id="mwA6Q">=</span><span class="p" id="mwA6U">[</span><span class="mi" id="mwA6Y">0</span><span class="p" id="mwA6c">,</span><span class="mi" id="mwA6g">100</span><span class="p" id="mwA6k">,</span><span class="mi" id="mwA6o">200</span><span class="p" id="mwA6s">,</span><span class="mi" id="mwA6w">300</span><span class="p" id="mwA60">],</span> <span class="n" id="mwA64">labels</span><span class="o" id="mwA68">=</span><span class="p" id="mwA7A">[</span><span class="s2" id="mwA7E">"Low"</span><span class="p" id="mwA7I">,</span> <span class="s2" id="mwA7M">"Medium"</span><span class="p" id="mwA7Q">,</span> <span class="s2" id="mwA7U">"High"</span><span class="p" id="mwA7Y">]</span> <span class="p" id="mwA7c">)</span>

    This splits numeric data into separate "bins" or intervals, thereby allowing continuous measurements to be analyzed as discrete categories.

Example 2: Resampling

Create example time series data, daily:

<span id="mwA8Q"></span><span class="n" id="mwA8U">periods</span> <span class="o" id="mwA8Y">=</span> <span class="mi" id="mwA8c">30</span> <span class="n" id="mwA8g">days</span> <span class="o" id="mwA8k">=</span> <span class="n" id="mwA8o">pd</span><span class="o" id="mwA8s">.</span><span class="n" id="mwA8w">date_range</span><span class="p" id="mwA80">(</span><span class="n" id="mwA84">start</span><span class="o" id="mwA88">=</span><span class="s1" id="mwA9A">'1 June 2019'</span><span class="p" id="mwA9E">,</span> <span class="n" id="mwA9I">periods</span><span class="o" id="mwA9M">=</span><span class="n" id="mwA9Q">periods</span><span class="p" id="mwA9U">)</span> <span class="n" id="mwA9Y">np</span><span class="o" id="mwA9c">.</span><span class="n" id="mwA9g">random</span><span class="o" id="mwA9k">.</span><span class="kp" id="mwA9o">seed</span><span class="p" id="mwA9s">(</span><span class="mi" id="mwA9w">0</span><span class="p" id="mwA90">)</span> <span class="c1" id="mwA94"># Seed the random number generator (RNG)</span> <span class="n" id="mwA98">values</span> <span class="o" id="mwA-A">=</span> <span class="n" id="mwA-E">np</span><span class="o" id="mwA-I">.</span><span class="n" id="mwA-M">random</span><span class="o" id="mwA-Q">.</span><span class="n" id="mwA-U">rand</span><span class="p" id="mwA-Y">(</span><span class="n" id="mwA-c">periods</span><span class="p" id="mwA-g">)</span> <span class="n" id="mwA-k">s_daily</span> <span class="o" id="mwA-o">=</span> <span class="n" id="mwA-s">pd</span><span class="o" id="mwA-w">.</span><span class="n" id="mwA-0">Series</span><span class="p" id="mwA-4">(</span><span class="n" id="mwA-8">values</span><span class="p" id="mwA_A">,</span> <span class="n" id="mwA_E">index</span><span class="o" id="mwA_I">=</span><span class="n" id="mwA_M">days</span><span class="p" id="mwA_Q">)</span> <span class="nb" id="mwA_U">print</span><span class="p" id="mwA_Y">(</span><span class="n" id="mwA_c">s_daily</span><span class="p" id="mwA_g">)</span>

2019-06-01 0.548814 2019-06-02 0.715189 2019-06-03 0.602763 ... 2019-06-28 0.944669 2019-06-29 0.521848 2019-06-30 0.414662 Freq: D, Length: 30, dtype: float64

Resample to weekly ending Monday:

<span id="mwBAU"></span><span class="n" id="mwBAY">s_weekly</span> <span class="o" id="mwBAc">=</span> <span class="n" id="mwBAg">s_daily</span><span class="o" id="mwBAk">.</span><span class="n" id="mwBAo">resample</span><span class="p" id="mwBAs">(</span><span class="s1" id="mwBAw">'W-Mon'</span><span class="p" id="mwBA0">)</span><span class="o" id="mwBA4">.</span><span class="kp" id="mwBA8">sum</span><span class="p" id="mwBBA">()</span> <span class="nb" id="mwBBE">print</span><span class="p" id="mwBBI">(</span><span class="n" id="mwBBM">s_weekly</span><span class="p" id="mwBBQ">)</span>

2019-06-03 1.866766 2019-06-10 4.290897 2019-06-17 2.992645 2019-06-24 5.500574 2019-07-01 2.782728 Freq: W-MON, dtype: float64

Version history

Pandas uses a version scheme where each x.y line is a feature release and x.y.z releases are patch/bugfix updates; only major feature releases are listed below.