{{ fullname }}{{ underline }}.. currentmodule:: {{ module.split('.')[0] }}.. autoaccessor:: {{ (module.split('.')[1:] + [objname]) | join('.') }}

{{ fullname }}{{ underline }}.. currentmodule:: {{ module.split('.')[0] }}.. autoaccessorattribute:: {{ (module.split('.')[1:] + [objname]) | join('.') }}

{{ fullname }}{{ underline }}.. currentmodule:: {{ module.split('.')[0] }}.. autoaccessorcallable:: {{ (module.split('.')[1:] + [objname]) | join('.') }}.__call__

{{ fullname }}{{ underline }}.. currentmodule:: {{ module.split('.')[0] }}.. autoaccessormethod:: {{ (module.split('.')[1:] + [objname]) | join('.') }}

{% extends "!autosummary/class.rst" %}{% block methods %}{% if methods %}..  HACK -- the point here is that we don't want this to appear in the output, but the autosummary should still generate the pages.  .. autosummary::  {% for item in all_methods %}  {%- if not item.startswith('_') or item in ['__call__'] %}  {{ name }}.{{ item }}  {%- endif -%}  {%- endfor %}{% endif %}{% endblock %}{% block attributes %}{% if attributes %}..  HACK -- the point here is that we don't want this to appear in the output, but the autosummary should still generate the pages.  .. autosummary::  {% for item in all_attributes %}  {%- if not item.startswith('_') %}  {{ name }}.{{ item }}  {%- endif -%}  {%- endfor %}{% endif %}{% endblock %}

{{ fullname }}{{ underline }}.. currentmodule:: {{ module }}.. autoclass:: {{ objname }}

Data cleaning and validation

Pyjanitor_
~~~  Pyjanitor provides a clean API for cleaning data, using method chaining.  Pandera_
~~~  Pandera provides a flexible and expressive API for performing data validation on dataframes
to make data processing pipelines more readable and robust.
Dataframes contain information that pandera explicitly validates at runtime. This is useful in
production-critical data pipelines or reproducible research settings.  pandas-path_
~~~  Since Python 3.4, pathlib has been
included in the Python standard library. Path objects provide a simple
and delightful way to interact with the file system. The pandas-path package enables the
Path API for pandas through a custom accessor ``.path``. Getting just the filenames from
a series of full file paths is as simple as ``my_files.path.name``. Other convenient operations like
joining paths, replacing file extensions, and checking if files exist are also available.  .. _ecosystem.stats:

Statistics and machine learning

pandas-tfrecords_
~~~  Easy saving pandas dataframe to tensorflow tfrecords format and reading tfrecords to pandas.  Statsmodels_
~~~  Statsmodels is the prominent Python "statistics and econometrics library" and it has
a long-standing special relationship with pandas. Statsmodels provides powerful statistics,
econometrics, analysis and modeling functionality that is out of pandas' scope.
Statsmodels leverages pandas objects as the underlying data container for computation.  sklearn-pandas_
~~~  Use pandas DataFrames in your scikit-learn_
ML pipeline.  Featuretools_
~~~  Featuretools is a Python library for automated feature engineering built on top of pandas. It excels at transforming temporal and relational datasets into feature matrices for machine learning using reusable feature engineering "primitives". Users can contribute their own primitives in Python and share them with the rest of the community.  Compose_
~~~  Compose is a machine learning tool for labeling data and prediction engineering. It allows you to structure the labeling process by parameterizing prediction problems and transforming time-driven relational data into target values with cutoff times that can be used for supervised learning.  STUMPY_
~~~  STUMPY is a powerful and scalable Python library for modern time series analysis.
At its core, STUMPY efficiently computes something called a
matrix profile_,
which can be used for a wide variety of time series data mining tasks.  .. _ecosystem.visualization:

Pandas has its own Styler class for table visualization, and while
`pandas also has built-in support for data visualization through charts with matplotlib <visualization>`,
there are a number of other pandas-compatible libraries.  Altair_
~~~  Altair is a declarative statistical visualization library for Python.
With Altair, you can spend more time understanding your data and its
meaning. Altair's API is simple, friendly and consistent and built on
top of the powerful Vega-Lite JSON specification. This elegant
simplicity produces beautiful and effective visualizations with a
minimal amount of code. Altair works with pandas DataFrames.  Bokeh_
~~~  Bokeh is a Python interactive visualization library for large datasets that natively uses
the latest web technologies. Its goal is to provide elegant, concise construction of novel
graphics in the style of Protovis/D3, while delivering high-performance interactivity over
large data to thin clients.  Pandas-Bokeh_ provides a high level API
for Bokeh that can be loaded as a native pandas plotting backend via  .. code:: python  pd.set_option("plotting.backend", "pandas_bokeh")  It is very similar to the matplotlib plotting backend, but provides interactive
web-based charts and maps.  Seaborn_
~~~  Seaborn is a Python visualization library based on
matplotlib_. It provides a high-level, dataset-oriented
interface for creating attractive statistical graphics. The plotting functions
in seaborn understand pandas objects and leverage pandas grouping operations
internally to support concise specification of complex visualizations. Seaborn
also goes beyond matplotlib and pandas with the option to perform statistical
estimation while plotting, aggregating across observations and visualizing the
fit of statistical models to emphasize patterns in a dataset.  plotnine_
~~~  Hadley Wickham's ggplot2_ is a foundational exploratory visualization package for the R language.
Based on "The Grammar of Graphics"_ it
provides a powerful, declarative and extremely general way to generate bespoke plots of any kind of data.
Various implementations to other languages are available.
A good implementation for Python users is has2k1/plotnine_.  IPython vega_
~~~  IPython Vega_ leverages `Vega
<https://github.com/vega/vega>`__ to create plots within Jupyter Notebook.  Plotly_
~~~  Plotly’s_ Python API_ enables interactive figures and web shareability. Maps, 2D, 3D, and live-streaming graphs are rendered with WebGL and D3.js_. The library supports plotting directly from a pandas DataFrame and cloud-based collaboration. Users of matplotlib, ggplot for Python, and Seaborn_ can convert figures into interactive web-based plots. Plots can be drawn in IPython Notebooks_ , edited with R or MATLAB, modified in a GUI, or embedded in apps and dashboards. Plotly is free for unlimited sharing, and has offline_, or on-premise_ accounts for private use.  Lux_
~~~  Lux_ is a Python library that facilitates fast and easy experimentation with data by automating the visual data exploration process. To use Lux, simply add an extra import alongside pandas:  .. code:: python  import lux  import pandas as pd  df = pd.read_csv("data.csv")  df  # discover interesting insights!  By printing out a dataframe, Lux automatically recommends a set of visualizations_ that highlights interesting trends and patterns in the dataframe. Users can leverage any existing pandas commands without modifying their code, while being able to visualize their pandas data structures (e.g., DataFrame, Series, Index) at the same time. Lux also offers a powerful, intuitive language_ that allow users to create  Altair_, matplotlib_, or Vega-Lite_ visualizations without having to think at the level of code.  Qtpandas_
~~~  Spun off from the main pandas library, the qtpandas_
library enables DataFrame visualization and manipulation in PyQt4 and PySide applications.  D-Tale_
~~~  D-Tale is a lightweight web client for visualizing pandas data structures. It

provides a rich spreadsheet-style grid which acts as a wrapper for a lot of
pandas functionality (query, sort, describe, corr...) so users can quickly
manipulate their data. There is also an interactive chart-builder using Plotly
Dash allowing users to build nice portable visualizations. D-Tale can be
invoked with the following command  .. code:: python  import dtale  dtale.show(df)  D-Tale integrates seamlessly with Jupyter notebooks, Python terminals, Kaggle
& Google Colab. Here are some demos of the grid_.  hvplot_
~~~  hvPlot is a high-level plotting API for the PyData ecosystem built on HoloViews_.
It can be loaded as a native pandas plotting backend via  .. code:: python  pd.set_option("plotting.backend", "hvplot")  .. _ecosystem.ide:

IDE

IPython_
~~~  IPython is an interactive command shell and distributed computing
environment. IPython tab completion works with pandas methods and also
attributes like DataFrame columns.  Jupyter Notebook / Jupyter Lab_
~~~
Jupyter Notebook is a web application for creating Jupyter notebooks.
A Jupyter notebook is a JSON document containing an ordered list
of input/output cells which can contain code, text, mathematics, plots
and rich media.
Jupyter notebooks can be converted to a number of open standard output formats
(HTML, HTML presentation slides, LaTeX, PDF, ReStructuredText, Markdown,
Python) through 'Download As' in the web interface and ``jupyter convert``
in a shell.  pandas DataFrames implement ``_repr_html_`` and ``_repr_latex`` methods
which are utilized by Jupyter Notebook for displaying
(abbreviated) HTML or LaTeX tables. LaTeX output is properly escaped.
(Note: HTML tables may or may not be
compatible with non-HTML Jupyter output formats.)  See `Options and Settings <options>` and
`Available Options <options.available>`
for pandas ``display.`` settings.  Quantopian/qgrid_
~~~  qgrid is "an interactive grid for sorting and filtering
DataFrames in IPython Notebook" built with SlickGrid.  Spyder_
~~~  Spyder is a cross-platform PyQt-based IDE combining the editing, analysis,
debugging and profiling functionality of a software development tool with the
data exploration, interactive execution, deep inspection and rich visualization
capabilities of a scientific environment like MATLAB or Rstudio.  Its Variable Explorer_
allows users to view, manipulate and edit pandas ``Index``, ``Series``,
and ``DataFrame`` objects like a "spreadsheet", including copying and modifying
values, sorting, displaying a "heatmap", converting data types and more.
pandas objects can also be renamed, duplicated, new columns added,
copied/pasted to/from the clipboard (as TSV), and saved/loaded to/from a file.
Spyder can also import data from a variety of plain text and binary files
or the clipboard into a new pandas DataFrame via a sophisticated import wizard.  Most pandas classes, methods and data attributes can be autocompleted in
Spyder's Editor_ and
IPython Console_,
and Spyder's Help pane_ can retrieve
and render Numpydoc documentation on pandas objects in rich text with Sphinx
both automatically and on-demand.  .. _ecosystem.api:

API

pandas-datareader_
~~~
``pandas-datareader`` is a remote data access library for pandas (PyPI:``pandas-datareader``).
It is based on functionality that was located in ``pandas.io.data`` and ``pandas.io.wb`` but was
split off in v0.19.
See more in the  pandas-datareader docs:  The following data feeds are available:  * Google Finance  * Tiingo  * Morningstar  * IEX  * Robinhood  * Enigma  * Quandl  * FRED  * Fama/French  * World Bank  * OECD  * Eurostat  * TSP Fund Data  * Nasdaq Trader Symbol Definitions  * Stooq Index Data  * MOEX Data  Quandl/Python_
~~~
Quandl API for Python wraps the Quandl REST API to return
pandas DataFrames with timeseries indexes.  Pydatastream_
~~~
PyDatastream is a Python interface to the
Refinitiv Datastream (DWS)_
REST API to return indexed pandas DataFrames with financial data.
This package requires valid credentials for this API (non free).  pandaSDMX_
~~~
pandaSDMX is a library to retrieve and acquire statistical data
and metadata disseminated in
SDMX 2.1, an ISO-standard
widely used by institutions such as statistics offices, central banks,
and international organisations. pandaSDMX can expose datasets and related
structural metadata including data flows, code-lists,
and data structure definitions as pandas Series
or MultiIndexed DataFrames.  fredapi_
~~~
fredapi is a Python interface to the Federal Reserve Economic Data (FRED)_
provided by the Federal Reserve Bank of St. Louis. It works with both the FRED database and ALFRED database that
contains point-in-time data (i.e. historic data revisions). fredapi provides a wrapper in Python to the FRED
HTTP API, and also provides several convenient methods for parsing and analyzing point-in-time data from ALFRED.
fredapi makes use of pandas and returns data in a Series or DataFrame. This module requires a FRED API key that
you can obtain for free on the FRED website.  dataframe_sql_
~~~
``dataframe_sql`` is a Python package that translates SQL syntax directly into
operations on pandas DataFrames. This is useful when migrating from a database to
using pandas or for users more comfortable with SQL looking for a way to interface
with pandas.  .. _ecosystem.domain:

Domain specific

Geopandas_
~~~  Geopandas extends pandas data objects to include geographic information which support
geometric operations. If your work entails maps and geographical coordinates, and
you love pandas, you should take a close look at Geopandas.  staircase_
~~~  staircase is a data analysis package, built upon pandas and numpy, for modelling and
manipulation of mathematical step functions. It provides a rich variety of arithmetic
operations, relational operations, logical operations, statistical operations and
aggregations for step functions defined over real numbers, datetime and timedelta domains.  xarray_
~~~  xarray brings the labeled data power of pandas to the physical sciences by
providing N-dimensional variants of the core pandas data structures. It aims to
provide a pandas-like and pandas-compatible toolkit for analytics on multi-
dimensional arrays, rather than the tabular data for which pandas excels.  .. _ecosystem.io:

IO

BCPandas_
~~~  BCPandas provides high performance writes from pandas to Microsoft SQL Server,
far exceeding the performance of the native ``df.to_sql`` method. Internally, it uses
Microsoft's BCP utility, but the complexity is fully abstracted away from the end user.
Rigorously tested, it is a complete replacement for ``df.to_sql``.  Deltalake_
~~~  Deltalake python package lets you access tables stored in
Delta Lake_ natively in Python without the need to use Spark or
JVM. It provides the ``delta_table.to_pyarrow_table().to_pandas()`` method to convert
any Delta table into Pandas dataframe.  .. _ecosystem.out-of-core:

Blaze_
~~~  Blaze provides a standard API for doing computations with various
in-memory and on-disk backends: NumPy, pandas, SQLAlchemy, MongoDB, PyTables,
PySpark.  Cylon_
~~~  Cylon is a fast, scalable, distributed memory parallel runtime with a pandas
like Python DataFrame API. ”Core Cylon” is implemented with C++ using Apache
Arrow format to represent the data in-memory. Cylon DataFrame API implements
most of the core operators of pandas such as merge, filter, join, concat,
group-by, drop_duplicates, etc. These operators are designed to work across
thousands of cores to scale applications. It can interoperate with pandas
DataFrame by reading data from pandas or converting data to pandas so users
can selectively scale parts of their pandas DataFrame applications.  .. code:: python  from pycylon import read_csv, DataFrame, CylonEnv  from pycylon.net import MPIConfig  # Initialize Cylon distributed environment  config: MPIConfig = MPIConfig()  env: CylonEnv = CylonEnv(config=config, distributed=True)  df1: DataFrame = read_csv('/tmp/csv1.csv')  df2: DataFrame = read_csv('/tmp/csv2.csv')  # Using 1000s of cores across the cluster to compute the join  df3: Table = df1.join(other=df2, on=[0], algorithm="hash", env=env)  print(df3)  Dask_
~~~  Dask is a flexible parallel computing library for analytics. Dask
provides a familiar ``DataFrame`` interface for out-of-core, parallel and distributed computing.  Dask-ML_
~~~  Dask-ML enables parallel and distributed machine learning using Dask alongside existing machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow.  Ibis_
~~~  Ibis offers a standard way to write analytics code, that can be run in multiple engines. It helps in bridging the gap between local Python environments (like pandas) and remote storage and execution systems like Hadoop components (like HDFS, Impala, Hive, Spark) and SQL databases (Postgres, etc.).  Koalas_
~~~  Koalas provides a familiar pandas DataFrame interface on top of Apache Spark. It enables users to leverage multi-cores on one machine or a cluster of machines to speed up or scale their DataFrame code.  Modin_
~~~  The ``modin.pandas`` DataFrame is a parallel and distributed drop-in replacement
for pandas. This means that you can use Modin with existing pandas code or write
new code with the existing pandas API. Modin can leverage your entire machine or
cluster to speed up and scale your pandas workloads, including traditionally
time-consuming tasks like ingesting data (``read_csv``, ``read_excel``,
``read_parquet``, etc.).  .. code:: python  # import pandas as pd  import modin.pandas as pd  df = pd.read_csv("big.csv")  # use all your cores!  Odo_
~~~  Odo provides a uniform API for moving data between different formats. It uses
pandas own ``read_csv`` for CSV IO and leverages many existing packages such as
PyTables, h5py, and pymongo to move data between non pandas formats. Its graph
based approach is also extensible by end users for custom formats that may be
too specific for the core of odo.  Pandarallel_
~~~  Pandarallel provides a simple way to parallelize your pandas operations on all your CPUs by changing only one line of code.
If also displays progress bars.  .. code:: python  from pandarallel import pandarallel  pandarallel.initialize(progress_bar=True)  # df.apply(func)  df.parallel_apply(func)  Vaex_

~~~  Increasingly, packages are being built on top of pandas to address specific needs in data preparation, analysis and visualization. Vaex is a Python library for Out-of-Core DataFrames (similar to pandas), to visualize and explore big tabular datasets. It can calculate statistics such as mean, sum, count, standard deviation etc, on an N-dimensional grid up to a billion (10\ `9`) objects/rows per second. Visualization is done using histograms, density plots and 3d volume rendering, allowing interactive exploration of big data. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).  * vaex.from_pandas  * vaex.to_pandas_df  .. _ecosystem.extensions:

Extension data types

pandas provides an interface for defining
`extension types <extending.extension-types>` to extend NumPy's type
system. The following libraries implement that interface to provide types not
found in NumPy or pandas, which work well with pandas' data containers.  `Cyberpandas`_
~~~  Cyberpandas provides an extension type for storing arrays of IP Addresses. These
arrays can be stored inside pandas' Series and DataFrame.  `Pandas-Genomics`_
~~~  Pandas-Genomics provides extension types, extension arrays, and extension accessors for working with genomics data  `Pint-Pandas`_
~~~  Pint-Pandas provides an extension type for
storing numeric arrays with units. These arrays can be stored inside pandas'
Series and DataFrame. Operations between Series and DataFrame columns which
use pint's extension array are then units aware.  `Text Extensions for Pandas`_
~~~  Text Extensions for Pandas
provides extension types to cover common data structures for representing natural language
data, plus library integrations that convert the outputs of popular natural language
processing libraries into Pandas DataFrames.  .. _ecosystem.accessors:

Accessors

A directory of projects providing
`extension accessors <extending.register-accessors>`. This is for users to
discover new accessors and for library authors to coordinate on the namespace.  === === === ===
Library  Accessor  Classes  Description
=== === === ===
`cyberpandas`_  ``ip``  ``Series``  Provides common operations for working with IP addresses.
`pdvega`_  ``vgplot``  ``Series``, ``DataFrame``  Provides plotting functions from the Altair_ library.
`pandas-genomics`_ ``genomics`` ``Series``, ``DataFrame``  Provides common operations for quality control and analysis of genomics data.
`pandas_path`_  ``path``  ``Index``, ``Series``  Provides `pathlib.Path`_ functions for Series.
`pint-pandas`_  ``pint``  ``Series``, ``DataFrame``  Provides units support for numeric Series and DataFrames.
`composeml`_  ``slice``  ``DataFrame``  Provides a generator for enhanced data slicing.
`datatest`_  ``validate`` ``Series``, ``DataFrame``, ``Index`` Provides validation, differences, and acceptance managers.
`woodwork`_  ``ww``  ``Series``, ``DataFrame``  Provides physical, logical, and semantic data typing information for Series and DataFrames.
`staircase`_  ``sc``  ``Series``  Provides methods for querying, aggregating and plotting step functions
=== === === ===  .. _cyberpandas: https://cyberpandas.readthedocs.io/en/latest
.. _pdvega: https://altair-viz.github.io/pdvega/
.. _Altair: https://altair-viz.github.io/
.. _pandas-genomics: https://pandas-genomics.readthedocs.io/en/latest/
.. _pandas_path: https://github.com/drivendataorg/pandas-path/
.. _pathlib.Path: https://docs.python.org/3/library/pathlib.html
.. _pint-pandas: https://github.com/hgrecco/pint-pandas
.. _composeml: https://github.com/alteryx/compose
.. _datatest: https://datatest.readthedocs.io/en/stable/
.. _woodwork: https://github.com/alteryx/woodwork
.. _staircase: https://www.staircase.dev/

Development tools

pandas-stubs_
~~~  While pandas repository is partially typed, the package itself doesn't expose this information for external use.
Install pandas-stubs to enable basic type coverage of pandas API.  Learn more by reading through `14468`, `26766`, `28142`.  See installation and usage instructions on the GitHub page_.

Contributor community

pandas is a community-driven open source project developed by a large group
of contributors
and a smaller group of maintainers.
The pandas leadership has made a strong commitment to creating an open,
inclusive, and positive community. Please read the pandas `Code of Conduct
<https://pandas.pydata.org/community/coc.html>`_ for guidance on how to
interact with others in a way that makes the community thrive.  We offer several meetings and communication channels to share knowledge and
connect with others within the pandas community.

Community meeting

The pandas Community Meeting is a regular sync meeting for the project's
maintainers which is open to the community. Everyone is welcome to attend and
contribute to conversations.  The meetings take place on the second and fourth Wednesdays of each month at 18:00 UTC.  The minutes of past meetings are available in this Google Document_.

New contributor meeting

On the third Wednesday of the month, we hold meetings to welcome and support
new contributors in our community.  | 👋 you all are invited
| 💬 everyone can present (add yourself to the hackMD agenda)
| 👀 anyone can sit in and listen  Attendees are new and experienced contributors, as well as a few maintainers.
We aim to answer questions about getting started, or help with work in
progress when possible, as well as get to know each other and share our
learnings and experiences.  The agenda for the next meeting and minutes of past meetings are available in
this HackMD_.

Calendar

This calendar shows all the community meetings. Our community meetings are
ideal for anyone wanting to contribute to pandas, or just curious to know how
current development is going.  .. raw:: html  <iframe src="https://calendar.google.com/calendar/embed?src=pgbn14p6poja8a1cf2dv2jhrmg%40group.calendar.google.com" style="border: 0" width="800" height="600" frameborder="0" scrolling="no"></iframe>  You can subscribe to this calendar with the following links:  * iCal_
* Google calendar_  Additionally, we'll sometimes have one-off meetings on specific topics.
These will be published on the same calendar.

GitHub issue tracker

The pandas contributor community conducts conversations mainly via this channel.
Any community member can open issues to:  - Report bugs, e.g. "I noticed the behavior of a certain function is  incorrect"
- Request features, e.g. "I would like this error message to be more readable"
- Request documentation improvements, e.g. "I found this section unclear"
- Ask questions, e.g. "I noticed the behavior of a certain function  changed between versions. Is this expected?".  Ideally, your questions should be related to how pandas works rather  than how you use pandas. StackOverflow is  better suited for answering usage questions, and we ask that all usage  questions are first asked on StackOverflow. Thank you for respecting our  time and wishes. 🙇  Maintainers and frequent contributors might also open issues to discuss the
ongoing development of the project. For example:  - Report issues with the CI, GitHub Actions, or the performance of pandas
- Open issues relating to the internals
- Start roadmap discussion aligning on proposals for what to do in future  releases or changes to the API.
- Open issues relating to the project's website, logo, or governance

The developer mailing list

The pandas mailing list `pandas-dev@python.org <mailto://pandas-dev@python
.org>`_ is used for long form
conversations and to engage people in the wider community who might not
be active on the issue tracker but we would like to include in discussions.  .. _community.slack:

Community slack

We have a chat platform for contributors, maintainers and potential
contributors. This is not a space for user questions, rather for questions about
contributing to pandas. The slack is a private space, specifically meant for
people who are hesitant to bring up their questions or ideas on a large public
mailing list or GitHub.  If this sounds like the right place for you, you are welcome to join! Email us
at slack@pandas.pydata.org and let us
know that you read and agree to our Code of Conduct
😉 to get an invite. And please remember that slack is not meant to replace the
mailing list or issue tracker - all important announcements and conversations
should still happen there.

Where to start?

All contributions, bug reports, bug fixes, documentation improvements,
enhancements, and ideas are welcome.  If you are brand new to pandas or open-source development, we recommend going
through the GitHub "issues" tab
to find issues that interest you. There are a number of issues listed under `Docs
<https://github.com/pandas-dev/pandas/issues?labels=Docs&sort=updated&state=open>`_
and `good first issue
<https://github.com/pandas-dev/pandas/issues?labels=good+first+issue&sort=updated&state=open>`_
where you could start out. Once you've found an interesting issue, you can
return here to get your development environment setup.  When you start working on an issue, it's a good idea to assign the issue to yourself,
so nobody else duplicates the work on it. GitHub restricts assigning issues to maintainers
of the project only. In most projects, and until recently in pandas, contributors added a
comment letting others know they are working on an issue. While this is ok, you need to
check each issue individually, and it's not possible to find the unassigned ones.  For this reason, we implemented a workaround consisting of adding a comment with the exact
text ``take``. When you do it, a GitHub action will automatically assign you the issue
(this will take seconds, and may require refreshing the page to see it).
By doing this, it's possible to filter the list of issues and find only the unassigned ones.  So, a good way to find an issue to start contributing to pandas is to check the list of
unassigned good first issues
and assign yourself one you like by writing a comment with the exact text ``take``.  If for whatever reason you are not able to continue working with the issue, please try to
unassign it, so other people know it's available again. You can check the list of
assigned issues, since people may not be working in them anymore. If you want to work on one
that is assigned, feel free to kindly ask the current assignee if you can take it
(please allow at least a week of inactivity before considering work in the issue discontinued).  We have several `contributor community <community>` communication channels, which you are
welcome to join, and ask questions as you figure things out. Among them are regular meetings for
new contributors, dev meetings, a dev mailing list, and a slack for the contributor community.
All pandas contributors are welcome to these spaces, where they can connect with each other. Even
maintainers who have been with us for a long time felt just like you when they started out, and
are happy to welcome you and support you as you get to know how we work, and where things are.
Take a look at the next sections to learn more.  .. _contributing.bug_reports:

Bug reports and enhancement requests

Bug reports are an important part of making pandas more stable. Having a complete bug report
will allow others to reproduce the bug and provide insight into fixing. See
this stackoverflow article and
this blogpost
for tips on writing a good bug report.  Trying the bug-producing code out on the *main* branch is often a worthwhile exercise
to confirm the bug still exists. It is also worth searching existing bug reports and pull requests
to see if the issue has already been reported and/or fixed.  Bug reports must:  #. Include a short, self-contained Python snippet reproducing the problem.  You can format the code nicely by using `GitHub Flavored Markdown  <https://github.github.com/github-flavored-markdown/>`_::  ```python  >>> from pandas import DataFrame  >>> df = DataFrame(...)  ...  ```  #. Include the full version string of pandas and its dependencies. You can use the built-in function::  >>> import pandas as pd  >>> pd.show_versions()  #. Explain why the current behavior is wrong/not desired and what you expect instead.  The issue will then show up to the pandas community and be open to comments/ideas from others.  .. _contributing.github:

Working with the code

Now that you have an issue you want to fix, enhancement to add, or documentation to improve,
you need to learn how to work with GitHub and the pandas code base.  .. _contributing.version_control:

Version control, Git, and GitHub

To the new user, working with Git is one of the more daunting aspects of contributing to pandas.
It can very quickly become overwhelming, but sticking to the guidelines below will help keep the process
straightforward and mostly trouble free.  As always, if you are having difficulties please
feel free to ask for help.  The code is hosted on GitHub. To
contribute you will need to sign up for a `free GitHub account
<https://github.com/signup/free>_. We use `Git for
version control to allow many people to work together on the project.  Some great resources for learning Git:  * the Git documentation.
* the GitHub help pages.
* the NumPy documentation.
* Matthew Brett's Pydagogue.

Getting started with Git

GitHub has instructions_ for installing git,
setting up your SSH key, and configuring git.  All these steps need to be completed before
you can work seamlessly between your local repository and GitHub.  .. _contributing.forking:

Forking

You will need your own fork to work on the code. Go to the `pandas project
page <https://github.com/pandas-dev/pandas>`_ and hit the ``Fork`` button. You will
want to clone your fork to your machine::  git clone https://github.com/your-user-name/pandas.git pandas-yourname  cd pandas-yourname  git remote add upstream https://github.com/pandas-dev/pandas.git  git fetch upstream  This creates the directory ``pandas-yourname`` and connects your repository to
the upstream (main project) *pandas* repository.  Note that performing a shallow clone (with ``--depth==N``, for some ``N`` greater
or equal to 1) might break some tests and features as ``pd.show_versions()``
as the version number cannot be computed anymore.

Creating a branch

You want your main branch to reflect only production-ready code, so create a
feature branch for making your changes. For example::  git branch shiny-new-feature  git checkout shiny-new-feature  The above can be simplified to::  git checkout -b shiny-new-feature  This changes your working directory to the shiny-new-feature branch.  Keep any
changes in this branch specific to one bug or feature so it is clear
what the branch brings to pandas. You can have many shiny-new-features
and switch in between them using the git checkout command.  When creating this branch, make sure your main branch is up to date with
the latest upstream main version. To update your local main branch, you
can do::  git checkout main  git pull upstream main --ff-only  When you want to update the feature branch with changes in main after
you created the branch, check the section on
`updating a PR <contributing.update-pr>`.

Contributing your changes to pandas

.. _contributing.commit-code:

Committing your code

Keep style fixes to a separate commit to make your pull request more readable.  Once you've made changes, you can see them by typing::  git status  If you have created a new file, it is not being tracked by git. Add it by typing::  git add path/to/file-to-be-added.py  Doing 'git status' again should give something like::  # On branch shiny-new-feature  #  #  modified:  /relative/path/to/file-you-added.py  #  Finally, commit your changes to your local repository with an explanatory commit
message::  git commit -m "your commit message goes here"  .. _contributing.push-code:

Pushing your changes

When you want your changes to appear publicly on your GitHub page, push your
forked feature branch's commits::  git push origin shiny-new-feature  Here ``origin`` is the default name given to your remote repository on GitHub.
You can see the remote repositories::  git remote -v  If you added the upstream repository as described above you will see something
like::  origin  git@github.com:yourname/pandas.git (fetch)  origin  git@github.com:yourname/pandas.git (push)  upstream  git://github.com/pandas-dev/pandas.git (fetch)  upstream  git://github.com/pandas-dev/pandas.git (push)  Now your code is on GitHub, but it is not yet a part of the pandas project. For that to
happen, a pull request needs to be submitted on GitHub.

Review your code

When you're ready to ask for a code review, file a pull request. Before you do, once
again make sure that you have followed all the guidelines outlined in this document
regarding code style, tests, performance tests, and documentation. You should also
double check your branch changes against the branch it was based on:  #. Navigate to your repository on GitHub -- https://github.com/your-user-name/pandas
#. Click on ``Branches``
#. Click on the ``Compare`` button for your feature branch
#. Select the ``base`` and ``compare`` branches, if necessary. This will be ``main`` and  ``shiny-new-feature``, respectively.

Finally, make the pull request

If everything looks good, you are ready to make a pull request. A pull request is how
code from a local repository becomes available to the GitHub community and can be looked
at and eventually merged into the main version. This pull request and its associated
changes will eventually be committed to the main branch and available in the next
release. To submit a pull request:  #. Navigate to your repository on GitHub
#. Click on the ``Compare & pull request`` button
#. You can then click on ``Commits`` and ``Files Changed`` to make sure everything looks  okay one last time
#. Write a descriptive title that includes prefixes. pandas uses a convention for title  prefixes. Here are some common ones along with general guidelines for when to use them:  * ENH: Enhancement, new functionality  * BUG: Bug fix  * DOC: Additions/updates to documentation  * TST: Additions/updates to tests  * BLD: Updates to the build process/scripts  * PERF: Performance improvement  * TYP: Type annotations  * CLN: Code cleanup  #. Write a description of your changes in the ``Preview Discussion`` tab
#. Click ``Send Pull Request``.  This request then goes to the repository maintainers, and they will review
the code.  .. _contributing.update-pr:

Updating your pull request

Based on the review you get on your pull request, you will probably need to make
some changes to the code. In that case, you can make them in your branch,
add a new commit to that branch, push it to GitHub, and the pull request will be
automatically updated.  Pushing them to GitHub again is done by::  git push origin shiny-new-feature  This will automatically update your pull request with the latest code and restart the
`Continuous Integration <contributing.ci>` tests.  Another reason you might need to update your pull request is to solve conflicts
with changes that have been merged into the main branch since you opened your
pull request.  To do this, you need to "merge upstream main" in your branch::  git checkout shiny-new-feature  git fetch upstream  git merge upstream/main  If there are no conflicts (or they could be fixed automatically), a file with a
default commit message will open, and you can simply save and quit this file.  If there are merge conflicts, you need to solve those conflicts. See for
example at https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/
for an explanation on how to do this.
Once the conflicts are merged and the files where the conflicts were solved are
added, you can run ``git commit`` to save those fixes.  If you have uncommitted changes at the moment you want to update the branch with
main, you will need to ``stash`` them prior to updating (see the
stash docs_).
This will effectively store your changes and they can be reapplied after updating.  After the feature branch has been update locally, you can now update your pull
request by pushing to the branch on GitHub::  git push origin shiny-new-feature

Autofixing formatting errors

We use several styling checks (e.g. ``black``, ``flake8``, ``isort``) which are run after
you make a pull request.  To automatically fix formatting errors on each commit you make, you can
set up pre-commit yourself. First, create a Python `environment
<contributing_environment>` and then set up `pre-commit <contributing.pre-commit>`.

Tips for a successful pull request

If you have made it to the `Review your code`_ phase, one of the core contributors may
take a look. Please note however that a handful of people are responsible for reviewing
all of the contributions, which can often lead to bottlenecks.  To improve the chances of your pull request being reviewed, you should:  - **Reference an open issue** for non-trivial changes to clarify the PR's purpose
- **Ensure you have appropriate tests**. These should be the first part of any PR
- **Keep your pull requests as simple as possible**. Larger PRs take longer to review
- **Ensure that CI is in a green state**. Reviewers may not even look otherwise
- **Keep** `Updating your pull request`_, either by request or every few days

Code standards

Writing good code is not just about what you write. It is also about *how* you
write it. During `Continuous Integration <contributing.ci>` testing, several
tools will be run to check your code for stylistic errors.
Generating any warnings will cause the test to fail.
Thus, good style is a requirement for submitting code to pandas.  There is a tool in pandas to help contributors verify their changes before
contributing them to the project::  ./ci/code_checks.sh  The script validates the doctests, formatting in docstrings, and
imported modules. It is possible to run the checks independently by using the
parameters ``docstrings``, ``code``, and ``doctests``
(e.g. ``./ci/code_checks.sh doctests``).  In addition, because a lot of people use our library, it is important that we
do not make sudden changes to the code that could have the potential to break
a lot of user code as a result, that is, we need it to be as *backwards compatible*
as possible to avoid mass breakages.  In addition to ``./ci/code_checks.sh``, some extra checks (including static type
checking) are run by ``pre-commit`` - see `here <contributing.pre-commit>`
for how to run them.  .. _contributing.pre-commit:

Pre-commit

Additionally, `Continuous Integration <contributing.ci>` will run code formatting checks
like ``black``, ``ruff``,
`isort``, and ``cpplint`` and more using `pre-commit hooks
Any warnings from these checks will cause the `Continuous Integration <contributing.ci>` to fail; therefore,
it is helpful to run the check yourself before submitting code. This
can be done by installing ``pre-commit``::  pip install pre-commit  and then running::  pre-commit install  from the root of the pandas repository. Now all of the styling checks will be
run each time you commit changes without your needing to run each one manually.
In addition, using ``pre-commit`` will also allow you to more easily
remain up-to-date with our code checks as they change.  Note that if needed, you can skip these checks with ``git commit --no-verify``.  If you don't want to use ``pre-commit`` as part of your workflow, you can still use it
to run its checks with::  pre-commit run --files <files you have modified>  without needing to have done ``pre-commit install`` beforehand.  If you want to run checks on all recently committed files on upstream/main you can use::  pre-commit run --from-ref=upstream/main --to-ref=HEAD --all-files  without needing to have done ``pre-commit install`` beforehand.  .. note::  You may want to periodically run ``pre-commit gc``, to clean up repos  which are no longer used.  .. note::  If you have conflicting installations of ``virtualenv``, then you may get an  error - see here.  Also, due to a bug in virtualenv,  you may run into issues if you're using conda. To solve this, you can downgrade  ``virtualenv`` to version ``20.0.33``.

Optional dependencies

Optional dependencies (e.g. matplotlib) should be imported with the private helper
``pandas.compat._optional.import_optional_dependency``. This ensures a
consistent error message when the dependency is not met.  All methods using an optional dependency should include a test asserting that an
``ImportError`` is raised when the optional dependency is not found. This test
should be skipped if the library is present.  All optional dependencies should be documented in
`install.optional_dependencies` and the minimum required version should be
set in the ``pandas.compat._optional.VERSIONS`` dict.

Backwards compatibility

Please try to maintain backward compatibility. pandas has lots of users with lots of
existing code, so don't break it if at all possible.  If you think breakage is required,
clearly state why as part of the pull request.  Also, be careful when changing method
signatures and add deprecation warnings where needed. Also, add the deprecated sphinx
directive to the deprecated functions or methods.  If a function with the same arguments as the one being deprecated exist, you can use
the ``pandas.util._decorators.deprecate``:  .. code-block:: python  from pandas.util._decorators import deprecate  deprecate('old_func', 'new_func', '1.1.0')  Otherwise, you need to do it manually:  .. code-block:: python  import warnings  from pandas.util._exceptions import find_stack_level  def old_func():  """Summary of the function.  .. deprecated:: 1.1.0  Use new_func instead.  """  warnings.warn(  'Use new_func instead.',  FutureWarning,  stacklevel=find_stack_level(),  )  new_func()  def new_func():  pass  You'll also need to  1. Write a new test that asserts a warning is issued when calling with the deprecated argument
2. Update all of pandas existing tests and code to use the new argument  See `contributing.warnings` for more.  .. _contributing.type_hints:

pandas strongly encourages the use of `484` style type hints. New development should contain type hints and pull requests to annotate existing code are accepted as well!  Style guidelines
~~~  Type imports should follow the ``from typing import ...`` convention. Some types do not need to be imported since `585` some builtin constructs, such as ``list`` and ``tuple``, can directly be used for type annotations. So rather than  .. code-block:: python  import typing  primes: typing.List[int] = []  You should write  .. code-block:: python  primes: list[int] = []  ``Optional`` should be  avoided in favor of the shorter ``| None``, so instead of  .. code-block:: python  from typing import Union  maybe_primes: list[Union[int, None]] = []  or  .. code-block:: python  from typing import Optional  maybe_primes: list[Optional[int]] = []  You should write  .. code-block:: python  from __future__ import annotations  # noqa: F404  maybe_primes: list[int | None] = []  In some cases in the code base classes may define class variables that shadow builtins. This causes an issue as described in Mypy 1775. The defensive solution here is to create an unambiguous alias of the builtin and use that without your annotation. For example, if you come across a definition like  .. code-block:: python  class SomeClass1:  str = None  The appropriate way to annotate this would be as follows  .. code-block:: python  str_type = str  class SomeClass2:  str: str_type = None  In some cases you may be tempted to use ``cast`` from the typing module when you know better than the analyzer. This occurs particularly when using custom inference functions. For example  .. code-block:: python  from typing import cast  from pandas.core.dtypes.common import is_number  def cannot_infer_bad(obj: Union[str, int, float]):  if is_number(obj):  ...  else:  # Reasonably only str objects would reach this but...  obj = cast(str, obj)  # Mypy complains without this!  return obj.upper()  The limitation here is that while a human can reasonably understand that `is_number`` would catch the ``int`` and ``float`` types mypy cannot make that same inference just yet (see `mypy #5206. While the above works, the use of ``cast`` is **strongly discouraged**. Where applicable a refactor of the code to appease static analysis is preferable  .. code-block:: python  def cannot_infer_good(obj: Union[str, int, float]):  if isinstance(obj, str):  return obj.upper()  else:  ...  With custom types and inference this is not always possible so exceptions are made, but every effort should be exhausted to avoid ``cast`` before going down such paths.  pandas-specific types
~~~  Commonly used types specific to pandas will appear in pandas._typing and you should use these where applicable. This module is private for now but ultimately this should be exposed to third party libraries who want to implement type checking against pandas.  For example, quite a few functions in pandas accept a ``dtype`` argument. This can be expressed as a string like ``"object"``, a ``numpy.dtype`` like ``np.int64`` or even a pandas ``ExtensionDtype`` like ``pd.CategoricalDtype``. Rather than burden the user with having to constantly annotate all of those options, this can simply be imported and reused from the pandas._typing module  .. code-block:: python  from pandas._typing import Dtype  def as_type(dtype: Dtype) -> ...:  ...  This module will ultimately house types for repeatedly used concepts like "path-like", "array-like", "numeric", etc... and can also hold aliases for commonly appearing parameters like ``axis``. Development of this module is active so be sure to refer to the source for the most up to date list of available types.  Validating type hints

~~~  pandas uses mypy and pyright to statically analyze the code base and type hints. After making any change you can ensure your type hints are correct by running  .. code-block:: shell  # the following might fail if the installed pandas version does not correspond to your local git version  pre-commit run --hook-stage manual --all-files  # if the above fails due to stubtest  SKIP=stubtest pre-commit run --hook-stage manual --all-files  in your activated python environment. A recent version of ``numpy`` (>=1.22.0) is required for type validation.  .. _contributing.ci:  Testing type hints in code using pandas
~~~  .. warning::  * Pandas is not yet a py.typed library (`561`)!  The primary purpose of locally declaring pandas as a py.typed library is to test and  improve the pandas-builtin type annotations.  Until pandas becomes a py.typed library, it is possible to easily experiment with the type
annotations shipped with pandas by creating an empty file named "py.typed" in the pandas
installation folder:  .. code-block:: none  python -c "import pandas; import pathlib; (pathlib.Path(pandas.__path__[0]) / 'py.typed').touch()"  The existence of the py.typed file signals to type checkers that pandas is already a py.typed
library. This makes type checkers aware of the type annotations shipped with pandas.

Testing with continuous integration

The pandas test suite will run automatically on GitHub Actions_
continuous integration services, once your pull request is submitted.
However, if you wish to run the test suite on a branch prior to submitting the pull request,
then the continuous integration services need to be hooked to your GitHub repository. Instructions are here
for GitHub Actions_.  A pull-request will be considered for merging when you have an all 'green' build. If any tests are failing,
then you will get a red 'X', where you can click through to see the individual failed tests.
This is an example of a green build.  .. _contributing.tdd:

pandas is serious about testing and strongly encourages contributors to embrace
test-driven development (TDD).
This development process "relies on the repetition of a very short development cycle:
first the developer writes an (initially failing) automated test case that defines a desired
improvement or new function, then produces the minimum amount of code to pass that test."
So, before actually writing any code, you should write your tests.  Often the test can be
taken from the original GitHub issue.  However, it is always worth considering additional
use cases and writing corresponding tests.  Adding tests is one of the most common requests after code is pushed to pandas.  Therefore,
it is worth getting in the habit of writing tests ahead of time so this is never an issue.  Writing tests
~~~  All tests should go into the ``tests`` subdirectory of the specific package.
This folder contains many current examples of tests, and we suggest looking to these for
inspiration.  As a general tip, you can use the search functionality in your integrated development
environment (IDE) or the git grep command in a terminal to find test files in which the method
is called. If you are unsure of the best location to put your test, take your best guess,
but note that reviewers may request that you move the test to a different location.  To use git grep, you can run the following command in a terminal:  ``git grep "function_name("``  This will search through all files in your repository for the text ``function_name(``.
This can be a useful way to quickly locate the function in the
codebase and determine the best location to add a test for it.  Ideally, there should be one, and only one, obvious place for a test to reside.
Until we reach that ideal, these are some rules of thumb for where a test should

be located.  1. Does your test depend only on code in ``pd._libs.tslibs``?  This test likely belongs in one of:  - tests.tslibs  .. note::  No file in ``tests.tslibs`` should import from any pandas modules  outside of ``pd._libs.tslibs``  - tests.scalar  - tests.tseries.offsets  2. Does your test depend only on code in pd._libs?  This test likely belongs in one of:  - tests.libs  - tests.groupby.test_libgroupby  3. Is your test for an arithmetic or comparison method?  This test likely belongs in one of:  - tests.arithmetic  .. note::  These are intended for tests that can be shared to test the behavior  of DataFrame/Series/Index/ExtensionArray using the ``box_with_array``  fixture.  - tests.frame.test_arithmetic  - tests.series.test_arithmetic  4. Is your test for a reduction method (min, max, sum, prod, ...)?  This test likely belongs in one of:  - tests.reductions  .. note::  These are intended for tests that can be shared to test the behavior  of DataFrame/Series/Index/ExtensionArray.  - tests.frame.test_reductions  - tests.series.test_reductions  - tests.test_nanops  5. Is your test for an indexing method?  This is the most difficult case for deciding where a test belongs, because  there are many of these tests, and many of them test more than one method  (e.g. both ``Series.__getitem__`` and ``Series.loc.__getitem__``)  A) Is the test specifically testing an Index method (e.g. ``Index.get_loc``,  ``Index.get_indexer``)?  This test likely belongs in one of:  - tests.indexes.test_indexing  - tests.indexes.fooindex.test_indexing  Within that files there should be a method-specific test class e.g.  ``TestGetLoc``.  In most cases, neither ``Series`` nor ``DataFrame`` objects should be  needed in these tests.  B) Is the test for a Series or DataFrame indexing method *other* than  ``__getitem__`` or ``__setitem__``, e.g. ``xs``, ``where``, ``take``,  ``mask``, ``lookup``, or ``insert``?  This test likely belongs in one of:  - tests.frame.indexing.test_methodname  - tests.series.indexing.test_methodname  C) Is the test for any of ``loc``, ``iloc``, ``at``, or ``iat``?  This test likely belongs in one of:  - tests.indexing.test_loc  - tests.indexing.test_iloc  - tests.indexing.test_at  - tests.indexing.test_iat  Within the appropriate file, test classes correspond to either types of  indexers (e.g. ``TestLocBooleanMask``) or major use cases  (e.g. ``TestLocSetitemWithExpansion``).  See the note in section D) about tests that test multiple indexing methods.  D) Is the test for ``Series.__getitem__``, ``Series.__setitem__``,  ``DataFrame.__getitem__``, or ``DataFrame.__setitem__``?  This test likely belongs in one of:  - tests.series.test_getitem  - tests.series.test_setitem  - tests.frame.test_getitem  - tests.frame.test_setitem  If many cases such a test may test multiple similar methods, e.g.  .. code-block:: python  import pandas as pd  import pandas._testing as tm  def test_getitem_listlike_of_ints():  ser = pd.Series(range(5))  result = ser[[3, 4]]  expected = pd.Series([2, 3])  tm.assert_series_equal(result, expected)  result = ser.loc[[3, 4]]  tm.assert_series_equal(result, expected)  In cases like this, the test location should be based on the *underlying*  method being tested.  Or in the case of a test for a bugfix, the location  of the actual bug.  So in this example, we know that ``Series.__getitem__``  calls ``Series.loc.__getitem__``, so this is *really* a test for  ``loc.__getitem__``.  So this test belongs in ``tests.indexing.test_loc``.  6. Is your test for a DataFrame or Series method?  A) Is the method a plotting method?  This test likely belongs in one of:  - tests.plotting  B) Is the method an IO method?  This test likely belongs in one of:  - tests.io  C) Otherwise  This test likely belongs in one of:  - tests.series.methods.test_mymethod  - tests.frame.methods.test_mymethod  .. note::  If a test can be shared between DataFrame/Series using the  ``frame_or_series`` fixture, by convention it goes in the  ``tests.frame`` file.  7. Is your test for an Index method, not depending on Series/DataFrame?  This test likely belongs in one of:  - tests.indexes  8) Is your test for one of the pandas-provided ExtensionArrays (``Categorical``,  ``DatetimeArray``, ``TimedeltaArray``, ``PeriodArray``, ``IntervalArray``,  ``PandasArray``, ``FloatArray``, ``BoolArray``, ``StringArray``)?  This test likely belongs in one of:  - tests.arrays  9) Is your test for *all* ExtensionArray subclasses (the "EA Interface")?  This test likely belongs in one of:  - tests.extension  Using ``pytest``

~~~  Test structure
^^^  pandas existing test structure is *mostly* class-based, meaning that you will typically find tests wrapped in a class.  .. code-block:: python  class TestReallyCoolFeature:  def test_cool_feature_aspect(self):  pass  We prefer a more *functional* style using the pytest_ framework, which offers a richer testing
framework that will facilitate testing and developing. Thus, instead of writing test classes, we will write test functions like this:  .. code-block:: python  def test_really_cool_feature():  pass  Preferred ``pytest`` idioms
^^^  * Functional tests named ``def test_*`` and *only* take arguments that are either fixtures or parameters.
* Use a bare ``assert`` for testing scalars and truth-testing
* Use ``tm.assert_series_equal(result, expected)`` and ``tm.assert_frame_equal(result, expected)`` for comparing `Series` and `DataFrame` results respectively.
* Use @pytest.mark.parameterize_ when testing multiple cases.
* Use pytest.mark.xfail_ when a test case is expected to fail.
* Use pytest.mark.skip_ when a test case is never expected to pass.
* Use pytest.param_ when a test case needs a particular mark.
* Use @pytest.fixture_ if multiple tests can share a setup object.  .. warning::  Do not use ``pytest.xfail`` (which is different than ``pytest.mark.xfail``) since it immediately stops the  test and does not check if the test will fail. If this is the behavior you desire, use ``pytest.skip`` instead.  If a test is known to fail but the manner in which it fails
is not meant to be captured, use ``pytest.mark.xfail`` It is common to use this method for a test that
exhibits buggy behavior or a non-implemented feature. If
the failing test has flaky behavior, use the argument ``strict=False``. This
will make it so pytest does not fail if the test happens to pass.  Prefer the decorator ``@pytest.mark.xfail`` and the argument ``pytest.param``
over usage within a test so that the test is appropriately marked during the
collection phase of pytest. For xfailing a test that involves multiple
parameters, a fixture, or a combination of these, it is only possible to
xfail during the testing phase. To do so, use the ``request`` fixture:  .. code-block:: python  def test_xfail(request):  mark = pytest.mark.xfail(raises=TypeError, reason="Indicate why here")  request.node.add_marker(mark)  xfail is not to be used for tests involving failure due to invalid user arguments.
For these tests, we need to verify the correct exception type and error message
is being raised, using ``pytest.raises`` instead.  .. _contributing.warnings:  Testing a warning
^^^  Use ``tm.assert_produces_warning`` as a context manager to check that a block of code raises a warning.  .. code-block:: python  with tm.assert_produces_warning(DeprecationWarning):  pd.deprecated_function()  If a warning should specifically not happen in a block of code, pass ``False`` into the context manager.  .. code-block:: python  with tm.assert_produces_warning(False):  pd.no_warning_function()  If you have a test that would emit a warning, but you aren't actually testing the
warning itself (say because it's going to be removed in the future, or because we're
matching a 3rd-party library's behavior), then use ``pytest.mark.filterwarnings`` to
ignore the error.  .. code-block:: python  @pytest.mark.filterwarnings("ignorecategory")  def test_thing(self):  pass  If you need finer-grained control, you can use Python's
warnings module_
to control whether a warning is ignored or raised at different places within
a single test.  .. code-block:: python  with warnings.catch_warnings():  warnings.simplefilter("ignore", FutureWarning)  Testing an exception
^^^  Use pytest.raises as a context manager
with the specific exception subclass (i.e. never use class:`Exception`) and the exception message in ``match``.  .. code-block:: python  with pytest.raises(ValueError, match="an error"):  raise ValueError("an error")  Testing involving files

^^^  The ``tm.ensure_clean`` context manager creates a temporary file for testing,
with a generated filename (or your filename if provided), that is automatically
deleted when the context block is exited.  .. code-block:: python  with tm.ensure_clean('my_file_path') as path:  # do something with the path  Testing involving network connectivity
^^^  It is highly discouraged to add a test that connects to the internet due to flakiness of network connections and
lack of ownership of the server that is being connected to. If network connectivity is absolutely required, use the
``tm.network`` decorator.  .. code-block:: python  @tm.network  # noqa  def test_network():  result = package.call_to_internet()  If the test requires data from a specific website, specify ``check_before_test=True`` and the site in the decorator.  .. code-block:: python  @tm.network("https://www.somespecificsite.com", check_before_test=True)  def test_network():  result = pd.read_html("https://www.somespecificsite.com")  Example
^^^  Here is an example of a self-contained set of tests in a file ``pandas/tests/test_cool_feature.py``
that illustrate multiple features that we like to use. Please remember to add the GitHub Issue Number
as a comment to a new test.  .. code-block:: python  import pytest  import numpy as np  import pandas as pd  @pytest.mark.parametrize('dtype', ['int8', 'int16', 'int32', 'int64'])  def test_dtypes(dtype):  assert str(np.dtype(dtype)) == dtype  @pytest.mark.parametrize(  'dtype', ['float32', pytest.param('int16', marks=pytest.mark.skip),  pytest.param('int32', marks=pytest.mark.xfail(  reason='to show how it works'))])  def test_mark(dtype):  assert str(np.dtype(dtype)) == 'float32'  @pytest.fixture  def series():  return pd.Series([1, 2, 3])  @pytest.fixture(params=['int8', 'int16', 'int32', 'int64'])  def dtype(request):  return request.param  def test_series(series, dtype):  # GH <issue_number>  result = series.astype(dtype)  assert result.dtype == dtype  expected = pd.Series([1, 2, 3], dtype=dtype)  tm.assert_series_equal(result, expected)  A test run of this yields  .. code-block:: shell  ((pandas) bash-3.2$ pytest  test_cool_feature.py  -v  === test session starts ===  platform darwin -- Python 3.6.2, pytest-3.6.0, py-1.4.31, pluggy-0.4.0  collected 11 items  tester.py::test_dtypes[int8] PASSED  tester.py::test_dtypes[int16] PASSED  tester.py::test_dtypes[int32] PASSED  tester.py::test_dtypes[int64] PASSED  tester.py::test_mark[float32] PASSED  tester.py::test_mark[int16] SKIPPED  tester.py::test_mark[int32] xfail  tester.py::test_series[int8] PASSED  tester.py::test_series[int16] PASSED  tester.py::test_series[int32] PASSED  tester.py::test_series[int64] PASSED  Tests that we have ``parametrized`` are now accessible via the test name, for example we could run these with ``-k int8`` to sub-select *only* those tests which match ``int8``.  .. code-block:: shell  ((pandas) bash-3.2$ pytest  test_cool_feature.py  -v -k int8  === test session starts ===  platform darwin -- Python 3.6.2, pytest-3.6.0, py-1.4.31, pluggy-0.4.0  collected 11 items  test_cool_feature.py::test_dtypes[int8] PASSED  test_cool_feature.py::test_series[int8] PASSED  .. _using-hypothesis:  Using ``hypothesis``
~~~  Hypothesis is a library for property-based testing. Instead of explicitly
parametrizing a test, you can describe *all* valid inputs and let Hypothesis
try to find a failing input.  Even better, no matter how many random examples
it tries, Hypothesis always reports a single minimal counterexample to your
assertions - often an example that you would never have thought to test.  See Getting Started with Hypothesis
for more of an introduction, then `refer to the Hypothesis documentation

for details <https://hypothesis.readthedocs.io/en/latest/index.html>`_.  .. code-block:: python  import json  from hypothesis import given, strategies as st  any_json_value = st.deferred(lambda: st.one_of(  st.none(), st.booleans(), st.floats(allow_nan=False), st.text(),  st.lists(any_json_value), st.dictionaries(st.text(), any_json_value)  ))  @given(value=any_json_value)  def test_json_roundtrip(value):  result = json.loads(json.dumps(value))  assert value == result  This test shows off several useful features of Hypothesis, as well as
demonstrating a good use-case: checking properties that should hold over
a large or complicated domain of inputs.  To keep the pandas test suite running quickly, parametrized tests are
preferred if the inputs or logic are simple, with Hypothesis tests reserved
for cases with complex logic or where there are too many combinations of
options or subtle interactions to test (or think of!) all of them.  .. _contributing.running_tests:

Running the test suite

The tests can then be run directly inside your Git clone (without having to
install pandas) by typing::  pytest pandas  .. note::  If a handful of tests don't pass, it may not be an issue with your pandas installation.  Some tests (e.g. some SQLAlchemy ones) require additional setup, others might start  failing because a non-pinned library released a new version, and others might be flaky  if run in parallel. As long as you can import pandas from your locally built version,  your installation is probably fine and you can start contributing!  Often it is worth running only a subset of tests first around your changes before running the
entire suite.  The easiest way to do this is with::  pytest pandas/path/to/test.py -k regex_matching_test_name  Or with one of the following constructs::  pytest pandas/tests/[test-module].py  pytest pandas/tests/[test-module].py::[TestClass]  pytest pandas/tests/[test-module].py::[TestClass]::[test_method]  Using pytest-xdist, which is
included in our 'pandas-dev' environment, one can speed up local testing on
multicore machines. The ``-n`` number flag then can be specified when running
pytest to parallelize a test run across the number of specified cores or auto to
utilize all the available cores on your machine.  .. code-block:: bash  # Utilize 4 cores  pytest -n 4 pandas  # Utilizes all available cores  pytest -n auto pandas  If you'd like to speed things along further a more advanced use of this
command would look like this  .. code-block:: bash  pytest pandas -n 4 -m "not slow and not network and not db and not single_cpu" -r sxX  In addition to the multithreaded performance increase this improves test
speed by skipping some tests using the ``-m`` mark flag:  - slow: any test taking long (think seconds rather than milliseconds)
- network: tests requiring network connectivity
- db: tests requiring a database (mysql or postgres)
- single_cpu: tests that should run on a single cpu only  You might want to enable the following option if it's relevant for you:  - arm_slow: any test taking long on arm64 architecture  These markers are defined in this toml file
, under ``[tool.pytest.ini_options]`` in a list called ``markers``, in case
you want to check if new ones have been created which are of interest to you.  The ``-r`` report flag will display a short summary info (see `pytest
documentation <https://docs.pytest.org/en/4.6.x/usage.html#detailed-summary-report>`_)
. Here we are displaying the number of:  - s: skipped tests
- x: xfailed tests
- X: xpassed tests  The summary is optional and can be removed if you don't need the added
information. Using the parallelization option can significantly reduce the
time it takes to locally run tests before submitting a pull request.  If you require assistance with the results,
which has happened in the past, please set a seed before running the command
and opening a bug report, that way we can reproduce it. Here's an example
for setting a seed on windows  .. code-block:: bash  set PYTHONHASHSEED=314159265  pytest pandas -n 4 -m "not slow and not network and not db and not single_cpu" -r sxX  On Unix use  .. code-block:: bash  export PYTHONHASHSEED=314159265  pytest pandas -n 4 -m "not slow and not network and not db and not single_cpu" -r sxX  For more, see the pytest documentation.  Furthermore one can run  .. code-block:: python  pd.test()  with an imported pandas to run tests similarly.

Running the performance test suite

Performance matters and it is worth considering whether your code has introduced
performance regressions. pandas is in the process of migrating to
asv benchmarks_
to enable easy monitoring of the performance of critical pandas operations.
These benchmarks are all found in the ``pandas/asv_bench`` directory, and the
test results can be found here_.  To use all features of asv, you will need either ``conda`` or
``virtualenv``. For more details please check the `asv installation
webpage <https://asv.readthedocs.io/en/latest/installing.html>`_.  To install asv::  pip install git+https://github.com/airspeed-velocity/asv  If you need to run a benchmark, change your directory to ``asv_bench/`` and run::  asv continuous -f 1.1 upstream/main HEAD  You can replace ``HEAD`` with the name of the branch you are working on,
and report benchmarks that changed by more than 10%.
The command uses ``conda`` by default for creating the benchmark
environments. If you want to use virtualenv instead, write::  asv continuous -f 1.1 -E virtualenv upstream/main HEAD  The ``-E virtualenv`` option should be added to all ``asv`` commands
that run benchmarks. The default value is defined in ``asv.conf.json``.  Running the full benchmark suite can be an all-day process, depending on your
hardware and its resource utilization. However, usually it is sufficient to paste
only a subset of the results into the pull request to show that the committed changes
do not cause unexpected performance regressions.  You can run specific benchmarks
using the ``-b`` flag, which takes a regular expression. For example, this will
only run benchmarks from a ``pandas/asv_bench/benchmarks/groupby.py`` file::  asv continuous -f 1.1 upstream/main HEAD -b ^groupby  If you want to only run a specific group of benchmarks from a file, you can do it
using ``.`` as a separator. For example::  asv continuous -f 1.1 upstream/main HEAD -b groupby.GroupByMethods  will only run the ``GroupByMethods`` benchmark defined in ``groupby.py``.  You can also run the benchmark suite using the version of ``pandas``
already installed in your current Python environment. This can be
useful if you do not have virtualenv or conda, or are using the
``setup.py develop`` approach discussed above; for the in-place build
you need to set ``PYTHONPATH``, e.g.
``PYTHONPATH="$PWD/.." asv [remaining arguments]``.
You can run benchmarks using an existing Python
environment by::  asv run -e -E existing  or, to use a specific Python interpreter,::  asv run -e -E existing:python3.6  This will display stderr from the benchmarks, and use your local
``python`` that comes from your ``$PATH``.  Information on how to write a benchmark and how to use asv can be found in the
asv documentation.

Documenting your code

Changes should be reflected in the release notes located in ``doc/source/whatsnew/vx.y.z.rst``.
This file contains an ongoing change log for each release.  Add an entry to this file to
document your fix, enhancement or (unavoidable) breaking change.  Make sure to include the
GitHub issue number when adding your entry (using ```1234``` where ``1234`` is the
issue/pull request number). Your entry should be written using full sentences and proper
grammar.  When mentioning parts of the API, use a Sphinx ```, ```, or ```
directive as appropriate. Not all public API functions and methods have a
documentation page; ideally links would only be added if they resolve. You can
usually find similar examples by checking the release notes for one of the previous
versions.  If your code is a bugfix, add your entry to the relevant bugfix section. Avoid
adding to the ``Other`` section; only in rare cases should entries go there.
Being as concise as possible, the description of the bug should include how the
user may encounter it and an indication of the bug itself, e.g.
"produces incorrect results" or "incorrectly raises". It may be necessary to also
indicate the new behavior.  If your code is an enhancement, it is most likely necessary to add usage
examples to the existing documentation.  This can be done following the section
regarding `documentation <contributing_documentation>`.
Further, to let users know when this feature was added, the ``versionadded``
directive is used. The sphinx syntax for that is:  .. code-block:: rst  .. versionadded:: 1.1.0  This will put the text *New in version 1.1.0* wherever you put the sphinx
directive. This should also be put in the docstring when adding a new function
or method (example_)
or a new keyword argument (example_).

About docstrings and standards

A Python docstring is a string used to document a Python module, class,
function or method, so programmers can understand what it does without having
to read the details of the implementation.  Also, it is a common practice to generate online (html) documentation
automatically from docstrings. Sphinx serves
this purpose.  The next example gives an idea of what a docstring looks like:  .. code-block:: python  def add(num1, num2):  """  Add up two integer numbers.  This function simply wraps the ``+`` operator, and does not  do anything interesting, except for illustrating what  the docstring of a very simple function looks like.

Parameters
  num1 : int  First number to add.  num2 : int  Second number to add.

Examples
  >>> add(2, 2)  4  >>> add(25, 0)  25  >>> add(10, -10)  0  """  return num1 + num2  Some standards regarding docstrings exist, which make them easier to read, and allow them
be easily exported to other formats such as html or pdf.  The first conventions every Python docstring should follow are defined in
PEP-257.  As PEP-257 is quite broad, other more specific standards also exist. In the
case of pandas, the NumPy docstring convention is followed. These conventions are
explained in this document:  * numpydoc docstring guide  numpydoc is a Sphinx extension to support the NumPy docstring convention.  The standard uses reStructuredText (reST). reStructuredText is a markup
language that allows encoding styles in plain text files. Documentation
about reStructuredText can be found in:  * Sphinx reStructuredText primer
* Quick reStructuredText reference
* Full reStructuredText specification  pandas has some helpers for sharing docstrings between related classes, see
`docstring.sharing`.  The rest of this document will summarize all the above guidelines, and will
provide additional conventions specific to the pandas project.  .. _docstring.tutorial:

.. _docstring.general:  General rules
~~~  Docstrings must be defined with three double-quotes. No blank lines should be
left before or after the docstring. The text starts in the next line after the
opening quotes. The closing quotes have their own line
(meaning that they are not at the end of the last sentence).  On rare occasions reST styles like bold text or italics will be used in
docstrings, but is it common to have inline code, which is presented between
backticks. The following are considered inline code:  * The name of a parameter
* Python code, a module, function, built-in, type, literal... (e.g. ``os``,  ``list``, ``numpy.abs``, ``datetime.date``, ``True``)
* A pandas class (in the form ```pandas.Series```)
* A pandas method (in the form ```pandas.Series.sum```)
* A pandas function (in the form ```pandas.to_datetime```)  .. note::  To display only the last component of the linked class, method or  function, prefix it with ``~``. For example, ```~pandas.Series```  will link to ``pandas.Series`` but only display the last part, ``Series``  as the link text. See `Sphinx cross-referencing syntax  <https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-syntax>`_  for details.  **Good:**  .. code-block:: python  def add_values(arr):  """  Add the values in ``arr``.  This is equivalent to Python ``sum`` of `pandas.Series.sum`.  Some sections are omitted here for simplicity.  """  return sum(arr)  **Bad:**  .. code-block:: python  def func():  """Some function.  With several mistakes in the docstring.  It has a blank like after the signature ``def func():``.  The text 'Some function' should go in the line after the  opening quotes of the docstring, not in the same line.  There is a blank line between the docstring and the first line  of code ``foo = 1``.  The closing quotes should be in the next line, not in this one."""  foo = 1  bar = 2  return foo + bar  .. _docstring.short_summary:  Section 1: short summary
~~~  The short summary is a single sentence that expresses what the function does in
a concise way.  The short summary must start with a capital letter, end with a dot, and fit in
a single line. It needs to express what the object does without providing
details. For functions and methods, the short summary must start with an
infinitive verb.  **Good:**  .. code-block:: python  def astype(dtype):  """  Cast Series type.  This section will provide further details.  """  pass  **Bad:**  .. code-block:: python  def astype(dtype):  """  Casts Series type.  Verb in third-person of the present simple, should be infinitive.  """  pass  .. code-block:: python  def astype(dtype):  """  Method to cast Series type.  Does not start with verb.  """  pass  .. code-block:: python  def astype(dtype):  """  Cast Series type  Missing dot at the end.  """  pass  .. code-block:: python  def astype(dtype):  """  Cast Series type from its current type to the new type defined in  the parameter dtype.  Summary is too verbose and doesn't fit in a single line.  """  pass  .. _docstring.extended_summary:  Section 2: extended summary
~~~  The extended summary provides details on what the function does. It should not
go into the details of the parameters, or discuss implementation notes, which
go in other sections.  A blank line is left between the short summary and the extended summary.
Every paragraph in the extended summary ends with a dot.  The extended summary should provide details on why the function is useful and
their use cases, if it is not too generic.  .. code-block:: python  def unstack():  """  Pivot a row index to columns.  When using a MultiIndex, a level can be pivoted so each value in  the index becomes a column. This is especially useful when a subindex  is repeated for the main index, and data is easier to visualize as a  pivot table.  The index level will be automatically removed from the index when added  as columns.  """  pass  .. _docstring.parameters:  Section 3: parameters

~~~  The details of the parameters will be added in this section. This section has
the title "Parameters", followed by a line with a hyphen under each letter of
the word "Parameters". A blank line is left before the section title, but not
after, and not between the line with the word "Parameters" and the one with
the hyphens.  After the title, each parameter in the signature must be documented, including
``*args`` and ``**kwargs``, but not ``self``.  The parameters are defined by their name, followed by a space, a colon, another
space, and the type (or types). Note that the space between the name and the
colon is important. Types are not defined for ``*args`` and ``**kwargs``, but must
be defined for all other parameters. After the parameter definition, it is
required to have a line with the parameter description, which is indented, and
can have multiple lines. The description must start with a capital letter, and
finish with a dot.  For keyword arguments with a default value, the default will be listed after a
comma at the end of the type. The exact form of the type in this case will be
"int, default 0". In some cases it may be useful to explain what the default
argument means, which can be added after a comma "int, default -1, meaning all
cpus".  In cases where the default value is ``None``, meaning that the value will not be
used. Instead of ``"str, default None"``, it is preferred to write ``"str, optional"``.
When ``None`` is a value being used, we will keep the form "str, default None".
For example, in ``df.to_csv(compression=None)``, ``None`` is not a value being used,
but means that compression is optional, and no compression is being used if not
provided. In this case we will use ``"str, optional"``. Only in cases like
``func(value=None)`` and ``None`` is being used in the same way as ``0`` or ``foo``
would be used, then we will specify "str, int or None, default None".  **Good:**  .. code-block:: python  class Series:  def plot(self, kind, color='blue', **kwargs):  """  Generate a plot.  Render the data in the Series as a matplotlib plot of the  specified kind.

  kind : str  Kind of matplotlib plot.  color : str, default 'blue'  Color name or rgb code.  **kwargs  These parameters will be passed to the matplotlib plotting  function.  """  pass  **Bad:**  .. code-block:: python  class Series:  def plot(self, kind, **kwargs):  """  Generate a plot.  Render the data in the Series as a matplotlib plot of the  specified kind.  Note the blank line between the parameters title and the first  parameter. Also, note that after the name of the parameter ``kind``  and before the colon, a space is missing.  Also, note that the parameter descriptions do not start with a  capital letter, and do not finish with a dot.  Finally, the ``**kwargs`` parameter is missing.

  kind: str  kind of matplotlib plot  """  pass  .. _docstring.parameter_types:  Parameter types
^^^  When specifying the parameter types, Python built-in data types can be used
directly (the Python type is preferred to the more verbose string, integer,
boolean, etc):  * int
* float
* str
* bool  For complex types, define the subtypes. For ``dict`` and ``tuple``, as more than
one type is present, we use the brackets to help read the type (curly brackets
for ``dict`` and normal brackets for ``tuple``):  * list of int
* dict of {str : int}
* tuple of (str, int, int)
* tuple of (str,)
* set of str  In case where there are just a set of values allowed, list them in curly
brackets and separated by commas (followed by a space). If the values are
ordinal and they have an order, list them in this order. Otherwise, list
the default value first, if there is one:  * {0, 10, 25}
* {'simple', 'advanced'}
* {'low', 'medium', 'high'}
* {'cat', 'dog', 'bird'}  If the type is defined in a Python module, the module must be specified:  * datetime.date
* datetime.datetime
* decimal.Decimal  If the type is in a package, the module must be also specified:  * numpy.ndarray
* scipy.sparse.coo_matrix  If the type is a pandas type, also specify pandas except for Series and
DataFrame:  * Series
* DataFrame
* pandas.Index
* pandas.Categorical
* pandas.arrays.SparseArray  If the exact type is not relevant, but must be compatible with a NumPy
array, array-like can be specified. If Any type that can be iterated is
accepted, iterable can be used:  * array-like
* iterable  If more than one type is accepted, separate them by commas, except the
last two types, that need to be separated by the word 'or':  * int or float
* float, decimal.Decimal or None
* str or list of str  If ``None`` is one of the accepted values, it always needs to be the last in
the list.  For axis, the convention is to use something like:  * axis : {0 or 'index', 1 or 'columns', None}, default None  .. _docstring.returns:  Section 4: returns or yields
~~~  If the method returns a value, it will be documented in this section. Also
if the method yields its output.  The title of the section will be defined in the same way as the "Parameters".
With the names "Returns" or "Yields" followed by a line with as many hyphens
as the letters in the preceding word.  The documentation of the return is also similar to the parameters. But in this
case, no name will be provided, unless the method returns or yields more than
one value (a tuple of values).  The types for "Returns" and "Yields" are the same as the ones for the
"Parameters". Also, the description must finish with a dot.  For example, with a single value:  .. code-block:: python  def sample():  """  Generate and return a random number.  The value is sampled from a continuous uniform distribution between  0 and 1.

Returns
  float  Random number generated.  """  return np.random.random()  With more than one value:  .. code-block:: python  import string  def random_letters():  """  Generate and return a sequence of random letters.  The length of the returned string is also random, and is also  returned.

Returns
  length : int  Length of the returned string.  letters : str  String of random letters.  """  length = np.random.randint(1, 10)  letters = ''.join(np.random.choice(string.ascii_lowercase)  for i in range(length))  return length, letters  If the method yields its value:  .. code-block:: python  def sample_values():  """  Generate an infinite sequence of random numbers.  The values are sampled from a continuous uniform distribution between  0 and 1.

Yields
  float  Random number generated.  """  while True:  yield np.random.random()  .. _docstring.see_also:  Section 5: see also
~~~  This section is used to let users know about pandas functionality
related to the one being documented. In rare cases, if no related methods
or functions can be found at all, this section can be skipped.  An obvious example would be the ``head()`` and ``tail()`` methods. As ``tail()`` does
the equivalent as ``head()`` but at the end of the ``Series`` or ``DataFrame``
instead of at the beginning, it is good to let the users know about it.  To give an intuition on what can be considered related, here there are some
examples:  * ``loc`` and ``iloc``, as they do the same, but in one case providing indices  and in the other positions
* ``max`` and ``min``, as they do the opposite
* ``iterrows``, ``itertuples`` and ``items``, as it is easy that a user  looking for the method to iterate over columns ends up in the method to  iterate over rows, and vice-versa
* ``fillna`` and ``dropna``, as both methods are used to handle missing values
* ``read_csv`` and ``to_csv``, as they are complementary
* ``merge`` and ``join``, as one is a generalization of the other
* ``astype`` and ``pandas.to_datetime``, as users may be reading the  documentation of ``astype`` to know how to cast as a date, and the way to do  it is with ``pandas.to_datetime``
* ``where`` is related to ``numpy.where``, as its functionality is based on it  When deciding what is related, you should mainly use your common sense and
think about what can be useful for the users reading the documentation,
especially the less experienced ones.  When relating to other libraries (mainly ``numpy``), use the name of the module
first (not an alias like ``np``). If the function is in a module which is not
the main one, like ``scipy.sparse``, list the full module (e.g.
``scipy.sparse.coo_matrix``).  This section has a header, "See Also" (note the capital
S and A), followed by the line with hyphens and preceded by a blank line.  After the header, we will add a line for each related method or function,
followed by a space, a colon, another space, and a short description that
illustrates what this method or function does, why is it relevant in this
context, and what the key differences are between the documented function and
the one being referenced. The description must also end with a dot.  Note that in "Returns" and "Yields", the description is located on the line
after the type. In this section, however, it is located on the same
line, with a colon in between. If the description does not fit on the same
line, it can continue onto other lines which must be further indented.  For example:  .. code-block:: python  class Series:  def head(self):  """  Return the first 5 elements of the Series.  This function is mainly useful to preview the values of the  Series without displaying the whole of it.

Returns
  Series  Subset of the original series with the 5 first values.

See Also
  Series.tail : Return the last 5 elements of the Series.  Series.iloc : Return a slice of the elements in the Series,  which can also be used to return the first or last n.  """  return self.iloc[:5]  .. _docstring.notes:  Section 6: notes
~~~  This is an optional section used for notes about the implementation of the
algorithm, or to document technical aspects of the function behavior.  Feel free to skip it, unless you are familiar with the implementation of the
algorithm, or you discover some counter-intuitive behavior while writing the
examples for the function.  This section follows the same format as the extended summary section.  .. _docstring.examples:  Section 7: examples
~~~  This is one of the most important sections of a docstring, despite being
placed in the last position, as often people understand concepts better
by example than through accurate explanations.  Examples in docstrings, besides illustrating the usage of the function or
method, must be valid Python code, that returns the given output in a
deterministic way, and that can be copied and run by users.  Examples are presented as a session in the Python terminal. ``>>>`` is used to
present code. ``...`` is used for code continuing from the previous line.
Output is presented immediately after the last line of code generating the
output (no blank lines in between). Comments describing the examples can
be added with blank lines before and after them.  The way to present examples is as follows:  1. Import required libraries (except ``numpy`` and ``pandas``)  2. Create the data required for the example  3. Show a very basic example that gives an idea of the most common use case  4. Add examples with explanations that illustrate how the parameters can be  used for extended functionality  A simple example could be:  .. code-block:: python  class Series:  def head(self, n=5):  """  Return the first elements of the Series.  This function is mainly useful to preview the values of the  Series without displaying all of it.

Return
  pandas.Series  Subset of the original series with the n first values.

Examples
  >>> s = pd.Series(['Ant', 'Bear', 'Cow', 'Dog', 'Falcon',  ...  'Lion', 'Monkey', 'Rabbit', 'Zebra'])  >>> s.head()  0  Ant  1  Bear  2  Cow  3  Dog  4  Falcon  dtype: object  With the ``n`` parameter, we can change the number of returned rows:  >>> s.head(n=3)  0  Ant  1  Bear  2  Cow  dtype: object  """  return self.iloc[:n]  The examples should be as concise as possible. In cases where the complexity of
the function requires long examples, is recommended to use blocks with headers
in bold. Use double star ``**`` to make a text bold, like in ``**this example**``.  .. _docstring.example_conventions:  Conventions for the examples
^^^  Code in examples is assumed to always start with these two lines which are not
shown:  .. code-block:: python  import numpy as np  import pandas as pd  Any other module used in the examples must be explicitly imported, one per line (as
recommended in `8#imports`)
and avoiding aliases. Avoid excessive imports, but if needed, imports from
the standard library go first, followed by third-party libraries (like
matplotlib).  When illustrating examples with a single ``Series`` use the name ``s``, and if
illustrating with a single ``DataFrame`` use the name ``df``. For indices,
``idx`` is the preferred name. If a set of homogeneous ``Series`` or
``DataFrame`` is used, name them ``s1``, ``s2``, ``s3``...  or ``df1``,
``df2``, ``df3``... If the data is not homogeneous, and more than one structure
is needed, name them with something meaningful, for example ``df_main`` and
``df_to_join``.  Data used in the example should be as compact as possible. The number of rows
is recommended to be around 4, but make it a number that makes sense for the
specific example. For example in the ``head`` method, it requires to be higher
than 5, to show the example with the default values. If doing the ``mean``, we
could use something like ``[1, 2, 3]``, so it is easy to see that the value
returned is the mean.  For more complex examples (grouping for example), avoid using data without
interpretation, like a matrix of random numbers with columns A, B, C, D...
And instead use a meaningful example, which makes it easier to understand the
concept. Unless required by the example, use names of animals, to keep examples
consistent. And numerical properties of them.  When calling the method, keywords arguments ``head(n=3)`` are preferred to
positional arguments ``head(3)``.  **Good:**  .. code-block:: python  class Series:  def mean(self):  """  Compute the mean of the input.

Examples
  >>> s = pd.Series([1, 2, 3])  >>> s.mean()  2  """  pass  def fillna(self, value):  """  Replace missing values by ``value``.

Examples
  >>> s = pd.Series([1, np.nan, 3])  >>> s.fillna(0)  [1, 0, 3]  """  pass  def groupby_mean(self):  """  Group by index and return mean.

Examples
  >>> s = pd.Series([380., 370., 24., 26],  ...  name='max_speed',  ...  index=['falcon', 'falcon', 'parrot', 'parrot'])  >>> s.groupby_mean()  index  falcon  375.0  parrot  25.0  Name: max_speed, dtype: float64  """  pass  def contains(self, pattern, case_sensitive=True, na=numpy.nan):  """  Return whether each value contains ``pattern``.  In this case, we are illustrating how to use sections, even  if the example is simple enough and does not require them.

Examples
  >>> s = pd.Series('Antelope', 'Lion', 'Zebra', np.nan)  >>> s.contains(pattern='a')  0  False  1  False  2  True  3  NaN  dtype: bool  **Case sensitivity**  With ``case_sensitive`` set to ``False`` we can match ``a`` with both  ``a`` and ``A``:  >>> s.contains(pattern='a', case_sensitive=False)  0  True  1  False  2  True  3  NaN  dtype: bool  **Missing values**  We can fill missing values in the output using the ``na`` parameter:  >>> s.contains(pattern='a', na=False)  0  False  1  False  2  True  3  False  dtype: bool  """  pass  **Bad:**  .. code-block:: python  def method(foo=None, bar=None):  """  A sample DataFrame method.  Do not import NumPy and pandas.  Try to use meaningful data, when it makes the example easier  to understand.  Try to avoid positional arguments like in ``df.method(1)``. They  can be all right if previously defined with a meaningful name,  like in ``present_value(interest_rate)``, but avoid them otherwise.  When presenting the behavior with different parameters, do not place  all the calls one next to the other. Instead, add a short sentence  explaining what the example shows.

Examples
  >>> import numpy as np  >>> import pandas as pd  >>> df = pd.DataFrame(np.random.randn(3, 3),  ...  columns=('a', 'b', 'c'))  >>> df.method(1)  21  >>> df.method(bar=14)  123  """  pass  .. _docstring.doctest_tips:  Tips for getting your examples pass the doctests
^^^  Getting the examples pass the doctests in the validation script can sometimes
be tricky. Here are some attention points:  * Import all needed libraries (except for pandas and NumPy, those are already  imported as ``import pandas as pd`` and ``import numpy as np``) and define  all variables you use in the example.  * Try to avoid using random data. However random data might be OK in some  cases, like if the function you are documenting deals with probability  distributions, or if the amount of data needed to make the function result  meaningful is too much, such that creating it manually is very cumbersome.  In those cases, always use a fixed random seed to make the generated examples  predictable. Example::  >>> np.random.seed(42)  >>> df = pd.DataFrame({'normal': np.random.normal(100, 5, 20)})  * If you have a code snippet that wraps multiple lines, you need to use '...'  on the continued lines: ::  >>> df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], index=['a', 'b', 'c'],  ...  columns=['A', 'B'])  * If you want to show a case where an exception is raised, you can do::  >>> pd.to_datetime(["712-01-01"])  Traceback (most recent call last):  OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 712-01-01 000  It is essential to include the "Traceback (most recent call last):", but for  the actual error only the error name is sufficient.  * If there is a small part of the result that can vary (e.g. a hash in an object  representation), you can use `  >>> s.plot()  <matplotlib.axes._subplots.AxesSubplot at 0x7efd0c0b0690>  However, you can do (notice the comment that needs to be added) ::  >>> s.plot()  # doctest: +ELLIPSIS  <matplotlib.axes._subplots.AxesSubplot at ...>  .. _docstring.example_plots:  Plots in examples
^^^  There are some methods in pandas returning plots. To render the plots generated
by the examples in the documentation, the ``` directive exists.  To use it, place the next code after the "Examples" header as shown below. The
plot will be generated automatically when building the documentation.  .. code-block:: python  class Series:  def plot(self):  """  Generate a plot with the ``Series`` data.

Examples
  .. plot::  close-figs  >>> s = pd.Series([1, 2, 3])  >>> s.plot()  """  pass  .. _docstring.sharing:

Sharing docstrings

pandas has a system for sharing docstrings, with slight variations, between
classes. This helps us keep docstrings consistent, while keeping things clear
for the user reading. It comes at the cost of some complexity when writing.  Each shared docstring will have a base template with variables, like
``{klass}``. The variables filled in later on using the ``doc`` decorator.
Finally, docstrings can also be appended to with the ``doc`` decorator.  In this example, we'll create a parent docstring normally (this is like
``pandas.core.generic.NDFrame``. Then we'll have two children (like
``pandas.core.series.Series`` and ``pandas.core.frame.DataFrame``). We'll
substitute the class names in this docstring.  .. code-block:: python  class Parent:  @doc(klass="Parent")  def my_function(self):  """Apply my function to {klass}."""  ...  class ChildA(Parent):  @doc(Parent.my_function, klass="ChildA")  def my_function(self):  ...  class ChildB(Parent):  @doc(Parent.my_function, klass="ChildB")  def my_function(self):  ...  The resulting docstrings are  .. code-block:: python  >>> print(Parent.my_function.__doc__)  Apply my function to Parent.  >>> print(ChildA.my_function.__doc__)  Apply my function to ChildA.  >>> print(ChildB.my_function.__doc__)  Apply my function to ChildB.  Notice:  1. We "append" the parent docstring to the children docstrings, which are  initially empty.  Our files will often contain a module-level ``_shared_doc_kwargs`` with some
common substitution values (things like ``klass``, ``axes``, etc).  You can substitute and append in one shot with something like  .. code-block:: python  @doc(template, **_shared_doc_kwargs)  def my_function(self):  ...  where ``template`` may come from a module-level ``_shared_docs`` dictionary
mapping function names to docstrings. Wherever possible, we prefer using
``doc``, since the docstring-writing processes is slightly closer to normal.  See ``pandas.core.generic.NDFrame.fillna`` for an example template, and
``pandas.core.series.Series.fillna`` and ``pandas.core.generic.frame.fillna``
for the filled versions.

Contributing to the documentation

Contributing to the documentation benefits everyone who uses pandas.
We encourage you to help us improve the documentation, and
you don't have to be an expert on pandas to do so! In fact,
there are sections of the docs that are worse off after being written by
experts. If something in the docs doesn't make sense to you, updating the
relevant section after you figure it out is a great way to ensure it will help
the next person. Please visit the issues page_
for a full list of issues that are currently open regarding the
Pandas documentation.  .. contents:: Documentation:

About the pandas documentation

The documentation is written in **reStructuredText**, which is almost like writing
in plain English, and built using Sphinx_. The
Sphinx Documentation has an excellent `introduction to reST
<https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`__. Review the Sphinx docs to perform more
complex changes to the documentation as well.  Some other important things to know about the docs:  * The pandas documentation consists of two parts: the docstrings in the code  itself and the docs in this folder ``doc/``.  The docstrings provide a clear explanation of the usage of the individual  functions, while the documentation in this folder consists of tutorial-like  overviews per topic together with some other information (what's new,  installation, etc).  * The docstrings follow a pandas convention, based on the **Numpy Docstring  Standard**. Follow the `pandas docstring guide <docstring>` for detailed  instructions on how to write a correct docstring.  .. toctree::  2  contributing_docstring.rst  * The tutorials make heavy use of the `IPython directive  <https://matplotlib.org/sampledoc/ipython_directive.html>`_ sphinx extension.  This directive lets you put code in the documentation which will be run  during the doc build. For example::  .. ipython:: python  x = 2  x**3  will be rendered as::  In [1]: x = 2  In [2]: x**3  Out[2]: 8  Almost all code examples in the docs are run (and the output saved) during the  doc build. This approach means that code examples will always be up to date,  but it does make the doc building a bit more complex.  * Our API documentation files in ``doc/source/reference`` house the auto-generated  documentation from the docstrings. For classes, there are a few subtleties  around controlling which methods and attributes have pages auto-generated.  We have two autosummary templates for classes.  1. ``_templates/autosummary/class.rst``. Use this when you want to  automatically generate a page for every public method and attribute on the  class. The ``Attributes`` and ``Methods`` sections will be automatically  added to the class' rendered documentation by numpydoc. See ``DataFrame``  for an example.  2. ``_templates/autosummary/class_without_autosummary``. Use this when you  want to pick a subset of methods / attributes to auto-generate pages for.  When using this template, you should include an ``Attributes`` and  ``Methods`` section in the class docstring. See ``CategoricalIndex`` for an  example.  Every method should be included in a ``toctree`` in one of the documentation files in  ``doc/source/reference``, else Sphinx  will emit a warning.  The utility script ``scripts/validate_docstrings.py`` can be used to get a csv
summary of the API documentation. And also validate common errors in the docstring
of a specific class, function or method. The summary also compares the list of
methods documented in the files in ``doc/source/reference`` (which is used to generate
the API Reference page)
and the actual public methods.
This will identify methods documented in ``doc/source/reference`` that are not actually
class methods, and existing methods that are not documented in ``doc/source/reference``.

Updating a pandas docstring

When improving a single function or method's docstring, it is not necessarily
needed to build the full documentation (see next section).
However, there is a script that checks a docstring (for example for the ``DataFrame.mean`` method)::  python scripts/validate_docstrings.py pandas.DataFrame.mean  This script will indicate some formatting errors if present, and will also
run and test the examples included in the docstring.
Check the `pandas docstring guide <docstring>` for a detailed guide
on how to format the docstring.  The examples in the docstring ('doctests') must be valid Python code,
that in a deterministic way returns the presented output, and that can be
copied and run by users. This can be checked with the script above, and is
also tested on Travis. A failing doctest will be a blocker for merging a PR.
Check the `examples <docstring.examples>` section in the docstring guide
for some tips and tricks to get the doctests passing.  When doing a PR with a docstring update, it is good to post the
output of the validation script in a comment on github.  .. _contributing.howto-build-docs:

How to build the pandas documentation

Requirements
~~~  First, you need to have a development environment to be able to build pandas
(see the docs on `creating a development environment <contributing_environment>`).  Building the documentation
~~~  So how do you build the docs? Navigate to your local
``doc/`` directory in the console and run::  python make.py html  Then you can find the HTML output in the folder ``doc/build/html/``.  The first time you build the docs, it will take quite a while because it has to run
all the code examples and build all the generated docstring pages. In subsequent
evocations, sphinx will try to only build the pages that have been modified.  If you want to do a full clean build, do::  python make.py clean  python make.py html  You can tell ``make.py`` to compile only a single section of the docs, greatly
reducing the turn-around time for checking your changes.  ::  # omit autosummary and API section  python make.py clean  python make.py --no-api  # compile the docs with only a single section, relative to the "source" folder.  # For example, compiling only this guide (doc/source/development/contributing.rst)  python make.py clean  python make.py --single development/contributing.rst  # compile the reference docs for a single function  python make.py clean  python make.py --single pandas.DataFrame.join  # compile whatsnew and API section (to resolve links in the whatsnew)  python make.py clean  python make.py --whatsnew  For comparison, a full documentation build may take 15 minutes, but a single
section may take 15 seconds. Subsequent builds, which only process portions
you have changed, will be faster.  The build will automatically use the number of cores available on your machine
to speed up the documentation build. You can override this::  python make.py html --num-jobs 4  Open the following file in a web browser to see the full documentation you
just built::  doc/build/html/index.html  And you'll have the satisfaction of seeing your new and improved documentation!  .. _contributing.dev_docs:  Building main branch documentation
~~~  When pull requests are merged into the pandas ``main`` branch, the main parts of
the documentation are also built by Travis-CI. These docs are then hosted `here
<https://pandas.pydata.org/docs/dev/>`__, see also
the `Continuous Integration <contributing.ci>` section.

Previewing changes

Once, the pull request is submitted, GitHub Actions will automatically build the
documentation. To view the built site:  #. Wait for the ``CI / Web and docs`` check to complete.
#. Click ``Details`` next to it.
#. From the ``Artifacts`` drop-down, click ``docs`` or ``website`` to download  the site as a ZIP file.

Creating a development environment

To test out code changes, you'll need to build pandas from source, which
requires a C/C++ compiler and Python environment. If you're making documentation
changes, you can skip to `contributing to the documentation <contributing_documentation>` but if you skip
creating the development environment you won't be able to build the documentation
locally before pushing your changes. It's recommended to also install the `pre-commit hooks <contributing.pre-commit>`.  .. toctree::  2  contributing_gitpod.rst

Step 1: install a C compiler

How to do this will depend on your platform. If you choose to user ``Docker``
in the next step, then you can skip this step.  **Windows**  You will need `Build Tools for Visual Studio 2022
<https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2022>`_.  .. note::  You DO NOT need to install Visual Studio 2022.  You only need "Build Tools for Visual Studio 2022" found by  scrolling down to "All downloads" -> "Tools for Visual Studio".  In the installer, select the "Desktop development with C++" Workloads.  Alternatively, you can install the necessary components on the commandline using
vs_BuildTools.exe  Alternatively, you could use the WSL
and consult the ``Linux`` instructions below.  **macOS**  To use the `mamba <contributing.mamba>`-based compilers, you will need to install the
Developer Tools using ``xcode-select --install``. Otherwise
information about compiler installation can be found here:
https://devguide.python.org/setup/#macos  **Linux**  For Linux-based `mamba <contributing.mamba>` installations, you won't have to install any
additional components outside of the mamba environment. The instructions
below are only needed if your setup isn't based on mamba environments.  Some Linux distributions will come with a pre-installed C compiler. To find out
which compilers (and versions) are installed on your system::  # for Debian/Ubuntu:  dpkg --list | grep compiler  # for Red Hat/RHEL/CentOS/Fedora:  yum list installed | grep -i --color compiler  GCC (GNU Compiler Collection), is a widely used
compiler, which supports C and a number of other languages. If GCC is listed
as an installed compiler nothing more is required.  If no C compiler is installed, or you wish to upgrade, or you're using a different
Linux distribution, consult your favorite search engine for compiler installation/update
instructions.  Let us know if you have any difficulties by opening an issue or reaching out on our contributor
community `Slack <community.slack>`.

Step 2: create an isolated environment

Before we begin, please:  * Make sure that you have `cloned the repository <contributing.forking>`
* ``cd`` to the pandas source directory you just created with the clone command  .. _contributing.mamba:  Option 1: using mamba (recommended)
~~~  * Install mamba
* Make sure your mamba is up to date (``mamba update mamba``)  .. code-block:: none  # Create and activate the build environment  mamba env create --file environment.yml  mamba activate pandas-dev  Option 2: using pip
~~~  You'll need to have at least the `minimum Python version <install.version>` that pandas supports.
You also need to have ``setuptools`` 51.0.0 or later to build pandas.  **Unix**/**macOS with virtualenv**  .. code-block:: bash  # Create a virtual environment  # Use an ENV_DIR of your choice. We'll use ~/virtualenvs/pandas-dev  # Any parent directories should already exist  python3 -m venv ~/virtualenvs/pandas-dev  # Activate the virtualenv  . ~/virtualenvs/pandas-dev/bin/activate  # Install the build dependencies  python -m pip install -r requirements-dev.txt  **Unix**/**macOS with pyenv**  Consult the docs for setting up pyenv here_.  .. code-block:: bash  # Create a virtual environment  # Use an ENV_DIR of your choice. We'll use ~/Users/<yourname>/.pyenv/versions/pandas-dev  pyenv virtualenv <version> <name-to-give-it>  # For instance:  pyenv virtualenv 3.9.10 pandas-dev  # Activate the virtualenv  pyenv activate pandas-dev  # Now install the build dependencies in the cloned pandas repo  python -m pip install -r requirements-dev.txt  **Windows**  Below is a brief overview on how to set-up a virtual environment with Powershell
under Windows. For details please refer to the
official virtualenv user guide_.  Use an ENV_DIR of your choice. We'll use ``~\\virtualenvs\\pandas-dev`` where
``~`` is the folder pointed to by either ``$env:USERPROFILE`` (Powershell) or
``%USERPROFILE%`` (cmd.exe) environment variable. Any parent directories
should already exist.  .. code-block:: powershell  # Create a virtual environment  python -m venv $env:USERPROFILE\virtualenvs\pandas-dev  # Activate the virtualenv. Use activate.bat for cmd.exe  ~\virtualenvs\pandas-dev\Scripts\Activate.ps1  # Install the build dependencies  python -m pip install -r requirements-dev.txt  Option 3: using Docker
~~~  pandas provides a ``DockerFile`` in the root directory to build a Docker image
with a full pandas development environment.  **Docker Commands**  Build the Docker image::  # Build the image  docker build -t pandas-dev .  Run Container::  # Run a container and bind your local repo to the container  # This command assumes you are running from your local repo  # but if not alter ${PWD} to match your local repo path  docker run -it --rm -v ${PWD}:/home/pandas pandas-dev  *Even easier, you can integrate Docker with the following IDEs:*  **Visual Studio Code**  You can use the DockerFile to launch a remote session with Visual Studio Code,
a popular free IDE, using the ``.devcontainer.json`` file.
See https://code.visualstudio.com/docs/remote/containers for details.  **PyCharm (Professional)**  Enable Docker support and use the Services tool window to build and manage images as well as
run and interact with containers.
See https://www.jetbrains.com/help/pycharm/docker.html for details.  Option 4: using Gitpod
~~~  Gitpod is an open-source platform that automatically creates the correct development
environment right in your browser, reducing the need to install local development
environments and deal with incompatible dependencies.  If you are a Windows user, unfamiliar with using the command line or building pandas
for the first time, it is often faster to build with Gitpod. Here are the in-depth instructions
for `building pandas with GitPod <contributing-gitpod>`.

Step 3: build and install pandas

You can now run::  # Build and install pandas  python setup.py build_ext -j 4  python -m pip install -e . --no-build-isolation --no-use-pep517  At this point you should be able to import pandas from your locally built version::  $ python  >>> import pandas  >>> print(pandas.__version__)  # note: the exact output may differ  2.0.0.dev0+880.g2b9e661fbb.dirty  This will create the new environment, and not touch any of your existing environments,
nor any existing Python installation.  .. note::  You will need to repeat this step each time the C extensions change, for example  if you modified any file in ``pandas/_libs`` or if you did a fetch and merge from ``upstream/main``.

Using Gitpod for pandas development

This section of the documentation will guide you through:  *  using Gitpod for your pandas development environment
*  creating a personal fork of the pandas repository on GitHub
*  a quick tour of pandas and VSCode
*  working on the pandas documentation in Gitpod

Gitpod

`Gitpod`_  is an open-source platform for automated and ready-to-code
development environments. It enables developers to describe their dev
environment as code and start instant and fresh development environments for
each new task directly from your browser. This reduces the need to install local
development environments and deal with incompatible dependencies.

Gitpod GitHub integration

To be able to use Gitpod, you will need to have the Gitpod app installed on your
GitHub account, so if
you do not have an account yet, you will need to create one first.  To get started just login at `Gitpod`_, and grant the appropriate permissions to GitHub.  We have built a python 3.8 environment and all development dependencies will
install when the environment starts.

Forking the pandas repository

The best way to work on pandas as a contributor is by making a fork of the
repository first.  #. Browse to the `pandas repository on GitHub`_ and `create your own fork`_.  #. Browse to your fork. Your fork will have a URL like  https://github.com/noatamir/pandas-dev, except with your GitHub username in place of  ``noatamir``.

Starting Gitpod
Once you have authenticated to Gitpod through GitHub, you can install the
Gitpod Chromium or Firefox browser extension
which will add a **Gitpod** button next to the **Code** button in the
repository:  pandas repository with Gitpod button screenshot  #. If you install the extension - you can click the **Gitpod** button to start  a new workspace.  #. Alternatively, if you do not want to install the browser extension, you can  visit https://gitpod.io/#https://github.com/USERNAME/pandas replacing  ``USERNAME`` with your GitHub username.  #. In both cases, this will open a new tab on your web browser and start  building your development environment. Please note this can take a few  minutes.  #. Once the build is complete, you will be directed to your workspace,  including the VSCode editor and all the dependencies you need to work on  pandas. The first time you start your workspace, you will notice that there  might be some actions running. This will ensure that you have a development  version of pandas installed.  #. When your workspace is ready, you can `test the build<contributing.running_tests>` by  entering::  $ python -m pytest pandas

Quick workspace tour
Gitpod uses VSCode as the editor. If you have not used this editor before, you
can check the Getting started `VSCode docs`_ to familiarize yourself with it.  Your workspace will look similar to the image below:  Gitpod workspace screenshot  We have marked some important sections in the editor:  #. Your current Python interpreter - by default, this is ``pandas-dev`` and  should be displayed in the status bar and on your terminal. You do not need  to activate the conda environment as this will always be activated for you.
#. Your current branch is always displayed in the status bar. You can also use  this button to change or create branches.
#. GitHub Pull Requests extension - you can use this to work with Pull Requests  from your workspace.
#. Marketplace extensions - we have added some essential extensions to the pandas  Gitpod. Still, you can also install other extensions or syntax highlighting  themes for your user, and these will be preserved for you.
#. Your workspace directory - by default, it is ``/workspace/pandas-dev``. **Do not  change this** as this is the only directory preserved in Gitpod.  We have also pre-installed a few tools and VSCode extensions to help with the
development experience:  *  VSCode rst extension
*  Markdown All in One
*  VSCode Gitlens extension
*  VSCode Git Graph extension

Development workflow with Gitpod
The  `contributing` section of this documentation contains
information regarding the pandas development workflow. Make sure to check this
before working on your contributions.  When using Gitpod, git is pre configured for you:  #. You do not need to configure your git username, and email as this should be  done for you as you authenticated through GitHub. Unless you are using GitHub  feature to keep email address private. You can check the git  configuration with the command ``git config --list`` in your terminal. Use  ``git config --global user.email “your-secret-email@users.noreply.github.com”``  to set your email address to the one you use to make commits with your github  profile.
#. As you started your workspace from your own pandas fork, you will by default  have both ``upstream`` and ``origin`` added as remotes. You can verify this by  typing ``git remote`` on your terminal or by clicking on the **branch name**  on the status bar (see image below).  Gitpod workspace branches plugin screenshot

Rendering the pandas documentation
You can find the detailed documentation on how rendering the documentation with
Sphinx works in the `contributing.howto-build-docs` section. To build the full
docs you need to run the following command in the docs directory::  $ cd docs  $ python make.py html  Alternatively you can build a single page with::  python make.py --single development/contributing_gitpod.rst  You have two main options to render the documentation in Gitpod.  Option 1: using Liveserve
~~~  #. View the documentation in ``pandas/doc/build/html``.
#. To see the rendered version of a page, you can right-click on the ``.html``  file and click on **Open with Live Serve**. Alternatively, you can open the  file in the editor and click on the **Go live** button on the status bar.  Gitpod workspace VSCode start live serve screenshot  #. A simple browser will open to the right-hand side of the editor. We recommend  closing it and click on the **Open in browser** button in the pop-up.
#. To stop the server click on the **Port: 5500** button on the status bar.  Option 2: using the rst extension
~~~  A quick and easy way to see live changes in a ``.rst`` file as you work on it
uses the rst extension with docutils.  .. note:: This will generate a simple live preview of the document without the  ``html`` theme, and some backlinks might not be added correctly. But it is an  easy and lightweight way to get instant feedback on your work, without  building the html files.  #. Open any of the source documentation files located in ``doc/source`` in the  editor.
#. Open VSCode Command Palette with `Cmd-Shift-P` in Mac or  `Ctrl-Shift-P` in Linux and Windows. Start typing "restructured"  and choose either "Open preview" or "Open preview to the Side".  Gitpod workspace VSCode open rst screenshot  #. As you work on the document, you will see a live rendering of it on the editor.  Gitpod workspace VSCode rst rendering screenshot  If you want to see the final output with the ``html`` theme you will need to
rebuild the docs with ``make html`` and use Live Serve as described in option 1.

FAQ's and troubleshooting

How long is my Gitpod workspace kept for?
~~~  Your stopped workspace will be kept for 14 days and deleted afterwards if you do
not use them.  Can I come back to a previous workspace?
~~~  Yes, let's say you stepped away for a while and you want to carry on working on
your pandas contributions. You need to visit https://gitpod.io/workspaces and
click on the workspace you want to spin up again. All your changes will be there
as you last left them.  Can I install additional VSCode extensions?
~~~  Absolutely! Any extensions you installed will be installed in your own workspace
and preserved.  I registered on Gitpod but I still cannot see a ``Gitpod`` button in my repositories.
~~~  Head to https://gitpod.io/integrations and make sure you are logged in.
Hover over GitHub and click on the three buttons that appear on the right.
Click on edit permissions and make sure you have ``user:email``,
``read:user``, and ``public_repo`` checked. Click on **Update Permissions**
and confirm the changes in the GitHub application page.  Gitpod integrations - edit GH permissions screenshot  How long does my workspace stay active if I'm not using it?
~~~  If you keep your workspace open in a browser tab but don't interact with it,
it will shut down after 30 minutes. If you close the browser tab, it will
shut down after 3 minutes.  My terminal is blank - there is no cursor and it's completely unresponsive
~~~  Unfortunately this is a known-issue on Gitpod's side. You can sort this
issue in two ways:  #. Create a new Gitpod workspace altogether.
#. Head to your Gitpod dashboard and locate  the running workspace. Hover on it and click on the **three dots menu**  and then click on **Stop**. When the workspace is completely stopped you  can click on its name to restart it again.  Gitpod dashboard and workspace menu screenshot  I authenticated through GitHub but I still cannot commit to the repository through Gitpod.
Click on edit permissions and make sure you have ``public_repo`` checked.
Click on **Update Permissions** and confirm the changes in the
GitHub application page.  Gitpod integrations - edit GH repository permissions screenshot

Acknowledgments

This page is lightly adapted from the `NumPy`_ project .  .. _Gitpod: https://www.gitpod.io/
.. _pandas repository on GitHub: https://github.com/pandas-dev/pandas
.. _create your own fork: https://help.github.com/en/articles/fork-a-repo
.. _VSCode docs: https://code.visualstudio.com/docs/getstarted/tips-and-tricks
.. _NumPy: https://www.numpy.org/

Reference tracking

To be able to determine, if we have to make a copy when writing into a DataFrame,
we have to be aware, if the values are shared with another DataFrame. pandas
keeps track of all ``Blocks`` that share values with another block internally to
be able to tell when a copy needs to be triggered. The reference tracking
mechanism is implemented on the Block level.  We use a custom reference tracker object, ``BlockValuesRefs``, that keeps
track of every block, whose values share memory with each other. The reference
is held through a weak-reference. Every two blocks that share some memory should
point to the same ``BlockValuesRefs`` object. If one block goes out of
scope, the reference to this block dies. As a consequence, the reference tracker
object always knows how many blocks are alive and share memory.  Whenever a `DataFrame` or `Series` object is sharing data with another
object, it is required that each of those objects have its own BlockManager and Block
objects. Thus, in other words, one Block instance (that is held by a DataFrame, not
necessarily for intermediate objects) should always be uniquely used for only
a single DataFrame/Series object. For example, when you want to use the same
Block for another object, you can create a shallow copy of the Block instance
with ``block.copy(deep=False)`` (which will create a new Block instance with
the same underlying values and which will correctly set up the references).  We can ask the reference tracking object if there is another block alive that shares
data with us before writing into the values. We can trigger a copy before
writing if there is in fact another block alive.

Debugging C extensions

Pandas uses select C extensions for high performance IO operations. In case you need to debug segfaults or general issues with those extensions, the following steps may be helpful.  First, be sure to compile the extensions with the appropriate flags to generate debug symbols and remove optimizations. This can be achieved as follows:  .. code-block:: sh  python setup.py build_ext --inplace -j4 --with-debugging-symbols

Using a debugger

Assuming you are on a Unix-like operating system, you can use either lldb or gdb to debug. The choice between either is largely dependent on your compilation toolchain - typically you would use lldb if using clang and gdb if using gcc. For macOS users, please note that ``gcc`` is on modern systems an alias for ``clang``, so if using Xcode you usually opt for lldb. Regardless of which debugger you choose, please refer to your operating systems instructions on how to install.  After installing a debugger you can create a script that hits the extension module you are looking to debug. For demonstration purposes, let's assume you have a script called ``debug_testing.py`` with the following contents:  .. code-block:: python  import pandas as pd  pd.DataFrame([[1, 2]]).to_json()  Place the ``debug_testing.py`` script in the project root and launch a Python process under your debugger. If using lldb:  .. code-block:: sh  lldb python  If using gdb:  .. code-block:: sh  gdb python  Before executing our script, let's set a breakpoint in our JSON serializer in its entry function called ``objToJSON``. The lldb syntax would look as follows:  .. code-block:: sh  breakpoint set --name objToJSON  Similarly for gdb:  .. code-block:: sh  break objToJSON  .. note::  You may get a warning that this breakpoint cannot be resolved in lldb. gdb may give a similar warning and prompt you to make the breakpoint on a future library load, which you should say yes to. This should only happen on the very first invocation as the module you wish to debug has not yet been loaded into memory.  Now go ahead and execute your script:  .. code-block:: sh  run <the_script>.py  Code execution will halt at the breakpoint defined or at the occurrence of any segfault. LLDB's GDB to LLDB command map provides a listing of debugger command that you can execute using either debugger.  Another option to execute the entire test suite under lldb would be to run the following:  .. code-block:: sh  lldb -- python -m pytest  Or for gdb  .. code-block:: sh  gdb --args python -m pytest  Once the process launches, simply type ``run`` and the test suite will begin, stopping at any segmentation fault that may occur.

Improve debugger printing

By default your debug will simply print the type and memory address of a PyObject. Assuming we passed a list containing ``["a", "b"]`` as an argument to a Cython-generated function with parameter ``obj``, debugging that object would look as follows:  .. code-block:: sh  (gdb) p __pyx_v_obj  $1 = (PyObject *) 0x5558b91e0  Dereferencing this will yield the standard PyObject struct members of the object, which provides some more visibility  .. code-block:: sh  (gdb) p *__pyx_v_obj  $2 = {ob_refcnt = 1, ob_type = 0x5558b91e0 <PyList_Type>}  If you are using gdb, CPython provides an extension that prints out more useful information about the object you are inspecting. The extension can be found in cpython/Tools/gdb/libpython.py; for best results be sure to use the gdb extension from the CPython branch that matches the version of your interpreter.  To activate the extension you will need to execute ``source <path_to_cpython_source>/Tools/gdb/libpython.py`` from an actively-running gdb session. After loading you will get more detailed information about the Python object you are inspecting.  .. code-block:: sh  (gdb) p __pyx_v_obj  $3 = ['a', 'b']  If you do not wish to explicitly source this file on every gdb run, you can alternately add it as a start up command to your gdbinit file.

Checking memory leaks with valgrind

You can use Valgrind to check for and log memory leaks in extensions. For instance, to check for a memory leak in a test from the suite you can run:  .. code-block:: sh  PYTHONMALLOC=malloc valgrind --leak-check=yes --track-origins=yes --log-file=valgrind-log.txt python -m pytest <path_to_a_test>  Note that code execution under valgrind will take much longer than usual. While you can run valgrind against extensions compiled with any optimization level, it is suggested to have optimizations turned off from compiled extensions to reduce the amount of false positives. The ``--with-debugging-symbols`` flag passed during package setup will do this for you automatically.  .. note::  For best results, you should run use a Python installation configured with Valgrind support (--with-valgrind)

Easier code navigation

Generating a `compile_commands.json`` file may make it easier to navigate the C extensions, as this allows your code editor to list references, jump to definitions, etc... To make this work with setuptools you can use `Bear.  .. code-block:: sh  bear -- python setup.py build_ext --inplace -j4 --with-debugging-symbols

The Apache Parquet_ format
provides key-value metadata at the file and column level, stored in the footer
of the Parquet file:  .. code-block:: shell  5: optional list<KeyValue> key_value_metadata  where ``KeyValue`` is  .. code-block:: shell  struct KeyValue {  1: required string key  2: optional string value  }  So that a ``pandas.DataFrame`` can be faithfully reconstructed, we store a
``pandas`` metadata key in the ``FileMetaData`` with the value stored as :  .. code-block:: text  {'index_columns': [<descr0>, <descr1>, ...],  'column_indexes': [<ci0>, <ci1>, ..., <ciN>],  'columns': [<c0>, <c1>, ...],  'pandas_version': $VERSION,  'creator': {  'library': $LIBRARY,  'version': $LIBRARY_VERSION  }}  The "descriptor" values ``<descr0>`` in the ``'index_columns'`` field are
strings (referring to a column) or dictionaries with values as described below.  The ``<c0>``/``<ci0>`` and so forth are dictionaries containing the metadata
for each column, *including the index columns*. This has JSON form:  .. code-block:: text  {'name': column_name,  'field_name': parquet_column_name,  'pandas_type': pandas_type,  'numpy_type': numpy_type,  'metadata': metadata}  See below for the detailed specification for these.  Index metadata descriptors
~~~  ``RangeIndex`` can be stored as metadata only, not requiring serialization. The
descriptor format for these as is follows:  .. code-block:: python  index = pd.RangeIndex(0, 10, 2)  {  "kind": "range",  "name": index.name,  "start": index.start,  "stop": index.stop,  "step": index.step,  }  Other index types must be serialized as data columns along with the other
DataFrame columns. The metadata for these is a string indicating the name of
the field in the data columns, for example ``'__index_level_0__'``.  If an index has a non-None ``name`` attribute, and there is no other column
with a name matching that value, then the ``index.name`` value can be used as
the descriptor. Otherwise (for unnamed indexes and ones with names colliding
with other column names) a disambiguating name with pattern matching
``__index_level_\d+__`` should be used. In cases of named indexes as data
columns, ``name`` attribute is always stored in the column descriptors as
above.  Column metadata
~~~  ``pandas_type`` is the logical type of the column, and is one of:  * Boolean: ``'bool'``
* Integers: ``'int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64'``
* Floats: ``'float16', 'float32', 'float64'``
* Date and Time Types: ``'datetime', 'datetimetz'``, ``'timedelta'``
* String: ``'unicode', 'bytes'``
* Categorical: ``'categorical'``
* Other Python objects: ``'object'``  The ``numpy_type`` is the physical storage type of the column, which is the
result of ``str(dtype)`` for the underlying NumPy array that holds the data. So
for ``datetimetz`` this is ``datetime64[ns]`` and for categorical, it may be
any of the supported integer categorical types.  The ``metadata`` field is ``None`` except for:  * ``datetimetz``: ``{'timezone': zone, 'unit': 'ns'}``, e.g. ``{'timezone',  'America/New_York', 'unit': 'ns'}``. The ``'unit'`` is optional, and if  omitted it is assumed to be nanoseconds.
* ``categorical``: ``{'num_categories': K, 'ordered': is_ordered, 'type': $TYPE}``  * Here ``'type'`` is optional, and can be a nested pandas type specification  here (but not categorical)  * ``unicode``: ``{'encoding': encoding}``  * The encoding is optional, and if not present is UTF-8  * ``object``: ``{'encoding': encoding}``. Objects can be serialized and stored  in ``BYTE_ARRAY`` Parquet columns. The encoding can be one of:  * ``'pickle'``  * ``'bson'``  * ``'json'``  * ``timedelta``: ``{'unit': 'ns'}``. The ``'unit'`` is optional, and if omitted  it is assumed to be nanoseconds. This metadata is optional altogether  For types other than these, the ``'metadata'`` key can be

omitted. Implementations can assume ``None`` if the key is not present.  As an example of fully-formed metadata:  .. code-block:: text  {'index_columns': ['__index_level_0__'],  'column_indexes': [  {'name': None,  'field_name': 'None',  'pandas_type': 'unicode',  'numpy_type': 'object',  'metadata': {'encoding': 'UTF-8'}}  ],  'columns': [  {'name': 'c0',  'field_name': 'c0',  'pandas_type': 'int8',  'numpy_type': 'int8',  'metadata': None},  {'name': 'c1',  'field_name': 'c1',  'pandas_type': 'bytes',  'numpy_type': 'object',  'metadata': None},  {'name': 'c2',  'field_name': 'c2',  'pandas_type': 'categorical',  'numpy_type': 'int16',  'metadata': {'num_categories': 1000, 'ordered': False}},  {'name': 'c3',  'field_name': 'c3',  'pandas_type': 'datetimetz',  'numpy_type': 'datetime64[ns]',  'metadata': {'timezone': 'America/Los_Angeles'}},  {'name': 'c4',  'field_name': 'c4',  'pandas_type': 'object',  'numpy_type': 'object',  'metadata': {'encoding': 'pickle'}},  {'name': None,  'field_name': '__index_level_0__',  'pandas_type': 'int64',  'numpy_type': 'int64',  'metadata': None}  ],  'pandas_version': '1.4.0',  'creator': {  'library': 'pyarrow',  'version': '0.13.0'  }}

Registering custom accessors

Libraries can use the decorators
`pandas.api.extensions.register_dataframe_accessor`,
`pandas.api.extensions.register_series_accessor`, and
`pandas.api.extensions.register_index_accessor`, to add additional
"namespaces" to pandas objects. All of these follow a similar convention: you
decorate a class, providing the name of attribute to add. The class's
``__init__`` method gets the object being decorated. For example:  .. code-block:: python  @pd.api.extensions.register_dataframe_accessor("geo")  class GeoAccessor:  def __init__(self, pandas_obj):  self._validate(pandas_obj)  self._obj = pandas_obj  @staticmethod  def _validate(obj):  # verify there is a column latitude and a column longitude  if "latitude" not in obj.columns or "longitude" not in obj.columns:  raise AttributeError("Must have 'latitude' and 'longitude'.")  @property  def center(self):  # return the geographic center point of this DataFrame  lat = self._obj.latitude  lon = self._obj.longitude  return (float(lon.mean()), float(lat.mean()))  def plot(self):  # plot this array's data on a map, e.g., using Cartopy  pass  Now users can access your methods using the ``geo`` namespace:  >>> ds = pd.DataFrame(  ...  {"longitude": np.linspace(0, 10), "latitude": np.linspace(0, 20)}  ... )  >>> ds.geo.center  (5.0, 10.0)  >>> ds.geo.plot()  # plots data on a map  This can be a convenient way to extend pandas objects without subclassing them.
If you write a custom accessor, make a pull request adding it to our
`ecosystem` page.  We highly recommend validating the data in your accessor's ``__init__``.
In our ``GeoAccessor``, we validate that the data contains the expected columns,
raising an ``AttributeError`` when the validation fails.
For a ``Series`` accessor, you should validate the ``dtype`` if the accessor
applies only to certain dtypes.  .. _extending.extension-types:

.. note::  The `pandas.api.extensions.ExtensionDtype` and `pandas.api.extensions.ExtensionArray` APIs were  experimental prior to pandas 1.5. Starting with version 1.5, future changes will follow  the `pandas deprecation policy <policies.version>`.  pandas defines an interface for implementing data types and arrays that *extend*
NumPy's type system. pandas itself uses the extension system for some types
that aren't built into NumPy (categorical, period, interval, datetime with
timezone).  Libraries can define a custom array and data type. When pandas encounters these
objects, they will be handled properly (i.e. not converted to an ndarray of
objects). Many methods like `pandas.isna` will dispatch to the extension
type's implementation.  If you're building a library that implements the interface, please publicize it
on `ecosystem.extensions`.  The interface consists of two classes.  `~pandas.api.extensions.ExtensionDtype`
^^^  A `pandas.api.extensions.ExtensionDtype` is similar to a ``numpy.dtype`` object. It describes the
data type. Implementors are responsible for a few unique items like the name.  One particularly important item is the ``type`` property. This should be the
class that is the scalar type for your data. For example, if you were writing an
extension array for IP Address data, this might be ``ipaddress.IPv4Address``.  See the `extension dtype source`_ for interface definition.  `pandas.api.extensions.ExtensionDtype` can be registered to pandas to allow creation via a string dtype name.
This allows one to instantiate ``Series`` and ``.astype()`` with a registered string name, for
example ``'category'`` is a registered string accessor for the ``CategoricalDtype``.  See the `extension dtype dtypes`_ for more on how to register dtypes.  `~pandas.api.extensions.ExtensionArray`
^^^  This class provides all the array-like functionality. ExtensionArrays are
limited to 1 dimension. An ExtensionArray is linked to an ExtensionDtype via the
``dtype`` attribute.  pandas makes no restrictions on how an extension array is created via its
``__new__`` or ``__init__``, and puts no restrictions on how you store your
data. We do require that your array be convertible to a NumPy array, even if
this is relatively expensive (as it is for ``Categorical``).  They may be backed by none, one, or many NumPy arrays. For example,
`pandas.Categorical` is an extension array backed by two arrays,
one for codes and one for categories. An array of IPv6 addresses may
be backed by a NumPy structured array with two fields, one for the
lower 64 bits and one for the upper 64 bits. Or they may be backed
by some other storage type, like Python lists.  See the `extension array source`_ for the interface definition. The docstrings
and comments contain guidance for properly implementing the interface.  .. _extending.extension.operator:  `~pandas.api.extensions.ExtensionArray` operator support
^^^  By default, there are no operators defined for the class `~pandas.api.extensions.ExtensionArray`.
There are two approaches for providing operator support for your ExtensionArray:  1. Define each of the operators on your ``ExtensionArray`` subclass.
2. Use an operator implementation from pandas that depends on operators that are already defined  on the underlying elements (scalars) of the ExtensionArray.  .. note::  Regardless of the approach, you may want to set ``__array_priority__``  if you want your implementation to be called when involved in binary operations  with NumPy arrays.  For the first approach, you define selected operators, e.g., ``__add__``, ``__le__``, etc. that
you want your ``ExtensionArray`` subclass to support.  The second approach assumes that the underlying elements (i.e., scalar type) of the ``ExtensionArray``
have the individual operators already defined.  In other words, if your ``ExtensionArray``
named ``MyExtensionArray`` is implemented so that each element is an instance
of the class ``MyExtensionElement``, then if the operators are defined

for ``MyExtensionElement``, the second approach will automatically
define the operators for ``MyExtensionArray``.  A mixin class, `~pandas.api.extensions.ExtensionScalarOpsMixin` supports this second
approach.  If developing an ``ExtensionArray`` subclass, for example ``MyExtensionArray``,
can simply include ``ExtensionScalarOpsMixin`` as a parent class of ``MyExtensionArray``,
and then call the methods `~MyExtensionArray._add_arithmetic_ops` and/or
`~MyExtensionArray._add_comparison_ops` to hook the operators into
your ``MyExtensionArray`` class, as follows:  .. code-block:: python  from pandas.api.extensions import ExtensionArray, ExtensionScalarOpsMixin  class MyExtensionArray(ExtensionArray, ExtensionScalarOpsMixin):  pass  MyExtensionArray._add_arithmetic_ops()  MyExtensionArray._add_comparison_ops()  .. note::  Since ``pandas`` automatically calls the underlying operator on each  element one-by-one, this might not be as performant as implementing your own  version of the associated operators directly on the ``ExtensionArray``.  For arithmetic operations, this implementation will try to reconstruct a new
``ExtensionArray`` with the result of the element-wise operation. Whether
or not that succeeds depends on whether the operation returns a result
that's valid for the ``ExtensionArray``. If an ``ExtensionArray`` cannot
be reconstructed, an ndarray containing the scalars returned instead.  For ease of implementation and consistency with operations between pandas
and NumPy ndarrays, we recommend *not* handling Series and Indexes in your binary ops.
Instead, you should detect these cases and return ``NotImplemented``.
When pandas encounters an operation like ``op(Series, ExtensionArray)``, pandas
will  1. unbox the array from the ``Series`` (``Series.array``)
2. call ``result = op(values, ExtensionArray)``
3. re-box the result in a ``Series``  .. _extending.extension.ufunc:  NumPy universal functions
^^^  `Series` implements ``__array_ufunc__``. As part of the implementation,
pandas unboxes the ``ExtensionArray`` from the `Series`, applies the ufunc,
and re-boxes it if necessary.  If applicable, we highly recommend that you implement ``__array_ufunc__`` in your
extension array to avoid coercion to an ndarray. See
the NumPy documentation_
for an example.  As part of your implementation, we require that you defer to pandas when a pandas
container (`Series`, `DataFrame`, `Index`) is detected in ``inputs``.
If any of those is present, you should return ``NotImplemented``. pandas will take care of
unboxing the array from the container and re-calling the ufunc with the unwrapped input.  .. _extending.extension.testing:  Testing extension arrays
^^^  We provide a test suite for ensuring that your extension arrays satisfy the expected
behavior. To use the test suite, you must provide several pytest fixtures and inherit
from the base test class. The required fixtures are found in
https://github.com/pandas-dev/pandas/blob/main/pandas/tests/extension/conftest.py.  To use a test, subclass it:  .. code-block:: python  from pandas.tests.extension import base  class TestConstructors(base.BaseConstructorsTests):  pass  See https://github.com/pandas-dev/pandas/blob/main/pandas/tests/extension/base/__init__.py
for a list of all the tests available.  .. _extending.extension.arrow:  Compatibility with Apache Arrow
^^^  An ``ExtensionArray`` can support conversion to / from ``pyarrow`` arrays
(and thus support for example serialization to the Parquet file format)
by implementing two methods: ``ExtensionArray.__arrow_array__`` and
``ExtensionDtype.__from_arrow__``.  The ``ExtensionArray.__arrow_array__`` ensures that ``pyarrow`` knowns how
to convert the specific extension array into a ``pyarrow.Array`` (also when

included as a column in a pandas DataFrame):  .. code-block:: python  class MyExtensionArray(ExtensionArray):  ...  def __arrow_array__(self, type=None):  # convert the underlying array values to a pyarrow Array  import pyarrow  return pyarrow.array(..., type=type)  The ``ExtensionDtype.__from_arrow__`` method then controls the conversion
back from pyarrow to a pandas ExtensionArray. This method receives a pyarrow
``Array`` or ``ChunkedArray`` as only argument and is expected to return the
appropriate pandas ``ExtensionArray`` for this dtype and the passed values:  .. code-block:: none  class ExtensionDtype:  ...  def __from_arrow__(self, array: pyarrow.Array/ChunkedArray) -> ExtensionArray:  ...  See more in the Arrow documentation_.  Those methods have been implemented for the nullable integer and string extension
dtypes included in pandas, and ensure roundtrip to pyarrow and the Parquet file format.  .. _extension dtype dtypes: https://github.com/pandas-dev/pandas/blob/main/pandas/core/dtypes/dtypes.py
.. _extension dtype source: https://github.com/pandas-dev/pandas/blob/main/pandas/core/dtypes/base.py
.. _extension array source: https://github.com/pandas-dev/pandas/blob/main/pandas/core/arrays/base.py  .. _extending.subclassing-pandas:

.. warning:: There are some easier alternatives before considering subclassing ``pandas`` data structures.  1. Extensible method chains with `pipe <basics.pipe>`  2. Use *composition*. See here.  3. Extending by `registering an accessor <extending.register-accessors>`  4. Extending by `extension type <extending.extension-types>`  This section describes how to subclass ``pandas`` data structures to meet more specific needs. There are two points that need attention:  1. Override constructor properties.
2. Define original properties  .. note::  You can find a nice example in geopandas project.  Override constructor properties
^^^  Each data structure has several *constructor properties* for returning a new
data structure as the result of an operation. By overriding these properties,
you can retain subclasses through ``pandas`` data manipulations.  There are 3 possible constructor properties to be defined on a subclass:  * ``DataFrame/Series._constructor``: Used when a manipulation result has the same dimension as the original.
* ``DataFrame._constructor_sliced``: Used when a ``DataFrame`` (sub-)class manipulation result should be a ``Series`` (sub-)class.
* ``Series._constructor_expanddim``: Used when a ``Series`` (sub-)class manipulation result should be a ``DataFrame`` (sub-)class, e.g. ``Series.to_frame()``.  Below example shows how to define ``SubclassedSeries`` and ``SubclassedDataFrame`` overriding constructor properties.  .. code-block:: python  class SubclassedSeries(pd.Series):  @property  def _constructor(self):  return SubclassedSeries  @property  def _constructor_expanddim(self):  return SubclassedDataFrame  class SubclassedDataFrame(pd.DataFrame):  @property  def _constructor(self):  return SubclassedDataFrame  @property  def _constructor_sliced(self):  return SubclassedSeries  .. code-block:: python  >>> s = SubclassedSeries([1, 2, 3])  >>> type(s)  <class '__main__.SubclassedSeries'>  >>> to_framed = s.to_frame()  >>> type(to_framed)  <class '__main__.SubclassedDataFrame'>  >>> df = SubclassedDataFrame({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})  >>> df  A  B  C  0  1  4  7  1  2  5  8  2  3  6  9  >>> type(df)  <class '__main__.SubclassedDataFrame'>  >>> sliced1 = df[["A", "B"]]  >>> sliced1  A  B  0  1  4  1  2  5  2  3  6  >>> type(sliced1)  <class '__main__.SubclassedDataFrame'>  >>> sliced2 = df["A"]  >>> sliced2  0  1  1  2  2  3  Name: A, dtype: int64  >>> type(sliced2)  <class '__main__.SubclassedSeries'>  Define original properties
^^^  To let original data structures have additional properties, you should let ``pandas`` know what properties are added. ``pandas`` maps unknown properties to data names overriding ``__getattribute__``. Defining original properties can be done in one of 2 ways:  1. Define ``_internal_names`` and ``_internal_names_set`` for temporary properties which WILL NOT be passed to manipulation results.

2. Define ``_metadata`` for normal properties which will be passed to manipulation results.  Below is an example to define two original properties, "internal_cache" as a temporary property and "added_property" as a normal property  .. code-block:: python  class SubclassedDataFrame2(pd.DataFrame):  # temporary properties  _internal_names = pd.DataFrame._internal_names + ["internal_cache"]  _internal_names_set = set(_internal_names)  # normal properties  _metadata = ["added_property"]  @property  def _constructor(self):  return SubclassedDataFrame2  .. code-block:: python  >>> df = SubclassedDataFrame2({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})  >>> df  A  B  C  0  1  4  7  1  2  5  8  2  3  6  9  >>> df.internal_cache = "cached"  >>> df.added_property = "property"  >>> df.internal_cache  cached  >>> df.added_property  property  # properties defined in _internal_names is reset after manipulation  >>> df[["A", "B"]].internal_cache  AttributeError: 'SubclassedDataFrame2' object has no attribute 'internal_cache'  # properties defined in _metadata are retained  >>> df[["A", "B"]].added_property  property  .. _extending.plotting-backends:

Plotting backends

Starting in 0.25 pandas can be extended with third-party plotting backends. The
main idea is letting users select a plotting backend different than the provided
one based on Matplotlib. For example:  .. code-block:: python  >>> pd.set_option("plotting.backend", "backend.module")  >>> pd.Series([1, 2, 3]).plot()  This would be more or less equivalent to:  .. code-block:: python  >>> import backend.module  >>> backend.module.plot(pd.Series([1, 2, 3]))  The backend module can then use other visualization tools (Bokeh, Altair,...)
to generate the plots.  Libraries implementing the plotting backend should use entry points_
to make their backend discoverable to pandas. The key is ``"pandas_plotting_backends"``. For example, pandas
registers the default "matplotlib" backend as follows.  .. code-block:: python  # in setup.py  setup(  # noqa: F821  ...,  entry_points={  "pandas_plotting_backends": [  "matplotlib = pandas:plotting._matplotlib",  ],  },  )  More information on how to implement a third-party plotting backend can be found at
https://github.com/pandas-dev/pandas/blob/main/pandas/plotting/__init__.py#L1.

Development

.. If you update this toctree, also update the manual toctree in the  main index.rst.template  .. toctree::  2  contributing  contributing_environment  contributing_documentation  contributing_codebase  maintaining  internals  copy_on_write  debugging_extensions  extending  developer  policies  roadmap  community

Indexing

In pandas there are a few objects implemented which can serve as valid
containers for the axis labels:  * `Index`: the generic "ordered set" object, an ndarray of object dtype  assuming nothing about its contents. The labels must be hashable (and  likely immutable) and unique. Populates a dict of label to location in  Cython to do ``O(1)`` lookups.
* `MultiIndex`: the standard hierarchical index object
* `DatetimeIndex`: An Index object with `Timestamp` boxed elements (impl are the int64 values)
* `TimedeltaIndex`: An Index object with `Timedelta` boxed elements (impl are the in64 values)
* `PeriodIndex`: An Index object with Period elements  There are functions that make the creation of a regular index easy:  * `date_range`: fixed frequency date range generated from a time rule or  DateOffset. An ndarray of Python datetime objects
* `period_range`: fixed frequency date range generated from a time rule or  DateOffset. An ndarray of `Period` objects, representing timespans  The motivation for having an ``Index`` class in the first place was to enable
different implementations of indexing. This means that it's possible for you,
the user, to implement a custom ``Index`` subclass that may be better suited to
a particular application than the ones provided in pandas.  From an internal implementation point of view, the relevant methods that an
``Index`` must define are one or more of the following (depending on how
incompatible the new object internals are with the ``Index`` functions):  * `~Index.get_loc`: returns an "indexer" (an integer, or in some cases a  slice object) for a label
* `~Index.slice_locs`: returns the "range" to slice between two labels
* `~Index.get_indexer`: Computes the indexing vector for reindexing / data  alignment purposes. See the source / docstrings for more on this
* `~Index.get_indexer_non_unique`: Computes the indexing vector for reindexing / data  alignment purposes when the index is non-unique. See the source / docstrings  for more on this
* `~Index.reindex`: Does any pre-conversion of the input index then calls  ``get_indexer``
* `~Index.union`, `~Index.intersection`: computes the union or intersection of two  Index objects
* `~Index.insert`: Inserts a new label into an Index, yielding a new object
* `~Index.delete`: Delete a label, yielding a new object
* `~Index.drop`: Deletes a set of labels
* `~Index.take`: Analogous to ndarray.take  MultiIndex
~~~  Internally, the `MultiIndex` consists of a few things: the **levels**, the
integer **codes**, and the level **names**:  .. ipython:: python  index = pd.MultiIndex.from_product(  [range(3), ["one", "two"]], names=["first", "second"]  )  index  index.levels  index.codes  index.names  You can probably guess that the codes determine which unique element is
identified with that location at each layer of the index. It's important to
note that sortedness is determined **solely** from the integer codes and does
not check (or care) whether the levels themselves are sorted. Fortunately, the
constructors `~MultiIndex.from_tuples` and `~MultiIndex.from_arrays` ensure
that this is true, but if you compute the levels and codes yourself, please be careful.  Values
~~~  pandas extends NumPy's type system with custom types, like `Categorical` or
datetimes with a timezone, so we have multiple notions of "values". For 1-D
containers (``Index`` classes and ``Series``) we have the following convention:  * ``cls._values`` refers is the "best possible" array. This could be an  ``ndarray`` or ``ExtensionArray``.  So, for example, ``Series[category]._values`` is a ``Categorical``.  .. _ref-subclassing-pandas:

Subclassing pandas data structures

This section has been moved to `extending.subclassing-pandas`.

Roles

pandas uses two levels of permissions: **triage** and **core** team members.  Triage members can label and close issues and pull requests.  Core team members can label and close issues and pull request, and can merge
pull requests.  GitHub publishes the full `list of permissions`_.

Tasks

pandas is largely a volunteer project, so these tasks shouldn't be read as
"expectations" of triage and maintainers. Rather, they're general descriptions
of what it means to be a maintainer.  * Triage newly filed issues (see `maintaining.triage`)
* Review newly opened pull requests
* Respond to updates on existing issues and pull requests
* Drive discussion and decisions on stalled issues and pull requests
* Provide experience / wisdom on API design questions to ensure consistency and maintainability
* Project organization (run / attend developer meetings, represent pandas)  https://matthewrocklin.com/blog/2019/05/18/maintainer may be interesting background
reading.  .. _maintaining.triage:

Issue triage
  Here's a typical workflow for triaging a newly opened issue.  1. **Thank the reporter for opening an issue**  The issue tracker is many people's first interaction with the pandas project itself,  beyond just using the library. As such, we want it to be a welcoming, pleasant  experience.  2. **Is the necessary information provided?**  Ideally reporters would fill out the issue template, but many don't.  If crucial information (like the version of pandas they used), is missing  feel free to ask for that and label the issue with "Needs info". The  report should follow the guidelines in `contributing.bug_reports`.  You may want to link to that if they didn't follow the template.  Make sure that the title accurately reflects the issue. Edit it yourself  if it's not clear.  3. **Is this a duplicate issue?**  We have many open issues. If a new issue is clearly a duplicate, label the  new issue as "Duplicate" assign the milestone "No Action", and close the issue  with a link to the original issue. Make sure to still thank the reporter, and  encourage them to chime in on the original issue, and perhaps try to fix it.  If the new issue provides relevant information, such as a better or slightly  different example, add it to the original issue as a comment or an edit to  the original post.  4. **Is the issue minimal and reproducible**?  For bug reports, we ask that the reporter provide a minimal reproducible  example. See https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports  for a good explanation. If the example is not reproducible, or if it's  *clearly* not minimal, feel free to ask the reporter if they can provide  and example or simplify the provided one. Do acknowledge that writing  minimal reproducible examples is hard work. If the reporter is struggling,  you can try to write one yourself and we'll edit the original post to include it.  If a reproducible example can't be provided, add the "Needs info" label.  If a reproducible example is provided, but you see a simplification,  edit the original post with your simpler reproducible example.  5. **Is this a clearly defined feature request?**  Generally, pandas prefers to discuss and design new features in issues, before  a pull request is made. Encourage the submitter to include a proposed API  for the new feature. Having them write a full docstring is a good way to  pin down specifics.  We'll need a discussion from several pandas maintainers before deciding whether  the proposal is in scope for pandas.  6. **Is this a usage question?**  We prefer that usage questions are asked on StackOverflow with the pandas  tag. https://stackoverflow.com/questions/tagged/pandas  If it's easy to answer, feel free to link to the relevant documentation section,  let them know that in the future this kind of question should be on  StackOverflow, and close the issue.  7. **What labels and milestones should I add?**  Apply the relevant labels. This is a bit of an art, and comes with experience.  Look at similar issues to get a feel for how things are labeled.  If the issue is clearly defined and the fix seems relatively straightforward,  label the issue as "Good first issue".  Typically, new issues will be assigned the "Contributions welcome" milestone,  unless it's know that this issue should be addressed in a specific release (say  because it's a large regression).  .. _maintaining.regressions:

Investigating regressions

Regressions are bugs that unintentionally break previously working code. The common way
to  investigate regressions is by using
git bisect,
which finds the first commit that introduced the bug.  For example: a user reports that ``pd.Series([1, 1]).sum()`` returns ``3``
in pandas version ``1.5.0`` while in version ``1.4.0`` it returned ``2``. To begin,
create a file ``t.py`` in your pandas directory, which contains  .. code-block:: python  import pandas as pd  assert pd.Series([1, 1]).sum() == 2  and then run::  git bisect start  git bisect good v1.4.0  git bisect bad v1.5.0  git bisect run bash -c "python setup.py build_ext -j 4; python t.py"  This finds the first commit that changed the behavior. The C extensions have to be
rebuilt at every step, so the search can take a while.  Exit bisect and rebuild the current version::  git bisect reset  python setup.py build_ext -j 4  Report your findings under the corresponding issue and ping the commit author to get
their input.  .. note::  In the ``bisect run`` command above, commits are considered good if ``t.py`` exits  with ``0`` and bad otherwise. When raising an exception is the desired behavior,  wrap the code in an appropriate ``try/except`` statement. See `35685` for  more examples.  .. _maintaining.closing:

Closing issues

Be delicate here: many people interpret closing an issue as us saying that the
conversation is over. It's typically best to give the reporter some time to
respond or self-close their issue if it's determined that the behavior is not a bug,
or the feature is out of scope. Sometimes reporters just go away though, and
we'll close the issue after the conversation has died.  .. _maintaining.reviewing:

Reviewing pull requests

Anybody can review a pull request: regular contributors, triagers, or core-team
members. But only core-team members can merge pull requests when they're ready.  Here are some things to check when reviewing a pull request.  * Tests should be in a sensible location: in the same file as closely related tests.
* New public APIs should be included somewhere in ``doc/source/reference/``.
* New / changed API should use the ``versionadded`` or ``versionchanged`` directives in the docstring.
* User-facing changes should have a whatsnew in the appropriate file.
* Regression tests should reference the original GitHub issue number like ``# GH-1234``.
* The pull request should be labeled and assigned the appropriate milestone (the next patch release  for regression fixes and small bug fixes, the next minor milestone otherwise)
* Changes should comply with our `policies.version`.  .. _maintaining.backporting:

Backporting

pandas supports point releases (e.g. ``1.4.3``) that aim to:  1. Fix bugs in new features introduced in the first minor version release.  * e.g. If a new feature was added in ``1.4`` and contains a bug, a fix can be applied in ``1.4.3``  2. Fix bugs that used to work in a few minor releases prior. There should be agreement between core team members that a backport is appropriate.  * e.g. If a feature worked in ``1.2`` and stopped working since ``1.3``, a fix can be applied in ``1.4.3``.  Since pandas minor releases are based on GitHub branches (e.g. point release of ``1.4`` are based off the ``1.4.x`` branch),
"backporting" means merging a pull request fix to the ``main`` branch and correct minor branch associated with the next point release.  By default, if a pull request is assigned to the next point release milestone within the GitHub interface,
the backporting process should happen automatically by the ``@meeseeksdev`` bot once the pull request is merged.
A new pull request will be made backporting the pull request to the correct version branch.
Sometimes due to merge conflicts, a manual pull request will need to be made addressing the code conflict.  If the bot does not automatically start the backporting process, you can also write a GitHub comment in the merged pull request
to trigger the backport::  @meeseeksdev backport version-branch  This will trigger a workflow which will backport a given change to a branch
(e.g. @meeseeksdev backport 1.4.x)

Cleaning up old issues

Every open issue in pandas has a cost. Open issues make finding duplicates harder,
and can make it harder to know what needs to be done in pandas. That said, closing
issues isn't a goal on its own. Our goal is to make pandas the best it can be,
and that's best done by ensuring that the quality of our open issues is high.  Occasionally, bugs are fixed but the issue isn't linked to in the Pull Request.
In these cases, comment that "This has been fixed, but could use a test." and
label the issue as "Good First Issue" and "Needs Test".  If an older issue doesn't follow our issue template, edit the original post to
include a minimal example, the actual output, and the expected output. Uniformity
in issue reports is valuable.  If an older issue lacks a reproducible example, label it as "Needs Info" and
ask them to provide one (or write one yourself if possible). If one isn't
provide reasonably soon, close it according to the policies in `maintaining.closing`.

Cleaning up old pull requests

Occasionally, contributors are unable to finish off a pull request.
If some time has passed (two weeks, say) since the last review requesting changes,
gently ask if they're still interested in working on this. If another two weeks or
so passes with no response, thank them for their work and close the pull request.
Comment on the original issue that "There's a stalled PR at #1234 that may be
helpful.", and perhaps label the issue as "Good first issue" if the PR was relatively
close to being accepted.  Additionally, core-team members can push to contributors branches. This can be
helpful for pushing an important PR across the line, or for fixing a small
merge conflict.

Becoming a pandas maintainer

The full process is outlined in our `governance documents`_. In summary,
we're happy to give triage permissions to anyone who shows interest by
being helpful on the issue tracker.  The required steps for adding a maintainer are:  1. Contact the contributor and ask their interest to join.
2. Add the contributor to the appropriate GitHub Team if accepted the invitation.  * ``pandas-core`` is for core team members  * ``pandas-triage`` is for pandas triage members  3. Add the contributor to the pandas Google group.
4. Create a pull request to add the contributor's GitHub handle to ``pandas-dev/pandas/web/pandas/config.yml``.
5. Create a pull request to add the contributor's name/GitHub handle to the governance document.  The current list of core-team members is at
https://github.com/pandas-dev/pandas-governance/blob/master/people.md  .. _maintaining.merging:

Merging pull requests

Only core team members can merge pull requests. We have a few guidelines.  1. You should typically not self-merge your own pull requests. Exceptions include  things like small changes to fix CI (e.g. pinning a package version).
2. You should not merge pull requests that have an active discussion, or pull  requests that has any ``-1`` votes from a core maintainer. pandas operates  by consensus.
3. For larger changes, it's good to have a +1 from at least two core team members.  In addition to the items listed in `maintaining.closing`, you should verify
that the pull request is assigned the correct milestone.  Pull requests merged with a patch-release milestone will typically be backported
by our bot. Verify that the bot noticed the merge (it will leave a comment within
a minute typically). If a manual backport is needed please do that, and remove
the "Needs backport" label once you've done it manually. If you forget to assign
a milestone before tagging, you can request the bot to backport it with:  .. code-block:: console  @Meeseeksdev backport <branch>  .. _maintaining.asv-machine:

Benchmark machine

The team currently owns dedicated hardware for hosting a website for pandas' ASV performance benchmark. The results
are published to https://asv-runner.github.io/asv-collection/pandas/  Configuration
```  The machine can be configured with the Ansible playbook in https://github.com/tomaugspurger/asv-runner.  Publishing
```  The results are published to another GitHub repository, https://github.com/tomaugspurger/asv-collection.
Finally, we have a cron job on our docs server to pull from https://github.com/tomaugspurger/asv-collection, to serve them from ``/speed``.
Ask Tom or Joris for access to the webserver.  Debugging
```  The benchmarks are scheduled by Airflow. It has a dashboard for viewing and debugging the results. You'll need to setup an SSH tunnel to view them  ssh -L 80808080 pandas@panda.likescandy.com  .. _maintaining.release:

The release process makes a snapshot of pandas (a git commit) available to users with
a particular version number. After the release the new pandas version will be available
in the next places:  - Git repo with a new tag
- Source distribution in a GitHub release
- Pip packages in the PyPI
- Conda/Mamba packages in conda-forge  The process for releasing a new version of pandas is detailed next section.  The instructions contain ``<version>`` which needs to be replaced with the version
to be released (e.g. ``1.5.2``). Also the branch to be released ``<branch>``, which
depends on whether the version being released is the release candidate of a new version,
or any other version. Release candidates are released from ``main``, while other
versions are released from their branch (e.g. ``1.5.x``).  Prerequisites
```  In order to be able to release a new pandas version, the next permissions are needed:  - Merge rights to the pandas,  pandas-wheels, and  pandas-feedstock repositories.
- Permissions to push to main in the pandas repository, to push the new tags.
- Write permissions to PyPI
- Access to the social media accounts, to publish the announcements.  Pre-release
```  1. Agree with the core team on the next topics:  - Release date (major/minor releases happen usually every 6 months, and patch releases  monthly until x.x.5, just before the next major/minor)  - Blockers (issues and PRs that must be part of the release)  - Next version after the one being released  2. Update and clean release notes for the version to be released, including:  - Set the final date of the release  - Remove any unused bullet point  - Make sure there are no formatting issues, typos, etc.  3. Make sure the CI is green for the last commit of the branch being released.  4. If not a release candidate, make sure all backporting pull requests to the branch  being released are merged.  5. Create a new issue and milestone for the version after the one being released.  If the release was a release candidate, we would usually want to create issues and  milestones for both the next major/minor, and the next patch release. In the  milestone of a patch release, we add the description ``on-merge: backport to <branch>``,  so tagged PRs are automatically backported to the release branch by our bot.  6. Change the milestone of all issues and PRs in the milestone being released to the  next milestone.  Release
```  1. Create an empty commit and a tag in the last commit of the branch to be released::  git checkout <branch>  git pull --ff-only upstream <branch>  git clean -xdf  git commit --allow-empty --author="Pandas Development Team <pandas-dev@python.org>" -m "RLS: <version>"  git tag -a v<version> -m "Version <version>"  # NOTE that the tag is v1.5.2 with "v" not 1.5.2  git push upstream <branch> --follow-tags  The docs for the new version will be built and published automatically with the docs job in the CI,

which will be triggered when the tag is pushed.  2. Only if the release is a release candidate, we want to create a new branch for it, immediately  after creating the tag. For example, if we are releasing pandas 1.4.0rc0, we would like to  create the branch 1.4.x to backport commits to the 1.4 versions. As well as create a tag to  mark the start of the development of 1.5.0 (assuming it is the next version)::  git checkout -b 1.4.x  git push upstream 1.4.x  git checkout main  git commit --allow-empty -m "Start 1.5.0"  git tag -a v1.5.0.dev0 -m "DEV: Start 1.5.0"  git push upstream main --follow-tags  3. Build the source distribution (git must be in the tag commit):  ./setup.py sdist --formats=gztar --quiet  4. Create a new GitHub release:  - Title: ``Pandas <version>``  - Tag: ``<version>``  - Files: ``pandas-<version>.tar.gz`` source distribution just generated  - Description: Copy the description of the last release of the same kind (release candidate, major/minor or patch release)  - Set as a pre-release: Only check for a release candidate  - Set as the latest release: Leave checked, unless releasing a patch release for an older version  (e.g. releasing 1.4.5 after 1.5 has been released)  5. The GitHub release will after some hours trigger an  automated conda-forge PR.  Merge it once the CI is green, and it will generate the conda-forge packages.  6. Packages for supported versions in PyPI are built in the  MacPython repo.  Open a PR updating the build commit to the released version, and merge it once the  CI is green. To do this type::  git checkout master  git pull --ff-only upstream master  git checkout -B RLS-<version>  sed -i 's/BUILD_COMMIT: "v.*/BUILD_COMMIT: "'v<version>'"/' azure/windows.yml azure/posix.yml  sed -i 's/BUILD_COMMIT="v.*/BUILD_COMMIT="'v<version>'"/' .travis.yml  git commit -am "RLS <version>"  git push -u origin RLS-<version>  7. Download all wheels from the Anaconda repository where MacPython uploads them:  https://anaconda.org/multibuild-wheels-staging/pandas/files?version=<version>  to the ``dist/`` directory in the local pandas copy. You can use the script  ``scripts/download_wheels.sh`` to download all wheels at once.  8. Upload wheels to PyPI:  twine upload pandas/dist/pandas-<version>*.{whl,tar.gz} --skip-existing  Post-Release
```  1. Update symlink to stable documentation by logging in to our web server, and  editing ``/var/www/html/pandas-docs/stable`` to point to ``version/<latest-version>``.  2. If releasing a major or minor release, open a PR in our source code to update  ``web/pandas/versions.json``, to have the desired versions in the documentation  dropdown menu.  3. Close the milestone and the issue for the released version.  4. Create a new issue for the next release, with the estimated date of release.  5. Open a PR with the placeholder for the release notes of the next version. See  for example the PR for 1.5.3.  6. Announce the new release in the official channels (use previous announcements  for reference):  - The pandas-dev and pydata mailing lists  - Twitter, Mastodon and Telegram  .. _governance documents: https://github.com/pandas-dev/pandas-governance
.. _list of permissions: https://docs.github.com/en/organizations/managing-access-to-your-organizations-repositories/repository-roles-for-an-organization

.. _develop.policies:***Policies***.. _policies.version:Version policy~~~.. versionchanged:: 1.0.0pandas uses a loose variant of semantic versioning (`SemVer`_) to governdeprecations, API compatibility, and version numbering.A pandas release number is made up of ``MAJOR.MINOR.PATCH``.API breaking changes should only occur in **major** releases. These changeswill be documented, with clear guidance on what is changing, why it's changing,and how to migrate existing code to the new behavior.Whenever possible, a deprecation path will be provided rather than an outrightbreaking change.pandas will introduce deprecations in **minor** releases. These deprecationswill preserve the existing behavior while emitting a warning that provideguidance on:* How to achieve similar behavior if an alternative is available* The pandas version in which the deprecation will be enforced.We will not introduce new deprecations in patch releases.Deprecations will only be enforced in **major** releases. For example, if abehavior is deprecated in pandas 1.2.0, it will continue to work, with awarning, for all releases in the 1.x series. The behavior will change and thedeprecation removed in the next major release (2.0.0)... note::  pandas will sometimes make *behavior changing* bug fixes, as part of  minor or patch releases. Whether or not a change is a bug fix or an  API-breaking change is a judgement call. We'll do our best, and we  invite you to participate in development discussion on the issue  tracker or mailing list.These policies do not apply to features marked as **experimental** in the documentation.pandas may change the behavior of experimental features at any time.Python support~~~pandas mirrors the NumPy guidelines for Python support_... _SemVer: https://semver.org

Roadmap

This page provides an overview of the major themes in pandas' development. Each of
these items requires a relatively large amount of effort to implement. These may
be achieved more quickly with dedicated funding or interest from contributors.  An item being on the roadmap does not mean that it will *necessarily* happen, even
with unlimited funding. During the implementation period we may discover issues
preventing the adoption of the feature.  Additionally, an item *not* being on the roadmap does not exclude it from inclusion
in pandas. The roadmap is intended for larger, fundamental changes to the project that
are likely to take months or years of developer time. Smaller-scoped items will continue
to be tracked on our issue tracker_.  See `roadmap.evolution` for proposing changes to this document.

Extensibility

pandas `extending.extension-types` allow for extending NumPy types with custom
data types and array storage. pandas uses extension types internally, and provides
an interface for 3rd-party libraries to define their own custom data types.  Many parts of pandas still unintentionally convert data to a NumPy array.
These problems are especially pronounced for nested data.  We'd like to improve the handling of extension arrays throughout the library,
making their behavior more consistent with the handling of NumPy arrays. We'll do this
by cleaning up pandas' internals and adding new methods to the extension array interface.

String data type

Currently, pandas stores text data in an ``object`` -dtype NumPy array.
The current implementation has two primary drawbacks: First, ``object`` -dtype
is not specific to strings: any Python object can be stored in an ``object`` -dtype
array, not just strings. Second: this is not efficient. The NumPy memory model
isn't especially well-suited to variable width text data.  To solve the first issue, we propose a new extension type for string data. This
will initially be opt-in, with users explicitly requesting ``dtype="string"``.
The array backing this string dtype may initially be the current implementation:
an ``object`` -dtype NumPy array of Python strings.  To solve the second issue (performance), we'll explore alternative in-memory
array libraries (for example, Apache Arrow). As part of the work, we may
need to implement certain operations expected by pandas users (for example
the algorithm used in, ``Series.str.upper``). That work may be done outside of
pandas.

Consistent missing value handling

Currently, pandas handles missing data differently for different data types. We
use different types to indicate that a value is missing (``np.nan`` for
floating-point data, ``np.nan`` or ``None`` for object-dtype data -- typically
strings or booleans -- with missing values, and ``pd.NaT`` for datetimelike
data). Integer data cannot store missing data or are cast to float. In addition,
pandas 1.0 introduced a new missing value sentinel, ``pd.NA``, which is being
used for the experimental nullable integer, boolean, and string data types.  These different missing values have different behaviors in user-facing
operations. Specifically, we introduced different semantics for the nullable
data types for certain operations (e.g. propagating in comparison operations
instead of comparing as False).  Long term, we want to introduce consistent missing data handling for all data
types. This includes consistent behavior in all operations (indexing, arithmetic
operations, comparisons, etc.). There has been discussion of eventually making
the new semantics the default.  This has been discussed at `28095` (and
linked issues), and described in more detail in this
design doc_.

Apache Arrow interoperability

Apache Arrow_ is a cross-language development
platform for in-memory data. The Arrow logical types are closely aligned with
typical pandas use cases.  We'd like to provide better-integrated support for Arrow memory and data types
within pandas. This will let us take advantage of its I/O capabilities and
provide for better interoperability with other languages and libraries
using Arrow.

Block manager rewrite

We'd like to replace pandas current internal data structures (a collection of
1 or 2-D arrays) with a simpler collection of 1-D arrays.  pandas internal data model is quite complex. A DataFrame is made up of
one or more 2-dimensional "blocks", with one or more blocks per dtype. This
collection of 2-D arrays is managed by the BlockManager.  The primary benefit of the BlockManager is improved performance on certain
operations (construction from a 2D array, binary operations, reductions across the columns),
especially for wide DataFrames. However, the BlockManager substantially increases the
complexity and maintenance burden of pandas.  By replacing the BlockManager we hope to achieve  * Substantially simpler code
* Easier extensibility with new logical types
* Better user control over memory use and layout
* Improved micro-performance
* Option to provide a C / Cython API to pandas' internals  See these design documents_
for more.

Decoupling of indexing and internals

The code for getting and setting values in pandas' data structures needs refactoring.
In particular, we must clearly separate code that converts keys (e.g., the argument
to ``DataFrame.loc``) to positions from code that uses these positions to get
or set values. This is related to the proposed BlockManager rewrite. Currently, the
BlockManager sometimes uses label-based, rather than position-based, indexing.
We propose that it should only work with positional indexing, and the translation of keys
to positions should be entirely done at a higher level.  Indexing is a complicated API with many subtleties. This refactor will require care
and attention. The following principles should inspire refactoring of indexing code and
should result on cleaner, simpler, and more performant code.  1. **Label indexing must never involve looking in an axis twice for the same label(s).**
This implies that any validation step must either:  * limit validation to general features (e.g. dtype/structure of the key/index), or  * reuse the result for the actual indexing.  2. **Indexers must never rely on an explicit call to other indexers.**
For instance, it is OK to have some internal method of ``.loc`` call some
internal method of ``__getitem__`` (or of their common base class),
but never in the code flow of ``.loc`` should ``the_obj[something]`` appear.  3. **Execution of positional indexing must never involve labels** (as currently, sadly, happens).
That is, the code flow of a getter call (or a setter call in which the right hand side is non-indexed)
to ``.iloc`` should never involve the axes of the object in any way.  4. **Indexing must never involve accessing/modifying values** (i.e., act on ``._data`` or ``.values``) **more than once.**
The following steps must hence be clearly decoupled:  * find positions we need to access/modify on each axis  * (if we are accessing) derive the type of object we need to return (dimensionality)  * actually access/modify the values  * (if we are accessing) construct the return object  5. As a corollary to the decoupling between 4.i and 4.iii, **any code which deals on how data is stored**
(including any combination of handling multiple dtypes, and sparse storage, categoricals, third-party types)
**must be independent from code that deals with identifying affected rows/columns**,
and take place only once step 4.i is completed.  * In particular, such code should most probably not live in ``pandas/core/indexing.py``  * ... and must not depend in any way on the type(s) of axes (e.g. no ``MultiIndex`` special cases)  6. As a corollary to point 1.i, **``Index`` (sub)classes must provide separate methods for any desired validity check of label(s) which does not involve actual lookup**,
on the one side, and for any required conversion/adaptation/lookup of label(s), on the other.  7. **Use of trial and error should be limited**, and anyway restricted to catch only exceptions
which are actually expected (typically ``KeyError``).  * In particular, code should never (intentionally) raise new exceptions in the ``except`` portion of a ``try... exception``  8. **Any code portion which is not specific to setters and getters must be shared**,
and when small differences in behavior are expected (e.g. getting with ``.loc`` raises for
missing labels, setting still doesn't), they can be managed with a specific parameter.

Numba-accelerated operations

Numba_ is a JIT compiler for Python code. We'd like to provide
ways for users to apply their own Numba-jitted functions where pandas accepts user-defined functions
(for example, `Series.apply`, `DataFrame.apply`, `DataFrame.applymap`,
and in groupby and window contexts). This will improve the performance of
user-defined-functions in these operations by staying within compiled code.

Performance monitoring

pandas uses airspeed velocity_ to
monitor for performance regressions. ASV itself is a fabulous tool, but requires
some additional work to be integrated into an open source project's workflow.  The asv-runner_ organization, currently made up
of pandas maintainers, provides tools built on top of ASV. We have a physical
machine for running a number of project's benchmarks, and tools managing the
benchmark runs and reporting on results.  We'd like to fund improvements and maintenance of these tools to  * Be more stable. Currently, they're maintained on the nights and weekends when  a maintainer has free time.
* Tune the system for benchmarks to improve stability, following  https://pyperf.readthedocs.io/en/latest/system.html
* Build a GitHub bot to request ASV runs *before* a PR is merged. Currently, the  benchmarks are only run nightly.  .. _roadmap.evolution:

Roadmap evolution

pandas continues to evolve. The direction is primarily determined by community
interest. Everyone is welcome to review existing items on the roadmap and
to propose a new item.  Each item on the roadmap should be a short summary of a larger design proposal.
The proposal should include  1. Short summary of the changes, which would be appropriate for inclusion in  the roadmap if accepted.
2. Motivation for the changes.
3. An explanation of why the change is in scope for pandas.
4. Detailed design: Preferably with example-usage (even if not implemented yet)  and API documentation
5. API Change: Any API changes that may result from the proposal.  That proposal may then be submitted as a GitHub issue, where the pandas maintainers
can review and comment on the design. The pandas mailing list_
should be notified of the proposal.  When there's agreement that an implementation
would be welcome, the roadmap should be updated to include the summary and a
link to the discussion issue.

Completed items

This section records now completed items from the pandas roadmap.  Documentation improvements
~~~  We improved the pandas documentation  * The pandas community worked with others to build the `pydata-sphinx-theme`_,  which is now used for https://pandas.pydata.org/docs/ (`15556`).
* `getting_started` contains a number of resources intended for new  pandas users coming from a variety of backgrounds (`26831`).  .. _pydata-sphinx-theme: https://github.com/pydata/pydata-sphinx-theme

Installation

.. panels::  + install-card  col-lg-6 col-md-6 col-sm-12 col-xs-12 p-3  Working with conda?  ^^^  pandas is part of the Anaconda_  distribution and can be installed with Anaconda or Miniconda:  +++  .. code-block:: bash  conda install pandas

Prefer pip?  ^^^  pandas can be installed via pip from PyPI_.  +++  .. code-block:: bash  pip install pandas

col-12 p-3  In-depth instructions?  ^^^  Installing a specific version? Installing from source? Check the advanced  installation page.  .. link-button:: ./install.html  url  Learn more  btn-secondary stretched-link  .. _gentle_intro:

.. raw:: html  <div class="container">  <div id="accordion" class="shadow tutorial-accordion">  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseOne">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  What kind of data does pandas handle?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_01_tableoriented>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseOne" class="collapse" data-parent="#accordion">  <div class="card-body">  When working with tabular data, such as data stored in spreadsheets or databases, pandas is the right tool for you. pandas will help you
to explore, clean, and process your data. In pandas, a data table is called a `DataFrame`.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_01_tableoriented>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <dsintro>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseTwo">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How do I read and write tabular data?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_02_read_write>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseTwo" class="collapse" data-parent="#accordion">  <div class="card-body">  pandas supports the integration with many file formats or data sources out of the box (csv, excel, sql, json, parquet,…). Importing data from each of these
data sources is provided by function with the prefix ``read_*``. Similarly, the ``to_*`` methods are used to store data.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_02_read_write>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <io>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseThree">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How do I select a subset of a table?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_03_subset>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseThree" class="collapse" data-parent="#accordion">  <div class="card-body">  Selecting or filtering specific rows and/or columns? Filtering the data on a condition? Methods for slicing, selecting, and extracting the
data you need are available in pandas.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_03_subset>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <indexing>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseFour">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to create plots in pandas?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_04_plotting>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseFour" class="collapse" data-parent="#accordion">  <div class="card-body">  pandas provides plotting your data out of the box, using the power of Matplotlib. You can pick the plot type (scatter, bar, boxplot,...)

corresponding to your data.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_04_plotting>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <visualization>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseFive">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to create new columns derived from existing columns?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_05_columns>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseFive" class="collapse" data-parent="#accordion">  <div class="card-body">  There is no need to loop over all rows of your data table to do calculations. Data manipulations on a column work elementwise.
Adding a column to a `DataFrame` based on existing data in other columns is straightforward.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_05_columns>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <basics.dataframe.sel_add_del>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseSix">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to calculate summary statistics?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_06_stats>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseSix" class="collapse" data-parent="#accordion">  <div class="card-body">  Basic statistics (mean, median, min, max, counts...) are easily calculable. These or custom aggregations can be applied on the entire
data set, a sliding window of the data, or grouped by categories. The latter is also known as the split-apply-combine approach.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_06_stats>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <groupby>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseSeven">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to reshape the layout of tables?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_07_reshape>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseSeven" class="collapse" data-parent="#accordion">  <div class="card-body">  Change the structure of your data table in multiple ways. You can `~pandas.melt` your data table from wide to long/tidy form or `~pandas.pivot`

from long to wide format. With aggregations built-in, a pivot table is created with a single command.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_07_reshape>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <reshaping>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseEight">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to combine data from multiple tables?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_08_combine>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseEight" class="collapse" data-parent="#accordion">  <div class="card-body">  Multiple tables can be concatenated both column wise and row wise as database-like join/merge operations are provided to combine multiple tables of data.  center  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_08_combine>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <merging>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseNine">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to handle time series data?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_09_timeseries>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseNine" class="collapse" data-parent="#accordion">  <div class="card-body">  pandas has great support for time series and has an extensive set of tools for working with dates, times, and time-indexed data.  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_09_timeseries>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <timeseries>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  <div class="card tutorial-card">  <div class="card-header collapsed card-link" data-toggle="collapse" data-target="#collapseTen">  <div class="d-flex flex-row tutorial-card-header-1">  <div class="d-flex flex-row tutorial-card-header-2">  <button class="btn btn-dark btn-sm"></button>  How to manipulate textual data?  </div>  <span class="badge gs-badge-link">  `Straight to tutorial...<10min_tut_10_text>`  .. raw:: html  </span>  </div>  </div>  <div id="collapseTen" class="collapse" data-parent="#accordion">  <div class="card-body">  Data sets do not only contain numerical data. pandas provides a wide range of functions to clean textual data and extract useful information from it.  .. raw:: html  <div class="d-flex flex-row">  <span class="badge gs-badge-link">  `To introduction tutorial <10min_tut_10_text>`  .. raw:: html  </span>  <span class="badge gs-badge-link">  `To user guide <text>`  .. raw:: html  </span>  </div>  </div>  </div>  </div>  </div>  </div>  .. _comingfrom:

Coming from...

Are you familiar with other software for manipulating tablular data? Learn
the pandas-equivalent operations compared to software you already know:  .. panels::  + comparison-card text-center shadow  col-lg-6 col-md-6 col-sm-6 col-xs-12 d-flex

+ comparison-card-r  :img-top: ../_static/logo_r.svg  The R programming language_ provides the  ``data.frame`` data structure and multiple packages, such as  tidyverse_ use and extend ``data.frame``  for convenient data handling functionalities similar to pandas.  +++  .. link-button:: compare_with_r  ref  Learn more  btn-secondary stretched-link

+ comparison-card-sql  :img-top: ../_static/logo_sql.svg  Already familiar to ``SELECT``, ``GROUP BY``, ``JOIN``, etc.?  Most of these SQL manipulations do have equivalents in pandas.  +++  .. link-button:: compare_with_sql  ref  Learn more  btn-secondary stretched-link

+ comparison-card-stata  :img-top: ../_static/logo_stata.svg  The `data set`` included in the `STATA_  statistical software suite corresponds to the pandas ``DataFrame``.  Many of the operations known from STATA have an equivalent in pandas.  +++  .. link-button:: compare_with_stata  ref  Learn more  btn-secondary stretched-link

+ comparison-card-excel  :img-top: ../_static/spreadsheets/logo_excel.svg  Users of Excel_  or other spreadsheet programs will find that many of the concepts are  transferrable to pandas.  +++  .. link-button:: compare_with_spreadsheets  ref  Learn more  btn-secondary stretched-link

+ comparison-card-sas  :img-top: ../_static/logo_sas.svg  The SAS_ statistical software suite  also provides the ``data set`` corresponding to the pandas ``DataFrame``.  Also SAS vectorized operations, filtering, string processing operations,  and more have similar functions in pandas.  +++  .. link-button:: compare_with_sas  ref  Learn more  btn-secondary stretched-link

Tutorials

For a quick overview of pandas functionality, see `10 Minutes to pandas<10min>`.  You can also reference the pandas cheat sheet
for a succinct guide for manipulating data with pandas.  The community produces a wide variety of tutorials available online. Some of the
material is enlisted in the community contributed `communitytutorials`.  .. If you update this toctree, also update the manual toctree in the  main index.rst.template  .. toctree::  2  install  overview  intro_tutorials/index  comparison/index  tutorials

The easiest way to install pandas is to install it
as part of the Anaconda_ distribution, a
cross platform distribution for data analysis and scientific computing.
This is the recommended installation method for most users.  Instructions for installing from source,
PyPI_, ActivePython_, various Linux distributions, or a
development version_ are also provided.  .. _install.version:

Python version support

Officially Python 3.8, 3.9, 3.10 and 3.11.

.. _install.anaconda:  Installing with Anaconda
~~~  Installing pandas and the rest of the NumPy_ and
SciPy_ stack can be a little
difficult for inexperienced users.  The simplest way to install not only pandas, but Python and the most popular
packages that make up the SciPy_ stack
(IPython_, NumPy_,
Matplotlib_, ...) is with
Anaconda_, a cross-platform
(Linux, macOS, Windows) Python distribution for data analytics and
scientific computing.  After running the installer, the user will have access to pandas and the
rest of the SciPy_ stack without needing to install
anything else, and without needing to wait for any software to be compiled.  Installation instructions for Anaconda_
can be found here_.  A full list of the packages available as part of the
Anaconda_ distribution
can be found here_.  Another advantage to installing Anaconda is that you don't need
admin rights to install it. Anaconda can install in the user's home directory,
which makes it trivial to delete Anaconda if you decide (just delete
that folder).  .. _install.miniconda:  Installing with Miniconda
~~~  The previous section outlined how to get pandas installed as part of the
Anaconda_ distribution.
However this approach means you will install well over one hundred packages
and involves downloading the installer which is a few hundred megabytes in size.  If you want to have more control on which packages, or have a limited internet
bandwidth, then installing pandas with
Miniconda_ may be a better solution.  Conda_ is the package manager that the
Anaconda_ distribution is built upon.
It is a package manager that is both cross-platform and language agnostic
(it can play a similar role to a pip and virtualenv combination).  Miniconda_ allows you to create a
minimal self contained Python installation, and then use the
Conda_ command to install additional packages.  First you will need Conda_ to be installed and
downloading and running the `Miniconda
<https://conda.pydata.org/miniconda.html>`__
will do this for you. The installer
can be found here_  The next step is to create a new conda environment. A conda environment is like a
virtualenv that allows you to specify a specific version of Python and set of libraries.
Run the following commands from a terminal window::  conda create -n name_of_my_env python  This will create a minimal environment with only Python installed in it.
To put your self inside this environment run::  source activate name_of_my_env  On Windows the command is::  activate name_of_my_env  The final step required is to install pandas. This can be done with the
following command::  conda install pandas  To install a specific pandas version::  conda install pandas=0.20.3  To install other packages, IPython for example::  conda install ipython  To install the full Anaconda_
distribution::  conda install anaconda  If you need packages that are available to pip but not conda, then
install pip, and then use pip to install those packages::  conda install pip  pip install django  Installing from PyPI
~~~  pandas can be installed via pip from
PyPI_.  .. note::  You must have ``pip>=19.3`` to install from PyPI.  ::  pip install pandas  pandas can also be installed with sets of optional dependencies to enable certain functionality. For example,
to install pandas with the optional dependencies to read Excel files.  ::  pip install "pandas[excel]"  The full list of extras that can be installed can be found in the `dependency section.<install.optional_dependencies>`  Installing with ActivePython
~~~  Installation instructions for
ActivePython_ can be found
here_. Versions
2.7, 3.5 and 3.6 include pandas.  Installing using your Linux distribution's package manager.

~~~  The commands in this table will install pandas for Python 3 from your distribution.  .. csv-table::  "Distribution", "Status", "Download / Repository Link", "Install method"  10, 10, 20, 50  Debian, stable, official Debian repository_ , ``sudo apt-get install python3-pandas``  Debian & Ubuntu, unstable (latest packages), NeuroDebian_ , ``sudo apt-get install python3-pandas``  Ubuntu, stable, official Ubuntu repository_ , ``sudo apt-get install python3-pandas``  OpenSuse, stable, OpenSuse Repository _ , ``zypper in python3-pandas``  Fedora, stable, official Fedora repository _ , ``dnf install python3-pandas``  Centos/RHEL, stable, EPEL repository_ , ``yum install python3-pandas``  **However**, the packages in the linux package managers are often a few versions behind, so
to get the newest version of pandas, it's recommended to install using the ``pip`` or ``conda``
methods described above.  Handling ImportErrors
~~~  If you encounter an ImportError, it usually means that Python couldn't find pandas in the list of available
libraries. Python internally has a list of directories it searches through, to find packages. You can
obtain these directories with::  import sys  sys.path  One way you could be encountering this error is if you have multiple Python installations on your system
and you don't have pandas installed in the Python installation you're currently using.
In Linux/Mac you can run ``which python`` on your terminal and it will tell you which Python installation you're
using. If it's something like "/usr/bin/python", you're using the Python from the system, which is not recommended.  It is highly recommended to use ``conda``, for quick installation and for package and dependency updates.
You can find simple installation instructions for pandas in this document: ``installation instructions </getting_started.html>``.  Installing from source
~~~  See the `contributing guide <contributing>` for complete instructions on building from the git source tree. Further, see `creating a development environment <contributing_environment>` if you wish to create a pandas development environment.  Installing the development version of pandas
~~~  Installing a nightly build is the quickest way to:  * Try a new feature that will be shipped in the next release (that is, a feature from a pull-request that was recently merged to the main branch).
* Check whether a bug you encountered has been fixed since the last release.  You can install the nightly build of pandas using the scipy-wheels-nightly index from the PyPI registry of anaconda.org with the following command::  pip install --pre --extra-index https://pypi.anaconda.org/scipy-wheels-nightly/simple pandas  Note that first uninstalling pandas might be required to be able to install nightly builds::  pip uninstall pandas -y

pandas is equipped with an exhaustive set of unit tests, covering about 97% of
the code base as of this writing. To run it on your machine to verify that
everything is working (and that you have all of the dependencies, soft and hard,
installed), make sure you have `pytest
<https://docs.pytest.org/en/latest/>`__ >= 7.0 and `Hypothesis
<https://hypothesis.readthedocs.io/en/latest/>`__ >= 6.34.2, then run:  ::  >>> pd.test()  running: pytest --skip-slow --skip-network --skip-db /home/user/anaconda3/lib/python3.9/site-packages/pandas  === test session starts ===  platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0  rootdir: /home/user  plugins: dash-1.19.0, anyio-3.5.0, hypothesis-6.29.3  collected 154975 items / 4 skipped / 154971 selected  ... [  0%]  ... [ 99%]  ...  [100%]  === ERRORS ===  === FAILURES ===  === warnings summary ===  === short test summary info ===  = 1 failed, 146194 passed, 7402 skipped, 1367 xfailed, 5 xpassed, 197 warnings, 10 errors in 1090.16s (010) =  This is just an example of what information is shown. You might see a slightly different result as what is shown above.  .. _install.dependencies:

.. _install.required_dependencies:  Required dependencies
~~~  pandas requires the following dependencies.  === ===
Package  Minimum supported version
=== ===
NumPy_  1.20.3
python-dateutil_ 2.8.2
pytz_  2020.1
=== ===  .. _install.optional_dependencies:  Optional dependencies
~~~  pandas has many optional dependencies that are only used for specific methods.
For example, `pandas.read_hdf` requires the ``pytables`` package, while
`DataFrame.to_markdown` requires the ``tabulate`` package. If the
optional dependency is not installed, pandas will raise an ``ImportError`` when
the method requiring that dependency is called.  If using pip, optional pandas dependencies can be installed or managed in a file (e.g. requirements.txt or pyproject.toml)
as optional extras (e.g.,``pandas[performance, aws]>=1.5.0``). All optional dependencies can be installed with ``pandas[all]``,
and specific sets of dependencies are listed in the sections below.  .. _install.recommended_dependencies:  Performance dependencies (recommended)
^^^  .. note::  You are highly encouraged to install these libraries, as they provide speed improvements, especially  when working with large data sets.  Installable with ``pip install "pandas[performance]"``  === === === ===
Dependency  Minimum Version  pip extra  Notes
numexpr_  2.7.3  performance  Accelerates certain numerical operations by using uses multiple cores as well as smart chunking and caching to achieve large speedups
bottleneck_ 1.3.2  performance  Accelerates certain types of ``nan`` by using specialized cython routines to achieve large speedup.
numba_  0.53.1  performance  Alternative execution engine for operations that accept ``engine="numba"`` using a JIT compiler that translates Python functions to optimized machine code using the LLVM compiler.
=== === === ===  Timezones
^^^  Installable with ``pip install "pandas[timezone]"``  === === === ===
tzdata  2022.1(pypi)/  timezone  Allows the use of ``zoneinfo`` timezones with pandas.  2022a(for system tzdata)  **Note**: You only need to install the pypi package if your  system does not already provide the IANA tz database.  However, the minimum tzdata version still applies, even if it  is not enforced through an error.  If you would like to keep your system tzdata version updated,  it is recommended to use the ``tzdata`` package from  conda-forge.
=== === === ===  Visualization
^^^  Installable with ``pip install "pandas[plot, output_formatting]"``.  === === === ===
matplotlib  3.6.1  plot  Plotting library
Jinja2  3.0.0  output_formatting  Conditional formatting with DataFrame.style
tabulate  0.8.9  output_formatting  Printing in Markdown-friendly format (see `tabulate`_)
=== === === ===  Computation
^^^  Installable with ``pip install "pandas[computation]"``.  === === === ===
SciPy  1.7.1  computation  Miscellaneous statistical functions
xarray  0.21.0  computation  pandas-like API for N-dimensional data
=== === === ===  Excel files
^^^  Installable with ``pip install "pandas[excel]"``.  === === === ===
xlrd  2.0.1  excel  Reading Excel
xlsxwriter  1.4.3  excel  Writing Excel
openpyxl  3.0.7  excel  Reading / writing for xlsx files
pyxlsb  1.0.8  excel  Reading for xlsb files
=== === === ===  HTML
^^^  Installable with ``pip install "pandas[html]"``.  === === === ===
BeautifulSoup4  4.9.3  html  HTML parser for read_html
html5lib  1.1  html  HTML parser for read_html
lxml  4.6.3  html  HTML parser for read_html
=== === === ===  One of the following combinations of libraries is needed to use the
top-level `~pandas.read_html` function:  * `BeautifulSoup4`_ and `html5lib`_
* `BeautifulSoup4`_ and `lxml`_
* `BeautifulSoup4`_ and `html5lib`_ and `lxml`_

* Only `lxml`_, although see `HTML Table Parsing <io.html.gotchas>`  for reasons as to why you should probably **not** take this approach.  .. warning::  * if you install `BeautifulSoup4`_ you must install either  `lxml`_ or `html5lib`_ or both.  `~pandas.read_html` will **not** work with *only*  `BeautifulSoup4`_ installed.  * You are highly encouraged to read `HTML Table Parsing gotchas <io.html.gotchas>`.  It explains issues surrounding the installation and  usage of the above three libraries.  .. _html5lib: https://github.com/html5lib/html5lib-python
.. _BeautifulSoup4: https://www.crummy.com/software/BeautifulSoup
.. _lxml: https://lxml.de
.. _tabulate: https://github.com/astanin/python-tabulate  XML
^^^  Installable with ``pip install "pandas[xml]"``.  === === === ===
lxml  4.6.3  xml  XML parser for read_xml and tree builder for to_xml
=== === === ===  SQL databases
^^^  Installable with ``pip install "pandas[postgresql, mysql, sql-other]"``.  === === === ===
SQLAlchemy  1.4.16  postgresql,  SQL support for databases other than sqlite  mysql,  sql-other
psycopg2  2.8.6  postgresql  PostgreSQL engine for sqlalchemy
pymysql  1.0.2  mysql  MySQL engine for sqlalchemy
=== === === ===  Other data sources
^^^  Installable with ``pip install "pandas[hdf5, parquet, feather, spss, excel]"``  === === === ===
PyTables  3.6.1  hdf5  HDF5-based reading / writing
blosc  1.21.0  hdf5  Compression for HDF5; only available on ``conda``
zlib  hdf5  Compression for HDF5
fastparquet  0.6.3  -  Parquet reading / writing (pyarrow is default)
pyarrow  7.0.0  parquet, feather Parquet, ORC, and feather reading / writing
pyreadstat  1.1.2  spss  SPSS files (.sav) reading
odfpy  1.4.1  excel  Open document format (.odf, .ods, .odt) reading / writing
=== === === ===  .. _install.warn_orc:  .. warning::  * If you want to use `~pandas.read_orc`, it is highly recommended to install pyarrow using conda.  The following is a summary of the environment in which `~pandas.read_orc` can work.  === === ===  System  Conda  PyPI  === === ===  Linux  Successful  Failed  macOS  Successful  Failed  Windows  Failed  Failed  === === ===  Access data in the cloud
^^^  Installable with ``pip install "pandas[fss, aws, gcp]"``  === === === ===
fsspec  2021.7.0  fss, gcp, aws  Handling files aside from simple local and HTTP (required  dependency of s3fs, gcsfs).
gcsfs  2021.7.0  gcp  Google Cloud Storage access
pandas-gbq  0.15.0  gcp  Google Big Query access
s3fs  2021.08.0  aws  Amazon S3 access
=== === === ===  Clipboard
^^^  Installable with ``pip install "pandas[clipboard]"``.  === === === ===
PyQt4/PyQt5  5.15.1  clipboard  Clipboard I/O
qtpy  2.2.0  clipboard  Clipboard I/O
=== === === ===  .. note::  Depending on operating system, system-level packages may need to installed.  For clipboard to operate on Linux one of the CLI tools ``xclip`` or ``xsel`` must be installed on your system.  Compression
^^^  Installable with ``pip install "pandas[compression]"``  === === === ===
brotli  0.7.0  compression  Brotli compression
python-snappy  0.6.0  compression  Snappy compression
Zstandard  0.15.2  compression  Zstandard compression

Data structures

.. csv-table::  "Dimensions", "Name", "Description"  15, 20, 50  1, "Series", "1D labeled homogeneously-typed array"  2, "DataFrame", "General 2D labeled, size-mutable tabular structure with potentially heterogeneously-typed column"  Why more than one data structure?
~~~  The best way to think about the pandas data structures is as flexible
containers for lower dimensional data. For example, DataFrame is a container
for Series, and Series is a container for scalars. We would like to be
able to insert and remove objects from these containers in a dictionary-like
fashion.  Also, we would like sensible default behaviors for the common API functions
which take into account the typical orientation of time series and
cross-sectional data sets. When using the N-dimensional array (ndarrays) to store 2- and 3-dimensional
data, a burden is placed on the user to consider the orientation of the data
set when writing functions; axes are considered more or less equivalent (except
when C- or Fortran-contiguousness matters for performance). In pandas, the axes
are intended to lend more semantic meaning to the data; i.e., for a particular
data set, there is likely to be a "right" way to orient the data. The goal,
then, is to reduce the amount of mental effort required to code up data
transformations in downstream functions.  For example, with tabular data (DataFrame) it is more semantically helpful to
think of the **index** (the rows) and the **columns** rather than axis 0 and
axis 1. Iterating through the columns of the DataFrame thus results in more
readable code:  ::  for col in df.columns:  series = df[col]  # do something with series

Mutability and copying of data

All pandas data structures are value-mutable (the values they contain can be
altered) but not always size-mutable. The length of a Series cannot be
changed, but, for example, columns can be inserted into a DataFrame. However,
the vast majority of methods produce new objects and leave the input data
untouched. In general we like to **favor immutability** where sensible.

Getting support

The first stop for pandas issues and ideas is the `GitHub Issue Tracker
<https://github.com/pandas-dev/pandas/issues>`__. If you have a general question,
pandas community experts can answer through `Stack Overflow
<https://stackoverflow.com/questions/tagged/pandas>`__.

Community

pandas is actively supported today by a community of like-minded individuals around
the world who contribute their valuable time and energy to help make open source
pandas possible. Thanks to all of our contributors_.  If you're interested in contributing, please visit the `contributing guide <contributing>`.  pandas is a NumFOCUS_ sponsored project.
This will help ensure the success of the development of pandas as a world-class open-source
project and makes it possible to donate_ to the project.

Project governance

The governance process that pandas project has used informally since its inception in 2008 is formalized in Project Governance documents_.
The documents clarify how decisions are made and how the various elements of our community interact, including the relationship between open source collaborative development and work that may be funded by for-profit or non-profit entities.  Wes McKinney is the Benevolent Dictator for Life (BDFL).

Development team

The list of the Core Team members and more detailed information can be found on the people’s page_ of the governance repo.

Institutional partners

The information about current institutional partners can be found on pandas website page_.

pandas cookbook by Julia Evans

The goal of this 2015 cookbook (by Julia Evans) is to
give you some concrete examples for getting started with pandas. These
are examples with real-world data, and all the bugs and weirdness that
entails.
For the table of contents, see the `pandas-cookbook GitHub
repository <https://github.com/jvns/pandas-cookbook>`_.

pandas workshop by Stefanie Molin

An introductory workshop by Stefanie Molin
designed to quickly get you up to speed with pandas using real-world datasets.
It covers getting started with pandas, data wrangling, and data visualization
(with some exposure to matplotlib and seaborn). The
pandas-workshop GitHub repository
features detailed environment setup instructions (including a Binder environment),
slides and notebooks for following along, and exercises to practice the concepts.
There is also a lab with new exercises on a dataset not covered in the workshop for
additional practice.

Learn pandas by Hernan Rojas

A set of lesson for new pandas users: https://bitbucket.org/hrojas/learn-pandas

Practical data analysis with Python

This guide is an introduction to the data analysis process using the Python data ecosystem and an interesting open dataset.
There are four sections covering selected topics as munging data_,
aggregating data, visualizing data
and time series.  .. _tutorial-exercises-new-users:

Exercises for new users
Practice your skills with real data sets and exercises.
For more resources, please visit the main repository_.  .. _tutorial-modern:

Modern pandas

Tutorial series written in 2016 by
Tom Augspurger.
The source may be found in the GitHub repository
TomAugspurger/effective-pandas.  * Modern Pandas
* Method Chaining
* Indexes
* Performance
* Tidy Data
* Visualization
* Timeseries

Excel charts with pandas, vincent and xlsxwriter

*  Using Pandas and XlsxWriter to create Excel charts

Joyful pandas

A tutorial written in Chinese by Yuanhao Geng. It covers the basic operations
for NumPy and pandas, 4 main data manipulation methods (including indexing, groupby, reshaping
and concatenation) and 4 main data types (including missing data, string data, categorical
data and time series data). At the end of each chapter, corresponding exercises are posted.
All the datasets and related materials can be found in the GitHub repository
datawhalechina/joyful-pandas.

Video tutorials

* Pandas From The Ground Up  (2015) (2:24)  GitHub repo_
* Introduction Into Pandas  (2016) (1:28)  GitHub repo_
* Pandas: .head() to .tail()  (2016) (1:26)  GitHub repo_
* Data analysis in Python with pandas  (2016-2018)  GitHub repo_ and  Jupyter Notebook_
* Best practices with pandas  (2018)  GitHub repo_ and  Jupyter Notebook_

Various tutorials

* Wes McKinney's (pandas BDFL) blog
* Statistical analysis made easy in Python with SciPy and pandas DataFrames, by Randal Olson
* Statistical Data Analysis in Python, tutorial videos, by Christopher Fonnesbeck from SciPy 2013
* Financial analysis in Python, by Thomas Wiecki
* Intro to pandas data structures, by Greg Reda
* Pandas and Python: Top 10, by Manish Amde
* Pandas DataFrames Tutorial, by Karlijn Willems
* A concise tutorial with real life examples
* 430+ Searchable Pandas recipes by Isshin Inada

Quick reference

We'll start off with a quick reference guide pairing some common R
operations using `dplyr
<https://cran.r-project.org/web/packages/dplyr/index.html>`__ with
pandas equivalents.  Querying, filtering, sampling
~~~  ===  ===
R  pandas
===  ===
``dim(df)``  ``df.shape``
``head(df)``  ``df.head()``
``slice(df, 1:10)``  ``df.iloc[:9]``
``filter(df, col1 == 1, col2 == 1)``  ``df.query('col1 == 1 & col2 == 1')``
``df[df$col1 == 1 & df$col2 == 1,]``  ``df[(df.col1 == 1) & (df.col2 == 1)]``
``select(df, col1, col2)``  ``df[['col1', 'col2']]``
``select(df, col1:col3)``  ``df.loc[:, 'col1':'col3']``
``select(df, -(col1:col3))``  ``df.drop(cols_to_drop, axis=1)`` but see [#select_range]_
``distinct(select(df, col1))``  ``df[['col1']].drop_duplicates()``
``distinct(select(df, col1, col2))``  ``df[['col1', 'col2']].drop_duplicates()``
``sample_n(df, 10)``  ``df.sample(n=10)``
``sample_frac(df, 0.01)``  ``df.sample(frac=0.01)``
===  ===  .. [#select_range] R's shorthand for a subrange of columns  (``select(df, col1:col3)``) can be approached  cleanly in pandas, if you have the list of columns,  for example ``df[cols[1:3]]`` or  ``df.drop(cols[1:3])``, but doing this by column  name is a bit messy.  Sorting
~~~  ===  ===
R  pandas
===  ===
``arrange(df, col1, col2)``  ``df.sort_values(['col1', 'col2'])``
``arrange(df, desc(col1))``  ``df.sort_values('col1', ascending=False)``
===  ===  Transforming
~~~  ===  ===
R  pandas
===  ===
``select(df, col_one = col1)``  ``df.rename(columns={'col1': 'col_one'})['col_one']``
``rename(df, col_one = col1)``  ``df.rename(columns={'col1': 'col_one'})``
``mutate(df, c=a-b)``  ``df.assign(c=df['a']-df['b'])``
===  ===  Grouping and summarizing
~~~  ===  ===
R  pandas
===  ===
``summary(df)``  ``df.describe()``
``gdf <- group_by(df, col1)``  ``gdf = df.groupby('col1')``
``summarise(gdf, avg=mean(col1, na.rm=TRUE))``  ``df.groupby('col1').agg({'col1': 'mean'})``
``summarise(gdf, total=sum(col1))``  ``df.groupby('col1').sum()``
===  ===

Slicing with R's |c|_
~~~  R makes it easy to access ``data.frame`` columns by name  .. code-block:: r  df <- data.frame(a=rnorm(5), b=rnorm(5), c=rnorm(5), d=rnorm(5), e=rnorm(5))  df[, c("a", "c", "e")]  or by integer location  .. code-block:: r  df <- data.frame(matrix(rnorm(1000), ncol=100))  df[, c(1:10, 25:30, 40, 50:100)]  Selecting multiple columns by name in pandas is straightforward  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 3), columns=list("abc"))  df[["a", "c"]]  df.loc[:, ["a", "c"]]  Selecting multiple noncontiguous columns by integer location can be achieved
with a combination of the ``iloc`` indexer attribute and ``numpy.r_``.  .. ipython:: python  named = list("abcdefg")  n = 30  columns = named + np.arange(len(named), n).tolist()  df = pd.DataFrame(np.random.randn(n, n), columns=columns)  df.iloc[:, np.r_[:10, 24:30]]  |aggregate|_
~~~  In R you may want to split data into subsets and compute the mean for each.
Using a data.frame called ``df`` and splitting it into groups ``by1`` and
``by2``:  .. code-block:: r  df <- data.frame(  v1 = c(1,3,5,7,8,3,5,NA,4,5,7,9),  v2 = c(11,33,55,77,88,33,55,NA,44,55,77,99),  by1 = c("red", "blue", 1, 2, NA, "big", 1, 2, "red", 1, NA, 12),  by2 = c("wet", "dry", 99, 95, NA, "damp", 95, 99, "red", 99, NA, NA))  aggregate(x=df[, c("v1", "v2")], by=list(mydf2$by1, mydf2$by2), FUN = mean)  The `~pandas.DataFrame.groupby` method is similar to base R ``aggregate``
function.  .. ipython:: python  df = pd.DataFrame(  {  "v1": [1, 3, 5, 7, 8, 3, 5, np.nan, 4, 5, 7, 9],  "v2": [11, 33, 55, 77, 88, 33, 55, np.nan, 44, 55, 77, 99],  "by1": ["red", "blue", 1, 2, np.nan, "big", 1, 2, "red", 1, np.nan, 12],  "by2": [  "wet",  "dry",  99,  95,  np.nan,  "damp",  95,  99,  "red",  99,  np.nan,  np.nan,  ],  }  )  g = df.groupby(["by1", "by2"])  g[["v1", "v2"]].mean()  For more details and examples see `the groupby documentation
<groupby.split>`.  |match|_
~~~  A common way to select data in R is using ``%in%`` which is defined using the
function ``match``. The operator ``%in%`` is used to return a logical vector
indicating if there is a match or not:  .. code-block:: r  s <- 0:4  s %in% c(2,4)  The `~pandas.DataFrame.isin` method is similar to R ``%in%`` operator:  .. ipython:: python  s = pd.Series(np.arange(5), dtype=np.float32)  s.isin([2, 4])  The ``match`` function returns a vector of the positions of matches
of its first argument in its second:  .. code-block:: r  s <- 0:4  match(s, c(2,4))  For more details and examples see `the reshaping documentation
<indexing.basics.indexing_isin>`.  |tapply|_
~~~  ``tapply`` is similar to ``aggregate``, but data can be in a ragged array,
since the subclass sizes are possibly irregular. Using a data.frame called
``baseball``, and retrieving information based on the array ``team``:  .. code-block:: r  baseball <-  data.frame(team = gl(5, 5,  labels = paste("Team", LETTERS[1:5])),  player = sample(letters, 25),  batting.average = runif(25, .200, .400))  tapply(baseball$batting.average, baseball.example$team,  max)  In pandas we may use `~pandas.pivot_table` method to handle this:  .. ipython:: python  import random  import string  baseball = pd.DataFrame(  {  "team": ["team %d" % (x + 1) for x in range(5)] * 5,  "player": random.sample(list(string.ascii_lowercase), 25),  "batting avg": np.random.uniform(0.200, 0.400, 25),  }  )  baseball.pivot_table(values="batting avg", columns="team", aggfunc=np.max)  For more details and examples see `the reshaping documentation
<reshaping.pivot>`.  |subset|_
~~~  The `~pandas.DataFrame.query` method is similar to the base R ``subset``
function. In R you might want to get the rows of a ``data.frame`` where one
column's values are less than another column's values:  .. code-block:: r  df <- data.frame(a=rnorm(10), b=rnorm(10))  subset(df, a <= b)  df[df$a <= df$b,]  # note the comma  In pandas, there are a few ways to perform subsetting. You can use
`~pandas.DataFrame.query` or pass an expression as if it were an

index/slice as well as standard boolean indexing:  .. ipython:: python  df = pd.DataFrame({"a": np.random.randn(10), "b": np.random.randn(10)})  df.query("a <= b")  df[df["a"] <= df["b"]]  df.loc[df["a"] <= df["b"]]  For more details and examples see `the query documentation
<indexing.query>`.  |with|_
~~~  An expression using a data.frame called ``df`` in R with the columns ``a`` and
``b`` would be evaluated using ``with`` like so:  .. code-block:: r  df <- data.frame(a=rnorm(10), b=rnorm(10))  with(df, a + b)  df$a + df$b  # same as the previous expression  In pandas the equivalent expression, using the
`~pandas.DataFrame.eval` method, would be:  .. ipython:: python  df = pd.DataFrame({"a": np.random.randn(10), "b": np.random.randn(10)})  df.eval("a + b")  df["a"] + df["b"]  # same as the previous expression  In certain cases `~pandas.DataFrame.eval` will be much faster than
evaluation in pure Python. For more details and examples see `the eval
documentation <enhancingperf.eval>`.

plyr

``plyr`` is an R library for the split-apply-combine strategy for data
analysis. The functions revolve around three data structures in R, ``a``
for ``arrays``, ``l`` for ``lists``, and ``d`` for ``data.frame``. The
table below shows how these data structures could be mapped in Python.  | R  | Python  |
+===+===+
| array  | list  |  | lists  | dictionary or list of objects |  | data.frame | dataframe  |  ddply
~~~  An expression using a data.frame called ``df`` in R where you want to
summarize ``x`` by ``month``:  .. code-block:: r  require(plyr)  df <- data.frame(  x = runif(120, 1, 168),  y = runif(120, 7, 334),  z = runif(120, 1.7, 20.7),  month = rep(c(5,6,7,8),30),  week = sample(1:4, 120, TRUE)  )  ddply(df, .(month, week), summarize,  mean = round(mean(x), 2),  sd = round(sd(x), 2))  In pandas the equivalent expression, using the
`~pandas.DataFrame.groupby` method, would be:  .. ipython:: python  df = pd.DataFrame(  {  "x": np.random.uniform(1.0, 168.0, 120),  "y": np.random.uniform(7.0, 334.0, 120),  "z": np.random.uniform(1.7, 20.7, 120),  "month": [5, 6, 7, 8] * 30,  "week": np.random.randint(1, 4, 120),  }  )  grouped = df.groupby(["month", "week"])  grouped["x"].agg([np.mean, np.std])  For more details and examples see `the groupby documentation
<groupby.aggregate>`.

meltarray
~~~  An expression using a 3 dimensional array called ``a`` in R where you want to
melt it into a data.frame:  .. code-block:: r  a <- array(c(1:23, NA), c(2,3,4))  data.frame(melt(a))  In Python, since ``a`` is a list, you can simply use list comprehension.  .. ipython:: python  a = np.array(list(range(1, 24)) + [np.NAN]).reshape(2, 3, 4)  pd.DataFrame([tuple(list(x) + [val]) for x, val in np.ndenumerate(a)])  meltlist
~~~  An expression using a list called ``a`` in R where you want to melt it
into a data.frame:  .. code-block:: r  a <- as.list(c(1:4, NA))  data.frame(melt(a))  In Python, this list would be a list of tuples, so
`~pandas.DataFrame` method would convert it to a dataframe as required.  .. ipython:: python  a = list(enumerate(list(range(1, 5)) + [np.NAN]))  pd.DataFrame(a)  For more details and examples see `the Into to Data Structures
documentation <dsintro>`.  meltdf
~~~  An expression using a data.frame called ``cheese`` in R where you want to
reshape the data.frame:  .. code-block:: r  cheese <- data.frame(  first = c('John', 'Mary'),  last = c('Doe', 'Bo'),  height = c(5.5, 6.0),  weight = c(130, 150)  )  melt(cheese, id=c("first", "last"))  In Python, the `~pandas.melt` method is the R equivalent:  .. ipython:: python  cheese = pd.DataFrame(  {  "first": ["John", "Mary"],  "last": ["Doe", "Bo"],  "height": [5.5, 6.0],  "weight": [130, 150],  }  )  pd.melt(cheese, id_vars=["first", "last"])  cheese.set_index(["first", "last"]).stack()  # alternative way  For more details and examples see `the reshaping documentation
<reshaping.melt>`.  cast
~~~  In R ``acast`` is an expression using a data.frame called ``df`` in R to cast
into a higher dimensional array:  .. code-block:: r  df <- data.frame(  x = runif(12, 1, 168),  y = runif(12, 7, 334),  z = runif(12, 1.7, 20.7),  month = rep(c(5,6,7),4),  week = rep(c(1,2), 6)  )  mdf <- melt(df, id=c("month", "week"))  acast(mdf, week ~ month ~ variable, mean)  In Python the best way is to make use of `~pandas.pivot_table`:  .. ipython:: python  df = pd.DataFrame(  {  "x": np.random.uniform(1.0, 168.0, 12),  "y": np.random.uniform(7.0, 334.0, 12),  "z": np.random.uniform(1.7, 20.7, 12),  "month": [5, 6, 7] * 4,  "week": [1, 2] * 6,  }  )  mdf = pd.melt(df, id_vars=["month", "week"])  pd.pivot_table(  mdf,  values="value",  index=["variable", "week"],  columns=["month"],  aggfunc=np.mean,  )  Similarly for ``dcast`` which uses a data.frame called ``df`` in R to
aggregate information based on ``Animal`` and ``FeedType``:  .. code-block:: r  df <- data.frame(  Animal = c('Animal1', 'Animal2', 'Animal3', 'Animal2', 'Animal1',  'Animal2', 'Animal3'),  FeedType = c('A', 'B', 'A', 'A', 'B', 'B', 'A'),  Amount = c(10, 7, 4, 2, 5, 6, 2)  )  dcast(df, Animal ~ FeedType, sum, fill=NaN)  # Alternative method using base R  with(df, tapply(Amount, list(Animal, FeedType), sum))  Python can approach this in two different ways. Firstly, similar to above
using `~pandas.pivot_table`:  .. ipython:: python  df = pd.DataFrame(  {  "Animal": [  "Animal1",  "Animal2",  "Animal3",  "Animal2",  "Animal1",  "Animal2",  "Animal3",  ],  "FeedType": ["A", "B", "A", "A", "B", "B", "A"],  "Amount": [10, 7, 4, 2, 5, 6, 2],  }  )  df.pivot_table(values="Amount", index="Animal", columns="FeedType", aggfunc="sum")  The second approach is to use the `~pandas.DataFrame.groupby` method:  .. ipython:: python  df.groupby(["Animal", "FeedType"])["Amount"].sum()  For more details and examples see `the reshaping documentation
<reshaping.pivot>` or `the groupby documentation<groupby.split>`.  |factor|_
~~~  pandas has a data type for categorical data.  .. code-block:: r  cut(c(1,2,3,4,5,6), 3)  factor(c(1,2,3,2,2,3))  In pandas this is accomplished with ``pd.cut`` and ``astype("category")``:  .. ipython:: python  pd.cut(pd.Series([1, 2, 3, 4, 5, 6]), 3)  pd.Series([1, 2, 3, 2, 2, 3]).astype("category")  For more details and examples see `categorical introduction <categorical>` and the

`API documentation <api.arrays.categorical>`. There is also a documentation regarding the
`differences to R's factor <categorical.rfactor>`.  .. |c| replace:: ``c``
.. _c: https://stat.ethz.ch/R-manual/R-patched/library/base/html/c.html  .. |aggregate| replace:: ``aggregate``
.. _aggregate: https://stat.ethz.ch/R-manual/R-patched/library/stats/html/aggregate.html  .. |match| replace:: ``match`` / ``%in%``
.. _match: https://stat.ethz.ch/R-manual/R-patched/library/base/html/match.html  .. |tapply| replace:: ``tapply``
.. _tapply: https://stat.ethz.ch/R-manual/R-patched/library/base/html/tapply.html  .. |with| replace:: ``with``
.. _with: https://stat.ethz.ch/R-manual/R-patched/library/base/html/with.html  .. |subset| replace:: ``subset``
.. _subset: https://stat.ethz.ch/R-manual/R-patched/library/base/html/subset.html  .. |factor| replace:: ``factor``
.. _factor: https://stat.ethz.ch/R-manual/R-devel/library/base/html/factor.html

General terminology translation
~~~  .. csv-table::  "pandas", "SAS"  20, 20  ``DataFrame``, data set  column, variable  row, observation  groupby, BY-group  ``NaN``, ``.``  ``DataFrame``
~~~  A ``DataFrame`` in pandas is analogous to a SAS data set - a two-dimensional
data source with labeled columns that can be of different types. As will be
shown in this document, almost any operation that can be applied to a data set
using SAS's ``DATA`` step, can also be accomplished in pandas.  ``Series``
~~~  A ``Series`` is the data structure that represents one column of a
``DataFrame``. SAS doesn't have a separate data structure for a single column,
but in general, working with a ``Series`` is analogous to referencing a column
in the ``DATA`` step.  ``Index``
~~~  Every ``DataFrame`` and ``Series`` has an ``Index`` - which are labels on the
*rows* of the data. SAS does not have an exactly analogous concept. A data set's
rows are essentially unlabeled, other than an implicit integer index that can be
accessed during the ``DATA`` step (``_N_``).  In pandas, if no index is specified, an integer index is also used by default
(first row = 0, second row = 1, and so on). While using a labeled ``Index`` or
``MultiIndex`` can enable sophisticated analyses and is ultimately an important
part of pandas to understand, for this comparison we will essentially ignore the
``Index`` and just treat the ``DataFrame`` as a collection of columns. Please
see the `indexing documentation<indexing>` for much more on how to use an
``Index`` effectively.  Copies vs. in place operations
~~~  .. include:: includes/copies.rst

Data input / output

Constructing a DataFrame from values
~~~  A SAS data set can be built from specified values by
placing the data after a ``datalines`` statement and
specifying the column names.  .. code-block:: sas  data df;  input x y;  datalines;  1 2  3 4  5 6  ;  run;  .. include:: includes/construct_dataframe.rst  Reading external data
~~~  Like SAS, pandas provides utilities for reading in data from
many formats.  The ``tips`` dataset, found within the pandas
tests (csv)
will be used in many of the following examples.  SAS provides ``PROC IMPORT`` to read csv data into a data set.  .. code-block:: sas  proc import datafile='tips.csv' dbms=csv out=tips replace;  getnames=yes;  run;  The pandas method is `read_csv`, which works similarly.  .. ipython:: python  url = (  "https://raw.githubusercontent.com/pandas-dev/"  "pandas/main/pandas/tests/io/data/csv/tips.csv"  )  tips = pd.read_csv(url)  tips  Like ``PROC IMPORT``, ``read_csv`` can take a number of parameters to specify
how the data should be parsed.  For example, if the data was instead tab delimited,
and did not have column names, the pandas command would be:  .. code-block:: python  tips = pd.read_csv("tips.csv", sep="\t", header=None)  # alternatively, read_table is an alias to read_csv with tab delimiter  tips = pd.read_table("tips.csv", header=None)  In addition to text/csv, pandas supports a variety of other data formats
such as Excel, HDF5, and SQL databases.  These are all read via a ``pd.read_*``
function.  See the `IO documentation<io>` for more details.  Limiting output
~~~  .. include:: includes/limit.rst  The equivalent in SAS would be:  .. code-block:: sas  proc print data=df(obs=5);  run;  Exporting data
~~~  The inverse of ``PROC IMPORT`` in SAS is ``PROC EXPORT``  .. code-block:: sas  proc export data=tips outfile='tips2.csv' dbms=csv;  run;  Similarly in pandas, the opposite of ``read_csv`` is `~DataFrame.to_csv`,
and other data formats follow a similar api.  .. code-block:: python  tips.to_csv("tips2.csv")

Data operations

Operations on columns
~~~  In the ``DATA`` step, arbitrary math expressions can
be used on new or existing columns.  .. code-block:: sas  data tips;  set tips;  total_bill = total_bill - 2;  new_bill = total_bill / 2;  run;  .. include:: includes/column_operations.rst  Filtering
~~~  Filtering in SAS is done with an ``if`` or ``where`` statement, on one
or more columns.  .. code-block:: sas  data tips;  set tips;  if total_bill > 10;  run;  data tips;  set tips;  where total_bill > 10;  /* equivalent in this case - where happens before the  DATA step begins and can also be used in PROC statements */  run;  .. include:: includes/filtering.rst  If/then logic
~~~  In SAS, if/then logic can be used to create new columns.  .. code-block:: sas  data tips;  set tips;  format bucket $4.;  if total_bill < 10 then bucket = 'low';  else bucket = 'high';  run;  .. include:: includes/if_then.rst  Date functionality
~~~  SAS provides a variety of functions to do operations on
date/datetime columns.  .. code-block:: sas  data tips;  set tips;  format date1 date2 date1_plusmonth mmddyy10.;  date1 = mdy(1, 15, 2013);  date2 = mdy(2, 15, 2015);  date1_year = year(date1);  date2_month = month(date2);  * shift date to beginning of next interval;  date1_next = intnx('MONTH', date1, 1);  * count intervals between dates;  months_between = intck('MONTH', date1, date2);  run;  The equivalent pandas operations are shown below.  In addition to these
functions pandas supports other Time Series features
not available in Base SAS (such as resampling and custom offsets) -
see the `timeseries documentation<timeseries>` for more details.  .. include:: includes/time_date.rst  Selection of columns
~~~  SAS provides keywords in the ``DATA`` step to select,
drop, and rename columns.  .. code-block:: sas  data tips;  set tips;  keep sex total_bill tip;  run;  data tips;  set tips;  drop sex;  run;  data tips;  set tips;  rename total_bill=total_bill_2;  run;  .. include:: includes/column_selection.rst  Sorting by values
~~~  Sorting in SAS is accomplished via ``PROC SORT``  .. code-block:: sas  proc sort data=tips;  by sex total_bill;  run;  .. include:: includes/sorting.rst

String processing

Finding length of string
~~~  SAS determines the length of a character string with the
LENGTHN_
and LENGTHC_
functions. ``LENGTHN`` excludes trailing blanks and ``LENGTHC`` includes trailing blanks.  .. code-block:: sas  data _null_;  set tips;  put(LENGTHN(time));  put(LENGTHC(time));  run;  .. include:: includes/length.rst  Finding position of substring
~~~  SAS determines the position of a character in a string with the
FINDW_ function.
``FINDW`` takes the string defined by the first argument and searches for the first position of the substring
you supply as the second argument.  .. code-block:: sas  data _null_;  set tips;  put(FINDW(sex,'ale'));  run;  .. include:: includes/find_substring.rst  Extracting substring by position
~~~  SAS extracts a substring from a string based on its position with the
SUBSTR_ function.  .. code-block:: sas  data _null_;  set tips;  put(substr(sex,1,1));  run;  .. include:: includes/extract_substring.rst  Extracting nth word
~~~  The SAS SCAN_
function returns the nth word from a string. The first argument is the string you want to parse and the
second argument specifies which word you want to extract.  .. code-block:: sas  data firstlast;  input String $60.;  First_Name = scan(string, 1);  Last_Name = scan(string, -1);  datalines2;  John Smith;  Jane Cook;  ;;;  run;  .. include:: includes/nth_word.rst  Changing case
~~~  The SAS UPCASE_
LOWCASE_ and
PROPCASE_
functions change the case of the argument.  .. code-block:: sas  data firstlast;  input String $60.;  string_up = UPCASE(string);  string_low = LOWCASE(string);  string_prop = PROPCASE(string);  datalines2;  John Smith;  Jane Cook;  ;;;  run;  .. include:: includes/case.rst

Merging

.. include:: includes/merge_setup.rst  In SAS, data must be explicitly sorted before merging.  Different
types of joins are accomplished using the ``in=`` dummy
variables to track whether a match was found in one or both
input frames.  .. code-block:: sas  proc sort data=df1;  by key;  run;  proc sort data=df2;  by key;  run;  data left_join inner_join right_join outer_join;  merge df1(in=a) df2(in=b);  if a and b then output inner_join;  if a then output left_join;  if b then output right_join;  if a or b then output outer_join;  run;  .. include:: includes/merge.rst

Missing data

Both pandas and SAS have a representation for missing data.  .. include:: includes/missing_intro.rst  One difference is that missing data cannot be compared to its sentinel value.
For example, in SAS you could do this to filter missing values.  .. code-block:: sas  data outer_join_nulls;  set outer_join;  if value_x = .;  run;  data outer_join_no_nulls;  set outer_join;  if value_x ^= .;  run;  .. include:: includes/missing.rst

GroupBy

Aggregation
~~~  SAS's ``PROC SUMMARY`` can be used to group by one or
more key variables and compute aggregations on
numeric columns.  .. code-block:: sas  proc summary data=tips nway;  class sex smoker;  var total_bill tip;  output out=tips_summed sum=;  run;  .. include:: includes/groupby.rst  Transformation
~~~  In SAS, if the group aggregations need to be used with
the original frame, it must be merged back together.  For
example, to subtract the mean for each observation by smoker group.  .. code-block:: sas  proc summary data=tips missing nway;  class smoker;  var total_bill;  output out=smoker_means mean(total_bill)=group_bill;  run;  proc sort data=tips;  by smoker;  run;  data tips;  merge tips(in=a) smoker_means(in=b);  by smoker;  adj_total_bill = total_bill - group_bill;  if a and b;  run;  .. include:: includes/transform.rst  By group processing
~~~  In addition to aggregation, pandas ``groupby`` can be used to
replicate most other by group processing from SAS. For example,
this ``DATA`` step reads the data by sex/smoker group and filters to
the first entry for each.  .. code-block:: sas  proc sort data=tips;  by sex smoker;  run;  data tips_first;  set tips;  by sex smoker;  if FIRST.sex or FIRST.smoker then output;  run;  In pandas this would be written as:  .. ipython:: python  tips.groupby(["sex", "smoker"]).first()

Other considerations

Disk vs memory
~~~  pandas operates exclusively in memory, where a SAS data set exists on disk.
This means that the size of data able to be loaded in pandas is limited by your
machine's memory, but also that the operations on that data may be faster.  If out of core processing is needed, one possibility is the
dask.dataframe
library (currently in development) which
provides a subset of pandas functionality for an on-disk ``DataFrame``  Data interop
~~~  pandas provides a `read_sas` method that can read SAS data saved in
the XPORT or SAS7BDAT binary format.  .. code-block:: sas  libname xportout xport 'transport-file.xpt';  data xportout.tips;  set tips(rename=(total_bill=tbill));  * xport variable names limited to 6 characters;  run;  .. code-block:: python  df = pd.read_sas("transport-file.xpt")  df = pd.read_sas("binary-file.sas7bdat")  You can also specify the file format directly. By default, pandas will try
to infer the file format based on its extension.  .. code-block:: python  df = pd.read_sas("transport-file.xpt", format="xport")  df = pd.read_sas("binary-file.sas7bdat", format="sas7bdat")  XPORT is a relatively limited format and the parsing of it is not as
optimized as some of the other pandas readers. An alternative way
to interop data between SAS and pandas is to serialize to csv.  .. code-block:: ipython  # version 0.17, 10M rows  In [8]: %time df = pd.read_sas('big.xpt')  Wall time: 14.6 s  In [9]: %time df = pd.read_csv('big.csv')  Wall time: 4.86 s

~~~  .. csv-table::  "pandas", "Excel"  20, 20  ``DataFrame``, worksheet  ``Series``, column  ``Index``, row headings  row, row  ``NaN``, empty cell  ``DataFrame``
~~~  A ``DataFrame`` in pandas is analogous to an Excel worksheet. While an Excel workbook can contain
multiple worksheets, pandas ``DataFrame``\s exist independently.  ``Series``
~~~  A ``Series`` is the data structure that represents one column of a ``DataFrame``. Working with a
``Series`` is analogous to referencing a column of a spreadsheet.  ``Index``
~~~  Every ``DataFrame`` and ``Series`` has an ``Index``, which are labels on the *rows* of the data. In
pandas, if no index is specified, a `~pandas.RangeIndex` is used by default (first row = 0,
second row = 1, and so on), analogous to row headings/numbers in spreadsheets.  In pandas, indexes can be set to one (or multiple) unique values, which is like having a column that
is used as the row identifier in a worksheet. Unlike most spreadsheets, these ``Index`` values can
actually be used to reference the rows. (Note that `this can be done in Excel with structured
references
<https://support.microsoft.com/en-us/office/using-structured-references-with-excel-tables-f5ed2452-2337-4f71-bed3-c8ae6d2b276e>`_.)
For example, in spreadsheets, you would reference the first row as ``A1:Z1``, while in pandas you
could use ``populations.loc['Chicago']``.  Index values are also persistent, so if you re-order the rows in a ``DataFrame``, the label for a
particular row don't change.  See the `indexing documentation<indexing>` for much more on how to use an ``Index``
effectively.  Copies vs. in place operations

~~~  In a spreadsheet, values can be typed directly into cells.  .. include:: includes/construct_dataframe.rst  Reading external data
~~~  Both Excel_
and `pandas <10min_tut_02_read_write>` can import data from various sources in various
formats.  CSV
'''  Let's load and display the tips
dataset from the pandas tests, which is a CSV file. In Excel, you would download and then
open the CSV.
In pandas, you pass the URL or local path of the CSV file to `~pandas.read_csv`:  .. ipython:: python  url = (  "https://raw.githubusercontent.com/pandas-dev"  "/pandas/main/pandas/tests/io/data/csv/tips.csv"  )  tips = pd.read_csv(url)  tips  Like Excel's Text Import Wizard,
``read_csv`` can take a number of parameters to specify how the data should be parsed. For
example, if the data was instead tab delimited, and did not have column names, the pandas command
would be:  .. code-block:: python  tips = pd.read_csv("tips.csv", sep="\t", header=None)  # alternatively, read_table is an alias to read_csv with tab delimiter  tips = pd.read_table("tips.csv", header=None)  Excel files
'''  Excel opens various Excel file formats
by double-clicking them, or using the Open menu.
In pandas, you use `special methods for reading and writing from/to Excel files <io.excel>`.  Let's first `create a new Excel file <io.excel_writer>` based on the ``tips`` dataframe in the above example:  .. code-block:: python  tips.to_excel("./tips.xlsx")  Should you wish to subsequently access the data in the ``tips.xlsx`` file, you can read it into your module using  .. code-block:: python  tips_df = pd.read_excel("./tips.xlsx", index_col=0)  You have just read in an Excel file using pandas!  Limiting output
~~~  Spreadsheet programs will only show one screenful of data at a time and then allow you to scroll, so
there isn't really a need to limit output. In pandas, you'll need to put a little more thought into
controlling how your ``DataFrame``\s are displayed.  .. include:: includes/limit.rst  Exporting data
~~~  By default, desktop spreadsheet software will save to its respective file format (`.xlsx``, ``.ods``, etc). You can, however, `save to other file formats.  `pandas can create Excel files <io.excel_writer>`, `CSV <io.store_in_csv>`, or `a number of other formats <io>`.

~~~  In spreadsheets, `formulas
<https://support.microsoft.com/en-us/office/overview-of-formulas-in-excel-ecfdc708-9162-49e8-b993-c311f47ca173>`_
are often created in individual cells and then `dragged
<https://support.microsoft.com/en-us/office/copy-a-formula-by-dragging-the-fill-handle-in-excel-for-mac-dd928259-622b-473f-9a33-83aa1a63e218>`_
into other cells to compute them for other columns. In pandas, you're able to do operations on whole
columns directly.  .. include:: includes/column_operations.rst  Note that we aren't having to tell it to do that subtraction cell-by-cell — pandas handles that for
us. See `how to create new columns derived from existing columns <10min_tut_05_columns>`.  Filtering
~~~  In Excel, filtering is done through a graphical menu.  Screenshot showing filtering of the total_bill column to values greater than 10  center  .. include:: includes/filtering.rst  If/then logic
~~~  Let's say we want to make a ``bucket`` column with values of ``low`` and ``high``, based on whether
the ``total_bill`` is less or more than $10.  In spreadsheets, logical comparison can be done with `conditional formulas
<https://support.microsoft.com/en-us/office/create-conditional-formulas-ca916c57-abd8-4b44-997c-c309b7307831>`_.
We'd use a formula of ``=IF(A2 < 10, "low", "high")``, dragged to all cells in a new ``bucket``
column.  Screenshot showing the formula from above in a bucket column of the tips spreadsheet  center  .. include:: includes/if_then.rst  Date functionality
~~~  *This section will refer to "dates", but timestamps are handled similarly.*  We can think of date functionality in two parts: parsing, and output. In spreadsheets, date values
are generally parsed automatically, though there is a `DATEVALUE
<https://support.microsoft.com/en-us/office/datevalue-function-df8b07d4-7761-4a93-bc33-b7471bbff252>`_
function if you need it. In pandas, you need to explicitly convert plain text to datetime objects,
either `while reading from a CSV <io.read_csv_table.datetime>` or `once in a DataFrame
<10min_tut_09_timeseries.properties>`.  Once parsed, spreadsheets display the dates in a default format, though `the format can be changed
<https://support.microsoft.com/en-us/office/format-a-date-the-way-you-want-8e10019e-d5d8-47a1-ba95-db95123d273e>`_.
In pandas, you'll generally want to keep dates as ``datetime`` objects while you're doing
calculations with them. Outputting *parts* of dates (such as the year) is done through `date
functions
<https://support.microsoft.com/en-us/office/date-and-time-functions-reference-fd1b5961-c1ae-4677-be58-074152f97b81>`_
in spreadsheets, and `datetime properties <10min_tut_09_timeseries.properties>` in pandas.  Given ``date1`` and ``date2`` in columns ``A`` and ``B`` of a spreadsheet, you might have these
formulas:  .. list-table::  :header-rows: 1  auto  * - column  - formula  * - ``date1_year``  - ``=YEAR(A2)``  * - ``date2_month``  - ``=MONTH(B2)``  * - ``date1_next``  - ``=DATE(YEAR(A2),MONTH(A2)+1,1)``  * - ``months_between``  - ``=DATEDIF(A2,B2,"M")``  The equivalent pandas operations are shown below.  .. include:: includes/time_date.rst  See `timeseries` for more details.  Selection of columns
~~~  In spreadsheets, you can select columns you want by:  - Hiding columns
- Deleting columns
- Referencing a range from one worksheet into another  Since spreadsheet columns are typically `named in a header row
<https://support.microsoft.com/en-us/office/turn-excel-table-headers-on-or-off-c91d1742-312c-4480-820f-cf4b534c8b3b>`_,
renaming a column is simply a matter of changing the text in that first cell.  .. include:: includes/column_selection.rst  Sorting by values
~~~  Sorting in spreadsheets is accomplished via the sort dialog.  Screenshot of dialog from Excel showing sorting by the sex then total_bill columns  center  .. include:: includes/sorting.rst

~~~  In spreadsheets, the number of characters in text can be found with the `LEN
<https://support.microsoft.com/en-us/office/len-lenb-functions-29236f94-cedc-429d-affd-b5e33d2c67cb>`_
function. This can be used with the `TRIM
<https://support.microsoft.com/en-us/office/trim-function-410388fa-c5df-49c6-b16c-9e5630b479f9>`_
function to remove extra whitespace.  ::  =LEN(TRIM(A2))  .. include:: includes/length.rst  Note this will still include multiple spaces within the string, so isn't 100% equivalent.  Finding position of substring
~~~  The `FIND
<https://support.microsoft.com/en-us/office/find-findb-functions-c7912941-af2a-4bdf-a553-d0d89b0a0628>`_
spreadsheet function returns the position of a substring, with the first character being ``1``.  Screenshot of FIND formula being used in Excel  center  .. include:: includes/find_substring.rst  Extracting substring by position
~~~  Spreadsheets have a `MID
<https://support.microsoft.com/en-us/office/mid-midb-functions-d5f9e25c-d7d6-472e-b568-4ecb12433028>`_
formula for extracting a substring from a given position. To get the first character::  =MID(A2,1,1)  .. include:: includes/extract_substring.rst  Extracting nth word
~~~  In Excel, you might use the `Text to Columns Wizard
<https://support.microsoft.com/en-us/office/split-text-into-different-columns-with-the-convert-text-to-columns-wizard-30b14928-5550-41f5-97ca-7a3e9c363ed7>`_
for splitting text and retrieving a specific column. (Note `it's possible to do so through a formula
as well <https://exceljet.net/formula/extract-nth-word-from-text-string>`_.)  .. include:: includes/nth_word.rst  Changing case
~~~  Spreadsheets provide `UPPER, LOWER, and PROPER functions
<https://support.microsoft.com/en-us/office/change-the-case-of-text-01481046-0fa7-4f3b-a693-496795a7a44d>`_
for converting text to upper, lower, and title case, respectively.  .. include:: includes/case.rst

Merging

.. include:: includes/merge_setup.rst  In Excel, there are `merging of tables can be done through a VLOOKUP
<https://support.microsoft.com/en-us/office/how-can-i-merge-two-or-more-tables-c80a9fce-c1ab-4425-bb96-497dd906d656>`_.  Screenshot showing a VLOOKUP formula between two tables in Excel, with some values being filled in and others with "#N/A"  center  .. include:: includes/merge.rst  ``merge`` has a number of advantages over ``VLOOKUP``:  * The lookup value doesn't need to be the first column of the lookup table
* If multiple rows are matched, there will be one row for each match, instead of just the first
* It will include all columns from the lookup table, instead of just a single specified column
* It supports `more complex join operations <merging.join>`

Fill Handle
~~~  Create a series of numbers following a set pattern in a certain set of cells. In
a spreadsheet, this would be done by shift+drag after entering the first number or by
entering the first two or three values and then dragging.  This can be achieved by creating a series and assigning it to the desired cells.  .. ipython:: python  df = pd.DataFrame({"AAA": [1] * 8, "BBB": list(range(0, 8))})  df  series = list(range(1, 5))  series  df.loc[2:5, "AAA"] = series  df  Drop Duplicates
~~~  Excel has built-in functionality for removing duplicate values.
This is supported in pandas via `~DataFrame.drop_duplicates`.  .. ipython:: python  df = pd.DataFrame(  {  "class": ["A", "A", "A", "B", "C", "D"],  "student_count": [42, 35, 42, 50, 47, 45],  "all_pass": ["Yes", "Yes", "Yes", "No", "No", "Yes"],  }  )  df.drop_duplicates()  df.drop_duplicates(["class", "student_count"])  Pivot Tables
~~~  PivotTables
from spreadsheets can be replicated in pandas through `reshaping`. Using the ``tips`` dataset again,
let's find the average gratuity by size of the party and sex of the server.  In Excel, we use the following configuration for the PivotTable:  Screenshot showing a PivotTable in Excel, using sex as the column, size as the rows, then average tip as the values  center  The equivalent in pandas:  .. ipython:: python  pd.pivot_table(  tips, values="tip", index=["size"], columns=["sex"], aggfunc=np.average  )  Adding a row
~~~  Assuming we are using a `~pandas.RangeIndex` (numbered ``0``, ``1``, etc.), we can use `concat` to add a row to the bottom of a ``DataFrame``.  .. ipython:: python  df  new_row = pd.DataFrame([["E", 51, True]],  columns=["class", "student_count", "all_pass"])  pd.concat([df, new_row])  Find and Replace
~~~  Excel's Find dialog
takes you to cells that match, one by one. In pandas, this operation is generally done for an
entire column or ``DataFrame`` at once through `conditional expressions <10min_tut_03_subset.rows_and_columns>`.  .. ipython:: python  tips  tips == "Sun"  tips["day"].str.contains("S")  pandas' `~DataFrame.replace` is comparable to Excel's ``Replace All``.  .. ipython:: python  tips.replace("Thu", "Thursday")

Copies vs. in place operations

.. include:: includes/copies.rst

SELECT
In SQL, selection is done using a comma-separated list of columns you'd like to select (or a ``*``
to select all columns):  .. code-block:: sql  SELECT total_bill, tip, smoker, time  FROM tips;  With pandas, column selection is done by passing a list of column names to your DataFrame:  .. ipython:: python  tips[["total_bill", "tip", "smoker", "time"]]  Calling the DataFrame without the list of column names would display all columns (akin to SQL's
``*``).  In SQL, you can add a calculated column:  .. code-block:: sql  SELECT *, tip/total_bill as tip_rate  FROM tips;  With pandas, you can use the `DataFrame.assign` method of a DataFrame to append a new column:  .. ipython:: python  tips.assign(tip_rate=tips["tip"] / tips["total_bill"])

WHERE
Filtering in SQL is done via a WHERE clause.  .. code-block:: sql  SELECT *  FROM tips  WHERE time = 'Dinner';  .. include:: includes/filtering.rst  Just like SQL's ``OR`` and ``AND``, multiple conditions can be passed to a DataFrame using ``|``
(``OR``) and ``&`` (``AND``).  Tips of more than $5 at Dinner meals:  .. code-block:: sql  SELECT *  FROM tips  WHERE time = 'Dinner' AND tip > 5.00;  .. ipython:: python  tips[(tips["time"] == "Dinner") & (tips["tip"] > 5.00)]  Tips by parties of at least 5 diners OR bill total was more than $45:  .. code-block:: sql  SELECT *  FROM tips  WHERE size >= 5 OR total_bill > 45;  .. ipython:: python  tips[(tips["size"] >= 5) | (tips["total_bill"] > 45)]  NULL checking is done using the `~pandas.Series.notna` and `~pandas.Series.isna`
methods.  .. ipython:: python  frame = pd.DataFrame(  {"col1": ["A", "B", np.NaN, "C", "D"], "col2": ["F", np.NaN, "G", "H", "I"]}  )  frame  Assume we have a table of the same structure as our DataFrame above. We can see only the records
where ``col2`` IS NULL with the following query:  .. code-block:: sql  SELECT *  FROM frame  WHERE col2 IS NULL;  .. ipython:: python  frame[frame["col2"].isna()]  Getting items where ``col1`` IS NOT NULL can be done with `~pandas.Series.notna`.  .. code-block:: sql  SELECT *  FROM frame  WHERE col1 IS NOT NULL;  .. ipython:: python  frame[frame["col1"].notna()]

GROUP BY
In pandas, SQL's ``GROUP BY`` operations are performed using the similarly named
`~pandas.DataFrame.groupby` method. `~pandas.DataFrame.groupby` typically refers to a
process where we'd like to split a dataset into groups, apply some function (typically aggregation)
, and then combine the groups together.  A common SQL operation would be getting the count of records in each group throughout a dataset.
For instance, a query getting us the number of tips left by sex:  .. code-block:: sql  SELECT sex, count(*)  FROM tips  GROUP BY sex;  /*  Female  87  Male  157  */  The pandas equivalent would be:  .. ipython:: python  tips.groupby("sex").size()  Notice that in the pandas code we used `~pandas.core.groupby.DataFrameGroupBy.size` and not
`~pandas.core.groupby.DataFrameGroupBy.count`. This is because
`~pandas.core.groupby.DataFrameGroupBy.count` applies the function to each column, returning
the number of ``NOT NULL`` records within each.  .. ipython:: python  tips.groupby("sex").count()  Alternatively, we could have applied the `~pandas.core.groupby.DataFrameGroupBy.count` method
to an individual column:  .. ipython:: python  tips.groupby("sex")["total_bill"].count()  Multiple functions can also be applied at once. For instance, say we'd like to see how tip amount
differs by day of the week - `~pandas.core.groupby.DataFrameGroupBy.agg` allows you to pass a dictionary
to your grouped DataFrame, indicating which functions to apply to specific columns.  .. code-block:: sql  SELECT day, AVG(tip), COUNT(*)  FROM tips  GROUP BY day;  /*  Fri  2.734737  19  Sat  2.993103  87  Sun  3.255132  76  Thu  2.771452  62  */  .. ipython:: python  tips.groupby("day").agg({"tip": np.mean, "day": np.size})  Grouping by more than one column is done by passing a list of columns to the
`~pandas.DataFrame.groupby` method.  .. code-block:: sql  SELECT smoker, day, COUNT(*), AVG(tip)  FROM tips  GROUP BY smoker, day;  /*  smoker day  No  Fri  4  2.812500  Sat  45  3.102889  Sun  57  3.167895  Thu  45  2.673778  Yes  Fri  15  2.714000  Sat  42  2.875476  Sun  19  3.516842  Thu  17  3.03000  */  .. ipython:: python  tips.groupby(["smoker", "day"]).agg({"tip": [np.size, np.mean]})  .. _compare_with_sql.join:

JOIN
``JOIN``\s can be performed with `~pandas.DataFrame.join` or `~pandas.merge`. By
default, `~pandas.DataFrame.join` will join the DataFrames on their indices. Each method has
parameters allowing you to specify the type of join to perform (``LEFT``, ``RIGHT``, ``INNER``,
``FULL``) or the columns to join on (column names or indices).  .. warning::  If both key columns contain rows where the key is a null value, those  rows will be matched against each other. This is different from usual SQL  join behaviour and can lead to unexpected results.  .. ipython:: python  df1 = pd.DataFrame({"key": ["A", "B", "C", "D"], "value": np.random.randn(4)})  df2 = pd.DataFrame({"key": ["B", "D", "D", "E"], "value": np.random.randn(4)})  Assume we have two database tables of the same name and structure as our DataFrames.  Now let's go over the various types of ``JOIN``\s.  INNER JOIN
~~~
.. code-block:: sql  SELECT *  FROM df1  INNER JOIN df2  ON df1.key = df2.key;  .. ipython:: python  # merge performs an INNER JOIN by default  pd.merge(df1, df2, on="key")  `~pandas.merge` also offers parameters for cases when you'd like to join one DataFrame's
column with another DataFrame's index.  .. ipython:: python  indexed_df2 = df2.set_index("key")  pd.merge(df1, indexed_df2, left_on="key", right_index=True)  LEFT OUTER JOIN
~~~  Show all records from ``df1``.  .. code-block:: sql  SELECT *  FROM df1  LEFT OUTER JOIN df2  ON df1.key = df2.key;  .. ipython:: python  pd.merge(df1, df2, on="key", how="left")  RIGHT JOIN
~~~  Show all records from ``df2``.  .. code-block:: sql  SELECT *  FROM df1  RIGHT OUTER JOIN df2  ON df1.key = df2.key;  .. ipython:: python  pd.merge(df1, df2, on="key", how="right")  FULL JOIN
~~~
pandas also allows for ``FULL JOIN``\s, which display both sides of the dataset, whether or not the
joined columns find a match. As of writing, ``FULL JOIN``\s are not supported in all RDBMS (MySQL).  Show all records from both tables.  .. code-block:: sql  SELECT *  FROM df1  FULL OUTER JOIN df2  ON df1.key = df2.key;  .. ipython:: python  pd.merge(df1, df2, on="key", how="outer")

UNION

``UNION ALL`` can be performed using `~pandas.concat`.  .. ipython:: python  df1 = pd.DataFrame(  {"city": ["Chicago", "San Francisco", "New York City"], "rank": range(1, 4)}  )  df2 = pd.DataFrame(  {"city": ["Chicago", "Boston", "Los Angeles"], "rank": [1, 4, 5]}  )  .. code-block:: sql  SELECT city, rank  FROM df1  UNION ALL  SELECT city, rank  FROM df2;  /*  city  rank  Chicago  1  San Francisco  2  New York City  3  Chicago  1  Boston  4  Los Angeles  5  */  .. ipython:: python  pd.concat([df1, df2])  SQL's ``UNION`` is similar to ``UNION ALL``, however ``UNION`` will remove duplicate rows.  .. code-block:: sql  SELECT city, rank  FROM df1  UNION  SELECT city, rank  FROM df2;  -- notice that there is only one Chicago record this time  /*  city  rank  Chicago  1  San Francisco  2  New York City  3  Boston  4  Los Angeles  5  */  In pandas, you can use `~pandas.concat` in conjunction with
`~pandas.DataFrame.drop_duplicates`.  .. ipython:: python  pd.concat([df1, df2]).drop_duplicates()

LIMIT

.. code-block:: sql  SELECT * FROM tips  LIMIT 10;  .. ipython:: python  tips.head(10)

pandas equivalents for some SQL analytic and aggregate functions

Top n rows with offset
~~~  .. code-block:: sql  -- MySQL  SELECT * FROM tips  ORDER BY tip DESC  LIMIT 10 OFFSET 5;  .. ipython:: python  tips.nlargest(10 + 5, columns="tip").tail(10)  Top n rows per group
~~~  .. code-block:: sql  -- Oracle's ROW_NUMBER() analytic function  SELECT * FROM (  SELECT  t.*,  ROW_NUMBER() OVER(PARTITION BY day ORDER BY total_bill DESC) AS rn  FROM tips t  )  WHERE rn < 3  ORDER BY day, rn;  .. ipython:: python  (  tips.assign(  rn=tips.sort_values(["total_bill"], ascending=False)  .groupby(["day"])  .cumcount()  + 1  )  .query("rn < 3")  .sort_values(["day", "rn"])  )  the same using ``rank(method='first')`` function  .. ipython:: python  (  tips.assign(  rnk=tips.groupby(["day"])["total_bill"].rank(  method="first", ascending=False  )  )  .query("rnk < 3")  .sort_values(["day", "rnk"])  )  .. code-block:: sql  -- Oracle's RANK() analytic function  SELECT * FROM (  SELECT  t.*,  RANK() OVER(PARTITION BY sex ORDER BY tip) AS rnk  FROM tips t  WHERE tip < 2  )  WHERE rnk < 3  ORDER BY sex, rnk;  Let's find tips with (rank < 3) per gender group for (tips < 2).
Notice that when using ``rank(method='min')`` function
``rnk_min`` remains the same for the same ``tip``
(as Oracle's ``RANK()`` function)  .. ipython:: python  (  tips[tips["tip"] < 2]  .assign(rnk_min=tips.groupby(["sex"])["tip"].rank(method="min"))  .query("rnk_min < 3")  .sort_values(["sex", "rnk_min"])  )

UPDATE

.. code-block:: sql  UPDATE tips  SET tip = tip*2  WHERE tip < 2;  .. ipython:: python  tips.loc[tips["tip"] < 2, "tip"] *= 2

DELETE

.. code-block:: sql  DELETE FROM tips  WHERE tip > 9;  In pandas we select the rows that should remain instead of deleting them:  .. ipython:: python  tips = tips.loc[tips["tip"] <= 9]

~~~  .. csv-table::  "pandas", "Stata"  20, 20  ``DataFrame``, data set  column, variable  row, observation  groupby, bysort  ``NaN``, ``.``  ``DataFrame``
~~~  A ``DataFrame`` in pandas is analogous to a Stata data set -- a two-dimensional
in Stata can also be accomplished in pandas.  ``Series``
``DataFrame``. Stata doesn't have a separate data structure for a single column,
of a data set in Stata.  ``Index``
~~~  Every ``DataFrame`` and ``Series`` has an ``Index`` -- labels on the
*rows* of the data. Stata does not have an exactly analogous concept. In Stata, a data set's
accessed with ``_n``.  In pandas, if no index is specified, an integer index is also used by default

~~~  A Stata data set can be built from specified values by
placing the data after an ``input`` statement and
specifying the column names.  .. code-block:: stata  input x y  1 2  3 4  5 6  end  .. include:: includes/construct_dataframe.rst  Reading external data
~~~  Like Stata, pandas provides utilities for reading in data from
many formats.  The ``tips`` data set, found within the pandas
will be used in many of the following examples.  Stata provides ``import delimited`` to read csv data into a data set in memory.
If the ``tips.csv`` file is in the current working directory, we can import it as follows.  .. code-block:: stata  import delimited tips.csv  The pandas method is `read_csv`, which works similarly. Additionally, it will automatically download
the data set if presented with a url.  .. ipython:: python  url = (  "https://raw.githubusercontent.com/pandas-dev"  "/pandas/main/pandas/tests/io/data/csv/tips.csv"  )  tips = pd.read_csv(url)  tips  Like ``import delimited``, `read_csv` can take a number of parameters to specify
how the data should be parsed.  For example, if the data were instead tab delimited,
did not have column names, and existed in the current working directory,
the pandas command would be:  .. code-block:: python  tips = pd.read_csv("tips.csv", sep="\t", header=None)  # alternatively, read_table is an alias to read_csv with tab delimiter  tips = pd.read_table("tips.csv", header=None)  pandas can also read Stata data sets in ``.dta`` format with the `read_stata` function.  .. code-block:: python  df = pd.read_stata("data.dta")  In addition to text/csv and Stata files, pandas supports a variety of other data formats
such as Excel, SAS, HDF5, Parquet, and SQL databases.  These are all read via a ``pd.read_*``
~~~  .. include:: includes/limit.rst  The equivalent in Stata would be:  .. code-block:: stata  list in 1/5  Exporting data
~~~  The inverse of ``import delimited`` in Stata is ``export delimited``  .. code-block:: stata  export delimited tips2.csv  Similarly in pandas, the opposite of ``read_csv`` is `DataFrame.to_csv`.  .. code-block:: python  tips.to_csv("tips2.csv")  pandas can also export to Stata file format with the `DataFrame.to_stata` method.  .. code-block:: python  tips.to_stata("tips2.dta")

~~~  In Stata, arbitrary math expressions can be used with the ``generate`` and
``replace`` commands on new or existing columns. The ``drop`` command drops
the column from the data set.  .. code-block:: stata  replace total_bill = total_bill - 2  generate new_bill = total_bill / 2  drop new_bill  .. include:: includes/column_operations.rst  Filtering
~~~  Filtering in Stata is done with an ``if`` clause on one or more columns.  .. code-block:: stata  list if total_bill > 10  .. include:: includes/filtering.rst  If/then logic
~~~  In Stata, an ``if`` clause can also be used to create new columns.  .. code-block:: stata  generate bucket = "low" if total_bill < 10  replace bucket = "high" if total_bill >= 10  .. include:: includes/if_then.rst  Date functionality
~~~  Stata provides a variety of functions to do operations on
date/datetime columns.  .. code-block:: stata  generate date1 = mdy(1, 15, 2013)  generate date2 = date("Feb152015", "MDY")  generate date1_year = year(date1)  generate date2_month = month(date2)  * shift date to beginning of next month  generate date1_next = mdy(month(date1) + 1, 1, year(date1)) if month(date1) != 12  replace date1_next = mdy(1, 1, year(date1) + 1) if month(date1) == 12  generate months_between = mofd(date2) - mofd(date1)  list date1 date2 date1_year date2_month date1_next months_between  The equivalent pandas operations are shown below.  In addition to these
functions, pandas supports other Time Series features
not available in Stata (such as time zone handling and custom offsets) --
~~~  Stata provides keywords to select, drop, and rename columns.  .. code-block:: stata  keep sex total_bill tip  drop sex  rename total_bill total_bill_2  .. include:: includes/column_selection.rst  Sorting by values
~~~  Sorting in Stata is accomplished via ``sort``  .. code-block:: stata  sort sex total_bill  .. include:: includes/sorting.rst

~~~  Stata determines the length of a character string with the `strlen` and
`ustrlen` functions for ASCII and Unicode strings, respectively.  .. code-block:: stata  generate strlen_time = strlen(time)  generate ustrlen_time = ustrlen(time)  .. include:: includes/length.rst  Finding position of substring
~~~  Stata determines the position of a character in a string with the `strpos` function.
This takes the string defined by the first argument and searches for the
first position of the substring you supply as the second argument.  .. code-block:: stata  generate str_position = strpos(sex, "ale")  .. include:: includes/find_substring.rst  Extracting substring by position
~~~  Stata extracts a substring from a string based on its position with the `substr` function.  .. code-block:: stata  generate short_sex = substr(sex, 1, 1)  .. include:: includes/extract_substring.rst  Extracting nth word
~~~  The Stata `word` function returns the nth word from a string.
The first argument is the string you want to parse and the
second argument specifies which word you want to extract.  .. code-block:: stata  clear  input str20 string  "John Smith"  "Jane Cook"  end  generate first_name = word(name, 1)  generate last_name = word(name, -1)  .. include:: includes/nth_word.rst  Changing case
~~~  The Stata `strupper`, `strlower`, `strproper`,
`ustrupper`, `ustrlower`, and `ustrtitle` functions
change the case of ASCII and Unicode strings, respectively.  .. code-block:: stata  clear  input str20 string  "John Smith"  "Jane Cook"  end  generate upper = strupper(string)  generate lower = strlower(string)  generate title = strproper(string)  list  .. include:: includes/case.rst

Merging

.. include:: includes/merge_setup.rst  In Stata, to perform a merge, one data set must be in memory
and the other must be referenced as a file name on disk. In
contrast, Python must have both ``DataFrames`` already in memory.  By default, Stata performs an outer join, where all observations
from both data sets are left in memory after the merge. One can
keep only observations from the initial data set, the merged data set,
or the intersection of the two by using the values created in the
``_merge`` variable.  .. code-block:: stata  * First create df2 and save to disk  clear  input str1 key  B  D  D  E  end  generate value = rnormal()  save df2.dta  * Now create df1 in memory  clear  input str1 key  A  B  C  D  end  generate value = rnormal()  preserve  * Left join  merge 1:n key using df2.dta  keep if _merge == 1  * Right join  restore, preserve  merge 1:n key using df2.dta  keep if _merge == 2  * Inner join  restore, preserve  merge 1:n key using df2.dta  keep if _merge == 3  * Outer join  restore  merge 1:n key using df2.dta  .. include:: includes/merge.rst

Both pandas and Stata have a representation for missing data.  .. include:: includes/missing_intro.rst  One difference is that missing data cannot be compared to its sentinel value.
For example, in Stata you could do this to filter missing values.  .. code-block:: stata  * Keep missing values  list if value_x == .  * Keep non-missing values  list if value_x != .  .. include:: includes/missing.rst

GroupBy

~~~  Stata's ``collapse`` can be used to group by one or
numeric columns.  .. code-block:: stata  collapse (sum) total_bill tip, by(sex smoker)  .. include:: includes/groupby.rst  Transformation
~~~  In Stata, if the group aggregations need to be used with the
original data set, one would usually use ``bysort`` with `egen`.
For example, to subtract the mean for each observation by smoker group.  .. code-block:: stata  bysort sex smoker: egen group_bill = mean(total_bill)  generate adj_total_bill = total_bill - group_bill  .. include:: includes/transform.rst  By group processing
replicate most other ``bysort`` processing from Stata. For example,
the following example lists the first observation in the current
sort order by sex/smoker group.  .. code-block:: stata  bysort sex smoker: list if _n == 1  In pandas this would be written as:  .. ipython:: python  tips.groupby(["sex", "smoker"]).first()

~~~  pandas and Stata both operate exclusively in memory. This means that the size of
data able to be loaded in pandas is limited by your machine's memory.
If out of core processing is needed, one possibility is the
library, which provides a subset of pandas functionality for an
on-disk ``DataFrame``.

Comparison with other tools

.. toctree::  2  comparison_with_r  comparison_with_sql  comparison_with_spreadsheets  comparison_with_sas  comparison_with_stata

The equivalent pandas methods are `Series.str.upper`, `Series.str.lower`, and`Series.str.title`... ipython:: python  firstlast = pd.DataFrame({"string": ["John Smith", "Jane Cook"]})  firstlast["upper"] = firstlast["string"].str.upper()  firstlast["lower"] = firstlast["string"].str.lower()  firstlast["title"] = firstlast["string"].str.title()  firstlast

pandas provides vectorized operations by specifying the individual ``Series`` in the``DataFrame``. New columns can be assigned in the same way. The `DataFrame.drop` method dropsa column from the ``DataFrame``... ipython:: python  tips["total_bill"] = tips["total_bill"] - 2  tips["new_bill"] = tips["total_bill"] / 2  tips  tips = tips.drop("new_bill", axis=1)

The same operations are expressed in pandas below.Keep certain columns'''.. ipython:: python  tips[["sex", "total_bill", "tip"]]Drop a column'''.. ipython:: python  tips.drop("sex", axis=1)Rename a column'''.. ipython:: python  tips.rename(columns={"total_bill": "total_bill_2"})

A pandas ``DataFrame`` can be constructed in many different ways,but for a small number of values, it is often convenient to specify it asa Python dictionary, where the keys are the column namesand the values are the data... ipython:: python  df = pd.DataFrame({"x": [1, 3, 5], "y": [2, 4, 6]})  df

Most pandas operations return copies of the ``Series``/``DataFrame``. To make the changes "stick",you'll need to either assign to a new variable:  .. code-block:: python  sorted_df = df.sort_values("col1")or overwrite the original one:  .. code-block:: python  df = df.sort_values("col1")

With pandas you can use ``[]`` notation to extract a substringfrom a string by position locations. Keep in mind that Pythonindexes are zero-based... ipython:: python  tips["sex"].str[0:1]

DataFrames can be filtered in multiple ways; the most intuitive of which is using`boolean indexing <indexing.boolean>`... ipython:: python  tips[tips["total_bill"] > 10]The above statement is simply passing a ``Series`` of ``True``/``False`` objects to the DataFrame,returning all rows with ``True``... ipython:: python  is_dinner = tips["time"] == "Dinner"  is_dinner  is_dinner.value_counts()  tips[is_dinner]

You can find the position of a character in a column of strings with the `Series.str.find`method. ``find`` searches for the first position of the substring. If the substring is found, themethod returns its position. If not found, it returns ``-1``. Keep in mind that Python indexes arezero-based... ipython:: python  tips["sex"].str.find("ale")

pandas provides a flexible ``groupby`` mechanism that allows similar aggregations. See the`groupby documentation<groupby>` for more details and examples... ipython:: python  tips_summed = tips.groupby(["sex", "smoker"])[["total_bill", "tip"]].sum()  tips_summed

The same operation in pandas can be accomplished usingthe ``where`` method from ``numpy``... ipython:: python  tips["bucket"] = np.where(tips["total_bill"] < 10, "low", "high")  tips.. ipython:: python  tips = tips.drop("bucket", axis=1)

If you're new to pandas, you might want to first read through `10 Minutes to pandas<10min>`to familiarize yourself with the library.As is customary, we import pandas and NumPy as follows:.. ipython:: python  import pandas as pd  import numpy as np

You can find the length of a character string with `Series.str.len`.In Python 3, all strings are Unicode strings. ``len`` includes trailing blanks.Use ``len`` and ``rstrip`` to exclude trailing blanks... ipython:: python  tips["time"].str.len()  tips["time"].str.rstrip().str.len()

By default, pandas will truncate output of large ``DataFrame``\s to show the first and last rows.This can be overridden by `changing the pandas options <options>`, or using`DataFrame.head` or `DataFrame.tail`... ipython:: python  tips.head(5)

pandas DataFrames have a `~DataFrame.merge` method, which provides similar functionality. Thedata does not have to be sorted ahead of time, and different join types are accomplished via the``how`` keyword... ipython:: python  inner_join = df1.merge(df2, on=["key"], how="inner")  inner_join  left_join = df1.merge(df2, on=["key"], how="left")  left_join  right_join = df1.merge(df2, on=["key"], how="right")  right_join  outer_join = df1.merge(df2, on=["key"], how="outer")  outer_join

The following tables will be used in the merge examples:.. ipython:: python  df1 = pd.DataFrame({"key": ["A", "B", "C", "D"], "value": np.random.randn(4)})  df1  df2 = pd.DataFrame({"key": ["B", "D", "D", "E"], "value": np.random.randn(4)})  df2

In pandas, `Series.isna` and `Series.notna` can be used to filter the rows... ipython:: python  outer_join[outer_join["value_x"].isna()]  outer_join[outer_join["value_x"].notna()]pandas provides `a variety of methods to work with missing data <missing_data>`. Here are some examples:Drop rows with missing values~~~.. ipython:: python  outer_join.dropna()Forward fill from previous rows~~~.. ipython:: python  outer_join.fillna(method="ffill")Replace missing values with a specified value~~~Using the mean:.. ipython:: python  outer_join["value_x"].fillna(outer_join["value_x"].mean())

pandas represents missing data with the special float value ``NaN`` (not a number).  Many of thesemantics are the same; for example missing data propagates through numeric operations, and isignored by default for aggregations... ipython:: python  outer_join  outer_join["value_x"] + outer_join["value_y"]  outer_join["value_x"].sum()

The simplest way to extract words in pandas is to split the strings by spaces, then reference theword by index. Note there are more powerful approaches should you need them... ipython:: python  firstlast = pd.DataFrame({"String": ["John Smith", "Jane Cook"]})  firstlast["First_Name"] = firstlast["String"].str.split(" ", expand=True)[0]  firstlast["Last_Name"] = firstlast["String"].str.rsplit(" ", expand=True)[1]  firstlast

pandas has a `DataFrame.sort_values` method, which takes a list of columns to sort by... ipython:: python  tips = tips.sort_values(["sex", "total_bill"])  tips

.. ipython:: python  tips["date1"] = pd.Timestamp("2013-01-15")  tips["date2"] = pd.Timestamp("2015-02-15")  tips["date1_year"] = tips["date1"].dt.year  tips["date2_month"] = tips["date2"].dt.month  tips["date1_next"] = tips["date1"] + pd.offsets.MonthBegin()  tips["months_between"] = tips["date2"].dt.to_period("M") - tips[  "date1"  ].dt.to_period("M")  tips[  ["date1", "date2", "date1_year", "date2_month", "date1_next", "months_between"]  ].. ipython:: python  tips = tips.drop(  ["date1", "date2", "date1_year", "date2_month", "date1_next", "months_between"],  axis=1,  )

pandas provides a `groupby.transform` mechanism that allows these type of operations to besuccinctly expressed in one operation... ipython:: python  gb = tips.groupby("smoker")["total_bill"]  tips["adj_total_bill"] = tips["total_bill"] - gb.transform("mean")  tips

.. raw:: html  <ul class="task-bullet">  <li>  I want to start using pandas  .. ipython:: python  import pandas as pd  To load the pandas package and start working with it, import the
package. The community agreed alias for pandas is ``pd``, so loading
pandas as ``pd`` is assumed standard practice for all of the pandas
documentation.  .. raw:: html  </li>  </ul>  pandas data table representation
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I want to store passenger data of the Titanic. For a number of passengers, I know the name (characters), age (integers) and sex (male/female) data.  .. ipython:: python  df = pd.DataFrame(  {  "Name": [  "Braund, Mr. Owen Harris",  "Allen, Mr. William Henry",  "Bonnell, Miss. Elizabeth",  ],  "Age": [22, 35, 58],  "Sex": ["male", "male", "female"],  }  )  df  To manually store data in a table, create a ``DataFrame``. When using a Python dictionary of lists, the dictionary keys will be used as column headers and
the values in each list as columns of the ``DataFrame``.  .. raw:: html  </li>  </ul>  A `DataFrame` is a 2-dimensional data structure that can store data of
different types (including characters, integers, floating point values,
categorical data and more) in columns. It is similar to a spreadsheet, a
SQL table or the ``data.frame`` in R.  -  The table has 3 columns, each of them with a column label. The column  labels are respectively ``Name``, ``Age`` and ``Sex``.
-  The column ``Name`` consists of textual data with each value a  string, the column ``Age`` are numbers and the column ``Sex`` is  textual data.  In spreadsheet software, the table representation of our data would look
very similar:  center  Each column in a ``DataFrame`` is a ``Series``
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I’m just interested in working with the data in the column ``Age``  .. ipython:: python  df["Age"]  When selecting a single column of a pandas `DataFrame`, the result is
a pandas `Series`. To select the column, use the column label in
between square brackets ``[]``.  .. raw:: html  </li>  </ul>  .. note::  If you are familiar to Python  `dictionaries <python:tut-dictionaries>`, the selection of a  single column is very similar to selection of dictionary values based on  the key.  You can create a ``Series`` from scratch as well:  .. ipython:: python  ages = pd.Series([22, 35, 58], name="Age")  ages  A pandas ``Series`` has no column labels, as it is just a single column
of a ``DataFrame``. A Series does have row labels.  Do something with a DataFrame or Series
~~~  .. raw:: html  <ul class="task-bullet">  <li>  I want to know the maximum Age of the passengers  We can do this on the ``DataFrame`` by selecting the ``Age`` column and
applying ``max()``:  .. ipython:: python  df["Age"].max()  Or to the ``Series``:  .. ipython:: python  ages.max()  .. raw:: html  </li>  </ul>  As illustrated by the ``max()`` method, you can *do* things with a
``DataFrame`` or ``Series``. pandas provides a lot of functionalities,
each of them a *method* you can apply to a ``DataFrame`` or ``Series``.
As methods are functions, do not forget to use parentheses ``()``.  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in some basic statistics of the numerical data of my data table  .. ipython:: python  df.describe()  The `~DataFrame.describe` method provides a quick overview of the numerical data in
a ``DataFrame``. As the ``Name`` and ``Sex`` columns are textual data,
these are by default not taken into account by the `~DataFrame.describe` method.  .. raw:: html  </li>  </ul>  Many pandas operations return a ``DataFrame`` or a ``Series``. The
`~DataFrame.describe` method is an example of a pandas operation returning a

pandas ``Series`` or a pandas ``DataFrame``.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  Check more options on ``describe`` in the user guide section about `aggregations with describe <basics.describe>`  .. raw:: html  </div>  .. note::  This is just a starting point. Similar to spreadsheet  software, pandas represents data as a table with columns and rows. Apart  from the representation, also the data manipulations and calculations  you would do in spreadsheet software are supported by pandas. Continue  reading the next tutorials to get started!  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Import the package, aka ``import pandas as pd``
-  A table of data is stored as a pandas ``DataFrame``
-  Each column in a ``DataFrame`` is a ``Series``
-  You can do things by applying a method to a ``DataFrame`` or ``Series``  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A more extended explanation to ``DataFrame`` and ``Series`` is provided in the `introduction to data structures <dsintro>`.  .. raw:: html  </div>

How do I read and write tabular data?
  center  .. raw:: html  <ul class="task-bullet">  <li>  I want to analyze the Titanic passenger data, available as a CSV file.  .. ipython:: python  titanic = pd.read_csv("data/titanic.csv")  pandas provides the `read_csv` function to read data stored as a csv
file into a pandas ``DataFrame``. pandas supports many different file
formats or data sources out of the box (csv, excel, sql, json, parquet,
…), each of them with the prefix ``read_*``.  .. raw:: html  </li>  </ul>  Make sure to always have a check on the data after reading in the
data. When displaying a ``DataFrame``, the first and last 5 rows will be
shown by default:  .. ipython:: python  titanic  .. raw:: html  <ul class="task-bullet">  <li>  I want to see the first 8 rows of a pandas DataFrame.  .. ipython:: python  titanic.head(8)  To see the first N rows of a ``DataFrame``, use the `~DataFrame.head` method with
the required number of rows (in this case 8) as argument.  .. raw:: html  </li>  </ul>  .. note::  Interested in the last N rows instead? pandas also provides a  `~DataFrame.tail` method. For example, ``titanic.tail(10)`` will return the last  10 rows of the DataFrame.  A check on how pandas interpreted each of the column data types can be
done by requesting the pandas ``dtypes`` attribute:  .. ipython:: python  titanic.dtypes  For each of the columns, the used data type is enlisted. The data types
in this ``DataFrame`` are integers (``int64``), floats (``float64``) and
strings (``object``).  .. note::  When asking for the ``dtypes``, no brackets are used!  ``dtypes`` is an attribute of a ``DataFrame`` and ``Series``. Attributes  of ``DataFrame`` or ``Series`` do not need brackets. Attributes  represent a characteristic of a ``DataFrame``/``Series``, whereas a  method (which requires brackets) *do* something with the  ``DataFrame``/``Series`` as introduced in the `first tutorial <10min_tut_01_tableoriented>`.  .. raw:: html  <ul class="task-bullet">  <li>  My colleague requested the Titanic data as a spreadsheet.  .. ipython:: python  titanic.to_excel("titanic.xlsx", sheet_name="passengers", index=False)  Whereas ``read_*`` functions are used to read data to pandas, the
``to_*`` methods are used to store data. The `~DataFrame.to_excel` method stores
the data as an excel file. In the example here, the ``sheet_name`` is
named *passengers* instead of the default *Sheet1*. By setting
``index=False`` the row index labels are not saved in the spreadsheet.  .. raw:: html  </li>  </ul>  The equivalent read function `~DataFrame.read_excel` will reload the data to a
``DataFrame``:  .. ipython:: python  titanic = pd.read_excel("titanic.xlsx", sheet_name="passengers")  .. ipython:: python  titanic.head()  .. ipython:: python  import os  os.remove("titanic.xlsx")  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in a technical summary of a ``DataFrame``  .. ipython:: python  titanic.info()  The method `~DataFrame.info` provides technical information about a
``DataFrame``, so let’s explain the output in more detail:  -  It is indeed a `DataFrame`.
-  There are 891 entries, i.e. 891 rows.
-  Each row has a row label (aka the ``index``) with values ranging from  0 to 890.
-  The table has 12 columns. Most columns have a value for each of the  rows (all 891 values are ``non-null``). Some columns do have missing  values and less than 891 ``non-null`` values.
-  The columns ``Name``, ``Sex``, ``Cabin`` and ``Embarked`` consists of  textual data (strings, aka ``object``). The other columns are  numerical data with some of them whole numbers (aka ``integer``) and  others are real numbers (aka ``float``).
-  The kind of data (characters, integers,…) in the different columns  are summarized by listing the ``dtypes``.

-  The approximate amount of RAM used to hold the DataFrame is provided  as well.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Getting data in to pandas from many different file formats or data  sources is supported by ``read_*`` functions.
-  Exporting data out of pandas is provided by different  ``to_*``\ methods.
-  The ``head``/``tail``/``info`` methods and the ``dtypes`` attribute  are convenient for a first check.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row bg-light gs-torefguide">  <span class="badge badge-info">To user guide</span>  For a complete overview of the input and output possibilities from and to pandas, see the user guide section about `reader and writer functions <io>`.  .. raw:: html  </div>

How do I select specific columns from a ``DataFrame``?
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in the age of the Titanic passengers.  .. ipython:: python  ages = titanic["Age"]  ages.head()  To select a single column, use square brackets ``[]`` with the column
name of the column of interest.  .. raw:: html  </li>  </ul>  Each column in a `DataFrame` is a `Series`. As a single column is
selected, the returned object is a pandas `Series`. We can verify this
by checking the type of the output:  .. ipython:: python  type(titanic["Age"])  And have a look at the ``shape`` of the output:  .. ipython:: python  titanic["Age"].shape  `DataFrame.shape` is an attribute (remember `tutorial on reading and writing <10min_tut_02_read_write>`, do not use parentheses for attributes) of a
pandas ``Series`` and ``DataFrame`` containing the number of rows and
columns: *(nrows, ncolumns)*. A pandas Series is 1-dimensional and only
the number of rows is returned.  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in the age and sex of the Titanic passengers.  .. ipython:: python  age_sex = titanic[["Age", "Sex"]]  age_sex.head()  To select multiple columns, use a list of column names within the
selection brackets ``[]``.  .. raw:: html  </li>  </ul>  .. note::  The inner square brackets define a  `Python list <python:tut-morelists>` with column names, whereas  the outer brackets are used to select the data from a pandas  ``DataFrame`` as seen in the previous example.  The returned data type is a pandas DataFrame:  .. ipython:: python  type(titanic[["Age", "Sex"]])  .. ipython:: python  titanic[["Age", "Sex"]].shape  The selection returned a ``DataFrame`` with 891 rows and 2 columns. Remember, a
``DataFrame`` is 2-dimensional with both a row and column dimension.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  For basic information on indexing, see the user guide section on `indexing and selecting data <indexing.basics>`.  .. raw:: html  </div>  How do I filter specific rows from a ``DataFrame``?
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in the passengers older than 35 years.  .. ipython:: python  above_35 = titanic[titanic["Age"] > 35]  above_35.head()  To select rows based on a conditional expression, use a condition inside
the selection brackets ``[]``.  .. raw:: html  </li>  </ul>  The condition inside the selection
brackets ``titanic["Age"] > 35`` checks for which rows the ``Age``
column has a value larger than 35:  .. ipython:: python  titanic["Age"] > 35  The output of the conditional expression (``>``, but also ``==``,
``!=``, ``<``, ``<=``,… would work) is actually a pandas ``Series`` of
boolean values (either ``True`` or ``False``) with the same number of
rows as the original ``DataFrame``. Such a ``Series`` of boolean values
can be used to filter the ``DataFrame`` by putting it in between the
selection brackets ``[]``. Only rows for which the value is ``True``
will be selected.  We know from before that the original Titanic ``DataFrame`` consists of
891 rows. Let’s have a look at the number of rows which satisfy the
condition by checking the ``shape`` attribute of the resulting
``DataFrame`` ``above_35``:  .. ipython:: python  above_35.shape  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in the Titanic passengers from cabin class 2 and 3.  .. ipython:: python  class_23 = titanic[titanic["Pclass"].isin([2, 3])]  class_23.head()  Similar to the conditional expression, the `~Series.isin` conditional function
returns a ``True`` for each row the values are in the provided list. To
filter the rows based on such a function, use the conditional function
inside the selection brackets ``[]``. In this case, the condition inside
the selection brackets ``titanic["Pclass"].isin([2, 3])`` checks for

which rows the ``Pclass`` column is either 2 or 3.  .. raw:: html  </li>  </ul>  The above is equivalent to filtering by rows for which the class is
either 2 or 3 and combining the two statements with an ``|`` (or)
operator:  .. ipython:: python  class_23 = titanic[(titanic["Pclass"] == 2) | (titanic["Pclass"] == 3)]  class_23.head()  .. note::  When combining multiple conditional statements, each condition  must be surrounded by parentheses ``()``. Moreover, you can not use  ``or``/``and`` but need to use the ``or`` operator ``|`` and the ``and``  operator ``&``.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  See the dedicated section in the user guide about `boolean indexing <indexing.boolean>` or about the `isin function <indexing.basics.indexing_isin>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  I want to work with passenger data for which the age is known.  .. ipython:: python  age_no_na = titanic[titanic["Age"].notna()]  age_no_na.head()  The `~Series.notna` conditional function returns a ``True`` for each row the
values are not a ``Null`` value. As such, this can be combined with the
selection brackets ``[]`` to filter the data table.  .. raw:: html  </li>  </ul>  You might wonder what actually changed, as the first 5 lines are still
the same values. One way to verify is to check if the shape has changed:  .. ipython:: python  age_no_na.shape  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  For more dedicated functions on missing values, see the user guide section about `handling missing data <missing_data>`.  .. raw:: html  </div>  .. _10min_tut_03_subset.rows_and_columns:  How do I select specific rows and columns from a ``DataFrame``?
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in the names of the passengers older than 35 years.  .. ipython:: python  adult_names = titanic.loc[titanic["Age"] > 35, "Name"]  adult_names.head()  In this case, a subset of both rows and columns is made in one go and
just using selection brackets ``[]`` is not sufficient anymore. The
``loc``/``iloc`` operators are required in front of the selection
brackets ``[]``. When using ``loc``/``iloc``, the part before the comma
is the rows you want, and the part after the comma is the columns you
want to select.  .. raw:: html  </li>  </ul>  When using the column names, row labels or a condition expression, use
the ``loc`` operator in front of the selection brackets ``[]``. For both
the part before and after the comma, you can use a single label, a list
of labels, a slice of labels, a conditional expression or a colon. Using
a colon specifies you want to select all rows or columns.  .. raw:: html  <ul class="task-bullet">  <li>  I’m interested in rows 10 till 25 and columns 3 to 5.  .. ipython:: python  titanic.iloc[9:25, 2:5]  Again, a subset of both rows and columns is made in one go and just
using selection brackets ``[]`` is not sufficient anymore. When
specifically interested in certain rows and/or columns based on their
position in the table, use the ``iloc`` operator in front of the
selection brackets ``[]``.  .. raw:: html  </li>  </ul>  When selecting specific rows and/or columns with ``loc`` or ``iloc``,
new values can be assigned to the selected data. For example, to assign

the name ``anonymous`` to the first 3 elements of the third column:  .. ipython:: python  titanic.iloc[0:3, 3] = "anonymous"  titanic.head()  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  See the user guide section on `different choices for indexing <indexing.choice>` to get more insight in the usage of ``loc`` and ``iloc``.  .. raw:: html  </div>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  When selecting subsets of data, square brackets ``[]`` are used.
-  Inside these brackets, you can use a single column/row label, a list  of column/row labels, a slice of labels, a conditional expression or  a colon.
-  Select specific rows and/or columns using ``loc`` when using the row  and column names.
-  Select specific rows and/or columns using ``iloc`` when using the  positions in the table.
-  You can assign new values to a selection based on ``loc``/``iloc``.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full overview of indexing is provided in the user guide pages on `indexing and selecting data <indexing>`.  .. raw:: html  </div>

How do I create plots in pandas?
  center  .. ipython:: python  import pandas as pd  import matplotlib.pyplot as plt  .. raw:: html  <div class="card gs-data">  <div class="card-header">  <div class="gs-data-title">  Data used for this tutorial:  </div>  </div>  <ul class="list-group list-group-flush">  <li class="list-group-item">  .. include:: includes/air_quality_no2.rst  .. ipython:: python  air_quality = pd.read_csv("data/air_quality_no2.csv", index_col=0, parse_dates=True)  air_quality.head()  .. note::  The usage of the ``index_col`` and ``parse_dates`` parameters of the ``read_csv`` function to define the first (0th) column as  index of the resulting ``DataFrame`` and convert the dates in the column to `Timestamp` objects, respectively.  .. raw:: html  </li>  </ul>  </div>  .. raw:: html  <ul class="task-bullet">  <li>  I want a quick visual check of the data.  .. ipython:: python  @savefig 04_airqual_quick.png  air_quality.plot()  plt.show()  With a ``DataFrame``, pandas creates by default one line plot for each of
the columns with numeric data.  .. raw:: html  </li>  </ul>  .. raw:: html  <ul class="task-bullet">  <li>  I want to plot only the columns of the data table with the data from Paris.  .. ipython:: python  # We need to clear the figure here as, within doc generation, the plot  # accumulates data on each plot(). This is not needed when running  # in a notebook, so is suppressed from output.  plt.clf()  .. ipython:: python  @savefig 04_airqual_paris.png  air_quality["station_paris"].plot()  plt.show()  To plot a specific column, use the selection method of the
`subset data tutorial <10min_tut_03_subset>` in combination with the `~DataFrame.plot`
method. Hence, the `~DataFrame.plot` method works on both ``Series`` and
``DataFrame``.  .. raw:: html  </li>  </ul>  .. raw:: html  <ul class="task-bullet">  <li>  I want to visually compare the `NO_2` values measured in London versus Paris.  .. ipython:: python  @savefig 04_airqual_scatter.png  air_quality.plot.scatter(x="station_london", y="station_paris", alpha=0.5)  plt.show()  .. raw:: html  </li>  </ul>  Apart from the default ``line`` plot when using the ``plot`` function, a
number of alternatives are available to plot data. Let’s use some
standard Python to get an overview of the available plot methods:  .. ipython:: python  [  method_name  for method_name in dir(air_quality.plot)  if not method_name.startswith("_")  ]  .. note::  In many development environments as well as IPython and  Jupyter Notebook, use the TAB button to get an overview of the available  methods, for example ``air_quality.plot.`` + TAB.  One of the options is `DataFrame.plot.box`, which refers to a
boxplot_. The ``box``
method is applicable on the air quality example data:  .. ipython:: python  @savefig 04_airqual_boxplot.png  air_quality.plot.box()  plt.show()  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  For an introduction to plots other than the default line plot, see the user guide section about `supported plot styles <visualization.other>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  I want each of the columns in a separate subplot.  .. ipython:: python  @savefig 04_airqual_area_subplot.png  axs = air_quality.plot.area(figsize=(12, 4), subplots=True)  plt.show()  Separate subplots for each of the data columns are supported by the ``subplots`` argument
of the ``plot`` functions. The builtin options available in each of the pandas plot

functions are worth reviewing.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  Some more formatting options are explained in the user guide section on `plot formatting <visualization.formatting>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  I want to further customize, extend or save the resulting plot.  .. ipython:: python  fig, axs = plt.subplots(figsize=(12, 4))  air_quality.plot.area(ax=axs)  axs.set_ylabel("NO$_2$ concentration")  @savefig 04_airqual_customized.png  fig.savefig("no2_concentrations.png")  plt.show()  .. ipython:: python  import os  os.remove("no2_concentrations.png")  .. raw:: html  </li>  </ul>  Each of the plot objects created by pandas is a
Matplotlib_ object. As Matplotlib provides
plenty of options to customize plots, making the link between pandas and
Matplotlib explicit enables all the power of Matplotlib to the plot.
This strategy is applied in the previous example:  ::  fig, axs = plt.subplots(figsize=(12, 4))  # Create an empty Matplotlib Figure and Axes  air_quality.plot.area(ax=axs)  # Use pandas to put the area plot on the prepared Figure/Axes  axs.set_ylabel("NO$_2$ concentration")  # Do any Matplotlib customization you like  fig.savefig("no2_concentrations.png")  # Save the Figure/Axes using the existing Matplotlib method.  plt.show()  # Display the plot  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  The ``.plot.*`` methods are applicable on both Series and DataFrames.
-  By default, each of the columns is plotted as a different element  (line, boxplot,…).
-  Any plot created by pandas is a Matplotlib object.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full overview of plotting in pandas is provided in the `visualization pages <visualization>`.  .. raw:: html  </div>

How to create new columns derived from existing columns
  center  .. raw:: html  <ul class="task-bullet">  <li>  I want to express the `NO_2` concentration of the station in London in mg/m\ `^3`.  (*If we assume temperature of 25 degrees Celsius and pressure of 1013
hPa, the conversion factor is 1.882*)  .. ipython:: python  air_quality["london_mg_per_cubic"] = air_quality["station_london"] * 1.882  air_quality.head()  To create a new column, use the ``[]`` brackets with the new column name
at the left side of the assignment.  .. raw:: html  </li>  </ul>  .. note::  The calculation of the values is done **element-wise**. This  means all values in the given column are multiplied by the value 1.882  at once. You do not need to use a loop to iterate each of the rows!  center  .. raw:: html  <ul class="task-bullet">  <li>  I want to check the ratio of the values in Paris versus Antwerp and save the result in a new column.  .. ipython:: python  air_quality["ratio_paris_antwerp"] = (  air_quality["station_paris"] / air_quality["station_antwerp"]  )  air_quality.head()  The calculation is again element-wise, so the ``/`` is applied *for the
values in each row*.  .. raw:: html  </li>  </ul>  Also other mathematical operators (``+``, ``-``, ``*``, ``/``,…) or
logical operators (``<``, ``>``, ``==``,…) work element-wise. The latter was already
used in the `subset data tutorial <10min_tut_03_subset>` to filter
rows of a table using a conditional expression.  If you need more advanced logic, you can use arbitrary Python code via `~DataFrame.apply`.  .. raw:: html  <ul class="task-bullet">  <li>  I want to rename the data columns to the corresponding station identifiers used by OpenAQ_.  .. ipython:: python  air_quality_renamed = air_quality.rename(  columns={  "station_antwerp": "BETR801",  "station_paris": "FR04014",  "station_london": "London Westminster",  }  )  .. ipython:: python  air_quality_renamed.head()  The `~DataFrame.rename` function can be used for both row labels and column
labels. Provide a dictionary with the keys the current names and the
values the new names to update the corresponding names.  .. raw:: html  </li>  </ul>  The mapping should not be restricted to fixed names only, but can be a
mapping function as well. For example, converting the column names to
lowercase letters can be done using a function as well:  .. ipython:: python  air_quality_renamed = air_quality_renamed.rename(columns=str.lower)  air_quality_renamed.head()  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  Details about column or row label renaming is provided in the user guide section on `renaming labels <basics.rename>`.  .. raw:: html  </div>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Create a new column by assigning the output to the DataFrame with a  new column name in between the ``[]``.
-  Operations are element-wise, no need to loop over rows.
-  Use ``rename`` with a dictionary or function to rename row labels or  column names.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  The user guide contains a separate section on `column addition and deletion <basics.dataframe.sel_add_del>`.  .. raw:: html  </div>

Aggregating statistics
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  What is the average age of the Titanic passengers?  .. ipython:: python  titanic["Age"].mean()  .. raw:: html  </li>  </ul>  Different statistics are available and can be applied to columns with
numerical data. Operations in general exclude missing data and operate
across rows by default.  center  .. raw:: html  <ul class="task-bullet">  <li>  What is the median age and ticket fare price of the Titanic passengers?  .. ipython:: python  titanic[["Age", "Fare"]].median()  The statistic applied to multiple columns of a ``DataFrame`` (the selection of two columns
returns a ``DataFrame``, see the `subset data tutorial <10min_tut_03_subset>`) is calculated for each numeric column.  .. raw:: html  </li>  </ul>  The aggregating statistic can be calculated for multiple columns at the
same time. Remember the ``describe`` function from the `first tutorial <10min_tut_01_tableoriented>`?  .. ipython:: python  titanic[["Age", "Fare"]].describe()  Instead of the predefined statistics, specific combinations of
aggregating statistics for given columns can be defined using the
`DataFrame.agg` method:  .. ipython:: python  titanic.agg(  {  "Age": ["min", "max", "median", "skew"],  "Fare": ["min", "max", "median", "mean"],  }  )  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  Details about descriptive statistics are provided in the user guide section on `descriptive statistics <basics.stats>`.  .. raw:: html  </div>  Aggregating statistics grouped by category
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  What is the average age for male versus female Titanic passengers?  .. ipython:: python  titanic[["Sex", "Age"]].groupby("Sex").mean()  As our interest is the average age for each gender, a subselection on
these two columns is made first: ``titanic[["Sex", "Age"]]``. Next, the
`~DataFrame.groupby` method is applied on the ``Sex`` column to make a group per
category. The average age *for each gender* is calculated and
returned.  .. raw:: html  </li>  </ul>  Calculating a given statistic (e.g. ``mean`` age) *for each category in
a column* (e.g. male/female in the ``Sex`` column) is a common pattern.
The ``groupby`` method is used to support this type of operations. This
fits in the more general ``split-apply-combine`` pattern:  -  **Split** the data into groups
-  **Apply** a function to each group independently
-  **Combine** the results into a data structure  The apply and combine steps are typically done together in pandas.  In the previous example, we explicitly selected the 2 columns first. If
not, the ``mean`` method is applied to each column containing numerical
columns by passing ``numeric_only=True``:  .. ipython:: python  titanic.groupby("Sex").mean(numeric_only=True)  It does not make much sense to get the average value of the ``Pclass``.
If we are only interested in the average age for each gender, the
selection of columns (rectangular brackets ``[]`` as usual) is supported
on the grouped data as well:  .. ipython:: python  titanic.groupby("Sex")["Age"].mean()  center  .. note::  The ``Pclass`` column contains numerical data but actually  represents 3 categories (or factors) with respectively the labels ‘1’,  ‘2’ and ‘3’. Calculating statistics on these does not make much sense.  Therefore, pandas provides a ``Categorical`` data type to handle this  type of data. More information is provided in the user guide  `categorical` section.  .. raw:: html  <ul class="task-bullet">  <li>  What is the mean ticket fare price for each of the sex and cabin class combinations?  .. ipython:: python  titanic.groupby(["Sex", "Pclass"])["Fare"].mean()  Grouping can be done by multiple columns at the same time. Provide the

column names as a list to the `~DataFrame.groupby` method.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full description on the split-apply-combine approach is provided in the user guide section on `groupby operations <groupby>`.  .. raw:: html  </div>  Count number of records by category
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  What is the number of passengers in each of the cabin classes?  .. ipython:: python  titanic["Pclass"].value_counts()  The `~Series.value_counts` method counts the number of records for each
category in a column.  .. raw:: html  </li>  </ul>  The function is a shortcut, as it is actually a groupby operation in combination with counting of the number of records
within each group:  .. ipython:: python  titanic.groupby("Pclass")["Pclass"].count()  .. note::  Both ``size`` and ``count`` can be used in combination with  ``groupby``. Whereas ``size`` includes ``NaN`` values and just provides  the number of rows (size of the table), ``count`` excludes the missing  values. In the ``value_counts`` method, use the ``dropna`` argument to  include or exclude the ``NaN`` values.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  The user guide has a dedicated section on ``value_counts`` , see the page on `discretization <basics.discretization>`.  .. raw:: html  </div>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Aggregation statistics can be calculated on entire columns or rows.
-  ``groupby`` provides the power of the *split-apply-combine* pattern.
-  ``value_counts`` is a convenient shortcut to count the number of  entries in each category of a variable.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full description on the split-apply-combine approach is provided in the user guide pages about `groupby operations <groupby>`.  .. raw:: html  </div>

Sort table rows
~~~  .. raw:: html  <ul class="task-bullet">  <li>  I want to sort the Titanic data according to the age of the passengers.  .. ipython:: python  titanic.sort_values(by="Age").head()  .. raw:: html  </li>  </ul>  .. raw:: html  <ul class="task-bullet">  <li>  I want to sort the Titanic data according to the cabin class and age in descending order.  .. ipython:: python  titanic.sort_values(by=['Pclass', 'Age'], ascending=False).head()  With `DataFrame.sort_values`, the rows in the table are sorted according to the
defined column(s). The index will follow the row order.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  More details about sorting of tables is provided in the user guide section on `sorting data <basics.sorting>`.  .. raw:: html  </div>  Long to wide table format
~~~  Let’s use a small subset of the air quality data set. We focus on
`NO_2` data and only use the first two measurements of each
location (i.e. the head of each group). The subset of data will be
called ``no2_subset``.  .. ipython:: python  # filter for no2 data only  no2 = air_quality[air_quality["parameter"] == "no2"]  .. ipython:: python  # use 2 measurements (head) for each location (groupby)  no2_subset = no2.sort_index().groupby(["location"]).head(2)  no2_subset  center  .. raw:: html  <ul class="task-bullet">  <li>  I want the values for the three stations as separate columns next to each other.  .. ipython:: python  no2_subset.pivot(columns="location", values="value")  The `~pandas.pivot` function is purely reshaping of the data: a single value
for each index/column combination is required.  .. raw:: html  </li>  </ul>  As pandas supports plotting of multiple columns (see `plotting tutorial <10min_tut_04_plotting>`) out of the box, the conversion from
*long* to *wide* table format enables the plotting of the different time
series at the same time:  .. ipython:: python  no2.head()  .. ipython:: python  @savefig 7_reshape_columns.png  no2.pivot(columns="location", values="value").plot()  .. note::  When the ``index`` parameter is not defined, the existing  index (row labels) is used.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  For more information about `~DataFrame.pivot`, see the user guide section on `pivoting DataFrame objects <reshaping.reshaping>`.  .. raw:: html  </div>  Pivot table
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I want the mean concentrations for `NO_2` and `PM_{2.5}` in each of the stations in table form.  .. ipython:: python  air_quality.pivot_table(  values="value", index="location", columns="parameter", aggfunc="mean"  )  In the case of `~DataFrame.pivot`, the data is only rearranged. When multiple
values need to be aggregated (in this specific case, the values on
different time steps), `~DataFrame.pivot_table` can be used, providing an
aggregation function (e.g. mean) on how to combine these values.  .. raw:: html  </li>  </ul>  Pivot table is a well known concept in spreadsheet software. When
interested in the row/column margins (subtotals) for each variable, set

the ``margins`` parameter to ``True``:  .. ipython:: python  air_quality.pivot_table(  values="value",  index="location",  columns="parameter",  aggfunc="mean",  margins=True,  )  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  For more information about `~DataFrame.pivot_table`, see the user guide section on `pivot tables <reshaping.pivot>`.  .. raw:: html  </div>  .. note::  In case you are wondering, `~DataFrame.pivot_table` is indeed directly linked  to `~DataFrame.groupby`. The same result can be derived by grouping on both  ``parameter`` and ``location``:  ::  air_quality.groupby(["parameter", "location"]).mean()  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  Have a look at `~DataFrame.groupby` in combination with `~DataFrame.unstack` at the user guide section on `combining stats and groupby <reshaping.combine_with_groupby>`.  .. raw:: html  </div>  Wide to long format
~~~  Starting again from the wide format table created in the previous
section, we add a new index to the ``DataFrame`` with `~DataFrame.reset_index`.  .. ipython:: python  no2_pivoted = no2.pivot(columns="location", values="value").reset_index()  no2_pivoted.head()  center  .. raw:: html  <ul class="task-bullet">  <li>  I want to collect all air quality `NO_2` measurements in a single column (long format).  .. ipython:: python  no_2 = no2_pivoted.melt(id_vars="date.utc")  no_2.head()  The `pandas.melt` method on a ``DataFrame`` converts the data table from wide
format to long format. The column headers become the variable names in a
newly created column.  .. raw:: html  </li>  </ul>  The solution is the short version on how to apply `pandas.melt`. The method
will *melt* all columns NOT mentioned in ``id_vars`` together into two
columns: A column with the column header names and a column with the
values itself. The latter column gets by default the name ``value``.  The parameters passed to `pandas.melt` can be defined in more detail:  .. ipython:: python  no_2 = no2_pivoted.melt(  id_vars="date.utc",  value_vars=["BETR801", "FR04014", "London Westminster"],  value_name="NO_2",  var_name="id_location",  )  no_2.head()  The additional parameters have the following effects:  -  ``value_vars`` defines which columns to *melt* together
-  ``value_name`` provides a custom column name for the values column  instead of the default column name ``value``
-  ``var_name`` provides a custom column name for the column collecting  the column header names. Otherwise it takes the index name or a  default ``variable``  Hence, the arguments ``value_name`` and ``var_name`` are just
user-defined names for the two generated columns. The columns to melt
are defined by ``id_vars`` and ``value_vars``.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  Conversion from wide to long format with `pandas.melt` is explained in the user guide section on `reshaping by melt <reshaping.melt>`.  .. raw:: html  </div>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Sorting by one or more columns is supported by ``sort_values``.
-  The ``pivot`` function is purely restructuring of the data,  ``pivot_table`` supports aggregations.
-  The reverse of ``pivot`` (long to wide format) is ``melt`` (wide to  long format).  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full overview is available in the user guide on the pages about `reshaping and pivoting <reshaping>`.  .. raw:: html  </div>

Concatenating objects
~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  I want to combine the measurements of `NO_2` and `PM_{25}`, two tables with a similar structure, in a single table.  .. ipython:: python  air_quality = pd.concat([air_quality_pm25, air_quality_no2], axis=0)  air_quality.head()  The `~pandas.concat` function performs concatenation operations of multiple
tables along one of the axes (row-wise or column-wise).  .. raw:: html  </li>  </ul>  By default concatenation is along axis 0, so the resulting table combines the rows
of the input tables. Let’s check the shape of the original and the
concatenated tables to verify the operation:  .. ipython:: python  print('Shape of the ``air_quality_pm25`` table: ', air_quality_pm25.shape)  print('Shape of the ``air_quality_no2`` table: ', air_quality_no2.shape)  print('Shape of the resulting ``air_quality`` table: ', air_quality.shape)  Hence, the resulting table has 3178 = 1110 + 2068 rows.  .. note::  The **axis** argument will return in a number of pandas  methods that can be applied **along an axis**. A ``DataFrame`` has two  corresponding axes: the first running vertically downwards across rows  (axis 0), and the second running horizontally across columns (axis 1).  Most operations like concatenation or summary statistics are by default  across rows (axis 0), but can be applied across columns as well.  Sorting the table on the datetime information illustrates also the
combination of both tables, with the ``parameter`` column defining the
origin of the table (either ``no2`` from table ``air_quality_no2`` or
``pm25`` from table ``air_quality_pm25``):  .. ipython:: python  air_quality = air_quality.sort_values("date.utc")  air_quality.head()  In this specific example, the ``parameter`` column provided by the data
ensures that each of the original tables can be identified. This is not
always the case. The ``concat`` function provides a convenient solution
with the ``keys`` argument, adding an additional (hierarchical) row
index. For example:  .. ipython:: python  air_quality_ = pd.concat([air_quality_pm25, air_quality_no2], keys=["PM25", "NO2"])  air_quality_.head()  .. note::  The existence of multiple row/column indices at the same time  has not been mentioned within these tutorials. *Hierarchical indexing*  or *MultiIndex* is an advanced and powerful pandas feature to analyze  higher dimensional data.  Multi-indexing is out of scope for this pandas introduction. For the  moment, remember that the function ``reset_index`` can be used to  convert any level of an index to a column, e.g.  ``air_quality.reset_index(level=0)``  .. raw:: html  <div class="d-flex flex-row  gs-torefguide">  <span class="badge badge-info">To user guide</span>  Feel free to dive into the world of multi-indexing at the user guide section on `advanced indexing <advanced>`.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  More options on table concatenation (row and column
wise) and how ``concat`` can be used to define the logic (union or
intersection) of the indexes on the other axes is provided at the section on
`object concatenation <merging.concat>`.  .. raw:: html  </div>  Join tables using a common identifier

~~~  center  .. raw:: html  <ul class="task-bullet">  <li>  Add the station coordinates, provided by the stations metadata table, to the corresponding rows in the measurements table.  .. warning::  The air quality measurement station coordinates are stored in a data  file ``air_quality_stations.csv``, downloaded using the  py-openaq_ package.  .. ipython:: python  stations_coord = pd.read_csv("data/air_quality_stations.csv")  stations_coord.head()  .. note::  The stations used in this example (FR04014, BETR801 and London  Westminster) are just three entries enlisted in the metadata table. We  only want to add the coordinates of these three to the measurements  table, each on the corresponding rows of the ``air_quality`` table.  .. ipython:: python  air_quality.head()  .. ipython:: python  air_quality = pd.merge(air_quality, stations_coord, how="left", on="location")  air_quality.head()  Using the `~pandas.merge` function, for each of the rows in the
``air_quality`` table, the corresponding coordinates are added from the
``air_quality_stations_coord`` table. Both tables have the column
``location`` in common which is used as a key to combine the
information. By choosing the ``left`` join, only the locations available
in the ``air_quality`` (left) table, i.e. FR04014, BETR801 and London
Westminster, end up in the resulting table. The ``merge`` function
supports multiple join options similar to database-style operations.  .. raw:: html  </li>  </ul>  .. raw:: html  <ul class="task-bullet">  <li>  Add the parameters' full description and name, provided by the parameters metadata table, to the measurements table.  .. warning::  The air quality parameters metadata are stored in a data file  ``air_quality_parameters.csv``, downloaded using the  py-openaq_ package.  .. ipython:: python  air_quality_parameters = pd.read_csv("data/air_quality_parameters.csv")  air_quality_parameters.head()  .. ipython:: python  air_quality = pd.merge(air_quality, air_quality_parameters,  how='left', left_on='parameter', right_on='id')  air_quality.head()  Compared to the previous example, there is no common column name.
However, the ``parameter`` column in the ``air_quality`` table and the
``id`` column in the ``air_quality_parameters_name`` both provide the
measured variable in a common format. The ``left_on`` and ``right_on``
arguments are used here (instead of just ``on``) to make the link
between the two tables.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  pandas supports also inner, outer, and right joins.
More information on join/merge of tables is provided in the user guide section on
`database style merging of tables <merging.join>`. Or have a look at the
`comparison with SQL<compare_with_sql.join>` page.  .. raw:: html  </div>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Multiple tables can be concatenated both column-wise and row-wise using  the ``concat`` function.
-  For database-like merging/joining of tables, use the ``merge``  function.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  See the user guide for a full description of the various `facilities to combine data tables <merging>`.  .. raw:: html  </div>

.. _10min_tut_09_timeseries.properties:  Using pandas datetime properties
~~~  .. raw:: html  <ul class="task-bullet">  <li>  I want to work with the dates in the column ``datetime`` as datetime objects instead of plain text  .. ipython:: python  air_quality["datetime"] = pd.to_datetime(air_quality["datetime"])  air_quality["datetime"]  Initially, the values in ``datetime`` are character strings and do not
provide any datetime operations (e.g. extract the year, day of the
week,…). By applying the ``to_datetime`` function, pandas interprets the
strings and convert these to datetime (i.e. ``datetime64[ns, UTC]``)
objects. In pandas we call these datetime objects similar to
``datetime.datetime`` from the standard library as `pandas.Timestamp`.  .. raw:: html  </li>  </ul>  .. note::  As many data sets do contain datetime information in one of  the columns, pandas input function like `pandas.read_csv` and `pandas.read_json`  can do the transformation to dates when reading the data using the  ``parse_dates`` parameter with a list of the columns to read as  Timestamp:  ::  pd.read_csv("../data/air_quality_no2_long.csv", parse_dates=["datetime"])  Why are these `pandas.Timestamp` objects useful? Let’s illustrate the added
value with some example cases.  What is the start and end date of the time series data set we are working  with?  .. ipython:: python  air_quality["datetime"].min(), air_quality["datetime"].max()  Using `pandas.Timestamp` for datetimes enables us to calculate with date
information and make them comparable. Hence, we can use this to get the
length of our time series:  .. ipython:: python  air_quality["datetime"].max() - air_quality["datetime"].min()  The result is a `pandas.Timedelta` object, similar to ``datetime.timedelta``
from the standard Python library and defining a time duration.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  The various time concepts supported by pandas are explained in the user guide section on `time related concepts <timeseries.overview>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  I want to add a new column to the ``DataFrame`` containing only the month of the measurement  .. ipython:: python  air_quality["month"] = air_quality["datetime"].dt.month  air_quality.head()  By using ``Timestamp`` objects for dates, a lot of time-related
properties are provided by pandas. For example the ``month``, but also
``year``, ``quarter``,… All of these properties are
accessible by the ``dt`` accessor.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  An overview of the existing date properties is given in the
`time and date components overview table <timeseries.components>`. More details about the ``dt`` accessor
to return datetime like properties are explained in a dedicated section on the  `dt accessor <basics.dt_accessors>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  What is the average `NO_2` concentration for each day of the week for each of the measurement locations?  .. ipython:: python  air_quality.groupby(  [air_quality["datetime"].dt.weekday, "location"])["value"].mean()  Remember the split-apply-combine pattern provided by ``groupby`` from the
`tutorial on statistics calculation <10min_tut_06_stats>`?
Here, we want to calculate a given statistic (e.g. mean `NO_2`)
**for each weekday** and **for each measurement location**. To group on
weekdays, we use the datetime property ``weekday`` (with Monday=0 and
Sunday=6) of pandas ``Timestamp``, which is also accessible by the
``dt`` accessor. The grouping on both locations and weekdays can be done

to split the calculation of the mean on each of these combinations.  .. danger::  As we are working with a very short time series in these  examples, the analysis does not provide a long-term representative  result!  .. raw:: html  </li>  </ul>  .. raw:: html  <ul class="task-bullet">  <li>  Plot the typical `NO_2` pattern during the day of our time series of all stations together. In other words, what is the average value for each hour of the day?  .. ipython:: python  fig, axs = plt.subplots(figsize=(12, 4))  air_quality.groupby(air_quality["datetime"].dt.hour)["value"].mean().plot(  kind='bar', rot=0, ax=axs  )  plt.xlabel("Hour of the day");  # custom x label using Matplotlib  @savefig 09_bar_chart.png  plt.ylabel("$NO_2 (µg/m^3)$");  Similar to the previous case, we want to calculate a given statistic
(e.g. mean `NO_2`) **for each hour of the day** and we can use the
split-apply-combine approach again. For this case, we use the datetime property ``hour``
of pandas ``Timestamp``, which is also accessible by the ``dt`` accessor.  .. raw:: html  </li>  </ul>  Datetime as index
~~~  In the `tutorial on reshaping <10min_tut_07_reshape>`,
`~pandas.pivot` was introduced to reshape the data table with each of the
measurements locations as a separate column:  .. ipython:: python  no_2 = air_quality.pivot(index="datetime", columns="location", values="value")  no_2.head()  .. note::  By pivoting the data, the datetime information became the  index of the table. In general, setting a column as an index can be  achieved by the ``set_index`` function.  Working with a datetime index (i.e. ``DatetimeIndex``) provides powerful
functionalities. For example, we do not need the ``dt`` accessor to get
the time series properties, but have these properties available on the
index directly:  .. ipython:: python  no_2.index.year, no_2.index.weekday  Some other advantages are the convenient subsetting of time period or
the adapted time scale on plots. Let’s apply this on our data.  .. raw:: html  <ul class="task-bullet">  <li>  Create a plot of the `NO_2` values in the different stations from the 20th of May till the end of 21st of May  .. ipython:: python  @savefig 09_time_section.png  no_2["2019-05-20":"2019-05-21"].plot();  By providing a **string that parses to a datetime**, a specific subset of the data can be selected on a ``DatetimeIndex``.  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  More information on the ``DatetimeIndex`` and the slicing by using strings is provided in the section on `time series indexing <timeseries.datetimeindex>`.  .. raw:: html  </div>  Resample a time series to another frequency
~~~  .. raw:: html  <ul class="task-bullet">  <li>  Aggregate the current hourly time series values to the monthly maximum value in each of the stations.  .. ipython:: python  monthly_max = no_2.resample("M").max()  monthly_max  A very powerful method on time series data with a datetime index, is the
ability to `~Series.resample` time series to another frequency (e.g.,
converting secondly data into 5-minutely data).  .. raw:: html  </li>  </ul>  The `~Series.resample` method is similar to a groupby operation:  -  it provides a time-based grouping, by using a string (e.g. ``M``,  ``5H``,…) that defines the target frequency
-  it requires an aggregation function such as ``mean``, ``max``,…  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  An overview of the aliases used to define time series frequencies is given in the `offset aliases overview table <timeseries.offset_aliases>`.  .. raw:: html  </div>  When defined, the frequency of the time series is provided by the

``freq`` attribute:  .. ipython:: python  monthly_max.index.freq  .. raw:: html  <ul class="task-bullet">  <li>  Make a plot of the daily mean `NO_2` value in each of the stations.  .. ipython:: python  @savefig 09_resample_mean.png  no_2.resample("D").mean().plot(style="-o", figsize=(10, 5));  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  More details on the power of time series ``resampling`` is provided in the user guide section on `resampling <timeseries.resampling>`.  .. raw:: html  </div>  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  Valid date strings can be converted to datetime objects using  ``to_datetime`` function or as part of read functions.
-  Datetime objects in pandas support calculations, logical operations  and convenient date-related properties using the ``dt`` accessor.
-  A ``DatetimeIndex`` contains these date-related properties and  supports convenient slicing.
-  ``Resample`` is a powerful method to change the frequency of a time  series.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full overview on time series is given on the pages on `time series and date functionality <timeseries>`.  .. raw:: html  </div>

.. raw:: html  <ul class="task-bullet">  <li>  Make all name characters lowercase.  .. ipython:: python  titanic["Name"].str.lower()  To make each of the strings in the ``Name`` column lowercase, select the ``Name`` column
(see the `tutorial on selection of data <10min_tut_03_subset>`), add the ``str`` accessor and
apply the ``lower`` method. As such, each of the strings is converted element-wise.  .. raw:: html  </li>  </ul>  Similar to datetime objects in the `time series tutorial <10min_tut_09_timeseries>`
having a ``dt`` accessor, a number of
specialized string methods are available when using the ``str``
accessor. These methods have in general matching names with the
equivalent built-in string methods for single elements, but are applied
element-wise (remember `element-wise calculations <10min_tut_05_columns>`?)
on each of the values of the columns.  .. raw:: html  <ul class="task-bullet">  <li>  Create a new column ``Surname`` that contains the surname of the passengers by extracting the part before the comma.  .. ipython:: python  titanic["Name"].str.split(",")  Using the `Series.str.split` method, each of the values is returned as a list of
2 elements. The first element is the part before the comma and the
second element is the part after the comma.  .. ipython:: python  titanic["Surname"] = titanic["Name"].str.split(",").str.get(0)  titanic["Surname"]  As we are only interested in the first part representing the surname
(element 0), we can again use the ``str`` accessor and apply `Series.str.get` to
extract the relevant part. Indeed, these string functions can be
concatenated to combine multiple functions at once!  .. raw:: html  </li>  </ul>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  More information on extracting parts of strings is available in the user guide section on `splitting and replacing strings <text.split>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  Extract the passenger data about the countesses on board of the Titanic.  .. ipython:: python  titanic["Name"].str.contains("Countess")  .. ipython:: python  titanic[titanic["Name"].str.contains("Countess")]  (*Interested in her story? See* Wikipedia_\ *!*)  The string method `Series.str.contains` checks for each of the values in the
column ``Name`` if the string contains the word ``Countess`` and returns
for each of the values ``True`` (``Countess`` is part of the name) or
``False`` (``Countess`` is not part of the name). This output can be used
to subselect the data using conditional (boolean) indexing introduced in
the `subsetting of data tutorial <10min_tut_03_subset>`. As there was
only one countess on the Titanic, we get one row as a result.  .. raw:: html  </li>  </ul>  .. note::  More powerful extractions on strings are supported, as the  `Series.str.contains` and `Series.str.extract` methods accept `regular  expressions <https://docs.python.org/3/library/re.html>`__, but out of  scope of this tutorial.  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  More information on extracting parts of strings is available in the user guide section on `string matching and extracting <text.extract>`.  .. raw:: html  </div>  .. raw:: html  <ul class="task-bullet">  <li>  Which passenger of the Titanic has the longest name?  .. ipython:: python  titanic["Name"].str.len()  To get the longest name we first have to get the lengths of each of the
names in the ``Name`` column. By using pandas string methods, the
`Series.str.len` function is applied to each of the names individually
(element-wise).  .. ipython:: python  titanic["Name"].str.len().idxmax()  Next, we need to get the corresponding location, preferably the index
label, in the table for which the name length is the largest. The
`~Series.idxmax` method does exactly that. It is not a string method and is

applied to integers, so no ``str`` is used.  .. ipython:: python  titanic.loc[titanic["Name"].str.len().idxmax(), "Name"]  Based on the index name of the row (``307``) and the column (``Name``),
we can do a selection using the ``loc`` operator, introduced in the
`tutorial on subsetting <10min_tut_03_subset>`.  .. raw:: html  </li>  </ul>  .. raw:: html  <ul class="task-bullet">  <li>  In the "Sex" column, replace values of "male" by "M" and values of "female" by "F".  .. ipython:: python  titanic["Sex_short"] = titanic["Sex"].replace({"male": "M", "female": "F"})  titanic["Sex_short"]  Whereas `~Series.replace` is not a string method, it provides a convenient way
to use mappings or vocabularies to translate certain values. It requires
a ``dictionary`` to define the mapping ``{from : to}``.  .. raw:: html  </li>  </ul>  .. warning::  There is also a `~Series.str.replace` method available to replace a  specific set of characters. However, when having a mapping of multiple  values, this would become:  ::  titanic["Sex_short"] = titanic["Sex"].str.replace("female", "F")  titanic["Sex_short"] = titanic["Sex_short"].str.replace("male", "M")  This would become cumbersome and easily lead to mistakes. Just think (or  try out yourself) what would happen if those two statements are applied  in the opposite order…  .. raw:: html  <div class="shadow gs-callout gs-callout-remember">  <h4>REMEMBER</h4>  -  String methods are available using the ``str`` accessor.
-  String methods work element-wise and can be used for conditional  indexing.
-  The ``replace`` method is a convenient method to convert values  according to a given dictionary.  .. raw:: html  </div>  .. raw:: html  <div class="d-flex flex-row gs-torefguide">  <span class="badge badge-info">To user guide</span>  A full overview is provided in the user guide pages on `working with text data <text>`.  .. raw:: html  </div>

Getting started tutorials

.. toctree::  1  01_table_oriented  02_read_write  03_subset_data  04_plotting  05_add_columns  06_calculate_statistics  07_reshape_table_layout  08_combine_dataframes  09_timeseries  10_text_data

.. raw:: html  <div data-toggle="collapse" href="#collapsedata" role="button" aria-expanded="false" aria-controls="collapsedata">  <span class="badge badge-dark">Air quality data</span>  </div>  <div class="collapse" id="collapsedata">  <div class="card-body">  <p class="card-text">For this tutorial, air quality data about `NO_2` is used, madeavailable by OpenAQ_ and using thepy-openaq_ package.The ``air_quality_no2.csv`` data set provides `NO_2` values forthe measurement stations *FR04014*, *BETR801* and *London Westminster*in respectively Paris, Antwerp and London... raw:: html  </p>  <a href="https://github.com/pandas-dev/pandas/tree/main/doc/data/air_quality_no2.csv" class="btn btn-dark btn-sm">To raw data</a>  </div>  </div>

.. raw:: html  <div data-toggle="collapse" href="#collapsedata" role="button" aria-expanded="false" aria-controls="collapsedata">  <span class="badge badge-dark">Titanic data</span>  </div>  <div class="collapse" id="collapsedata">  <div class="card-body">  <p class="card-text">This tutorial uses the Titanic data set, stored as CSV. The dataconsists of the following data columns:-  PassengerId: Id of every passenger.-  Survived: Indication whether passenger survived. ``0`` for yes and ``1`` for no.-  Pclass: One out of the 3 ticket classes: Class ``1``, Class ``2`` and Class ``3``.-  Name: Name of passenger.-  Sex: Gender of passenger.-  Age: Age of passenger in years.-  SibSp: Number of siblings or spouses aboard.-  Parch: Number of parents or children aboard.-  Ticket: Ticket number of passenger.-  Fare: Indicating the fare.-  Cabin: Cabin number of passenger.-  Embarked: Port of embarkation... raw:: html  </p>  <a href="https://github.com/pandas-dev/pandas/raw/main/doc/data/titanic.csv" class="btn btn-dark btn-sm">To raw data</a>  </div>  </div>

pandas arrays, scalars, and data types

***
Objects
***  .. currentmodule:: pandas  For most data types, pandas uses NumPy arrays as the concrete
objects contained with a `Index`, `Series`, or
`DataFrame`.  For some data types, pandas extends NumPy's type system. String aliases for these types
can be found at `basics.dtypes`.  === === === ===
Kind of Data  pandas Data Type  Scalar  Array
TZ-aware datetime  `DatetimeTZDtype`  `Timestamp`  `api.arrays.datetime`
Timedeltas  (none)  `Timedelta`  `api.arrays.timedelta`
Period (time spans) `PeriodDtype`  `Period`  `api.arrays.period`
Intervals  `IntervalDtype`  `Interval`  `api.arrays.interval`
Nullable Integer  `Int64Dtype`, ...  (none)  `api.arrays.integer_na`
Nullable Float  `Float64Dtype`, ... (none)  `api.arrays.float_na`
Categorical  `CategoricalDtype`  (none)  `api.arrays.categorical`
Sparse  `SparseDtype`  (none)  `api.arrays.sparse`
Strings  `StringDtype`  `str`  `api.arrays.string`
Nullable Boolean  `BooleanDtype`  `bool`  `api.arrays.bool`
PyArrow  `ArrowDtype`  Python Scalars or `NA` `api.arrays.arrow`
=== === === ===  pandas and third-party libraries can extend NumPy's type system (see `extending.extension-types`).
The top-level `array` method can be used to create a new array, which may be
stored in a `Series`, `Index`, or as a column in a `DataFrame`.  .. autosummary::  api/  array  .. _api.arrays.arrow:

PyArrow

.. warning::  This feature is experimental, and the API can change in a future release without warning.  The `arrays.ArrowExtensionArray` is backed by a :external+pyarrowclass:`pyarrow.ChunkedArray` with a
:external+pyarrowclass:`pyarrow.DataType` instead of a NumPy array and data type. The ``.dtype`` of a `arrays.ArrowExtensionArray`
is an `ArrowDtype`.  Pyarrow_ provides similar array and data type_
support as NumPy including first-class nullability support for all data types, immutability and more.  The table below shows the equivalent pyarrow-backed (``pa``), pandas extension, and numpy (``np``) types that are recognized by pandas.
Pyarrow-backed types below need to be passed into `ArrowDtype` to be recognized by pandas e.g. ``pd.ArrowDtype(pa.bool_())``  === === ===
PyArrow type  pandas extension type  NumPy type
=== === ===
:external+pyarrowfunc:`pyarrow.bool_`  `BooleanDtype`  ``np.bool_``
:external+pyarrowfunc:`pyarrow.int8`  `Int8Dtype`  ``np.int8``
:external+pyarrowfunc:`pyarrow.int16`  `Int16Dtype`  ``np.int16``
:external+pyarrowfunc:`pyarrow.int32`  `Int32Dtype`  ``np.int32``
:external+pyarrowfunc:`pyarrow.int64`  `Int64Dtype`  ``np.int64``
:external+pyarrowfunc:`pyarrow.uint8`  `UInt8Dtype`  ``np.uint8``
:external+pyarrowfunc:`pyarrow.uint16`  `UInt16Dtype`  ``np.uint16``
:external+pyarrowfunc:`pyarrow.uint32`  `UInt32Dtype`  ``np.uint32``
:external+pyarrowfunc:`pyarrow.uint64`  `UInt64Dtype`  ``np.uint64``
:external+pyarrowfunc:`pyarrow.float32`  `Float32Dtype`  ``np.float32``
:external+pyarrowfunc:`pyarrow.float64`  `Float64Dtype`  ``np.float64``
:external+pyarrowfunc:`pyarrow.time32`  (none)  (none)
:external+pyarrowfunc:`pyarrow.time64`  (none)  (none)
:external+pyarrowfunc:`pyarrow.timestamp`  `DatetimeTZDtype`  ``np.datetime64``
:external+pyarrowfunc:`pyarrow.date32`  (none)  (none)
:external+pyarrowfunc:`pyarrow.date64`  (none)  (none)
:external+pyarrowfunc:`pyarrow.duration`  (none)  ``np.timedelta64``
:external+pyarrowfunc:`pyarrow.binary`  (none)  (none)
:external+pyarrowfunc:`pyarrow.string`  `StringDtype`  ``np.str_``
:external+pyarrowfunc:`pyarrow.decimal128`  (none)  (none)
:external+pyarrowfunc:`pyarrow.list_`  (none)  (none)
:external+pyarrowfunc:`pyarrow.map_`  (none)  (none)
:external+pyarrowfunc:`pyarrow.dictionary`  `CategoricalDtype`  (none)
=== === ===  .. note::  For string types (``pyarrow.string()``, ``string[pyarrow]``), PyArrow support is still facilitated  by `arrays.ArrowStringArray` and ``StringDtype("pyarrow")``. See the `string section <api.arrays.string>`  below.  While individual values in an `arrays.ArrowExtensionArray` are stored as a PyArrow objects, scalars are **returned**
as Python scalars corresponding to the data type, e.g. a PyArrow int64 will be returned as Python int, or `NA` for missing
values.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.ArrowExtensionArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  ArrowDtype  .. _api.arrays.datetime:

Datetimes

NumPy cannot natively represent timezone-aware datetimes. pandas supports this
with the `arrays.DatetimeArray` extension array, which can hold timezone-naive
or timezone-aware values.  `Timestamp`, a subclass of `datetime.datetime`, is pandas'
scalar type for timezone-naive or timezone-aware datetime data. `NaT`
is the missing value for datetime data.  .. autosummary::  api/  Timestamp  .. autosummary::  api/  autosummary/class_without_autosummary.rst  NaT  Properties
~~~
.. autosummary::  api/  Timestamp.asm8  Timestamp.day  Timestamp.dayofweek  Timestamp.day_of_week  Timestamp.dayofyear  Timestamp.day_of_year  Timestamp.days_in_month  Timestamp.daysinmonth  Timestamp.fold  Timestamp.hour  Timestamp.is_leap_year  Timestamp.is_month_end  Timestamp.is_month_start  Timestamp.is_quarter_end  Timestamp.is_quarter_start  Timestamp.is_year_end  Timestamp.is_year_start  Timestamp.max  Timestamp.microsecond  Timestamp.min  Timestamp.minute  Timestamp.month  Timestamp.nanosecond  Timestamp.quarter  Timestamp.resolution  Timestamp.second  Timestamp.tz  Timestamp.tzinfo  Timestamp.unit  Timestamp.value  Timestamp.week  Timestamp.weekofyear  Timestamp.year  Methods
~~~
.. autosummary::  api/  Timestamp.as_unit  Timestamp.astimezone  Timestamp.ceil  Timestamp.combine  Timestamp.ctime  Timestamp.date  Timestamp.day_name  Timestamp.dst  Timestamp.floor  Timestamp.fromordinal  Timestamp.fromtimestamp  Timestamp.isocalendar  Timestamp.isoformat  Timestamp.isoweekday  Timestamp.month_name  Timestamp.normalize  Timestamp.now  Timestamp.replace  Timestamp.round  Timestamp.strftime  Timestamp.strptime  Timestamp.time  Timestamp.timestamp  Timestamp.timetuple  Timestamp.timetz  Timestamp.to_datetime64  Timestamp.to_numpy  Timestamp.to_julian_date  Timestamp.to_period  Timestamp.to_pydatetime  Timestamp.today  Timestamp.toordinal  Timestamp.tz_convert  Timestamp.tz_localize  Timestamp.tzname  Timestamp.utcfromtimestamp  Timestamp.utcnow  Timestamp.utcoffset  Timestamp.utctimetuple  Timestamp.weekday  A collection of timestamps may be stored in a `arrays.DatetimeArray`.
For timezone-aware data, the ``.dtype`` of a `arrays.DatetimeArray` is a
`DatetimeTZDtype`. For timezone-naive data, ``np.dtype("datetime64[ns]")``
is used.  If the data are timezone-aware, then every value in the array must have the same timezone.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.DatetimeArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  DatetimeTZDtype  .. _api.arrays.timedelta:

Timedeltas

NumPy can natively represent timedeltas. pandas provides `Timedelta`
for symmetry with `Timestamp`. `NaT`
is the missing value for timedelta data.  .. autosummary::  api/  Timedelta  .. autosummary::  api/  autosummary/class_without_autosummary.rst  NaT  Properties
~~~
.. autosummary::  api/  Timedelta.asm8  Timedelta.components  Timedelta.days  Timedelta.max  Timedelta.microseconds  Timedelta.min  Timedelta.nanoseconds  Timedelta.resolution  Timedelta.seconds  Timedelta.unit  Timedelta.value  Timedelta.view  Methods
~~~
.. autosummary::  api/  Timedelta.as_unit  Timedelta.ceil  Timedelta.floor  Timedelta.isoformat  Timedelta.round  Timedelta.to_pytimedelta  Timedelta.to_timedelta64  Timedelta.to_numpy  Timedelta.total_seconds  A collection of `Timedelta` may be stored in a `TimedeltaArray`.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.TimedeltaArray  .. _api.arrays.period:

Period
.. autosummary::  api/  Period  Properties
~~~
.. autosummary::  api/  Period.day  Period.dayofweek  Period.day_of_week  Period.dayofyear  Period.day_of_year  Period.days_in_month  Period.daysinmonth  Period.end_time  Period.freq  Period.freqstr  Period.hour  Period.is_leap_year  Period.minute  Period.month  Period.ordinal  Period.quarter  Period.qyear  Period.second  Period.start_time  Period.week  Period.weekday  Period.weekofyear  Period.year  Methods
~~~
.. autosummary::  api/  Period.asfreq  Period.now  Period.strftime  Period.to_timestamp  A collection of `Period` may be stored in a `arrays.PeriodArray`.
Every period in a `arrays.PeriodArray` must have the same ``freq``.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.PeriodArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  PeriodDtype  .. _api.arrays.interval:

Intervals

Arbitrary intervals can be represented as `Interval` objects.  .. autosummary::  api/  Interval  Properties
~~~
.. autosummary::  api/  Interval.closed  Interval.closed_left  Interval.closed_right  Interval.is_empty  Interval.left  Interval.length  Interval.mid  Interval.open_left  Interval.open_right  Interval.overlaps  Interval.right  A collection of intervals may be stored in an `arrays.IntervalArray`.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.IntervalArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  IntervalDtype  .. Those attributes and methods are included in the API because the docstrings
.. of IntervalIndex and IntervalArray are shared. Including it here to make
.. sure a docstring page is built for them to avoid warnings  ..  .. autosummary::  api/  arrays.IntervalArray.left  arrays.IntervalArray.right  arrays.IntervalArray.closed  arrays.IntervalArray.mid  arrays.IntervalArray.length  arrays.IntervalArray.is_empty  arrays.IntervalArray.is_non_overlapping_monotonic  arrays.IntervalArray.from_arrays  arrays.IntervalArray.from_tuples  arrays.IntervalArray.from_breaks  arrays.IntervalArray.contains  arrays.IntervalArray.overlaps  arrays.IntervalArray.set_closed  arrays.IntervalArray.to_tuples  .. _api.arrays.integer_na:

Nullable integer

`numpy.ndarray` cannot natively represent integer-data with missing values.
pandas provides this through `arrays.IntegerArray`.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.IntegerArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  Int8Dtype  Int16Dtype  Int32Dtype  Int64Dtype  UInt8Dtype  UInt16Dtype  UInt32Dtype  UInt64Dtype  NA  .. _api.arrays.float_na:

Nullable float

.. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.FloatingArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  Float32Dtype  Float64Dtype  NA  .. _api.arrays.categorical:

Categoricals

pandas defines a custom data type for representing data that can take only a
limited, fixed set of values. The dtype of a `Categorical` can be described by
a `CategoricalDtype`.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  CategoricalDtype  .. autosummary::  api/  CategoricalDtype.categories  CategoricalDtype.ordered  Categorical data can be stored in a `pandas.Categorical`  .. autosummary::  api/  autosummary/class_without_autosummary.rst  Categorical  The alternative `Categorical.from_codes` constructor can be used when you
have the categories and integer codes already:  .. autosummary::  api/  Categorical.from_codes  The dtype information is available on the `Categorical`  .. autosummary::  api/  Categorical.dtype  Categorical.categories  Categorical.ordered  Categorical.codes  ``np.asarray(categorical)`` works by implementing the array interface. Be aware, that this converts
the `Categorical` back to a NumPy array, so categories and order information is not preserved!  .. autosummary::  api/  Categorical.__array__  A `Categorical` can be stored in a `Series` or `DataFrame`.
To create a Series of dtype ``category``, use ``cat = s.astype(dtype)`` or
``Series(..., dtype=dtype)`` where ``dtype`` is either  * the string ``'category'``
* an instance of `CategoricalDtype`.  If the `Series` is of dtype `CategoricalDtype`, ``Series.cat`` can be used to change the categorical
data. See `api.series.cat` for more.  .. _api.arrays.sparse:

Sparse

Data where a single value is repeated many times (e.g. ``0`` or ``NaN``) may
be stored efficiently as a `arrays.SparseArray`.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.SparseArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  SparseDtype  The ``Series.sparse`` accessor may be used to access sparse-specific attributes
and methods if the `Series` contains sparse values. See
`api.series.sparse` and `the user guide <sparse>` for more.  .. _api.arrays.string:

Strings

When working with text data, where each valid element is a string or missing,
we recommend using `StringDtype` (with the alias ``"string"``).  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.StringArray  arrays.ArrowStringArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  StringDtype  The ``Series.str`` accessor is available for `Series` backed by a `arrays.StringArray`.
See `api.series.str` for more.  .. _api.arrays.bool:

Nullable Boolean

The boolean dtype (with the alias ``"boolean"``) provides support for storing
boolean data (``True``, ``False``) with missing values, which is not possible
with a bool `numpy.ndarray`.  .. autosummary::  api/  autosummary/class_without_autosummary.rst  arrays.BooleanArray  .. autosummary::  api/  autosummary/class_without_autosummary.rst  BooleanDtype  NA  .. Dtype attributes which are manually listed in their docstrings: including
.. it here to make sure a docstring page is built for them  ..  .. autosummary::  api/  DatetimeTZDtype.unit  DatetimeTZDtype.tz  PeriodDtype.freq  IntervalDtype.subtype  ***
Utilities
***

Constructors
.. autosummary::  api/  api.types.union_categoricals  api.types.infer_dtype  api.types.pandas_dtype  Data type introspection
~~~
.. autosummary::  api/  api.types.is_any_real_numeric_dtype  api.types.is_bool_dtype  api.types.is_categorical_dtype  api.types.is_complex_dtype  api.types.is_datetime64_any_dtype  api.types.is_datetime64_dtype  api.types.is_datetime64_ns_dtype  api.types.is_datetime64tz_dtype  api.types.is_extension_array_dtype  api.types.is_float_dtype  api.types.is_int64_dtype  api.types.is_integer_dtype  api.types.is_interval_dtype  api.types.is_numeric_dtype  api.types.is_object_dtype  api.types.is_period_dtype  api.types.is_signed_integer_dtype  api.types.is_string_dtype  api.types.is_timedelta64_dtype  api.types.is_timedelta64_ns_dtype  api.types.is_unsigned_integer_dtype  api.types.is_sparse  Iterable introspection
~~~
.. autosummary::  api/  api.types.is_dict_like  api.types.is_file_like  api.types.is_list_like  api.types.is_named_tuple  api.types.is_iterator  Scalar introspection
~~~
.. autosummary::  api/  api.types.is_bool  api.types.is_complex  api.types.is_float  api.types.is_hashable  api.types.is_integer  api.types.is_interval  api.types.is_number  api.types.is_re  api.types.is_re_compilable  api.types.is_scalar

Extensions
.. currentmodule:: pandas  These are primarily intended for library authors looking to extend pandas
objects.  .. autosummary::  api/  api.extensions.register_extension_dtype  api.extensions.register_dataframe_accessor  api.extensions.register_series_accessor  api.extensions.register_index_accessor  api.extensions.ExtensionDtype  .. autosummary::  api/  autosummary/class_without_autosummary.rst  api.extensions.ExtensionArray  arrays.PandasArray  .. We need this autosummary so that methods and attributes are generated.
.. Separate block, since they aren't classes.  .. autosummary::  api/  api.extensions.ExtensionArray._accumulate  api.extensions.ExtensionArray._concat_same_type  api.extensions.ExtensionArray._formatter  api.extensions.ExtensionArray._from_factorized  api.extensions.ExtensionArray._from_sequence  api.extensions.ExtensionArray._from_sequence_of_strings  api.extensions.ExtensionArray._reduce  api.extensions.ExtensionArray._values_for_argsort  api.extensions.ExtensionArray._values_for_factorize  api.extensions.ExtensionArray.argsort  api.extensions.ExtensionArray.astype  api.extensions.ExtensionArray.copy  api.extensions.ExtensionArray.view  api.extensions.ExtensionArray.dropna  api.extensions.ExtensionArray.equals  api.extensions.ExtensionArray.factorize  api.extensions.ExtensionArray.fillna  api.extensions.ExtensionArray.insert  api.extensions.ExtensionArray.isin  api.extensions.ExtensionArray.isna  api.extensions.ExtensionArray.ravel  api.extensions.ExtensionArray.repeat  api.extensions.ExtensionArray.searchsorted  api.extensions.ExtensionArray.shift  api.extensions.ExtensionArray.take  api.extensions.ExtensionArray.unique  api.extensions.ExtensionArray.dtype  api.extensions.ExtensionArray.nbytes  api.extensions.ExtensionArray.ndim  api.extensions.ExtensionArray.shape  api.extensions.ExtensionArray.tolist  Additionally, we have some utility methods for ensuring your object
behaves correctly.  .. autosummary::  api/  api.indexers.check_array_indexer  The sentinel ``pandas.api.extensions.no_default`` is used as the default
value in some methods. Use an ``is`` comparison to check if the user
provides a non-default value.

DataFrame
.. currentmodule:: pandas  Constructor
~~~
.. autosummary::  api/  DataFrame  Attributes and underlying data
~~~
**Axes**  .. autosummary::  api/  DataFrame.index  DataFrame.columns  .. autosummary::  api/  DataFrame.dtypes  DataFrame.info  DataFrame.select_dtypes  DataFrame.values  DataFrame.axes  DataFrame.ndim  DataFrame.size  DataFrame.shape  DataFrame.memory_usage  DataFrame.empty  DataFrame.set_flags  Conversion
~~~
.. autosummary::  api/  DataFrame.astype  DataFrame.convert_dtypes  DataFrame.infer_objects  DataFrame.copy  DataFrame.bool  Indexing, iteration
~~~
.. autosummary::  api/  DataFrame.head  DataFrame.at  DataFrame.iat  DataFrame.loc  DataFrame.iloc  DataFrame.insert  DataFrame.__iter__  DataFrame.items  DataFrame.keys  DataFrame.iterrows  DataFrame.itertuples  DataFrame.pop  DataFrame.tail  DataFrame.xs  DataFrame.get  DataFrame.isin  DataFrame.where  DataFrame.mask  DataFrame.query  For more information on ``.at``, ``.iat``, ``.loc``, and
``.iloc``,  see the `indexing documentation <indexing>`.  Binary operator functions
~~~
.. autosummary::  api/  DataFrame.__add__  DataFrame.add  DataFrame.sub  DataFrame.mul  DataFrame.div  DataFrame.truediv  DataFrame.floordiv  DataFrame.mod  DataFrame.pow  DataFrame.dot  DataFrame.radd  DataFrame.rsub  DataFrame.rmul  DataFrame.rdiv  DataFrame.rtruediv  DataFrame.rfloordiv  DataFrame.rmod  DataFrame.rpow  DataFrame.lt  DataFrame.gt  DataFrame.le  DataFrame.ge  DataFrame.ne  DataFrame.eq  DataFrame.combine  DataFrame.combine_first  Function application, GroupBy & window
~~~
.. autosummary::  api/  DataFrame.apply  DataFrame.applymap  DataFrame.pipe  DataFrame.agg  DataFrame.aggregate  DataFrame.transform  DataFrame.groupby  DataFrame.rolling  DataFrame.expanding  DataFrame.ewm  .. _api.dataframe.stats:  Computations / descriptive stats
~~~
.. autosummary::  api/  DataFrame.abs  DataFrame.all  DataFrame.any  DataFrame.clip  DataFrame.corr  DataFrame.corrwith  DataFrame.count  DataFrame.cov  DataFrame.cummax  DataFrame.cummin  DataFrame.cumprod  DataFrame.cumsum  DataFrame.describe  DataFrame.diff  DataFrame.eval  DataFrame.kurt  DataFrame.kurtosis  DataFrame.max  DataFrame.mean  DataFrame.median  DataFrame.min  DataFrame.mode  DataFrame.pct_change  DataFrame.prod  DataFrame.product  DataFrame.quantile  DataFrame.rank  DataFrame.round  DataFrame.sem  DataFrame.skew  DataFrame.sum  DataFrame.std  DataFrame.var  DataFrame.nunique  DataFrame.value_counts  Reindexing / selection / label manipulation
~~~
.. autosummary::  api/  DataFrame.add_prefix  DataFrame.add_suffix  DataFrame.align  DataFrame.at_time  DataFrame.between_time  DataFrame.drop  DataFrame.drop_duplicates  DataFrame.duplicated  DataFrame.equals  DataFrame.filter  DataFrame.first  DataFrame.head  DataFrame.idxmax  DataFrame.idxmin  DataFrame.last  DataFrame.reindex  DataFrame.reindex_like  DataFrame.rename  DataFrame.rename_axis  DataFrame.reset_index  DataFrame.sample  DataFrame.set_axis  DataFrame.set_index  DataFrame.tail  DataFrame.take  DataFrame.truncate  .. _api.dataframe.missing:  Missing data handling
~~~
.. autosummary::  api/  DataFrame.backfill  DataFrame.bfill  DataFrame.dropna  DataFrame.ffill  DataFrame.fillna  DataFrame.interpolate  DataFrame.isna  DataFrame.isnull  DataFrame.notna  DataFrame.notnull  DataFrame.pad  DataFrame.replace  Reshaping, sorting, transposing
~~~
.. autosummary::  api/  DataFrame.droplevel  DataFrame.pivot  DataFrame.pivot_table  DataFrame.reorder_levels  DataFrame.sort_values  DataFrame.sort_index  DataFrame.nlargest  DataFrame.nsmallest  DataFrame.swaplevel  DataFrame.stack  DataFrame.unstack  DataFrame.swapaxes  DataFrame.melt  DataFrame.explode  DataFrame.squeeze  DataFrame.to_xarray  DataFrame.T  DataFrame.transpose  Combining / comparing / joining / merging
~~~
.. autosummary::  api/  DataFrame.assign  DataFrame.compare  DataFrame.join  DataFrame.merge  DataFrame.update  Time Series-related
~~~

~~~
~~~
.. autosummary::  api/  DataFrame.asfreq  DataFrame.asof  DataFrame.shift  DataFrame.first_valid_index  DataFrame.last_valid_index  DataFrame.resample  DataFrame.to_period  DataFrame.to_timestamp  DataFrame.tz_convert  DataFrame.tz_localize  .. _api.frame.flags:  Flags
~~~  Flags refer to attributes of the pandas object. Properties of the dataset (like
the date is was recorded, the URL it was accessed from, etc.) should be stored
in `DataFrame.attrs`.  .. autosummary::  api/  Flags  .. _api.frame.metadata:  Metadata
~~~  `DataFrame.attrs` is a dictionary for storing global metadata for this DataFrame.  .. warning:: ``DataFrame.attrs`` is considered experimental and may change without warning.  .. autosummary::  api/  DataFrame.attrs  .. _api.dataframe.plotting:  Plotting
~~~
``DataFrame.plot`` is both a callable method and a namespace attribute for
specific plotting methods of the form ``DataFrame.plot.<kind>``.  .. autosummary::  api/  autosummary/accessor_callable.rst  DataFrame.plot  .. autosummary::  api/  autosummary/accessor_method.rst  DataFrame.plot.area  DataFrame.plot.bar  DataFrame.plot.barh  DataFrame.plot.box  DataFrame.plot.density  DataFrame.plot.hexbin  DataFrame.plot.hist  DataFrame.plot.kde  DataFrame.plot.line  DataFrame.plot.pie  DataFrame.plot.scatter  .. autosummary::  api/  DataFrame.boxplot  DataFrame.hist  .. _api.frame.sparse:  Sparse accessor
~~~  Sparse-dtype specific methods and attributes are provided under the
``DataFrame.sparse`` accessor.  .. autosummary::  api/  autosummary/accessor_attribute.rst  DataFrame.sparse.density  .. autosummary::  api/  autosummary/accessor_method.rst  DataFrame.sparse.from_spmatrix  DataFrame.sparse.to_coo  DataFrame.sparse.to_dense  Serialization / IO / conversion
~~~
.. autosummary::  api/  DataFrame.from_dict  DataFrame.from_records  DataFrame.to_orc  DataFrame.to_parquet  DataFrame.to_pickle  DataFrame.to_csv  DataFrame.to_hdf  DataFrame.to_sql  DataFrame.to_dict  DataFrame.to_excel  DataFrame.to_json  DataFrame.to_html  DataFrame.to_feather  DataFrame.to_latex  DataFrame.to_stata  DataFrame.to_gbq  DataFrame.to_records  DataFrame.to_string  DataFrame.to_clipboard  DataFrame.to_markdown  DataFrame.style  DataFrame.__dataframe__

General functions
.. currentmodule:: pandas  Data manipulations
~~~
.. autosummary::  api/  melt  pivot  pivot_table  crosstab  cut  qcut  merge  merge_ordered  merge_asof  concat  get_dummies  from_dummies  factorize  unique  lreshape  wide_to_long  Top-level missing data
~~~
.. autosummary::  api/  isna  isnull  notna  notnull  Top-level dealing with numeric data
~~~
.. autosummary::  api/  to_numeric  Top-level dealing with datetimelike data
~~~
.. autosummary::  api/  to_datetime  to_timedelta  date_range  bdate_range  period_range  timedelta_range  infer_freq  Top-level dealing with Interval data
~~~
.. autosummary::  api/  interval_range  Top-level evaluation
~~~
.. autosummary::  api/  eval  Hashing
~~~
.. autosummary::  api/  util.hash_array  util.hash_pandas_object  Importing from other DataFrame libraries
~~~
.. autosummary::  api/  api.interchange.from_dataframe

GroupBy
.. currentmodule:: pandas.core.groupby  GroupBy objects are returned by groupby calls: `pandas.DataFrame.groupby`, `pandas.Series.groupby`, etc.

Indexing, iteration
.. autosummary::  api/  DataFrameGroupBy.__iter__  SeriesGroupBy.__iter__  DataFrameGroupBy.groups  SeriesGroupBy.groups  DataFrameGroupBy.indices  SeriesGroupBy.indices  DataFrameGroupBy.get_group  SeriesGroupBy.get_group  .. currentmodule:: pandas  .. autosummary::  api/  autosummary/class_without_autosummary.rst  Grouper

Function application helper
.. autosummary::  api/  NamedAgg  .. currentmodule:: pandas.core.groupby

Function application
.. autosummary::  api/  SeriesGroupBy.apply  DataFrameGroupBy.apply  SeriesGroupBy.agg  DataFrameGroupBy.agg  SeriesGroupBy.aggregate  DataFrameGroupBy.aggregate  SeriesGroupBy.transform  DataFrameGroupBy.transform  SeriesGroupBy.pipe  DataFrameGroupBy.pipe  DataFrameGroupBy.filter  SeriesGroupBy.filter

``DataFrameGroupBy`` computations / descriptive stats
.. autosummary::  api/  DataFrameGroupBy.all  DataFrameGroupBy.any  DataFrameGroupBy.bfill  DataFrameGroupBy.corr  DataFrameGroupBy.corrwith  DataFrameGroupBy.count  DataFrameGroupBy.cov  DataFrameGroupBy.cumcount  DataFrameGroupBy.cummax  DataFrameGroupBy.cummin  DataFrameGroupBy.cumprod  DataFrameGroupBy.cumsum  DataFrameGroupBy.describe  DataFrameGroupBy.diff  DataFrameGroupBy.ffill  DataFrameGroupBy.fillna  DataFrameGroupBy.first  DataFrameGroupBy.head  DataFrameGroupBy.idxmax  DataFrameGroupBy.idxmin  DataFrameGroupBy.last  DataFrameGroupBy.max  DataFrameGroupBy.mean  DataFrameGroupBy.median  DataFrameGroupBy.min  DataFrameGroupBy.ngroup  DataFrameGroupBy.nth  DataFrameGroupBy.nunique  DataFrameGroupBy.ohlc  DataFrameGroupBy.pct_change  DataFrameGroupBy.prod  DataFrameGroupBy.quantile  DataFrameGroupBy.rank  DataFrameGroupBy.resample  DataFrameGroupBy.rolling  DataFrameGroupBy.sample  DataFrameGroupBy.sem  DataFrameGroupBy.shift  DataFrameGroupBy.size  DataFrameGroupBy.skew  DataFrameGroupBy.std  DataFrameGroupBy.sum  DataFrameGroupBy.var  DataFrameGroupBy.tail  DataFrameGroupBy.take  DataFrameGroupBy.value_counts

``SeriesGroupBy`` computations / descriptive stats
.. autosummary::  api/  SeriesGroupBy.all  SeriesGroupBy.any  SeriesGroupBy.bfill  SeriesGroupBy.corr  SeriesGroupBy.count  SeriesGroupBy.cov  SeriesGroupBy.cumcount  SeriesGroupBy.cummax  SeriesGroupBy.cummin  SeriesGroupBy.cumprod  SeriesGroupBy.cumsum  SeriesGroupBy.describe  SeriesGroupBy.diff  SeriesGroupBy.ffill  SeriesGroupBy.fillna  SeriesGroupBy.first  SeriesGroupBy.head  SeriesGroupBy.last  SeriesGroupBy.idxmax  SeriesGroupBy.idxmin  SeriesGroupBy.is_monotonic_increasing  SeriesGroupBy.is_monotonic_decreasing  SeriesGroupBy.max  SeriesGroupBy.mean  SeriesGroupBy.median  SeriesGroupBy.min  SeriesGroupBy.ngroup  SeriesGroupBy.nlargest  SeriesGroupBy.nsmallest  SeriesGroupBy.nth  SeriesGroupBy.nunique  SeriesGroupBy.unique  SeriesGroupBy.ohlc  SeriesGroupBy.pct_change  SeriesGroupBy.prod  SeriesGroupBy.quantile  SeriesGroupBy.rank  SeriesGroupBy.resample  SeriesGroupBy.rolling  SeriesGroupBy.sample  SeriesGroupBy.sem  SeriesGroupBy.shift  SeriesGroupBy.size  SeriesGroupBy.skew  SeriesGroupBy.std  SeriesGroupBy.sum  SeriesGroupBy.var  SeriesGroupBy.tail  SeriesGroupBy.take  SeriesGroupBy.value_counts

Plotting and visualization
.. autosummary::  api/  DataFrameGroupBy.boxplot  DataFrameGroupBy.hist  SeriesGroupBy.hist  DataFrameGroupBy.plot  SeriesGroupBy.plot

API reference

This page gives an overview of all public pandas objects, functions and
methods. All classes and functions exposed in ``pandas.*`` namespace are public.  Some subpackages are public which include ``pandas.errors``,
``pandas.plotting``, and ``pandas.testing``. Public functions in
``pandas.io`` and ``pandas.tseries`` submodules are mentioned in
the documentation. ``pandas.api.types`` subpackage holds some
public functions related to data types in pandas.  .. warning::  The ``pandas.core``, ``pandas.compat``, and ``pandas.util`` top-level modules are PRIVATE. Stable functionality in such modules is not guaranteed.  .. If you update this toctree, also update the manual toctree in the  main index.rst.template  .. toctree::  2  io  general_functions  series  frame  arrays  indexing  offset_frequency  window  groupby  resampling  style  plotting  options  extensions  testing  .. This is to prevent warnings in the doc build. We don't want to encourage
.. these methods.  ..  .. toctree::  api/pandas.Index.holds_integer  api/pandas.Index.nlevels  api/pandas.Index.sort  .. Can't convince sphinx to generate toctree for this class attribute.
.. So we do it manually to avoid a warning  ..  .. toctree::  api/pandas.api.extensions.ExtensionDtype.na_value

Index
.. currentmodule:: pandas  **Many of these methods or variants thereof are available on the objects
that contain an index (Series/DataFrame) and those should most likely be
used before calling these methods directly.**  .. autosummary::  api/  Index  Properties
~~~
.. autosummary::  api/  Index.values  Index.is_monotonic_increasing  Index.is_monotonic_decreasing  Index.is_unique  Index.has_duplicates  Index.hasnans  Index.dtype  Index.inferred_type  Index.shape  Index.name  Index.names  Index.nbytes  Index.ndim  Index.size  Index.empty  Index.T  Index.memory_usage  Modifying and computations
~~~
.. autosummary::  api/  Index.all  Index.any  Index.argmin  Index.argmax  Index.copy  Index.delete  Index.drop  Index.drop_duplicates  Index.duplicated  Index.equals  Index.factorize  Index.identical  Index.insert  Index.is_  Index.is_boolean  Index.is_categorical  Index.is_floating  Index.is_integer  Index.is_interval  Index.is_numeric  Index.is_object  Index.min  Index.max  Index.reindex  Index.rename  Index.repeat  Index.where  Index.take  Index.putmask  Index.unique  Index.nunique  Index.value_counts  Compatibility with MultiIndex
~~~
.. autosummary::  api/  Index.set_names  Index.droplevel  Missing values
~~~
.. autosummary::  api/  Index.fillna  Index.dropna  Index.isna  Index.notna  Conversion
~~~
.. autosummary::  api/  Index.astype  Index.item  Index.map  Index.ravel  Index.to_list  Index.to_series  Index.to_frame  Index.view  Sorting
~~~
.. autosummary::  api/  Index.argsort  Index.searchsorted  Index.sort_values  Time-specific operations
~~~
.. autosummary::  api/  Index.shift  Combining / joining / set operations
~~~
.. autosummary::  api/  Index.append  Index.join  Index.intersection  Index.union  Index.difference  Index.symmetric_difference  Selecting
~~~
.. autosummary::  api/  Index.asof  Index.asof_locs  Index.get_indexer  Index.get_indexer_for  Index.get_indexer_non_unique  Index.get_level_values  Index.get_loc  Index.get_slice_bound  Index.isin  Index.slice_indexer  Index.slice_locs  .. _api.numericindex:

Numeric Index
.. autosummary::  api/  autosummary/class_without_autosummary.rst  RangeIndex  .. We need this autosummary so that the methods are generated.
.. Separate block, since they aren't classes.  .. autosummary::  api/  RangeIndex.start  RangeIndex.stop  RangeIndex.step  RangeIndex.from_range  .. _api.categoricalindex:

CategoricalIndex
.. autosummary::  api/  autosummary/class_without_autosummary.rst  CategoricalIndex  Categorical components
~~~
.. autosummary::  api/  CategoricalIndex.codes  CategoricalIndex.categories  CategoricalIndex.ordered  CategoricalIndex.rename_categories  CategoricalIndex.reorder_categories  CategoricalIndex.add_categories  CategoricalIndex.remove_categories  CategoricalIndex.remove_unused_categories  CategoricalIndex.set_categories  CategoricalIndex.as_ordered  CategoricalIndex.as_unordered  Modifying and computations
~~~
.. autosummary::  api/  CategoricalIndex.map  CategoricalIndex.equals  .. _api.intervalindex:

IntervalIndex
.. autosummary::  api/  autosummary/class_without_autosummary.rst  IntervalIndex  IntervalIndex components
~~~
.. autosummary::  api/  IntervalIndex.from_arrays  IntervalIndex.from_tuples  IntervalIndex.from_breaks  IntervalIndex.left  IntervalIndex.right  IntervalIndex.mid  IntervalIndex.closed  IntervalIndex.length  IntervalIndex.values  IntervalIndex.is_empty  IntervalIndex.is_non_overlapping_monotonic  IntervalIndex.is_overlapping  IntervalIndex.get_loc  IntervalIndex.get_indexer  IntervalIndex.set_closed  IntervalIndex.contains  IntervalIndex.overlaps  IntervalIndex.to_tuples  .. _api.multiindex:

MultiIndex
.. autosummary::  api/  IndexSlice  .. autosummary::  api/  autosummary/class_without_autosummary.rst  MultiIndex  MultiIndex constructors
~~~
.. autosummary::  api/  MultiIndex.from_arrays  MultiIndex.from_tuples  MultiIndex.from_product  MultiIndex.from_frame  MultiIndex properties
~~~
.. autosummary::  api/  MultiIndex.names  MultiIndex.levels  MultiIndex.codes  MultiIndex.nlevels  MultiIndex.levshape  MultiIndex.dtypes  MultiIndex components
~~~
.. autosummary::  api/  MultiIndex.set_levels  MultiIndex.set_codes  MultiIndex.to_flat_index  MultiIndex.to_frame  MultiIndex.sortlevel  MultiIndex.droplevel  MultiIndex.swaplevel  MultiIndex.reorder_levels  MultiIndex.remove_unused_levels  MultiIndex.drop  MultiIndex.copy  MultiIndex.append  MultiIndex.truncate  MultiIndex selecting
~~~
.. autosummary::  api/  MultiIndex.get_loc  MultiIndex.get_locs  MultiIndex.get_loc_level  MultiIndex.get_indexer  MultiIndex.get_level_values  .. _api.datetimeindex:

DatetimeIndex
.. autosummary::  api/  autosummary/class_without_autosummary.rst  DatetimeIndex  Time/date components
~~~
.. autosummary::  api/  DatetimeIndex.year  DatetimeIndex.month  DatetimeIndex.day  DatetimeIndex.hour  DatetimeIndex.minute  DatetimeIndex.second  DatetimeIndex.microsecond  DatetimeIndex.nanosecond  DatetimeIndex.date  DatetimeIndex.time  DatetimeIndex.timetz  DatetimeIndex.dayofyear  DatetimeIndex.day_of_year  DatetimeIndex.dayofweek  DatetimeIndex.day_of_week  DatetimeIndex.weekday  DatetimeIndex.quarter  DatetimeIndex.tz  DatetimeIndex.freq  DatetimeIndex.freqstr  DatetimeIndex.is_month_start  DatetimeIndex.is_month_end  DatetimeIndex.is_quarter_start  DatetimeIndex.is_quarter_end  DatetimeIndex.is_year_start  DatetimeIndex.is_year_end  DatetimeIndex.is_leap_year  DatetimeIndex.inferred_freq  Selecting
~~~
.. autosummary::  api/  DatetimeIndex.indexer_at_time  DatetimeIndex.indexer_between_time  Time-specific operations
~~~
.. autosummary::  api/  DatetimeIndex.normalize  DatetimeIndex.strftime  DatetimeIndex.snap  DatetimeIndex.tz_convert  DatetimeIndex.tz_localize  DatetimeIndex.round  DatetimeIndex.floor  DatetimeIndex.ceil  DatetimeIndex.month_name  DatetimeIndex.day_name  Conversion
~~~
.. autosummary::  api/  DatetimeIndex.as_unit  DatetimeIndex.to_period  DatetimeIndex.to_pydatetime  DatetimeIndex.to_series  DatetimeIndex.to_frame  Methods
~~~
.. autosummary::  api/  DatetimeIndex.mean  DatetimeIndex.std

TimedeltaIndex
.. autosummary::  api/  autosummary/class_without_autosummary.rst  TimedeltaIndex  Components
~~~
.. autosummary::  api/  TimedeltaIndex.days  TimedeltaIndex.seconds  TimedeltaIndex.microseconds  TimedeltaIndex.nanoseconds  TimedeltaIndex.components  TimedeltaIndex.inferred_freq  Conversion
~~~
.. autosummary::  api/  TimedeltaIndex.as_unit  TimedeltaIndex.to_pytimedelta  TimedeltaIndex.to_series  TimedeltaIndex.round  TimedeltaIndex.floor  TimedeltaIndex.ceil  TimedeltaIndex.to_frame  Methods
~~~
.. autosummary::  api/  TimedeltaIndex.mean  .. currentmodule:: pandas

PeriodIndex
.. autosummary::  api/  autosummary/class_without_autosummary.rst  PeriodIndex  Properties
~~~
.. autosummary::  api/  PeriodIndex.day  PeriodIndex.dayofweek  PeriodIndex.day_of_week  PeriodIndex.dayofyear  PeriodIndex.day_of_year  PeriodIndex.days_in_month  PeriodIndex.daysinmonth  PeriodIndex.end_time  PeriodIndex.freq  PeriodIndex.freqstr  PeriodIndex.hour  PeriodIndex.is_leap_year  PeriodIndex.minute  PeriodIndex.month  PeriodIndex.quarter  PeriodIndex.qyear  PeriodIndex.second  PeriodIndex.start_time  PeriodIndex.week  PeriodIndex.weekday  PeriodIndex.weekofyear  PeriodIndex.year  Methods
~~~
.. autosummary::  api/  PeriodIndex.asfreq  PeriodIndex.strftime  PeriodIndex.to_timestamp

Input/output
.. currentmodule:: pandas  Pickling
~~~
.. autosummary::  api/  read_pickle  DataFrame.to_pickle  Flat file
~~~
.. autosummary::  api/  read_table  read_csv  DataFrame.to_csv  read_fwf  Clipboard
~~~
.. autosummary::  api/  read_clipboard  DataFrame.to_clipboard  Excel
~~~
.. autosummary::  api/  read_excel  DataFrame.to_excel  ExcelFile  ExcelFile.book  ExcelFile.sheet_names  ExcelFile.parse  .. currentmodule:: pandas.io.formats.style  .. autosummary::  api/  Styler.to_excel  .. currentmodule:: pandas  .. autosummary::  api/  ExcelWriter  .. currentmodule:: pandas  JSON
~~~
.. autosummary::  api/  read_json  json_normalize  DataFrame.to_json  .. currentmodule:: pandas.io.json  .. autosummary::  api/  build_table_schema  .. currentmodule:: pandas  HTML
~~~
.. autosummary::  api/  read_html  DataFrame.to_html  .. currentmodule:: pandas.io.formats.style  .. autosummary::  api/  Styler.to_html  .. currentmodule:: pandas  XML
~~~
.. autosummary::  api/  read_xml  DataFrame.to_xml  Latex
~~~
.. autosummary::  api/  DataFrame.to_latex  .. currentmodule:: pandas.io.formats.style  .. autosummary::  api/  Styler.to_latex  .. currentmodule:: pandas  HDFStore: PyTables (HDF5)
~~~
.. autosummary::  api/  read_hdf  HDFStore.put  HDFStore.append  HDFStore.get  HDFStore.select  HDFStore.info  HDFStore.keys  HDFStore.groups  HDFStore.walk  .. warning::  One can store a subclass of `DataFrame` or `Series` to HDF5,  but the type of the subclass is lost upon storing.  Feather
~~~
.. autosummary::  api/  read_feather  DataFrame.to_feather  Parquet
~~~
.. autosummary::  api/  read_parquet  DataFrame.to_parquet  ORC
~~~
.. autosummary::  api/  read_orc  DataFrame.to_orc  SAS
~~~
.. autosummary::  api/  read_sas  SPSS
~~~
.. autosummary::  api/  read_spss  SQL
~~~
.. autosummary::  api/  read_sql_table  read_sql_query  read_sql  DataFrame.to_sql  Google BigQuery
~~~
.. autosummary::  api/  read_gbq  STATA
~~~
.. autosummary::  api/  read_stata  DataFrame.to_stata  .. currentmodule:: pandas.io.stata  .. autosummary::  api/  StataReader.data_label  StataReader.value_labels  StataReader.variable_labels  StataWriter.write_file

DateOffset
.. autosummary::  api/  DateOffset  Properties
~~~
.. autosummary::  api/  DateOffset.freqstr  DateOffset.kwds  DateOffset.name  DateOffset.nanos  DateOffset.normalize  DateOffset.rule_code  DateOffset.n  DateOffset.is_month_start  DateOffset.is_month_end  Methods
~~~
.. autosummary::  api/  DateOffset.copy  DateOffset.is_anchored  DateOffset.is_on_offset  DateOffset.is_month_start  DateOffset.is_month_end  DateOffset.is_quarter_start  DateOffset.is_quarter_end  DateOffset.is_year_start  DateOffset.is_year_end

BusinessDay

.. autosummary::  api/  BusinessDay  Alias:  .. autosummary::  api/  autosummary/class_without_autosummary.rst  BDay  Properties
~~~
.. autosummary::  api/  BusinessDay.freqstr  BusinessDay.kwds  BusinessDay.name  BusinessDay.nanos  BusinessDay.normalize  BusinessDay.rule_code  BusinessDay.n  BusinessDay.weekmask  BusinessDay.holidays  BusinessDay.calendar  Methods
~~~
.. autosummary::  api/  BusinessDay.copy  BusinessDay.is_anchored  BusinessDay.is_on_offset  BusinessDay.is_month_start  BusinessDay.is_month_end  BusinessDay.is_quarter_start  BusinessDay.is_quarter_end  BusinessDay.is_year_start  BusinessDay.is_year_end

BusinessHour
.. autosummary::  api/  BusinessHour  Properties
~~~
.. autosummary::  api/  BusinessHour.freqstr  BusinessHour.kwds  BusinessHour.name  BusinessHour.nanos  BusinessHour.normalize  BusinessHour.rule_code  BusinessHour.n  BusinessHour.start  BusinessHour.end  BusinessHour.weekmask  BusinessHour.holidays  BusinessHour.calendar  Methods
~~~
.. autosummary::  api/  BusinessHour.copy  BusinessHour.is_anchored  BusinessHour.is_on_offset  BusinessHour.is_month_start  BusinessHour.is_month_end  BusinessHour.is_quarter_start  BusinessHour.is_quarter_end  BusinessHour.is_year_start  BusinessHour.is_year_end

CustomBusinessDay

.. autosummary::  api/  CustomBusinessDay  Alias:  .. autosummary::  api/  autosummary/class_without_autosummary.rst  CDay  Properties
~~~
.. autosummary::  api/  CustomBusinessDay.freqstr  CustomBusinessDay.kwds  CustomBusinessDay.name  CustomBusinessDay.nanos  CustomBusinessDay.normalize  CustomBusinessDay.rule_code  CustomBusinessDay.n  CustomBusinessDay.weekmask  CustomBusinessDay.calendar  CustomBusinessDay.holidays  Methods
~~~
.. autosummary::  api/  CustomBusinessDay.copy  CustomBusinessDay.is_anchored  CustomBusinessDay.is_on_offset  CustomBusinessDay.is_month_start  CustomBusinessDay.is_month_end  CustomBusinessDay.is_quarter_start  CustomBusinessDay.is_quarter_end  CustomBusinessDay.is_year_start  CustomBusinessDay.is_year_end

CustomBusinessHour
.. autosummary::  api/  CustomBusinessHour  Properties
~~~
.. autosummary::  api/  CustomBusinessHour.freqstr  CustomBusinessHour.kwds  CustomBusinessHour.name  CustomBusinessHour.nanos  CustomBusinessHour.normalize  CustomBusinessHour.rule_code  CustomBusinessHour.n  CustomBusinessHour.weekmask  CustomBusinessHour.calendar  CustomBusinessHour.holidays  CustomBusinessHour.start  CustomBusinessHour.end  Methods
~~~
.. autosummary::  api/  CustomBusinessHour.copy  CustomBusinessHour.is_anchored  CustomBusinessHour.is_on_offset  CustomBusinessHour.is_month_start  CustomBusinessHour.is_month_end  CustomBusinessHour.is_quarter_start  CustomBusinessHour.is_quarter_end  CustomBusinessHour.is_year_start  CustomBusinessHour.is_year_end

MonthEnd
.. autosummary::  api/  MonthEnd  Properties
~~~
.. autosummary::  api/  MonthEnd.freqstr  MonthEnd.kwds  MonthEnd.name  MonthEnd.nanos  MonthEnd.normalize  MonthEnd.rule_code  MonthEnd.n  Methods
~~~
.. autosummary::  api/  MonthEnd.copy  MonthEnd.is_anchored  MonthEnd.is_on_offset  MonthEnd.is_month_start  MonthEnd.is_month_end  MonthEnd.is_quarter_start  MonthEnd.is_quarter_end  MonthEnd.is_year_start  MonthEnd.is_year_end

MonthBegin
.. autosummary::  api/  MonthBegin  Properties
~~~
.. autosummary::  api/  MonthBegin.freqstr  MonthBegin.kwds  MonthBegin.name  MonthBegin.nanos  MonthBegin.normalize  MonthBegin.rule_code  MonthBegin.n  Methods
~~~
.. autosummary::  api/  MonthBegin.copy  MonthBegin.is_anchored  MonthBegin.is_on_offset  MonthBegin.is_month_start  MonthBegin.is_month_end  MonthBegin.is_quarter_start  MonthBegin.is_quarter_end  MonthBegin.is_year_start  MonthBegin.is_year_end

BusinessMonthEnd

.. autosummary::  api/  BusinessMonthEnd  Alias:  .. autosummary::  api/  autosummary/class_without_autosummary.rst  BMonthEnd  Properties
~~~
.. autosummary::  api/  BusinessMonthEnd.freqstr  BusinessMonthEnd.kwds  BusinessMonthEnd.name  BusinessMonthEnd.nanos  BusinessMonthEnd.normalize  BusinessMonthEnd.rule_code  BusinessMonthEnd.n  Methods
~~~
.. autosummary::  api/  BusinessMonthEnd.copy  BusinessMonthEnd.is_anchored  BusinessMonthEnd.is_on_offset  BusinessMonthEnd.is_month_start  BusinessMonthEnd.is_month_end  BusinessMonthEnd.is_quarter_start  BusinessMonthEnd.is_quarter_end  BusinessMonthEnd.is_year_start  BusinessMonthEnd.is_year_end

BusinessMonthBegin

.. autosummary::  api/  BusinessMonthBegin  Alias:  .. autosummary::  api/  autosummary/class_without_autosummary.rst  BMonthBegin  Properties
~~~
.. autosummary::  api/  BusinessMonthBegin.freqstr  BusinessMonthBegin.kwds  BusinessMonthBegin.name  BusinessMonthBegin.nanos  BusinessMonthBegin.normalize  BusinessMonthBegin.rule_code  BusinessMonthBegin.n  Methods
~~~
.. autosummary::  api/  BusinessMonthBegin.copy  BusinessMonthBegin.is_anchored  BusinessMonthBegin.is_on_offset  BusinessMonthBegin.is_month_start  BusinessMonthBegin.is_month_end  BusinessMonthBegin.is_quarter_start  BusinessMonthBegin.is_quarter_end  BusinessMonthBegin.is_year_start  BusinessMonthBegin.is_year_end

CustomBusinessMonthEnd

.. autosummary::  api/  CustomBusinessMonthEnd  Alias:  .. autosummary::  api/  autosummary/class_without_autosummary.rst  CBMonthEnd  Properties
~~~
.. autosummary::  api/  CustomBusinessMonthEnd.freqstr  CustomBusinessMonthEnd.kwds  CustomBusinessMonthEnd.m_offset  CustomBusinessMonthEnd.name  CustomBusinessMonthEnd.nanos  CustomBusinessMonthEnd.normalize  CustomBusinessMonthEnd.rule_code  CustomBusinessMonthEnd.n  CustomBusinessMonthEnd.weekmask  CustomBusinessMonthEnd.calendar  CustomBusinessMonthEnd.holidays  Methods
~~~
.. autosummary::  api/  CustomBusinessMonthEnd.copy  CustomBusinessMonthEnd.is_anchored  CustomBusinessMonthEnd.is_on_offset  CustomBusinessMonthEnd.is_month_start  CustomBusinessMonthEnd.is_month_end  CustomBusinessMonthEnd.is_quarter_start  CustomBusinessMonthEnd.is_quarter_end  CustomBusinessMonthEnd.is_year_start  CustomBusinessMonthEnd.is_year_end

CustomBusinessMonthBegin

.. autosummary::  api/  CustomBusinessMonthBegin  Alias:  .. autosummary::  api/  autosummary/class_without_autosummary.rst  CBMonthBegin  Properties
~~~
.. autosummary::  api/  CustomBusinessMonthBegin.freqstr  CustomBusinessMonthBegin.kwds  CustomBusinessMonthBegin.m_offset  CustomBusinessMonthBegin.name  CustomBusinessMonthBegin.nanos  CustomBusinessMonthBegin.normalize  CustomBusinessMonthBegin.rule_code  CustomBusinessMonthBegin.n  CustomBusinessMonthBegin.weekmask  CustomBusinessMonthBegin.calendar  CustomBusinessMonthBegin.holidays  Methods
~~~
.. autosummary::  api/  CustomBusinessMonthBegin.copy  CustomBusinessMonthBegin.is_anchored  CustomBusinessMonthBegin.is_on_offset  CustomBusinessMonthBegin.is_month_start  CustomBusinessMonthBegin.is_month_end  CustomBusinessMonthBegin.is_quarter_start  CustomBusinessMonthBegin.is_quarter_end  CustomBusinessMonthBegin.is_year_start  CustomBusinessMonthBegin.is_year_end

SemiMonthEnd
.. autosummary::  api/  SemiMonthEnd  Properties
~~~
.. autosummary::  api/  SemiMonthEnd.freqstr  SemiMonthEnd.kwds  SemiMonthEnd.name  SemiMonthEnd.nanos  SemiMonthEnd.normalize  SemiMonthEnd.rule_code  SemiMonthEnd.n  SemiMonthEnd.day_of_month  Methods
~~~
.. autosummary::  api/  SemiMonthEnd.copy  SemiMonthEnd.is_anchored  SemiMonthEnd.is_on_offset  SemiMonthEnd.is_month_start  SemiMonthEnd.is_month_end  SemiMonthEnd.is_quarter_start  SemiMonthEnd.is_quarter_end  SemiMonthEnd.is_year_start  SemiMonthEnd.is_year_end

SemiMonthBegin
.. autosummary::  api/  SemiMonthBegin  Properties
~~~
.. autosummary::  api/  SemiMonthBegin.freqstr  SemiMonthBegin.kwds  SemiMonthBegin.name  SemiMonthBegin.nanos  SemiMonthBegin.normalize  SemiMonthBegin.rule_code  SemiMonthBegin.n  SemiMonthBegin.day_of_month  Methods
~~~
.. autosummary::  api/  SemiMonthBegin.copy  SemiMonthBegin.is_anchored  SemiMonthBegin.is_on_offset  SemiMonthBegin.is_month_start  SemiMonthBegin.is_month_end  SemiMonthBegin.is_quarter_start  SemiMonthBegin.is_quarter_end  SemiMonthBegin.is_year_start  SemiMonthBegin.is_year_end

Week
.. autosummary::  api/  Week  Properties
~~~
.. autosummary::  api/  Week.freqstr  Week.kwds  Week.name  Week.nanos  Week.normalize  Week.rule_code  Week.n  Week.weekday  Methods
~~~
.. autosummary::  api/  Week.copy  Week.is_anchored  Week.is_on_offset  Week.is_month_start  Week.is_month_end  Week.is_quarter_start  Week.is_quarter_end  Week.is_year_start  Week.is_year_end

WeekOfMonth
.. autosummary::  api/  WeekOfMonth  Properties
~~~
.. autosummary::  api/  WeekOfMonth.freqstr  WeekOfMonth.kwds  WeekOfMonth.name  WeekOfMonth.nanos  WeekOfMonth.normalize  WeekOfMonth.rule_code  WeekOfMonth.n  WeekOfMonth.week  Methods
~~~
.. autosummary::  api/  WeekOfMonth.copy  WeekOfMonth.is_anchored  WeekOfMonth.is_on_offset  WeekOfMonth.weekday  WeekOfMonth.is_month_start  WeekOfMonth.is_month_end  WeekOfMonth.is_quarter_start  WeekOfMonth.is_quarter_end  WeekOfMonth.is_year_start  WeekOfMonth.is_year_end

LastWeekOfMonth
.. autosummary::  api/  LastWeekOfMonth  Properties
~~~
.. autosummary::  api/  LastWeekOfMonth.freqstr  LastWeekOfMonth.kwds  LastWeekOfMonth.name  LastWeekOfMonth.nanos  LastWeekOfMonth.normalize  LastWeekOfMonth.rule_code  LastWeekOfMonth.n  LastWeekOfMonth.weekday  LastWeekOfMonth.week  Methods
~~~
.. autosummary::  api/  LastWeekOfMonth.copy  LastWeekOfMonth.is_anchored  LastWeekOfMonth.is_on_offset  LastWeekOfMonth.is_month_start  LastWeekOfMonth.is_month_end  LastWeekOfMonth.is_quarter_start  LastWeekOfMonth.is_quarter_end  LastWeekOfMonth.is_year_start  LastWeekOfMonth.is_year_end

BQuarterEnd
.. autosummary::  api/  BQuarterEnd  Properties
~~~
.. autosummary::  api/  BQuarterEnd.freqstr  BQuarterEnd.kwds  BQuarterEnd.name  BQuarterEnd.nanos  BQuarterEnd.normalize  BQuarterEnd.rule_code  BQuarterEnd.n  BQuarterEnd.startingMonth  Methods
~~~
.. autosummary::  api/  BQuarterEnd.copy  BQuarterEnd.is_anchored  BQuarterEnd.is_on_offset  BQuarterEnd.is_month_start  BQuarterEnd.is_month_end  BQuarterEnd.is_quarter_start  BQuarterEnd.is_quarter_end  BQuarterEnd.is_year_start  BQuarterEnd.is_year_end

BQuarterBegin
.. autosummary::  api/  BQuarterBegin  Properties
~~~
.. autosummary::  api/  BQuarterBegin.freqstr  BQuarterBegin.kwds  BQuarterBegin.name  BQuarterBegin.nanos  BQuarterBegin.normalize  BQuarterBegin.rule_code  BQuarterBegin.n  BQuarterBegin.startingMonth  Methods
~~~
.. autosummary::  api/  BQuarterBegin.copy  BQuarterBegin.is_anchored  BQuarterBegin.is_on_offset  BQuarterBegin.is_month_start  BQuarterBegin.is_month_end  BQuarterBegin.is_quarter_start  BQuarterBegin.is_quarter_end  BQuarterBegin.is_year_start  BQuarterBegin.is_year_end

QuarterEnd
.. autosummary::  api/  QuarterEnd  Properties
~~~
.. autosummary::  api/  QuarterEnd.freqstr  QuarterEnd.kwds  QuarterEnd.name  QuarterEnd.nanos  QuarterEnd.normalize  QuarterEnd.rule_code  QuarterEnd.n  QuarterEnd.startingMonth  Methods
~~~
.. autosummary::  api/  QuarterEnd.copy  QuarterEnd.is_anchored  QuarterEnd.is_on_offset  QuarterEnd.is_month_start  QuarterEnd.is_month_end  QuarterEnd.is_quarter_start  QuarterEnd.is_quarter_end  QuarterEnd.is_year_start  QuarterEnd.is_year_end

QuarterBegin
.. autosummary::  api/  QuarterBegin  Properties
~~~
.. autosummary::  api/  QuarterBegin.freqstr  QuarterBegin.kwds  QuarterBegin.name  QuarterBegin.nanos  QuarterBegin.normalize  QuarterBegin.rule_code  QuarterBegin.n  QuarterBegin.startingMonth  Methods
~~~
.. autosummary::  api/  QuarterBegin.copy  QuarterBegin.is_anchored  QuarterBegin.is_on_offset  QuarterBegin.is_month_start  QuarterBegin.is_month_end  QuarterBegin.is_quarter_start  QuarterBegin.is_quarter_end  QuarterBegin.is_year_start  QuarterBegin.is_year_end

BYearEnd
.. autosummary::  api/  BYearEnd  Properties
~~~
.. autosummary::  api/  BYearEnd.freqstr  BYearEnd.kwds  BYearEnd.name  BYearEnd.nanos  BYearEnd.normalize  BYearEnd.rule_code  BYearEnd.n  BYearEnd.month  Methods
~~~
.. autosummary::  api/  BYearEnd.copy  BYearEnd.is_anchored  BYearEnd.is_on_offset  BYearEnd.is_month_start  BYearEnd.is_month_end  BYearEnd.is_quarter_start  BYearEnd.is_quarter_end  BYearEnd.is_year_start  BYearEnd.is_year_end

BYearBegin
.. autosummary::  api/  BYearBegin  Properties
~~~
.. autosummary::  api/  BYearBegin.freqstr  BYearBegin.kwds  BYearBegin.name  BYearBegin.nanos  BYearBegin.normalize  BYearBegin.rule_code  BYearBegin.n  BYearBegin.month  Methods
~~~
.. autosummary::  api/  BYearBegin.copy  BYearBegin.is_anchored  BYearBegin.is_on_offset  BYearBegin.is_month_start  BYearBegin.is_month_end  BYearBegin.is_quarter_start  BYearBegin.is_quarter_end  BYearBegin.is_year_start  BYearBegin.is_year_end

YearEnd
.. autosummary::  api/  YearEnd  Properties
~~~
.. autosummary::  api/  YearEnd.freqstr  YearEnd.kwds  YearEnd.name  YearEnd.nanos  YearEnd.normalize  YearEnd.rule_code  YearEnd.n  YearEnd.month  Methods
~~~
.. autosummary::  api/  YearEnd.copy  YearEnd.is_anchored  YearEnd.is_on_offset  YearEnd.is_month_start  YearEnd.is_month_end  YearEnd.is_quarter_start  YearEnd.is_quarter_end  YearEnd.is_year_start  YearEnd.is_year_end

YearBegin
.. autosummary::  api/  YearBegin  Properties
~~~
.. autosummary::  api/  YearBegin.freqstr  YearBegin.kwds  YearBegin.name  YearBegin.nanos  YearBegin.normalize  YearBegin.rule_code  YearBegin.n  YearBegin.month  Methods
~~~
.. autosummary::  api/  YearBegin.copy  YearBegin.is_anchored  YearBegin.is_on_offset  YearBegin.is_month_start  YearBegin.is_month_end  YearBegin.is_quarter_start  YearBegin.is_quarter_end  YearBegin.is_year_start  YearBegin.is_year_end

FY5253
.. autosummary::  api/  FY5253  Properties
~~~
.. autosummary::  api/  FY5253.freqstr  FY5253.kwds  FY5253.name  FY5253.nanos  FY5253.normalize  FY5253.rule_code  FY5253.n  FY5253.startingMonth  FY5253.variation  FY5253.weekday  Methods
~~~
.. autosummary::  api/  FY5253.copy  FY5253.get_rule_code_suffix  FY5253.get_year_end  FY5253.is_anchored  FY5253.is_on_offset  FY5253.is_month_start  FY5253.is_month_end  FY5253.is_quarter_start  FY5253.is_quarter_end  FY5253.is_year_start  FY5253.is_year_end

FY5253Quarter
.. autosummary::  api/  FY5253Quarter  Properties
~~~
.. autosummary::  api/  FY5253Quarter.freqstr  FY5253Quarter.kwds  FY5253Quarter.name  FY5253Quarter.nanos  FY5253Quarter.normalize  FY5253Quarter.rule_code  FY5253Quarter.n  FY5253Quarter.qtr_with_extra_week  FY5253Quarter.startingMonth  FY5253Quarter.variation  FY5253Quarter.weekday  Methods
~~~
.. autosummary::  api/  FY5253Quarter.copy  FY5253Quarter.get_rule_code_suffix  FY5253Quarter.get_weeks  FY5253Quarter.is_anchored  FY5253Quarter.is_on_offset  FY5253Quarter.year_has_extra_week  FY5253Quarter.is_month_start  FY5253Quarter.is_month_end  FY5253Quarter.is_quarter_start  FY5253Quarter.is_quarter_end  FY5253Quarter.is_year_start  FY5253Quarter.is_year_end

Easter
.. autosummary::  api/  Easter  Properties
~~~
.. autosummary::  api/  Easter.freqstr  Easter.kwds  Easter.name  Easter.nanos  Easter.normalize  Easter.rule_code  Easter.n  Methods
~~~
.. autosummary::  api/  Easter.copy  Easter.is_anchored  Easter.is_on_offset  Easter.is_month_start  Easter.is_month_end  Easter.is_quarter_start  Easter.is_quarter_end  Easter.is_year_start  Easter.is_year_end

Tick
.. autosummary::  api/  Tick  Properties
~~~
.. autosummary::  api/  Tick.delta  Tick.freqstr  Tick.kwds  Tick.name  Tick.nanos  Tick.normalize  Tick.rule_code  Tick.n  Methods
~~~
.. autosummary::  api/  Tick.copy  Tick.is_anchored  Tick.is_on_offset  Tick.is_month_start  Tick.is_month_end  Tick.is_quarter_start  Tick.is_quarter_end  Tick.is_year_start  Tick.is_year_end

Day
.. autosummary::  api/  Day  Properties
~~~
.. autosummary::  api/  Day.delta  Day.freqstr  Day.kwds  Day.name  Day.nanos  Day.normalize  Day.rule_code  Day.n  Methods
~~~
.. autosummary::  api/  Day.copy  Day.is_anchored  Day.is_on_offset  Day.is_month_start  Day.is_month_end  Day.is_quarter_start  Day.is_quarter_end  Day.is_year_start  Day.is_year_end

Hour
.. autosummary::  api/  Hour  Properties
~~~
.. autosummary::  api/  Hour.delta  Hour.freqstr  Hour.kwds  Hour.name  Hour.nanos  Hour.normalize  Hour.rule_code  Hour.n  Methods
~~~
.. autosummary::  api/  Hour.copy  Hour.is_anchored  Hour.is_on_offset  Hour.is_month_start  Hour.is_month_end  Hour.is_quarter_start  Hour.is_quarter_end  Hour.is_year_start  Hour.is_year_end

Minute
.. autosummary::  api/  Minute  Properties
~~~
.. autosummary::  api/  Minute.delta  Minute.freqstr  Minute.kwds  Minute.name  Minute.nanos  Minute.normalize  Minute.rule_code  Minute.n  Methods
~~~
.. autosummary::  api/  Minute.copy  Minute.is_anchored  Minute.is_on_offset  Minute.is_month_start  Minute.is_month_end  Minute.is_quarter_start  Minute.is_quarter_end  Minute.is_year_start  Minute.is_year_end

Second
.. autosummary::  api/  Second  Properties
~~~
.. autosummary::  api/  Second.delta  Second.freqstr  Second.kwds  Second.name  Second.nanos  Second.normalize  Second.rule_code  Second.n  Methods
~~~
.. autosummary::  api/  Second.copy  Second.is_anchored  Second.is_on_offset  Second.is_month_start  Second.is_month_end  Second.is_quarter_start  Second.is_quarter_end  Second.is_year_start  Second.is_year_end

Milli
.. autosummary::  api/  Milli  Properties
~~~
.. autosummary::  api/  Milli.delta  Milli.freqstr  Milli.kwds  Milli.name  Milli.nanos  Milli.normalize  Milli.rule_code  Milli.n  Methods
~~~
.. autosummary::  api/  Milli.copy  Milli.is_anchored  Milli.is_on_offset  Milli.is_month_start  Milli.is_month_end  Milli.is_quarter_start  Milli.is_quarter_end  Milli.is_year_start  Milli.is_year_end

Micro
.. autosummary::  api/  Micro  Properties
~~~
.. autosummary::  api/  Micro.delta  Micro.freqstr  Micro.kwds  Micro.name  Micro.nanos  Micro.normalize  Micro.rule_code  Micro.n  Methods
~~~
.. autosummary::  api/  Micro.copy  Micro.is_anchored  Micro.is_on_offset  Micro.is_month_start  Micro.is_month_end  Micro.is_quarter_start  Micro.is_quarter_end  Micro.is_year_start  Micro.is_year_end

Nano
.. autosummary::  api/  Nano  Properties
~~~
.. autosummary::  api/  Nano.delta  Nano.freqstr  Nano.kwds  Nano.name  Nano.nanos  Nano.normalize  Nano.rule_code  Nano.n  Methods
~~~
.. autosummary::  api/  Nano.copy  Nano.is_anchored  Nano.is_on_offset  Nano.is_month_start  Nano.is_month_end  Nano.is_quarter_start  Nano.is_quarter_end  Nano.is_year_start  Nano.is_year_end  .. _api.frequencies:

Frequencies
.. currentmodule:: pandas.tseries.frequencies  .. _api.offsets:  .. autosummary::  api/  to_offset

Options and settings
.. currentmodule:: pandas  API for configuring global behavior. See `the User Guide <options>` for more.

Working with options
.. autosummary::  api/  describe_option  reset_option  get_option  set_option  option_context

Plotting
.. currentmodule:: pandas.plotting  The following functions are contained in the ``pandas.plotting`` module.  .. autosummary::  api/  andrews_curves  autocorrelation_plot  bootstrap_plot  boxplot  deregister_matplotlib_converters  lag_plot  parallel_coordinates  plot_params  radviz  register_matplotlib_converters  scatter_matrix  table

Resampling
.. currentmodule:: pandas.core.resample  Resampler objects are returned by resample calls: `pandas.DataFrame.resample`, `pandas.Series.resample`.  Indexing, iteration
~~~
.. autosummary::  api/  Resampler.__iter__  Resampler.groups  Resampler.indices  Resampler.get_group  Function application
~~~
.. autosummary::  api/  Resampler.apply  Resampler.aggregate  Resampler.transform  Resampler.pipe  Upsampling
~~~
.. autosummary::  api/  Resampler.ffill  Resampler.bfill  Resampler.nearest  Resampler.fillna  Resampler.asfreq  Resampler.interpolate  Computations / descriptive stats
~~~
.. autosummary::  api/  Resampler.count  Resampler.nunique  Resampler.first  Resampler.last  Resampler.max  Resampler.mean  Resampler.median  Resampler.min  Resampler.ohlc  Resampler.prod  Resampler.size  Resampler.sem  Resampler.std  Resampler.sum  Resampler.var  Resampler.quantile

Attributes
**Axes**  .. autosummary::  api/  Series.index  Series.array  Series.values  Series.dtype  Series.shape  Series.nbytes  Series.ndim  Series.size  Series.T  Series.memory_usage  Series.hasnans  Series.empty  Series.dtypes  Series.name  Series.flags  Series.set_flags

Conversion
.. autosummary::  api/  Series.astype  Series.convert_dtypes  Series.infer_objects  Series.copy  Series.bool  Series.to_numpy  Series.to_period  Series.to_timestamp  Series.to_list  Series.__array__

.. autosummary::  api/  Series.get  Series.at  Series.iat  Series.loc  Series.iloc  Series.__iter__  Series.items  Series.keys  Series.pop  Series.item  Series.xs  For more information on ``.at``, ``.iat``, ``.loc``, and
``.iloc``,  see the `indexing documentation <indexing>`.

Binary operator functions
.. autosummary::  api/  Series.add  Series.sub  Series.mul  Series.div  Series.truediv  Series.floordiv  Series.mod  Series.pow  Series.radd  Series.rsub  Series.rmul  Series.rdiv  Series.rtruediv  Series.rfloordiv  Series.rmod  Series.rpow  Series.combine  Series.combine_first  Series.round  Series.lt  Series.gt  Series.le  Series.ge  Series.ne  Series.eq  Series.product  Series.dot

Function application, GroupBy & window
.. autosummary::  api/  Series.apply  Series.agg  Series.aggregate  Series.transform  Series.map  Series.groupby  Series.rolling  Series.expanding  Series.ewm  Series.pipe  .. _api.series.stats:

Computations / descriptive stats
.. autosummary::  api/  Series.abs  Series.all  Series.any  Series.autocorr  Series.between  Series.clip  Series.corr  Series.count  Series.cov  Series.cummax  Series.cummin  Series.cumprod  Series.cumsum  Series.describe  Series.diff  Series.factorize  Series.kurt  Series.max  Series.mean  Series.median  Series.min  Series.mode  Series.nlargest  Series.nsmallest  Series.pct_change  Series.prod  Series.quantile  Series.rank  Series.sem  Series.skew  Series.std  Series.sum  Series.var  Series.kurtosis  Series.unique  Series.nunique  Series.is_unique  Series.is_monotonic_increasing  Series.is_monotonic_decreasing  Series.value_counts

Reindexing / selection / label manipulation
.. autosummary::  api/  Series.align  Series.drop  Series.droplevel  Series.drop_duplicates  Series.duplicated  Series.equals  Series.first  Series.head  Series.idxmax  Series.idxmin  Series.isin  Series.last  Series.reindex  Series.reindex_like  Series.rename  Series.rename_axis  Series.reset_index  Series.sample  Series.set_axis  Series.take  Series.tail  Series.truncate  Series.where  Series.mask  Series.add_prefix  Series.add_suffix  Series.filter

Missing data handling
.. autosummary::  api/  Series.backfill  Series.bfill  Series.dropna  Series.ffill  Series.fillna  Series.interpolate  Series.isna  Series.isnull  Series.notna  Series.notnull  Series.pad  Series.replace

Reshaping, sorting
.. autosummary::  api/  Series.argsort  Series.argmin  Series.argmax  Series.reorder_levels  Series.sort_values  Series.sort_index  Series.swaplevel  Series.unstack  Series.explode  Series.searchsorted  Series.ravel  Series.repeat  Series.squeeze  Series.view

Combining / comparing / joining / merging
.. autosummary::  api/  Series.compare  Series.update

Time Series-related
.. autosummary::  api/  Series.asfreq  Series.asof  Series.shift  Series.first_valid_index  Series.last_valid_index  Series.resample  Series.tz_convert  Series.tz_localize  Series.at_time  Series.between_time

pandas provides dtype-specific methods under various accessors.
These are separate namespaces within `Series` that only apply
to specific data types.  === ===
Data Type  Accessor
=== ===
Datetime, Timedelta, Period `dt <api.series.dt>`
String  `str <api.series.str>`
Categorical  `cat <api.series.cat>`
Sparse  `sparse <api.series.sparse>`
=== ===  .. _api.series.dt:  Datetimelike properties
~~~  ``Series.dt`` can be used to access the values of the series as
datetimelike and return several properties.
These can be accessed like ``Series.dt.<property>``.  Datetime properties
^^^  .. autosummary::  api/  autosummary/accessor_attribute.rst  Series.dt.date  Series.dt.time  Series.dt.timetz  Series.dt.year  Series.dt.month  Series.dt.day  Series.dt.hour  Series.dt.minute  Series.dt.second  Series.dt.microsecond  Series.dt.nanosecond  Series.dt.dayofweek  Series.dt.day_of_week  Series.dt.weekday  Series.dt.dayofyear  Series.dt.day_of_year  Series.dt.quarter  Series.dt.is_month_start  Series.dt.is_month_end  Series.dt.is_quarter_start  Series.dt.is_quarter_end  Series.dt.is_year_start  Series.dt.is_year_end  Series.dt.is_leap_year  Series.dt.daysinmonth  Series.dt.days_in_month  Series.dt.tz  Series.dt.freq  Series.dt.unit  Datetime methods
^^^  .. autosummary::  api/  autosummary/accessor_method.rst  Series.dt.isocalendar  Series.dt.to_period  Series.dt.to_pydatetime  Series.dt.tz_localize  Series.dt.tz_convert  Series.dt.normalize  Series.dt.strftime  Series.dt.round  Series.dt.floor  Series.dt.ceil  Series.dt.month_name  Series.dt.day_name  Series.dt.as_unit  Period properties
^^^  .. autosummary::  api/  autosummary/accessor_attribute.rst  Series.dt.qyear  Series.dt.start_time  Series.dt.end_time  Timedelta properties
^^^  .. autosummary::  api/  autosummary/accessor_attribute.rst  Series.dt.days  Series.dt.seconds  Series.dt.microseconds  Series.dt.nanoseconds  Series.dt.components  Series.dt.unit  Timedelta methods
^^^  .. autosummary::  api/  autosummary/accessor_method.rst  Series.dt.to_pytimedelta  Series.dt.total_seconds  Series.dt.as_unit  .. _api.series.str:  String handling
~~~  ``Series.str`` can be used to access the values of the series as
strings and apply several methods to it. These can be accessed like
``Series.str.<function/property>``.  .. autosummary::  api/  autosummary/accessor_method.rst  Series.str.capitalize  Series.str.casefold  Series.str.cat  Series.str.center  Series.str.contains  Series.str.count  Series.str.decode  Series.str.encode  Series.str.endswith  Series.str.extract  Series.str.extractall  Series.str.find  Series.str.findall  Series.str.fullmatch  Series.str.get  Series.str.index  Series.str.join  Series.str.len  Series.str.ljust  Series.str.lower  Series.str.lstrip  Series.str.match  Series.str.normalize  Series.str.pad  Series.str.partition  Series.str.removeprefix  Series.str.removesuffix  Series.str.repeat  Series.str.replace  Series.str.rfind  Series.str.rindex  Series.str.rjust  Series.str.rpartition  Series.str.rstrip  Series.str.slice  Series.str.slice_replace  Series.str.split  Series.str.rsplit  Series.str.startswith  Series.str.strip  Series.str.swapcase  Series.str.title  Series.str.translate  Series.str.upper  Series.str.wrap  Series.str.zfill  Series.str.isalnum  Series.str.isalpha  Series.str.isdigit  Series.str.isspace  Series.str.islower  Series.str.isupper  Series.str.istitle  Series.str.isnumeric  Series.str.isdecimal  Series.str.get_dummies  ..  The following is needed to ensure the generated pages are created with the  correct template (otherwise they would be created in the Series/Index class page)  ..  .. autosummary::  api/  autosummary/accessor.rst  Series.str  Series.cat  Series.dt  Series.sparse  DataFrame.sparse  Index.str  .. _api.series.cat:  Categorical accessor
~~~  Categorical-dtype specific methods and attributes are available under

the ``Series.cat`` accessor.  .. autosummary::  api/  autosummary/accessor_attribute.rst  Series.cat.categories  Series.cat.ordered  Series.cat.codes  .. autosummary::  api/  autosummary/accessor_method.rst  Series.cat.rename_categories  Series.cat.reorder_categories  Series.cat.add_categories  Series.cat.remove_categories  Series.cat.remove_unused_categories  Series.cat.set_categories  Series.cat.as_ordered  Series.cat.as_unordered  .. _api.series.sparse:  Sparse accessor
``Series.sparse`` accessor.  .. autosummary::  api/  autosummary/accessor_attribute.rst  Series.sparse.npoints  Series.sparse.density  Series.sparse.fill_value  Series.sparse.sp_values  .. autosummary::  api/  autosummary/accessor_method.rst  Series.sparse.from_coo  Series.sparse.to_coo  .. _api.series.flags:  Flags
in `Series.attrs`.  .. autosummary::  api/  Flags  .. _api.series.metadata:  Metadata
~~~  `Series.attrs` is a dictionary for storing global metadata for this Series.  .. warning:: ``Series.attrs`` is considered experimental and may change without warning.  .. autosummary::  api/  Series.attrs

Plotting
``Series.plot`` is both a callable method and a namespace attribute for
specific plotting methods of the form ``Series.plot.<kind>``.  .. autosummary::  api/  autosummary/accessor_callable.rst  Series.plot  .. autosummary::  api/  autosummary/accessor_method.rst  Series.plot.area  Series.plot.bar  Series.plot.barh  Series.plot.box  Series.plot.density  Series.plot.hist  Series.plot.kde  Series.plot.line  Series.plot.pie  .. autosummary::  api/  Series.hist

Serialization / IO / conversion
.. autosummary::  api/  Series.to_pickle  Series.to_csv  Series.to_dict  Series.to_excel  Series.to_frame  Series.to_xarray  Series.to_hdf  Series.to_sql  Series.to_json  Series.to_string  Series.to_clipboard  Series.to_latex  Series.to_markdown

Style
.. currentmodule:: pandas.io.formats.style  ``Styler`` objects are returned by `pandas.DataFrame.style`.

Styler constructor
.. autosummary::  api/  Styler  Styler.from_custom_template

Styler properties
.. autosummary::  api/  Styler.env  Styler.template_html  Styler.template_html_style  Styler.template_html_table  Styler.template_latex  Styler.template_string  Styler.loader

Style application
.. autosummary::  api/  Styler.apply  Styler.applymap  Styler.apply_index  Styler.applymap_index  Styler.format  Styler.format_index  Styler.relabel_index  Styler.hide  Styler.concat  Styler.set_td_classes  Styler.set_table_styles  Styler.set_table_attributes  Styler.set_tooltips  Styler.set_caption  Styler.set_sticky  Styler.set_properties  Styler.set_uuid  Styler.clear  Styler.pipe

Builtin styles
.. autosummary::  api/  Styler.highlight_null  Styler.highlight_max  Styler.highlight_min  Styler.highlight_between  Styler.highlight_quantile  Styler.background_gradient  Styler.text_gradient  Styler.bar

Style export and import
.. autosummary::  api/  Styler.to_html  Styler.to_latex  Styler.to_excel  Styler.to_string  Styler.export  Styler.use

Assertion functions
.. autosummary::  api/  testing.assert_frame_equal  testing.assert_series_equal  testing.assert_index_equal  testing.assert_extension_array_equal

Exceptions and warnings
.. autosummary::  api/  errors.AbstractMethodError  errors.AccessorRegistrationWarning  errors.AttributeConflictWarning  errors.CategoricalConversionWarning  errors.ChainedAssignmentError  errors.ClosedFileError  errors.CSSWarning  errors.DatabaseError  errors.DataError  errors.DtypeWarning  errors.DuplicateLabelError  errors.EmptyDataError  errors.IncompatibilityWarning  errors.IndexingError  errors.InvalidColumnName  errors.InvalidComparison  errors.InvalidIndexError  errors.InvalidVersion  errors.IntCastingNaNError  errors.LossySetitemError  errors.MergeError  errors.NoBufferPresent  errors.NullFrequencyError  errors.NumbaUtilError  errors.NumExprClobberingError  errors.OptionError  errors.OutOfBoundsDatetime  errors.OutOfBoundsTimedelta  errors.ParserError  errors.ParserWarning  errors.PerformanceWarning  errors.PossibleDataLossError  errors.PossiblePrecisionLoss  errors.PyperclipException  errors.PyperclipWindowsException  errors.SettingWithCopyError  errors.SettingWithCopyWarning  errors.SpecificationError  errors.UndefinedVariableError  errors.UnsortedIndexError  errors.UnsupportedFunctionCall  errors.ValueLabelTypeMismatch

Window

Rolling objects are returned by ``.rolling`` calls: `pandas.DataFrame.rolling` and `pandas.Series.rolling`.
Expanding objects are returned by ``.expanding`` calls: `pandas.DataFrame.expanding` and `pandas.Series.expanding`.
ExponentialMovingWindow objects are returned by ``.ewm`` calls: `pandas.DataFrame.ewm` and `pandas.Series.ewm`.  .. _api.functions_rolling:

Rolling window functions
.. currentmodule:: pandas.core.window.rolling  .. autosummary::  api/  Rolling.count  Rolling.sum  Rolling.mean  Rolling.median  Rolling.var  Rolling.std  Rolling.min  Rolling.max  Rolling.corr  Rolling.cov  Rolling.skew  Rolling.kurt  Rolling.apply  Rolling.aggregate  Rolling.quantile  Rolling.sem  Rolling.rank  .. _api.functions_window:

Weighted window functions
.. currentmodule:: pandas.core.window.rolling  .. autosummary::  api/  Window.mean  Window.sum  Window.var  Window.std  .. _api.functions_expanding:

Expanding window functions
.. currentmodule:: pandas.core.window.expanding  .. autosummary::  api/  Expanding.count  Expanding.sum  Expanding.mean  Expanding.median  Expanding.var  Expanding.std  Expanding.min  Expanding.max  Expanding.corr  Expanding.cov  Expanding.skew  Expanding.kurt  Expanding.apply  Expanding.aggregate  Expanding.quantile  Expanding.sem  Expanding.rank  .. _api.functions_ewm:

Exponentially-weighted window functions
.. currentmodule:: pandas.core.window.ewm  .. autosummary::  api/  ExponentialMovingWindow.mean  ExponentialMovingWindow.sum  ExponentialMovingWindow.std  ExponentialMovingWindow.var  ExponentialMovingWindow.corr  ExponentialMovingWindow.cov  .. _api.indexers_window:

Window indexer
.. currentmodule:: pandas  Base class for defining custom window boundaries.  .. autosummary::  api/  api.indexers.BaseIndexer  api.indexers.FixedForwardWindowIndexer  api.indexers.VariableOffsetWindowIndexer

Object creation

See the `Intro to data structures section <dsintro>`.  Creating a `Series` by passing a list of values, letting pandas create
a default integer index:  .. ipython:: python  s = pd.Series([1, 3, 5, np.nan, 6, 8])  s  Creating a `DataFrame` by passing a NumPy array, with a datetime index using `date_range`
and labeled columns:  .. ipython:: python  dates = pd.date_range("20130101", periods=6)  dates  df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list("ABCD"))  df  Creating a `DataFrame` by passing a dictionary of objects that can be
converted into a series-like structure:  .. ipython:: python  df2 = pd.DataFrame(  {  "A": 1.0,  "B": pd.Timestamp("20130102"),  "C": pd.Series(1, index=list(range(4)), dtype="float32"),  "D": np.array([3] * 4, dtype="int32"),  "E": pd.Categorical(["test", "train", "test", "train"]),  "F": "foo",  }  )  df2  The columns of the resulting `DataFrame` have different
`dtypes <basics.dtypes>`:  .. ipython:: python  df2.dtypes  If you're using IPython, tab completion for column names (as well as public
attributes) is automatically enabled. Here's a subset of the attributes that
will be completed:  .. ipython::  @verbatim  In [1]: df2.<TAB>  # noqa: E225, E999  df2.A  df2.bool  df2.abs  df2.boxplot  df2.add  df2.C  df2.add_prefix  df2.clip  df2.add_suffix  df2.columns  df2.align  df2.copy  df2.all  df2.count  df2.any  df2.combine  df2.append  df2.D  df2.apply  df2.describe  df2.applymap  df2.diff  df2.B  df2.duplicated  As you can see, the columns ``A``, ``B``, ``C``, and ``D`` are automatically
tab completed. ``E`` and ``F`` are there as well; the rest of the attributes have been
truncated for brevity.

Viewing data

See the `Basics section <basics>`.  Use `DataFrame.head` and `DataFrame.tail` to view the top and bottom rows of the frame
respectively:  .. ipython:: python  df.head()  df.tail(3)  Display the `DataFrame.index` or `DataFrame.columns`:  .. ipython:: python  df.index  df.columns  `DataFrame.to_numpy` gives a NumPy representation of the underlying data.
Note that this can be an expensive operation when your `DataFrame` has
columns with different data types, which comes down to a fundamental difference
between pandas and NumPy: **NumPy arrays have one dtype for the entire array,
while pandas DataFrames have one dtype per column**. When you call
`DataFrame.to_numpy`, pandas will find the NumPy dtype that can hold *all*
of the dtypes in the DataFrame. This may end up being ``object``, which requires
casting every value to a Python object.  For ``df``, our `DataFrame` of all floating-point values, and
`DataFrame.to_numpy` is fast and doesn't require copying data:  .. ipython:: python  df.to_numpy()  For ``df2``, the `DataFrame` with multiple dtypes,
`DataFrame.to_numpy` is relatively expensive:  .. ipython:: python  df2.to_numpy()  .. note::  `DataFrame.to_numpy` does *not* include the index or column  labels in the output.  `~DataFrame.describe` shows a quick statistic summary of your data:  .. ipython:: python  df.describe()  Transposing your data:  .. ipython:: python  df.T  `DataFrame.sort_index` sorts by an axis:  .. ipython:: python  df.sort_index(axis=1, ascending=False)  `DataFrame.sort_values` sorts by values:  .. ipython:: python  df.sort_values(by="B")

Selection

.. note::  While standard Python / NumPy expressions for selecting and setting are  intuitive and come in handy for interactive work, for production code, we  recommend the optimized pandas data access methods, `DataFrame.at`, `DataFrame.iat`,  `DataFrame.loc` and `DataFrame.iloc`.  See the indexing documentation `Indexing and Selecting Data <indexing>` and `MultiIndex / Advanced Indexing <advanced>`.  Getting
~~~  Selecting a single column, which yields a `Series`,
equivalent to ``df.A``:  .. ipython:: python  df["A"]  Selecting via ``[]`` (``__getitem__``), which slices the rows:  .. ipython:: python  df[0:3]  df["20130102":"20130104"]  Selection by label
~~~  See more in `Selection by Label <indexing.label>` using `DataFrame.loc` or `DataFrame.at`.  For getting a cross section using a label:  .. ipython:: python  df.loc[dates[0]]  Selecting on a multi-axis by label:  .. ipython:: python  df.loc[:, ["A", "B"]]  Showing label slicing, both endpoints are *included*:  .. ipython:: python  df.loc["20130102":"20130104", ["A", "B"]]  Reduction in the dimensions of the returned object:  .. ipython:: python  df.loc["20130102", ["A", "B"]]  For getting a scalar value:  .. ipython:: python  df.loc[dates[0], "A"]  For getting fast access to a scalar (equivalent to the prior method):  .. ipython:: python  df.at[dates[0], "A"]  Selection by position
~~~  See more in `Selection by Position <indexing.integer>` using `DataFrame.iloc` or `DataFrame.iat`.  Select via the position of the passed integers:  .. ipython:: python  df.iloc[3]  By integer slices, acting similar to NumPy/Python:  .. ipython:: python  df.iloc[3:5, 0:2]  By lists of integer position locations, similar to the NumPy/Python style:  .. ipython:: python  df.iloc[[1, 2, 4], [0, 2]]  For slicing rows explicitly:  .. ipython:: python  df.iloc[1:3, :]  For slicing columns explicitly:  .. ipython:: python  df.iloc[:, 1:3]  For getting a value explicitly:  .. ipython:: python  df.iloc[1, 1]  For getting fast access to a scalar (equivalent to the prior method):  .. ipython:: python  df.iat[1, 1]  Boolean indexing
~~~  Using a single column's values to select data:  .. ipython:: python  df[df["A"] > 0]  Selecting values from a DataFrame where a boolean condition is met:  .. ipython:: python  df[df > 0]  Using the `~Series.isin` method for filtering:  .. ipython:: python  df2 = df.copy()  df2["E"] = ["one", "one", "two", "three", "four", "three"]  df2  df2[df2["E"].isin(["two", "four"])]  Setting
~~~  Setting a new column automatically aligns the data
by the indexes:  .. ipython:: python  s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range("20130102", periods=6))  s1  df["F"] = s1  Setting values by label:  .. ipython:: python  df.at[dates[0], "A"] = 0  Setting values by position:  .. ipython:: python  df.iat[0, 1] = 0  Setting by assigning with a NumPy array:  .. ipython:: python  df.loc[:, "D"] = np.array([5] * len(df))  The result of the prior setting operations:  .. ipython:: python  df  A ``where`` operation with setting:  .. ipython:: python  df2 = df.copy()  df2[df2 > 0] = -df2  df2

pandas primarily uses the value ``np.nan`` to represent missing data. It is by
default not included in computations. See the `Missing Data section
<missing_data>`.  Reindexing allows you to change/add/delete the index on a specified axis. This
returns a copy of the data:  .. ipython:: python  df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ["E"])  df1.loc[dates[0] : dates[1], "E"] = 1  df1  `DataFrame.dropna` drops any rows that have missing data:  .. ipython:: python  df1.dropna(how="any")  `DataFrame.fillna` fills missing data:  .. ipython:: python  df1.fillna(value=5)  `isna` gets the boolean mask where values are ``nan``:  .. ipython:: python  pd.isna(df1)

Operations

See the `Basic section on Binary Ops <basics.binop>`.  Stats
~~~  Operations in general *exclude* missing data.  Performing a descriptive statistic:  .. ipython:: python  df.mean()  Same operation on the other axis:  .. ipython:: python  df.mean(1)  Operating with objects that have different dimensionality and need alignment.
In addition, pandas automatically broadcasts along the specified dimension:  .. ipython:: python  s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2)  s  df.sub(s, axis="index")  Apply
~~~  `DataFrame.apply` applies a user defined function to the data:  .. ipython:: python  df.apply(np.cumsum)  df.apply(lambda x: x.max() - x.min())  Histogramming
~~~  See more at `Histogramming and Discretization <basics.discretization>`.  .. ipython:: python  s = pd.Series(np.random.randint(0, 7, size=10))  s  s.value_counts()  String Methods
~~~  Series is equipped with a set of string processing methods in the ``str``
attribute that make it easy to operate on each element of the array, as in the
code snippet below. Note that pattern-matching in ``str`` generally uses `regular
expressions <https://docs.python.org/3/library/re.html>`__ by default (and in
some cases always uses them). See more at `Vectorized String Methods
<text.string_methods>`.  .. ipython:: python  s = pd.Series(["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"])  s.str.lower()

Merge

Concat
~~~  pandas provides various facilities for easily combining together Series and
DataFrame objects with various kinds of set logic for the indexes
and relational algebra functionality in the case of join / merge-type
operations.  See the `Merging section <merging>`.  Concatenating pandas objects together along an axis with `concat`:  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 4))  df  # break it into pieces  pieces = [df[:3], df[3:7], df[7:]]  pd.concat(pieces)  .. note::  Adding a column to a `DataFrame` is relatively fast. However, adding  a row requires a copy, and may be expensive. We recommend passing a  pre-built list of records to the `DataFrame` constructor instead  of building a `DataFrame` by iteratively appending records to it.  Join
~~~  `merge` enables SQL style join types along specific columns. See the `Database style joining <merging.join>` section.  .. ipython:: python  left = pd.DataFrame({"key": ["foo", "foo"], "lval": [1, 2]})  right = pd.DataFrame({"key": ["foo", "foo"], "rval": [4, 5]})  left  right  pd.merge(left, right, on="key")  Another example that can be given is:  .. ipython:: python  left = pd.DataFrame({"key": ["foo", "bar"], "lval": [1, 2]})  right = pd.DataFrame({"key": ["foo", "bar"], "rval": [4, 5]})  left  right  pd.merge(left, right, on="key")

Grouping

By "group by" we are referring to a process involving one or more of the
following steps:  - **Splitting** the data into groups based on some criteria  - **Applying** a function to each group independently  - **Combining** the results into a data structure  See the `Grouping section <groupby>`.  .. ipython:: python  df = pd.DataFrame(  {  "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],  "B": ["one", "one", "two", "three", "two", "two", "one", "three"],  "C": np.random.randn(8),  "D": np.random.randn(8),  }  )  df  Grouping and then applying the `~pandas.core.groupby.DataFrameGroupBy.sum` function to the resulting
groups:  .. ipython:: python  df.groupby("A")[["C", "D"]].sum()  Grouping by multiple columns forms a hierarchical index, and again we can
apply the `~pandas.core.groupby.DataFrameGroupBy.sum` function:  .. ipython:: python  df.groupby(["A", "B"]).sum()

Reshaping

See the sections on `Hierarchical Indexing <advanced.hierarchical>` and
`Reshaping <reshaping.stacking>`.  Stack
~~~  .. ipython:: python  tuples = list(  zip(  ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],  ["one", "two", "one", "two", "one", "two", "one", "two"],  )  )  index = pd.MultiIndex.from_tuples(tuples, names=["first", "second"])  df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=["A", "B"])  df2 = df[:4]  df2  The `~DataFrame.stack` method "compresses" a level in the DataFrame's
columns:  .. ipython:: python  stacked = df2.stack()  stacked  With a "stacked" DataFrame or Series (having a `MultiIndex` as the
``index``), the inverse operation of `~DataFrame.stack` is
`~DataFrame.unstack`, which by default unstacks the **last level**:  .. ipython:: python  stacked.unstack()  stacked.unstack(1)  stacked.unstack(0)  Pivot tables
~~~
See the section on `Pivot Tables <reshaping.pivot>`.  .. ipython:: python  df = pd.DataFrame(  {  "A": ["one", "one", "two", "three"] * 3,  "B": ["A", "B", "C"] * 4,  "C": ["foo", "foo", "foo", "bar", "bar", "bar"] * 2,  "D": np.random.randn(12),  "E": np.random.randn(12),  }  )  df  `pivot_table` pivots a `DataFrame` specifying the ``values``, ``index`` and ``columns``  .. ipython:: python  pd.pivot_table(df, values="D", index=["A", "B"], columns=["C"])

Time series

pandas has simple, powerful, and efficient functionality for performing
resampling operations during frequency conversion (e.g., converting secondly
data into 5-minutely data). This is extremely common in, but not limited to,
financial applications. See the `Time Series section <timeseries>`.  .. ipython:: python  rng = pd.date_range("1/1/2012", periods=100, freq="S")  ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)  ts.resample("5Min").sum()  `Series.tz_localize` localizes a time series to a time zone:  .. ipython:: python  rng = pd.date_range("3/6/2012 00:00", periods=5, freq="D")  ts = pd.Series(np.random.randn(len(rng)), rng)  ts  ts_utc = ts.tz_localize("UTC")  ts_utc  `Series.tz_convert` converts a timezones aware time series to another time zone:  .. ipython:: python  ts_utc.tz_convert("US/Eastern")  Converting between time span representations:  .. ipython:: python  rng = pd.date_range("1/1/2012", periods=5, freq="M")  ts = pd.Series(np.random.randn(len(rng)), index=rng)  ts  ps = ts.to_period()  ps  ps.to_timestamp()  Converting between period and timestamp enables some convenient arithmetic
functions to be used. In the following example, we convert a quarterly
frequency with year ending in November to 9am of the end of the month following
the quarter end:  .. ipython:: python  prng = pd.period_range("1990Q1", "2000Q4", freq="Q-NOV")  ts = pd.Series(np.random.randn(len(prng)), prng)  ts.index = (prng.asfreq("M", "e") + 1).asfreq("H", "s") + 9  ts.head()

pandas can include categorical data in a `DataFrame`. For full docs, see the
`categorical introduction <categorical>` and the `API documentation <api.arrays.categorical>`.  .. ipython:: python  df = pd.DataFrame(  {"id": [1, 2, 3, 4, 5, 6], "raw_grade": ["a", "b", "b", "a", "a", "e"]}  )  Converting the raw grades to a categorical data type:  .. ipython:: python  df["grade"] = df["raw_grade"].astype("category")  df["grade"]  Rename the categories to more meaningful names:  .. ipython:: python  new_categories = ["very good", "good", "very bad"]  df["grade"] = df["grade"].cat.rename_categories(new_categories)  Reorder the categories and simultaneously add the missing categories (methods under `Series.cat` return a new `Series` by default):  .. ipython:: python  df["grade"] = df["grade"].cat.set_categories(  ["very bad", "bad", "medium", "good", "very good"]  )  df["grade"]  Sorting is per order in the categories, not lexical order:  .. ipython:: python  df.sort_values(by="grade")  Grouping by a categorical column also shows empty categories:  .. ipython:: python  df.groupby("grade").size()

Plotting

See the `Plotting <visualization>` docs.  We use the standard convention for referencing the matplotlib API:  .. ipython:: python  import matplotlib.pyplot as plt  plt.close("all")  The `plt.close`` method is used to `close_ a figure window:  .. ipython:: python  ts = pd.Series(np.random.randn(1000), index=pd.date_range("1/1/2000", periods=1000))  ts = ts.cumsum()  @savefig series_plot_basic.png  ts.plot();  If running under Jupyter Notebook, the plot will appear on `~Series.plot`.  Otherwise use
matplotlib.pyplot.show_ to show it or
matplotlib.pyplot.savefig_ to write it to a file.  .. ipython:: python  plt.show();  On a DataFrame, the `~DataFrame.plot` method is a convenience to plot all
of the columns with labels:  .. ipython:: python  df = pd.DataFrame(  np.random.randn(1000, 4), index=ts.index, columns=["A", "B", "C", "D"]  )  df = df.cumsum()  plt.figure();  df.plot();  @savefig frame_plot_basic.png  plt.legend(loc='best');

Importing and exporting data

CSV
~~~  `Writing to a csv file: <io.store_in_csv>` using `DataFrame.to_csv`  .. ipython:: python  df.to_csv("foo.csv")  `Reading from a csv file: <io.read_csv_table>` using `read_csv`  .. ipython:: python  pd.read_csv("foo.csv")  .. ipython:: python  import os  os.remove("foo.csv")  HDF5
~~~  Reading and writing to `HDFStores <io.hdf5>`.  Writing to a HDF5 Store using `DataFrame.to_hdf`:  .. ipython:: python  df.to_hdf("foo.h5", "df")  Reading from a HDF5 Store using `read_hdf`:  .. ipython:: python  pd.read_hdf("foo.h5", "df")  .. ipython:: python  os.remove("foo.h5")  Excel
~~~  Reading and writing to `Excel <io.excel>`.  Writing to an excel file using `DataFrame.to_excel`:  .. ipython:: python  df.to_excel("foo.xlsx", sheet_name="Sheet1")  Reading from an excel file using `read_excel`:  .. ipython:: python  pd.read_excel("foo.xlsx", "Sheet1", index_col=None, na_values=["NA"])  .. ipython:: python  os.remove("foo.xlsx")

Gotchas

If you are attempting to perform a boolean operation on a `Series` or `DataFrame`
you might see an exception like:  .. ipython:: python  if pd.Series([False, True, False]):  print("I was true")  See `Comparisons<basics.compare>` and `Gotchas<gotchas>` for an explanation and what to do.

Hierarchical / Multi-level indexing is very exciting as it opens the door to some
quite sophisticated data analysis and manipulation, especially for working with
higher dimensional data. In essence, it enables you to store and manipulate
data with an arbitrary number of dimensions in lower dimensional data
structures like ``Series`` (1d) and ``DataFrame`` (2d).  In this section, we will show what exactly we mean by "hierarchical" indexing
and how it integrates with all of the pandas indexing functionality
described above and in prior sections. Later, when discussing `group by
<groupby>` and `pivoting and reshaping data <reshaping>`, we'll show
non-trivial applications to illustrate how it aids in structuring data for
analysis.  See the `cookbook<cookbook.multi_index>` for some advanced strategies.  Creating a MultiIndex (hierarchical index) object
~~~  The `MultiIndex` object is the hierarchical analogue of the standard
`Index` object which typically stores the axis labels in pandas objects. You
can think of ``MultiIndex`` as an array of tuples where each tuple is unique. A
``MultiIndex`` can be created from a list of arrays (using
`MultiIndex.from_arrays`), an array of tuples (using
`MultiIndex.from_tuples`), a crossed set of iterables (using
`MultiIndex.from_product`), or a `DataFrame` (using
`MultiIndex.from_frame`).  The ``Index`` constructor will attempt to return
a ``MultiIndex`` when it is passed a list of tuples.  The following examples
demonstrate different ways to initialize MultiIndexes.  .. ipython:: python  arrays = [  ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],  ["one", "two", "one", "two", "one", "two", "one", "two"],  ]  tuples = list(zip(*arrays))  tuples  index = pd.MultiIndex.from_tuples(tuples, names=["first", "second"])  index  s = pd.Series(np.random.randn(8), index=index)  s  When you want every pairing of the elements in two iterables, it can be easier
to use the `MultiIndex.from_product` method:  .. ipython:: python  iterables = [["bar", "baz", "foo", "qux"], ["one", "two"]]  pd.MultiIndex.from_product(iterables, names=["first", "second"])  You can also construct a ``MultiIndex`` from a ``DataFrame`` directly, using
the method `MultiIndex.from_frame`. This is a complementary method to
`MultiIndex.to_frame`.  .. ipython:: python  df = pd.DataFrame(  [["bar", "one"], ["bar", "two"], ["foo", "one"], ["foo", "two"]],  columns=["first", "second"],  )  pd.MultiIndex.from_frame(df)  As a convenience, you can pass a list of arrays directly into ``Series`` or
``DataFrame`` to construct a ``MultiIndex`` automatically:  .. ipython:: python  arrays = [  np.array(["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"]),  np.array(["one", "two", "one", "two", "one", "two", "one", "two"]),  ]  s = pd.Series(np.random.randn(8), index=arrays)  s  df = pd.DataFrame(np.random.randn(8, 4), index=arrays)  df  All of the ``MultiIndex`` constructors accept a ``names`` argument which stores
string names for the levels themselves. If no names are provided, ``None`` will
be assigned:  .. ipython:: python  df.index.names  This index can back any axis of a pandas object, and the number of **levels**
of the index is up to you:  .. ipython:: python  df = pd.DataFrame(np.random.randn(3, 8), index=["A", "B", "C"], columns=index)  df  pd.DataFrame(np.random.randn(6, 6), index=index[:6], columns=index[:6])  We've "sparsified" the higher levels of the indexes to make the console output a
bit easier on the eyes. Note that how the index is displayed can be controlled using the
``multi_sparse`` option in ``pandas.set_options()``:  .. ipython:: python  with pd.option_context("display.multi_sparse", False):  df  It's worth keeping in mind that there's nothing preventing you from using
tuples as atomic labels on an axis:  .. ipython:: python  pd.Series(np.random.randn(8), index=tuples)  The reason that the ``MultiIndex`` matters is that it can allow you to do
grouping, selection, and reshaping operations as we will describe below and in

subsequent areas of the documentation. As you will see in later sections, you
can find yourself working with hierarchically-indexed data without creating a
``MultiIndex`` explicitly yourself. However, when loading data from a file, you
may wish to generate your own ``MultiIndex`` when preparing the data set.  .. _advanced.get_level_values:  Reconstructing the level labels
~~~  The method `~MultiIndex.get_level_values` will return a vector of the labels for each
location at a particular level:  .. ipython:: python  index.get_level_values(0)  index.get_level_values("second")  Basic indexing on axis with MultiIndex
~~~  One of the important features of hierarchical indexing is that you can select
data by a "partial" label identifying a subgroup in the data. **Partial**
selection "drops" levels of the hierarchical index in the result in a
completely analogous way to selecting a column in a regular DataFrame:  .. ipython:: python  df["bar"]  df["bar", "one"]  df["bar"]["one"]  s["qux"]  See `Cross-section with hierarchical index <advanced.xs>` for how to select
on a deeper level.  .. _advanced.shown_levels:  Defined levels
~~~  The `MultiIndex` keeps all the defined levels of an index, even
if they are not actually used. When slicing an index, you may notice this.
For example:  .. ipython:: python  df.columns.levels  # original MultiIndex  df[["foo","qux"]].columns.levels  # sliced  This is done to avoid a recomputation of the levels in order to make slicing
highly performant. If you want to see only the used levels, you can use the
`~MultiIndex.get_level_values` method.  .. ipython:: python  df[["foo", "qux"]].columns.to_numpy()  # for a specific level  df[["foo", "qux"]].columns.get_level_values(0)  To reconstruct the ``MultiIndex`` with only the used levels, the
`~MultiIndex.remove_unused_levels` method may be used.  .. ipython:: python  new_mi = df[["foo", "qux"]].columns.remove_unused_levels()  new_mi.levels  Data alignment and using ``reindex``
~~~  Operations between differently-indexed objects having ``MultiIndex`` on the
axes will work as you expect; data alignment will work the same as an Index of
tuples:  .. ipython:: python  s + s[:-2]  s + s[::2]  The `~DataFrame.reindex` method of ``Series``/``DataFrames`` can be
called with another ``MultiIndex``, or even a list or array of tuples:  .. ipython:: python  s.reindex(index[:3])  s.reindex([("foo", "two"), ("bar", "one"), ("qux", "one"), ("baz", "one")])  .. _advanced.advanced_hierarchical:

Syntactically integrating ``MultiIndex`` in advanced indexing with ``.loc`` is a
bit challenging, but we've made every effort to do so. In general, MultiIndex
keys take the form of tuples. For example, the following works as you would expect:  .. ipython:: python  df = df.T  df  df.loc[("bar", "two")]  Note that ``df.loc['bar', 'two']`` would also work in this example, but this shorthand
notation can lead to ambiguity in general.  If you also want to index a specific column with ``.loc``, you must use a tuple
like this:  .. ipython:: python  df.loc[("bar", "two"), "A"]  You don't have to specify all levels of the ``MultiIndex`` by passing only the
first elements of the tuple. For example, you can use "partial" indexing to
get all elements with ``bar`` in the first level as follows:  .. ipython:: python  df.loc["bar"]  This is a shortcut for the slightly more verbose notation ``df.loc[('bar',),]`` (equivalent
to ``df.loc['bar',]`` in this example).  "Partial" slicing also works quite nicely.  .. ipython:: python  df.loc["baz":"foo"]  You can slice with a 'range' of values, by providing a slice of tuples.  .. ipython:: python  df.loc[("baz", "two"):("qux", "one")]  df.loc[("baz", "two"):"foo"]  Passing a list of labels or tuples works similar to reindexing:  .. ipython:: python  df.loc[[("bar", "two"), ("qux", "one")]]  .. note::  It is important to note that tuples and lists are not treated identically  in pandas when it comes to indexing. Whereas a tuple is interpreted as one  multi-level key, a list is used to specify several keys. Or in other words,  tuples go horizontally (traversing levels), lists go vertically (scanning levels).  Importantly, a list of tuples indexes several complete ``MultiIndex`` keys,
whereas a tuple of lists refer to several values within a level:  .. ipython:: python  s = pd.Series(  [1, 2, 3, 4, 5, 6],  index=pd.MultiIndex.from_product([["A", "B"], ["c", "d", "e"]]),  )  s.loc[[("A", "c"), ("B", "d")]]  # list of tuples  s.loc[(["A", "B"], ["c", "d"])]  # tuple of lists  .. _advanced.mi_slicers:  Using slicers
~~~  You can slice a ``MultiIndex`` by providing multiple indexers.  You can provide any of the selectors as if you are indexing by label, see `Selection by Label <indexing.label>`,
including slices, lists of labels, labels, and boolean indexers.  You can use ``slice(None)`` to select all the contents of *that* level. You do not need to specify all the
*deeper* levels, they will be implied as ``slice(None)``.  As usual, **both sides** of the slicers are included as this is label indexing.  .. warning::  You should specify all axes in the ``.loc`` specifier, meaning the indexer for the **index** and  for the **columns**. There are some ambiguous cases where the passed indexer could be mis-interpreted  as indexing *both* axes, rather than into say the ``MultiIndex`` for the rows.  You should do this:  .. code-block:: python  df.loc[(slice("A1", "A3"), ...), :]  # noqa: E999  You should **not** do this:  .. code-block:: python  df.loc[(slice("A1", "A3"), ...)]  # noqa: E999  .. ipython:: python  def mklbl(prefix, n):  return ["%s%s" % (prefix, i) for i in range(n)]  miindex = pd.MultiIndex.from_product(  [mklbl("A", 4), mklbl("B", 2), mklbl("C", 4), mklbl("D", 2)]  )  micolumns = pd.MultiIndex.from_tuples(  [("a", "foo"), ("a", "bar"), ("b", "foo"), ("b", "bah")], names=["lvl0", "lvl1"]  )  dfmi = (  pd.DataFrame(  np.arange(len(miindex) * len(micolumns)).reshape(  (len(miindex), len(micolumns))  ),  index=miindex,  columns=micolumns,  )  .sort_index()  .sort_index(axis=1)  )  dfmi  Basic MultiIndex slicing using slices, lists, and labels.  .. ipython:: python  dfmi.loc[(slice("A1", "A3"), slice(None), ["C1", "C3"]), :]  You can use `pandas.IndexSlice` to facilitate a more natural syntax

using ``:``, rather than using ``slice(None)``.  .. ipython:: python  idx = pd.IndexSlice  dfmi.loc[idx[:, :, ["C1", "C3"]], idx[:, "foo"]]  It is possible to perform quite complicated selections using this method on multiple
axes at the same time.  .. ipython:: python  dfmi.loc["A1", (slice(None), "foo")]  dfmi.loc[idx[:, :, ["C1", "C3"]], idx[:, "foo"]]  Using a boolean indexer you can provide selection related to the *values*.  .. ipython:: python  mask = dfmi[("a", "foo")] > 200  dfmi.loc[idx[mask, :, ["C1", "C3"]], idx[:, "foo"]]  You can also specify the ``axis`` argument to ``.loc`` to interpret the passed
slicers on a single axis.  .. ipython:: python  dfmi.loc(axis=0)[:, :, ["C1", "C3"]]  Furthermore, you can *set* the values using the following methods.  .. ipython:: python  df2 = dfmi.copy()  df2.loc(axis=0)[:, :, ["C1", "C3"]] = -10  df2  You can use a right-hand-side of an alignable object as well.  .. ipython:: python  df2 = dfmi.copy()  df2.loc[idx[:, :, ["C1", "C3"]], :] = df2 * 1000  df2  .. _advanced.xs:  Cross-section
~~~  The `~DataFrame.xs` method of ``DataFrame`` additionally takes a level argument to make
selecting data at a particular level of a ``MultiIndex`` easier.  .. ipython:: python  df  df.xs("one", level="second")  .. ipython:: python  # using the slicers  df.loc[(slice(None), "one"), :]  You can also select on the columns with ``xs``, by
providing the axis argument.  .. ipython:: python  df = df.T  df.xs("one", level="second", axis=1)  .. ipython:: python  # using the slicers  df.loc[:, (slice(None), "one")]  ``xs`` also allows selection with multiple keys.  .. ipython:: python  df.xs(("one", "bar"), level=("second", "first"), axis=1)  .. ipython:: python  # using the slicers  df.loc[:, ("bar", "one")]  You can pass ``drop_level=False`` to ``xs`` to retain
the level that was selected.  .. ipython:: python  df.xs("one", level="second", axis=1, drop_level=False)  Compare the above with the result using ``drop_level=True`` (the default value).  .. ipython:: python  df.xs("one", level="second", axis=1, drop_level=True)  .. ipython:: python  df = df.T  .. _advanced.advanced_reindex:  Advanced reindexing and alignment
~~~  Using the parameter ``level`` in the `~DataFrame.reindex` and
`~DataFrame.align` methods of pandas objects is useful to broadcast
values across a level. For instance:  .. ipython:: python  midx = pd.MultiIndex(  levels=[["zero", "one"], ["x", "y"]], codes=[[1, 1, 0, 0], [1, 0, 1, 0]]  )  df = pd.DataFrame(np.random.randn(4, 2), index=midx)  df  df2 = df.groupby(level=0).mean()  df2  df2.reindex(df.index, level=0)  # aligning  df_aligned, df2_aligned = df.align(df2, level=0)  df_aligned  df2_aligned  Swapping levels with ``swaplevel``
~~~  The `~MultiIndex.swaplevel` method can switch the order of two levels:  .. ipython:: python  df[:5]  df[:5].swaplevel(0, 1, axis=0)  .. _advanced.reorderlevels:  Reordering levels with ``reorder_levels``
~~~  The `~MultiIndex.reorder_levels` method generalizes the ``swaplevel``
method, allowing you to permute the hierarchical index levels in one step:  .. ipython:: python  df[:5].reorder_levels([1, 0], axis=0)  .. _advanced.index_names:  Renaming names of an ``Index`` or ``MultiIndex``
~~~  The `~DataFrame.rename` method is used to rename the labels of a
``MultiIndex``, and is typically used to rename the columns of a ``DataFrame``.
The ``columns`` argument of ``rename`` allows a dictionary to be specified
that includes only the columns you wish to rename.  .. ipython:: python  df.rename(columns={0: "col0", 1: "col1"})  This method can also be used to rename specific labels of the main index
of the ``DataFrame``.  .. ipython:: python  df.rename(index={"one": "two", "y": "z"})  The `~DataFrame.rename_axis` method is used to rename the name of a
``Index`` or ``MultiIndex``. In particular, the names of the levels of a
``MultiIndex`` can be specified, which is useful if ``reset_index()`` is later

used to move the values from the ``MultiIndex`` to a column.  .. ipython:: python  df.rename_axis(index=["abc", "def"])  Note that the columns of a ``DataFrame`` are an index, so that using
``rename_axis`` with the ``columns`` argument will change the name of that
index.  .. ipython:: python  df.rename_axis(columns="Cols").columns  Both ``rename`` and ``rename_axis`` support specifying a dictionary,
``Series`` or a mapping function to map labels/names to new values.  When working with an ``Index`` object directly, rather than via a ``DataFrame``,
`Index.set_names` can be used to change the names.  .. ipython:: python  mi = pd.MultiIndex.from_product([[1, 2], ["a", "b"]], names=["x", "y"])  mi.names  mi2 = mi.rename("new name", level=0)  mi2  You cannot set the names of the MultiIndex via a level.  .. ipython:: python  mi.levels[0].name = "name via level"  Use `Index.set_names` instead.

Sorting a ``MultiIndex``

For `MultiIndex`-ed objects to be indexed and sliced effectively,
they need to be sorted. As with any index, you can use `~DataFrame.sort_index`.  .. ipython:: python  import random  random.shuffle(tuples)  s = pd.Series(np.random.randn(8), index=pd.MultiIndex.from_tuples(tuples))  s  s.sort_index()  s.sort_index(level=0)  s.sort_index(level=1)  .. _advanced.sortlevel_byname:  You may also pass a level name to ``sort_index`` if the ``MultiIndex`` levels
are named.  .. ipython:: python  s.index = s.index.set_names(["L1", "L2"])  s.sort_index(level="L1")  s.sort_index(level="L2")  On higher dimensional objects, you can sort any of the other axes by level if
they have a ``MultiIndex``:  .. ipython:: python  df.T.sort_index(level=1, axis=1)  Indexing will work even if the data are not sorted, but will be rather
inefficient (and show a ``PerformanceWarning``). It will also
return a copy of the data rather than a view:  .. ipython:: python  dfm = pd.DataFrame(  {"jim": [0, 0, 1, 1], "joe": ["x", "x", "z", "y"], "jolie": np.random.rand(4)}  )  dfm = dfm.set_index(["jim", "joe"])  dfm  .. code-block:: ipython  In [4]: dfm.loc[(1, 'z')]  PerformanceWarning: indexing past lexsort depth may impact performance.  Out[4]:  jolie  jim joe  1  z  0.64094  .. _advanced.unsorted:  Furthermore, if you try to index something that is not fully lexsorted, this can raise:  .. code-block:: ipython  In [5]: dfm.loc[(0, 'y'):(1, 'z')]  UnsortedIndexError: 'Key length (2) was greater than MultiIndex lexsort depth (1)'  The `~MultiIndex.is_monotonic_increasing` method on a ``MultiIndex`` shows if the
index is sorted:  .. ipython:: python  dfm.index.is_monotonic_increasing  .. ipython:: python  dfm = dfm.sort_index()  dfm  dfm.index.is_monotonic_increasing  And now selection works as expected.  .. ipython:: python  dfm.loc[(0, "y"):(1, "z")]

Take methods

.. _advanced.take:  Similar to NumPy ndarrays, pandas ``Index``, ``Series``, and ``DataFrame`` also provides
the `~DataFrame.take` method that retrieves elements along a given axis at the given
indices. The given indices must be either a list or an ndarray of integer
index positions. ``take`` will also accept negative integers as relative positions to the end of the object.  .. ipython:: python  index = pd.Index(np.random.randint(0, 1000, 10))  index  positions = [0, 9, 3]  index[positions]  index.take(positions)  ser = pd.Series(np.random.randn(10))  ser.iloc[positions]  ser.take(positions)  For DataFrames, the given indices should be a 1d list or ndarray that specifies
row or column positions.  .. ipython:: python  frm = pd.DataFrame(np.random.randn(5, 3))  frm.take([1, 4, 3])  frm.take([0, 2], axis=1)  It is important to note that the ``take`` method on pandas objects are not
intended to work on boolean indices and may return unexpected results.  .. ipython:: python  arr = np.random.randn(10)  arr.take([False, False, True, True])  arr[[0, 1]]  ser = pd.Series(np.random.randn(10))  ser.take([False, False, True, True])  ser.iloc[[0, 1]]  Finally, as a small note on performance, because the ``take`` method handles
a narrower range of inputs, it can offer performance that is a good deal
faster than fancy indexing.  .. ipython:: python  arr = np.random.randn(1000, 5)  indexer = np.arange(1000)  random.shuffle(indexer)  %timeit arr[indexer]  %timeit arr.take(indexer, axis=0)  .. ipython:: python  ser = pd.Series(arr[:, 0])  %timeit ser.iloc[indexer]  %timeit ser.take(indexer)  .. _advanced.index_types:

We have discussed ``MultiIndex`` in the previous sections pretty extensively.
Documentation about ``DatetimeIndex`` and ``PeriodIndex`` are shown `here <timeseries.overview>`,
and documentation about ``TimedeltaIndex`` is found `here <timedeltas.index>`.  In the following sub-sections we will highlight some other index types.  .. _advanced.categoricalindex:  CategoricalIndex
~~~  `CategoricalIndex` is a type of index that is useful for supporting
indexing with duplicates. This is a container around a `Categorical`
and allows efficient indexing and storage of an index with a large number of duplicated elements.  .. ipython:: python  from pandas.api.types import CategoricalDtype  df = pd.DataFrame({"A": np.arange(6), "B": list("aabbca")})  df["B"] = df["B"].astype(CategoricalDtype(list("cab")))  df  df.dtypes  df["B"].cat.categories  Setting the index will create a ``CategoricalIndex``.  .. ipython:: python  df2 = df.set_index("B")  df2.index  Indexing with ``__getitem__/.iloc/.loc`` works similarly to an ``Index`` with duplicates.
The indexers **must** be in the category or the operation will raise a ``KeyError``.  .. ipython:: python  df2.loc["a"]  The ``CategoricalIndex`` is **preserved** after indexing:  .. ipython:: python  df2.loc["a"].index  Sorting the index will sort by the order of the categories (recall that we
created the index with ``CategoricalDtype(list('cab'))``, so the sorted
order is ``cab``).  .. ipython:: python  df2.sort_index()  Groupby operations on the index will preserve the index nature as well.  .. ipython:: python  df2.groupby(level=0).sum()  df2.groupby(level=0).sum().index  Reindexing operations will return a resulting index based on the type of the passed
indexer. Passing a list will return a plain-old ``Index``; indexing with
a ``Categorical`` will return a ``CategoricalIndex``, indexed according to the categories
of the **passed** ``Categorical`` dtype. This allows one to arbitrarily index these even with
values **not** in the categories, similarly to how you can reindex **any** pandas index.  .. ipython:: python  df3 = pd.DataFrame(  {"A": np.arange(3), "B": pd.Series(list("abc")).astype("category")}  )  df3 = df3.set_index("B")  df3  .. ipython:: python  df3.reindex(["a", "e"])  df3.reindex(["a", "e"]).index  df3.reindex(pd.Categorical(["a", "e"], categories=list("abe")))  df3.reindex(pd.Categorical(["a", "e"], categories=list("abe"))).index  .. warning::  Reshaping and Comparison operations on a ``CategoricalIndex`` must have the same categories  or a ``TypeError`` will be raised.  .. ipython:: python  df4 = pd.DataFrame({"A": np.arange(2), "B": list("ba")})  df4["B"] = df4["B"].astype(CategoricalDtype(list("ab")))  df4 = df4.set_index("B")  df4.index  df5 = pd.DataFrame({"A": np.arange(2), "B": list("bc")})  df5["B"] = df5["B"].astype(CategoricalDtype(list("bc")))  df5 = df5.set_index("B")  df5.index  .. code-block:: ipython  In [1]: pd.concat([df4, df5])  TypeError: categories must match existing categories when appending  .. _advanced.rangeindex:  RangeIndex
~~~  `RangeIndex` is a sub-class of `Index`  that provides the default index for all `DataFrame` and `Series` objects.
`RangeIndex`` is an optimized version of ``Index`` that can represent a monotonic ordered set. These are analogous to Python `range types_.
A ``RangeIndex`` will always have an ``int64`` dtype.  .. ipython:: python  idx = pd.RangeIndex(5)  idx  ``RangeIndex`` is the default index for all `DataFrame` and `Series` objects:  .. ipython:: python  ser = pd.Series([1, 2, 3])  ser.index  df = pd.DataFrame([[1, 2], [3, 4]])  df.index  df.columns  A ``RangeIndex`` will behave similarly to a `Index` with an ``int64`` dtype and operations on a ``RangeIndex``,
whose result cannot be represented by a ``RangeIndex``, but should have an integer dtype, will be converted to an ``Index`` with ``int64``.
For example:  .. ipython:: python  idx[[0, 2]]  .. _advanced.intervalindex:  IntervalIndex

~~~  `IntervalIndex` together with its own dtype, `~pandas.api.types.IntervalDtype`
as well as the `Interval` scalar type,  allow first-class support in pandas
for interval notation.  The ``IntervalIndex`` allows some unique indexing and is also used as a
return type for the categories in `cut` and `qcut`.  Indexing with an ``IntervalIndex``
^^^  An ``IntervalIndex`` can be used in ``Series`` and in ``DataFrame`` as the index.  .. ipython:: python  df = pd.DataFrame(  {"A": [1, 2, 3, 4]}, index=pd.IntervalIndex.from_breaks([0, 1, 2, 3, 4])  )  df  Label based indexing via ``.loc`` along the edges of an interval works as you would expect,
selecting that particular interval.  .. ipython:: python  df.loc[2]  df.loc[[2, 3]]  If you select a label *contained* within an interval, this will also select the interval.  .. ipython:: python  df.loc[2.5]  df.loc[[2.5, 3.5]]  Selecting using an ``Interval`` will only return exact matches (starting from pandas 0.25.0).  .. ipython:: python  df.loc[pd.Interval(1, 2)]  Trying to select an ``Interval`` that is not exactly contained in the ``IntervalIndex`` will raise a ``KeyError``.  .. code-block:: python

In [7]: df.loc[pd.Interval(0.5, 2.5)]
  KeyError: Interval(0.5, 2.5, closed='right')  Selecting all ``Intervals`` that overlap a given ``Interval`` can be performed using the
`~IntervalIndex.overlaps` method to create a boolean indexer.  .. ipython:: python  idxr = df.index.overlaps(pd.Interval(0.5, 2.5))  idxr  df[idxr]  Binning data with ``cut`` and ``qcut``
^^^  `cut` and `qcut` both return a ``Categorical`` object, and the bins they
create are stored as an ``IntervalIndex`` in its ``.categories`` attribute.  .. ipython:: python  c = pd.cut(range(4), bins=2)  c  c.categories  `cut` also accepts an ``IntervalIndex`` for its ``bins`` argument, which enables
a useful pandas idiom. First, We call `cut` with some data and ``bins`` set to a
fixed number, to generate the bins. Then, we pass the values of ``.categories`` as the
``bins`` argument in subsequent calls to `cut`, supplying new data which will be
binned into the same bins.  .. ipython:: python  pd.cut([0, 3, 5, 1], bins=c.categories)  Any value which falls outside all bins will be assigned a ``NaN`` value.  Generating ranges of intervals
^^^  If we need intervals on a regular frequency, we can use the `interval_range` function
to create an ``IntervalIndex`` using various combinations of ``start``, ``end``, and ``periods``.
The default frequency for ``interval_range`` is a 1 for numeric intervals, and calendar day for
datetime-like intervals:  .. ipython:: python  pd.interval_range(start=0, end=5)  pd.interval_range(start=pd.Timestamp("2017-01-01"), periods=4)  pd.interval_range(end=pd.Timedelta("3 days"), periods=3)  The ``freq`` parameter can used to specify non-default frequencies, and can utilize a variety
of `frequency aliases <timeseries.offset_aliases>` with datetime-like intervals:  .. ipython:: python  pd.interval_range(start=0, periods=5, freq=1.5)  pd.interval_range(start=pd.Timestamp("2017-01-01"), periods=4, freq="W")  pd.interval_range(start=pd.Timedelta("0 days"), periods=3, freq="9H")  Additionally, the ``closed`` parameter can be used to specify which side(s) the intervals
are closed on.  Intervals are closed on the right side by default.  .. ipython:: python  pd.interval_range(start=0, end=4, closed="both")  pd.interval_range(start=0, end=4, closed="neither")  Specifying ``start``, ``end``, and ``periods`` will generate a range of evenly spaced
intervals from ``start`` to ``end`` inclusively, with ``periods`` number of elements
in the resulting ``IntervalIndex``:  .. ipython:: python  pd.interval_range(start=0, end=6, periods=4)  pd.interval_range(pd.Timestamp("2018-01-01"), pd.Timestamp("2018-02-28"), periods=3)

Integer indexing
~~~  Label-based indexing with integer axis labels is a thorny topic. It has been
discussed heavily on mailing lists and among various members of the scientific
Python community. In pandas, our general viewpoint is that labels matter more
than integer locations. Therefore, with an integer axis index *only*
label-based indexing is possible with the standard tools like ``.loc``. The
following code will generate exceptions:  .. ipython:: python  s = pd.Series(range(5))  s[-1]  df = pd.DataFrame(np.random.randn(5, 4))  df  df.loc[-2:]  This deliberate decision was made to prevent ambiguities and subtle bugs (many
users reported finding bugs when the API change was made to stop "falling back"
on position-based indexing).  Non-monotonic indexes require exact matches
~~~  If the index of a ``Series`` or ``DataFrame`` is monotonically increasing or decreasing, then the bounds
of a label-based slice can be outside the range of the index, much like slice indexing a
normal Python ``list``. Monotonicity of an index can be tested with the `~Index.is_monotonic_increasing` and
`~Index.is_monotonic_decreasing` attributes.  .. ipython:: python  df = pd.DataFrame(index=[2, 3, 3, 4, 5], columns=["data"], data=list(range(5)))  df.index.is_monotonic_increasing  # no rows 0 or 1, but still returns rows 2, 3 (both of them), and 4:  df.loc[0:4, :]  # slice is are outside the index, so empty DataFrame is returned  df.loc[13:15, :]  On the other hand, if the index is not monotonic, then both slice bounds must be
*unique* members of the index.  .. ipython:: python  df = pd.DataFrame(index=[2, 3, 1, 4, 3, 5], columns=["data"], data=list(range(6)))  df.index.is_monotonic_increasing  # OK because 2 and 4 are in the index  df.loc[2:4, :]  .. code-block:: ipython  # 0 is not in the index  In [9]: df.loc[0:4, :]  KeyError: 0  # 3 is not a unique label  In [11]: df.loc[2:3, :]  KeyError: 'Cannot get right slice bound for non-unique label: 3'  ``Index.is_monotonic_increasing`` and ``Index.is_monotonic_decreasing`` only check that
an index is weakly monotonic. To check for strict monotonicity, you can combine one of those with
the `~Index.is_unique` attribute.  .. ipython:: python  weakly_monotonic = pd.Index(["a", "b", "c", "c"])  weakly_monotonic  weakly_monotonic.is_monotonic_increasing  weakly_monotonic.is_monotonic_increasing & weakly_monotonic.is_unique  .. _advanced.endpoints_are_inclusive:  Endpoints are inclusive
~~~  Compared with standard Python sequence slicing in which the slice endpoint is
not inclusive, label-based slicing in pandas **is inclusive**. The primary
reason for this is that it is often not possible to easily determine the
"successor" or next element after a particular label in an index. For example,
consider the following ``Series``:  .. ipython:: python  s = pd.Series(np.random.randn(6), index=list("abcdef"))  s  Suppose we wished to slice from ``c`` to ``e``, using integers this would be
accomplished as such:  .. ipython:: python  s[2:5]  However, if you only had ``c`` and ``e``, determining the next element in the
index can be somewhat complicated. For example, the following does not work:  ::  s.loc['c':'e' + 1]  A very common use case is to limit a time series to start and end at two
specific dates. To enable this, we made the design choice to make label-based
slicing include both endpoints:  .. ipython:: python  s.loc["c":"e"]  This is most definitely a "practicality beats purity" sort of thing, but it is
something to watch out for if you expect label-based slicing to behave exactly
in the way that standard Python integer slicing works.  Indexing potentially changes underlying Series dtype

~~~  The different indexing operation can potentially change the dtype of a ``Series``.  .. ipython:: python  series1 = pd.Series([1, 2, 3])  series1.dtype  res = series1.reindex([0, 4])  res.dtype  res  .. ipython:: python  series2 = pd.Series([True])  series2.dtype  res = series2.reindex_like(series1)  res.dtype  res  This is because the (re)indexing operations above silently inserts ``NaNs`` and the ``dtype``
changes accordingly.  This can cause some issues when using ``numpy`` ``ufuncs``
such as ``numpy.logical_and``.  See the `2388` for a more
detailed discussion.

Essential basic functionality

Here we discuss a lot of the essential functionality common to the pandas data
structures. To begin, let's create some example objects like we did in
the `10 minutes to pandas <10min>` section:  .. ipython:: python  index = pd.date_range("1/1/2000", periods=8)  s = pd.Series(np.random.randn(5), index=["a", "b", "c", "d", "e"])  df = pd.DataFrame(np.random.randn(8, 3), index=index, columns=["A", "B", "C"])  .. _basics.head_tail:

Head and tail

To view a small sample of a Series or DataFrame object, use the
`~DataFrame.head` and `~DataFrame.tail` methods. The default number
of elements to display is five, but you may pass a custom number.  .. ipython:: python  long_series = pd.Series(np.random.randn(1000))  long_series.head()  long_series.tail(3)  .. _basics.attrs:

pandas objects have a number of attributes enabling you to access the metadata  * **shape**: gives the axis dimensions of the object, consistent with ndarray
* Axis labels  * **Series**: *index* (only axis)  * **DataFrame**: *index* (rows) and *columns*  Note, **these attributes can be safely assigned to**!  .. ipython:: python  df[:2]  df.columns = [x.lower() for x in df.columns]  df  pandas objects (`Index`, `Series`, `DataFrame`) can be
thought of as containers for arrays, which hold the actual data and do the
actual computation. For many types, the underlying array is a
`numpy.ndarray`. However, pandas and 3rd party libraries may *extend*
NumPy's type system to add support for custom arrays
(see `basics.dtypes`).  To get the actual data inside a `Index` or `Series`, use
the ``.array`` property  .. ipython:: python  s.array  s.index.array  `~Series.array` will always be an `~pandas.api.extensions.ExtensionArray`.
The exact details of what an `~pandas.api.extensions.ExtensionArray` is and why pandas uses them are a bit
beyond the scope of this introduction. See `basics.dtypes` for more.  If you know you need a NumPy array, use `~Series.to_numpy`
or `numpy.asarray`.  .. ipython:: python  s.to_numpy()  np.asarray(s)  When the Series or Index is backed by
an `~pandas.api.extensions.ExtensionArray`, `~Series.to_numpy`
may involve copying data and coercing values. See `basics.dtypes` for more.  `~Series.to_numpy` gives some control over the ``dtype`` of the
resulting `numpy.ndarray`. For example, consider datetimes with timezones.
NumPy doesn't have a dtype to represent timezone-aware datetimes, so there
are two possibly useful representations:  1. An object-dtype `numpy.ndarray` with `Timestamp` objects, each  with the correct ``tz``
2. A ``datetime64[ns]`` -dtype `numpy.ndarray`, where the values have  been converted to UTC and the timezone discarded  Timezones may be preserved with ``dtype=object``  .. ipython:: python  ser = pd.Series(pd.date_range("2000", periods=2, tz="CET"))  ser.to_numpy(dtype=object)  Or thrown away with ``dtype='datetime64[ns]'``  .. ipython:: python  ser.to_numpy(dtype="datetime64[ns]")  Getting the "raw data" inside a `DataFrame` is possibly a bit more
complex. When your ``DataFrame`` only has a single data type for all the
columns, `DataFrame.to_numpy` will return the underlying data:  .. ipython:: python  df.to_numpy()  If a DataFrame contains homogeneously-typed data, the ndarray can
actually be modified in-place, and the changes will be reflected in the data
structure. For heterogeneous data (e.g. some of the DataFrame's columns are not
all the same dtype), this will not be the case. The values attribute itself,
unlike the axis labels, cannot be assigned to.  .. note::  When working with heterogeneous data, the dtype of the resulting ndarray  will be chosen to accommodate all of the data involved. For example, if  strings are involved, the result will be of object dtype. If there are only  floats and integers, the resulting array will be of float dtype.  In the past, pandas recommended `Series.values` or `DataFrame.values`
for extracting the data from a Series or DataFrame. You'll still find references
to these in old code bases and online. Going forward, we recommend avoiding
``.values`` and using ``.array`` or ``.to_numpy()``. ``.values`` has the following
drawbacks:  1. When your Series contains an `extension type <extending.extension-types>`, it's  unclear whether `Series.values` returns a NumPy array or the extension array.  `Series.array` will always return an `~pandas.api.extensions.ExtensionArray`, and will never  copy data. `Series.to_numpy` will always return a NumPy array,  potentially at the cost of copying / coercing values.

2. When your DataFrame contains a mixture of data types, `DataFrame.values` may  involve copying data and coercing values to a common dtype, a relatively expensive  operation. `DataFrame.to_numpy`, being a method, makes it clearer that the  returned NumPy array may not be a view on the same data in the DataFrame.  .. _basics.accelerate:

Accelerated operations

pandas has support for accelerating certain types of binary numerical and boolean operations using
the ``numexpr`` library and the ``bottleneck`` libraries.  These libraries are especially useful when dealing with large data sets, and provide large
speedups. ``numexpr`` uses smart chunking, caching, and multiple cores. ``bottleneck`` is
a set of specialized cython routines that are especially fast when dealing with arrays that have
``nans``.  Here is a sample (using 100 column x 100,000 row ``DataFrames``):  .. csv-table::  "Operation", "0.11.0 (ms)", "Prior Version (ms)", "Ratio to Prior"  25, 25, 25, 25  ;  ``df1 > df2``; 13.32; 125.35;  0.1063  ``df1 * df2``; 21.71;  36.63;  0.5928  ``df1 + df2``; 22.04;  36.50;  0.6039  You are highly encouraged to install both libraries. See the section
`Recommended Dependencies <install.recommended_dependencies>` for more installation info.  These are both enabled to be used by default, you can control this by setting the options:  .. code-block:: python  pd.set_option("compute.use_bottleneck", False)  pd.set_option("compute.use_numexpr", False)  .. _basics.binop:

With binary operations between pandas data structures, there are two key points
of interest:  * Broadcasting behavior between higher- (e.g. DataFrame) and  lower-dimensional (e.g. Series) objects.
* Missing data in computations.  We will demonstrate how to manage these issues independently, though they can
be handled simultaneously.  Matching / broadcasting behavior
~~~  DataFrame has the methods `~DataFrame.add`, `~DataFrame.sub`,
`~DataFrame.mul`, `~DataFrame.div` and related functions
`~DataFrame.radd`, `~DataFrame.rsub`, ...
for carrying out binary operations. For broadcasting behavior,
Series input is of primary interest. Using these functions, you can use to
either match on the *index* or *columns* via the **axis** keyword:  .. ipython:: python  df = pd.DataFrame(  {  "one": pd.Series(np.random.randn(3), index=["a", "b", "c"]),  "two": pd.Series(np.random.randn(4), index=["a", "b", "c", "d"]),  "three": pd.Series(np.random.randn(3), index=["b", "c", "d"]),  }  )  df  row = df.iloc[1]  column = df["two"]  df.sub(row, axis="columns")  df.sub(row, axis=1)  df.sub(column, axis="index")  df.sub(column, axis=0)  .. ipython:: python  df_orig = df  Furthermore you can align a level of a MultiIndexed DataFrame with a Series.  .. ipython:: python  dfmi = df.copy()  dfmi.index = pd.MultiIndex.from_tuples(  [(1, "a"), (1, "b"), (1, "c"), (2, "a")], names=["first", "second"]  )  dfmi.sub(column, axis=0, level="second")  Series and Index also support the `divmod` builtin. This function takes
the floor division and modulo operation at the same time returning a two-tuple
of the same type as the left hand side. For example:  .. ipython:: python  s = pd.Series(np.arange(10))  s  div, rem = divmod(s, 3)  div  rem  idx = pd.Index(np.arange(10))  idx  div, rem = divmod(idx, 3)  div  rem  We can also do elementwise `divmod`:  .. ipython:: python  div, rem = divmod(s, [2, 2, 3, 3, 4, 4, 5, 5, 6, 6])  div  rem  Missing data / operations with fill values
~~~  In Series and DataFrame, the arithmetic functions have the option of inputting
a *fill_value*, namely a value to substitute when at most one of the values at
a location are missing. For example, when adding two DataFrame objects, you may
wish to treat NaN as 0 unless both DataFrames are missing that value, in which
case the result will be NaN (you can later replace NaN with some other value
using ``fillna`` if you wish).  .. ipython:: python  df2 = df.copy()  df2["three"]["a"] = 1.0  .. ipython:: python  df  df2  df + df2  df.add(df2, fill_value=0)  .. _basics.compare:  Flexible comparisons
~~~  Series and DataFrame have the binary comparison methods ``eq``, ``ne``, ``lt``, ``gt``,
``le``, and ``ge`` whose behavior is analogous to the binary
arithmetic operations described above:  .. ipython:: python  df.gt(df2)  df2.ne(df)  These operations produce a pandas object of the same type as the left-hand-side
input that is of dtype ``bool``. These ``boolean`` objects can be used in
indexing operations, see the section on `Boolean indexing<indexing.boolean>`.  .. _basics.reductions:  Boolean reductions
~~~  You can apply the reductions: `~DataFrame.empty`, `~DataFrame.any`,
`~DataFrame.all`, and `~DataFrame.bool` to provide a
way to summarize a boolean result.  .. ipython:: python  (df > 0).all()  (df > 0).any()  You can reduce to a final boolean value.  .. ipython:: python  (df > 0).any().any()  You can test if a pandas object is empty, via the `~DataFrame.empty` property.  .. ipython:: python  df.empty  pd.DataFrame(columns=list("ABC")).empty  To evaluate single-element pandas objects in a boolean context, use the method

`~DataFrame.bool`:  .. ipython:: python  pd.Series([True]).bool()  pd.Series([False]).bool()  pd.DataFrame([[True]]).bool()  pd.DataFrame([[False]]).bool()  .. warning::  You might be tempted to do the following:  .. code-block:: python  >>> if df:  ...  pass  Or  .. code-block:: python  >>> df and df2  These will both raise errors, as you are trying to compare multiple values.::  ValueError: The truth value of an array is ambiguous. Use a.empty, a.any() or a.all().  See `gotchas<gotchas.truth>` for a more detailed discussion.  .. _basics.equals:  Comparing if objects are equivalent
~~~  Often you may find that there is more than one way to compute the same
result.  As a simple example, consider ``df + df`` and ``df * 2``. To test
that these two computations produce the same result, given the tools
shown above, you might imagine using ``(df + df == df * 2).all()``. But in
fact, this expression is False:  .. ipython:: python  df + df == df * 2  (df + df == df * 2).all()  Notice that the boolean DataFrame ``df + df == df * 2`` contains some False values!
This is because NaNs do not compare as equals:  .. ipython:: python  np.nan == np.nan  So, NDFrames (such as Series and DataFrames)
have an `~DataFrame.equals` method for testing equality, with NaNs in
corresponding locations treated as equal.  .. ipython:: python  (df + df).equals(df * 2)  Note that the Series or DataFrame index needs to be in the same order for
equality to be True:  .. ipython:: python  df1 = pd.DataFrame({"col": ["foo", 0, np.nan]})  df2 = pd.DataFrame({"col": [np.nan, 0, "foo"]}, index=[2, 1, 0])  df1.equals(df2)  df1.equals(df2.sort_index())  Comparing array-like objects
~~~  You can conveniently perform element-wise comparisons when comparing a pandas
data structure with a scalar value:  .. ipython:: python  pd.Series(["foo", "bar", "baz"]) == "foo"  pd.Index(["foo", "bar", "baz"]) == "foo"  pandas also handles element-wise comparisons between different array-like
objects of the same length:  .. ipython:: python  pd.Series(["foo", "bar", "baz"]) == pd.Index(["foo", "bar", "qux"])  pd.Series(["foo", "bar", "baz"]) == np.array(["foo", "bar", "qux"])  Trying to compare ``Index`` or ``Series`` objects of different lengths will
raise a ValueError:  .. code-block:: ipython  In [55]: pd.Series(['foo', 'bar', 'baz']) == pd.Series(['foo', 'bar'])  ValueError: Series lengths must match to compare  In [56]: pd.Series(['foo', 'bar', 'baz']) == pd.Series(['foo'])  ValueError: Series lengths must match to compare  Note that this is different from the NumPy behavior where a comparison can
be broadcast:  .. ipython:: python  np.array([1, 2, 3]) == np.array([2])  or it can return False if broadcasting can not be done:  .. ipython:: python  np.array([1, 2, 3]) == np.array([1, 2])  Combining overlapping data sets
~~~  A problem occasionally arising is the combination of two similar data sets
where values in one are preferred over the other. An example would be two data
series representing a particular economic indicator where one is considered to
be of "higher quality". However, the lower quality series might extend further
back in history or have more complete data coverage. As such, we would like to
combine two DataFrame objects where missing values in one DataFrame are
conditionally filled with like-labeled values from the other DataFrame. The
function implementing this operation is `~DataFrame.combine_first`,
which we illustrate:  .. ipython:: python  df1 = pd.DataFrame(  {"A": [1.0, np.nan, 3.0, 5.0, np.nan], "B": [np.nan, 2.0, 3.0, np.nan, 6.0]}  )  df2 = pd.DataFrame(  {  "A": [5.0, 2.0, 4.0, np.nan, 3.0, 7.0],  "B": [np.nan, np.nan, 3.0, 4.0, 6.0, 8.0],  }  )  df1  df2  df1.combine_first(df2)  General DataFrame combine
~~~  The `~DataFrame.combine_first` method above calls the more general
`DataFrame.combine`. This method takes another DataFrame
and a combiner function, aligns the input DataFrame and then passes the combiner

function pairs of Series (i.e., columns whose names are the same).  So, for instance, to reproduce `~DataFrame.combine_first` as above:  .. ipython:: python  def combiner(x, y):  return np.where(pd.isna(x), y, x)  df1.combine(df2, combiner)  .. _basics.stats:

There exists a large number of methods for computing descriptive statistics and
other related operations on `Series <api.series.stats>`, `DataFrame
<api.dataframe.stats>`. Most of these
are aggregations (hence producing a lower-dimensional result) like
`~DataFrame.sum`, `~DataFrame.mean`, and `~DataFrame.quantile`,
but some of them, like `~DataFrame.cumsum` and `~DataFrame.cumprod`,
produce an object of the same size. Generally speaking, these methods take an
**axis** argument, just like *ndarray.{sum, std, ...}*, but the axis can be
specified by name or integer:  * **Series**: no axis argument needed
* **DataFrame**: "index" (axis=0, default), "columns" (axis=1)  For example:  .. ipython:: python  df  df.mean(0)  df.mean(1)  All such methods have a ``skipna`` option signaling whether to exclude missing
data (``True`` by default):  .. ipython:: python  df.sum(0, skipna=False)  df.sum(axis=1, skipna=True)  Combined with the broadcasting / arithmetic behavior, one can describe various
statistical procedures, like standardization (rendering data zero mean and
standard deviation of 1), very concisely:  .. ipython:: python  ts_stand = (df - df.mean()) / df.std()  ts_stand.std()  xs_stand = df.sub(df.mean(1), axis=0).div(df.std(1), axis=0)  xs_stand.std(1)  Note that methods like `~DataFrame.cumsum` and `~DataFrame.cumprod`
preserve the location of ``NaN`` values. This is somewhat different from
`~DataFrame.expanding` and `~DataFrame.rolling` since ``NaN`` behavior
is furthermore dictated by a ``min_periods`` parameter.  .. ipython:: python  df.cumsum()  Here is a quick reference summary table of common functions. Each also takes an
optional ``level`` parameter which applies only if the object has a
`hierarchical index<advanced.hierarchical>`.  .. csv-table::  "Function", "Description"  20, 80  ``count``, Number of non-NA observations  ``sum``, Sum of values  ``mean``, Mean of values  ``median``, Arithmetic median of values  ``min``, Minimum  ``max``, Maximum  ``mode``, Mode  ``abs``, Absolute Value  ``prod``, Product of values  ``std``, Bessel-corrected sample standard deviation  ``var``, Unbiased variance  ``sem``, Standard error of the mean  ``skew``, Sample skewness (3rd moment)  ``kurt``, Sample kurtosis (4th moment)  ``quantile``, Sample quantile (value at %)  ``cumsum``, Cumulative sum  ``cumprod``, Cumulative product  ``cummax``, Cumulative maximum  ``cummin``, Cumulative minimum  Note that by chance some NumPy methods, like ``mean``, ``std``, and ``sum``,
will exclude NAs on Series input by default:  .. ipython:: python  np.mean(df["one"])  np.mean(df["one"].to_numpy())  `Series.nunique` will return the number of unique non-NA values in a
Series:  .. ipython:: python  series = pd.Series(np.random.randn(500))  series[20:500] = np.nan  series[10:20] = 5  series.nunique()  .. _basics.describe:  Summarizing data: describe
~~~  There is a convenient `~DataFrame.describe` function which computes a variety of summary
statistics about a Series or the columns of a DataFrame (excluding NAs of
course):  .. ipython:: python  series = pd.Series(np.random.randn(1000))  series[::2] = np.nan  series.describe()  frame = pd.DataFrame(np.random.randn(1000, 5), columns=["a", "b", "c", "d", "e"])  frame.iloc[::2] = np.nan  frame.describe()  You can select specific percentiles to include in the output:  .. ipython:: python  series.describe(percentiles=[0.05, 0.25, 0.75, 0.95])  By default, the median is always included.  For a non-numerical Series object, `~Series.describe` will give a simple
summary of the number of unique values and most frequently occurring values:  .. ipython:: python  s = pd.Series(["a", "a", "b", "b", "a", "a", np.nan, "c", "d", "a"])  s.describe()  Note that on a mixed-type DataFrame object, `~DataFrame.describe` will
restrict the summary to include only numerical columns or, if none are, only

categorical columns:  .. ipython:: python  frame = pd.DataFrame({"a": ["Yes", "Yes", "No", "No"], "b": range(4)})  frame.describe()  This behavior can be controlled by providing a list of types as ``include``/``exclude``
arguments. The special value ``all`` can also be used:  .. ipython:: python  frame.describe(include=["object"])  frame.describe(include=["number"])  frame.describe(include="all")  That feature relies on `select_dtypes <basics.selectdtypes>`. Refer to
there for details about accepted inputs.  .. _basics.idxmin:  Index of min/max values
~~~  The `~DataFrame.idxmin` and `~DataFrame.idxmax` functions on Series
and DataFrame compute the index labels with the minimum and maximum
corresponding values:  .. ipython:: python  s1 = pd.Series(np.random.randn(5))  s1  s1.idxmin(), s1.idxmax()  df1 = pd.DataFrame(np.random.randn(5, 3), columns=["A", "B", "C"])  df1  df1.idxmin(axis=0)  df1.idxmax(axis=1)  When there are multiple rows (or columns) matching the minimum or maximum
value, `~DataFrame.idxmin` and `~DataFrame.idxmax` return the first
matching index:  .. ipython:: python  df3 = pd.DataFrame([2, 1, 1, 3, np.nan], columns=["A"], index=list("edcba"))  df3  df3["A"].idxmin()  .. note::  ``idxmin`` and ``idxmax`` are called ``argmin`` and ``argmax`` in NumPy.  .. _basics.discretization:  Value counts (histogramming) / mode
~~~  The `~Series.value_counts` Series method and top-level function computes a histogram
of a 1D array of values. It can also be used as a function on regular arrays:  .. ipython:: python  data = np.random.randint(0, 7, size=50)  data  s = pd.Series(data)  s.value_counts()  pd.value_counts(data)  .. versionadded:: 1.1.0  The `~DataFrame.value_counts` method can be used to count combinations across multiple columns.
By default all columns are used but a subset can be selected using the ``subset`` argument.  .. ipython:: python  data = {"a": [1, 2, 3, 4], "b": ["x", "x", "y", "y"]}  frame = pd.DataFrame(data)  frame.value_counts()  Similarly, you can get the most frequently occurring value(s), i.e. the mode, of the values in a Series or DataFrame:  .. ipython:: python  s5 = pd.Series([1, 1, 3, 3, 3, 5, 5, 7, 7, 7])  s5.mode()  df5 = pd.DataFrame(  {  "A": np.random.randint(0, 7, size=50),  "B": np.random.randint(-10, 15, size=50),  }  )  df5.mode()  Discretization and quantiling
~~~  Continuous values can be discretized using the `cut` (bins based on values)
and `qcut` (bins based on sample quantiles) functions:  .. ipython:: python  arr = np.random.randn(20)  factor = pd.cut(arr, 4)  factor  factor = pd.cut(arr, [-5, -1, 0, 1, 5])  factor  `qcut` computes sample quantiles. For example, we could slice up some
normally distributed data into equal-size quartiles like so:  .. ipython:: python  arr = np.random.randn(30)  factor = pd.qcut(arr, [0, 0.25, 0.5, 0.75, 1])  factor  pd.value_counts(factor)  We can also pass infinite values to define the bins:  .. ipython:: python  arr = np.random.randn(20)  factor = pd.cut(arr, [-np.inf, 0, np.inf])  factor  .. _basics.apply:

To apply your own or another library's functions to pandas objects,
you should be aware of the three methods below. The appropriate
method to use depends on whether your function expects to operate
on an entire ``DataFrame`` or ``Series``, row- or column-wise, or elementwise.  1. `Tablewise Function Application`_: `~DataFrame.pipe`
2. `Row or Column-wise Function Application`_: `~DataFrame.apply`
3. `Aggregation API`_: `~DataFrame.agg` and `~DataFrame.transform`
4. `Applying Elementwise Functions`_: `~DataFrame.applymap`  .. _basics.pipe:  Tablewise function application
~~~  ``DataFrames`` and ``Series`` can be passed into functions.
However, if the function needs to be called in a chain, consider using the `~DataFrame.pipe` method.  First some setup:  .. ipython:: python  def extract_city_name(df):  """  Chicago, IL -> Chicago for city_name column  """  df["city_name"] = df["city_and_code"].str.split(",").str.get(0)  return df  def add_country_name(df, country_name=None):  """  Chicago -> Chicago-US for city_name column  """  col = "city_name"  df["city_and_country"] = df[col] + country_name  return df  df_p = pd.DataFrame({"city_and_code": ["Chicago, IL"]})  ``extract_city_name`` and ``add_country_name`` are functions taking and returning ``DataFrames``.  Now compare the following:  .. ipython:: python  add_country_name(extract_city_name(df_p), country_name="US")  Is equivalent to:  .. ipython:: python  df_p.pipe(extract_city_name).pipe(add_country_name, country_name="US")  pandas encourages the second style, which is known as method chaining.
``pipe`` makes it easy to use your own or another library's functions
in method chains, alongside pandas' methods.  In the example above, the functions ``extract_city_name`` and ``add_country_name`` each expected a ``DataFrame`` as the first positional argument.
What if the function you wish to apply takes its data as, say, the second argument?
In this case, provide ``pipe`` with a tuple of ``(callable, data_keyword)``.
``.pipe`` will route the ``DataFrame`` to the argument specified in the tuple.  For example, we can fit a regression using statsmodels. Their API expects a formula first and a ``DataFrame`` as the second argument, ``data``. We pass in the function, keyword pair ``(sm.ols, 'data')`` to ``pipe``:  .. code-block:: ipython  In [147]: import statsmodels.formula.api as sm  In [148]: bb = pd.read_csv("data/baseball.csv", index_col="id")  In [149]: (  ...:  bb.query("h > 0")  ...:  .assign(ln_h=lambda df: np.log(df.h))  ...:  .pipe((sm.ols, "data"), "hr ~ ln_h + year + g + C(lg)")  ...:  .fit()  ...:  .summary()  ...: )  ...:  Out[149]:  <class 'statsmodels.iolib.summary.Summary'>  """

OLS Regression Results
  Dep. Variable:  hr  R-squared:  0.685  Model:  OLS  Adj. R-squared:  0.665  Method:  Least Squares  F-statistic:  34.28  Date:  Tue, 22 Nov 2022  Prob (F-statistic):  3.48e-15  Time:  0517  Log-Likelihood:  -205.92  No. Observations:  68  AIC:  421.8  Df Residuals:  63  BIC:  432.9  Df Model:  4

coef    std err          t      P>|t|      [0.025      0.975]
  Intercept  -8484.7720  4664.146  -1.819  0.074  -1.78e+04  835.780  C(lg)[T.NL]  -2.2736  1.325  -1.716  0.091  -4.922  0.375  ln_h  -1.3542  0.875  -1.547  0.127  -3.103  0.395  year  4.2277  2.324  1.819  0.074  -0.417  8.872

g               0.1841      0.029      6.258      0.000       0.125       0.243
  Omnibus:  10.875  Durbin-Watson:  1.999  Prob(Omnibus):  0.004  Jarque-Bera (JB):  17.298  Skew:  0.537  Prob(JB):  0.000175

Kurtosis:                       5.225   Cond. No.                     1.49e+07
  Notes:  [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.  [2] The condition number is large, 1.49e+07. This might indicate that there are  strong multicollinearity or other numerical problems.  """  The pipe method is inspired by unix pipes and more recently dplyr_ and magrittr_, which
have introduced the popular ``(%>%)`` (read pipe) operator for R_.
The implementation of ``pipe`` here is quite clean and feels right at home in Python.
We encourage you to view the source code of `~DataFrame.pipe`.  .. _dplyr: https://github.com/tidyverse/dplyr
.. _magrittr: https://github.com/tidyverse/magrittr
.. _R: https://www.r-project.org  Row or column-wise function application
~~~  Arbitrary functions can be applied along the axes of a DataFrame
using the `~DataFrame.apply` method, which, like the descriptive
statistics methods, takes an optional ``axis`` argument:  .. ipython:: python  df.apply(np.mean)  df.apply(np.mean, axis=1)  df.apply(lambda x: x.max() - x.min())  df.apply(np.cumsum)  df.apply(np.exp)  The `~DataFrame.apply` method will also dispatch on a string method name.  .. ipython:: python  df.apply("mean")  df.apply("mean", axis=1)  The return type of the function passed to `~DataFrame.apply` affects the
type of the final output from ``DataFrame.apply`` for the default behaviour:  * If the applied function returns a ``Series``, the final output is a ``DataFrame``.  The columns match the index of the ``Series`` returned by the applied function.
* If the applied function returns any other type, the final output is a ``Series``.  This default behaviour can be overridden using the ``result_type``, which
accepts three options: ``reduce``, ``broadcast``, and ``expand``.
These will determine how list-likes return values expand (or not) to a ``DataFrame``.  `~DataFrame.apply` combined with some cleverness can be used to answer many questions
about a data set. For example, suppose we wanted to extract the date where the
maximum value for each column occurred:  .. ipython:: python  tsdf = pd.DataFrame(  np.random.randn(1000, 3),  columns=["A", "B", "C"],  index=pd.date_range("1/1/2000", periods=1000),  )  tsdf.apply(lambda x: x.idxmax())  You may also pass additional arguments and keyword arguments to the `~DataFrame.apply`
method. For instance, consider the following function you would like to apply:  .. code-block:: python  def subtract_and_divide(x, sub, divide=1):  return (x - sub) / divide  You may then apply this function as follows:  .. code-block:: python  df.apply(subtract_and_divide, args=(5,), divide=3)  Another useful feature is the ability to pass Series methods to carry out some
Series operation on each column or row:  .. ipython:: python  tsdf = pd.DataFrame(  np.random.randn(10, 3),  columns=["A", "B", "C"],  index=pd.date_range("1/1/2000", periods=10),  )  tsdf.iloc[3:7] = np.nan  .. ipython:: python  tsdf  tsdf.apply(pd.Series.interpolate)  Finally, `~DataFrame.apply` takes an argument ``raw`` which is False by default, which
converts each row or column into a Series before applying the function. When
set to True, the passed function will instead receive an ndarray object, which
has positive performance implications if you do not need the indexing
functionality.  .. _basics.aggregate:  Aggregation API
~~~  The aggregation API allows one to express possibly multiple aggregation operations in a single concise way.
This API is similar across pandas objects, see `groupby API <groupby.aggregate>`, the
`window API <window.overview>`, and the `resample API <timeseries.aggregate>`.
The entry point for aggregation is `DataFrame.aggregate`, or the alias

`DataFrame.agg`.  We will use a similar starting frame from above:  .. ipython:: python  tsdf = pd.DataFrame(  np.random.randn(10, 3),  columns=["A", "B", "C"],  index=pd.date_range("1/1/2000", periods=10),  )  tsdf.iloc[3:7] = np.nan  tsdf  Using a single function is equivalent to `~DataFrame.apply`. You can also
pass named methods as strings. These will return a ``Series`` of the aggregated
output:  .. ipython:: python  tsdf.agg(np.sum)  tsdf.agg("sum")  # these are equivalent to a ``.sum()`` because we are aggregating  # on a single function  tsdf.sum()  Single aggregations on a ``Series`` this will return a scalar value:  .. ipython:: python  tsdf["A"].agg("sum")  Aggregating with multiple functions
+++  You can pass multiple aggregation arguments as a list.
The results of each of the passed functions will be a row in the resulting ``DataFrame``.
These are naturally named from the aggregation function.  .. ipython:: python  tsdf.agg(["sum"])  Multiple functions yield multiple rows:  .. ipython:: python  tsdf.agg(["sum", "mean"])  On a ``Series``, multiple functions return a ``Series``, indexed by the function names:  .. ipython:: python  tsdf["A"].agg(["sum", "mean"])  Passing a ``lambda`` function will yield a ``<lambda>`` named row:  .. ipython:: python  tsdf["A"].agg(["sum", lambda x: x.mean()])  Passing a named function will yield that name for the row:  .. ipython:: python  def mymean(x):  return x.mean()  tsdf["A"].agg(["sum", mymean])  Aggregating with a dict
+++  Passing a dictionary of column names to a scalar or a list of scalars, to ``DataFrame.agg``
allows you to customize which functions are applied to which columns. Note that the results
are not in any particular order, you can use an ``OrderedDict`` instead to guarantee ordering.  .. ipython:: python  tsdf.agg({"A": "mean", "B": "sum"})  Passing a list-like will generate a ``DataFrame`` output. You will get a matrix-like output
of all of the aggregators. The output will consist of all unique functions. Those that are
not noted for a particular column will be ``NaN``:  .. ipython:: python  tsdf.agg({"A": ["mean", "min"], "B": "sum"})  .. _basics.aggregation.custom_describe:  Custom describe
+++  With ``.agg()`` it is possible to easily create a custom describe function, similar
to the built in `describe function <basics.describe>`.  .. ipython:: python  from functools import partial  q_25 = partial(pd.Series.quantile, q=0.25)  q_25.__name__ = "25%"  q_75 = partial(pd.Series.quantile, q=0.75)  q_75.__name__ = "75%"  tsdf.agg(["count", "mean", "std", "min", q_25, "median", q_75, "max"])  .. _basics.transform:  Transform API
~~~  The `~DataFrame.transform` method returns an object that is indexed the same (same size)
as the original. This API allows you to provide *multiple* operations at the same
time rather than one-by-one. Its API is quite similar to the ``.agg`` API.  We create a frame similar to the one used in the above sections.  .. ipython:: python  tsdf = pd.DataFrame(  np.random.randn(10, 3),  columns=["A", "B", "C"],  index=pd.date_range("1/1/2000", periods=10),  )  tsdf.iloc[3:7] = np.nan  tsdf  Transform the entire frame. ``.transform()`` allows input functions as: a NumPy function, a string
function name or a user defined function.  .. ipython:: python  tsdf.transform(np.abs)  tsdf.transform("abs")  tsdf.transform(lambda x: x.abs())  Here `~DataFrame.transform` received a single function; this is equivalent to a `ufunc
<https://numpy.org/doc/stable/reference/ufuncs.html>`__ application.  .. ipython:: python  np.abs(tsdf)  Passing a single function to ``.transform()`` with a ``Series`` will yield a single ``Series`` in return.  .. ipython:: python  tsdf["A"].transform(np.abs)  Transform with multiple functions
+++  Passing multiple functions will yield a column MultiIndexed DataFrame.

The first level will be the original frame column names; the second level
will be the names of the transforming functions.  .. ipython:: python  tsdf.transform([np.abs, lambda x: x + 1])  Passing multiple functions to a Series will yield a DataFrame. The
resulting column names will be the transforming functions.  .. ipython:: python  tsdf["A"].transform([np.abs, lambda x: x + 1])  Transforming with a dict
+++  Passing a dict of functions will allow selective transforming per column.  .. ipython:: python  tsdf.transform({"A": np.abs, "B": lambda x: x + 1})  Passing a dict of lists will generate a MultiIndexed DataFrame with these
selective transforms.  .. ipython:: python  tsdf.transform({"A": np.abs, "B": [lambda x: x + 1, "sqrt"]})  .. _basics.elementwise:  Applying elementwise functions
~~~  Since not all functions can be vectorized (accept NumPy arrays and return
another array or value), the methods `~DataFrame.applymap` on DataFrame
and analogously `~Series.map` on Series accept any Python function taking
a single value and returning a single value. For example:  .. ipython:: python  df4 = df_orig.copy()  .. ipython:: python  df4  def f(x):  return len(str(x))  df4["one"].map(f)  df4.applymap(f)  `Series.map` has an additional feature; it can be used to easily
"link" or "map" values defined by a secondary series. This is closely related
to `merging/joining functionality <merging>`:  .. ipython:: python  s = pd.Series(  ["six", "seven", "six", "seven", "six"], index=["a", "b", "c", "d", "e"]  )  t = pd.Series({"six": 6.0, "seven": 7.0})  s  s.map(t)  .. _basics.reindexing:

`~Series.reindex` is the fundamental data alignment method in pandas.
It is used to implement nearly all other features relying on label-alignment
functionality. To *reindex* means to conform the data to match a given set of
labels along a particular axis. This accomplishes several things:  * Reorders the existing data to match a new set of labels
* Inserts missing value (NA) markers in label locations where no data for  that label existed
* If specified, **fill** data for missing labels using logic (highly relevant  to working with time series data)  Here is a simple example:  .. ipython:: python  s = pd.Series(np.random.randn(5), index=["a", "b", "c", "d", "e"])  s  s.reindex(["e", "b", "f", "d"])  Here, the ``f`` label was not contained in the Series and hence appears as
``NaN`` in the result.  With a DataFrame, you can simultaneously reindex the index and columns:  .. ipython:: python  df  df.reindex(index=["c", "f", "b"], columns=["three", "two", "one"])  Note that the ``Index`` objects containing the actual axis labels can be
**shared** between objects. So if we have a Series and a DataFrame, the
following can be done:  .. ipython:: python  rs = s.reindex(df.index)  rs  rs.index is df.index  This means that the reindexed Series's index is the same Python object as the
DataFrame's index.  `DataFrame.reindex` also supports an "axis-style" calling convention,
where you specify a single ``labels`` argument and the ``axis`` it applies to.  .. ipython:: python  df.reindex(["c", "f", "b"], axis="index")  df.reindex(["three", "two", "one"], axis="columns")  .. seealso::  `MultiIndex / Advanced Indexing <advanced>` is an even more concise way of  doing reindexing.  .. note::  When writing performance-sensitive code, there is a good reason to spend  some time becoming a reindexing ninja: **many operations are faster on  pre-aligned data**. Adding two unaligned DataFrames internally triggers a  reindexing step. For exploratory analysis you will hardly notice the  difference (because ``reindex`` has been heavily optimized), but when CPU  cycles matter sprinkling a few explicit ``reindex`` calls here and there can  have an impact.  .. _basics.reindex_like:  Reindexing to align with another object
~~~  You may wish to take an object and reindex its axes to be labeled the same as
another object. While the syntax for this is straightforward albeit verbose, it
is a common enough operation that the `~DataFrame.reindex_like` method is
available to make this simpler:  .. ipython:: python  df2 = df.reindex(["a", "b", "c"], columns=["one", "two"])  df3 = df2 - df2.mean()  .. ipython:: python  df2  df3  df.reindex_like(df2)  .. _basics.align:  Aligning objects with each other with ``align``
~~~  The `~Series.align` method is the fastest way to simultaneously align two objects. It
supports a ``join`` argument (related to `joining and merging <merging>`):  - ``join='outer'``: take the union of the indexes (default)  - ``join='left'``: use the calling object's index  - ``join='right'``: use the passed object's index  - ``join='inner'``: intersect the indexes  It returns a tuple with both of the reindexed Series:  .. ipython:: python  s = pd.Series(np.random.randn(5), index=["a", "b", "c", "d", "e"])  s1 = s[:4]  s2 = s[1:]  s1.align(s2)  s1.align(s2, join="inner")  s1.align(s2, join="left")  .. _basics.df_join:  For DataFrames, the join method will be applied to both the index and the
columns by default:  .. ipython:: python  df.align(df2, join="inner")  You can also pass an ``axis`` option to only align on the specified axis:  .. ipython:: python  df.align(df2, join="inner", axis=0)  .. _basics.align.frame.series:  If you pass a Series to `DataFrame.align`, you can choose to align both
objects either on the DataFrame's index or columns using the ``axis`` argument:  .. ipython:: python  df.align(df2.iloc[0], axis=1)  .. _basics.reindex_fill:  Filling while reindexing
~~~  `~Series.reindex` takes an optional parameter ``method`` which is a

filling method chosen from the following table:  .. csv-table::  "Method", "Action"  30, 50  pad / ffill, Fill values forward  bfill / backfill, Fill values backward  nearest, Fill from the nearest index value  We illustrate these fill methods on a simple Series:  .. ipython:: python  rng = pd.date_range("1/3/2000", periods=8)  ts = pd.Series(np.random.randn(8), index=rng)  ts2 = ts[[0, 3, 6]]  ts  ts2  ts2.reindex(ts.index)  ts2.reindex(ts.index, method="ffill")  ts2.reindex(ts.index, method="bfill")  ts2.reindex(ts.index, method="nearest")  These methods require that the indexes are **ordered** increasing or
decreasing.  Note that the same result could have been achieved using
`fillna <missing_data.fillna>` (except for ``method='nearest'``) or
`interpolate <missing_data.interpolate>`:  .. ipython:: python  ts2.reindex(ts.index).fillna(method="ffill")  `~Series.reindex` will raise a ValueError if the index is not monotonically
increasing or decreasing. `~Series.fillna` and `~Series.interpolate`
will not perform any checks on the order of the index.  .. _basics.limits_on_reindex_fill:  Limits on filling while reindexing
~~~  The ``limit`` and ``tolerance`` arguments provide additional control over
filling while reindexing. Limit specifies the maximum count of consecutive
matches:  .. ipython:: python  ts2.reindex(ts.index, method="ffill", limit=1)  In contrast, tolerance specifies the maximum distance between the index and
indexer values:  .. ipython:: python  ts2.reindex(ts.index, method="ffill", tolerance="1 day")  Notice that when used on a ``DatetimeIndex``, ``TimedeltaIndex`` or
``PeriodIndex``, ``tolerance`` will coerced into a ``Timedelta`` if possible.
This allows you to specify tolerance with appropriate strings.  .. _basics.drop:  Dropping labels from an axis
~~~  A method closely related to ``reindex`` is the `~DataFrame.drop` function.
It removes a set of labels from an axis:  .. ipython:: python  df  df.drop(["a", "d"], axis=0)  df.drop(["one"], axis=1)  Note that the following also works, but is a bit less obvious / clean:  .. ipython:: python  df.reindex(df.index.difference(["a", "d"]))  .. _basics.rename:  Renaming / mapping labels
~~~  The `~DataFrame.rename` method allows you to relabel an axis based on some
mapping (a dict or Series) or an arbitrary function.  .. ipython:: python  s  s.rename(str.upper)  If you pass a function, it must return a value when called with any of the
labels (and must produce a set of unique values). A dict or
Series can also be used:  .. ipython:: python  df.rename(  columns={"one": "foo", "two": "bar"},  index={"a": "apple", "b": "banana", "d": "durian"},  )  If the mapping doesn't include a column/index label, it isn't renamed. Note that
extra labels in the mapping don't throw an error.  `DataFrame.rename` also supports an "axis-style" calling convention, where
you specify a single ``mapper`` and the ``axis`` to apply that mapping to.  .. ipython:: python  df.rename({"one": "foo", "two": "bar"}, axis="columns")  df.rename({"a": "apple", "b": "banana", "d": "durian"}, axis="index")  Finally, `~Series.rename` also accepts a scalar or list-like
for altering the ``Series.name`` attribute.  .. ipython:: python  s.rename("scalar-name")  .. _basics.rename_axis:  The methods `DataFrame.rename_axis` and `Series.rename_axis`
allow specific names of a ``MultiIndex`` to be changed (as opposed to the
labels).  .. ipython:: python  df = pd.DataFrame(  {"x": [1, 2, 3, 4, 5, 6], "y": [10, 20, 30, 40, 50, 60]},  index=pd.MultiIndex.from_product(  [["a", "b", "c"], [1, 2]], names=["let", "num"]  ),  )  df  df.rename_axis(index={"let": "abc"})  df.rename_axis(index=str.upper)  .. _basics.iteration:

The behavior of basic iteration over pandas objects depends on the type.
When iterating over a Series, it is regarded as array-like, and basic iteration
produces the values. DataFrames follow the dict-like convention of iterating
over the "keys" of the objects.  In short, basic iteration (``for i in object``) produces:  * **Series**: values
* **DataFrame**: column labels  Thus, for example, iterating over a DataFrame gives you the column names:  .. ipython:: python  df = pd.DataFrame(  {"col1": np.random.randn(3), "col2": np.random.randn(3)}, index=["a", "b", "c"]  )  for col in df:  print(col)  pandas objects also have the dict-like `~DataFrame.items` method to
iterate over the (key, value) pairs.  To iterate over the rows of a DataFrame, you can use the following methods:  * `~DataFrame.iterrows`: Iterate over the rows of a DataFrame as (index, Series) pairs.  This converts the rows to Series objects, which can change the dtypes and has some  performance implications.
* `~DataFrame.itertuples`: Iterate over the rows of a DataFrame  as namedtuples of the values.  This is a lot faster than  `~DataFrame.iterrows`, and is in most cases preferable to use  to iterate over the values of a DataFrame.  .. warning::  Iterating through pandas objects is generally **slow**. In many cases,  iterating manually over the rows is not needed and can be avoided with  one of the following approaches:  * Look for a *vectorized* solution: many operations can be performed using  built-in methods or NumPy functions, (boolean) indexing, ...  * When you have a function that cannot work on the full DataFrame/Series  at once, it is better to use `~DataFrame.apply` instead of iterating  over the values. See the docs on `function application <basics.apply>`.  * If you need to do iterative manipulations on the values but performance is  important, consider writing the inner loop with cython or numba.  See the `enhancing performance <enhancingperf>` section for some  examples of this approach.  .. warning::  You should **never modify** something you are iterating over.  This is not guaranteed to work in all cases. Depending on the  data types, the iterator returns a copy and not a view, and writing  to it will have no effect!  For example, in the following case setting the value has no effect:  .. ipython:: python  df = pd.DataFrame({"a": [1, 2, 3], "b": ["a", "b", "c"]})  for index, row in df.iterrows():  row["a"] = 10  df  items
~~~  Consistent with the dict-like interface, `~DataFrame.items` iterates
through key-value pairs:  * **Series**: (index, scalar value) pairs
* **DataFrame**: (column, Series) pairs  For example:  .. ipython:: python  for label, ser in df.items():  print(label)  print(ser)  .. _basics.iterrows:  iterrows
~~~  `~DataFrame.iterrows` allows you to iterate through the rows of a
DataFrame as Series objects. It returns an iterator yielding each

index value along with a Series containing the data in each row:  .. ipython:: python  for row_index, row in df.iterrows():  print(row_index, row, sep="\n")  .. note::  Because `~DataFrame.iterrows` returns a Series for each row,  it does **not** preserve dtypes across the rows (dtypes are  preserved across columns for DataFrames). For example,  .. ipython:: python  df_orig = pd.DataFrame([[1, 1.5]], columns=["int", "float"])  df_orig.dtypes  row = next(df_orig.iterrows())[1]  row  All values in ``row``, returned as a Series, are now upcasted  to floats, also the original integer value in column ``x``:  .. ipython:: python  row["int"].dtype  df_orig["int"].dtype  To preserve dtypes while iterating over the rows, it is better  to use `~DataFrame.itertuples` which returns namedtuples of the values  and which is generally much faster than `~DataFrame.iterrows`.  For instance, a contrived way to transpose the DataFrame would be:  .. ipython:: python  df2 = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})  print(df2)  print(df2.T)  df2_t = pd.DataFrame({idx: values for idx, values in df2.iterrows()})  print(df2_t)  itertuples
~~~  The `~DataFrame.itertuples` method will return an iterator
yielding a namedtuple for each row in the DataFrame. The first element
of the tuple will be the row's corresponding index value, while the
remaining values are the row values.  For instance:  .. ipython:: python  for row in df.itertuples():  print(row)  This method does not convert the row to a Series object; it merely
returns the values inside a namedtuple. Therefore,
`~DataFrame.itertuples` preserves the data type of the values
and is generally faster as `~DataFrame.iterrows`.  .. note::  The column names will be renamed to positional names if they are  invalid Python identifiers, repeated, or start with an underscore.  With a large number of columns (>255), regular tuples are returned.  .. _basics.dt_accessors:

.dt accessor

``Series`` has an accessor to succinctly return datetime like properties for the
*values* of the Series, if it is a datetime/period like Series.
This will return a Series, indexed like the existing Series.  .. ipython:: python  # datetime  s = pd.Series(pd.date_range("20130101 0912", periods=4))  s  s.dt.hour  s.dt.second  s.dt.day  This enables nice expressions like this:  .. ipython:: python  s[s.dt.day == 2]  You can easily produces tz aware transformations:  .. ipython:: python  stz = s.dt.tz_localize("US/Eastern")  stz  stz.dt.tz  You can also chain these types of operations:  .. ipython:: python  s.dt.tz_localize("UTC").dt.tz_convert("US/Eastern")  You can also format datetime values as strings with `Series.dt.strftime` which
supports the same format as the standard `~datetime.datetime.strftime`.  .. ipython:: python  # DatetimeIndex  s = pd.Series(pd.date_range("20130101", periods=4))  s  s.dt.strftime("%Y/%m/%d")  .. ipython:: python  # PeriodIndex  s = pd.Series(pd.period_range("20130101", periods=4))  s  s.dt.strftime("%Y/%m/%d")  The ``.dt`` accessor works for period and timedelta dtypes.  .. ipython:: python  # period  s = pd.Series(pd.period_range("20130101", periods=4, freq="D"))  s  s.dt.year  s.dt.day  .. ipython:: python  # timedelta  s = pd.Series(pd.timedelta_range("1 day 0005", periods=4, freq="s"))  s  s.dt.days  s.dt.seconds  s.dt.components  .. note::  ``Series.dt`` will raise a ``TypeError`` if you access with a non-datetime-like values.

Vectorized string methods

Series is equipped with a set of string processing methods that make it easy to
operate on each element of the array. Perhaps most importantly, these methods
exclude missing/NA values automatically. These are accessed via the Series's
``str`` attribute and generally have names matching the equivalent (scalar)
built-in string methods. For example:  .. ipython:: python  s = pd.Series(  ["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"], dtype="string"  )  s.str.lower()  Powerful pattern-matching methods are provided as well, but note that
pattern-matching generally uses `regular expressions
<https://docs.python.org/3/library/re.html>`__ by default (and in some cases
always uses them).  .. note::  Prior to pandas 1.0, string methods were only available on ``object`` -dtype  ``Series``. pandas 1.0 added the `StringDtype` which is dedicated  to strings. See `text.types` for more.  Please see `Vectorized String Methods <text.string_methods>` for a complete
description.  .. _basics.sorting:

pandas supports three kinds of sorting: sorting by index labels,
sorting by column values, and sorting by a combination of both.  .. _basics.sort_index:  By index
~~~  The `Series.sort_index` and `DataFrame.sort_index` methods are
used to sort a pandas object by its index levels.  .. ipython:: python  df = pd.DataFrame(  {  "one": pd.Series(np.random.randn(3), index=["a", "b", "c"]),  "two": pd.Series(np.random.randn(4), index=["a", "b", "c", "d"]),  "three": pd.Series(np.random.randn(3), index=["b", "c", "d"]),  }  )  unsorted_df = df.reindex(  index=["a", "d", "c", "b"], columns=["three", "two", "one"]  )  unsorted_df  # DataFrame  unsorted_df.sort_index()  unsorted_df.sort_index(ascending=False)  unsorted_df.sort_index(axis=1)  # Series  unsorted_df["three"].sort_index()  .. _basics.sort_index_key:  .. versionadded:: 1.1.0  Sorting by index also supports a ``key`` parameter that takes a callable
function to apply to the index being sorted. For ``MultiIndex`` objects,
the key is applied per-level to the levels specified by ``level``.  .. ipython:: python  s1 = pd.DataFrame({"a": ["B", "a", "C"], "b": [1, 2, 3], "c": [2, 3, 4]}).set_index(  list("ab")  )  s1  .. ipython:: python  s1.sort_index(level="a")  s1.sort_index(level="a", key=lambda idx: idx.str.lower())  For information on key sorting by value, see `value sorting
<basics.sort_value_key>`.  .. _basics.sort_values:  By values
~~~  The `Series.sort_values` method is used to sort a ``Series`` by its values. The
`DataFrame.sort_values` method is used to sort a ``DataFrame`` by its column or row values.
The optional ``by`` parameter to `DataFrame.sort_values` may used to specify one or more columns
to use to determine the sorted order.  .. ipython:: python  df1 = pd.DataFrame(  {"one": [2, 1, 1, 1], "two": [1, 3, 2, 4], "three": [5, 4, 3, 2]}  )  df1.sort_values(by="two")  The ``by`` parameter can take a list of column names, e.g.:  .. ipython:: python  df1[["one", "two", "three"]].sort_values(by=["one", "two"])  These methods have special treatment of NA values via the ``na_position``
argument:  .. ipython:: python  s[2] = np.nan  s.sort_values()  s.sort_values(na_position="first")  .. _basics.sort_value_key:  .. versionadded:: 1.1.0  Sorting also supports a ``key`` parameter that takes a callable function
to apply to the values being sorted.  .. ipython:: python  s1 = pd.Series(["B", "a", "C"])  .. ipython:: python  s1.sort_values()  s1.sort_values(key=lambda x: x.str.lower())  ``key`` will be given the `Series` of values and should return a ``Series``
or array of the same shape with the transformed values. For ``DataFrame`` objects,
the key is applied per column, so the key should still expect a Series and return
a Series, e.g.  .. ipython:: python  df = pd.DataFrame({"a": ["B", "a", "C"], "b": [1, 2, 3]})  .. ipython:: python  df.sort_values(by="a")  df.sort_values(by="a", key=lambda col: col.str.lower())  The name or type of each column can be used to apply different functions to
different columns.  .. _basics.sort_indexes_and_values:  By indexes and values
~~~  Strings passed as the ``by`` parameter to `DataFrame.sort_values` may
refer to either columns or index level names.  .. ipython:: python  # Build MultiIndex  idx = pd.MultiIndex.from_tuples(  [("a", 1), ("a", 2), ("a", 2), ("b", 2), ("b", 1), ("b", 1)]  )  idx.names = ["first", "second"]  # Build DataFrame  df_multi = pd.DataFrame({"A": np.arange(6, 0, -1)}, index=idx)  df_multi  Sort by 'second' (index) and 'A' (column)  .. ipython:: python  df_multi.sort_values(by=["second", "A"])  .. note::  If a string matches both a column name and an index level name then a  warning is issued and the column takes precedence. This will result in an  ambiguity error in a future version.  .. _basics.searchsorted:  searchsorted
~~~  Series has the `~Series.searchsorted` method, which works similarly to

`numpy.ndarray.searchsorted`.  .. ipython:: python  ser = pd.Series([1, 2, 3])  ser.searchsorted([0, 3])  ser.searchsorted([0, 4])  ser.searchsorted([1, 3], side="right")  ser.searchsorted([1, 3], side="left")  ser = pd.Series([3, 1, 2])  ser.searchsorted([0, 3], sorter=np.argsort(ser))  .. _basics.nsorted:  smallest / largest values
~~~  ``Series`` has the `~Series.nsmallest` and `~Series.nlargest` methods which return the
smallest or largest `n` values. For a large ``Series`` this can be much
faster than sorting the entire Series and calling ``head(n)`` on the result.  .. ipython:: python  s = pd.Series(np.random.permutation(10))  s  s.sort_values()  s.nsmallest(3)  s.nlargest(3)  ``DataFrame`` also has the ``nlargest`` and ``nsmallest`` methods.  .. ipython:: python  df = pd.DataFrame(  {  "a": [-2, -1, 1, 10, 8, 11, -1],  "b": list("abdceff"),  "c": [1.0, 2.0, 4.0, 3.2, np.nan, 3.0, 4.0],  }  )  df.nlargest(3, "a")  df.nlargest(5, ["a", "c"])  df.nsmallest(3, "a")  df.nsmallest(5, ["a", "c"])  .. _basics.multiindex_sorting:  Sorting by a MultiIndex column
~~~  You must be explicit about sorting when the column is a MultiIndex, and fully specify
all levels to ``by``.  .. ipython:: python  df1.columns = pd.MultiIndex.from_tuples(  [("a", "one"), ("a", "two"), ("b", "three")]  )  df1.sort_values(by=("a", "two"))

Copying

The `~DataFrame.copy` method on pandas objects copies the underlying data (though not
the axis indexes, since they are immutable) and returns a new object. Note that
**it is seldom necessary to copy objects**. For example, there are only a
handful of ways to alter a DataFrame *in-place*:  * Inserting, deleting, or modifying a column.
* Assigning to the ``index`` or ``columns`` attributes.
* For homogeneous data, directly modifying the values via the ``values``  attribute or advanced indexing.  To be clear, no pandas method has the side effect of modifying your data;
almost every method returns a new object, leaving the original object
untouched. If the data is modified, it is because you did so explicitly.  .. _basics.dtypes:

For the most part, pandas uses NumPy arrays and dtypes for Series or individual
columns of a DataFrame. NumPy provides support for ``float``,
``int``, ``bool``, ``timedelta64[ns]`` and ``datetime64[ns]`` (note that NumPy
does not support timezone-aware datetimes).  pandas and third-party libraries *extend* NumPy's type system in a few places.
This section describes the extensions pandas has made internally.
See `extending.extension-types` for how to write your own extension that
works with pandas. See `ecosystem.extensions` for a list of third-party
libraries that have implemented an extension.  The following table lists all of pandas extension types. For methods requiring ``dtype``
arguments, strings can be specified as indicated. See the respective
documentation sections for more on each type.  +---+---+---+---+---+
| Kind of Data  | Data Type  | Scalar  | Array  | String Aliases  |
+===+===+===+===+===+===+===+
| `tz-aware datetime <timeseries.timezone>`  | `DatetimeTZDtype`  | `Timestamp` | `arrays.DatetimeArray` | ``'datetime64[ns, <tz>]'``  |
|  |  |  |  |  |
+---+---+---+---+---+---+
| `Categorical <categorical>`  | `CategoricalDtype` | (none)  | `Categorical`  | ``'category'``  |
+---+---+---+---+---+
| `period (time spans) <timeseries.periods>` | `PeriodDtype`  | `Period`  | `arrays.PeriodArray`  | ``'period[<freq>]'``,  |
|  |  |  | ``'Period[<freq>]'``  |  |
| `sparse <sparse>`  | `SparseDtype`  | (none)  | `arrays.SparseArray`  | ``'Sparse'``, ``'Sparse[int]'``,  |
|  |  |  |  | ``'Sparse[float]'``  |
| `intervals <advanced.intervalindex>`  | `IntervalDtype`  | `Interval`  | `arrays.IntervalArray` | ``'interval'``, ``'Interval'``,  |
|  |  |  |  | ``'Interval[<numpy_dtype>]'``,  |
|  |  |  |  | ``'Interval[datetime64[ns, <tz>]]'``,  |
|  |  |  |  | ``'Interval[timedelta64[<freq>]]'``  |
| `nullable integer <integer_na>`  | `Int64Dtype`, ...  | (none)  | `arrays.IntegerArray`  | ``'Int8'``, ``'Int16'``, ``'Int32'``,  |
|  |  |  |  | ``'Int64'``, ``'UInt8'``, ``'UInt16'``,|
|  |  |  |  | ``'UInt32'``, ``'UInt64'``  |
| `Strings <text>`  | `StringDtype`  | `str`  | `arrays.StringArray`  | ``'string'``  |
| `Boolean (with NA) <api.arrays.bool>`  | `BooleanDtype`  | `bool`  | `arrays.BooleanArray`  | ``'boolean'``  |
+---+---+---+---+---+  pandas has two ways to store strings.  1. ``object`` dtype, which can hold any Python object, including strings.
2. `StringDtype`, which is dedicated to strings.  Generally, we recommend using `StringDtype`. See `text.types` for more.  Finally, arbitrary objects may be stored using the ``object`` dtype, but should
be avoided to the extent possible (for performance and interoperability with
other libraries and methods. See `basics.object_conversion`).  A convenient `~DataFrame.dtypes` attribute for DataFrame returns a Series
with the data type of each column.  .. ipython:: python  dft = pd.DataFrame(  {  "A": np.random.rand(3),  "B": 1,  "C": "foo",  "D": pd.Timestamp("20010102"),  "E": pd.Series([1.0] * 3).astype("float32"),  "F": False,  "G": pd.Series([1] * 3, dtype="int8"),  }  )  dft  dft.dtypes  On a ``Series`` object, use the `~Series.dtype` attribute.  .. ipython:: python  dft["A"].dtype  If a pandas object contains data with multiple dtypes *in a single column*, the
dtype of the column will be chosen to accommodate all of the data types
(``object`` is the most general).  .. ipython:: python  # these ints are coerced to floats  pd.Series([1, 2, 3, 4, 5, 6.0])  # string data forces an ``object`` dtype  pd.Series([1, 2, 3, 6.0, "foo"])  The number of columns of each type in a ``DataFrame`` can be found by calling
``DataFrame.dtypes.value_counts()``.  .. ipython:: python  dft.dtypes.value_counts()  Numeric dtypes will propagate and can coexist in DataFrames.
If a dtype is passed (either directly via the ``dtype`` keyword, a passed ``ndarray``,

or a passed ``Series``), then it will be preserved in DataFrame operations. Furthermore,
different numeric dtypes will **NOT** be combined. The following example will give you a taste.  .. ipython:: python  df1 = pd.DataFrame(np.random.randn(8, 1), columns=["A"], dtype="float32")  df1  df1.dtypes  df2 = pd.DataFrame(  {  "A": pd.Series(np.random.randn(8), dtype="float16"),  "B": pd.Series(np.random.randn(8)),  "C": pd.Series(np.array(np.random.randn(8), dtype="uint8")),  }  )  df2  df2.dtypes  defaults
~~~  By default integer types are ``int64`` and float types are ``float64``,
*regardless* of platform (32-bit or 64-bit).
The following will all result in ``int64`` dtypes.  .. ipython:: python  pd.DataFrame([1, 2], columns=["a"]).dtypes  pd.DataFrame({"a": [1, 2]}).dtypes  pd.DataFrame({"a": 1}, index=list(range(2))).dtypes  Note that Numpy will choose *platform-dependent* types when creating arrays.
The following **WILL** result in ``int32`` on 32-bit platform.  .. ipython:: python  frame = pd.DataFrame(np.array([1, 2]))  upcasting
~~~  Types can potentially be *upcasted* when combined with other types, meaning they are promoted
from the current type (e.g. ``int`` to ``float``).  .. ipython:: python  df3 = df1.reindex_like(df2).fillna(value=0.0) + df2  df3  df3.dtypes  `DataFrame.to_numpy` will return the *lower-common-denominator* of the dtypes, meaning
the dtype that can accommodate **ALL** of the types in the resulting homogeneous dtyped NumPy array. This can
force some *upcasting*.  .. ipython:: python  df3.to_numpy().dtype  astype
~~~  .. _basics.cast:  You can use the `~DataFrame.astype` method to explicitly convert dtypes from one to another. These will by default return a copy,
even if the dtype was unchanged (pass ``copy=False`` to change this behavior). In addition, they will raise an
exception if the astype operation is invalid.  Upcasting is always according to the **NumPy** rules. If two different dtypes are involved in an operation,
then the more *general* one will be used as the result of the operation.  .. ipython:: python  df3  df3.dtypes  # conversion of dtypes  df3.astype("float32").dtypes  Convert a subset of columns to a specified type using `~DataFrame.astype`.  .. ipython:: python  dft = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": [7, 8, 9]})  dft[["a", "b"]] = dft[["a", "b"]].astype(np.uint8)  dft  dft.dtypes  Convert certain columns to a specific dtype by passing a dict to `~DataFrame.astype`.  .. ipython:: python  dft1 = pd.DataFrame({"a": [1, 0, 1], "b": [4, 5, 6], "c": [7, 8, 9]})  dft1 = dft1.astype({"a": np.bool_, "c": np.float64})  dft1  dft1.dtypes  .. note::  When trying to convert a subset of columns to a specified type using `~DataFrame.astype`  and `~DataFrame.loc`, upcasting occurs.  `~DataFrame.loc` tries to fit in what we are assigning to the current dtypes, while ``[]`` will overwrite them taking the dtype from the right hand side. Therefore the following piece of code produces the unintended result.  .. ipython:: python  dft = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": [7, 8, 9]})  dft.loc[:, ["a", "b"]].astype(np.uint8).dtypes  dft.loc[:, ["a", "b"]] = dft.loc[:, ["a", "b"]].astype(np.uint8)  dft.dtypes  .. _basics.object_conversion:  object conversion
~~~  pandas offers various functions to try to force conversion of types from the ``object`` dtype to other types.
In cases where the data is already of the correct type, but stored in an ``object`` array, the
`DataFrame.infer_objects` and `Series.infer_objects` methods can be used to soft convert
to the correct type.  .. ipython:: python  import datetime  df = pd.DataFrame(  [  [1, 2],  ["a", "b"],  [datetime.datetime(2016, 3, 2), datetime.datetime(2016, 3, 2)],  ]  )  df = df.T  df  df.dtypes  Because the data was transposed the original inference stored all columns as object, which

``infer_objects`` will correct.  .. ipython:: python  df.infer_objects().dtypes  The following functions are available for one dimensional object arrays or scalars to perform
hard conversion of objects to a specified type:  * `~pandas.to_numeric` (conversion to numeric dtypes)  .. ipython:: python  m = ["1.1", 2, 3]  pd.to_numeric(m)  * `~pandas.to_datetime` (conversion to datetime objects)  .. ipython:: python  import datetime  m = ["2016-07-09", datetime.datetime(2016, 3, 2)]  pd.to_datetime(m)  * `~pandas.to_timedelta` (conversion to timedelta objects)  .. ipython:: python  m = ["5us", pd.Timedelta("1day")]  pd.to_timedelta(m)  To force a conversion, we can pass in an ``errors`` argument, which specifies how pandas should deal with elements
that cannot be converted to desired dtype or object. By default, ``errors='raise'``, meaning that any errors encountered
will be raised during the conversion process. However, if ``errors='coerce'``, these errors will be ignored and pandas
will convert problematic elements to ``pd.NaT`` (for datetime and timedelta) or ``np.nan`` (for numeric). This might be
useful if you are reading in data which is mostly of the desired dtype (e.g. numeric, datetime), but occasionally has
non-conforming elements intermixed that you want to represent as missing:  .. ipython:: python  import datetime  m = ["apple", datetime.datetime(2016, 3, 2)]  pd.to_datetime(m, errors="coerce")  m = ["apple", 2, 3]  pd.to_numeric(m, errors="coerce")  m = ["apple", pd.Timedelta("1day")]  pd.to_timedelta(m, errors="coerce")  The ``errors`` parameter has a third option of ``errors='ignore'``, which will simply return the passed in data if it
encounters any errors with the conversion to a desired data type:  .. ipython:: python  import datetime  m = ["apple", datetime.datetime(2016, 3, 2)]  pd.to_datetime(m, errors="ignore")  m = ["apple", 2, 3]  pd.to_numeric(m, errors="ignore")  m = ["apple", pd.Timedelta("1day")]  pd.to_timedelta(m, errors="ignore")  In addition to object conversion, `~pandas.to_numeric` provides another argument ``downcast``, which gives the
option of downcasting the newly (or already) numeric data to a smaller dtype, which can conserve memory:  .. ipython:: python  m = ["1", 2, 3]  pd.to_numeric(m, downcast="integer")  # smallest signed int dtype  pd.to_numeric(m, downcast="signed")  # same as 'integer'  pd.to_numeric(m, downcast="unsigned")  # smallest unsigned int dtype  pd.to_numeric(m, downcast="float")  # smallest float dtype  As these methods apply only to one-dimensional arrays, lists or scalars; they cannot be used directly on multi-dimensional objects such
as DataFrames. However, with `~pandas.DataFrame.apply`, we can "apply" the function over each column efficiently:  .. ipython:: python  import datetime  df = pd.DataFrame([["2016-07-09", datetime.datetime(2016, 3, 2)]] * 2, dtype="O")  df  df.apply(pd.to_datetime)  df = pd.DataFrame([["1.1", 2, 3]] * 2, dtype="O")  df  df.apply(pd.to_numeric)  df = pd.DataFrame([["5us", pd.Timedelta("1day")]] * 2, dtype="O")  df  df.apply(pd.to_timedelta)  gotchas
~~~  Performing selection operations on ``integer`` type data can easily upcast the data to ``floating``.
The dtype of the input data will be preserved in cases where ``nans`` are not introduced.
See also `Support for integer NA <gotchas.intna>`.  .. ipython:: python  dfi = df3.astype("int32")  dfi["E"] = 1  dfi  dfi.dtypes  casted = dfi[dfi > 0]  casted  casted.dtypes  While float dtypes are unchanged.  .. ipython:: python  dfa = df3.copy()  dfa["A"] = dfa["A"].astype("float32")  dfa.dtypes  casted = dfa[df2 > 0]  casted  casted.dtypes

Selecting columns based on ``dtype``

.. _basics.selectdtypes:  The `~DataFrame.select_dtypes` method implements subsetting of columns
based on their ``dtype``.  First, let's create a `DataFrame` with a slew of different
dtypes:  .. ipython:: python  df = pd.DataFrame(  {  "string": list("abc"),  "int64": list(range(1, 4)),  "uint8": np.arange(3, 6).astype("u1"),  "float64": np.arange(4.0, 7.0),  "bool1": [True, False, True],  "bool2": [False, True, False],  "dates": pd.date_range("now", periods=3),  "category": pd.Series(list("ABC")).astype("category"),  }  )  df["tdeltas"] = df.dates.diff()  df["uint64"] = np.arange(3, 6).astype("u8")  df["other_dates"] = pd.date_range("20130101", periods=3)  df["tz_aware_dates"] = pd.date_range("20130101", periods=3, tz="US/Eastern")  df  And the dtypes:  .. ipython:: python  df.dtypes  `~DataFrame.select_dtypes` has two parameters ``include`` and ``exclude`` that allow you to
say "give me the columns *with* these dtypes" (``include``) and/or "give the
columns *without* these dtypes" (``exclude``).  For example, to select ``bool`` columns:  .. ipython:: python  df.select_dtypes(include=[bool])  You can also pass the name of a dtype in the `NumPy dtype hierarchy
<https://numpy.org/doc/stable/reference/arrays.scalars.html>`__:  .. ipython:: python  df.select_dtypes(include=["bool"])  `~pandas.DataFrame.select_dtypes` also works with generic dtypes as well.  For example, to select all numeric and boolean columns while excluding unsigned
integers:  .. ipython:: python  df.select_dtypes(include=["number", "bool"], exclude=["unsignedinteger"])  To select string columns you must use the ``object`` dtype:  .. ipython:: python  df.select_dtypes(include=["object"])  To see all the child dtypes of a generic ``dtype`` like ``numpy.number`` you
can define a function that returns a tree of child dtypes:  .. ipython:: python  def subdtypes(dtype):  subs = dtype.__subclasses__()  if not subs:  return dtype  return [dtype, [subdtypes(dt) for dt in subs]]  All NumPy dtypes are subclasses of ``numpy.generic``:  .. ipython:: python  subdtypes(np.generic)  .. note::  pandas also defines the types ``category``, and ``datetime64[ns, tz]``, which are not integrated into the normal  NumPy hierarchy and won't show up with the above function.

Indexing with NA values

pandas allows indexing with ``NA`` values in a boolean array, which are treated as ``False``.  .. versionchanged:: 1.0.2  .. ipython:: python  s = pd.Series([1, 2, 3])  mask = pd.array([True, False, pd.NA], dtype="boolean")  s[mask]  If you would prefer to keep the ``NA`` values you can manually fill them with ``fillna(True)``.  .. ipython:: python  s[mask.fillna(True)]  .. _boolean.kleene:

Kleene logical operations

`arrays.BooleanArray` implements `Kleene Logic`_ (sometimes called three-value logic) for
logical operations like ``&`` (and), ``|`` (or) and ``^`` (exclusive-or).  This table demonstrates the results for every combination. These operations are symmetrical,
so flipping the left- and right-hand side makes no difference in the result.  === ===
Expression  Result
=== ===
``True & True``  ``True``
``True & False``  ``False``
``True & NA``  ``NA``
``False & False`` ``False``
``False & NA``  ``False``
``NA & NA``  ``NA``
``True | True``  ``True``
``True | False``  ``True``
``True | NA``  ``True``
``False | False`` ``False``
``False | NA``  ``NA``
``NA | NA``  ``NA``
``True ^ True``  ``False``
``True ^ False``  ``True``
``True ^ NA``  ``NA``
``False ^ False`` ``False``
``False ^ NA``  ``NA``
``NA ^ NA``  ``NA``
=== ===  When an ``NA`` is present in an operation, the output value is ``NA`` only if
the result cannot be determined solely based on the other input. For example,
``True | NA`` is ``True``, because both ``True | True`` and ``True | False``
are ``True``. In that case, we don't actually need to consider the value
of the ``NA``.  On the other hand, ``True & NA`` is ``NA``. The result depends on whether
the ``NA`` really is ``True`` or ``False``, since ``True & True`` is ``True``,
but ``True & False`` is ``False``, so we can't determine the output.  This differs from how ``np.nan`` behaves in logical operations. pandas treated
``np.nan`` is *always false in the output*.  In ``or``  .. ipython:: python  pd.Series([True, False, np.nan], dtype="object") | True  pd.Series([True, False, np.nan], dtype="boolean") | True  In ``and``  .. ipython:: python  pd.Series([True, False, np.nan], dtype="object") & True  pd.Series([True, False, np.nan], dtype="boolean") & True  .. _Kleene Logic: https://en.wikipedia.org/wiki/Three-valued_logic#Kleene_and_Priest_logics

Series creation
~~~  Categorical ``Series`` or columns in a ``DataFrame`` can be created in several ways:  By specifying ``dtype="category"`` when constructing a ``Series``:  .. ipython:: python  s = pd.Series(["a", "b", "c", "a"], dtype="category")  s  By converting an existing ``Series`` or column to a ``category`` dtype:  .. ipython:: python  df = pd.DataFrame({"A": ["a", "b", "c", "a"]})  df["B"] = df["A"].astype("category")  df  By using special functions, such as `~pandas.cut`, which groups data into
discrete bins. See the `example on tiling <reshaping.tile.cut>` in the docs.  .. ipython:: python  df = pd.DataFrame({"value": np.random.randint(0, 100, 20)})  labels = ["{0} - {1}".format(i, i + 9) for i in range(0, 100, 10)]  df["group"] = pd.cut(df.value, range(0, 105, 10), right=False, labels=labels)  df.head(10)  By passing a `pandas.Categorical` object to a ``Series`` or assigning it to a ``DataFrame``.  .. ipython:: python  raw_cat = pd.Categorical(  ["a", "b", "c", "a"], categories=["b", "c", "d"], ordered=False  )  s = pd.Series(raw_cat)  s  df = pd.DataFrame({"A": ["a", "b", "c", "a"]})  df["B"] = raw_cat  df  Categorical data has a specific ``category`` `dtype <basics.dtypes>`:  .. ipython:: python  df.dtypes  DataFrame creation
~~~  Similar to the previous section where a single column was converted to categorical, all columns in a
``DataFrame`` can be batch converted to categorical either during or after construction.  This can be done during construction by specifying ``dtype="category"`` in the ``DataFrame`` constructor:  .. ipython:: python  df = pd.DataFrame({"A": list("abca"), "B": list("bccd")}, dtype="category")  df.dtypes  Note that the categories present in each column differ; the conversion is done column by column, so
only labels present in a given column are categories:  .. ipython:: python  df["A"]  df["B"]  Analogously, all columns in an existing ``DataFrame`` can be batch converted using `DataFrame.astype`:  .. ipython:: python  df = pd.DataFrame({"A": list("abca"), "B": list("bccd")})  df_cat = df.astype("category")  df_cat.dtypes  This conversion is likewise done column by column:  .. ipython:: python  df_cat["A"]  df_cat["B"]  Controlling behavior
~~~  In the examples above where we passed ``dtype='category'``, we used the default
behavior:  1. Categories are inferred from the data.
2. Categories are unordered.  To control those behaviors, instead of passing ``'category'``, use an instance
of `~pandas.api.types.CategoricalDtype`.  .. ipython:: python  from pandas.api.types import CategoricalDtype  s = pd.Series(["a", "b", "c", "a"])  cat_type = CategoricalDtype(categories=["b", "c", "d"], ordered=True)  s_cat = s.astype(cat_type)  s_cat  Similarly, a ``CategoricalDtype`` can be used with a ``DataFrame`` to ensure that categories
are consistent among all columns.  .. ipython:: python  from pandas.api.types import CategoricalDtype  df = pd.DataFrame({"A": list("abca"), "B": list("bccd")})  cat_type = CategoricalDtype(categories=list("abcd"), ordered=True)  df_cat = df.astype(cat_type)  df_cat["A"]  df_cat["B"]  .. note::  To perform table-wise conversion, where all labels in the entire ``DataFrame`` are used as  categories for each column, the ``categories`` parameter can be determined programmatically by  ``categories = pd.unique(df.to_numpy().ravel())``.  If you already have ``codes`` and ``categories``, you can use the
`~pandas.Categorical.from_codes` constructor to save the factorize step
during normal constructor mode:  .. ipython:: python  splitter = np.random.choice([0, 1], 5, p=[0.5, 0.5])  s = pd.Series(pd.Categorical.from_codes(splitter, categories=["train", "test"]))  Regaining original data
~~~  To get back to the original ``Series`` or NumPy array, use

``Series.astype(original_dtype)`` or ``np.asarray(categorical)``:  .. ipython:: python  s = pd.Series(["a", "b", "c", "a"])  s  s2 = s.astype("category")  s2  s2.astype(str)  np.asarray(s2)  .. note::  In contrast to R's ``factor`` function, categorical data is not converting input values to  strings; categories will end up the same data type as the original values.  .. note::  In contrast to R's ``factor`` function, there is currently no way to assign/change labels at  creation time. Use ``categories`` to change the categories after creation time.  .. _categorical.categoricaldtype:

CategoricalDtype

A categorical's type is fully described by  1. ``categories``: a sequence of unique values and no missing values
2. ``ordered``: a boolean  This information can be stored in a `~pandas.api.types.CategoricalDtype`.
The ``categories`` argument is optional, which implies that the actual categories
should be inferred from whatever is present in the data when the
`pandas.Categorical` is created. The categories are assumed to be unordered
by default.  .. ipython:: python  from pandas.api.types import CategoricalDtype  CategoricalDtype(["a", "b", "c"])  CategoricalDtype(["a", "b", "c"], ordered=True)  CategoricalDtype()  A `~pandas.api.types.CategoricalDtype` can be used in any place pandas
expects a ``dtype``. For example `pandas.read_csv`,
`pandas.DataFrame.astype`, or in the ``Series`` constructor.  .. note::  As a convenience, you can use the string ``'category'`` in place of a  `~pandas.api.types.CategoricalDtype` when you want the default behavior of  the categories being unordered, and equal to the set values present in the  array. In other words, ``dtype='category'`` is equivalent to  ``dtype=CategoricalDtype()``.  Equality semantics
~~~  Two instances of `~pandas.api.types.CategoricalDtype` compare equal
whenever they have the same categories and order. When comparing two
unordered categoricals, the order of the ``categories`` is not considered.  .. ipython:: python  c1 = CategoricalDtype(["a", "b", "c"], ordered=False)  # Equal, since order is not considered when ordered=False  c1 == CategoricalDtype(["b", "c", "a"], ordered=False)  # Unequal, since the second CategoricalDtype is ordered  c1 == CategoricalDtype(["a", "b", "c"], ordered=True)  All instances of ``CategoricalDtype`` compare equal to the string ``'category'``.  .. ipython:: python  c1 == "category"  .. warning::  Since ``dtype='category'`` is essentially ``CategoricalDtype(None, False)``,  and since all instances ``CategoricalDtype`` compare equal to ``'category'``,  all instances of ``CategoricalDtype`` compare equal to a  ``CategoricalDtype(None, False)``, regardless of ``categories`` or  ``ordered``.

Description

Using `~DataFrame.describe` on categorical data will produce similar
output to a ``Series`` or ``DataFrame`` of type ``string``.  .. ipython:: python  cat = pd.Categorical(["a", "c", "c", np.nan], categories=["b", "a", "c"])  df = pd.DataFrame({"cat": cat, "s": ["a", "c", "c", np.nan]})  df.describe()  df["cat"].describe()  .. _categorical.cat:

Working with categories

Categorical data has a ``categories`` and a ``ordered`` property, which list their
possible values and whether the ordering matters or not. These properties are
exposed as ``s.cat.categories`` and ``s.cat.ordered``. If you don't manually
specify categories and ordering, they are inferred from the passed arguments.  .. ipython:: python  s = pd.Series(["a", "b", "c", "a"], dtype="category")  s.cat.categories  s.cat.ordered  It's also possible to pass in the categories in a specific order:  .. ipython:: python  s = pd.Series(pd.Categorical(["a", "b", "c", "a"], categories=["c", "b", "a"]))  s.cat.categories  s.cat.ordered  .. note::  New categorical data are **not** automatically ordered. You must explicitly  pass ``ordered=True`` to indicate an ordered ``Categorical``.  .. note::  The result of `~Series.unique` is not always the same as ``Series.cat.categories``,  because ``Series.unique()`` has a couple of guarantees, namely that it returns categories  in the order of appearance, and it only includes values that are actually present.  .. ipython:: python  s = pd.Series(list("babc")).astype(CategoricalDtype(list("abcd")))  s  # categories  s.cat.categories  # uniques  s.unique()  Renaming categories
~~~  Renaming categories is done by using the
`~pandas.Categorical.rename_categories` method:  .. ipython:: python  s = pd.Series(["a", "b", "c", "a"], dtype="category")  s  new_categories = ["Group %s" % g for g in s.cat.categories]  s = s.cat.rename_categories(new_categories)  s  # You can also pass a dict-like object to map the renaming  s = s.cat.rename_categories({1: "x", 2: "y", 3: "z"})  s  .. note::  In contrast to R's ``factor``, categorical data can have categories of other types than string.  Categories must be unique or a ``ValueError`` is raised:  .. ipython:: python  try:  s = s.cat.rename_categories([1, 1, 1])  except ValueError as e:  print("ValueError:", str(e))  Categories must also not be ``NaN`` or a ``ValueError`` is raised:  .. ipython:: python  try:  s = s.cat.rename_categories([1, 2, np.nan])  except ValueError as e:  print("ValueError:", str(e))  Appending new categories
~~~  Appending categories can be done by using the
`~pandas.Categorical.add_categories` method:  .. ipython:: python  s = s.cat.add_categories([4])  s.cat.categories  s  Removing categories
~~~  Removing categories can be done by using the
`~pandas.Categorical.remove_categories` method. Values which are removed
are replaced by ``np.nan``.:  .. ipython:: python  s = s.cat.remove_categories([4])  s  Removing unused categories
~~~  Removing unused categories can also be done:  .. ipython:: python  s = pd.Series(pd.Categorical(["a", "b", "a"], categories=["a", "b", "c", "d"]))  s  s.cat.remove_unused_categories()  Setting categories
~~~  If you want to do remove and add new categories in one step (which has some
speed advantage), or simply set the categories to a predefined scale,
use `~pandas.Categorical.set_categories`.  .. ipython:: python  s = pd.Series(["one", "two", "four", "-"], dtype="category")  s  s = s.cat.set_categories(["one", "two", "three", "four"])  s  .. note::  Be aware that `Categorical.set_categories` cannot know whether some category is omitted  intentionally or because it is misspelled or (under Python3) due to a type difference (e.g.,  NumPy S1 dtype and Python strings). This can result in surprising behaviour!

Sorting and order

.. _categorical.sort:  If categorical data is ordered (``s.cat.ordered == True``), then the order of the categories has a
meaning and certain operations are possible. If the categorical is unordered, ``.min()/.max()`` will raise a ``TypeError``.  .. ipython:: python  s = pd.Series(pd.Categorical(["a", "b", "c", "a"], ordered=False))  s = s.sort_values()  s = pd.Series(["a", "b", "c", "a"]).astype(CategoricalDtype(ordered=True))  s = s.sort_values()  s  s.min(), s.max()  You can set categorical data to be ordered by using ``as_ordered()`` or unordered by using ``as_unordered()``. These will by
default return a *new* object.  .. ipython:: python  s.cat.as_ordered()  s.cat.as_unordered()  Sorting will use the order defined by categories, not any lexical order present on the data type.
This is even true for strings and numeric data:  .. ipython:: python  s = pd.Series([1, 2, 3, 1], dtype="category")  s = s.cat.set_categories([2, 3, 1], ordered=True)  s  s = s.sort_values()  s  s.min(), s.max()  Reordering
~~~  Reordering the categories is possible via the `Categorical.reorder_categories` and
the `Categorical.set_categories` methods. For `Categorical.reorder_categories`, all
old categories must be included in the new categories and no new categories are allowed. This will
necessarily make the sort order the same as the categories order.  .. ipython:: python  s = pd.Series([1, 2, 3, 1], dtype="category")  s = s.cat.reorder_categories([2, 3, 1], ordered=True)  s  s = s.sort_values()  s  s.min(), s.max()  .. note::  Note the difference between assigning new categories and reordering the categories: the first  renames categories and therefore the individual values in the ``Series``, but if the first  position was sorted last, the renamed value will still be sorted last. Reordering means that the  way values are sorted is different afterwards, but not that individual values in the  ``Series`` are changed.  .. note::  If the ``Categorical`` is not ordered, `Series.min` and `Series.max` will raise  ``TypeError``. Numeric operations like ``+``, ``-``, ``*``, ``/`` and operations based on them  (e.g. `Series.median`, which would need to compute the mean between two values if the length  of an array is even) do not work and raise a ``TypeError``.  Multi column sorting
~~~  A categorical dtyped column will participate in a multi-column sort in a similar manner to other columns.
The ordering of the categorical is determined by the ``categories`` of that column.  .. ipython:: python  dfs = pd.DataFrame(  {  "A": pd.Categorical(  list("bbeebbaa"),  categories=["e", "a", "b"],  ordered=True,  ),  "B": [1, 2, 1, 2, 2, 1, 2, 1],  }  )  dfs.sort_values(by=["A", "B"])  Reordering the ``categories`` changes a future sort.  .. ipython:: python  dfs["A"] = dfs["A"].cat.reorder_categories(["a", "b", "e"])  dfs.sort_values(by=["A", "B"])

Comparisons

Comparing categorical data with other objects is possible in three cases:  * Comparing equality (``==`` and ``!=``) to a list-like object (list, Series, array,  ...) of the same length as the categorical data.
* All comparisons (``==``, ``!=``, ``>``, ``>=``, ``<``, and ``<=``) of categorical data to  another categorical Series, when ``ordered==True`` and the ``categories`` are the same.
* All comparisons of a categorical data to a scalar.  All other comparisons, especially "non-equality" comparisons of two categoricals with different
categories or a categorical with any list-like object, will raise a ``TypeError``.  .. note::  Any "non-equality" comparisons of categorical data with a ``Series``, ``np.array``, ``list`` or  categorical data with different categories or ordering will raise a ``TypeError`` because custom  categories ordering could be interpreted in two ways: one with taking into account the  ordering and one without.  .. ipython:: python  cat = pd.Series([1, 2, 3]).astype(CategoricalDtype([3, 2, 1], ordered=True))  cat_base = pd.Series([2, 2, 2]).astype(CategoricalDtype([3, 2, 1], ordered=True))  cat_base2 = pd.Series([2, 2, 2]).astype(CategoricalDtype(ordered=True))  cat  cat_base  cat_base2  Comparing to a categorical with the same categories and ordering or to a scalar works:  .. ipython:: python  cat > cat_base  cat > 2  Equality comparisons work with any list-like object of same length and scalars:  .. ipython:: python  cat == cat_base  cat == np.array([1, 2, 3])  cat == 2  This doesn't work because the categories are not the same:  .. ipython:: python  try:  cat > cat_base2  except TypeError as e:  print("TypeError:", str(e))  If you want to do a "non-equality" comparison of a categorical series with a list-like object
which is not categorical data, you need to be explicit and convert the categorical data back to
the original values:  .. ipython:: python  base = np.array([1, 2, 3])  try:  cat > base  except TypeError as e:  print("TypeError:", str(e))  np.asarray(cat) > base  When you compare two unordered categoricals with the same categories, the order is not considered:  .. ipython:: python  c1 = pd.Categorical(["a", "b"], categories=["a", "b"], ordered=False)  c2 = pd.Categorical(["a", "b"], categories=["b", "a"], ordered=False)  c1 == c2

Apart from `Series.min`, `Series.max` and `Series.mode`, the
following operations are possible with categorical data:  ``Series`` methods like `Series.value_counts` will use all categories,
even if some categories are not present in the data:  .. ipython:: python  s = pd.Series(pd.Categorical(["a", "b", "c", "c"], categories=["c", "a", "b", "d"]))  s.value_counts()  ``DataFrame`` methods like `DataFrame.sum` also show "unused" categories.  .. ipython:: python  columns = pd.Categorical(  ["One", "One", "Two"], categories=["One", "Two", "Three"], ordered=True  )  df = pd.DataFrame(  data=[[1, 2, 3], [4, 5, 6]],  columns=pd.MultiIndex.from_arrays([["A", "B", "B"], columns]),  )  df.groupby(axis=1, level=1).sum()  Groupby will also show "unused" categories:  .. ipython:: python  cats = pd.Categorical(  ["a", "b", "b", "b", "c", "c", "c"], categories=["a", "b", "c", "d"]  )  df = pd.DataFrame({"cats": cats, "values": [1, 2, 2, 2, 3, 4, 5]})  df.groupby("cats").mean()  cats2 = pd.Categorical(["a", "a", "b", "b"], categories=["a", "b", "c"])  df2 = pd.DataFrame(  {  "cats": cats2,  "B": ["c", "d", "c", "d"],  "values": [1, 2, 3, 4],  }  )  df2.groupby(["cats", "B"]).mean()  Pivot tables:  .. ipython:: python  raw_cat = pd.Categorical(["a", "a", "b", "b"], categories=["a", "b", "c"])  df = pd.DataFrame({"A": raw_cat, "B": ["c", "d", "c", "d"], "values": [1, 2, 3, 4]})  pd.pivot_table(df, values="values", index=["A", "B"])

The optimized pandas data access methods  ``.loc``, ``.iloc``, ``.at``, and ``.iat``,
work as normal. The only difference is the return type (for getting) and
that only values already in ``categories`` can be assigned.  Getting
~~~  If the slicing operation returns either a ``DataFrame`` or a column of type
``Series``, the ``category`` dtype is preserved.  .. ipython:: python  idx = pd.Index(["h", "i", "j", "k", "l", "m", "n"])  cats = pd.Series(["a", "b", "b", "b", "c", "c", "c"], dtype="category", index=idx)  values = [1, 2, 2, 2, 3, 4, 5]  df = pd.DataFrame({"cats": cats, "values": values}, index=idx)  df.iloc[2:4, :]  df.iloc[2:4, :].dtypes  df.loc["h":"j", "cats"]  df[df["cats"] == "b"]  An example where the category type is not preserved is if you take one single
row: the resulting ``Series`` is of dtype ``object``:  .. ipython:: python  # get the complete "h" row as a Series  df.loc["h", :]  Returning a single item from categorical data will also return the value, not a categorical
of length "1".  .. ipython:: python  df.iat[0, 0]  df["cats"] = df["cats"].cat.rename_categories(["x", "y", "z"])  df.at["h", "cats"]  # returns a string  .. note::  The is in contrast to R's ``factor`` function, where ``factor(c(1,2,3))[1]``  returns a single value ``factor``.  To get a single value ``Series`` of type ``category``, you pass in a list with
a single value:  .. ipython:: python  df.loc[["h"], "cats"]  String and datetime accessors
~~~  The accessors  ``.dt`` and ``.str`` will work if the ``s.cat.categories`` are of
an appropriate type:  .. ipython:: python  str_s = pd.Series(list("aabb"))  str_cat = str_s.astype("category")  str_cat  str_cat.str.contains("a")  date_s = pd.Series(pd.date_range("1/1/2015", periods=5))  date_cat = date_s.astype("category")  date_cat  date_cat.dt.day  .. note::  The returned ``Series`` (or ``DataFrame``) is of the same type as if you used the  ``.str.<method>`` / ``.dt.<method>`` on a ``Series`` of that type (and not of  type ``category``!).  That means, that the returned values from methods and properties on the accessors of a
``Series`` and the returned values from methods and properties on the accessors of this
``Series`` transformed to one of type ``category`` will be equal:  .. ipython:: python  ret_s = str_s.str.contains("a")  ret_cat = str_cat.str.contains("a")  ret_s.dtype == ret_cat.dtype  ret_s == ret_cat  .. note::  The work is done on the ``categories`` and then a new ``Series`` is constructed. This has  some performance implication if you have a ``Series`` of type string, where lots of elements  are repeated (i.e. the number of unique elements in the ``Series`` is a lot smaller than the  length of the ``Series``). In this case it can be faster to convert the original ``Series``  to one of type ``category`` and use ``.str.<method>`` or ``.dt.<property>`` on that.  Setting
~~~  Setting values in a categorical column (or ``Series``) works as long as the

value is included in the ``categories``:  .. ipython:: python  idx = pd.Index(["h", "i", "j", "k", "l", "m", "n"])  cats = pd.Categorical(["a", "a", "a", "a", "a", "a", "a"], categories=["a", "b"])  values = [1, 1, 1, 1, 1, 1, 1]  df = pd.DataFrame({"cats": cats, "values": values}, index=idx)  df.iloc[2:4, :] = [["b", 2], ["b", 2]]  df  try:  df.iloc[2:4, :] = [["c", 3], ["c", 3]]  except TypeError as e:  print("TypeError:", str(e))  Setting values by assigning categorical data will also check that the ``categories`` match:  .. ipython:: python  df.loc["j":"k", "cats"] = pd.Categorical(["a", "a"], categories=["a", "b"])  df  try:  df.loc["j":"k", "cats"] = pd.Categorical(["b", "b"], categories=["a", "b", "c"])  except TypeError as e:  print("TypeError:", str(e))  Assigning a ``Categorical`` to parts of a column of other types will use the values:  .. ipython:: python  df = pd.DataFrame({"a": [1, 1, 1, 1, 1], "b": ["a", "a", "a", "a", "a"]})  df.loc[1:2, "a"] = pd.Categorical(["b", "b"], categories=["a", "b"])  df.loc[2:3, "b"] = pd.Categorical(["b", "b"], categories=["a", "b"])  df  df.dtypes  .. _categorical.merge:
.. _categorical.concat:  Merging / concatenation
~~~  By default, combining ``Series`` or ``DataFrames`` which contain the same
categories results in ``category`` dtype, otherwise results will depend on the
dtype of the underlying categories. Merges that result in non-categorical
dtypes will likely have higher memory usage. Use ``.astype`` or
``union_categoricals`` to ensure ``category`` results.  .. ipython:: python  from pandas.api.types import union_categoricals  # same categories  s1 = pd.Series(["a", "b"], dtype="category")  s2 = pd.Series(["a", "b", "a"], dtype="category")  pd.concat([s1, s2])  # different categories  s3 = pd.Series(["b", "c"], dtype="category")  pd.concat([s1, s3])  # Output dtype is inferred based on categories values  int_cats = pd.Series([1, 2], dtype="category")  float_cats = pd.Series([3.0, 4.0], dtype="category")  pd.concat([int_cats, float_cats])  pd.concat([s1, s3]).astype("category")  union_categoricals([s1.array, s3.array])  The following table summarizes the results of merging ``Categoricals``:  +---+---+---+---+
| arg1  | arg2  |  identical  | result  |
+===+===+===+===+
| category  | category  | True  | category  |
+---+---+---+---+
| category (object) | category (object)  | False  | object (dtype is inferred)  |
| category (int)  | category (float)  | False  | float (dtype is inferred)  |
+---+---+---+---+  See also the section on `merge dtypes<merging.dtypes>` for notes about
preserving merge dtypes and performance.  .. _categorical.union:  Unioning
~~~  If you want to combine categoricals that do not necessarily have the same
categories, the `~pandas.api.types.union_categoricals` function will
combine a list-like of categoricals. The new categories will be the union of
the categories being combined.  .. ipython:: python  from pandas.api.types import union_categoricals  a = pd.Categorical(["b", "c"])  b = pd.Categorical(["a", "b"])  union_categoricals([a, b])  By default, the resulting categories will be ordered as
they appear in the data. If you want the categories to
be lexsorted, use ``sort_categories=True`` argument.  .. ipython:: python  union_categoricals([a, b], sort_categories=True)  ``union_categoricals`` also works with the "easy" case of combining two
categoricals of the same categories and order information

(e.g. what you could also ``append`` for).  .. ipython:: python  a = pd.Categorical(["a", "b"], ordered=True)  b = pd.Categorical(["a", "b", "a"], ordered=True)  union_categoricals([a, b])  The below raises ``TypeError`` because the categories are ordered and not identical.  .. code-block:: ipython  In [1]: a = pd.Categorical(["a", "b"], ordered=True)  In [2]: b = pd.Categorical(["a", "b", "c"], ordered=True)  In [3]: union_categoricals([a, b])  Out[3]:  TypeError: to union ordered Categoricals, all categories must be the same  Ordered categoricals with different categories or orderings can be combined by
using the ``ignore_ordered=True`` argument.  .. ipython:: python  a = pd.Categorical(["a", "b", "c"], ordered=True)  b = pd.Categorical(["c", "b", "a"], ordered=True)  union_categoricals([a, b], ignore_order=True)  `~pandas.api.types.union_categoricals` also works with a
``CategoricalIndex``, or ``Series`` containing categorical data, but note that
the resulting array will always be a plain ``Categorical``:  .. ipython:: python  a = pd.Series(["b", "c"], dtype="category")  b = pd.Series(["a", "b"], dtype="category")  union_categoricals([a, b])  .. note::  ``union_categoricals`` may recode the integer codes for categories  when combining categoricals.  This is likely what you want,  but if you are relying on the exact numbering of the categories, be  aware.  .. ipython:: python  c1 = pd.Categorical(["b", "c"])  c2 = pd.Categorical(["a", "b"])  c1  # "b" is coded to 0  c1.codes  c2  # "b" is coded to 1  c2.codes  c = union_categoricals([c1, c2])  c  # "b" is coded to 0 throughout, same as c1, different from c2  c.codes

Getting data in/out

You can write data that contains ``category`` dtypes to a ``HDFStore``.
See `here <io.hdf5-categorical>` for an example and caveats.  It is also possible to write data to and reading data from *Stata* format files.
See `here <io.stata-categorical>` for an example and caveats.  Writing to a CSV file will convert the data, effectively removing any information about the
categorical (categories and ordering). So if you read back the CSV file you have to convert the
relevant columns back to ``category`` and assign the right categories and categories ordering.  .. ipython:: python  import io  s = pd.Series(pd.Categorical(["a", "b", "b", "a", "a", "d"]))  # rename the categories  s = s.cat.rename_categories(["very good", "good", "bad"])  # reorder the categories and add missing categories  s = s.cat.set_categories(["very bad", "bad", "medium", "good", "very good"])  df = pd.DataFrame({"cats": s, "vals": [1, 2, 3, 4, 5, 6]})  csv = io.StringIO()  df.to_csv(csv)  df2 = pd.read_csv(io.StringIO(csv.getvalue()))  df2.dtypes  df2["cats"]  # Redo the category  df2["cats"] = df2["cats"].astype("category")  df2["cats"] = df2["cats"].cat.set_categories(  ["very bad", "bad", "medium", "good", "very good"]  )  df2.dtypes  df2["cats"]  The same holds for writing to a SQL database with ``to_sql``.

<missing_data>`.  Missing values should **not** be included in the Categorical's ``categories``,
only in the ``values``.
Instead, it is understood that NaN is different, and is always a possibility.
When working with the Categorical's ``codes``, missing values will always have
a code of ``-1``.  .. ipython:: python  s = pd.Series(["a", "b", np.nan, "a"], dtype="category")  # only two categories  s  s.cat.codes  Methods for working with missing data, e.g. `~Series.isna`, `~Series.fillna`,
`~Series.dropna`, all work normally:  .. ipython:: python  s = pd.Series(["a", "b", np.nan], dtype="category")  s  pd.isna(s)  s.fillna("a")

Differences to R's ``factor``

The following differences to R's factor functions can be observed:  * R's ``levels`` are named ``categories``.
* R's ``levels`` are always of type string, while ``categories`` in pandas can be of any dtype.
* It's not possible to specify labels at creation time. Use ``s.cat.rename_categories(new_labels)``  afterwards.
* In contrast to R's ``factor`` function, using categorical data as the sole input to create a  new categorical series will *not* remove unused categories but create a new categorical series  which is equal to the passed in one!
* R allows for missing values to be included in its ``levels`` (pandas' ``categories``). pandas  does not allow ``NaN`` categories, but missing values can still be in the ``values``.

.. _categorical.rfactor:  Memory usage
~~~  .. _categorical.memory:  The memory usage of a ``Categorical`` is proportional to the number of categories plus the length of the data. In contrast,
an ``object`` dtype is a constant times the length of the data.  .. ipython:: python  s = pd.Series(["foo", "bar"] * 1000)  # object dtype  s.nbytes  # category dtype  s.astype("category").nbytes  .. note::  If the number of categories approaches the length of the data, the ``Categorical`` will use nearly the same or  more memory than an equivalent ``object`` dtype representation.  .. ipython:: python  s = pd.Series(["foo%04d" % i for i in range(2000)])  # object dtype  s.nbytes  # category dtype  s.astype("category").nbytes  ``Categorical`` is not a ``numpy`` array
~~~  Currently, categorical data and the underlying ``Categorical`` is implemented as a Python
object and not as a low-level NumPy array dtype. This leads to some problems.  NumPy itself doesn't know about the new ``dtype``:  .. ipython:: python  try:  np.dtype("category")  except TypeError as e:  print("TypeError:", str(e))  dtype = pd.Categorical(["a"]).dtype  try:  np.dtype(dtype)  except TypeError as e:  print("TypeError:", str(e))  Dtype comparisons work:  .. ipython:: python  dtype == np.str_  np.str_ == dtype  To check if a Series contains Categorical data, use ``hasattr(s, 'cat')``:  .. ipython:: python  hasattr(pd.Series(["a"], dtype="category"), "cat")  hasattr(pd.Series(["a"]), "cat")  Using NumPy functions on a ``Series`` of type ``category`` should not work as ``Categoricals``
are not numeric data (even in the case that ``.categories`` is numeric).  .. ipython:: python  s = pd.Series(pd.Categorical([1, 2, 3, 4]))  try:  np.sum(s)  # same with np.log(s),...  except TypeError as e:  print("TypeError:", str(e))  .. note::  If such a function works, please file a bug at https://github.com/pandas-dev/pandas!  dtype in apply
~~~  pandas currently does not preserve the dtype in apply functions: If you apply along rows you get
a ``Series`` of ``object`` ``dtype`` (same as getting a row -> getting one element will return a
basic type) and applying along columns will also convert to object. ``NaN`` values are unaffected.
You can use ``fillna`` to handle missing values before applying a function.  .. ipython:: python  df = pd.DataFrame(  {  "a": [1, 2, 3, 4],  "b": ["a", "b", "c", "d"],  "cats": pd.Categorical([1, 2, 3, 2]),  }  )  df.apply(lambda row: type(row["cats"]), axis=1)  df.apply(lambda col: col.dtype, axis=0)  Categorical index
~~~  ``CategoricalIndex`` is a type of index that is useful for supporting
indexing with duplicates. This is a container around a ``Categorical``
and allows efficient indexing and storage of an index with a large number of duplicated elements.
See the `advanced indexing docs <advanced.categoricalindex>` for a more detailed
explanation.  Setting the index will create a ``CategoricalIndex``:  .. ipython:: python  cats = pd.Categorical([1, 2, 3, 4], categories=[4, 2, 3, 1])  strings = ["a", "b", "c", "d"]  values = [4, 2, 3, 1]  df = pd.DataFrame({"strings": strings, "values": values}, index=cats)  df.index  # This now sorts by the categories order  df.sort_index()  Side effects
~~~  Constructing a ``Series`` from a ``Categorical`` will not copy the input
``Categorical``. This means that changes to the ``Series`` will in most cases

change the original ``Categorical``:  .. ipython:: python  cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])  s = pd.Series(cat, name="cat")  cat  s.iloc[0:2] = 10  cat  Use ``copy=True`` to prevent such a behaviour or simply don't reuse ``Categoricals``:  .. ipython:: python  cat = pd.Categorical([1, 2, 3, 10], categories=[1, 2, 3, 4, 10])  s = pd.Series(cat, name="cat", copy=True)  cat  s.iloc[0:2] = 10  cat  .. note::  This also happens in some cases when you supply a NumPy array instead of a ``Categorical``:  using an int array (e.g. ``np.array([1,2,3,4])``) will exhibit the same behavior, while using  a string array (e.g. ``np.array(["a","b","c","a"])``) will not.

Idioms

.. _cookbook.idioms:  These are some neat pandas ``idioms``  `if-then/if-then-else on one column, and assignment to another one or more columns:
<https://stackoverflow.com/questions/17128302/python-pandas-idiom-for-if-then-else>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  if-then...
***  An if-then on one column  .. ipython:: python  df.loc[df.AAA >= 5, "BBB"] = -1  df  An if-then with assignment to 2 columns:  .. ipython:: python  df.loc[df.AAA >= 5, ["BBB", "CCC"]] = 555  df  Add another line with different logic, to do the -else  .. ipython:: python  df.loc[df.AAA < 5, ["BBB", "CCC"]] = 2000  df  Or use pandas where after you've set up a mask  .. ipython:: python  df_mask = pd.DataFrame(  {"AAA": [True] * 4, "BBB": [False] * 4, "CCC": [True, False] * 2}  )  df.where(df_mask, -1000)  `if-then-else using NumPy's where()
<https://stackoverflow.com/questions/19913659/pandas-conditional-creation-of-a-series-dataframe-column>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  df["logic"] = np.where(df["AAA"] > 5, "high", "low")  df  Splitting
***  `Split a frame with a boolean criterion
<https://stackoverflow.com/questions/14957116/how-to-split-a-dataframe-according-to-a-boolean-criterion>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  df[df.AAA <= 5]  df[df.AAA > 5]  Building criteria
***  `Select with multi-column criteria
<https://stackoverflow.com/questions/15315452/selecting-with-complex-criteria-from-pandas-dataframe>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  ...and (without assignment returns a Series)  .. ipython:: python  df.loc[(df["BBB"] < 25) & (df["CCC"] >= -40), "AAA"]  ...or (without assignment returns a Series)  .. ipython:: python  df.loc[(df["BBB"] > 25) | (df["CCC"] >= -40), "AAA"]  ...or (with assignment modifies the DataFrame.)  .. ipython:: python  df.loc[(df["BBB"] > 25) | (df["CCC"] >= 75), "AAA"] = 0.1  df  `Select rows with data closest to certain value using argsort
<https://stackoverflow.com/questions/17758023/return-rows-in-a-dataframe-closest-to-a-user-defined-number>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  aValue = 43.0  df.loc[(df.CCC - aValue).abs().argsort()]  `Dynamically reduce a list of criteria using a binary operators
<https://stackoverflow.com/questions/21058254/pandas-boolean-operation-in-a-python-list/21058331>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  Crit1 = df.AAA <= 5.5  Crit2 = df.BBB == 10.0  Crit3 = df.CCC > -40.0  One could hard code:  .. ipython:: python  AllCrit = Crit1 & Crit2 & Crit3  ...Or it can be done with a list of dynamically built criteria  .. ipython:: python  import functools  CritList = [Crit1, Crit2, Crit3]  AllCrit = functools.reduce(lambda x, y: x & y, CritList)  df[AllCrit]  .. _cookbook.selection:

Selection

Dataframes
***  The `indexing <indexing>` docs.  `Using both row labels and value conditionals
<https://stackoverflow.com/questions/14725068/pandas-using-row-labels-in-boolean-indexing>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  df[(df.AAA <= 6) & (df.index.isin([0, 2, 4]))]  Use loc for label-oriented slicing and iloc positional slicing `2904`  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]},  index=["foo", "bar", "boo", "kar"],  )  There are 2 explicit slicing methods, with a third general case  1. Positional-oriented (Python slicing style : exclusive of end)
2. Label-oriented (Non-Python slicing style : inclusive of end)
3. General (Either slicing style : depends on if the slice contains labels or positions)  .. ipython:: python  df.iloc[0:3]  # Positional  df.loc["bar":"kar"]  # Label  # Generic  df[0:3]  df["bar":"kar"]  Ambiguity arises when an index consists of integers with a non-zero start or non-unit increment.  .. ipython:: python  data = {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  df2 = pd.DataFrame(data=data, index=[1, 2, 3, 4])  # Note index starts at 1.  df2.iloc[1:3]  # Position-oriented  df2.loc[1:3]  # Label-oriented  `Using inverse operator (~) to take the complement of a mask
<https://stackoverflow.com/q/14986510>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [4, 5, 6, 7], "BBB": [10, 20, 30, 40], "CCC": [100, 50, -30, -50]}  )  df  df[~((df.AAA <= 6) & (df.index.isin([0, 2, 4])))]  New columns
***  `Efficiently and dynamically creating new columns using applymap
<https://stackoverflow.com/questions/16575868/efficiently-creating-additional-columns-in-a-pandas-dataframe-using-map>`__  .. ipython:: python  df = pd.DataFrame({"AAA": [1, 2, 1, 3], "BBB": [1, 1, 2, 2], "CCC": [2, 1, 3, 1]})  df  source_cols = df.columns  # Or some subset would work too  new_cols = [str(x) + "_cat" for x in source_cols]  categories = {1: "Alpha", 2: "Beta", 3: "Charlie"}  df[new_cols] = df[source_cols].applymap(categories.get)  df  `Keep other columns when using min() with groupby
<https://stackoverflow.com/q/23394476>`__  .. ipython:: python  df = pd.DataFrame(  {"AAA": [1, 1, 1, 2, 2, 2, 3, 3], "BBB": [2, 1, 3, 4, 5, 1, 2, 3]}  )  df  Method 1 : idxmin() to get the index of the minimums  .. ipython:: python  df.loc[df.groupby("AAA")["BBB"].idxmin()]  Method 2 : sort then take first of each  .. ipython:: python  df.sort_values(by="BBB").groupby("AAA", as_index=False).first()  Notice the same results, with the exception of the index.  .. _cookbook.multi_index:

Multiindexing

The `multindexing <advanced.hierarchical>` docs.  `Creating a MultiIndex from a labeled frame
<https://stackoverflow.com/questions/14916358/reshaping-dataframes-in-pandas-based-on-column-labels>`__  .. ipython:: python  df = pd.DataFrame(  {  "row": [0, 1, 2],  "One_X": [1.1, 1.1, 1.1],  "One_Y": [1.2, 1.2, 1.2],  "Two_X": [1.11, 1.11, 1.11],  "Two_Y": [1.22, 1.22, 1.22],  }  )  df  # As Labelled Index  df = df.set_index("row")  df  # With Hierarchical Columns  df.columns = pd.MultiIndex.from_tuples([tuple(c.split("_")) for c in df.columns])  df  # Now stack & Reset  df = df.stack(0).reset_index(1)  df  # And fix the labels (Notice the label 'level_1' got added automatically)  df.columns = ["Sample", "All_X", "All_Y"]  df  Arithmetic
***  `Performing arithmetic with a MultiIndex that needs broadcasting
<https://stackoverflow.com/questions/19501510/divide-entire-pandas-multiindex-dataframe-by-dataframe-variable/19502176#19502176>`__  .. ipython:: python  cols = pd.MultiIndex.from_tuples(  [(x, y) for x in ["A", "B", "C"] for y in ["O", "I"]]  )  df = pd.DataFrame(np.random.randn(2, 6), index=["n", "m"], columns=cols)  df  df = df.div(df["C"], level=1)  df  Slicing
***  `Slicing a MultiIndex with xs
<https://stackoverflow.com/questions/12590131/how-to-slice-multindex-columns-in-pandas-dataframes>`__  .. ipython:: python  coords = [("AA", "one"), ("AA", "six"), ("BB", "one"), ("BB", "two"), ("BB", "six")]  index = pd.MultiIndex.from_tuples(coords)  df = pd.DataFrame([11, 22, 33, 44, 55], index, ["MyData"])  df  To take the cross section of the 1st level and 1st axis the index:  .. ipython:: python  # Note : level and axis are optional, and default to zero  df.xs("BB", level=0, axis=0)  ...and now the 2nd level of the 1st axis.  .. ipython:: python  df.xs("six", level=1, axis=0)  `Slicing a MultiIndex with xs, method #2
<https://stackoverflow.com/questions/14964493/multiindex-based-indexing-in-pandas>`__  .. ipython:: python  import itertools  index = list(itertools.product(["Ada", "Quinn", "Violet"], ["Comp", "Math", "Sci"]))  headr = list(itertools.product(["Exams", "Labs"], ["I", "II"]))  indx = pd.MultiIndex.from_tuples(index, names=["Student", "Course"])  cols = pd.MultiIndex.from_tuples(headr)  # Notice these are un-named  data = [[70 + x + y + (x * y) % 3 for x in range(4)] for y in range(9)]  df = pd.DataFrame(data, indx, cols)  df  All = slice(None)  df.loc["Violet"]  df.loc[(All, "Math"), All]  df.loc[(slice("Ada", "Quinn"), "Math"), All]  df.loc[(All, "Math"), ("Exams")]  df.loc[(All, "Math"), (All, "II")]  `Setting portions of a MultiIndex with xs
<https://stackoverflow.com/questions/19319432/pandas-selecting-a-lower-level-in-a-dataframe-to-do-a-ffill>`__  Sorting
***  `Sort by specific column or an ordered list of columns, with a MultiIndex
<https://stackoverflow.com/q/14733871>`__  .. ipython:: python  df.sort_values(by=("Labs", "II"), ascending=False)  Partial selection, the need for sortedness `2995`  Levels
***  `Prepending a level to a multiindex
<https://stackoverflow.com/questions/14744068/prepend-a-level-to-a-pandas-multiindex>`__  `Flatten Hierarchical columns
<https://stackoverflow.com/q/14507794>`__  .. _cookbook.missing_data:

The `missing data<missing_data>` docs.  Fill forward a reversed timeseries  .. ipython:: python  df = pd.DataFrame(  np.random.randn(6, 1),  index=pd.date_range("2013-08-01", periods=6, freq="B"),  columns=list("A"),  )  df.loc[df.index[3], "A"] = np.nan  df  df.bfill()  `cumsum reset at NaN values
<https://stackoverflow.com/questions/18196811/cumsum-reset-at-nan>`__  Replace
***  `Using replace with backrefs
<https://stackoverflow.com/questions/16818871/extracting-value-and-creating-new-column-out-of-it>`__  .. _cookbook.grouping:

The `grouping <groupby>` docs.  `Basic grouping with apply
<https://stackoverflow.com/questions/15322632/python-pandas-df-groupy-agg-column-reference-in-agg>`__  Unlike agg, apply's callable is passed a sub-DataFrame which gives you access to all the columns  .. ipython:: python  df = pd.DataFrame(  {  "animal": "cat dog cat fish dog cat cat".split(),  "size": list("SSMMMLL"),  "weight": [8, 10, 11, 1, 20, 12, 12],  "adult": [False] * 5 + [True] * 2,  }  )  df  # List the size of the animals with the highest weight.  df.groupby("animal").apply(lambda subf: subf["size"][subf["weight"].idxmax()])  `Using get_group
<https://stackoverflow.com/questions/14734533/how-to-access-pandas-groupby-dataframe-by-key>`__  .. ipython:: python  gb = df.groupby(["animal"])  gb.get_group("cat")  `Apply to different items in a group
<https://stackoverflow.com/questions/15262134/apply-different-functions-to-different-items-in-group-object-python-pandas>`__  .. ipython:: python  def GrowUp(x):  avg_weight = sum(x[x["size"] == "S"].weight * 1.5)  avg_weight += sum(x[x["size"] == "M"].weight * 1.25)  avg_weight += sum(x[x["size"] == "L"].weight)  avg_weight /= len(x)  return pd.Series(["L", avg_weight, True], index=["size", "weight", "adult"])  expected_df = gb.apply(GrowUp)  expected_df  `Expanding apply
<https://stackoverflow.com/questions/14542145/reductions-down-a-column-in-pandas>`__  .. ipython:: python  S = pd.Series([i / 100.0 for i in range(1, 11)])  def cum_ret(x, y):  return x * (1 + y)  def red(x):  return functools.reduce(cum_ret, x, 1.0)  S.expanding().apply(red, raw=True)  `Replacing some values with mean of the rest of a group
<https://stackoverflow.com/questions/14760757/replacing-values-with-groupby-means>`__  .. ipython:: python  df = pd.DataFrame({"A": [1, 1, 2, 2], "B": [1, -1, 1, 2]})  gb = df.groupby("A")  def replace(g):  mask = g < 0  return g.where(~mask, g[~mask].mean())  gb.transform(replace)  `Sort groups by aggregated data
<https://stackoverflow.com/questions/14941366/pandas-sort-by-group-aggregate-and-column>`__  .. ipython:: python  df = pd.DataFrame(  {  "code": ["foo", "bar", "baz"] * 2,  "data": [0.16, -0.21, 0.33, 0.45, -0.59, 0.62],  "flag": [False, True] * 3,  }  )  code_groups = df.groupby("code")  agg_n_sort_order = code_groups[["data"]].transform(sum).sort_values(by="data")  sorted_df = df.loc[agg_n_sort_order.index]  sorted_df  `Create multiple aggregated columns
<https://stackoverflow.com/questions/14897100/create-multiple-columns-in-pandas-aggregation-function>`__  .. ipython:: python  rng = pd.date_range(start="2014-10-07", periods=10, freq="2min")  ts = pd.Series(data=list(range(10)), index=rng)  def MyCust(x):  if len(x) > 2:  return x[1] * 1.234  return pd.NaT  mhc = {"Mean": np.mean, "Max": np.max, "Custom": MyCust}  ts.resample("5min").apply(mhc)  ts  `Create a value counts column and reassign back to the DataFrame
<https://stackoverflow.com/q/17709270>`__  .. ipython:: python  df = pd.DataFrame(  {"Color": "Red Red Red Blue".split(), "Value": [100, 150, 50, 50]}  )  df  df["Counts"] = df.groupby(["Color"]).transform(len)  df  `Shift groups of the values in a column based on the index
<https://stackoverflow.com/q/23198053/190597>`__  .. ipython:: python  df = pd.DataFrame(  {"line_race": [10, 10, 8, 10, 10, 8], "beyer": [99, 102, 103, 103, 88, 100]},  index=[  "Last Gunfighter",  "Last Gunfighter",  "Last Gunfighter",  "Paynter",  "Paynter",  "Paynter",  ],  )  df  df["beyer_shifted"] = df.groupby(level=0)["beyer"].shift(1)  df  `Select row with maximum value from each group
<https://stackoverflow.com/q/26701849/190597>`__  .. ipython:: python  df = pd.DataFrame(  {  "host": ["other", "other", "that", "this", "this"],  "service": ["mail", "web", "mail", "mail", "web"],  "no": [1, 2, 1, 2, 1],  }  ).set_index(["host", "service"])  mask = df.groupby(level=0).agg("idxmax")  df_count = df.loc[mask["no"]].reset_index()  df_count  `Grouping like Python's itertools.groupby

<https://stackoverflow.com/q/29142487/846892>`__  .. ipython:: python  df = pd.DataFrame([0, 1, 0, 1, 1, 1, 0, 1, 1], columns=["A"])  df["A"].groupby((df["A"] != df["A"].shift()).cumsum()).groups  df["A"].groupby((df["A"] != df["A"].shift()).cumsum()).cumsum()  Expanding data
***  `Alignment and to-date
<https://stackoverflow.com/questions/15489011/python-time-series-alignment-and-to-date-functions>`__  `Rolling Computation window based on values instead of counts
<https://stackoverflow.com/questions/14300768/pandas-rolling-computation-with-window-based-on-values-instead-of-counts>`__  `Rolling Mean by Time Interval
<https://stackoverflow.com/questions/15771472/pandas-rolling-mean-by-time-interval>`__  Splitting
***  `Splitting a frame
<https://stackoverflow.com/questions/13353233/best-way-to-split-a-dataframe-given-an-edge/15449992#15449992>`__  Create a list of dataframes, split using a delineation based on logic included in rows.  .. ipython:: python  df = pd.DataFrame(  data={  "Case": ["A", "A", "A", "B", "A", "A", "B", "A", "A"],  "Data": np.random.randn(9),  }  )  dfs = list(  zip(  *df.groupby(  (1 * (df["Case"] == "B"))  .cumsum()  .rolling(window=3, min_periods=1)  .median()  )  )  )[-1]  dfs[0]  dfs[1]  dfs[2]  .. _cookbook.pivot:  Pivot
***
The `Pivot <reshaping.pivot>` docs.  `Partial sums and subtotals
<https://stackoverflow.com/a/15574875>`__  .. ipython:: python  df = pd.DataFrame(  data={  "Province": ["ON", "QC", "BC", "AL", "AL", "MN", "ON"],  "City": [  "Toronto",  "Montreal",  "Vancouver",  "Calgary",  "Edmonton",  "Winnipeg",  "Windsor",  ],  "Sales": [13, 6, 16, 8, 4, 3, 1],  }  )  table = pd.pivot_table(  df,  values=["Sales"],  index=["Province"],  columns=["City"],  aggfunc=np.sum,  margins=True,  )  table.stack("City")  `Frequency table like plyr in R
<https://stackoverflow.com/questions/15589354/frequency-tables-in-pandas-like-plyr-in-r>`__  .. ipython:: python  grades = [48, 99, 75, 80, 42, 80, 72, 68, 36, 78]  df = pd.DataFrame(  {  "ID": ["x%d" % r for r in range(10)],  "Gender": ["F", "M", "F", "M", "F", "M", "F", "M", "M", "M"],  "ExamYear": [  "2007",  "2007",  "2007",  "2008",  "2008",  "2008",  "2008",  "2009",  "2009",  "2009",  ],  "Class": [  "algebra",  "stats",  "bio",  "algebra",  "algebra",  "stats",  "stats",  "algebra",  "bio",  "bio",  ],  "Participated": [  "yes",  "yes",  "yes",  "yes",  "no",  "yes",  "yes",  "yes",  "yes",  "yes",  ],  "Passed": ["yes" if x > 50 else "no" for x in grades],  "Employed": [  True,  True,  True,  False,  False,  False,  False,  True,  True,  False,  ],  "Grade": grades,  }  )  df.groupby("ExamYear").agg(  {  "Participated": lambda x: x.value_counts()["yes"],  "Passed": lambda x: sum(x == "yes"),  "Employed": lambda x: sum(x),  "Grade": lambda x: sum(x) / len(x),  }  )  `Plot pandas DataFrame with year over year data
<https://stackoverflow.com/questions/30379789/plot-pandas-data-frame-with-year-over-year-data>`__  To create year and month cross tabulation:  .. ipython:: python  df = pd.DataFrame(  {"value": np.random.randn(36)},  index=pd.date_range("2011-01-01", freq="M", periods=36),  )  pd.pivot_table(  df, index=df.index.month, columns=df.index.year, values="value", aggfunc="sum"  )  Apply
***  `Rolling apply to organize - Turning embedded lists into a MultiIndex frame
<https://stackoverflow.com/questions/17349981/converting-pandas-dataframe-with-categorical-values-into-binary-values>`__  .. ipython:: python  df = pd.DataFrame(  data={  "A": [[2, 4, 8, 16], [100, 200], [10, 20, 30]],  "B": [["a", "b", "c"], ["jj", "kk"], ["ccc"]],  },  index=["I", "II", "III"],  )  def SeriesFromSubList(aList):  return pd.Series(aList)  df_orgz = pd.concat(  {ind: row.apply(SeriesFromSubList) for ind, row in df.iterrows()}  )  df_orgz  `Rolling apply with a DataFrame returning a Series

<https://stackoverflow.com/questions/19121854/using-rolling-apply-on-a-dataframe-object>`__  Rolling Apply to multiple columns where function calculates a Series before a Scalar from the Series is returned  .. ipython:: python  df = pd.DataFrame(  data=np.random.randn(2000, 2) / 1000,  index=pd.date_range("2001-01-01", periods=2000),  columns=["A", "B"],  )  df  def gm(df, const):  v = (((df["A"] + df["B"]) + 1).cumprod()) - 1) * const  return v.iloc[-1]  s = pd.Series(  {  df.index[i]: gm(df.iloc[i: min(i + 51, len(df) - 1)], 5)  for i in range(len(df) - 50)  }  )  s  `Rolling apply with a DataFrame returning a Scalar
<https://stackoverflow.com/questions/21040766/python-pandas-rolling-apply-two-column-input-into-function/21045831#21045831>`__  Rolling Apply to multiple columns where function returns a Scalar (Volume Weighted Average Price)  .. ipython:: python  rng = pd.date_range(start="2014-01-01", periods=100)  df = pd.DataFrame(  {  "Open": np.random.randn(len(rng)),  "Close": np.random.randn(len(rng)),  "Volume": np.random.randint(100, 2000, len(rng)),  },  index=rng,  )  df  def vwap(bars):  return (bars.Close * bars.Volume).sum() / bars.Volume.sum()  window = 5  s = pd.concat(  [  (pd.Series(vwap(df.iloc[i: i + window]), index=[df.index[i + window]]))  for i in range(len(df) - window)  ]  )  s.round(2)

Timeseries

`Between times
<https://stackoverflow.com/questions/14539992/pandas-drop-rows-outside-of-time-range>`__  `Using indexer between time
<https://stackoverflow.com/questions/17559885/pandas-dataframe-mask-based-on-index>`__  `Constructing a datetime range that excludes weekends and includes only certain times
<https://stackoverflow.com/a/24014440>`__  `Vectorized Lookup
<https://stackoverflow.com/questions/13893227/vectorized-look-up-of-values-in-pandas-dataframe>`__  `Aggregation and plotting time series
<https://nipunbatra.github.io/blog/visualisation/2013/05/01/aggregation-timeseries.html>`__  Turn a matrix with hours in columns and days in rows into a continuous row sequence in the form of a time series.
`How to rearrange a Python pandas DataFrame?
<https://stackoverflow.com/questions/15432659/how-to-rearrange-a-python-pandas-dataframe>`__  `Dealing with duplicates when reindexing a timeseries to a specified frequency
<https://stackoverflow.com/questions/22244383/pandas-df-refill-adding-two-columns-of-different-shape>`__  Calculate the first day of the month for each entry in a DatetimeIndex  .. ipython:: python  dates = pd.date_range("2000-01-01", periods=5)  dates.to_period(freq="M").to_timestamp()  .. _cookbook.resample:  Resampling
***  The `Resample <timeseries.resampling>` docs.  `Using Grouper instead of TimeGrouper for time grouping of values
<https://stackoverflow.com/questions/15297053/how-can-i-divide-single-values-of-a-dataframe-by-monthly-averages>`__  `Time grouping with some missing values
<https://stackoverflow.com/questions/33637312/pandas-grouper-by-frequency-with-completeness-requirement>`__  Valid frequency arguments to Grouper `Timeseries <timeseries.offset_aliases>`  `Grouping using a MultiIndex
<https://stackoverflow.com/questions/41483763/pandas-timegrouper-on-multiindex>`__  Using TimeGrouper and another grouping to create subgroups, then apply a custom function `3791`  `Resampling with custom periods
<https://stackoverflow.com/questions/15408156/resampling-with-custom-periods>`__  `Resample intraday frame without adding new days
<https://stackoverflow.com/questions/14898574/resample-intrday-pandas-dataframe-without-add-new-days>`__  `Resample minute data
<https://stackoverflow.com/questions/14861023/resampling-minute-data>`__  Resample with groupby_  .. _cookbook.merge:

Merge

The `Join <merging.join>` docs.  `Concatenate two dataframes with overlapping index (emulate R rbind)
<https://stackoverflow.com/questions/14988480/pandas-version-of-rbind>`__  .. ipython:: python  rng = pd.date_range("2000-01-01", periods=6)  df1 = pd.DataFrame(np.random.randn(6, 3), index=rng, columns=["A", "B", "C"])  df2 = df1.copy()  Depending on df construction, ``ignore_index`` may be needed  .. ipython:: python  df = pd.concat([df1, df2], ignore_index=True)  df  Self Join of a DataFrame `2996`  .. ipython:: python  df = pd.DataFrame(  data={  "Area": ["A"] * 5 + ["C"] * 2,  "Bins": [110] * 2 + [160] * 3 + [40] * 2,  "Test_0": [0, 1, 0, 1, 2, 0, 1],  "Data": np.random.randn(7),  }  )  df  df["Test_1"] = df["Test_0"] - 1  pd.merge(  df,  df,  left_on=["Bins", "Area", "Test_0"],  right_on=["Bins", "Area", "Test_1"],  suffixes=("_L", "_R"),  )  `How to set the index and join
<https://stackoverflow.com/questions/14341805/pandas-merge-pd-merge-how-to-set-the-index-and-join>`__  `KDB like asof join
<https://stackoverflow.com/questions/12322289/kdb-like-asof-join-for-timeseries-data-in-pandas/12336039#12336039>`__  `Join with a criteria based on the values
<https://stackoverflow.com/questions/15581829/how-to-perform-an-inner-or-outer-join-of-dataframes-with-pandas-on-non-simplisti>`__  `Using searchsorted to merge based on values inside a range
<https://stackoverflow.com/questions/25125626/pandas-merge-with-logic/2512764>`__  .. _cookbook.plotting:

Plotting

The `Plotting <visualization>` docs.  `Make Matplotlib look like R
<https://stackoverflow.com/questions/14349055/making-matplotlib-graphs-look-like-r-by-default>`__  `Setting x-axis major and minor labels
<https://stackoverflow.com/questions/12945971/pandas-timeseries-plot-setting-x-axis-major-and-minor-ticks-and-labels>`__  `Plotting multiple charts in an IPython Jupyter notebook
<https://stackoverflow.com/questions/16392921/make-more-than-one-chart-in-same-ipython-notebook-cell>`__  `Creating a multi-line plot
<https://stackoverflow.com/questions/16568964/make-a-multiline-plot-from-csv-file-in-matplotlib>`__  `Plotting a heatmap
<https://stackoverflow.com/questions/17050202/plot-timeseries-of-histograms-in-python>`__  `Annotate a time-series plot
<https://stackoverflow.com/questions/11067368/annotate-time-series-plot-in-matplotlib>`__  `Annotate a time-series plot #2
<https://stackoverflow.com/questions/17891493/annotating-points-from-a-pandas-dataframe-in-matplotlib-plot>`__  `Generate Embedded plots in excel files using Pandas, Vincent and xlsxwriter
<https://pandas-xlsxwriter-charts.readthedocs.io/>`__  `Boxplot for each quartile of a stratifying variable
<https://stackoverflow.com/questions/23232989/boxplot-stratified-by-column-in-python-pandas>`__  .. ipython:: python  df = pd.DataFrame(  {  "stratifying_var": np.random.uniform(0, 100, 20),  "price": np.random.normal(100, 5, 20),  }  )  df["quartiles"] = pd.qcut(  df["stratifying_var"], 4, labels=["0-25%", "25-50%", "50-75%", "75-100%"]  )  @savefig quartile_boxplot.png  df.boxplot(column="price", by="quartiles")

`Performance comparison of SQL vs HDF5
<https://stackoverflow.com/q/16628329>`__  .. _cookbook.csv:  CSV
***  The `CSV <io.read_csv_table>` docs  read_csv in action_  `appending to a csv
<https://stackoverflow.com/questions/17134942/pandas-dataframe-output-end-of-csv>`__  `Reading a csv chunk-by-chunk
<https://stackoverflow.com/questions/11622652/large-persistent-dataframe-in-pandas/12193309#12193309>`__  `Reading only certain rows of a csv chunk-by-chunk
<https://stackoverflow.com/questions/19674212/pandas-data-frame-select-rows-and-clear-memory>`__  `Reading the first few lines of a frame
<https://stackoverflow.com/questions/15008970/way-to-read-first-few-lines-for-pandas-dataframe>`__  Reading a file that is compressed but not by ``gzip/bz2`` (the native compressed formats which ``read_csv`` understands).
This example shows a ``WinZipped`` file, but is a general application of opening the file within a context manager and
using that handle to read.
`See here
<https://stackoverflow.com/questions/17789907/pandas-convert-winzipped-csv-file-to-data-frame>`__  `Inferring dtypes from a file
<https://stackoverflow.com/questions/1555005/get-inferred-dataframe-types-iteratively-using-chunksize>`__  Dealing with bad lines `2886`  `Write a multi-row index CSV without writing duplicates
<https://stackoverflow.com/questions/17349574/pandas-write-multiindex-rows-with-to-csv>`__  .. _cookbook.csv.multiple_files:  Reading multiple files to create a single DataFrame
^^^  The best way to combine multiple files into a single DataFrame is to read the individual frames one by one, put all
of the individual frames into a list, and then combine the frames in the list using `pd.concat`:  .. ipython:: python  for i in range(3):  data = pd.DataFrame(np.random.randn(10, 4))  data.to_csv("file_{}.csv".format(i))  files = ["file_0.csv", "file_1.csv", "file_2.csv"]  result = pd.concat([pd.read_csv(f) for f in files], ignore_index=True)  You can use the same approach to read all files matching a pattern.  Here is an example using ``glob``:  .. ipython:: python  import glob  import os  files = glob.glob("file_*.csv")  result = pd.concat([pd.read_csv(f) for f in files], ignore_index=True)  Finally, this strategy will work with the other ``pd.read_*(...)`` functions described in the `io docs<io>`.  .. ipython:: python  for i in range(3):  os.remove("file_{}.csv".format(i))  Parsing date components in multi-columns
^^^  Parsing date components in multi-columns is faster with a format  .. ipython:: python  i = pd.date_range("2000101", periods=1000)  df = pd.DataFrame({"year": i.year, "month": i.month, "day": i.day})  df.head()  %timeit pd.to_datetime(df.year * 1000 + df.month * 100 + df.day, format='%Y%m%d')  ds = df.apply(lambda x: "%04d%02d%02d" % (x["year"], x["month"], x["day"]), axis=1)  ds.head()  %timeit pd.to_datetime(ds)  Skip row between header and data
^^^  .. ipython:: python  data = """;;;  ;;;  ;;;  ;;;  ;;;  ;;;  ;;;  ;;;  ;;;  ;;;  date;Param1;Param2;Param4;Param5  ;m²;°C;m²;m  ;;;  01.01.1990 00:00;1;1;2;3  01.01.1990 01:00;5;3;4;5  01.01.1990 02:00;9;5;6;7  01.01.1990 03:00;13;7;8;9  01.01.1990 04:00;17;9;10;11  01.01.1990 05:00;21;11;12;13  """  Option 1: pass rows explicitly to skip rows
"""  .. ipython:: python  from io import StringIO  pd.read_csv(  StringIO(data),  sep=";",  skiprows=[11, 12],  index_col=0,  parse_dates=True,  header=10,  )  Option 2: read column names and then data
"""  .. ipython:: python  pd.read_csv(StringIO(data), sep=";", header=10, nrows=10).columns  columns = pd.read_csv(StringIO(data), sep=";", header=10, nrows=10).columns  pd.read_csv(  StringIO(data), sep=";", index_col=0, header=12, parse_dates=True, names=columns  )  .. _cookbook.sql:  SQL
***  The `SQL <io.sql>` docs  `Reading from databases with SQL
<https://stackoverflow.com/questions/10065051/python-pandas-and-databases-like-mysql>`__  .. _cookbook.excel:  Excel
***  The `Excel <io.excel>` docs  `Reading from a filelike handle

<https://stackoverflow.com/questions/15588713/sheets-of-excel-workbook-from-a-url-into-a-pandas-dataframe>`__  `Modifying formatting in XlsxWriter output
<https://pbpython.com/improve-pandas-excel-output.html>`__  Loading only visible sheets `19842#issuecomment-892150745`  .. _cookbook.html:  HTML
***  `Reading HTML tables from a server that cannot handle the default request
header <https://stackoverflow.com/a/18939272/564538>`__  .. _cookbook.hdf:  HDFStore
***  The `HDFStores <io.hdf5>` docs  `Simple queries with a Timestamp Index
<https://stackoverflow.com/questions/13926089/selecting-columns-from-pandas-hdfstore-table>`__  Managing heterogeneous data using a linked multiple table hierarchy `3032`  `Merging on-disk tables with millions of rows
<https://stackoverflow.com/questions/14614512/merging-two-tables-with-millions-of-rows-in-python/14617925#14617925>`__  `Avoiding inconsistencies when writing to a store from multiple processes/threads
<https://stackoverflow.com/a/29014295/2858145>`__  De-duplicating a large store by chunks, essentially a recursive reduction operation. Shows a function for taking in data from
csv file and creating a store by chunks, with date parsing as well.
`See here
<https://stackoverflow.com/questions/16110252/need-to-compare-very-large-files-around-1-5gb-in-python/16110391#16110391>`__  `Creating a store chunk-by-chunk from a csv file
<https://stackoverflow.com/questions/20428355/appending-column-to-frame-of-hdf-file-in-pandas/20428786#20428786>`__  `Appending to a store, while creating a unique index
<https://stackoverflow.com/questions/16997048/how-does-one-append-large-amounts-of-data-to-a-pandas-hdfstore-and-get-a-natural/16999397#16999397>`__  `Large Data work flows
<https://stackoverflow.com/q/14262433>`__  `Reading in a sequence of files, then providing a global unique index to a store while appending
<https://stackoverflow.com/questions/16997048/how-does-one-append-large-amounts-of-data-to-a-pandas-hdfstore-and-get-a-natural>`__  `Groupby on a HDFStore with low group density
<https://stackoverflow.com/questions/15798209/pandas-group-by-query-on-large-data-in-hdfstore>`__  `Groupby on a HDFStore with high group density
<https://stackoverflow.com/questions/25459982/trouble-with-grouby-on-millions-of-keys-on-a-chunked-file-in-python-pandas/25471765#25471765>`__  `Hierarchical queries on a HDFStore
<https://stackoverflow.com/questions/22777284/improve-query-performance-from-a-large-hdfstore-table-with-pandas/22820780#22820780>`__  `Counting with a HDFStore
<https://stackoverflow.com/questions/20497897/converting-dict-of-dicts-into-pandas-dataframe-memory-issues>`__  `Troubleshoot HDFStore exceptions
<https://stackoverflow.com/questions/15488809/how-to-trouble-shoot-hdfstore-exception-cannot-find-the-correct-atom-type>`__  `Setting min_itemsize with strings
<https://stackoverflow.com/questions/15988871/hdfstore-appendstring-dataframe-fails-when-string-column-contents-are-longer>`__  `Using ptrepack to create a completely-sorted-index on a store
<https://stackoverflow.com/questions/17893370/ptrepack-sortby-needs-full-index>`__  Storing Attributes to a group node  .. ipython:: python  df = pd.DataFrame(np.random.randn(8, 3))  store = pd.HDFStore("test.h5")  store.put("df", df)  # you can store an arbitrary Python object via pickle  store.get_storer("df").attrs.my_attribute = {"A": 10}  store.get_storer("df").attrs.my_attribute  .. ipython:: python  store.close()  os.remove("test.h5")  You can create or load a HDFStore in-memory  by passing the ``driver``
parameter to PyTables. Changes are only written to disk when the HDFStore

is closed.  .. ipython:: python  store = pd.HDFStore("test.h5", "w", driver="H5FD_CORE")  df = pd.DataFrame(np.random.randn(8, 3))  store["test"] = df  # only after closing the store, data is written to disk:  store.close()  .. ipython:: python  os.remove("test.h5")  .. _cookbook.binary:  Binary files
***  pandas readily accepts NumPy record arrays, if you need to read in a binary
file consisting of an array of C structs. For example, given this C program
in a file called ``main.c`` compiled with ``gcc main.c -std=gnu99`` on a
64-bit machine,  .. code-block:: c  #include <stdio.h>  #include <stdint.h>  typedef struct _Data  {  int32_t count;  double avg;  float scale;  } Data;  int main(int argc, const char *argv[])  {  size_t n = 10;  Data d[n];  for (int i = 0; i < n; ++i)  {  d[i].count = i;  d[i].avg = i + 1.0;  d[i].scale = (float) i + 2.0f;  }  FILE *file = fopen("binary.dat", "wb");  fwrite(&d, sizeof(Data), n, file);  fclose(file);  return 0;  }  the following Python code will read the binary file ``'binary.dat'`` into a
pandas ``DataFrame``, where each element of the struct corresponds to a column
in the frame:  .. code-block:: python  names = "count", "avg", "scale"  # note that the offsets are larger than the size of the type because of  # struct padding  offsets = 0, 8, 16  formats = "i4", "f8", "f4"  dt = np.dtype({"names": names, "offsets": offsets, "formats": formats}, align=True)  df = pd.DataFrame(np.fromfile("binary.dat", dt))  .. note::  The offsets of the structure elements may be different depending on the  architecture of the machine on which the file was created. Using a raw  binary file format like this for general data storage is not recommended, as  it is not cross platform. We recommended either HDF5 or parquet, both of  which are supported by pandas' IO facilities.

Computation

`Numerical integration (sample-based) of a time series
<https://nbviewer.ipython.org/gist/metakermit/5720498>`__  Correlation
***  Often it's useful to obtain the lower (or upper) triangular form of a correlation matrix calculated from `DataFrame.corr`.  This can be achieved by passing a boolean mask to ``where`` as follows:  .. ipython:: python  df = pd.DataFrame(np.random.random(size=(100, 5)))  corr_mat = df.corr()  mask = np.tril(np.ones_like(corr_mat, dtype=np.bool_), k=-1)  corr_mat.where(mask)  The `method`` argument within ``DataFrame.corr`` can accept a callable in addition to the named correlation types.  Here we compute the `distance correlation_ matrix for a ``DataFrame`` object.  .. ipython:: python  def distcorr(x, y):  n = len(x)  a = np.zeros(shape=(n, n))  b = np.zeros(shape=(n, n))  for i in range(n):  for j in range(i + 1, n):  a[i, j] = abs(x[i] - x[j])  b[i, j] = abs(y[i] - y[j])  a += a.T  b += b.T  a_bar = np.vstack([np.nanmean(a, axis=0)] * n)  b_bar = np.vstack([np.nanmean(b, axis=0)] * n)  A = a - a_bar - a_bar.T + np.full(shape=(n, n), fill_value=a_bar.mean())  B = b - b_bar - b_bar.T + np.full(shape=(n, n), fill_value=b_bar.mean())  cov_ab = np.sqrt(np.nansum(A * B)) / n  std_a = np.sqrt(np.sqrt(np.nansum(A ** 2)) / n)  std_b = np.sqrt(np.sqrt(np.nansum(B ** 2)) / n)  return cov_ab / std_a / std_b  df = pd.DataFrame(np.random.normal(size=(100, 3)))  df.corr(method=distcorr)

The `Timedeltas <timedeltas.timedeltas>` docs.  `Using timedeltas
<https://github.com/pandas-dev/pandas/pull/2899>`__  .. ipython:: python  import datetime  s = pd.Series(pd.date_range("2012-1-1", periods=3, freq="D"))  s - s.max()  s.max() - s  s - datetime.datetime(2011, 1, 1, 3, 5)  s + datetime.timedelta(minutes=5)  datetime.datetime(2011, 1, 1, 3, 5) - s  datetime.timedelta(minutes=5) + s  `Adding and subtracting deltas and dates
<https://stackoverflow.com/questions/16385785/add-days-to-dates-in-dataframe>`__  .. ipython:: python  deltas = pd.Series([datetime.timedelta(days=i) for i in range(3)])  df = pd.DataFrame({"A": s, "B": deltas})  df  df["New Dates"] = df["A"] + df["B"]  df["Delta"] = df["A"] - df["New Dates"]  df  df.dtypes  `Another example
<https://stackoverflow.com/questions/15683588/iterating-through-a-pandas-dataframe>`__  Values can be set to NaT using np.nan, similar to datetime  .. ipython:: python  y = s - s.shift()  y  y[1] = np.nan  y

Creating example data

To create a dataframe from every combination of some given values, like R's ``expand.grid()``
function, we can create a dict where the keys are column names and the values are lists
of the data values:  .. ipython:: python  def expand_grid(data_dict):  rows = itertools.product(*data_dict.values())  return pd.DataFrame.from_records(rows, columns=data_dict.keys())  df = expand_grid(  {"height": [60, 70], "weight": [100, 140, 180], "sex": ["Male", "Female"]}  )  df

`Series` is a one-dimensional labeled array capable of holding any data
type (integers, strings, floating point numbers, Python objects, etc.). The axis
labels are collectively referred to as the **index**. The basic method to create a `Series` is to call:  ::  >>> s = pd.Series(data, index=index)  Here, ``data`` can be many different things:  * a Python dict
* an ndarray
* a scalar value (like 5)  The passed **index** is a list of axis labels. Thus, this separates into a few
cases depending on what **data is**:  **From ndarray**  If ``data`` is an ndarray, **index** must be the same length as **data**. If no
index is passed, one will be created having values ``[0, ..., len(data) - 1]``.  .. ipython:: python  s = pd.Series(np.random.randn(5), index=["a", "b", "c", "d", "e"])  s  s.index  pd.Series(np.random.randn(5))  .. note::  pandas supports non-unique index values. If an operation  that does not support duplicate index values is attempted, an exception  will be raised at that time.  **From dict**  `Series` can be instantiated from dicts:  .. ipython:: python  d = {"b": 1, "a": 0, "c": 2}  pd.Series(d)  If an index is passed, the values in data corresponding to the labels in the
index will be pulled out.  .. ipython:: python  d = {"a": 0.0, "b": 1.0, "c": 2.0}  pd.Series(d)  pd.Series(d, index=["b", "c", "d", "a"])  .. note::  NaN (not a number) is the standard missing data marker used in pandas.  **From scalar value**  If ``data`` is a scalar value, an index must be
provided. The value will be repeated to match the length of **index**.  .. ipython:: python  pd.Series(5.0, index=["a", "b", "c", "d", "e"])  Series is ndarray-like
~~~  `Series` acts very similarly to a ``ndarray`` and is a valid argument to most NumPy functions.
However, operations such as slicing will also slice the index.  .. ipython:: python  s[0]  s[:3]  s[s > s.median()]  s[[4, 3, 1]]  np.exp(s)  .. note::  We will address array-based indexing like ``s[[4, 3, 1]]``  in `section on indexing <indexing>`.  Like a NumPy array, a pandas `Series` has a single `~Series.dtype`.  .. ipython:: python  s.dtype  This is often a NumPy dtype. However, pandas and 3rd-party libraries
extend NumPy's type system in a few places, in which case the dtype would
be an `~pandas.api.extensions.ExtensionDtype`. Some examples within
pandas are `categorical` and `integer_na`. See `basics.dtypes`
for more.  If you need the actual array backing a `Series`, use `Series.array`.  .. ipython:: python  s.array  Accessing the array can be useful when you need to do some operation without the
index (to disable `automatic alignment <dsintro.alignment>`, for example).  `Series.array` will always be an `~pandas.api.extensions.ExtensionArray`.
Briefly, an ExtensionArray is a thin wrapper around one or more *concrete* arrays like a
`numpy.ndarray`. pandas knows how to take an `~pandas.api.extensions.ExtensionArray` and
store it in a `Series` or a column of a `DataFrame`.
See `basics.dtypes` for more.  While `Series` is ndarray-like, if you need an *actual* ndarray, then use
`Series.to_numpy`.  .. ipython:: python  s.to_numpy()  Even if the `Series` is backed by a `~pandas.api.extensions.ExtensionArray`,
`Series.to_numpy` will return a NumPy ndarray.  Series is dict-like
~~~  A `Series` is also like a fixed-size dict in that you can get and set values by index
label:  .. ipython:: python  s["a"]  s["e"] = 12.0  s  "e" in s  "f" in s  If a label is not contained in the index, an exception is raised:  .. ipython:: python  s["f"]  Using the `Series.get` method, a missing label will return None or specified default:  .. ipython:: python  s.get("f")  s.get("f", np.nan)  These labels can also be accessed by `attribute<indexing.attribute_access>`.  Vectorized operations and label alignment with Series
~~~  When working with raw NumPy arrays, looping through value-by-value is usually
not necessary. The same is true when working with `Series` in pandas.

`Series` can also be passed into most NumPy methods expecting an ndarray.  .. ipython:: python  s + s  s * 2  np.exp(s)  A key difference between `Series` and ndarray is that operations between `Series`
automatically align the data based on label. Thus, you can write computations
without giving consideration to whether the `Series` involved have the same
labels.  .. ipython:: python  s[1:] + s[:-1]  The result of an operation between unaligned `Series` will have the **union** of
the indexes involved. If a label is not found in one `Series` or the other, the
result will be marked as missing ``NaN``. Being able to write code without doing
any explicit data alignment grants immense freedom and flexibility in
interactive data analysis and research. The integrated data alignment features
of the pandas data structures set pandas apart from the majority of related
tools for working with labeled data.  .. note::  In general, we chose to make the default result of operations between  differently indexed objects yield the **union** of the indexes in order to  avoid loss of information. Having an index label, though the data is  missing, is typically important information as part of a computation. You  of course have the option of dropping labels with missing data via the  **dropna** function.  Name attribute
~~~  .. _dsintro.name_attribute:  `Series` also has a ``name`` attribute:  .. ipython:: python  s = pd.Series(np.random.randn(5), name="something")  s  s.name  The `Series` ``name`` can be assigned automatically in many cases, in particular,
when selecting a single column from a `DataFrame`, the ``name`` will be assigned
the column label.  You can rename a `Series` with the `pandas.Series.rename` method.  .. ipython:: python  s2 = s.rename("different")  s2.name  Note that ``s`` and ``s2`` refer to different objects.  .. _basics.dataframe:

`DataFrame` is a 2-dimensional labeled data structure with columns of
potentially different types. You can think of it like a spreadsheet or SQL
table, or a dict of Series objects. It is generally the most commonly used
pandas object. Like Series, DataFrame accepts many different kinds of input:  * Dict of 1D ndarrays, lists, dicts, or `Series`
* 2-D numpy.ndarray
* `Structured or record  <https://numpy.org/doc/stable/user/basics.rec.html>`__ ndarray
* A `Series`
* Another `DataFrame`  Along with the data, you can optionally pass **index** (row labels) and
**columns** (column labels) arguments. If you pass an index and / or columns,
you are guaranteeing the index and / or columns of the resulting
DataFrame. Thus, a dict of Series plus a specific index will discard all data
not matching up to the passed index.  If axis labels are not passed, they will be constructed from the input data
based on common sense rules.  From dict of Series or dicts
~~~  The resulting **index** will be the **union** of the indexes of the various
Series. If there are any nested dicts, these will first be converted to
Series. If no columns are passed, the columns will be the ordered list of dict
keys.  .. ipython:: python  d = {  "one": pd.Series([1.0, 2.0, 3.0], index=["a", "b", "c"]),  "two": pd.Series([1.0, 2.0, 3.0, 4.0], index=["a", "b", "c", "d"]),  }  df = pd.DataFrame(d)  df  pd.DataFrame(d, index=["d", "b", "a"])  pd.DataFrame(d, index=["d", "b", "a"], columns=["two", "three"])  The row and column labels can be accessed respectively by accessing the
**index** and **columns** attributes:  .. note::  When a particular set of columns is passed along with a dict of data, the  passed columns override the keys in the dict.  .. ipython:: python  df.index  df.columns  From dict of ndarrays / lists
~~~  The ndarrays must all be the same length. If an index is passed, it must
also be the same length as the arrays. If no index is passed, the
result will be ``range(n)``, where ``n`` is the array length.  .. ipython:: python  d = {"one": [1.0, 2.0, 3.0, 4.0], "two": [4.0, 3.0, 2.0, 1.0]}  pd.DataFrame(d)  pd.DataFrame(d, index=["a", "b", "c", "d"])  From structured or record array
~~~  This case is handled identically to a dict of arrays.  .. ipython:: python  data = np.zeros((2,), dtype=[("A", "i4"), ("B", "f4"), ("C", "a10")])  data[:] = [(1, 2.0, "Hello"), (2, 3.0, "World")]  pd.DataFrame(data)  pd.DataFrame(data, index=["first", "second"])  pd.DataFrame(data, columns=["C", "A", "B"])  .. note::  DataFrame is not intended to work exactly like a 2-dimensional NumPy  ndarray.  .. _basics.dataframe.from_list_of_dicts:  From a list of dicts
~~~  .. ipython:: python  data2 = [{"a": 1, "b": 2}, {"a": 5, "b": 10, "c": 20}]  pd.DataFrame(data2)  pd.DataFrame(data2, index=["first", "second"])  pd.DataFrame(data2, columns=["a", "b"])  .. _basics.dataframe.from_dict_of_tuples:  From a dict of tuples
~~~  You can automatically create a MultiIndexed frame by passing a tuples
dictionary.  .. ipython:: python  pd.DataFrame(  {  ("a", "b"): {("A", "B"): 1, ("A", "C"): 2},  ("a", "a"): {("A", "C"): 3, ("A", "B"): 4},  ("a", "c"): {("A", "B"): 5, ("A", "C"): 6},  ("b", "a"): {("A", "C"): 7, ("A", "B"): 8},  ("b", "b"): {("A", "D"): 9, ("A", "B"): 10},  }  )  .. _basics.dataframe.from_series:  From a Series
~~~  The result will be a DataFrame with the same index as the input Series, and
with one column whose name is the original name of the Series (only if no other
column name provided).  .. ipython:: python  ser = pd.Series(range(3), index=list("abc"), name="ser")  pd.DataFrame(ser)  .. _basics.dataframe.from_list_namedtuples:  From a list of namedtuples
~~~  The field names of the first ``namedtuple`` in the list determine the columns
of the `DataFrame`. The remaining namedtuples (or tuples) are simply unpacked
and their values are fed into the rows of the `DataFrame`. If any of those
tuples is shorter than the first ``namedtuple`` then the later columns in the

corresponding row are marked as missing values. If any are longer than the
first ``namedtuple``, a ``ValueError`` is raised.  .. ipython:: python  from collections import namedtuple  Point = namedtuple("Point", "x y")  pd.DataFrame([Point(0, 0), Point(0, 3), (2, 3)])  Point3D = namedtuple("Point3D", "x y z")  pd.DataFrame([Point3D(0, 0, 0), Point3D(0, 3, 5), Point(2, 3)])  .. _basics.dataframe.from_list_dataclasses:  From a list of dataclasses
~~~  .. versionadded:: 1.1.0  Data Classes as introduced in PEP557_,
can be passed into the DataFrame constructor.
Passing a list of dataclasses is equivalent to passing a list of dictionaries.  Please be aware, that all values in the list should be dataclasses, mixing
types in the list would result in a ``TypeError``.  .. ipython:: python  from dataclasses import make_dataclass  Point = make_dataclass("Point", [("x", int), ("y", int)])  pd.DataFrame([Point(0, 0), Point(0, 3), Point(2, 3)])  **Missing data**  To construct a DataFrame with missing data, we use ``np.nan`` to
represent missing values. Alternatively, you may pass a ``numpy.MaskedArray``
as the data argument to the DataFrame constructor, and its masked entries will
be considered missing. See `Missing data <missing_data>` for more.  Alternate constructors
~~~  .. _basics.dataframe.from_dict:  **DataFrame.from_dict**  `DataFrame.from_dict` takes a dict of dicts or a dict of array-like sequences
and returns a DataFrame. It operates like the `DataFrame` constructor except
for the ``orient`` parameter which is ``'columns'`` by default, but which can be
set to ``'index'`` in order to use the dict keys as row labels.  .. ipython:: python  pd.DataFrame.from_dict(dict([("A", [1, 2, 3]), ("B", [4, 5, 6])]))  If you pass ``orient='index'``, the keys will be the row labels. In this
case, you can also pass the desired column names:  .. ipython:: python  pd.DataFrame.from_dict(  dict([("A", [1, 2, 3]), ("B", [4, 5, 6])]),  orient="index",  columns=["one", "two", "three"],  )  .. _basics.dataframe.from_records:  **DataFrame.from_records**  `DataFrame.from_records` takes a list of tuples or an ndarray with structured
dtype. It works analogously to the normal `DataFrame` constructor, except that
the resulting DataFrame index may be a specific field of the structured
dtype.  .. ipython:: python  data  pd.DataFrame.from_records(data, index="C")  .. _basics.dataframe.sel_add_del:  Column selection, addition, deletion
~~~  You can treat a `DataFrame` semantically like a dict of like-indexed `Series`
objects. Getting, setting, and deleting columns works with the same syntax as
the analogous dict operations:  .. ipython:: python  df["one"]  df["three"] = df["one"] * df["two"]  df["flag"] = df["one"] > 2  df  Columns can be deleted or popped like with a dict:  .. ipython:: python  del df["two"]  three = df.pop("three")  df  When inserting a scalar value, it will naturally be propagated to fill the
column:  .. ipython:: python  df["foo"] = "bar"  df  When inserting a `Series` that does not have the same index as the `DataFrame`, it
will be conformed to the DataFrame's index:  .. ipython:: python  df["one_trunc"] = df["one"][:2]  df  You can insert raw ndarrays but their length must match the length of the
DataFrame's index.  By default, columns get inserted at the end. `DataFrame.insert`
inserts at a particular location in the columns:  .. ipython:: python  df.insert(1, "bar", df["one"])  df  .. _dsintro.chained_assignment:  Assigning new columns in method chains
~~~  Inspired by `dplyr's
<https://dplyr.tidyverse.org/reference/mutate.html>`__
``mutate`` verb, DataFrame has an `~pandas.DataFrame.assign`
method that allows you to easily create new columns that are potentially

derived from existing columns.  .. ipython:: python  iris = pd.read_csv("data/iris.data")  iris.head()  iris.assign(sepal_ratio=iris["SepalWidth"] / iris["SepalLength"]).head()  In the example above, we inserted a precomputed value. We can also pass in
a function of one argument to be evaluated on the DataFrame being assigned to.  .. ipython:: python  iris.assign(sepal_ratio=lambda x: (x["SepalWidth"] / x["SepalLength"])).head()  `~pandas.DataFrame.assign` **always** returns a copy of the data, leaving the original
DataFrame untouched.  Passing a callable, as opposed to an actual value to be inserted, is
useful when you don't have a reference to the DataFrame at hand. This is
common when using `~pandas.DataFrame.assign` in a chain of operations. For example,
we can limit the DataFrame to just those observations with a Sepal Length
greater than 5, calculate the ratio, and plot:  .. ipython:: python  @savefig basics_assign.png  (  iris.query("SepalLength > 5")  .assign(  SepalRatio=lambda x: x.SepalWidth / x.SepalLength,  PetalRatio=lambda x: x.PetalWidth / x.PetalLength,  )  .plot(kind="scatter", x="SepalRatio", y="PetalRatio")  )  Since a function is passed in, the function is computed on the DataFrame
being assigned to. Importantly, this is the DataFrame that's been filtered
to those rows with sepal length greater than 5. The filtering happens first,
and then the ratio calculations. This is an example where we didn't
have a reference to the *filtered* DataFrame available.  The function signature for `~pandas.DataFrame.assign` is simply ``**kwargs``. The keys
are the column names for the new fields, and the values are either a value
to be inserted (for example, a `Series` or NumPy array), or a function
of one argument to be called on the `DataFrame`. A *copy* of the original
`DataFrame` is returned, with the new values inserted.  The order of ``**kwargs`` is preserved. This allows
for *dependent* assignment, where an expression later in ``**kwargs`` can refer
to a column created earlier in the same `~DataFrame.assign`.  .. ipython:: python  dfa = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})  dfa.assign(C=lambda x: x["A"] + x["B"], D=lambda x: x["A"] + x["C"])  In the second expression, ``x['C']`` will refer to the newly created column,
that's equal to ``dfa['A'] + dfa['B']``.  Indexing / selection
~~~
The basics of indexing are as follows:  .. csv-table::  "Operation", "Syntax", "Result"  30, 20, 10  Select column, ``df[col]``, Series  Select row by label, ``df.loc[label]``, Series  Select row by integer location, ``df.iloc[loc]``, Series  Slice rows, ``df[5:10]``, DataFrame  Select rows by boolean vector, ``df[bool_vec]``, DataFrame  Row selection, for example, returns a `Series` whose index is the columns of the
`DataFrame`:  .. ipython:: python  df.loc["b"]  df.iloc[2]  For a more exhaustive treatment of sophisticated label-based indexing and
slicing, see the `section on indexing <indexing>`. We will address the
fundamentals of reindexing / conforming to new sets of labels in the
`section on reindexing <basics.reindexing>`.  .. _dsintro.alignment:  Data alignment and arithmetic
~~~  Data alignment between `DataFrame` objects automatically align on **both the
columns and the index (row labels)**. Again, the resulting object will have the
union of the column and row labels.  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 4), columns=["A", "B", "C", "D"])  df2 = pd.DataFrame(np.random.randn(7, 3), columns=["A", "B", "C"])  df + df2  When doing an operation between `DataFrame` and `Series`, the default behavior is
to align the `Series` **index** on the `DataFrame` **columns**, thus `broadcasting
<https://numpy.org/doc/stable/user/basics.broadcasting.html>`__

row-wise. For example:  .. ipython:: python  df - df.iloc[0]  For explicit control over the matching and broadcasting behavior, see the
section on `flexible binary operations <basics.binop>`.  Arithmetic operations with scalars operate element-wise:  .. ipython:: python  df * 5 + 2  1 / df  df ** 4  .. _dsintro.boolean:  Boolean operators operate element-wise as well:  .. ipython:: python  df1 = pd.DataFrame({"a": [1, 0, 1], "b": [0, 1, 1]}, dtype=bool)  df2 = pd.DataFrame({"a": [0, 1, 1], "b": [1, 1, 0]}, dtype=bool)  df1 & df2  df1 | df2  df1 ^ df2  -df1  Transposing
~~~  To transpose, access the ``T`` attribute or `DataFrame.transpose`,
similar to an ndarray:  .. ipython:: python  # only show the first 5 rows  df[:5].T  .. _dsintro.numpy_interop:  DataFrame interoperability with NumPy functions
~~~  Most NumPy functions can be called directly on `Series` and `DataFrame`.  .. ipython:: python  np.exp(df)  np.asarray(df)  `DataFrame` is not intended to be a drop-in replacement for ndarray as its
indexing semantics and data model are quite different in places from an n-dimensional
array.  `Series` implements ``__array_ufunc__``, which allows it to work with NumPy's
universal functions.  The ufunc is applied to the underlying array in a `Series`.  .. ipython:: python  ser = pd.Series([1, 2, 3, 4])  np.exp(ser)  When multiple `Series` are passed to a ufunc, they are aligned before
performing the operation.  Like other parts of the library, pandas will automatically align labeled inputs
as part of a ufunc with multiple inputs. For example, using `numpy.remainder`
on two `Series` with differently ordered labels will align before the operation.  .. ipython:: python  ser1 = pd.Series([1, 2, 3], index=["a", "b", "c"])  ser2 = pd.Series([1, 3, 5], index=["b", "a", "c"])  ser1  ser2  np.remainder(ser1, ser2)  As usual, the union of the two indices is taken, and non-overlapping values are filled
with missing values.  .. ipython:: python  ser3 = pd.Series([2, 4, 6], index=["b", "c", "d"])  ser3  np.remainder(ser1, ser3)  When a binary ufunc is applied to a `Series` and `Index`, the `Series`
implementation takes precedence and a `Series` is returned.  .. ipython:: python  ser = pd.Series([1, 2, 3])  idx = pd.Index([4, 5, 6])  np.maximum(ser, idx)  NumPy ufuncs are safe to apply to `Series` backed by non-ndarray arrays,
for example `arrays.SparseArray` (see `sparse.calculation`). If possible,
the ufunc is applied without converting the underlying data to an ndarray.  Console display
~~~  A very large `DataFrame` will be truncated to display them in the console.
You can also get a summary using `~pandas.DataFrame.info`.
(The **baseball** dataset is from the **plyr** R package):  .. ipython:: python  # force a summary to be printed  pd.set_option("display.max_rows", 5)  .. ipython:: python  baseball = pd.read_csv("data/baseball.csv")  print(baseball)  baseball.info()  .. ipython:: python  # restore GlobalPrintConfig  pd.reset_option(r"^display\.")  However, using `DataFrame.to_string` will return a string representation of the
`DataFrame` in tabular form, though it won't always fit the console width:  .. ipython:: python  print(baseball.iloc[-20:, :12].to_string())  Wide DataFrames will be printed across multiple rows by
default:  .. ipython:: python  pd.DataFrame(np.random.randn(3, 12))  You can change how much to print on a single row by setting the ``display.width``

option:  .. ipython:: python  pd.set_option("display.width", 40)  # default is 80  pd.DataFrame(np.random.randn(3, 12))  You can adjust the max width of the individual columns by setting ``display.max_colwidth``  .. ipython:: python  datafile = {  "filename": ["filename_01", "filename_02"],  "path": [  "media/user_name/storage/folder_01/filename_01",  "media/user_name/storage/folder_02/filename_02",  ],  }  pd.set_option("display.max_colwidth", 30)  pd.DataFrame(datafile)  pd.set_option("display.max_colwidth", 100)  pd.DataFrame(datafile)  .. ipython:: python  pd.reset_option("display.width")  pd.reset_option("display.max_colwidth")  You can also disable this feature via the ``expand_frame_repr`` option.
This will print the table in one block.  DataFrame column attribute access and IPython completion
~~~  If a `DataFrame` column label is a valid Python variable name, the column can be
accessed like an attribute:  .. ipython:: python  df = pd.DataFrame({"foo1": np.random.randn(5), "foo2": np.random.randn(5)})  df  df.foo1  The columns are also connected to the IPython_
completion mechanism so they can be tab-completed:  .. code-block:: ipython  In [5]: df.foo<TAB>  # noqa: E225, E999  df.foo1  df.foo2

.. _duplicates:***Duplicate Labels***`Index` objects are not required to be unique; you can have duplicate rowor column labels. This may be a bit confusing at first. If you're familiar withSQL, you know that row labels are similar to a primary key on a table, and youwould never want duplicates in a SQL table. But one of pandas' roles is to cleanmessy, real-world data before it goes to some downstream system. And real-worlddata has duplicates, even in fields that are supposed to be unique.This section describes how duplicate labels change the behavior of certainoperations, and how prevent duplicates from arising during operations, or todetect them if they do... ipython:: python  import pandas as pd  import numpy as npConsequences of Duplicate Labels~~~Some pandas methods (`Series.reindex` for example) just don't work withduplicates present. The output can't be determined, and so pandas raises... ipython:: python  s1 = pd.Series([0, 1, 2], index=["a", "b", "b"])  s1.reindex(["a", "b", "c"])Other methods, like indexing, can give very surprising results. Typicallyindexing with a scalar will *reduce dimensionality*. Slicing a ``DataFrame``with a scalar will return a ``Series``. Slicing a ``Series`` with a scalar willreturn a scalar. But with duplicates, this isn't the case... ipython:: python  df1 = pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "A", "B"])  df1We have duplicates in the columns. If we slice ``'B'``, we get back a ``Series``.. ipython:: python  df1["B"]  # a seriesBut slicing ``'A'`` returns a ``DataFrame``.. ipython:: python  df1["A"]  # a DataFrameThis applies to row labels as well.. ipython:: python  df2 = pd.DataFrame({"A": [0, 1, 2]}, index=["a", "a", "b"])  df2  df2.loc["b", "A"]  # a scalar  df2.loc["a", "A"]  # a SeriesDuplicate Label Detection~~~You can check whether an `Index` (storing the row or column labels) isunique with `Index.is_unique`:.. ipython:: python  df2  df2.index.is_unique  df2.columns.is_unique.. note::  Checking whether an index is unique is somewhat expensive for large datasets.  pandas does cache this result, so re-checking on the same index is very fast.`Index.duplicated` will return a boolean ndarray indicating whether alabel is repeated... ipython:: python  df2.index.duplicated()Which can be used as a boolean filter to drop duplicate rows... ipython:: python  df2.loc[~df2.index.duplicated(), :]If you need additional logic to handle duplicate labels, rather than justdropping the repeats, using `~DataFrame.groupby` on the index is a commontrick. For example, we'll resolve duplicates by taking the average of all rowswith the same label... ipython:: python  df2.groupby(level=0).mean().. _duplicates.disallow:Disallowing Duplicate Labels~~~.. versionadded:: 1.2.0As noted above, handling duplicates is an important feature when reading in rawdata. That said, you may want to avoid introducing duplicates as part of a dataprocessing pipeline (from methods like `pandas.concat`,`~DataFrame.rename`, etc.). Both `Series` and `DataFrame`*disallow* duplicate labels by calling ``.set_flags(allows_duplicate_labels=False)``.(the default is to allow them). If there are duplicate labels, an exceptionwill be raised... ipython:: python  pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)This applies to both row and column labels for a `DataFrame`.. ipython:: python  pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "B", "C"],).set_flags(  allows_duplicate_labels=False  )This attribute can be checked or set with `~DataFrame.flags.allows_duplicate_labels`,which indicates whether that object can have duplicate labels... ipython:: python  df = pd.DataFrame({"A": [0, 1, 2, 3]}, index=["x", "y", "X", "Y"]).set_flags(  allows_duplicate_labels=False  )  df  df.flags.allows_duplicate_labels`DataFrame.set_flags` can be used to return a new ``DataFrame`` with attributeslike ``allows_duplicate_labels`` set to some value.. ipython:: python  df2 = df.set_flags(allows_duplicate_labels=True)  df2.flags.allows_duplicate_labelsThe new ``DataFrame`` returned is a view on the same data as the old ``DataFrame``.Or the property can just be set directly on the same object.. ipython:: python  df2.flags.allows_duplicate_labels = False  df2.flags.allows_duplicate_labelsWhen processing raw, messy data you might initially read in the messy data(which potentially has duplicate labels), deduplicate, and then disallow duplicatesgoing forward, to ensure that your data pipeline doesn't introduce duplicates... code-block:: python  >>> raw = pd.read_csv("...")  >>> deduplicated = raw.groupby(level=0).first()  # remove duplicates

and then disallow duplicatesgoing forward, to ensure that your data pipeline doesn't introduce duplicates... code-block:: python  >>> raw = pd.read_csv("...")  >>> deduplicated = raw.groupby(level=0).first()  # remove duplicates  >>> deduplicated.flags.allows_duplicate_labels = False  # disallow going forwardSetting ``allows_duplicate_labels=False`` on a ``Series`` or ``DataFrame`` with duplicatelabels or performing an operation that introduces duplicate labels on a ``Series`` or``DataFrame`` that disallows duplicates will raise an`errors.DuplicateLabelError`... ipython:: python  df.rename(str.upper)This error message contains the labels that are duplicated, and the numeric positionsof all the duplicates (including the "original") in the ``Series`` or ``DataFrame``Duplicate Label Propagation^^^In general, disallowing duplicates is "sticky". It's preserved throughoperations... ipython:: python  s1 = pd.Series(0, index=["a", "b"]).set_flags(allows_duplicate_labels=False)  s1  s1.head().rename({"a": "b"}).. warning::  This is an experimental feature. Currently, many methods fail to  propagate the ``allows_duplicate_labels`` value. In future versions  it is expected that every method taking or returning one or more  DataFrame or Series objects will propagate ``allows_duplicate_labels``.

For many use cases writing pandas in pure Python and NumPy is sufficient. In some
computationally heavy applications however, it can be possible to achieve sizable
speed-ups by offloading work to cython_.  This tutorial assumes you have refactored as much as possible in Python, for example
by trying to remove for-loops and making use of NumPy vectorization. It's always worth
optimising in Python first.  This tutorial walks through a "typical" process of cythonizing a slow computation.
We use an example from the Cython documentation_
but in the context of pandas. Our final cythonized solution is around 100 times
faster than the pure Python solution.  .. _enhancingperf.pure:  Pure Python
~~~  We have a `DataFrame` to which we want to apply a function row-wise.  .. ipython:: python  df = pd.DataFrame(  {  "a": np.random.randn(1000),  "b": np.random.randn(1000),  "N": np.random.randint(100, 1000, (1000)),  "x": "x",  }  )  df  Here's the function in pure Python:  .. ipython:: python  def f(x):  return x * (x - 1)  def integrate_f(a, b, N):  s = 0  dx = (b - a) / N  for i in range(N):  s += f(a + i * dx)  return s * dx  We achieve our result by using `DataFrame.apply` (row-wise):  .. ipython:: python  %timeit df.apply(lambda x: integrate_f(x["a"], x["b"], x["N"]), axis=1)  But clearly this isn't fast enough for us. Let's take a look and see where the
time is spent during this operation (limited to the most time consuming
four calls) using the prun ipython magic function_:  .. ipython:: python  %prun -l 4 df.apply(lambda x: integrate_f(x["a"], x["b"], x["N"]), axis=1)  # noqa E999  By far the majority of time is spend inside either ``integrate_f`` or ``f``,
hence we'll concentrate our efforts cythonizing these two functions.  .. _enhancingperf.plain:  Plain Cython
~~~  First we're going to need to import the Cython magic function to IPython:  .. ipython:: python  %load_ext Cython  Now, let's simply copy our functions over to Cython as is (the suffix
is here to distinguish between function versions):  .. ipython::  In [2]: %%cython  ...: def f_plain(x):  ...:  return x * (x - 1)  ...: def integrate_f_plain(a, b, N):  ...:  s = 0  ...:  dx = (b - a) / N  ...:  for i in range(N):  ...:  s += f_plain(a + i * dx)  ...:  return s * dx  ...:  .. note::  If you're having trouble pasting the above into your ipython, you may need  to be using bleeding edge IPython for paste to play well with cell magics.  .. ipython:: python  %timeit df.apply(lambda x: integrate_f_plain(x["a"], x["b"], x["N"]), axis=1)  Already this has shaved a third off, not too bad for a simple copy and paste.  .. _enhancingperf.type:  Adding type
~~~  We get another huge improvement simply by providing type information:  .. ipython::  In [3]: %%cython  ...: cdef double f_typed(double x) except? -2:  ...:  return x * (x - 1)  ...: cpdef double integrate_f_typed(double a, double b, int N):  ...:  cdef int i  ...:  cdef double s, dx  ...:  s = 0  ...:  dx = (b - a) / N  ...:  for i in range(N):  ...:  s += f_typed(a + i * dx)  ...:  return s * dx  ...:  .. ipython:: python  %timeit df.apply(lambda x: integrate_f_typed(x["a"], x["b"], x["N"]), axis=1)  Now, we're talking! It's now over ten times faster than the original Python
implementation, and we haven't *really* modified the code. Let's have another
look at what's eating up time:  .. ipython:: python  %prun -l 4 df.apply(lambda x: integrate_f_typed(x["a"], x["b"], x["N"]), axis=1)  .. _enhancingperf.ndarray:  Using ndarray
~~~  It's calling series a lot! It's creating a `Series` from each row, and calling get from both
the index and the series (three times for each row). Function calls are expensive

in Python, so maybe we could minimize these by cythonizing the apply part.  .. note::  We are now passing ndarrays into the Cython function, fortunately Cython plays  very nicely with NumPy.  .. ipython::  In [4]: %%cython  ...: cimport numpy as np  ...: import numpy as np  ...: cdef double f_typed(double x) except? -2:  ...:  return x * (x - 1)  ...: cpdef double integrate_f_typed(double a, double b, int N):  ...:  cdef int i  ...:  cdef double s, dx  ...:  s = 0  ...:  dx = (b - a) / N  ...:  for i in range(N):  ...:  s += f_typed(a + i * dx)  ...:  return s * dx  ...: cpdef np.ndarray[double] apply_integrate_f(np.ndarray col_a, np.ndarray col_b,  ...:  np.ndarray col_N):  ...:  assert (col_a.dtype == np.float_  ...:  and col_b.dtype == np.float_ and col_N.dtype == np.int_)  ...:  cdef Py_ssize_t i, n = len(col_N)  ...:  assert (len(col_a) == len(col_b) == n)  ...:  cdef np.ndarray[double] res = np.empty(n)  ...:  for i in range(len(col_a)):  ...:  res[i] = integrate_f_typed(col_a[i], col_b[i], col_N[i])  ...:  return res  ...:  The implementation is simple, it creates an array of zeros and loops over
the rows, applying our ``integrate_f_typed``, and putting this in the zeros array.  .. warning::  You can **not pass** a `Series` directly as a ``ndarray`` typed parameter  to a Cython function. Instead pass the actual ``ndarray`` using the  `Series.to_numpy`. The reason is that the Cython  definition is specific to an ndarray and not the passed `Series`.  So, do not do this:  .. code-block:: python  apply_integrate_f(df["a"], df["b"], df["N"])  But rather, use `Series.to_numpy` to get the underlying ``ndarray``:  .. code-block:: python  apply_integrate_f(df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy())  .. note::  Loops like this would be *extremely* slow in Python, but in Cython looping  over NumPy arrays is *fast*.  .. ipython:: python  %timeit apply_integrate_f(df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy())  We've gotten another big improvement. Let's check again where the time is spent:  .. ipython:: python  %prun -l 4 apply_integrate_f(df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy())  As one might expect, the majority of the time is now spent in ``apply_integrate_f``,
so if we wanted to make anymore efficiencies we must continue to concentrate our
efforts here.  .. _enhancingperf.boundswrap:  More advanced techniques
~~~  There is still hope for improvement. Here's an example of using some more
advanced Cython techniques:  .. ipython::  In [5]: %%cython  ...: cimport cython  ...: cimport numpy as np  ...: import numpy as np  ...: cdef np.float64_t f_typed(np.float64_t x) except? -2:  ...:  return x * (x - 1)  ...: cpdef np.float64_t integrate_f_typed(np.float64_t a, np.float64_t b, np.int64_t N):  ...:  cdef np.int64_t i  ...:  cdef np.float64_t s = 0.0, dx  ...:  dx = (b - a) / N  ...:  for i in range(N):  ...:  s += f_typed(a + i * dx)  ...:  return s * dx  ...: @cython.boundscheck(False)  ...: @cython.wraparound(False)  ...: cpdef np.ndarray[np.float64_t] apply_integrate_f_wrap(  ...:  np.ndarray[np.float64_t] col_a,  ...:  np.ndarray[np.float64_t] col_b,  ...:  np.ndarray[np.int64_t] col_N  ...: ):  ...:  cdef np.int64_t i, n = len(col_N)  ...:  assert len(col_a) == len(col_b) == n  ...:  cdef np.ndarray[np.float64_t] res = np.empty(n, dtype=np.float64)  ...:  for i in range(n):  ...:  res[i] = integrate_f_typed(col_a[i], col_b[i], col_N[i])  ...:  return res  ...:  .. ipython:: python  %timeit apply_integrate_f_wrap(df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy())  Even faster, with the caveat that a bug in our Cython code (an off-by-one error,
for example) might cause a segfault because memory access isn't checked.
For more about ``boundscheck`` and ``wraparound``, see the Cython docs on

compiler directives_.  .. _enhancingperf.numba:

An alternative to statically compiling Cython code is to use a dynamic just-in-time (JIT) compiler with Numba_.  Numba allows you to write a pure Python function which can be JIT compiled to native machine instructions, similar in performance to C, C++ and Fortran,
by decorating your function with ``@jit``.  Numba works by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool).
Numba supports compilation of Python to run on either CPU or GPU hardware and is designed to integrate with the Python scientific software stack.  .. note::  The ``@jit`` compilation will add overhead to the runtime of the function, so performance benefits may not be realized especially when using small data sets.  Consider caching_ your function to avoid compilation overhead each time your function is run.  Numba can be used in 2 ways with pandas:  #. Specify the ``engine="numba"`` keyword in select pandas methods
#. Define your own Python function decorated with ``@jit`` and pass the underlying NumPy array of `Series` or `DataFrame` (using ``to_numpy()``) into the function  pandas Numba Engine
~~~  If Numba is installed, one can specify ``engine="numba"`` in select pandas methods to execute the method using Numba.
Methods that support ``engine="numba"`` will also have an ``engine_kwargs`` keyword that accepts a dictionary that allows one to specify
``"nogil"``, ``"nopython"`` and ``"parallel"`` keys with boolean values to pass into the ``@jit`` decorator.
If ``engine_kwargs`` is not specified, it defaults to ``{"nogil": False, "nopython": True, "parallel": False}`` unless otherwise specified.  In terms of performance, **the first time a function is run using the Numba engine will be slow**
as Numba will have some function compilation overhead. However, the JIT compiled functions are cached,
and subsequent calls will be fast. In general, the Numba engine is performant with
a larger amount of data points (e.g. 1+ million).  .. code-block:: ipython  In [1]: data = pd.Series(range(1_000_000))  # noqa: E225  In [2]: roll = data.rolling(10)  In [3]: def f(x):  ...:  return np.sum(x) + 5  # Run the first time, compilation time will affect performance  In [4]: %timeit -r 1 -n 1 roll.apply(f, engine='numba', raw=True)  1.23 s ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)  # Function is cached and performance will improve  In [5]: %timeit roll.apply(f, engine='numba', raw=True)  188 ms ± 1.93 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [6]: %timeit roll.apply(f, engine='cython', raw=True)  3.92 s ± 59 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  If your compute hardware contains multiple CPUs, the largest performance gain can be realized by setting ``parallel`` to ``True``
to leverage more than 1 CPU. Internally, pandas leverages numba to parallelize computations over the columns of a `DataFrame`;
therefore, this performance benefit is only beneficial for a `DataFrame` with a large number of columns.  .. code-block:: ipython  In [1]: import numba  In [2]: numba.set_num_threads(1)  In [3]: df = pd.DataFrame(np.random.randn(10_000, 100))  In [4]: roll = df.rolling(100)  In [5]: %timeit roll.mean(engine="numba", engine_kwargs={"parallel": True})  347 ms ± 26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [6]: numba.set_num_threads(2)  In [7]: %timeit roll.mean(engine="numba", engine_kwargs={"parallel": True})  201 ms ± 2.97 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  Custom Function Examples
~~~  A custom Python function decorated with ``@jit`` can be used with pandas objects by passing their NumPy array

representations with ``to_numpy()``.  .. code-block:: python  import numba  @numba.jit  def f_plain(x):  return x * (x - 1)  @numba.jit  def integrate_f_numba(a, b, N):  s = 0  dx = (b - a) / N  for i in range(N):  s += f_plain(a + i * dx)  return s * dx  @numba.jit  def apply_integrate_f_numba(col_a, col_b, col_N):  n = len(col_N)  result = np.empty(n, dtype="float64")  assert len(col_a) == len(col_b) == n  for i in range(n):  result[i] = integrate_f_numba(col_a[i], col_b[i], col_N[i])  return result  def compute_numba(df):  result = apply_integrate_f_numba(  df["a"].to_numpy(), df["b"].to_numpy(), df["N"].to_numpy()  )  return pd.Series(result, index=df.index, name="result")  .. code-block:: ipython  In [4]: %timeit compute_numba(df)  1000 loops, best of 3: 798 us per loop  In this example, using Numba was faster than Cython.  Numba can also be used to write vectorized functions that do not require the user to explicitly
loop over the observations of a vector; a vectorized function will be applied to each row automatically.
Consider the following example of doubling each observation:  .. code-block:: python  import numba  def double_every_value_nonumba(x):  return x * 2  @numba.vectorize  def double_every_value_withnumba(x):  # noqa E501  return x * 2  .. code-block:: ipython  # Custom function without numba  In [5]: %timeit df["col1_doubled"] = df["a"].apply(double_every_value_nonumba)  # noqa E501  1000 loops, best of 3: 797 us per loop  # Standard implementation (faster than a custom function)  In [6]: %timeit df["col1_doubled"] = df["a"] * 2  1000 loops, best of 3: 233 us per loop  # Custom function with numba  In [7]: %timeit df["col1_doubled"] = double_every_value_withnumba(df["a"].to_numpy())  1000 loops, best of 3: 145 us per loop  Caveats
~~~  Numba is best at accelerating functions that apply numerical functions to NumPy
arrays. If you try to `@jit`` a function that contains unsupported `Python_
or NumPy_
code, compilation will revert object mode_ which
will mostly likely not speed up your function. If you would
prefer that Numba throw an error if it cannot compile a function in a way that
speeds up your code, pass Numba the argument
``nopython=True`` (e.g.  ``@jit(nopython=True)``). For more on
troubleshooting Numba modes, see the `Numba troubleshooting page
<https://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#the-compiled-code-is-too-slow>`__.  Using ``parallel=True`` (e.g. ``@jit(parallel=True)``) may result in a ``SIGABRT`` if the threading layer leads to unsafe
behavior. You can first specify a safe threading layer_
before running a JIT function with ``parallel=True``.  Generally if the you encounter a segfault (``SIGSEGV``) while using Numba, please report the issue
to the Numba issue tracker._  .. _enhancingperf.eval:

The top-level function `pandas.eval` implements expression evaluation of
`~pandas.Series` and `~pandas.DataFrame` objects.  .. note::  To benefit from using `~pandas.eval` you need to  install ``numexpr``. See the `recommended dependencies section  <install.recommended_dependencies>` for more details.  The point of using `~pandas.eval` for expression evaluation rather than
plain Python is two-fold: 1) large `~pandas.DataFrame` objects are
evaluated more efficiently and 2) large arithmetic and boolean expressions are
evaluated all at once by the underlying engine (by default ``numexpr`` is used
for evaluation).  .. note::  You should not use `~pandas.eval` for simple  expressions or for expressions involving small DataFrames. In fact,  `~pandas.eval` is many orders of magnitude slower for  smaller expressions/objects than plain ol' Python. A good rule of thumb is  to only use `~pandas.eval` when you have a  `~pandas.core.frame.DataFrame` with more than 10,000 rows.  `~pandas.eval` supports all arithmetic expressions supported by the
engine in addition to some extensions available only in pandas.  .. note::  The larger the frame and the larger the expression the more speedup you will  see from using `~pandas.eval`.  Supported syntax
~~~  These operations are supported by `pandas.eval`:  * Arithmetic operations except for the left shift (``<<``) and right shift  (``>>``) operators, e.g., ``df + 2 * pi / s ** 4 % 42 - the_golden_ratio``
* Comparison operations, including chained comparisons, e.g., ``2 < df < df2``
* Boolean operations, e.g., ``df < df2 and df3 < df4 or not df_bool``
* ``list`` and ``tuple`` literals, e.g., ``[1, 2]`` or ``(1, 2)``
* Attribute access, e.g., ``df.a``
* Subscript expressions, e.g., ``df[0]``
* Simple variable evaluation, e.g., ``pd.eval("df")`` (this is not very useful)
* Math functions: ``sin``, ``cos``, ``exp``, ``log``, ``expm1``, ``log1p``,  ``sqrt``, ``sinh``, ``cosh``, ``tanh``, ``arcsin``, ``arccos``, ``arctan``, ``arccosh``,  ``arcsinh``, ``arctanh``, ``abs``, ``arctan2`` and ``log10``.  This Python syntax is **not** allowed:  * Expressions  * Function calls other than math functions.  * ``is``/``is not`` operations  * ``if`` expressions  * ``lambda`` expressions  * ``list``/``set``/``dict`` comprehensions  * Literal ``dict`` and ``set`` expressions  * ``yield`` expressions  * Generator expressions  * Boolean expressions consisting of only scalar values  * Statements  * Neither simple_  nor compound_  statements are allowed. This includes things like ``for``, ``while``, and  ``if``.  `~pandas.eval` examples
~~~  `pandas.eval` works well with expressions containing large arrays.  First let's create a few decent-sized arrays to play with:  .. ipython:: python  nrows, ncols = 2000, 100  df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols)) for _ in range(4)]  Now let's compare adding them together using plain ol' Python versus
`~pandas.eval`:  .. ipython:: python  %timeit df1 + df2 + df3 + df4  .. ipython:: python  %timeit pd.eval("df1 + df2 + df3 + df4")  Now let's do the same thing but with comparisons:  .. ipython:: python  %timeit (df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)  .. ipython:: python  %timeit pd.eval("(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)")  `~pandas.eval` also works with unaligned pandas objects:  .. ipython:: python  s = pd.Series(np.random.randn(50))  %timeit df1 + df2 + df3 + df4 + s  .. ipython:: python  %timeit pd.eval("df1 + df2 + df3 + df4 + s")  .. note::  Operations such as  .. code-block:: python  1 and 2  # would parse to 1 & 2, but should evaluate to 2  3 or 4  # would parse to 3 | 4, but should evaluate to 3  ~1  # this is okay, but slower when using eval  should be performed in Python. An exception will be raised if you try to  perform any boolean/bitwise operations with scalar operands that are not  of type ``bool`` or ``np.bool_``. Again, you should perform these kinds of  operations in plain Python.  The `DataFrame.eval` method

~~~  In addition to the top level `pandas.eval` function you can also
evaluate an expression in the "context" of a `~pandas.DataFrame`.  .. ipython:: python  try:  del a  except NameError:  pass  try:  del b  except NameError:  pass  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 2), columns=["a", "b"])  df.eval("a + b")  Any expression that is a valid `pandas.eval` expression is also a valid
`DataFrame.eval` expression, with the added benefit that you don't have to
prefix the name of the `~pandas.DataFrame` to the column(s) you're
interested in evaluating.  In addition, you can perform assignment of columns within an expression.
This allows for *formulaic evaluation*.  The assignment target can be a
new column name or an existing column name, and it must be a valid Python
identifier.  .. ipython:: python  df = pd.DataFrame(dict(a=range(5), b=range(5, 10)))  df = df.eval("c = a + b")  df = df.eval("d = a + b + c")  df = df.eval("a = 1")  df  A copy of the `DataFrame` with the
new or modified columns is returned and the original frame is unchanged.  .. ipython:: python  df  df.eval("e = a - c")  df  As a convenience, multiple assignments can be performed by using a
multi-line string.  .. ipython:: python  df.eval(  """  c = a + b  d = a + b + c  a = 1""",  )  The equivalent in standard Python would be  .. ipython:: python  df = pd.DataFrame(dict(a=range(5), b=range(5, 10)))  df["c"] = df["a"] + df["b"]  df["d"] = df["a"] + df["b"] + df["c"]  df["a"] = 1  df  Local variables
~~~  You must *explicitly reference* any local variable that you want to use in an
expression by placing the ``@`` character in front of the name. This mechanism is
the same for both `DataFrame.query` and `DataFrame.eval`. For example,  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 2), columns=list("ab"))  newcol = np.random.randn(len(df))  df.eval("b + @newcol")  df.query("b < @newcol")  If you don't prefix the local variable with ``@``, pandas will raise an
exception telling you the variable is undefined.  When using `DataFrame.eval` and `DataFrame.query`, this allows you
to have a local variable and a `~pandas.DataFrame` column with the same
name in an expression.  .. ipython:: python  a = np.random.randn()  df.query("@a < a")  df.loc[a < df["a"]]  # same as the previous expression  With `pandas.eval` you cannot use the ``@`` prefix *at all*, because it
isn't defined in that context. pandas will let you know this if you try to
use ``@`` in a top-level call to `pandas.eval`. For example,  .. ipython:: python  a, b = 1, 2  pd.eval("@a + b")  In this case, you should simply refer to the variables like you would in
standard Python.  .. ipython:: python  pd.eval("a + b")  `pandas.eval` parsers
~~~  There are two different parsers and two different engines you can use as
the backend.  The default ``'pandas'`` parser allows a more intuitive syntax for expressing
query-like operations (comparisons, conjunctions and disjunctions). In
particular, the precedence of the ``&`` and ``|`` operators is made equal to
the precedence of the corresponding boolean operations ``and`` and ``or``.  For example, the above conjunction can be written without parentheses.
Alternatively, you can use the ``'python'`` parser to enforce strict Python
semantics.  .. ipython:: python  expr = "(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)"  x = pd.eval(expr, parser="python")  expr_no_parens = "df1 > 0 & df2 > 0 & df3 > 0 & df4 > 0"  y = pd.eval(expr_no_parens, parser="pandas")  np.all(x == y)  The same expression can be "anded" together with the word `and` as
well:  .. ipython:: python  expr = "(df1 > 0) & (df2 > 0) & (df3 > 0) & (df4 > 0)"  x = pd.eval(expr, parser="python")  expr_with_ands = "df1 > 0 and df2 > 0 and df3 > 0 and df4 > 0"  y = pd.eval(expr_with_ands, parser="pandas")  np.all(x == y)  The ``and`` and ``or`` operators here have the same precedence that they would
in vanilla Python.  `pandas.eval` backends

~~~  There's also the option to make `~pandas.eval` operate identical to plain
ol' Python.  .. note::  Using the ``'python'`` engine is generally *not* useful, except for testing  other evaluation engines against it. You will achieve **no** performance  benefits using `~pandas.eval` with ``engine='python'`` and in fact may  incur a performance hit.  You can see this by using `pandas.eval` with the ``'python'`` engine. It
is a bit slower (not by much) than evaluating the same expression in Python  .. ipython:: python  %timeit df1 + df2 + df3 + df4  .. ipython:: python  %timeit pd.eval("df1 + df2 + df3 + df4", engine="python")  `pandas.eval` performance
~~~  `~pandas.eval` is intended to speed up certain kinds of operations. In
particular, those operations involving complex expressions with large
`~pandas.DataFrame`/`~pandas.Series` objects should see a
significant performance benefit.  Here is a plot showing the running time of
`pandas.eval` as function of the size of the frame involved in the
computation. The two lines are two different engines.  ..  The eval-perf.png figure below was generated with /doc/scripts/eval_performance.py  You will only see the performance benefits of using the ``numexpr`` engine with `pandas.eval` if your frame has more than approximately 100,000 rows.  This plot was created using a `DataFrame` with 3 columns each containing
floating point values generated using ``numpy.random.randn()``.  Technical minutia regarding expression evaluation
~~~  Expressions that would result in an object dtype or involve datetime operations
(because of ``NaT``) must be evaluated in Python space. The main reason for
this behavior is to maintain backwards compatibility with versions of NumPy <
1.7. In those versions of NumPy a call to ``ndarray.astype(str)`` will
truncate any strings that are more than 60 characters in length. Second, we
can't pass ``object`` arrays to ``numexpr`` thus string comparisons must be
evaluated in Python space.  The upshot is that this *only* applies to object-dtype expressions. So, if
you have an expression--for example  .. ipython:: python  df = pd.DataFrame(  {"strings": np.repeat(list("cba"), 3), "nums": np.repeat(range(3), 3)}  )  df  df.query("strings == 'a' and nums == 1")  the numeric part of the comparison (``nums == 1``) will be evaluated by
``numexpr``.  In general, `DataFrame.query`/`pandas.eval` will
evaluate the subexpressions that *can* be evaluated by ``numexpr`` and those
that must be evaluated in Python space transparently to the user. This is done
by inferring the result type of an expression from its arguments and operators.

DataFrame memory usage
The memory usage of a `DataFrame` (including the index) is shown when calling
the `~DataFrame.info`. A configuration option, ``display.memory_usage``
(see `the list of options <options.available>`), specifies if the
`DataFrame` memory usage will be displayed when invoking the ``df.info()``
method.  For example, the memory usage of the `DataFrame` below is shown
when calling `~DataFrame.info`:  .. ipython:: python  dtypes = [  "int64",  "float64",  "datetime64[ns]",  "timedelta64[ns]",  "complex128",  "object",  "bool",  ]  n = 5000  data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}  df = pd.DataFrame(data)  df["categorical"] = df["object"].astype("category")  df.info()  The ``+`` symbol indicates that the true memory usage could be higher, because
pandas does not count the memory used by values in columns with
``dtype=object``.  Passing ``memory_usage='deep'`` will enable a more accurate memory usage report,
accounting for the full usage of the contained objects. This is optional
as it can be expensive to do this deeper introspection.  .. ipython:: python  df.info(memory_usage="deep")  By default the display option is set to ``True`` but can be explicitly
overridden by passing the ``memory_usage`` argument when invoking ``df.info()``.  The memory usage of each column can be found by calling the
`~DataFrame.memory_usage` method. This returns a `Series` with an index
represented by column names and memory usage of each column shown in bytes. For
the `DataFrame` above, the memory usage of each column and the total memory
usage can be found with the ``memory_usage`` method:  .. ipython:: python  df.memory_usage()  # total memory usage of dataframe  df.memory_usage().sum()  By default the memory usage of the `DataFrame` index is shown in the
returned `Series`, the memory usage of the index can be suppressed by passing
the ``index=False`` argument:  .. ipython:: python  df.memory_usage(index=False)  The memory usage displayed by the `~DataFrame.info` method utilizes the
`~DataFrame.memory_usage` method to determine the memory usage of a
`DataFrame` while also formatting the output in human-readable units (base-2
representation; i.e. 1KB = 1024 bytes).  See also `Categorical Memory Usage <categorical.memory>`.  .. _gotchas.truth:

Using if/truth statements with pandas

pandas follows the NumPy convention of raising an error when you try to convert
something to a ``bool``. This happens in an ``if``-statement or when using the
boolean operations: ``and``, ``or``, and ``not``. It is not clear what the result
of the following code should be:  .. code-block:: python  >>> if pd.Series([False, True, False]):  ...  pass  Should it be ``True`` because it's not zero-length, or ``False`` because there
are ``False`` values? It is unclear, so instead, pandas raises a ``ValueError``:  .. ipython:: python  if pd.Series([False, True, False]):  print("I was true")  You need to explicitly choose what you want to do with the `DataFrame`, e.g.
use `~DataFrame.any`, `~DataFrame.all` or `~DataFrame.empty`.
Alternatively, you might want to compare if the pandas object is ``None``:  .. ipython:: python  if pd.Series([False, True, False]) is not None:  print("I was not None")  Below is how to check if any of the values are ``True``:  .. ipython:: python  if pd.Series([False, True, False]).any():  print("I am any")  To evaluate single-element pandas objects in a boolean context, use the method
`~DataFrame.bool`:  .. ipython:: python  pd.Series([True]).bool()  pd.Series([False]).bool()  pd.DataFrame([[True]]).bool()  pd.DataFrame([[False]]).bool()  Bitwise boolean
~~~  Bitwise boolean operators like ``==`` and ``!=`` return a boolean `Series`
which performs an element-wise comparison when compared to a scalar.  .. ipython:: python  s = pd.Series(range(5))  s == 4  See `boolean comparisons<basics.compare>` for more examples.  Using the ``in`` operator
~~~  Using the Python ``in`` operator on a `Series` tests for membership in the
**index**, not membership among the values.  .. ipython:: python  s = pd.Series(range(5), index=list("abcde"))  2 in s  'b' in s  If this behavior is surprising, keep in mind that using ``in`` on a Python
dictionary tests keys, not values, and `Series` are dict-like.
To test for membership in the values, use the method `~pandas.Series.isin`:  .. ipython:: python  s.isin([2])  s.isin([2]).any()  For `DataFrame`, likewise, ``in`` applies to the column axis,
testing for membership in the list of column names.  .. _gotchas.udf-mutation:

Mutating with User Defined Function (UDF) methods

This section applies to pandas methods that take a UDF. In particular, the methods
``.apply``, ``.aggregate``, ``.transform``, and ``.filter``.  It is a general rule in programming that one should not mutate a container
while it is being iterated over. Mutation will invalidate the iterator,
causing unexpected behavior. Consider the example:  .. ipython:: python  values = [0, 1, 2, 3, 4, 5]  n_removed = 0  for k, value in enumerate(values):  idx = k - n_removed  if value % 2 == 1:  del values[idx]  n_removed += 1  else:  values[idx] = value + 1  values  One probably would have expected that the result would be ``[1, 3, 5]``.
When using a pandas method that takes a UDF, internally pandas is often
iterating over the
`DataFrame` or other pandas object. Therefore, if the UDF mutates (changes)
the `DataFrame`, unexpected behavior can arise.  Here is a similar example with `DataFrame.apply`:  .. ipython:: python  def f(s):  s.pop("a")  return s  df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})  try:  df.apply(f, axis="columns")  except Exception as err:  print(repr(err))  To resolve this issue, one can make a copy so that the mutation does
not apply to the container being iterated over.  .. ipython:: python  values = [0, 1, 2, 3, 4, 5]  n_removed = 0  for k, value in enumerate(values.copy()):  idx = k - n_removed  if value % 2 == 1:  del values[idx]  n_removed += 1  else:  values[idx] = value + 1  values  .. ipython:: python  def f(s):  s = s.copy()  s.pop("a")  return s  df = pd.DataFrame({"a": [1, 2, 3], 'b': [4, 5, 6]})  df.apply(f, axis="columns")

Choice of ``NA`` representation
~~~  For lack of ``NA`` (missing) support from the ground up in NumPy and Python in
general, we were given the difficult choice between either:  * A *masked array* solution: an array of data and an array of boolean values  indicating whether a value is there or is missing.
* Using a special sentinel value, bit pattern, or set of sentinel values to  denote ``NA`` across the dtypes.  For many reasons we chose the latter. After years of production use it has
proven, at least in my opinion, to be the best decision given the state of
affairs in NumPy and Python in general. The special value ``NaN``
(Not-A-Number) is used everywhere as the ``NA`` value, and there are API
functions `DataFrame.isna` and `DataFrame.notna` which can be used across the dtypes to
detect NA values.  However, it comes with it a couple of trade-offs which I most certainly have
not ignored.  .. _gotchas.intna:  Support for integer ``NA``
~~~  In the absence of high performance ``NA`` support being built into NumPy from
the ground up, the primary casualty is the ability to represent NAs in integer
arrays. For example:  .. ipython:: python  s = pd.Series([1, 2, 3, 4, 5], index=list("abcde"))  s  s.dtype  s2 = s.reindex(["a", "b", "c", "f", "u"])  s2  s2.dtype  This trade-off is made largely for memory and performance reasons, and also so
that the resulting `Series` continues to be "numeric".  If you need to represent integers with possibly missing values, use one of
the nullable-integer extension dtypes provided by pandas  * `Int8Dtype`
* `Int16Dtype`
* `Int32Dtype`
* `Int64Dtype`  .. ipython:: python  s_int = pd.Series([1, 2, 3, 4, 5], index=list("abcde"), dtype=pd.Int64Dtype())  s_int  s_int.dtype  s2_int = s_int.reindex(["a", "b", "c", "f", "u"])  s2_int  s2_int.dtype  See `integer_na` for more.  ``NA`` type promotions
~~~  When introducing NAs into an existing `Series` or `DataFrame` via
`~Series.reindex` or some other means, boolean and integer types will be
promoted to a different dtype in order to store the NAs. The promotions are
summarized in this table:  .. csv-table::  "Typeclass","Promotion dtype for storing NAs"  40,60  ``floating``, no change  ``object``, no change  ``integer``, cast to ``float64``  ``boolean``, cast to ``object``  While this may seem like a heavy trade-off, I have found very few cases where
this is an issue in practice i.e. storing values greater than 2**53. Some
explanation for the motivation is in the next section.  Why not make NumPy like R?
~~~  Many people have suggested that NumPy should simply emulate the ``NA`` support
present in the more domain-specific statistical programming language `R
<https://www.r-project.org/>`__. Part of the reason is the NumPy type hierarchy:  .. csv-table::  "Typeclass","Dtypes"  30,70  |  ``numpy.floating`` | ``float16, float32, float64, float128``  ``numpy.integer`` | ``int8, int16, int32, int64``  ``numpy.unsignedinteger`` | ``uint8, uint16, uint32, uint64``  ``numpy.object_`` | ``object_``  ``numpy.bool_`` | ``bool_``  ``numpy.character`` | ``string_, unicode_``  The R language, by contrast, only has a handful of built-in data types:
``integer``, ``numeric`` (floating-point), ``character``, and
``boolean``. ``NA`` types are implemented by reserving special bit patterns for
each type to be used as the missing value. While doing this with the full NumPy
type hierarchy would be possible, it would be a more substantial trade-off
(especially for the 8- and 16-bit data types) and implementation undertaking.  An alternate approach is that of using masked arrays. A masked array is an
array of data with an associated boolean *mask* denoting whether each value
should be considered ``NA`` or not. I am personally not in love with this
approach as I feel that overall it places a fairly heavy burden on the user and
the library implementer. Additionally, it exacts a fairly high performance cost
when working with numerical data compared with the simple approach of using

``NaN``. Thus, I have chosen the Pythonic "practicality beats purity" approach
and traded integer ``NA`` capability for a much simpler approach of using a
special value in float and object arrays to denote ``NA``, and promoting
integer arrays to floating when NAs must be introduced.

Differences with NumPy
For `Series` and `DataFrame` objects, `~DataFrame.var` normalizes by
`N-1`` to produce `unbiased estimates of the population variance_, while NumPy's
`numpy.var` normalizes by N, which measures the variance of the sample. Note that
`~DataFrame.cov` normalizes by ``N-1`` in both pandas and NumPy.  .. _gotchas.thread-safety:

Thread-safety

pandas is not 100% thread safe. The known issues relate to
the `~DataFrame.copy` method. If you are doing a lot of copying of
`DataFrame` objects shared among threads, we recommend holding locks inside
the threads where the data copying occurs.  See this link_
for more information.

Byte-ordering issues
Occasionally you may have to deal with data that were created on a machine with
a different byte order than the one on which you are running Python. A common
symptom of this issue is an error like::  Traceback  ...  ValueError: Big-endian buffer not supported on little-endian compiler  To deal
with this issue you should convert the underlying NumPy array to the native
system byte order *before* passing it to `Series` or `DataFrame`
constructors using something similar to the following:  .. ipython:: python  x = np.array(list(range(10)), ">i4")  # big endian  newx = x.byteswap().newbyteorder()  # force native byteorder  s = pd.Series(newx)  See `the NumPy documentation on byte order
<https://numpy.org/doc/stable/user/basics.byteswapping.html>`__ for more
details.

pandas objects can be split on any of their axes. The abstract definition of
grouping is to provide a mapping of labels to group names. To create a GroupBy
object (more on what the GroupBy object is later), you may do the following:  .. ipython:: python  df = pd.DataFrame(  [  ("bird", "Falconiformes", 389.0),  ("bird", "Psittaciformes", 24.0),  ("mammal", "Carnivora", 80.2),  ("mammal", "Primates", np.nan),  ("mammal", "Carnivora", 58),  ],  index=["falcon", "parrot", "lion", "monkey", "leopard"],  columns=("class", "order", "max_speed"),  )  df  # default is axis=0  grouped = df.groupby("class")  grouped = df.groupby("order", axis="columns")  grouped = df.groupby(["class", "order"])  The mapping can be specified many different ways:  * A Python function, to be called on each of the axis labels.
* A list or NumPy array of the same length as the selected axis.
* A dict or ``Series``, providing a ``label -> group name`` mapping.
* For ``DataFrame`` objects, a string indicating either a column name or  an index level name to be used to group.
* ``df.groupby('A')`` is just syntactic sugar for ``df.groupby(df['A'])``.
* A list of any of the above things.  Collectively we refer to the grouping objects as the **keys**. For example,
consider the following ``DataFrame``:  .. note::  A string passed to ``groupby`` may refer to either a column or an index level.  If a string matches both a column name and an index level name, a  ``ValueError`` will be raised.  .. ipython:: python  df = pd.DataFrame(  {  "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],  "B": ["one", "one", "two", "three", "two", "two", "one", "three"],  "C": np.random.randn(8),  "D": np.random.randn(8),  }  )  df  On a DataFrame, we obtain a GroupBy object by calling `~DataFrame.groupby`.
We could naturally group by either the ``A`` or ``B`` columns, or both:  .. ipython:: python  grouped = df.groupby("A")  grouped = df.groupby(["A", "B"])  If we also have a MultiIndex on columns ``A`` and ``B``, we can group by all
but the specified columns  .. ipython:: python  df2 = df.set_index(["A", "B"])  grouped = df2.groupby(level=df2.index.names.difference(["B"]))  grouped.sum()  These will split the DataFrame on its index (rows). We could also split by the
columns:  .. ipython::  In [4]: def get_letter_type(letter):  ...:  if letter.lower() in 'aeiou':  ...:  return 'vowel'  ...:  else:  ...:  return 'consonant'  ...:  In [5]: grouped = df.groupby(get_letter_type, axis=1)  pandas `~pandas.Index` objects support duplicate values. If a
non-unique index is used as the group key in a groupby operation, all values
for the same index value will be considered to be in one group and thus the
output of aggregation functions will only contain unique index values:  .. ipython:: python  lst = [1, 2, 3, 1, 2, 3]  s = pd.Series([1, 2, 3, 10, 20, 30], lst)  grouped = s.groupby(level=0)  grouped.first()  grouped.last()  grouped.sum()  Note that **no splitting occurs** until it's needed. Creating the GroupBy object
only verifies that you've passed a valid mapping.  .. note::  Many kinds of complicated data manipulations can be expressed in terms of  GroupBy operations (though can't be guaranteed to be the most  efficient). You can get quite creative with the label mapping functions.  .. _groupby.sorting:  GroupBy sorting
~~~  By default the group keys are sorted during the ``groupby`` operation. You may however pass ``sort=False`` for potential speedups:  .. ipython:: python  df2 = pd.DataFrame({"X": ["B", "B", "A", "A"], "Y": [1, 2, 3, 4]})  df2.groupby(["X"]).sum()  df2.groupby(["X"], sort=False).sum()  Note that ``groupby`` will preserve the order in which *observations* are sorted *within* each group.

For example, the groups created by ``groupby()`` below are in the order they appeared in the original ``DataFrame``:  .. ipython:: python  df3 = pd.DataFrame({"X": ["A", "B", "A", "B"], "Y": [1, 4, 3, 2]})  df3.groupby(["X"]).get_group("A")  df3.groupby(["X"]).get_group("B")  .. _groupby.dropna:  .. versionadded:: 1.1.0  GroupBy dropna
^^^  By default ``NA`` values are excluded from group keys during the ``groupby`` operation. However,
in case you want to include ``NA`` values in group keys, you could pass ``dropna=False`` to achieve it.  .. ipython:: python  df_list = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]]  df_dropna = pd.DataFrame(df_list, columns=["a", "b", "c"])  df_dropna  .. ipython:: python  # Default ``dropna`` is set to True, which will exclude NaNs in keys  df_dropna.groupby(by=["b"], dropna=True).sum()  # In order to allow NaN in keys, set ``dropna`` to False  df_dropna.groupby(by=["b"], dropna=False).sum()  The default setting of ``dropna`` argument is ``True`` which means ``NA`` are not included in group keys.  .. _groupby.attributes:  GroupBy object attributes
~~~  The ``groups`` attribute is a dict whose keys are the computed unique groups
and corresponding values being the axis labels belonging to each group. In the
above example we have:  .. ipython:: python  df.groupby("A").groups  df.groupby(get_letter_type, axis=1).groups  Calling the standard Python ``len`` function on the GroupBy object just returns
the length of the ``groups`` dict, so it is largely just a convenience:  .. ipython:: python  grouped = df.groupby(["A", "B"])  grouped.groups  len(grouped)  .. _groupby.tabcompletion:  ``GroupBy`` will tab complete column names (and other attributes):  .. ipython:: python  n = 10  weight = np.random.normal(166, 20, size=n)  height = np.random.normal(60, 10, size=n)  time = pd.date_range("1/1/2000", periods=n)  gender = np.random.choice(["male", "female"], size=n)  df = pd.DataFrame(  {"height": height, "weight": weight, "gender": gender}, index=time  )  .. ipython:: python  df  gb = df.groupby("gender")  .. ipython::  @verbatim  In [1]: gb.<TAB>  # noqa: E225, E999  gb.agg  gb.boxplot  gb.cummin  gb.describe  gb.filter  gb.get_group  gb.height  gb.last  gb.median  gb.ngroups  gb.plot  gb.rank  gb.std  gb.transform  gb.aggregate  gb.count  gb.cumprod  gb.dtype  gb.first  gb.groups  gb.hist  gb.max  gb.min  gb.nth  gb.prod  gb.resample  gb.sum  gb.var  gb.apply  gb.cummax  gb.cumsum  gb.fillna  gb.gender  gb.head  gb.indices  gb.mean  gb.name  gb.ohlc  gb.quantile  gb.size  gb.tail  gb.weight  .. _groupby.multiindex:  GroupBy with MultiIndex
~~~  With `hierarchically-indexed data <advanced.hierarchical>`, it's quite
natural to group by one of the levels of the hierarchy.  Let's create a Series with a two-level ``MultiIndex``.  .. ipython:: python  arrays = [  ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],  ["one", "two", "one", "two", "one", "two", "one", "two"],  ]  index = pd.MultiIndex.from_arrays(arrays, names=["first", "second"])  s = pd.Series(np.random.randn(8), index=index)  s  We can then group by one of the levels in ``s``.  .. ipython:: python  grouped = s.groupby(level=0)  grouped.sum()  If the MultiIndex has names specified, these can be passed instead of the level

number:  .. ipython:: python  s.groupby(level="second").sum()  Grouping with multiple levels is supported.  .. ipython:: python  arrays = [  ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],  ["doo", "doo", "bee", "bee", "bop", "bop", "bop", "bop"],  ["one", "two", "one", "two", "one", "two", "one", "two"],  ]  tuples = list(zip(*arrays))  index = pd.MultiIndex.from_tuples(tuples, names=["first", "second", "third"])  s = pd.Series(np.random.randn(8), index=index)  .. ipython:: python  s  s.groupby(level=["first", "second"]).sum()  Index level names may be supplied as keys.  .. ipython:: python  s.groupby(["first", "second"]).sum()  More on the ``sum`` function and aggregation later.  Grouping DataFrame with Index levels and columns
~~~
A DataFrame may be grouped by a combination of columns and index levels by
specifying the column names as strings and the index levels as ``pd.Grouper``
objects.  .. ipython:: python  arrays = [  ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],  ["one", "two", "one", "two", "one", "two", "one", "two"],  ]  index = pd.MultiIndex.from_arrays(arrays, names=["first", "second"])  df = pd.DataFrame({"A": [1, 1, 1, 1, 2, 2, 3, 3], "B": np.arange(8)}, index=index)  df  The following example groups ``df`` by the ``second`` index level and
the ``A`` column.  .. ipython:: python  df.groupby([pd.Grouper(level=1), "A"]).sum()  Index levels may also be specified by name.  .. ipython:: python  df.groupby([pd.Grouper(level="second"), "A"]).sum()  Index level names may be specified as keys directly to ``groupby``.  .. ipython:: python  df.groupby(["second", "A"]).sum()  DataFrame column selection in GroupBy
~~~  Once you have created the GroupBy object from a DataFrame, you might want to do
something different for each of the columns. Thus, using ``[]`` similar to
getting a column from a DataFrame, you can do:  .. ipython:: python  df = pd.DataFrame(  {  "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],  "B": ["one", "one", "two", "three", "two", "two", "one", "three"],  "C": np.random.randn(8),  "D": np.random.randn(8),  }  )  df  grouped = df.groupby(["A"])  grouped_C = grouped["C"]  grouped_D = grouped["D"]  This is mainly syntactic sugar for the alternative and much more verbose:  .. ipython:: python  df["C"].groupby(df["A"])  Additionally this method avoids recomputing the internal grouping information
derived from the passed key.  .. _groupby.iterating-label:

Iterating through groups

With the GroupBy object in hand, iterating through the grouped data is very
natural and functions similarly to func:`itertools.groupby`:  .. ipython::  In [4]: grouped = df.groupby('A')  In [5]: for name, group in grouped:  ...:  print(name)  ...:  print(group)  ...:  In the case of grouping by multiple keys, the group name will be a tuple:  .. ipython::  In [5]: for name, group in df.groupby(['A', 'B']):  ...:  print(name)  ...:  print(group)  ...:  See `timeseries.iterating-label`.

Selecting a group

A single group can be selected using
`~pandas.core.groupby.DataFrameGroupBy.get_group`:  .. ipython:: python  grouped.get_group("bar")  Or for an object grouped on multiple columns:  .. ipython:: python  df.groupby(["A", "B"]).get_group(("bar", "one"))  .. _groupby.aggregate:

Once the GroupBy object has been created, several methods are available to
perform a computation on the grouped data. These operations are similar to the
`aggregating API <basics.aggregate>`, `window API <window.overview>`,
and `resample API <timeseries.aggregate>`.  An obvious one is aggregation via the
`~pandas.core.groupby.DataFrameGroupBy.aggregate` or equivalently
`~pandas.core.groupby.DataFrameGroupBy.agg` method:  .. ipython:: python  grouped = df.groupby("A")  grouped[["C", "D"]].aggregate(np.sum)  grouped = df.groupby(["A", "B"])  grouped.aggregate(np.sum)  As you can see, the result of the aggregation will have the group names as the
new index along the grouped axis. In the case of multiple keys, the result is a
`MultiIndex <advanced.hierarchical>` by default, though this can be
changed by using the ``as_index`` option:  .. ipython:: python  grouped = df.groupby(["A", "B"], as_index=False)  grouped.aggregate(np.sum)  df.groupby("A", as_index=False)[["C", "D"]].sum()  Note that you could use the ``reset_index`` DataFrame function to achieve the
same result as the column names are stored in the resulting ``MultiIndex``:  .. ipython:: python  df.groupby(["A", "B"]).sum().reset_index()  Another simple aggregation example is to compute the size of each group.
This is included in GroupBy as the ``size`` method. It returns a Series whose
index are the group names and whose values are the sizes of each group.  .. ipython:: python  grouped.size()  .. ipython:: python  grouped.describe()  Another aggregation example is to compute the number of unique values of each group. This is similar to the ``value_counts`` function, except that it only counts unique values.  .. ipython:: python  ll = [['foo', 1], ['foo', 2], ['foo', 2], ['bar', 1], ['bar', 1]]  df4 = pd.DataFrame(ll, columns=["A", "B"])  df4  df4.groupby("A")["B"].nunique()  .. note::  Aggregation functions **will not** return the groups that you are aggregating over  if they are named *columns*, when ``as_index=True``, the default. The grouped columns will  be the **indices** of the returned object.  Passing ``as_index=False`` **will** return the groups that you are aggregating over, if they are  named *columns*.  Aggregating functions are the ones that reduce the dimension of the returned objects.
Some common aggregating functions are tabulated below:  .. csv-table::  "Function", "Description"  20, 80  ;  `~pd.core.groupby.DataFrameGroupBy.mean`;Compute mean of groups  `~pd.core.groupby.DataFrameGroupBy.sum`;Compute sum of group values  `~pd.core.groupby.DataFrameGroupBy.size`;Compute group sizes  `~pd.core.groupby.DataFrameGroupBy.count`;Compute count of group  `~pd.core.groupby.DataFrameGroupBy.std`;Standard deviation of groups  `~pd.core.groupby.DataFrameGroupBy.var`;Compute variance of groups  `~pd.core.groupby.DataFrameGroupBy.sem`;Standard error of the mean of groups  `~pd.core.groupby.DataFrameGroupBy.describe`;Generates descriptive statistics  `~pd.core.groupby.DataFrameGroupBy.first`;Compute first of group values  `~pd.core.groupby.DataFrameGroupBy.last`;Compute last of group values  `~pd.core.groupby.DataFrameGroupBy.nth`;Take nth value, or a subset if n is a list  `~pd.core.groupby.DataFrameGroupBy.min`;Compute min of group values  `~pd.core.groupby.DataFrameGroupBy.max`;Compute max of group values  The aggregating functions above will exclude NA values. Any function which
reduces a `Series` to a scalar value is an aggregation function and will work,
a trivial example is ``df.groupby('A').agg(lambda ser: 1)``. Note that
`~pd.core.groupby.DataFrameGroupBy.nth` can act as a reducer *or* a
filter, see `here <groupby.nth>`.  .. _groupby.aggregate.multifunc:  Applying multiple functions at once
~~~  With grouped ``Series`` you can also pass a list or dict of functions to do

aggregation with, outputting a DataFrame:  .. ipython:: python  grouped = df.groupby("A")  grouped["C"].agg([np.sum, np.mean, np.std])  On a grouped ``DataFrame``, you can pass a list of functions to apply to each
column, which produces an aggregated result with a hierarchical index:  .. ipython:: python  grouped[["C", "D"]].agg([np.sum, np.mean, np.std])  The resulting aggregations are named for the functions themselves. If you
need to rename, then you can add in a chained operation for a ``Series`` like this:  .. ipython:: python  (  grouped["C"]  .agg([np.sum, np.mean, np.std])  .rename(columns={"sum": "foo", "mean": "bar", "std": "baz"})  )  For a grouped ``DataFrame``, you can rename in a similar manner:  .. ipython:: python  (  grouped[["C", "D"]].agg([np.sum, np.mean, np.std]).rename(  columns={"sum": "foo", "mean": "bar", "std": "baz"}  )  )  .. note::  In general, the output column names should be unique. You can't apply  the same function (or two functions with the same name) to the same  column.  .. ipython:: python  grouped["C"].agg(["sum", "sum"])  pandas *does* allow you to provide multiple lambdas. In this case, pandas  will mangle the name of the (nameless) lambda functions, appending ``_<i>``  to each subsequent lambda.  .. ipython:: python  grouped["C"].agg([lambda x: x.max() - x.min(), lambda x: x.median() - x.mean()])  .. _groupby.aggregate.named:  Named aggregation
~~~  To support column-specific aggregation *with control over the output column names*, pandas
accepts the special syntax in `DataFrameGroupBy.agg` and `SeriesGroupBy.agg`, known as "named aggregation", where  - The keywords are the *output* column names
- The values are tuples whose first element is the column to select  and the second element is the aggregation to apply to that column. pandas  provides the ``pandas.NamedAgg`` namedtuple with the fields ``['column', 'aggfunc']``  to make it clearer what the arguments are. As usual, the aggregation can  be a callable or a string alias.  .. ipython:: python  animals = pd.DataFrame(  {  "kind": ["cat", "dog", "cat", "dog"],  "height": [9.1, 6.0, 9.5, 34.0],  "weight": [7.9, 7.5, 9.9, 198.0],  }  )  animals  animals.groupby("kind").agg(  min_height=pd.NamedAgg(column="height", aggfunc="min"),  max_height=pd.NamedAgg(column="height", aggfunc="max"),  average_weight=pd.NamedAgg(column="weight", aggfunc=np.mean),  )  ``pandas.NamedAgg`` is just a ``namedtuple``. Plain tuples are allowed as well.  .. ipython:: python  animals.groupby("kind").agg(  min_height=("height", "min"),  max_height=("height", "max"),  average_weight=("weight", np.mean),  )  If your desired output column names are not valid Python keywords, construct a dictionary
and unpack the keyword arguments  .. ipython:: python  animals.groupby("kind").agg(  **{  "total weight": pd.NamedAgg(column="weight", aggfunc=sum)  }  )  Additional keyword arguments are not passed through to the aggregation functions. Only pairs
of ``(column, aggfunc)`` should be passed as ``**kwargs``. If your aggregation functions
requires additional arguments, partially apply them with `functools.partial`.  .. note::  For Python 3.5 and earlier, the order of ``**kwargs`` in a functions was not  preserved. This means that the output column ordering would not be  consistent. To ensure consistent ordering, the keys (and so output columns)  will always be sorted for Python 3.5.  Named aggregation is also valid for Series groupby aggregations. In this case there's
no column selection, so the values are just the functions.  .. ipython:: python  animals.groupby("kind").height.agg(  min_height="min",  max_height="max",  )  Applying different functions to DataFrame columns
~~~  By passing a dict to ``aggregate`` you can apply a different aggregation to the

columns of a DataFrame:  .. ipython:: python  grouped.agg({"C": np.sum, "D": lambda x: np.std(x, ddof=1)})  The function names can also be strings. In order for a string to be valid it
must be either implemented on GroupBy or available via `dispatching
<groupby.dispatch>`:  .. ipython:: python  grouped.agg({"C": "sum", "D": "std"})  .. _groupby.aggregate.cython:  Cython-optimized aggregation functions
~~~  Some common aggregations, currently only ``sum``, ``mean``, ``std``, and ``sem``, have
optimized Cython implementations:  .. ipython:: python  df.groupby("A")[["C", "D"]].sum()  df.groupby(["A", "B"]).mean()  Of course ``sum`` and ``mean`` are implemented on pandas objects, so the above
code would work even without the special versions via dispatching (see below).  .. _groupby.aggregate.udfs:  Aggregations with User-Defined Functions
~~~  Users can also provide their own functions for custom aggregations. When aggregating
with a User-Defined Function (UDF), the UDF should not mutate the provided ``Series``, see
`gotchas.udf-mutation` for more information.  .. ipython:: python  animals.groupby("kind")[["height"]].agg(lambda x: set(x))  The resulting dtype will reflect that of the aggregating function. If the results from different groups have
different dtypes, then a common dtype will be determined in the same way as ``DataFrame`` construction.  .. ipython:: python  animals.groupby("kind")[["height"]].agg(lambda x: x.astype(int).sum())  .. _groupby.transform:

The ``transform`` method returns an object that is indexed the same
as the one being grouped. The transform function must:  * Return a result that is either the same size as the group chunk or  broadcastable to the size of the group chunk (e.g., a scalar,  ``grouped.transform(lambda x: x.iloc[-1])``).
* Operate column-by-column on the group chunk.  The transform is applied to  the first group chunk using chunk.apply.
* Not perform in-place operations on the group chunk. Group chunks should  be treated as immutable, and changes to a group chunk may produce unexpected  results.
* (Optionally) operates on the entire group chunk. If this is supported, a  fast path is used starting from the *second* chunk.  .. versionchanged:: 2.0.0  When using ``.transform`` on a grouped DataFrame and the transformation function  returns a DataFrame, pandas now aligns the result's index  with the input's index. You can call ``.to_numpy()`` on the  result of the transformation function to avoid alignment.  Similar to `groupby.aggregate.udfs`, the resulting dtype will reflect that of the
transformation function. If the results from different groups have different dtypes, then
a common dtype will be determined in the same way as ``DataFrame`` construction.  Suppose we wished to standardize the data within each group:  .. ipython:: python  index = pd.date_range("10/1/1999", periods=1100)  ts = pd.Series(np.random.normal(0.5, 2, 1100), index)  ts = ts.rolling(window=100, min_periods=100).mean().dropna()  ts.head()  ts.tail()  transformed = ts.groupby(lambda x: x.year).transform(  lambda x: (x - x.mean()) / x.std()  )  We would expect the result to now have mean 0 and standard deviation 1 within
each group, which we can easily check:  .. ipython:: python  # Original Data  grouped = ts.groupby(lambda x: x.year)  grouped.mean()  grouped.std()  # Transformed Data  grouped_trans = transformed.groupby(lambda x: x.year)  grouped_trans.mean()  grouped_trans.std()  We can also visually compare the original and transformed data sets.  .. ipython:: python  compare = pd.DataFrame({"Original": ts, "Transformed": transformed})  @savefig groupby_transform_plot.png  compare.plot()  Transformation functions that have lower dimension outputs are broadcast to
match the shape of the input array.  .. ipython:: python  ts.groupby(lambda x: x.year).transform(lambda x: x.max() - x.min())  Alternatively, the built-in methods could be used to produce the same outputs.  .. ipython:: python  max_ts = ts.groupby(lambda x: x.year).transform("max")  min_ts = ts.groupby(lambda x: x.year).transform("min")  max_ts - min_ts  Another common data transform is to replace missing data with the group mean.  .. ipython:: python  cols = ["A", "B", "C"]  values = np.random.randn(1000, 3)  values[np.random.randint(0, 1000, 100), 0] = np.nan  values[np.random.randint(0, 1000, 50), 1] = np.nan  values[np.random.randint(0, 1000, 200), 2] = np.nan  data_df = pd.DataFrame(values, columns=cols)  .. ipython:: python  data_df  countries = np.array(["US", "UK", "GR", "JP"])  key = countries[np.random.randint(0, 4, 1000)]  grouped = data_df.groupby(key)  # Non-NA count in each group  grouped.count()  transformed = grouped.transform(lambda x: x.fillna(x.mean()))  We can verify that the group means have not changed in the transformed data

and that the transformed data contains no NAs.  .. ipython:: python  grouped_trans = transformed.groupby(key)  grouped.mean()  # original group means  grouped_trans.mean()  # transformation did not change group means  grouped.count()  # original has some missing data points  grouped_trans.count()  # counts after transformation  grouped_trans.size()  # Verify non-NA count equals group size  .. note::  Some functions will automatically transform the input when applied to a  GroupBy object, but returning an object of the same shape as the original.  Passing ``as_index=False`` will not affect these transformation methods.  For example: ``fillna, ffill, bfill, shift.``.  .. ipython:: python  grouped.ffill()  .. _groupby.transform.window_resample:  Window and resample operations
~~~  It is possible to use ``resample()``, ``expanding()`` and
``rolling()`` as methods on groupbys.  The example below will apply the ``rolling()`` method on the samples of
the column B based on the groups of column A.  .. ipython:: python  df_re = pd.DataFrame({"A": [1] * 10 + [5] * 10, "B": np.arange(20)})  df_re  df_re.groupby("A").rolling(4).B.mean()  The ``expanding()`` method will accumulate a given operation
(``sum()`` in the example) for all the members of each particular
group.  .. ipython:: python  df_re.groupby("A").expanding().sum()  Suppose you want to use the ``resample()`` method to get a daily
frequency in each group of your dataframe and wish to complete the
missing values with the ``ffill()`` method.  .. ipython:: python  df_re = pd.DataFrame(  {  "date": pd.date_range(start="2016-01-01", periods=4, freq="W"),  "group": [1, 1, 2, 2],  "val": [5, 6, 7, 8],  }  ).set_index("date")  df_re  df_re.groupby("group").resample("1D").ffill()  .. _groupby.filter:

Filtration

The ``filter`` method returns a subset of the original object. Suppose we
want to take only elements that belong to groups with a group sum greater
than 2.  .. ipython:: python  sf = pd.Series([1, 1, 2, 3, 3, 3])  sf.groupby(sf).filter(lambda x: x.sum() > 2)  The argument of ``filter`` must be a function that, applied to the group as a
whole, returns ``True`` or ``False``.  Another useful operation is filtering out elements that belong to groups
with only a couple members.  .. ipython:: python  dff = pd.DataFrame({"A": np.arange(8), "B": list("aabbbcc")})  dff.groupby("B").filter(lambda x: len(x) > 2)  Alternatively, instead of dropping the offending groups, we can return a
like-indexed objects where the groups that do not pass the filter are filled
with NaNs.  .. ipython:: python  dff.groupby("B").filter(lambda x: len(x) > 2, dropna=False)  For DataFrames with multiple columns, filters should explicitly specify a column as the filter criterion.  .. ipython:: python  dff["C"] = np.arange(8)  dff.groupby("B").filter(lambda x: len(x["C"]) > 2)  .. note::  Some functions when applied to a groupby object will act as a **filter** on the input, returning  a reduced shape of the original (and potentially eliminating groups), but with the index unchanged.  Passing ``as_index=False`` will not affect these transformation methods.  For example: ``head, tail``.  .. ipython:: python  dff.groupby("B").head(2)  .. _groupby.dispatch:

Dispatching to instance methods

When doing an aggregation or transformation, you might just want to call an
instance method on each data group. This is pretty easy to do by passing lambda
functions:  .. ipython:: python  grouped = df.groupby("A")[["C", "D"]]  grouped.agg(lambda x: x.std())  But, it's rather verbose and can be untidy if you need to pass additional
arguments. Using a bit of metaprogramming cleverness, GroupBy now has the
ability to "dispatch" method calls to the groups:  .. ipython:: python  grouped.std()  What is actually happening here is that a function wrapper is being
generated. When invoked, it takes any passed arguments and invokes the function
with any arguments on each group (in the above example, the ``std``
function). The results are then combined together much in the style of ``agg``
and ``transform`` (it actually uses ``apply`` to infer the gluing, documented
next). This enables some operations to be carried out rather succinctly:  .. ipython:: python  tsdf = pd.DataFrame(  np.random.randn(1000, 3),  index=pd.date_range("1/1/2000", periods=1000),  columns=["A", "B", "C"],  )  tsdf.iloc[::2] = np.nan  grouped = tsdf.groupby(lambda x: x.year)  grouped.fillna(method="pad")  In this example, we chopped the collection of time series into yearly chunks
then independently called `fillna <missing_data.fillna>` on the
groups.  The ``nlargest`` and ``nsmallest`` methods work on ``Series`` style groupbys:  .. ipython:: python  s = pd.Series([9, 8, 7, 5, 19, 1, 4.2, 3.3])  g = pd.Series(list("abababab"))  gb = s.groupby(g)  gb.nlargest(3)  gb.nsmallest(3)  .. _groupby.apply:

Flexible ``apply``

Some operations on the grouped data might not fit into either the aggregate or
transform categories. Or, you may simply want GroupBy to infer how to combine
the results. For these, use the ``apply`` function, which can be substituted
for both ``aggregate`` and ``transform`` in many standard use cases. However,
``apply`` can handle some exceptional use cases.  .. note::  ``apply`` can act as a reducer, transformer, *or* filter function, depending  on exactly what is passed to it. It can depend on the passed function and  exactly what you are grouping. Thus the grouped column(s) may be included in  the output as well as set the indices.  .. ipython:: python  df  grouped = df.groupby("A")  # could also just call .describe()  grouped["C"].apply(lambda x: x.describe())  The dimension of the returned result can also change:  .. ipython:: python  grouped = df.groupby('A')['C']  def f(group):  return pd.DataFrame({'original': group,  'demeaned': group - group.mean()})  grouped.apply(f)  ``apply`` on a Series can operate on a returned value from the applied function,
that is itself a series, and possibly upcast the result to a DataFrame:  .. ipython:: python  def f(x):  return pd.Series([x, x ** 2], index=["x", "x^2"])  s = pd.Series(np.random.rand(5))  s  s.apply(f)  Control grouped column(s) placement with ``group_keys``
~~~  .. note::  If ``group_keys=True`` is specified when calling `~DataFrame.groupby`,  functions passed to ``apply`` that return like-indexed outputs will have the  group keys added to the result index. Previous versions of pandas would add  the group keys only when the result from the applied function had a different  index than the input. If ``group_keys`` is not specified, the group keys will  not be added for like-indexed outputs. In the future this behavior  will change to always respect ``group_keys``, which defaults to ``True``.  .. versionchanged:: 1.5.0  To control whether the grouped column(s) are included in the indices, you can use
the argument ``group_keys``. Compare  .. ipython:: python  df.groupby("A", group_keys=True).apply(lambda x: x)  with  .. ipython:: python  df.groupby("A", group_keys=False).apply(lambda x: x)  Similar to `groupby.aggregate.udfs`, the resulting dtype will reflect that of the
apply function. If the results from different groups have different dtypes, then
a common dtype will be determined in the same way as ``DataFrame`` construction.

Numba Accelerated Routines

.. versionadded:: 1.1  If Numba_ is installed as an optional dependency, the ``transform`` and
``aggregate`` methods support ``engine='numba'`` and ``engine_kwargs`` arguments.
See `enhancing performance with Numba <enhancingperf.numba>` for general usage of the arguments
and performance considerations.  The function signature must start with ``values, index`` **exactly** as the data belonging to each group
will be passed into ``values``, and the group index will be passed into ``index``.  .. warning::  When using ``engine='numba'``, there will be no "fall back" behavior internally. The group  data and group index will be passed as NumPy arrays to the JITed user defined function, and no  alternative execution attempts will be tried.

Automatic exclusion of "nuisance" columns
~~~  Again consider the example DataFrame we've been looking at:  .. ipython:: python  df  Suppose we wish to compute the standard deviation grouped by the ``A``
column. There is a slight problem, namely that we don't care about the data in
column ``B``. We refer to this as a "nuisance" column. You can avoid nuisance
columns by specifying ``numeric_only=True``:  .. ipython:: python  df.groupby("A").std(numeric_only=True)  Note that ``df.groupby('A').colname.std().`` is more efficient than
``df.groupby('A').std().colname``, so if the result of an aggregation function
is only interesting over one column (here ``colname``), it may be filtered
*before* applying the aggregation function.  .. note::  Any object column, also if it contains numerical values such as ``Decimal``  objects, is considered as a "nuisance" columns. They are excluded from  aggregate functions automatically in groupby.  If you do wish to include decimal or object columns in an aggregation with  other non-nuisance data types, you must do so explicitly.  .. warning::  The automatic dropping of nuisance columns has been deprecated and will be removed  in a future version of pandas. If columns are included that cannot be operated  on, pandas will instead raise an error. In order to avoid this, either select  the columns you wish to operate on or specify ``numeric_only=True``.  .. ipython:: python  from decimal import Decimal  df_dec = pd.DataFrame(  {  "id": [1, 2, 1, 2],  "int_column": [1, 2, 3, 4],  "dec_column": [  Decimal("0.50"),  Decimal("0.15"),  Decimal("0.25"),  Decimal("0.40"),  ],  }  )  # Decimal columns can be sum'd explicitly by themselves...  df_dec.groupby(["id"])[["dec_column"]].sum()  # ...but cannot be combined with standard data types or they will be excluded  df_dec.groupby(["id"])[["int_column", "dec_column"]].sum()  # Use .agg function to aggregate over standard and "nuisance" data types  # at the same time  df_dec.groupby(["id"]).agg({"int_column": "sum", "dec_column": "sum"})  .. _groupby.observed:  Handling of (un)observed Categorical values
~~~  When using a ``Categorical`` grouper (as a single grouper, or as part of multiple groupers), the ``observed`` keyword
controls whether to return a cartesian product of all possible groupers values (``observed=False``) or only those
that are observed groupers (``observed=True``).  Show all values:  .. ipython:: python  pd.Series([1, 1, 1]).groupby(  pd.Categorical(["a", "a", "a"], categories=["a", "b"]), observed=False  ).count()  Show only the observed values:  .. ipython:: python  pd.Series([1, 1, 1]).groupby(  pd.Categorical(["a", "a", "a"], categories=["a", "b"]), observed=True  ).count()  The returned dtype of the grouped will *always* include *all* of the categories that were grouped.  .. ipython:: python  s = (  pd.Series([1, 1, 1])  .groupby(pd.Categorical(["a", "a", "a"], categories=["a", "b"]), observed=False)  .count()  )  s.index.dtype  .. _groupby.missing:  NA and NaT group handling
~~~  If there are any NaN or NaT values in the grouping key, these will be
automatically excluded. In other words, there will never be an "NA group" or
"NaT group". This was not the case in older versions of pandas, but users were
generally discarding the NA group anyway (and supporting it was an
implementation headache).  Grouping with ordered factors
~~~  Categorical variables represented as instance of pandas's ``Categorical`` class
can be used as group keys. If so, the order of the levels will be preserved:  .. ipython:: python  data = pd.Series(np.random.randn(100))  factor = pd.qcut(data, [0, 0.25, 0.5, 0.75, 1.0])  data.groupby(factor).mean()  .. _groupby.specify:  Grouping with a grouper specification
~~~  You may need to specify a bit more data to properly group. You can

use the ``pd.Grouper`` to provide this local control.  .. ipython:: python  import datetime  df = pd.DataFrame(  {  "Branch": "A A A A A A A B".split(),  "Buyer": "Carl Mark Carl Carl Joe Joe Joe Carl".split(),  "Quantity": [1, 3, 5, 1, 8, 1, 9, 3],  "Date": [  datetime.datetime(2013, 1, 1, 13, 0),  datetime.datetime(2013, 1, 1, 13, 5),  datetime.datetime(2013, 10, 1, 20, 0),  datetime.datetime(2013, 10, 2, 10, 0),  datetime.datetime(2013, 10, 1, 20, 0),  datetime.datetime(2013, 10, 2, 10, 0),  datetime.datetime(2013, 12, 2, 12, 0),  datetime.datetime(2013, 12, 2, 14, 0),  ],  }  )  df  Groupby a specific column with the desired frequency. This is like resampling.  .. ipython:: python  df.groupby([pd.Grouper(freq="1M", key="Date"), "Buyer"])[["Quantity"]].sum()  You have an ambiguous specification in that you have a named index and a column
that could be potential groupers.  .. ipython:: python  df = df.set_index("Date")  df["Date"] = df.index + pd.offsets.MonthEnd(2)  df.groupby([pd.Grouper(freq="6M", key="Date"), "Buyer"])[["Quantity"]].sum()  df.groupby([pd.Grouper(freq="6M", level="Date"), "Buyer"])[["Quantity"]].sum()  Taking the first rows of each group
~~~  Just like for a DataFrame or Series you can call head and tail on a groupby:  .. ipython:: python  df = pd.DataFrame([[1, 2], [1, 4], [5, 6]], columns=["A", "B"])  df  g = df.groupby("A")  g.head(1)  g.tail(1)  This shows the first or last n rows from each group.  .. _groupby.nth:  Taking the nth row of each group
~~~  To select the nth item from each group, use `.DataFrameGroupBy.nth` or
`.SeriesGroupBy.nth`. Arguments supplied can be any integer, lists of integers,
slices, or lists of slices; see below for examples. When the nth element of a group
does not exist an error is *not* raised; instead no corresponding rows are returned.  In general this operation acts as a filtration. In certain cases it will also return
one row per group, making it also a reduction. However because in general it can
return zero or multiple rows per group, pandas treats it as a filtration in all cases.  .. ipython:: python  df = pd.DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=["A", "B"])  g = df.groupby("A")  g.nth(0)  g.nth(-1)  g.nth(1)  If the nth element of a group does not exist, then no corresponding row is included
in the result. In particular, if the specified ``n`` is larger than any group, the
result will be an empty DataFrame.  .. ipython:: python  g.nth(5)  If you want to select the nth not-null item, use the ``dropna`` kwarg. For a DataFrame this should be either ``'any'`` or ``'all'`` just like you would pass to dropna:  .. ipython:: python  # nth(0) is the same as g.first()  g.nth(0, dropna="any")  g.first()  # nth(-1) is the same as g.last()  g.nth(-1, dropna="any")  g.last()  g.B.nth(0, dropna="all")  You can also select multiple rows from each group by specifying multiple nth values as a list of ints.  .. ipython:: python  business_dates = pd.date_range(start="4/1/2014", end="6/30/2014", freq="B")  df = pd.DataFrame(1, index=business_dates, columns=["a", "b"])  # get the first, 4th, and last date index for each month  df.groupby([df.index.year, df.index.month]).nth([0, 3, -1])  You may also use a slices or lists of slices.  .. ipython:: python  df.groupby([df.index.year, df.index.month]).nth[1:]  df.groupby([df.index.year, df.index.month]).nth[1:, :-1]  Enumerate group items
~~~  To see the order in which each row appears within its group, use the
``cumcount`` method:  .. ipython:: python  dfg = pd.DataFrame(list("aaabba"), columns=["A"])  dfg  dfg.groupby("A").cumcount()  dfg.groupby("A").cumcount(ascending=False)  .. _groupby.ngroup:  Enumerate groups
~~~  To see the ordering of the groups (as opposed to the order of rows
within a group given by ``cumcount``) you can use
`~pandas.core.groupby.DataFrameGroupBy.ngroup`.  Note that the numbers given to the groups match the order in which the

groups would be seen when iterating over the groupby object, not the
order they are first observed.  .. ipython:: python  dfg = pd.DataFrame(list("aaabba"), columns=["A"])  dfg  dfg.groupby("A").ngroup()  dfg.groupby("A").ngroup(ascending=False)  Plotting
~~~  Groupby also works with some plotting methods.  For example, suppose we
suspect that some features in a DataFrame may differ by group, in this case,
the values in column 1 where the group is "B" are 3 higher on average.  .. ipython:: python  np.random.seed(1234)  df = pd.DataFrame(np.random.randn(50, 2))  df["g"] = np.random.choice(["A", "B"], size=50)  df.loc[df["g"] == "B", 1] += 3  We can easily visualize this with a boxplot:  .. ipython:: python  @savefig groupby_boxplot.png  df.groupby("g").boxplot()  The result of calling ``boxplot`` is a dictionary whose keys are the values
of our grouping column ``g`` ("A" and "B"). The values of the resulting dictionary
can be controlled by the ``return_type`` keyword of ``boxplot``.
See the `visualization documentation<visualization.box>` for more.  .. warning::  For historical reasons, ``df.groupby("g").boxplot()`` is not equivalent  to ``df.boxplot(by="g")``. See `here<visualization.box.return>` for  an explanation.  .. _groupby.pipe:  Piping function calls
~~~  Similar to the functionality provided by ``DataFrame`` and ``Series``, functions
that take ``GroupBy`` objects can be chained together using a ``pipe`` method to
allow for a cleaner, more readable syntax. To read about ``.pipe`` in general terms,
see `here <basics.pipe>`.  Combining ``.groupby`` and ``.pipe`` is often useful when you need to reuse
GroupBy objects.  As an example, imagine having a DataFrame with columns for stores, products,
revenue and quantity sold. We'd like to do a groupwise calculation of *prices*
(i.e. revenue/quantity) per store and per product. We could do this in a
multi-step operation, but expressing it in terms of piping can make the
code more readable. First we set the data:  .. ipython:: python  n = 1000  df = pd.DataFrame(  {  "Store": np.random.choice(["Store_1", "Store_2"], n),  "Product": np.random.choice(["Product_1", "Product_2"], n),  "Revenue": (np.random.random(n) * 50 + 10).round(2),  "Quantity": np.random.randint(1, 10, size=n),  }  )  df.head(2)  Now, to find prices per store/product, we can simply do:  .. ipython:: python  (  df.groupby(["Store", "Product"])  .pipe(lambda grp: grp.Revenue.sum() / grp.Quantity.sum())  .unstack()  .round(2)  )  Piping can also be expressive when you want to deliver a grouped object to some
arbitrary function, for example:  .. ipython:: python  def mean(groupby):  return groupby.mean()  df.groupby(["Store", "Product"]).pipe(mean)  where ``mean`` takes a GroupBy object and finds the mean of the Revenue and Quantity
columns respectively for each Store-Product combination. The ``mean`` function can
be any function that takes in a GroupBy object; the ``.pipe`` will pass the GroupBy
object as a parameter into the function you specify.

Examples

Regrouping by factor
~~~  Regroup columns of a DataFrame according to their sum, and sum the aggregated ones.  .. ipython:: python  df = pd.DataFrame({"a": [1, 0, 0], "b": [0, 1, 0], "c": [1, 0, 0], "d": [2, 3, 4]})  df  df.groupby(df.sum(), axis=1).sum()  .. _groupby.multicolumn_factorization:  Multi-column factorization
~~~  By using `~pandas.core.groupby.DataFrameGroupBy.ngroup`, we can extract
information about the groups in a way similar to `factorize` (as described
further in the `reshaping API <reshaping.factorize>`) but which applies
naturally to multiple columns of mixed type and different
sources. This can be useful as an intermediate categorical-like step
in processing, when the relationships between the group rows are more
important than their content, or as input to an algorithm which only
accepts the integer encoding. (For more information about support in
pandas for full categorical data, see the `Categorical
introduction <categorical>` and the
`API documentation <api.arrays.categorical>`.)  .. ipython:: python  dfg = pd.DataFrame({"A": [1, 1, 2, 3, 2], "B": list("aaaba")})  dfg  dfg.groupby(["A", "B"]).ngroup()  dfg.groupby(["A", [0, 0, 0, 1, 1]]).ngroup()  Groupby by indexer to 'resample' data
~~~  Resampling produces new hypothetical samples (resamples) from already existing observed data or from a model that generates data. These new samples are similar to the pre-existing samples.  In order to resample to work on indices that are non-datetimelike, the following procedure can be utilized.  In the following examples, **df.index // 5** returns a binary array which is used to determine what gets selected for the groupby operation.  .. note:: The below example shows how we can downsample by consolidation of samples into fewer samples. Here by using **df.index // 5**, we are aggregating the samples in bins. By applying **std()** function, we aggregate the information contained in many samples into a small subset of values which is their standard deviation thereby reducing the number of samples.  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 2))  df  df.index // 5  df.groupby(df.index // 5).std()  Returning a Series to propagate names
~~~  Group DataFrame columns, compute a set of metrics and return a named Series.
The Series name is used as the name for the column index. This is especially
useful in conjunction with reshaping operations such as stacking in which the
column index name will be used as the name of the inserted column:  .. ipython:: python  df = pd.DataFrame(  {  "a": [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2],  "b": [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1],  "c": [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0],  "d": [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1],  }  )  def compute_metrics(x):  result = {"b_sum": x["b"].sum(), "c_mean": x["c"].mean()}  return pd.Series(result, name="metrics")  result = df.groupby("a").apply(compute_metrics)  result  result.stack()

User Guide

The User Guide covers all of pandas by topic area. Each of the subsections
introduces a topic (such as "working with missing data"), and discusses how
pandas approaches the problem, with many examples throughout.  Users brand-new to pandas should start with `10min`.  For a high level summary of the pandas fundamentals, see `dsintro` and `basics`.  Further information on any specific method can be obtained in the
`api`.

How to read these guides
In these guides you will see input code inside code blocks such as:  ::  import pandas as pd  pd.DataFrame({'A': [1, 2, 3]})  or:  .. ipython:: python  import pandas as pd  pd.DataFrame({'A': [1, 2, 3]})  The first block is a standard python input, while in the second the `In [1]:`` indicates the input is inside a `notebook_. In Jupyter Notebooks the last line is printed and plots are shown inline.  For example:  .. ipython:: python  a = 1  a
is equivalent to:  ::  a = 1  print(a)

Guides

.. If you update this toctree, also update the manual toctree in the  main index.rst.template  .. toctree::  2  10min  dsintro  basics  io  indexing  advanced  merging  reshaping  text  missing_data  duplicates  categorical  integer_na  boolean  visualization  style  groupby  window  timeseries  timedeltas  options  enhancingperf  scale  sparse  gotchas  cookbook

Different choices for indexing

Object selection has had a number of user-requested additions in order to
support more explicit location based indexing. pandas now supports three types
of multi-axis indexing.  * ``.loc`` is primarily label based, but may also be used with a boolean array. ``.loc`` will raise ``KeyError`` when the items are not found. Allowed inputs are:  * A single label, e.g. ``5`` or ``'a'`` (Note that ``5`` is interpreted as a  *label* of the index. This use is **not** an integer position along the  index.).  * A list or array of labels ``['a', 'b', 'c']``.  * A slice object with labels ``'a':'f'`` (Note that contrary to usual Python  slices, **both** the start and the stop are included, when present in the  index! See `Slicing with labels <indexing.slicing_with_labels>`  and `Endpoints are inclusive <advanced.endpoints_are_inclusive>`.)  * A boolean array (any ``NA`` values will be treated as ``False``).  * A ``callable`` function with one argument (the calling Series or DataFrame) and  that returns valid output for indexing (one of the above).  See more at `Selection by Label <indexing.label>`.  * ``.iloc`` is primarily integer position based (from ``0`` to  ``length-1`` of the axis), but may also be used with a boolean  array.  ``.iloc`` will raise ``IndexError`` if a requested  indexer is out-of-bounds, except *slice* indexers which allow  out-of-bounds indexing.  (this conforms with Python/NumPy *slice*  semantics).  Allowed inputs are:  * An integer e.g. ``5``.  * A list or array of integers ``[4, 3, 0]``.  * A slice object with ints ``1:7``.  * A boolean array (any ``NA`` values will be treated as ``False``).  * A ``callable`` function with one argument (the calling Series or DataFrame) and  that returns valid output for indexing (one of the above).  See more at `Selection by Position <indexing.integer>`,  `Advanced Indexing <advanced>` and `Advanced  Hierarchical <advanced.advanced_hierarchical>`.  * ``.loc``, ``.iloc``, and also ``[]`` indexing can accept a ``callable`` as indexer. See more at `Selection By Callable <indexing.callable>`.  Getting values from an object with multi-axes selection uses the following
notation (using ``.loc`` as an example, but the following applies to ``.iloc`` as
well). Any of the axes accessors may be the null slice ``:``. Axes left out of
the specification are assumed to be ``:``, e.g. ``p.loc['a']`` is equivalent to
``p.loc['a', :]``.  .. csv-table::  "Object Type", "Indexers"  30, 50  ;  Series; ``s.loc[indexer]``  DataFrame; ``df.loc[row_indexer,column_indexer]``  .. _indexing.basics:

Basics

As mentioned when introducing the data structures in the `last section
<basics>`, the primary function of indexing with ``[]`` (a.k.a. ``__getitem__``
for those familiar with implementing class behavior in Python) is selecting out
lower-dimensional slices. The following table shows return type values when
indexing pandas objects with ``[]``:  .. csv-table::  "Object Type", "Selection", "Return Value Type"  30, 30, 60  ;  Series; ``series[label]``; scalar value  DataFrame; ``frame[colname]``; ``Series`` corresponding to colname  Here we construct a simple time series data set to use for illustrating the
indexing functionality:  .. ipython:: python  dates = pd.date_range('1/1/2000', periods=8)  df = pd.DataFrame(np.random.randn(8, 4),  index=dates, columns=['A', 'B', 'C', 'D'])  df  .. note::  None of the indexing functionality is time series specific unless  specifically stated.  Thus, as per above, we have the most basic indexing using ``[]``:  .. ipython:: python  s = df['A']  s[dates[5]]  You can pass a list of columns to ``[]`` to select columns in that order.
If a column is not contained in the DataFrame, an exception will be
raised. Multiple columns can also be set in this manner:  .. ipython:: python  df  df[['B', 'A']] = df[['A', 'B']]  df  You may find this useful for applying a transform (in-place) to a subset of the
columns.  .. warning::  pandas aligns all AXES when setting ``Series`` and ``DataFrame`` from ``.loc``, and ``.iloc``.  This will **not** modify ``df`` because the column alignment is before value assignment.  .. ipython:: python  df[['A', 'B']]  df.loc[:, ['B', 'A']] = df[['A', 'B']]  df[['A', 'B']]  The correct way to swap column values is by using raw values:  .. ipython:: python  df.loc[:, ['B', 'A']] = df[['A', 'B']].to_numpy()  df[['A', 'B']]

Attribute access

.. _indexing.columns.multiple:  .. _indexing.df_cols:  .. _indexing.attribute_access:  You may access an index on a ``Series`` or  column on a ``DataFrame`` directly
as an attribute:  .. ipython:: python  sa = pd.Series([1, 2, 3], index=list('abc'))  dfa = df.copy()  .. ipython:: python  sa.b  dfa.A  .. ipython:: python  sa.a = 5  sa  dfa.A = list(range(len(dfa.index)))  # ok if A already exists  dfa  dfa['A'] = list(range(len(dfa.index)))  # use this form to create a new column  dfa  .. warning::  - You can use this access only if the index element is a valid Python identifier, e.g. ``s.1`` is not allowed.  See `here for an explanation of valid identifiers  <https://docs.python.org/3/reference/lexical_analysis.html#identifiers>`__.  - The attribute will not be available if it conflicts with an existing method name, e.g. ``s.min`` is not allowed, but ``s['min']`` is possible.  - Similarly, the attribute will not be available if it conflicts with any of the following list: ``index``,  ``major_axis``, ``minor_axis``, ``items``.  - In any of these cases, standard indexing will still work, e.g. ``s['1']``, ``s['min']``, and ``s['index']`` will  access the corresponding element or column.  If you are using the IPython environment, you may also use tab-completion to
see these accessible attributes.  You can also assign a ``dict`` to a row of a ``DataFrame``:  .. ipython:: python  x = pd.DataFrame({'x': [1, 2, 3], 'y': [3, 4, 5]})  x.iloc[1] = {'x': 9, 'y': 99}  x  You can use attribute access to modify an existing element of a Series or column of a DataFrame, but be careful;
if you try to use attribute access to create a new column, it creates a new attribute rather than a
new column and will this raise a ``UserWarning``:  .. code-block:: ipython  In [1]: df = pd.DataFrame({'one': [1., 2., 3.]})  In [2]: df.two = [4, 5, 6]  UserWarning: Pandas doesn't allow Series to be assigned into nonexistent columns - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute_access  In [3]: df  Out[3]:  one  0  1.0  1  2.0  2  3.0

Slicing ranges

The most robust and consistent way of slicing ranges along arbitrary axes is
described in the `Selection by Position <indexing.integer>` section
detailing the ``.iloc`` method. For now, we explain the semantics of slicing using the ``[]`` operator.  With Series, the syntax works exactly as with an ndarray, returning a slice of
the values and the corresponding labels:  .. ipython:: python  s[:5]  s[::2]  s[::-1]  Note that setting works as well:  .. ipython:: python  s2 = s.copy()  s2[:5] = 0  s2  With DataFrame, slicing inside of ``[]`` **slices the rows**. This is provided
largely as a convenience since it is such a common operation.  .. ipython:: python  df[:3]  df[::-1]  .. _indexing.label:

.. warning::  Whether a copy or a reference is returned for a setting operation, may depend on the context.  This is sometimes called ``chained assignment`` and should be avoided.  See `Returning a View versus Copy <indexing.view_versus_copy>`.  .. warning::  ``.loc`` is strict when you present slicers that are not compatible (or convertible) with the index type. For example  using integers in a ``DatetimeIndex``. These will raise a ``TypeError``.  .. ipython:: python  dfl = pd.DataFrame(np.random.randn(5, 4),  columns=list('ABCD'),  index=pd.date_range('20130101', periods=5))  dfl  .. code-block:: ipython  In [4]: dfl.loc[2:3]  TypeError: cannot do slice indexing on <class 'pandas.tseries.index.DatetimeIndex'> with these indexers [2] of <type 'int'>  String likes in slicing *can* be convertible to the type of the index and lead to natural slicing.  .. ipython:: python  dfl.loc['20130102':'20130104']  .. warning::  .. versionchanged:: 1.0.0  pandas will raise a ``KeyError`` if indexing with a list with missing labels. See `list-like Using loc with  missing keys in a list is Deprecated <indexing.deprecate_loc_reindex_listlike>`.  pandas provides a suite of methods in order to have **purely label based indexing**. This is a strict inclusion based protocol.
Every label asked for must be in the index, or a ``KeyError`` will be raised.
When slicing, both the start bound **AND** the stop bound are *included*, if present in the index.
Integers are valid labels, but they refer to the label **and not the position**.  The ``.loc`` attribute is the primary access method. The following are valid inputs:  * A single label, e.g. ``5`` or ``'a'`` (Note that ``5`` is interpreted as a *label* of the index. This use is **not** an integer position along the index.).
* A list or array of labels ``['a', 'b', 'c']``.
* A slice object with labels ``'a':'f'`` (Note that contrary to usual Python  slices, **both** the start and the stop are included, when present in the  index! See `Slicing with labels <indexing.slicing_with_labels>`.
* A boolean array.
* A ``callable``, see `Selection By Callable <indexing.callable>`.  .. ipython:: python  s1 = pd.Series(np.random.randn(6), index=list('abcdef'))  s1  s1.loc['c':]  s1.loc['b']  Note that setting works as well:  .. ipython:: python  s1.loc['c':] = 0  s1  With a DataFrame:  .. ipython:: python  df1 = pd.DataFrame(np.random.randn(6, 4),  index=list('abcdef'),  columns=list('ABCD'))  df1  df1.loc[['a', 'b', 'd'], :]  Accessing via label slices:  .. ipython:: python  df1.loc['d':, 'A':'C']  For getting a cross section using a label (equivalent to ``df.xs('a')``):  .. ipython:: python  df1.loc['a']  For getting values with a boolean array:  .. ipython:: python  df1.loc['a'] > 0  df1.loc[:, df1.loc['a'] > 0]  NA values in a boolean array propagate as ``False``:  .. versionchanged:: 1.0.2  .. ipython:: python  mask = pd.array([True, False, True, False, pd.NA, False], dtype="boolean")  mask  df1[mask]  For getting a value explicitly:  .. ipython:: python  # this is also equivalent to ``df1.at['a','A']``  df1.loc['a', 'A']  .. _indexing.slicing_with_labels:  Slicing with labels
~~~  When using ``.loc`` with slices, if both the start and the stop labels are
present in the index, then elements *located* between the two (including them)
are returned:  .. ipython:: python  s = pd.Series(list('abcde'), index=[0, 3, 2, 5, 4])  s.loc[3:5]  If at least one of the two is absent, but the index is sorted, and can be
compared against start and stop labels, then slicing will still work as
expected, by selecting labels which *rank* between the two:  .. ipython:: python  s.sort_index()  s.sort_index().loc[1:6]  However, if at least one of the two is absent *and* the index is not sorted, an
error will be raised (since doing otherwise would be computationally expensive,
as well as potentially ambiguous for mixed type indexes). For instance, in the

above example, ``s.loc[1:6]`` would raise ``KeyError``.  For the rationale behind this behavior, see
`Endpoints are inclusive <advanced.endpoints_are_inclusive>`.  .. ipython:: python  s = pd.Series(list('abcdef'), index=[0, 3, 2, 5, 4, 2])  s.loc[3:5]  Also, if the index has duplicate labels *and* either the start or the stop label is duplicated,
an error will be raised. For instance, in the above example, ``s.loc[2:5]`` would raise a ``KeyError``.  For more information about duplicate labels, see
`Duplicate Labels <duplicates>`.  .. _indexing.integer:

Selection by position

.. warning::  Whether a copy or a reference is returned for a setting operation, may depend on the context.  This is sometimes called ``chained assignment`` and should be avoided.  See `Returning a View versus Copy <indexing.view_versus_copy>`.  pandas provides a suite of methods in order to get **purely integer based indexing**. The semantics follow closely Python and NumPy slicing. These are ``0-based`` indexing. When slicing, the start bound is *included*, while the upper bound is *excluded*. Trying to use a non-integer, even a **valid** label will raise an ``IndexError``.  The ``.iloc`` attribute is the primary access method. The following are valid inputs:  * An integer e.g. ``5``.
* A list or array of integers ``[4, 3, 0]``.
* A slice object with ints ``1:7``.
* A ``callable``, see `Selection By Callable <indexing.callable>`.  .. ipython:: python  s1 = pd.Series(np.random.randn(5), index=list(range(0, 10, 2)))  s1  s1.iloc[:3]  s1.iloc[3]  Note that setting works as well:  .. ipython:: python  s1.iloc[:3] = 0  s1  With a DataFrame:  .. ipython:: python  df1 = pd.DataFrame(np.random.randn(6, 4),  index=list(range(0, 12, 2)),  columns=list(range(0, 8, 2)))  df1  Select via integer slicing:  .. ipython:: python  df1.iloc[:3]  df1.iloc[1:5, 2:4]  Select via integer list:  .. ipython:: python  df1.iloc[[1, 3, 5], [1, 3]]  .. ipython:: python  df1.iloc[1:3, :]  .. ipython:: python  df1.iloc[:, 1:3]  .. ipython:: python  # this is also equivalent to ``df1.iat[1,1]``  df1.iloc[1, 1]  For getting a cross section using an integer position (equiv to ``df.xs(1)``):  .. ipython:: python  df1.iloc[1]  Out of range slice indexes are handled gracefully just as in Python/NumPy.  .. ipython:: python  # these are allowed in Python/NumPy.  x = list('abcdef')  x  x[4:10]  x[8:10]  s = pd.Series(x)  s  s.iloc[4:10]  s.iloc[8:10]  Note that using slices that go out of bounds can result in
an empty axis (e.g. an empty DataFrame being returned).  .. ipython:: python  dfl = pd.DataFrame(np.random.randn(5, 2), columns=list('AB'))  dfl  dfl.iloc[:, 2:3]  dfl.iloc[:, 1:3]  dfl.iloc[4:6]  A single indexer that is out of bounds will raise an ``IndexError``.
A list of indexers where any element is out of bounds will raise an
``IndexError``.  .. code-block:: python  >>> dfl.iloc[[4, 5, 6]]  IndexError: positional indexers are out-of-bounds  >>> dfl.iloc[:, 4]  IndexError: single positional indexer is out-of-bounds  .. _indexing.callable:

Selection by callable

``.loc``, ``.iloc``, and also ``[]`` indexing can accept a ``callable`` as indexer.
The ``callable`` must be a function with one argument (the calling Series or DataFrame) that returns valid output for indexing.  .. ipython:: python  df1 = pd.DataFrame(np.random.randn(6, 4),  index=list('abcdef'),  columns=list('ABCD'))  df1  df1.loc[lambda df: df['A'] > 0, :]  df1.loc[:, lambda df: ['A', 'B']]  df1.iloc[:, lambda df: [0, 1]]  df1[lambda df: df.columns[0]]  You can use callable indexing in ``Series``.  .. ipython:: python  df1['A'].loc[lambda s: s > 0]  Using these methods / indexers, you can chain data selection operations
without using a temporary variable.  .. ipython:: python  bb = pd.read_csv('data/baseball.csv', index_col='id')  (bb.groupby(['year', 'team']).sum(numeric_only=True)  .loc[lambda df: df['r'] > 100])  .. _combining_positional_and_label_based_indexing:

Combining positional and label-based indexing

If you wish to get the 0th and the 2nd elements from the index in the 'A' column, you can do:  .. ipython:: python  dfd = pd.DataFrame({'A': [1, 2, 3],  'B': [4, 5, 6]},  index=list('abc'))  dfd  dfd.loc[dfd.index[[0, 2]], 'A']  This can also be expressed using ``.iloc``, by explicitly getting locations on the indexers, and using
*positional* indexing to select things.  .. ipython:: python  dfd.iloc[[0, 2], dfd.columns.get_loc('A')]  For getting *multiple* indexers, using ``.get_indexer``:  .. ipython:: python  dfd.iloc[[0, 2], dfd.columns.get_indexer(['A', 'B'])]  .. _deprecate_loc_reindex_listlike:
.. _indexing.deprecate_loc_reindex_listlike:

Indexing with list with missing labels is deprecated

.. warning::  .. versionchanged:: 1.0.0  Using ``.loc`` or ``[]`` with a list with one or more missing labels will no longer reindex, in favor of ``.reindex``.  In prior versions, using ``.loc[list-of-labels]`` would work as long as *at least 1* of the keys was found (otherwise it
would raise a ``KeyError``). This behavior was changed and will now raise a ``KeyError`` if at least one label is missing.
The recommended alternative is to use ``.reindex()``.  For example.  .. ipython:: python  s = pd.Series([1, 2, 3])  s  Selection with all keys found is unchanged.  .. ipython:: python  s.loc[[1, 2]]  Previous behavior  .. code-block:: ipython  In [4]: s.loc[[1, 2, 3]]  Out[4]:  1  2.0  2  3.0  3  NaN  dtype: float64  Current behavior  .. code-block:: ipython  In [4]: s.loc[[1, 2, 3]]  Passing list-likes to .loc with any non-matching elements will raise  KeyError in the future, you can use .reindex() as an alternative.  See the documentation here:  https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike  Out[4]:  1  2.0  2  3.0  3  NaN  dtype: float64  Reindexing
~~~  The idiomatic way to achieve selecting potentially not-found elements is via ``.reindex()``. See also the section on `reindexing <basics.reindexing>`.  .. ipython:: python  s.reindex([1, 2, 3])  Alternatively, if you want to select only *valid* keys, the following is idiomatic and efficient; it is guaranteed to preserve the dtype of the selection.  .. ipython:: python  labels = [1, 2, 3]  s.loc[s.index.intersection(labels)]  Having a duplicated index will raise for a ``.reindex()``:  .. ipython:: python  s = pd.Series(np.arange(4), index=['a', 'a', 'b', 'c'])  labels = ['c', 'd']  .. code-block:: ipython  In [17]: s.reindex(labels)  ValueError: cannot reindex on an axis with duplicate labels  Generally, you can intersect the desired labels with the current
axis, and then reindex.  .. ipython:: python  s.loc[s.index.intersection(labels)].reindex(labels)  However, this would *still* raise if your resulting index is duplicated.  .. code-block:: ipython  In [41]: labels = ['a', 'd']  In [42]: s.loc[s.index.intersection(labels)].reindex(labels)  ValueError: cannot reindex on an axis with duplicate labels  .. _indexing.basics.partial_setting:

Selecting random samples

A random selection of rows or columns from a Series or DataFrame with the `~DataFrame.sample` method. The method will sample rows by default, and accepts a specific number of rows/columns to return, or a fraction of rows.  .. ipython:: python  s = pd.Series([0, 1, 2, 3, 4, 5])  # When no arguments are passed, returns 1 row.  s.sample()  # One may specify either a number of rows:  s.sample(n=3)  # Or a fraction of the rows:  s.sample(frac=0.5)  By default, ``sample`` will return each row at most once, but one can also sample with replacement
using the ``replace`` option:  .. ipython:: python  s = pd.Series([0, 1, 2, 3, 4, 5])  # Without replacement (default):  s.sample(n=6, replace=False)  # With replacement:  s.sample(n=6, replace=True)  By default, each row has an equal probability of being selected, but if you want rows
to have different probabilities, you can pass the ``sample`` function sampling weights as
``weights``. These weights can be a list, a NumPy array, or a Series, but they must be of the same length as the object you are sampling. Missing values will be treated as a weight of zero, and inf values are not allowed. If weights do not sum to 1, they will be re-normalized by dividing all weights by the sum of the weights. For example:  .. ipython:: python  s = pd.Series([0, 1, 2, 3, 4, 5])  example_weights = [0, 0, 0.2, 0.2, 0.2, 0.4]  s.sample(n=3, weights=example_weights)  # Weights will be re-normalized automatically  example_weights2 = [0.5, 0, 0, 0, 0, 0]  s.sample(n=1, weights=example_weights2)  When applied to a DataFrame, you can use a column of the DataFrame as sampling weights
(provided you are sampling rows and not columns) by simply passing the name of the column
as a string.  .. ipython:: python  df2 = pd.DataFrame({'col1': [9, 8, 7, 6],  'weight_column': [0.5, 0.4, 0.1, 0]})  df2.sample(n=3, weights='weight_column')  ``sample`` also allows users to sample columns instead of rows using the ``axis`` argument.  .. ipython:: python  df3 = pd.DataFrame({'col1': [1, 2, 3], 'col2': [2, 3, 4]})  df3.sample(n=1, axis=1)  Finally, one can also set a seed for ``sample``'s random number generator using the ``random_state`` argument, which will accept either an integer (as a seed) or a NumPy RandomState object.  .. ipython:: python  df4 = pd.DataFrame({'col1': [1, 2, 3], 'col2': [2, 3, 4]})  # With a given seed, the sample will always draw the same rows.  df4.sample(n=2, random_state=2)  df4.sample(n=2, random_state=2)

Setting with enlargement

The ``.loc/[]`` operations can perform enlargement when setting a non-existent key for that axis.  In the ``Series`` case this is effectively an appending operation.  .. ipython:: python  se = pd.Series([1, 2, 3])  se  se[5] = 5.  se  A ``DataFrame`` can be enlarged on either axis via ``.loc``.  .. ipython:: python  dfi = pd.DataFrame(np.arange(6).reshape(3, 2),  columns=['A', 'B'])  dfi  dfi.loc[:, 'C'] = dfi.loc[:, 'A']  dfi  This is like an ``append`` operation on the ``DataFrame``.  .. ipython:: python  dfi.loc[3] = 5  dfi  .. _indexing.basics.get_value:

Fast scalar value getting and setting

Since indexing with ``[]`` must handle a lot of cases (single-label access,
slicing, boolean indexing, etc.), it has a bit of overhead in order to figure
out what you're asking for. If you only want to access a scalar value, the
fastest way is to use the ``at`` and ``iat`` methods, which are implemented on
all of the data structures.  Similarly to ``loc``, ``at`` provides **label** based scalar lookups, while, ``iat`` provides **integer** based lookups analogously to ``iloc``  .. ipython:: python  s.iat[5]  df.at[dates[5], 'A']  df.iat[3, 0]  You can also set using these same indexers.  .. ipython:: python  df.at[dates[5], 'E'] = 7  df.iat[3, 0] = 7  ``at`` may enlarge the object in-place as above if the indexer is missing.  .. ipython:: python  df.at[dates[-1] + pd.Timedelta('1 day'), 0] = 7  df

Boolean indexing

.. _indexing.boolean:  Another common operation is the use of boolean vectors to filter the data.
The operators are: ``|`` for ``or``, ``&`` for ``and``, and ``~`` for ``not``.
These **must** be grouped by using parentheses, since by default Python will
evaluate an expression such as ``df['A'] > 2 & df['B'] < 3`` as
``df['A'] > (2 & df['B']) < 3``, while the desired evaluation order is
``(df['A'] > 2) & (df['B'] < 3)``.  Using a boolean vector to index a Series works exactly as in a NumPy ndarray:  .. ipython:: python  s = pd.Series(range(-3, 4))  s  s[s > 0]  s[(s < -1) | (s > 0.5)]  s[~(s < 0)]  You may select rows from a DataFrame using a boolean vector the same length as
the DataFrame's index (for example, something derived from one of the columns
of the DataFrame):  .. ipython:: python  df[df['A'] > 0]  List comprehensions and the ``map`` method of Series can also be used to produce
more complex criteria:  .. ipython:: python  df2 = pd.DataFrame({'a': ['one', 'one', 'two', 'three', 'two', 'one', 'six'],  'b': ['x', 'y', 'y', 'x', 'y', 'x', 'x'],  'c': np.random.randn(7)})  # only want 'two' or 'three'  criterion = df2['a'].map(lambda x: x.startswith('t'))  df2[criterion]  # equivalent but slower  df2[[x.startswith('t') for x in df2['a']]]  # Multiple criteria  df2[criterion & (df2['b'] == 'x')]  With the choice methods `Selection by Label <indexing.label>`, `Selection by Position <indexing.integer>`,
and `Advanced Indexing <advanced>` you may select along more than one axis using boolean vectors combined with other indexing expressions.  .. ipython:: python  df2.loc[criterion & (df2['b'] == 'x'), 'b':'c']  .. warning::  ``iloc`` supports two kinds of boolean indexing. If the indexer is a boolean ``Series``,  an error will be raised. For instance, in the following example, ``df.iloc[s.values, 1]`` is ok.  The boolean indexer is an array. But ``df.iloc[s, 1]`` would raise ``ValueError``.  .. ipython:: python  df = pd.DataFrame([[1, 2], [3, 4], [5, 6]],  index=list('abc'),  columns=['A', 'B'])  s = (df['A'] > 2)  s  df.loc[s, 'B']  df.iloc[s.values, 1]  .. _indexing.basics.indexing_isin:

Indexing with isin

Consider the `~Series.isin` method of ``Series``, which returns a boolean
vector that is true wherever the ``Series`` elements exist in the passed list.
This allows you to select rows where one or more columns have values you want:  .. ipython:: python  s = pd.Series(np.arange(5), index=np.arange(5)[::-1], dtype='int64')  s  s.isin([2, 4, 6])  s[s.isin([2, 4, 6])]  The same method is available for ``Index`` objects and is useful for the cases
when you don't know which of the sought labels are in fact present:  .. ipython:: python  s[s.index.isin([2, 4, 6])]  # compare it to the following  s.reindex([2, 4, 6])  In addition to that, ``MultiIndex`` allows selecting a separate level to use
in the membership check:  .. ipython:: python  s_mi = pd.Series(np.arange(6),  index=pd.MultiIndex.from_product([[0, 1], ['a', 'b', 'c']]))  s_mi  s_mi.iloc[s_mi.index.isin([(1, 'a'), (2, 'b'), (0, 'c')])]  s_mi.iloc[s_mi.index.isin(['a', 'c', 'e'], level=1)]  DataFrame also has an `~DataFrame.isin` method.  When calling ``isin``, pass a set of
values as either an array or dict.  If values is an array, ``isin`` returns
a DataFrame of booleans that is the same shape as the original DataFrame, with True
wherever the element is in the sequence of values.  .. ipython:: python  df = pd.DataFrame({'vals': [1, 2, 3, 4], 'ids': ['a', 'b', 'f', 'n'],  'ids2': ['a', 'n', 'c', 'n']})  values = ['a', 'b', 1, 3]  df.isin(values)  Oftentimes you'll want to match certain values with certain columns.
Just make values a ``dict`` where the key is the column, and the value is
a list of items you want to check for.  .. ipython:: python  values = {'ids': ['a', 'b'], 'vals': [1, 3]}  df.isin(values)  To return the DataFrame of booleans where the values are *not* in the original DataFrame,
use the ``~`` operator:  .. ipython:: python  values = {'ids': ['a', 'b'], 'vals': [1, 3]}  ~df.isin(values)  Combine DataFrame's ``isin`` with the ``any()`` and ``all()`` methods to
quickly select subsets of your data that meet a given criteria.
To select a row where each column meets its own criterion:  .. ipython:: python  values = {'ids': ['a', 'b'], 'ids2': ['a', 'c'], 'vals': [1, 3]}  row_mask = df.isin(values).all(1)  df[row_mask]  .. _indexing.where_mask:

The `~pandas.DataFrame.where` Method and Masking

Selecting values from a Series with a boolean vector generally returns a
subset of the data. To guarantee that selection output has the same shape as
the original data, you can use the ``where`` method in ``Series`` and ``DataFrame``.  To return only the selected rows:  .. ipython:: python  s[s > 0]  To return a Series of the same shape as the original:  .. ipython:: python  s.where(s > 0)  Selecting values from a DataFrame with a boolean criterion now also preserves
input data shape. ``where`` is used under the hood as the implementation.
The code below is equivalent to ``df.where(df < 0)``.  .. ipython:: python  dates = pd.date_range('1/1/2000', periods=8)  df = pd.DataFrame(np.random.randn(8, 4),  index=dates, columns=['A', 'B', 'C', 'D'])  .. ipython:: python  df[df < 0]  In addition, ``where`` takes an optional ``other`` argument for replacement of
values where the condition is False, in the returned copy.  .. ipython:: python  df.where(df < 0, -df)  You may wish to set values based on some boolean criteria.
This can be done intuitively like so:  .. ipython:: python  s2 = s.copy()  s2[s2 < 0] = 0  s2  df2 = df.copy()  df2[df2 < 0] = 0  df2  ``where`` returns a modified copy of the data.  .. note::  The signature for `DataFrame.where` differs from `numpy.where`.  Roughly ``df1.where(m, df2)`` is equivalent to ``np.where(m, df1, df2)``.  .. ipython:: python  df.where(df < 0, -df) == np.where(df < 0, df, -df)  **Alignment**  Furthermore, ``where`` aligns the input boolean condition (ndarray or DataFrame),
such that partial selection with setting is possible. This is analogous to
partial setting via ``.loc`` (but on the contents rather than the axis labels).  .. ipython:: python  df2 = df.copy()  df2[df2[1:4] > 0] = 3  df2  Where can also accept ``axis`` and ``level`` parameters to align the input when
performing the ``where``.  .. ipython:: python  df2 = df.copy()  df2.where(df2 > 0, df2['A'], axis='index')  This is equivalent to (but faster than) the following.  .. ipython:: python  df2 = df.copy()  df.apply(lambda x, y: x.where(x > 0, y), y=df['A'])  ``where`` can accept a callable as condition and ``other`` arguments. The function must
be with one argument (the calling Series or DataFrame) and that returns valid output
as condition and ``other`` argument.  .. ipython:: python  df3 = pd.DataFrame({'A': [1, 2, 3],  'B': [4, 5, 6],  'C': [7, 8, 9]})  df3.where(lambda x: x > 4, lambda x: x + 10)  Mask
~~~  `~pandas.DataFrame.mask` is the inverse boolean operation of ``where``.  .. ipython:: python  s.mask(s >= 0)  df.mask(df >= 0)  .. _indexing.np_where:

Setting with enlargement conditionally using `numpy`

An alternative to `~pandas.DataFrame.where` is to use `numpy.where`.
Combined with setting a new column, you can use it to enlarge a DataFrame where the
values are determined conditionally.  Consider you have two choices to choose from in the following DataFrame. And you want to
set a new column color to 'green' when the second column has 'Z'.  You can do the
following:  .. ipython:: python  df = pd.DataFrame({'col1': list('ABBC'), 'col2': list('ZZXY')})  df['color'] = np.where(df['col2'] == 'Z', 'green', 'red')  df  If you have multiple conditions, you can use `numpy.select` to achieve that.  Say
corresponding to three conditions there are three choice of colors, with a fourth color
as a fallback, you can do the following.  .. ipython:: python  conditions = [  (df['col2'] == 'Z') & (df['col1'] == 'A'),  (df['col2'] == 'Z') & (df['col1'] == 'B'),  (df['col1'] == 'B')  ]  choices = ['yellow', 'blue', 'purple']  df['color'] = np.select(conditions, choices, default='black')  df  .. _indexing.query:

`~pandas.DataFrame` objects have a `~pandas.DataFrame.query`
method that allows selection using an expression.  You can get the value of the frame where column ``b`` has values
between the values of columns ``a`` and ``c``. For example:  .. ipython:: python  n = 10  df = pd.DataFrame(np.random.rand(n, 3), columns=list('abc'))  df  # pure python  df[(df['a'] < df['b']) & (df['b'] < df['c'])]  # query  df.query('(a < b) & (b < c)')  Do the same thing but fall back on a named index if there is no column
with the name ``a``.  .. ipython:: python  df = pd.DataFrame(np.random.randint(n / 2, size=(n, 2)), columns=list('bc'))  df.index.name = 'a'  df  df.query('a < b and b < c')  If instead you don't want to or cannot name your index, you can use the name
``index`` in your query expression:  .. ipython:: python  df = pd.DataFrame(np.random.randint(n, size=(n, 2)), columns=list('bc'))  df  df.query('index < b < c')  .. note::  If the name of your index overlaps with a column name, the column name is  given precedence. For example,  .. ipython:: python  df = pd.DataFrame({'a': np.random.randint(5, size=5)})  df.index.name = 'a'  df.query('a > 2')  # uses the column 'a', not the index  You can still use the index in a query expression by using the special  identifier 'index':  .. ipython:: python  df.query('index > 2')  If for some reason you have a column named ``index``, then you can refer to  the index as ``ilevel_0`` as well, but at this point you should consider  renaming your columns to something less ambiguous.  `~pandas.MultiIndex` `~pandas.DataFrame.query` Syntax
~~~  You can also use the levels of a ``DataFrame`` with a
`~pandas.MultiIndex` as if they were columns in the frame:  .. ipython:: python  n = 10  colors = np.random.choice(['red', 'green'], size=n)  foods = np.random.choice(['eggs', 'ham'], size=n)  colors  foods  index = pd.MultiIndex.from_arrays([colors, foods], names=['color', 'food'])  df = pd.DataFrame(np.random.randn(n, 2), index=index)  df  df.query('color == "red"')  If the levels of the ``MultiIndex`` are unnamed, you can refer to them using
special names:  .. ipython:: python  df.index.names = [None, None]  df  df.query('ilevel_0 == "red"')  The convention is ``ilevel_0``, which means "index level 0" for the 0th level
of the ``index``.  `~pandas.DataFrame.query` Use Cases
~~~  A use case for `~pandas.DataFrame.query` is when you have a collection of
`~pandas.DataFrame` objects that have a subset of column names (or index
levels/names) in common. You can pass the same query to both frames *without*
having to specify which frame you're interested in querying  .. ipython:: python  df = pd.DataFrame(np.random.rand(n, 3), columns=list('abc'))  df  df2 = pd.DataFrame(np.random.rand(n + 2, 3), columns=df.columns)  df2  expr = '0.0 <= a <= c <= 0.5'  map(lambda frame: frame.query(expr), [df, df2])  `~pandas.DataFrame.query` Python versus pandas Syntax Comparison
~~~  Full numpy-like syntax:  .. ipython:: python  df = pd.DataFrame(np.random.randint(n, size=(n, 3)), columns=list('abc'))  df  df.query('(a < b) & (b < c)')  df[(df['a'] < df['b']) & (df['b'] < df['c'])]  Slightly nicer by removing the parentheses (comparison operators bind tighter
than ``&`` and ``|``):  .. ipython:: python  df.query('a < b & b < c')  Use English instead of symbols:  .. ipython:: python  df.query('a < b and b < c')  Pretty close to how you might write it on paper:  .. ipython:: python  df.query('a < b < c')  The ``in`` and ``not in`` operators
~~~  `~pandas.DataFrame.query` also supports special use of Python's ``in`` and
``not in`` comparison operators, providing a succinct syntax for calling the

``isin`` method of a ``Series`` or ``DataFrame``.  .. ipython:: python  # get all rows where columns "a" and "b" have overlapping values  df = pd.DataFrame({'a': list('aabbccddeeff'), 'b': list('aaabbbccc'),  'c': np.random.randint(5, size=12),  'd': np.random.randint(9, size=12)})  df  df.query('a in b')  # How you'd do it in pure Python  df[df['a'].isin(df['b'])]  df.query('a not in b')  # pure Python  df[~df['a'].isin(df['b'])]  You can combine this with other expressions for very succinct queries:  .. ipython:: python  # rows where cols a and b have overlapping values  # and col c's values are less than col d's  df.query('a in b and c < d')  # pure Python  df[df['b'].isin(df['a']) & (df['c'] < df['d'])]  .. note::  Note that ``in`` and ``not in`` are evaluated in Python, since ``numexpr``  has no equivalent of this operation. However, **only the** ``in``/``not in``  **expression itself** is evaluated in vanilla Python. For example, in the  expression  .. code-block:: python  df.query('a in b + c + d')  ``(b + c + d)`` is evaluated by ``numexpr`` and *then* the ``in``  operation is evaluated in plain Python. In general, any operations that can  be evaluated using ``numexpr`` will be.  Special use of the ``==`` operator with ``list`` objects
~~~  Comparing a ``list`` of values to a column using ``==``/``!=`` works similarly
to ``in``/``not in``.  .. ipython:: python  df.query('b == ["a", "b", "c"]')  # pure Python  df[df['b'].isin(["a", "b", "c"])]  df.query('c == [1, 2]')  df.query('c != [1, 2]')  # using in/not in  df.query('[1, 2] in c')  df.query('[1, 2] not in c')  # pure Python  df[df['c'].isin([1, 2])]  Boolean operators
~~~  You can negate boolean expressions with the word ``not`` or the ``~`` operator.  .. ipython:: python  df = pd.DataFrame(np.random.rand(n, 3), columns=list('abc'))  df['bools'] = np.random.rand(len(df)) > 0.5  df.query('~bools')  df.query('not bools')  df.query('not bools') == df[~df['bools']]  Of course, expressions can be arbitrarily complex too:  .. ipython:: python  # short query syntax  shorter = df.query('a < b < c and (not bools) or bools > 2')  # equivalent in pure Python  longer = df[(df['a'] < df['b'])  & (df['b'] < df['c'])  & (~df['bools'])  | (df['bools'] > 2)]  shorter  longer  shorter == longer  Performance of `~pandas.DataFrame.query`
~~~  ``DataFrame.query()`` using ``numexpr`` is slightly faster than Python for
large frames.  ..  The eval-perf.png figure below was generated with /doc/scripts/eval_performance.py  You will only see the performance benefits of using the ``numexpr`` engine
with ``DataFrame.query()`` if your frame has more than approximately 100,000
rows.  This plot was created using a ``DataFrame`` with 3 columns each containing
floating point values generated using ``numpy.random.randn()``.  .. ipython:: python  df = pd.DataFrame(np.random.randn(8, 4),  index=dates, columns=['A', 'B', 'C', 'D'])  df2 = df.copy()

Duplicate data

.. _indexing.duplicate:  If you want to identify and remove duplicate rows in a DataFrame,  there are
two methods that will help: ``duplicated`` and ``drop_duplicates``. Each
takes as an argument the columns to use to identify duplicated rows.  * ``duplicated`` returns a boolean vector whose length is the number of rows, and which indicates whether a row is duplicated.
* ``drop_duplicates`` removes duplicate rows.  By default, the first observed row of a duplicate set is considered unique, but
each method has a ``keep`` parameter to specify targets to be kept.  * ``keep='first'`` (default): mark / drop duplicates except for the first occurrence.
* ``keep='last'``: mark / drop duplicates except for the last occurrence.
* ``keep=False``: mark  / drop all duplicates.  .. ipython:: python  df2 = pd.DataFrame({'a': ['one', 'one', 'two', 'two', 'two', 'three', 'four'],  'b': ['x', 'y', 'x', 'y', 'x', 'x', 'x'],  'c': np.random.randn(7)})  df2  df2.duplicated('a')  df2.duplicated('a', keep='last')  df2.duplicated('a', keep=False)  df2.drop_duplicates('a')  df2.drop_duplicates('a', keep='last')  df2.drop_duplicates('a', keep=False)  Also, you can pass a list of columns to identify duplications.  .. ipython:: python  df2.duplicated(['a', 'b'])  df2.drop_duplicates(['a', 'b'])  To drop duplicates by index value, use ``Index.duplicated`` then perform slicing.
The same set of options are available for the ``keep`` parameter.  .. ipython:: python  df3 = pd.DataFrame({'a': np.arange(6),  'b': np.random.randn(6)},  index=['a', 'a', 'b', 'c', 'b', 'a'])  df3  df3.index.duplicated()  df3[~df3.index.duplicated()]  df3[~df3.index.duplicated(keep='last')]  df3[~df3.index.duplicated(keep=False)]  .. _indexing.dictionarylike:

Dictionary-like `~pandas.DataFrame.get` method

Each of Series or DataFrame have a ``get`` method which can return a
default value.  .. ipython:: python  s = pd.Series([1, 2, 3], index=['a', 'b', 'c'])  s.get('a')  # equivalent to s['a']  s.get('x', default=-1)  .. _indexing.lookup:

Looking up values by index/column labels

Sometimes you want to extract a set of values given a sequence of row labels
and column labels, this can be achieved by ``pandas.factorize``  and NumPy indexing.
For instance:  .. ipython:: python  df = pd.DataFrame({'col': ["A", "A", "B", "B"],  'A': [80, 23, np.nan, 22],  'B': [80, 55, 76, 67]})  df  idx, cols = pd.factorize(df['col'])  df.reindex(cols, axis=1).to_numpy()[np.arange(len(df)), idx]  Formerly this could be achieved with the dedicated ``DataFrame.lookup`` method
which was deprecated in version 1.2.0 and removed in version 2.0.0.  .. _indexing.class:

Index objects

The pandas `~pandas.Index` class and its subclasses can be viewed as
implementing an *ordered multiset*. Duplicates are allowed. However, if you try
to convert an `~pandas.Index` object with duplicate entries into a
``set``, an exception will be raised.  `~pandas.Index` also provides the infrastructure necessary for
lookups, data alignment, and reindexing. The easiest way to create an
`~pandas.Index` directly is to pass a ``list`` or other sequence to
`~pandas.Index`:  .. ipython:: python  index = pd.Index(['e', 'd', 'a', 'b'])  index  'd' in index  or using numbers:  .. ipython:: python  index = pd.Index([1, 5, 12])  index  5 in index  If no dtype is given, ``Index`` tries to infer the dtype from the data.
It is also possible to give an explicit dtype when instantiating an `Index`:  .. ipython:: python  index = pd.Index(['e', 'd', 'a', 'b'], dtype="string")  index  index = pd.Index([1, 5, 12], dtype="int8")  index  index = pd.Index([1, 5, 12], dtype="float32")  index  You can also pass a ``name`` to be stored in the index:  .. ipython:: python  index = pd.Index(['e', 'd', 'a', 'b'], name='something')  index.name  The name, if set, will be shown in the console display:  .. ipython:: python  index = pd.Index(list(range(5)), name='rows')  columns = pd.Index(['A', 'B', 'C'], name='cols')  df = pd.DataFrame(np.random.randn(5, 3), index=index, columns=columns)  df  df['A']  .. _indexing.set_metadata:  Setting metadata
~~~  Indexes are "mostly immutable", but it is possible to set and change their
``name`` attribute. You can use the ``rename``, ``set_names`` to set these attributes
directly, and they default to returning a copy.  See `Advanced Indexing <advanced>` for usage of MultiIndexes.  .. ipython:: python  ind = pd.Index([1, 2, 3])  ind.rename("apple")  ind  ind = ind.set_names(["apple"])  ind.name = "bob"  ind  ``set_names``, ``set_levels``, and ``set_codes`` also take an optional
``level`` argument  .. ipython:: python  index = pd.MultiIndex.from_product([range(3), ['one', 'two']], names=['first', 'second'])  index  index.levels[1]  index.set_levels(["a", "b"], level=1)  .. _indexing.set_ops:  Set operations on Index objects
~~~  The two main operations are ``union`` and ``intersection``.
Difference is provided via the ``.difference()`` method.  .. ipython:: python  a = pd.Index(['c', 'b', 'a'])  b = pd.Index(['c', 'e', 'd'])  a.difference(b)  Also available is the ``symmetric_difference`` operation, which returns elements
that appear in either ``idx1`` or ``idx2``, but not in both. This is
equivalent to the Index created by ``idx1.difference(idx2).union(idx2.difference(idx1))``,
with duplicates dropped.  .. ipython:: python  idx1 = pd.Index([1, 2, 3, 4])  idx2 = pd.Index([2, 3, 4, 5])  idx1.symmetric_difference(idx2)  .. note::  The resulting index from a set operation will be sorted in ascending order.  When performing `Index.union` between indexes with different dtypes, the indexes
must be cast to a common dtype. Typically, though not always, this is object dtype. The
exception is when performing a union between integer and float data. In this case, the
integer values are converted to float  .. ipython:: python  idx1 = pd.Index([0, 1, 2])  idx2 = pd.Index([0.5, 1.5])  idx1.union(idx2)  .. _indexing.missing:  Missing values
~~~  .. important::  Even though ``Index`` can hold missing values (``NaN``), it should be avoided  if you do not want any unexpected results. For example, some operations  exclude missing values implicitly.  ``Index.fillna`` fills missing values with specified scalar value.  .. ipython:: python  idx1 = pd.Index([1, np.nan, 3, 4])  idx1  idx1.fillna(2)  idx2 = pd.DatetimeIndex([pd.Timestamp('2011-01-01'),  pd.NaT,  pd.Timestamp('2011-01-03')])  idx2  idx2.fillna(pd.Timestamp('2011-01-02'))

Set / reset index

Occasionally you will load or create a data set into a DataFrame and want to
add an index after you've already done so. There are a couple of different
ways.  .. _indexing.set_index:  Set an index
~~~  DataFrame has a `~DataFrame.set_index` method which takes a column name
(for a regular ``Index``) or a list of column names (for a ``MultiIndex``).
To create a new, re-indexed DataFrame:  .. ipython:: python  data = pd.DataFrame({'a': ['bar', 'bar', 'foo', 'foo'],  'b': ['one', 'two', 'one', 'two'],  'c': ['z', 'y', 'x', 'w'],  'd': [1., 2., 3, 4]})  .. ipython:: python  data  indexed1 = data.set_index('c')  indexed1  indexed2 = data.set_index(['a', 'b'])  indexed2  The ``append`` keyword option allow you to keep the existing index and append
the given columns to a MultiIndex:  .. ipython:: python  frame = data.set_index('c', drop=False)  frame = frame.set_index(['a', 'b'], append=True)  frame  Other options in ``set_index`` allow you not drop the index columns.  .. ipython:: python  data.set_index('c', drop=False)  Reset the index
~~~  As a convenience, there is a new function on DataFrame called
`~DataFrame.reset_index` which transfers the index values into the
DataFrame's columns and sets a simple integer index.
This is the inverse operation of `~DataFrame.set_index`.  .. ipython:: python  data  data.reset_index()  The output is more similar to a SQL table or a record array. The names for the
columns derived from the index are the ones stored in the ``names`` attribute.  You can use the ``level`` keyword to remove only a portion of the index:  .. ipython:: python  frame  frame.reset_index(level=1)  ``reset_index`` takes an optional parameter ``drop`` which if true simply
discards the index, instead of putting index values in the DataFrame's columns.  Adding an ad hoc index
~~~  If you create an index yourself, you can just assign it to the ``index`` field:  .. code-block:: python  data.index = index  .. _indexing.view_versus_copy:

When setting values in a pandas object, care must be taken to avoid what is called
``chained indexing``. Here is an example.  .. ipython:: python  dfmi = pd.DataFrame([list('abcd'),  list('efgh'),  list('ijkl'),  list('mnop')],  columns=pd.MultiIndex.from_product([['one', 'two'],  ['first', 'second']]))  dfmi  Compare these two access methods:  .. ipython:: python  dfmi['one']['second']  .. ipython:: python  dfmi.loc[:, ('one', 'second')]  These both yield the same results, so which should you use? It is instructive to understand the order
of operations on these and why method 2 (``.loc``) is much preferred over method 1 (chained ``[]``).  ``dfmi['one']`` selects the first level of the columns and returns a DataFrame that is singly-indexed.
Then another Python operation ``dfmi_with_one['second']`` selects the series indexed by ``'second'``.
This is indicated by the variable ``dfmi_with_one`` because pandas sees these operations as separate events.
e.g. separate calls to ``__getitem__``, so it has to treat them as linear operations, they happen one after another.  Contrast this to ``df.loc[:,('one','second')]`` which passes a nested tuple of ``(slice(None),('one','second'))`` to a single call to
``__getitem__``. This allows pandas to deal with this as a single entity. Furthermore this order of operations *can* be significantly
faster, and allows one to index *both* axes if so desired.  Why does assignment fail when using chained indexing?
~~~  The problem in the previous section is just a performance issue. What's up with
the ``SettingWithCopy`` warning? We don't **usually** throw warnings around when
you do something that might cost a few extra milliseconds!  But it turns out that assigning to the product of chained indexing has
inherently unpredictable results. To see this, think about how the Python
interpreter executes this code:  .. ipython:: python  value = None  .. code-block:: python  dfmi.loc[:, ('one', 'second')] = value  # becomes  dfmi.loc.__setitem__((slice(None), ('one', 'second')), value)  But this code is handled differently:  .. code-block:: python  dfmi['one']['second'] = value  # becomes  dfmi.__getitem__('one').__setitem__('second', value)  See that ``__getitem__`` in there? Outside of simple cases, it's very hard to
predict whether it will return a view or a copy (it depends on the memory layout
of the array, about which pandas makes no guarantees), and therefore whether
the ``__setitem__`` will modify ``dfmi`` or a temporary object that gets thrown
out immediately afterward. **That's** what ``SettingWithCopy`` is warning you
about!  .. note:: You may be wondering whether we should be concerned about the ``loc``  property in the first example. But ``dfmi.loc`` is guaranteed to be ``dfmi``  itself with modified indexing behavior, so ``dfmi.loc.__getitem__`` /  ``dfmi.loc.__setitem__`` operate on ``dfmi`` directly. Of course,  ``dfmi.loc.__getitem__(idx)`` may be a view or a copy of ``dfmi``.  Sometimes a ``SettingWithCopy`` warning will arise at times when there's no
obvious chained indexing going on. **These** are the bugs that
``SettingWithCopy`` is designed to catch! pandas is probably trying to warn you
that you've done this:  .. code-block:: python  def do_something(df):  foo = df[['bar', 'baz']]  # Is foo a view? A copy? Nobody knows!  # ... many lines here ...  # We don't know whether this will modify df or not!  foo['quux'] = value  return foo  Yikes!  .. _indexing.evaluation_order:  Evaluation order matters
~~~  When you use chained indexing, the order and type of the indexing operation
partially determine whether the result is a slice into the original object, or
a copy of the slice.  pandas has the ``SettingWithCopyWarning`` because assigning to a copy of a
slice is frequently not intentional, but a mistake caused by chained indexing
returning a copy where a slice was expected.  If you would like pandas to be more or less trusting about assignment to a
chained indexing expression, you can set the `option <options>`

``mode.chained_assignment`` to one of these values:  * ``'warn'``, the default, means a ``SettingWithCopyWarning`` is printed.
* ``'raise'`` means pandas will raise a ``SettingWithCopyError``  you have to deal with.
* ``None`` will suppress the warnings entirely.  .. ipython:: python  dfb = pd.DataFrame({'a': ['one', 'one', 'two',  'three', 'two', 'one', 'six'],  'c': np.arange(7)})  # This will show the SettingWithCopyWarning  # but the frame values will be set  dfb['c'][dfb['a'].str.startswith('o')] = 42  This however is operating on a copy and will not work.  ::  >>> pd.set_option('mode.chained_assignment','warn')  >>> dfb[dfb['a'].str.startswith('o')]['c'] = 42  Traceback (most recent call last)  ...  SettingWithCopyWarning:  A value is trying to be set on a copy of a slice from a DataFrame.  Try using .loc[row_index,col_indexer] = value instead  A chained assignment can also crop up in setting in a mixed dtype frame.  .. note::  These setting rules apply to all of ``.loc/.iloc``.  The following is the recommended access method using ``.loc`` for multiple items (using ``mask``) and a single item using a fixed index:  .. ipython:: python  dfc = pd.DataFrame({'a': ['one', 'one', 'two',  'three', 'two', 'one', 'six'],  'c': np.arange(7)})  dfd = dfc.copy()  # Setting multiple items using a mask  mask = dfd['a'].str.startswith('o')  dfd.loc[mask, 'c'] = 42  dfd  # Setting a single item  dfd = dfc.copy()  dfd.loc[2, 'a'] = 11  dfd  The following *can* work at times, but it is not guaranteed to, and therefore should be avoided:  .. ipython:: python  dfd = dfc.copy()  dfd['a'][2] = 111  dfd  Last, the subsequent example will **not** work at all, and so should be avoided:  ::  >>> pd.set_option('mode.chained_assignment','raise')  >>> dfd.loc[0]['a'] = 111  Traceback (most recent call last)  ...  SettingWithCopyError:  A value is trying to be set on a copy of a slice from a DataFrame.  Try using .loc[row_index,col_indexer] = value instead  .. warning::  The chained assignment warnings / exceptions are aiming to inform the user of a possibly invalid  assignment. There may be false positives; situations where a chained assignment is inadvertently  reported.

Construction

pandas can represent integer data with possibly missing values using
`arrays.IntegerArray`. This is an `extension type <extending.extension-types>`
implemented within pandas.  .. ipython:: python  arr = pd.array([1, 2, None], dtype=pd.Int64Dtype())  arr  Or the string alias ``"Int64"`` (note the capital ``"I"``, to differentiate from
NumPy's ``'int64'`` dtype:  .. ipython:: python  pd.array([1, 2, np.nan], dtype="Int64")  All NA-like values are replaced with `pandas.NA`.  .. ipython:: python  pd.array([1, 2, np.nan, None, pd.NA], dtype="Int64")  This array can be stored in a `DataFrame` or `Series` like any
NumPy array.  .. ipython:: python  pd.Series(arr)  You can also pass the list-like object to the `Series` constructor
with the dtype.  .. warning::  Currently `pandas.array` and `pandas.Series` use different  rules for dtype inference. `pandas.array` will infer a nullable-  integer dtype  .. ipython:: python  pd.array([1, None])  pd.array([1, 2])  For backwards-compatibility, `Series` infers these as either  integer or float dtype  .. ipython:: python  pd.Series([1, None])  pd.Series([1, 2])  We recommend explicitly providing the dtype to avoid confusion.  .. ipython:: python  pd.array([1, None], dtype="Int64")  pd.Series([1, None], dtype="Int64")  In the future, we may provide an option for `Series` to infer a  nullable-integer dtype.

Operations involving an integer array will behave similar to NumPy arrays.
Missing values will be propagated, and the data will be coerced to another
dtype if needed.  .. ipython:: python  s = pd.Series([1, 2, None], dtype="Int64")  # arithmetic  s + 1  # comparison  s == 1  # indexing  s.iloc[1:3]  # operate with other dtypes  s + s.iloc[1:3].astype("Int8")  # coerce when needed  s + 0.01  These dtypes can operate as part of ``DataFrame``.  .. ipython:: python  df = pd.DataFrame({"A": s, "B": [1, 1, 3], "C": list("aab")})  df  df.dtypes  These dtypes can be merged & reshaped & casted.  .. ipython:: python  pd.concat([df[["A"]], df[["B", "C"]]], axis=1).dtypes  df["A"].astype(float)  Reduction and groupby operations such as 'sum' work as well.  .. ipython:: python  df.sum()  df.groupby("B").A.sum()

Scalar NA Value

`arrays.IntegerArray` uses `pandas.NA` as its scalar
missing value. Slicing a single element that's missing will return
`pandas.NA`  .. ipython:: python  a = pd.array([1, None], dtype="Int64")  a[1]

IO tools (text, CSV, HDF5, ...)

The pandas I/O API is a set of top level ``reader`` functions accessed like
`pandas.read_csv` that generally return a pandas object. The corresponding
``writer`` functions are object methods that are accessed like
`DataFrame.to_csv`. Below is a table containing available ``readers`` and
``writers``.  .. csv-table::  "Format Type", "Data Description", "Reader", "Writer"  30, 100, 60, 60  ;  text;CSV_;`read_csv<io.read_csv_table>`;`to_csv<io.store_in_csv>`  text;Fixed-Width Text File;`read_fwf<io.fwf_reader>`  text;JSON_;`read_json<io.json_reader>`;`to_json<io.json_writer>`  text;HTML_;`read_html<io.read_html>`;`to_html<io.html>`  text;LaTeX_;;`Styler.to_latex<io.latex>`  text;XML_;`read_xml<io.read_xml>`;`to_xml<io.xml>`  text; Local clipboard;`read_clipboard<io.clipboard>`;`to_clipboard<io.clipboard>`  binary;MS Excel_;`read_excel<io.excel_reader>`;`to_excel<io.excel_writer>`  binary;OpenDocument_;`read_excel<io.ods>`;  binary;HDF5 Format_;`read_hdf<io.hdf5>`;`to_hdf<io.hdf5>`  binary;Feather Format_;`read_feather<io.feather>`;`to_feather<io.feather>`  binary;Parquet Format_;`read_parquet<io.parquet>`;`to_parquet<io.parquet>`  binary;ORC Format_;`read_orc<io.orc>`;`to_orc<io.orc>`  binary;Stata_;`read_stata<io.stata_reader>`;`to_stata<io.stata_writer>`  binary;SAS_;`read_sas<io.sas_reader>`;  binary;SPSS_;`read_spss<io.spss_reader>`;  binary;Python Pickle Format_;`read_pickle<io.pickle>`;`to_pickle<io.pickle>`  SQL;SQL_;`read_sql<io.sql>`;`to_sql<io.sql>`  SQL;Google BigQuery_;`read_gbq<io.bigquery>`;`to_gbq<io.bigquery>`  `Here <io.perf>` is an informal performance comparison for some of these IO methods.  .. note::  For examples that use the ``StringIO`` class, make sure you import it  with ``from io import StringIO`` for Python 3.  .. _io.read_csv_table:

The workhorse function for reading text files (a.k.a. flat files) is
`read_csv`. See the `cookbook<cookbook.csv>` for some advanced strategies.  Parsing options
'''  `read_csv` accepts the following common arguments:  Basic
+++  filepath_or_buffer : various  Either a path to a file (a `python:str`, `python:pathlib.Path`,  or `py:py._path.local.LocalPath`), URL (including http, ftp, and S3  locations), or any object with a ``read()`` method (such as an open file or  `~python:io.StringIO`).
sep : str, defaults to ``','`` for `read_csv`, ``\t`` for `read_table`  Delimiter to use. If sep is ``None``, the C engine cannot automatically detect  the separator, but the Python parsing engine can, meaning the latter will be  used and automatically detect the separator by Python's builtin sniffer tool,  `python:csv.Sniffer`. In addition, separators longer than 1 character and  different from ``'\s+'`` will be interpreted as regular expressions and  will also force the use of the Python parsing engine. Note that regex  delimiters are prone to ignoring quoted data. Regex example: ``'\\r\\t'``.
delimiter : str, default ``None``  Alternative argument name for sep.
delim_whitespace : boolean, default False  Specifies whether or not whitespace (e.g. ``' '`` or ``'\t'``)  will be used as the delimiter. Equivalent to setting ``sep='\s+'``.  If this option is set to ``True``, nothing should be passed in for the  ``delimiter`` parameter.  Column and index locations and names
+++  header : int or list of ints, default ``'infer'``  Row number(s) to use as the column names, and the start of the  data. Default behavior is to infer the column names: if no names are  passed the behavior is identical to ``header=0`` and column names  are inferred from the first line of the file, if column names are  passed explicitly then the behavior is identical to  ``header=None``. Explicitly pass ``header=0`` to be able to replace  existing names.  The header can be a list of ints that specify row locations  for a MultiIndex on the columns e.g. ``[0,1,3]``. Intervening rows  that are not specified will be skipped (e.g. 2 in this example is  skipped). Note that this parameter ignores commented lines and empty  lines if ``skip_blank_lines=True``, so header=0 denotes the first  line of data rather than the first line of the file.
names : array-like, default ``None``  List of column names to use. If file contains no header row, then you should  explicitly pass ``header=None``. Duplicates in this list are not allowed.
index_col : int, str, sequence of int / str, or False, optional, default ``None``  Column(s) to use as the row labels of the ``DataFrame``, either given as  string name or column index. If a sequence of int / str is given, a  MultiIndex is used.  .. note::  ``index_col=False`` can be used to force pandas to *not* use the first  column as the index, e.g. when you have a malformed file with delimiters at  the end of each line.  The default value of ``None`` instructs pandas to guess. If the number of  fields in the column header row is equal to the number of fields in the body  of the data file, then a default index is used.  If it is larger, then  the first columns are used as index so that the remaining number of fields in  the body are equal to the number of fields in the header.  The first row after the header is used to determine the number of columns,  which will go into the index. If the subsequent rows contain less columns  than the first row, they are filled with ``NaN``.  This can be avoided through ``usecols``. This ensures that the columns are  taken as is and the trailing data are ignored.

usecols : list-like or callable, default ``None``  Return a subset of the columns. If list-like, all elements must either  be positional (i.e. integer indices into the document columns) or strings  that correspond to column names provided either by the user in ``names`` or  inferred from the document header row(s). If ``names`` are given, the document  header row(s) are not taken into account. For example, a valid list-like  ``usecols`` parameter would be ``[0, 1, 2]`` or ``['foo', 'bar', 'baz']``.  Element order is ignored, so ``usecols=[0, 1]`` is the same as ``[1, 0]``. To  instantiate a DataFrame from ``data`` with element order preserved use  ``pd.read_csv(data, usecols=['foo', 'bar'])[['foo', 'bar']]`` for columns  in ``['foo', 'bar']`` order or  ``pd.read_csv(data, usecols=['foo', 'bar'])[['bar', 'foo']]`` for  ``['bar', 'foo']`` order.  If callable, the callable function will be evaluated against the column names,  returning names where the callable function evaluates to True:  .. ipython:: python  import pandas as pd  from io import StringIO  data = "col1,col2,col3\na,b,1\na,b,2\nc,d,3"  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data), usecols=lambda x: x.upper() in ["COL1", "COL3"])  Using this parameter results in much faster parsing time and lower memory usage  when using the c engine. The Python engine loads the data first before deciding  which columns to drop.  General parsing configuration
+++  dtype : Type name or dict of column -> type, default ``None``  Data type for data or columns. E.g. ``{'a': np.float64, 'b': np.int32, 'c': 'Int64'}``  Use ``str`` or ``object`` together with suitable ``na_values`` settings to preserve  and not interpret dtype. If converters are specified, they will be applied INSTEAD  of dtype conversion.  .. versionadded:: 1.5.0  Support for defaultdict was added. Specify a defaultdict as input where  the default determines the dtype of the columns which are not explicitly  listed.  use_nullable_dtypes : bool = False  Whether or not to use nullable dtypes as default when reading data. If  set to True, nullable dtypes are used for all dtypes that have a nullable  implementation, even if no nulls are present.  .. versionadded:: 2.0  engine : {``'c'``, ``'python'``, ``'pyarrow'``}  Parser engine to use. The C and pyarrow engines are faster, while the python engine  is currently more feature-complete. Multithreading is currently only supported by  the pyarrow engine.  .. versionadded:: 1.4.0  The "pyarrow" engine was added as an *experimental* engine, and some features  are unsupported, or may not work correctly, with this engine.
converters : dict, default ``None``  Dict of functions for converting values in certain columns. Keys can either be  integers or column labels.
true_values : list, default ``None``  Values to consider as ``True``.
false_values : list, default ``None``  Values to consider as ``False``.
skipinitialspace : boolean, default ``False``  Skip spaces after delimiter.
skiprows : list-like or integer, default ``None``  Line numbers to skip (0-indexed) or number of lines to skip (int) at the start  of the file.  If callable, the callable function will be evaluated against the row  indices, returning True if the row should be skipped and False otherwise:  .. ipython:: python  data = "col1,col2,col3\na,b,1\na,b,2\nc,d,3"  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data), skiprows=lambda x: x % 2 != 0)  skipfooter : int, default ``0``  Number of lines at bottom of file to skip (unsupported with engine='c').  nrows : int, default ``None``  Number of rows of file to read. Useful for reading pieces of large files.

low_memory : boolean, default ``True``  Internally process the file in chunks, resulting in lower memory use  while parsing, but possibly mixed type inference.  To ensure no mixed  types either set ``False``, or specify the type with the ``dtype`` parameter.  Note that the entire file is read into a single ``DataFrame`` regardless,  use the ``chunksize`` or ``iterator`` parameter to return the data in chunks.  (Only valid with C parser)
memory_map : boolean, default False  If a filepath is provided for ``filepath_or_buffer``, map the file object  directly onto memory and access the data directly from there. Using this  option can improve performance because there is no longer any I/O overhead.  NA and missing data handling
+++  na_values : scalar, str, list-like, or dict, default ``None``  Additional strings to recognize as NA/NaN. If dict passed, specific per-column  NA values. See `na values const <io.navaluesconst>` below  for a list of the values interpreted as NaN by default.  keep_default_na : boolean, default ``True``  Whether or not to include the default NaN values when parsing the data.  Depending on whether ``na_values`` is passed in, the behavior is as follows:  * If ``keep_default_na`` is ``True``, and ``na_values`` are specified, ``na_values``  is appended to the default NaN values used for parsing.  * If ``keep_default_na`` is ``True``, and ``na_values`` are not specified, only  the default NaN values are used for parsing.  * If ``keep_default_na`` is ``False``, and ``na_values`` are specified, only  the NaN values specified ``na_values`` are used for parsing.  * If ``keep_default_na`` is ``False``, and ``na_values`` are not specified, no  strings will be parsed as NaN.  Note that if ``na_filter`` is passed in as ``False``, the ``keep_default_na`` and  ``na_values`` parameters will be ignored.
na_filter : boolean, default ``True``  Detect missing value markers (empty strings and the value of na_values). In  data without any NAs, passing ``na_filter=False`` can improve the performance  of reading a large file.
verbose : boolean, default ``False``  Indicate number of NA values placed in non-numeric columns.
skip_blank_lines : boolean, default ``True``  If ``True``, skip over blank lines rather than interpreting as NaN values.  .. _io.read_csv_table.datetime:  Datetime handling
+++  parse_dates : boolean or list of ints or names or list of lists or dict, default ``False``.  * If ``True`` -> try parsing the index.  * If ``[1, 2, 3]`` ->  try parsing columns 1, 2, 3 each as a separate date  column.  * If ``[[1, 3]]`` -> combine columns 1 and 3 and parse as a single date  column.  * If ``{'foo': [1, 3]}`` -> parse columns 1, 3 as date and call result 'foo'.  .. note::  A fast-path exists for iso8601-formatted dates.
infer_datetime_format : boolean, default ``False``  If ``True`` and parse_dates is enabled for a column, attempt to infer the  datetime format to speed up the processing.  .. deprecated:: 2.0.0  A strict version of this argument is now the default, passing it has no effect.
keep_date_col : boolean, default ``False``  If ``True`` and parse_dates specifies combining multiple columns then keep the  original columns.
date_parser : function, default ``None``  Function to use for converting a sequence of string columns to an array of  datetime instances. The default uses ``dateutil.parser.parser`` to do the  conversion. pandas will try to call date_parser in three different ways,  advancing to the next if an exception occurs: 1) Pass one or more arrays (as  defined by parse_dates) as arguments; 2) concatenate (row-wise) the string  values from the columns defined by parse_dates into a single array and pass  that; and 3) call date_parser once for each row using one or more strings  (corresponding to the columns defined by parse_dates) as arguments.
dayfirst : boolean, default ``False``  DD/MM format dates, international and European format.

cache_dates : boolean, default True  If True, use a cache of unique, converted dates to apply the datetime  conversion. May produce significant speed-up when parsing duplicate  date strings, especially ones with timezone offsets.  Iteration
+++  iterator : boolean, default ``False``  Return ``TextFileReader`` object for iteration or getting chunks with  ``get_chunk()``.
chunksize : int, default ``None``  Return ``TextFileReader`` object for iteration. See `iterating and chunking  <io.chunking>` below.  Quoting, compression, and file format
+++  compression : {``'infer'``, ``'gzip'``, ``'bz2'``, ``'zip'``, ``'xz'``, ``'zstd'``, ``None``, ``dict``}, default ``'infer'``  For on-the-fly decompression of on-disk data. If 'infer', then use gzip,  bz2, zip, xz, or zstandard if ``filepath_or_buffer`` is path-like ending in '.gz', '.bz2',  '.zip', '.xz', '.zst', respectively, and no decompression otherwise. If using 'zip',  the ZIP file must contain only one data file to be read in.  Set to ``None`` for no decompression. Can also be a dict with key ``'method'``  set to one of {``'zip'``, ``'gzip'``, ``'bz2'``, ``'zstd'``} and other key-value pairs are  forwarded to ``zipfile.ZipFile``, ``gzip.GzipFile``, ``bz2.BZ2File``, or ``zstandard.ZstdDecompressor``.  As an example, the following could be passed for faster compression and to  create a reproducible gzip archive:  ``compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}``.  .. versionchanged:: 1.1.0 dict option extended to support ``gzip`` and ``bz2``.  .. versionchanged:: 1.2.0 Previous versions forwarded dict entries for 'gzip' to ``gzip.open``.
thousands : str, default ``None``  Thousands separator.
decimal : str, default ``'.'``  Character to recognize as decimal point. E.g. use ``','`` for European data.
float_precision : string, default None  Specifies which converter the C engine should use for floating-point values.  The options are ``None`` for the ordinary converter, ``high`` for the  high-precision converter, and ``round_trip`` for the round-trip converter.
lineterminator : str (length 1), default ``None``  Character to break file into lines. Only valid with C parser.
quotechar : str (length 1)  The character used to denote the start and end of a quoted item. Quoted items  can include the delimiter and it will be ignored.
quoting : int or ``csv.QUOTE_*`` instance, default ``0``  Control field quoting behavior per ``csv.QUOTE_*`` constants. Use one of  ``QUOTE_MINIMAL`` (0), ``QUOTE_ALL`` (1), ``QUOTE_NONNUMERIC`` (2) or  ``QUOTE_NONE`` (3).
doublequote : boolean, default ``True``  When ``quotechar`` is specified and ``quoting`` is not ``QUOTE_NONE``,  indicate whether or not to interpret two consecutive ``quotechar`` elements  **inside** a field as a single ``quotechar`` element.
escapechar : str (length 1), default ``None``  One-character string used to escape delimiter when quoting is ``QUOTE_NONE``.
comment : str, default ``None``  Indicates remainder of line should not be parsed. If found at the beginning of  a line, the line will be ignored altogether. This parameter must be a single  character. Like empty lines (as long as ``skip_blank_lines=True``), fully  commented lines are ignored by the parameter ``header`` but not by ``skiprows``.  For example, if ``comment='#'``, parsing '#empty\\na,b,c\\n1,2,3' with  ``header=0`` will result in 'a,b,c' being treated as the header.
encoding : str, default ``None``  Encoding to use for UTF when reading/writing (e.g. ``'utf-8'``). `List of  Python standard encodings  <https://docs.python.org/3/library/codecs.html#standard-encodings>`_.

dialect : str or `python:csv.Dialect` instance, default ``None``  If provided, this parameter will override values (default or not) for the  following parameters: ``delimiter``, ``doublequote``, ``escapechar``,  ``skipinitialspace``, ``quotechar``, and ``quoting``. If it is necessary to  override values, a ParserWarning will be issued. See `python:csv.Dialect`  documentation for more details.  Error handling
+++  on_bad_lines : {{'error', 'warn', 'skip'}}, default 'error'  Specifies what to do upon encountering a bad line (a line with too many fields).  Allowed values are :  - 'error', raise an ParserError when a bad line is encountered.  - 'warn', print a warning when a bad line is encountered and skip that line.  - 'skip', skip bad lines without raising or warning when they are encountered.  .. versionadded:: 1.3.0  .. _io.dtypes:  Specifying column data types
'''  You can indicate the data type for the whole ``DataFrame`` or individual
columns:  .. ipython:: python  import numpy as np  data = "a,b,c,d\n1,2,3,4\n5,6,7,8\n9,10,11"  print(data)  df = pd.read_csv(StringIO(data), dtype=object)  df  df["a"][0]  df = pd.read_csv(StringIO(data), dtype={"b": object, "c": np.float64, "d": "Int64"})  df.dtypes  Fortunately, pandas offers more than one way to ensure that your column(s)
contain only one ``dtype``. If you're unfamiliar with these concepts, you can
see `here<basics.dtypes>` to learn more about dtypes, and
`here<basics.object_conversion>` to learn more about ``object`` conversion in
pandas.  For instance, you can use the ``converters`` argument
of `~pandas.read_csv`:  .. ipython:: python  data = "col_1\n1\n2\n'A'\n4.22"  df = pd.read_csv(StringIO(data), converters={"col_1": str})  df  df["col_1"].apply(type).value_counts()  Or you can use the `~pandas.to_numeric` function to coerce the
dtypes after reading in the data,  .. ipython:: python  df2 = pd.read_csv(StringIO(data))  df2["col_1"] = pd.to_numeric(df2["col_1"], errors="coerce")  df2  df2["col_1"].apply(type).value_counts()  which will convert all valid parsing to floats, leaving the invalid parsing
as ``NaN``.  Ultimately, how you deal with reading in columns containing mixed dtypes
depends on your specific needs. In the case above, if you wanted to ``NaN`` out
the data anomalies, then `~pandas.to_numeric` is probably your best option.
However, if you wanted for all the data to be coerced, no matter the type, then
using the ``converters`` argument of `~pandas.read_csv` would certainly be
worth trying.  .. note::  In some cases, reading in abnormal data with columns containing mixed dtypes  will result in an inconsistent dataset. If you rely on pandas to infer the  dtypes of your columns, the parsing engine will go and infer the dtypes for  different chunks of the data, rather than the whole dataset at once. Consequently,  you can end up with column(s) with mixed dtypes. For example,  .. ipython:: python  col_1 = list(range(5000)) + ["a", "b"] + list(range(5000))  df = pd.DataFrame({"col_1": col_1})  df.to_csv("foo.csv")  mixed_df = pd.read_csv("foo.csv")  mixed_df["col_1"].apply(type).value_counts()  mixed_df["col_1"].dtype  will result with ``mixed_df`` containing an ``int`` dtype for certain chunks  of the column, and ``str`` for others due to the mixed dtypes from the  data that was read in. It is important to note that the overall column will be  marked with a ``dtype`` of ``object``, which is used for columns with mixed dtypes.  .. ipython:: python  import os  os.remove("foo.csv")  Setting ``use_nullable_dtypes=True`` will result in nullable dtypes for every column.  .. ipython:: python  data = """a,b,c,d,e,f,g,h,i,j  1,2.5,True,a,,,12-31-2019,  3,4.5,False,b,6,7.5,True,a,12-31-2019,  """  df = pd.read_csv(StringIO(data), use_nullable_dtypes=True, parse_dates=["i"])  df  df.dtypes  .. _io.categorical:  Specifying categorical dtype
'''  ``Categorical`` columns can be parsed directly by specifying ``dtype='category'`` or

``dtype=CategoricalDtype(categories, ordered)``.  .. ipython:: python  data = "col1,col2,col3\na,b,1\na,b,2\nc,d,3"  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data)).dtypes  pd.read_csv(StringIO(data), dtype="category").dtypes  Individual columns can be parsed as a ``Categorical`` using a dict
specification:  .. ipython:: python  pd.read_csv(StringIO(data), dtype={"col1": "category"}).dtypes  Specifying ``dtype='category'`` will result in an unordered ``Categorical``
whose ``categories`` are the unique values observed in the data. For more
control on the categories and order, create a
`~pandas.api.types.CategoricalDtype` ahead of time, and pass that for
that column's ``dtype``.  .. ipython:: python  from pandas.api.types import CategoricalDtype  dtype = CategoricalDtype(["d", "c", "b", "a"], ordered=True)  pd.read_csv(StringIO(data), dtype={"col1": dtype}).dtypes  When using ``dtype=CategoricalDtype``, "unexpected" values outside of
``dtype.categories`` are treated as missing values.  .. ipython:: python  dtype = CategoricalDtype(["a", "b", "d"])  # No 'c'  pd.read_csv(StringIO(data), dtype={"col1": dtype}).col1  This matches the behavior of `Categorical.set_categories`.  .. note::  With ``dtype='category'``, the resulting categories will always be parsed  as strings (object dtype). If the categories are numeric they can be  converted using the `to_numeric` function, or as appropriate, another  converter such as `to_datetime`.  When ``dtype`` is a ``CategoricalDtype`` with homogeneous ``categories`` (  all numeric, all datetimes, etc.), the conversion is done automatically.  .. ipython:: python  df = pd.read_csv(StringIO(data), dtype="category")  df.dtypes  df["col3"]  new_categories = pd.to_numeric(df["col3"].cat.categories)  df["col3"] = df["col3"].cat.rename_categories(new_categories)  df["col3"]  Naming and using columns
'''  .. _io.headers:  Handling column names
+++  A file may or may not have a header row. pandas assumes the first row should be
used as the column names:  .. ipython:: python  data = "a,b,c\n1,2,3\n4,5,6\n7,8,9"  print(data)  pd.read_csv(StringIO(data))  By specifying the ``names`` argument in conjunction with ``header`` you can
indicate other names to use and whether or not to throw away the header row (if
any):  .. ipython:: python  print(data)  pd.read_csv(StringIO(data), names=["foo", "bar", "baz"], header=0)  pd.read_csv(StringIO(data), names=["foo", "bar", "baz"], header=None)  If the header is in a row other than the first, pass the row number to
``header``. This will skip the preceding rows:  .. ipython:: python  data = "skip this skip it\na,b,c\n1,2,3\n4,5,6\n7,8,9"  pd.read_csv(StringIO(data), header=1)  .. note::  Default behavior is to infer the column names: if no names are  passed the behavior is identical to ``header=0`` and column names  are inferred from the first non-blank line of the file, if column  names are passed explicitly then the behavior is identical to  ``header=None``.  .. _io.dupe_names:  Duplicate names parsing
'''  If the file or header contains duplicate names, pandas will by default
distinguish between them so as to prevent overwriting data:  .. ipython:: python  data = "a,b,a\n0,1,2\n3,4,5"  pd.read_csv(StringIO(data))  There is no more duplicate data because duplicate columns 'X', ..., 'X' become
'X', 'X.1', ..., 'X.N'.  .. _io.usecols:  Filtering columns (``usecols``)
+++  The ``usecols`` argument allows you to select any subset of the columns in a
file, either using the column names, position numbers or a callable:  .. ipython:: python  data = "a,b,c,d\n1,2,3,foo\n4,5,6,bar\n7,8,9,baz"  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data), usecols=["b", "d"])  pd.read_csv(StringIO(data), usecols=[0, 2, 3])  pd.read_csv(StringIO(data), usecols=lambda x: x.upper() in ["A", "C"])  The ``usecols`` argument can also be used to specify which columns not to

use in the final result:  .. ipython:: python  pd.read_csv(StringIO(data), usecols=lambda x: x not in ["a", "c"])  In this case, the callable is specifying that we exclude the "a" and "c"
columns from the output.  Comments and empty lines
'''  .. _io.skiplines:  Ignoring line comments and empty lines
+++  If the ``comment`` parameter is specified, then completely commented lines will
be ignored. By default, completely blank lines will be ignored as well.  .. ipython:: python  data = "\na,b,c\n  \n# commented line\n1,2,3\n\n4,5,6"  print(data)  pd.read_csv(StringIO(data), comment="#")  If ``skip_blank_lines=False``, then ``read_csv`` will not ignore blank lines:  .. ipython:: python  data = "a,b,c\n\n1,2,3\n\n\n4,5,6"  pd.read_csv(StringIO(data), skip_blank_lines=False)  .. warning::  The presence of ignored lines might create ambiguities involving line numbers;  the parameter ``header`` uses row numbers (ignoring commented/empty  lines), while ``skiprows`` uses line numbers (including commented/empty lines):  .. ipython:: python  data = "#comment\na,b,c\nA,B,C\n1,2,3"  pd.read_csv(StringIO(data), comment="#", header=1)  data = "A,B,C\n#comment\na,b,c\n1,2,3"  pd.read_csv(StringIO(data), comment="#", skiprows=2)  If both ``header`` and ``skiprows`` are specified, ``header`` will be  relative to the end of ``skiprows``. For example:  .. ipython:: python  data = (  "# empty\n"  "# second empty line\n"  "# third emptyline\n"  "X,Y,Z\n"  "1,2,3\n"  "A,B,C\n"  "1,2.,4.\n"  "5.,NaN,10.0\n"  )  print(data)  pd.read_csv(StringIO(data), comment="#", skiprows=4, header=1)  .. _io.comments:  Comments
+++  Sometimes comments or meta data may be included in a file:  .. ipython:: python  data = (  "ID,level,category\n"  "Patient1,123000,x # really unpleasant\n"  "Patient2,23000,y # wouldn't take his medicine\n"  "Patient3,1234018,z # awesome"  )  with open("tmp.csv", "w") as fh:  fh.write(data)  .. ipython:: python  print(open("tmp.csv").read())  By default, the parser includes the comments in the output:  .. ipython:: python  df = pd.read_csv("tmp.csv")  df  We can suppress the comments using the ``comment`` keyword:  .. ipython:: python  df = pd.read_csv("tmp.csv", comment="#")  df  .. ipython:: python  os.remove("tmp.csv")  .. _io.unicode:  Dealing with Unicode data
'''  The ``encoding`` argument should be used for encoded unicode data, which will
result in byte strings being decoded to unicode in the result:  .. ipython:: python  from io import BytesIO  data = b"word,length\n" b"Tr\xc3\xa4umen,7\n" b"Gr\xc3\xbc\xc3\x9fe,5"  data = data.decode("utf8").encode("latin-1")  df = pd.read_csv(BytesIO(data), encoding="latin-1")  df  df["word"][1]  Some formats which encode all characters as multiple bytes, like UTF-16, won't
parse correctly at all without specifying the encoding. `Full list of Python
standard encodings
<https://docs.python.org/3/library/codecs.html#standard-encodings>`_.  .. _io.index_col:  Index columns and trailing delimiters
'''  If a file has one more column of data than the number of column names, the
first column will be used as the ``DataFrame``'s row names:  .. ipython:: python  data = "a,b,c\n4,apple,bat,5.7\n8,orange,cow,10"  pd.read_csv(StringIO(data))  .. ipython:: python  data = "index,a,b,c\n4,apple,bat,5.7\n8,orange,cow,10"  pd.read_csv(StringIO(data), index_col=0)  Ordinarily, you can achieve this behavior using the ``index_col`` option.  There are some exception cases when a file has been prepared with delimiters at
the end of each data line, confusing the parser. To explicitly disable the
index column inference and discard the last column, pass ``index_col=False``:  .. ipython:: python  data = "a,b,c\n4,apple,bat,\n8,orange,cow,"  print(data)  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data), index_col=False)  If a subset of data is being parsed using the ``usecols`` option, the

``index_col`` specification is based on that subset, not the original data.  .. ipython:: python  data = "a,b,c\n4,apple,bat,\n8,orange,cow,"  print(data)  pd.read_csv(StringIO(data), usecols=["b", "c"])  pd.read_csv(StringIO(data), usecols=["b", "c"], index_col=0)  .. _io.parse_dates:  Date Handling
'''  Specifying date columns
+++  To better facilitate working with datetime data, `read_csv`
uses the keyword arguments ``parse_dates`` and ``date_parser``
to allow users to specify a variety of columns and date/time formats to turn the
input text data into ``datetime`` objects.  The simplest case is to just pass in ``parse_dates=True``:  .. ipython:: python  with open("foo.csv", mode="w") as f:  f.write("date,A,B,C\n20090101,a,1,2\n20090102,b,3,4\n20090103,c,4,5")  # Use a column as an index, and parse it as dates.  df = pd.read_csv("foo.csv", index_col=0, parse_dates=True)  df  # These are Python datetime objects  df.index  It is often the case that we may want to store date and time data separately,
or store various date fields separately. the ``parse_dates`` keyword can be
used to specify a combination of columns to parse the dates and/or times from.  You can specify a list of column lists to ``parse_dates``, the resulting date
columns will be prepended to the output (so as to not affect the existing column
order) and the new column names will be the concatenation of the component
column names:  .. ipython:: python  data = (  "KORD,19990127, 1900, 1800, 0.8100\n"  "KORD,19990127, 2000, 1900, 0.0100\n"  "KORD,19990127, 2100, 2000, -0.5900\n"  "KORD,19990127, 2100, 2100, -0.9900\n"  "KORD,19990127, 2200, 2100, -0.5900\n"  "KORD,19990127, 2300, 2200, -0.5900"  )  with open("tmp.csv", "w") as fh:  fh.write(data)  df = pd.read_csv("tmp.csv", header=None, parse_dates=[[1, 2], [1, 3]])  df  By default the parser removes the component date columns, but you can choose
to retain them via the ``keep_date_col`` keyword:  .. ipython:: python  df = pd.read_csv(  "tmp.csv", header=None, parse_dates=[[1, 2], [1, 3]], keep_date_col=True  )  df  Note that if you wish to combine multiple columns into a single date column, a
nested list must be used. In other words, ``parse_dates=[1, 2]`` indicates that
the second and third columns should each be parsed as separate date columns
while ``parse_dates=[[1, 2]]`` means the two columns should be parsed into a
single column.  You can also use a dict to specify custom name columns:  .. ipython:: python  date_spec = {"nominal": [1, 2], "actual": [1, 3]}  df = pd.read_csv("tmp.csv", header=None, parse_dates=date_spec)  df  It is important to remember that if multiple text columns are to be parsed into
a single date column, then a new column is prepended to the data. The ``index_col``
specification is based off of this new set of columns rather than the original
data columns:  .. ipython:: python  date_spec = {"nominal": [1, 2], "actual": [1, 3]}  df = pd.read_csv(  "tmp.csv", header=None, parse_dates=date_spec, index_col=0  )  # index is the nominal column  df  .. note::  If a column or index contains an unparsable date, the entire column or  index will be returned unaltered as an object data type. For non-standard  datetime parsing, use `to_datetime` after ``pd.read_csv``.  .. note::  read_csv has a fast_path for parsing datetime strings in iso8601 format,  e.g "2000-01-01T0002+00:00" and similar variations. If you can arrange  for your data to store datetimes in this format, load times will be  significantly faster, ~20x has been observed.  Date parsing functions
+++  Finally, the parser allows you to specify a custom ``date_parser`` function to
take full advantage of the flexibility of the date parsing API:  .. ipython:: python  df = pd.read_csv(  "tmp.csv", header=None, parse_dates=date_spec, date_parser=pd.to_datetime  )  df  pandas will try to call the ``date_parser`` function in three different ways. If

an exception is raised, the next one is tried:  1. ``date_parser`` is first called with one or more arrays as arguments,  as defined using ``parse_dates`` (e.g., ``date_parser(['2013', '2013'], ['1', '2'])``).  2. If #1 fails, ``date_parser`` is called with all the columns  concatenated row-wise into a single array (e.g., ``date_parser(['2013 1', '2013 2'])``).  Note that performance-wise, you should try these methods of parsing dates in order:  1. If you know the format, use ``pd.to_datetime()``:  ``date_parser=lambda x: pd.to_datetime(x, format=...)``.  2. If you have a really non-standard format, use a custom ``date_parser`` function.  For optimal performance, this should be vectorized, i.e., it should accept arrays  as arguments.  .. ipython:: python  os.remove("tmp.csv")  .. _io.csv.mixed_timezones:  Parsing a CSV with mixed timezones
+++  pandas cannot natively represent a column or index with mixed timezones. If your CSV
file contains columns with a mixture of timezones, the default result will be
an object-dtype column with strings, even with ``parse_dates``.  .. ipython:: python  content = """\  a  2000-01-01T000+05:00  2000-01-01T000+06:00"""  df = pd.read_csv(StringIO(content), parse_dates=["a"])  df["a"]  To parse the mixed-timezone values as a datetime column, pass a partially-applied
`to_datetime` with ``utc=True`` as the ``date_parser``.  .. ipython:: python  df = pd.read_csv(  StringIO(content),  parse_dates=["a"],  date_parser=lambda col: pd.to_datetime(col, utc=True),  )  df["a"]  .. _io.dayfirst:  Inferring datetime format
+++  Here are some examples of datetime strings that can be guessed (all
representing December 30th, 2011 at 000):  * "20111230"
* "2011/12/30"
* "20111230 000"
* "12/30/2011 000"
* "30/Dec/2011 000"
* "30/December/2011 000"  Note that format inference is sensitive to ``dayfirst``.  With
``dayfirst=True``, it will guess "01/12/2011" to be December 1st. With
``dayfirst=False`` (default) it will guess "01/12/2011" to be January 12th.  If you try to parse a column of date strings, pandas will attempt to guess the format
from the first non-NaN element, and will then parse the rest of the column with that
format. If pandas fails to guess the format (for example if your first string is
``'01 December US/Pacific 2000'``), then a warning will be raised and each
row will be parsed individually by ``dateutil.parser.parse``. The safest
way to parse dates is to explicitly set ``format=``.  .. ipython:: python  df = pd.read_csv(  "foo.csv",  index_col=0,  parse_dates=True,  )  df  In the case that you have mixed datetime formats within the same column, you can
pass  ``format='mixed'``  .. ipython:: python  data = io.StringIO("date\n12 Jan 2000\n2000-01-13\n")  df = pd.read_csv(data)  df['date'] = pd.to_datetime(df['date'], format='mixed')  df  or, if your datetime formats are all ISO8601 (possibly not identically-formatted):  .. ipython:: python  data = io.StringIO("date\n2020-01-01\n2020-01-01 03:00\n")  df = pd.read_csv(data)  df['date'] = pd.to_datetime(df['date'], format='ISO8601')  df  .. ipython:: python  os.remove("foo.csv")  International date formats
+++  While US date formats tend to be MM/DD/YYY, many international formats use
DD/MM/YYY instead. For convenience, a ``dayfirst`` keyword is provided:  .. ipython:: python  data = "date,value,cat\n1/6/2000,5,a\n2/6/2000,10,b\n3/6/2000,15,c"  print(data)  with open("tmp.csv", "w") as fh:  fh.write(data)  pd.read_csv("tmp.csv", parse_dates=[0])  pd.read_csv("tmp.csv", dayfirst=True, parse_dates=[0])  .. ipython:: python  os.remove("tmp.csv")  Writing CSVs to binary file objects
+++  .. versionadded:: 1.2.0  ``df.to_csv(..., mode="wb")`` allows writing a CSV to a file object
opened binary mode. In most cases, it is not necessary to specify
``mode`` as Pandas will auto-detect whether the file object is

opened in text or binary mode.  .. ipython:: python  import io  data = pd.DataFrame([0, 1, 2])  buffer = io.BytesIO()  data.to_csv(buffer, encoding="utf-8", compression="gzip")  .. _io.float_precision:  Specifying method for floating-point conversion
'''  The parameter ``float_precision`` can be specified in order to use
a specific floating-point converter during parsing with the C engine.
The options are the ordinary converter, the high-precision converter, and
the round-trip converter (which is guaranteed to round-trip values after
writing to a file). For example:  .. ipython:: python  val = "0.3066101993807095471566981359501369297504425048828125"  data = "a,b,c\n1,2,{0}".format(val)  abs(  pd.read_csv(  StringIO(data),  engine="c",  float_precision=None,  )["c"][0] - float(val)  )  abs(  pd.read_csv(  StringIO(data),  engine="c",  float_precision="high",  )["c"][0] - float(val)  )  abs(  pd.read_csv(StringIO(data), engine="c", float_precision="round_trip")["c"][0]  - float(val)  )  .. _io.thousands:  Thousand separators
'''  For large numbers that have been written with a thousands separator, you can
set the ``thousands`` keyword to a string of length 1 so that integers will be parsed
correctly:  By default, numbers with a thousands separator will be parsed as strings:  .. ipython:: python  data = (  "ID|level|category\n"  "Patient1|123,000|x\n"  "Patient2|23,000|y\n"  "Patient3|1,234,018|z"  )  with open("tmp.csv", "w") as fh:  fh.write(data)  df = pd.read_csv("tmp.csv", sep="|")  df  df.level.dtype  The ``thousands`` keyword allows integers to be parsed correctly:  .. ipython:: python  df = pd.read_csv("tmp.csv", sep="|", thousands=",")  df  df.level.dtype  .. ipython:: python  os.remove("tmp.csv")  .. _io.na_values:  NA values
'''  To control which values are parsed as missing values (which are signified by
``NaN``), specify a string in ``na_values``. If you specify a list of strings,
then all values in it are considered to be missing values. If you specify a
number (a ``float``, like ``5.0`` or an ``integer`` like ``5``), the
corresponding equivalent values will also imply a missing value (in this case
effectively ``[5.0, 5]`` are recognized as ``NaN``).  To completely override the default values that are recognized as missing, specify ``keep_default_na=False``.  .. _io.navaluesconst:  The default ``NaN`` recognized values are ``['-1.#IND', '1.#QNAN', '1.#IND', '-1.#QNAN', '#N/A N/A', '#N/A', 'N/A',
'n/a', 'NA', '<NA>', '#NA', 'NULL', 'null', 'NaN', '-NaN', 'nan', '-nan', 'None', '']``.  Let us consider some examples:  .. code-block:: python  pd.read_csv("path_to_file.csv", na_values=[5])  In the example above ``5`` and ``5.0`` will be recognized as ``NaN``, in
addition to the defaults. A string will first be interpreted as a numerical
``5``, then as a ``NaN``.  .. code-block:: python  pd.read_csv("path_to_file.csv", keep_default_na=False, na_values=[""])  Above, only an empty field will be recognized as ``NaN``.  .. code-block:: python  pd.read_csv("path_to_file.csv", keep_default_na=False, na_values=["NA", "0"])  Above, both ``NA`` and ``0`` as strings are ``NaN``.  .. code-block:: python  pd.read_csv("path_to_file.csv", na_values=["Nope"])  The default values, in addition to the string ``"Nope"`` are recognized as
``NaN``.  .. _io.infinity:  Infinity
'''  ``inf`` like values will be parsed as ``np.inf`` (positive infinity), and ``-inf`` as ``-np.inf`` (negative infinity).
These will ignore the case of the value, meaning ``Inf``, will also be parsed as ``np.inf``.  .. _io.boolean:  Boolean values
'''  The common values ``True``, ``False``, ``TRUE``, and ``FALSE`` are all
recognized as boolean. Occasionally you might want to recognize other values
as being boolean. To do this, use the ``true_values`` and ``false_values``

options as follows:  .. ipython:: python  data = "a,b,c\n1,Yes,2\n3,No,4"  print(data)  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data), true_values=["Yes"], false_values=["No"])  .. _io.bad_lines:  Handling "bad" lines
'''  Some files may have malformed lines with too few fields or too many. Lines with
too few fields will have NA values filled in the trailing fields. Lines with
too many fields will raise an error by default:  .. ipython:: python  data = "a,b,c\n1,2,3\n4,5,6,7\n8,9,10"  pd.read_csv(StringIO(data))  You can elect to skip bad lines:  .. code-block:: ipython  In [29]: pd.read_csv(StringIO(data), on_bad_lines="warn")  Skipping line 3: expected 3 fields, saw 4  Out[29]:  a  b  c  0  1  2  3  1  8  9  10  Or pass a callable function to handle the bad line if ``engine="python"``.
The bad line will be a list of strings that was split by the ``sep``:  .. code-block:: ipython  In [29]: external_list = []  In [30]: def bad_lines_func(line):  ...:  external_list.append(line)  ...:  return line[-3:]  In [31]: pd.read_csv(StringIO(data), on_bad_lines=bad_lines_func, engine="python")  Out[31]:  a  b  c  0  1  2  3  1  5  6  7  2  8  9  10  In [32]: external_list  Out[32]: [4, 5, 6, 7]  .. versionadded:: 1.4.0  Note that the callable function will handle only a line with too many fields.
Bad lines caused by other errors will be silently skipped.  For example:  .. code-block:: ipython  def bad_lines_func(line):  print(line)  data = 'name,type\nname a,a is of type a\nname b,"b\" is of type b"'  data  pd.read_csv(data, on_bad_lines=bad_lines_func, engine="python")  The line was not processed in this case, as a "bad line" here is caused by an escape character.  You can also use the ``usecols`` parameter to eliminate extraneous column
data that appear in some lines but not others:  .. code-block:: ipython  In [33]: pd.read_csv(StringIO(data), usecols=[0, 1, 2])  Out[33]:  a  b  c  0  1  2  3  1  4  5  6  2  8  9  10  In case you want to keep all data including the lines with too many fields, you can
specify a sufficient number of ``names``. This ensures that lines with not enough
fields are filled with ``NaN``.  .. code-block:: ipython  In [34]: pd.read_csv(StringIO(data), names=['a', 'b', 'c', 'd'])  Out[34]:  a  b  c  d  0  1  2  3  NaN  1  4  5  6  7  2  8  9  10  NaN  .. _io.dialect:  Dialect
'''  The ``dialect`` keyword gives greater flexibility in specifying the file format.
By default it uses the Excel dialect but you can specify either the dialect name
or a `python:csv.Dialect` instance.  Suppose you had data with unenclosed quotes:  .. ipython:: python  data = "label1,label2,label3\n" 'index1,"a,c,e\n' "index2,b,d,f"  print(data)  By default, ``read_csv`` uses the Excel dialect and treats the double quote as
the quote character, which causes it to fail when it finds a newline before it
finds the closing double quote.  We can get around this using ``dialect``:  .. ipython:: python  import csv  dia = csv.excel()  dia.quoting = csv.QUOTE_NONE  pd.read_csv(StringIO(data), dialect=dia)  All of the dialect options can be specified separately by keyword arguments:  .. ipython:: python  data = "a,b,c~1,2,3~4,5,6"  pd.read_csv(StringIO(data), lineterminator="~")  Another common dialect option is ``skipinitialspace``, to skip any whitespace
after a delimiter:  .. ipython:: python  data = "a, b, c\n1, 2, 3\n4, 5, 6"  print(data)  pd.read_csv(StringIO(data), skipinitialspace=True)  The parsers make every attempt to "do the right thing" and not be fragile. Type
inference is a pretty big deal. If a column can be coerced to integer dtype
without altering the contents, the parser will do so. Any non-numeric
columns will come through as object dtype as with the rest of pandas objects.  .. _io.quoting:  Quoting and Escape Characters

'''  Quotes (and other escape characters) in embedded fields can be handled in any
number of ways. One way is to use backslashes; to properly parse this data, you
should pass the ``escapechar`` option:  .. ipython:: python  data = 'a,b\n"hello, \\"Bob\\", nice to see you",5'  print(data)  pd.read_csv(StringIO(data), escapechar="\\")  .. _io.fwf_reader:
.. _io.fwf:  Files with fixed width columns
'''  While `read_csv` reads delimited data, the `read_fwf` function works
with data files that have known and fixed column widths. The function parameters
to ``read_fwf`` are largely the same as ``read_csv`` with two extra parameters, and
a different usage of the ``delimiter`` parameter:  * ``colspecs``: A list of pairs (tuples) giving the extents of the  fixed-width fields of each line as half-open intervals (i.e.,  [from, to[ ).  String value 'infer' can be used to instruct the parser to try detecting  the column specifications from the first 100 rows of the data. Default  behavior, if not specified, is to infer.
* ``widths``: A list of field widths which can be used instead of 'colspecs'  if the intervals are contiguous.
* ``delimiter``: Characters to consider as filler characters in the fixed-width file.  Can be used to specify the filler character of the fields  if it is not spaces (e.g., '~').  Consider a typical fixed-width data file:  .. ipython:: python  data1 = (  "id8141  360.242940  149.910199  11950.7\n"  "id1594  444.953632  166.985655  11788.4\n"  "id1849  364.136849  183.628767  11806.2\n"  "id1230  413.836124  184.375703  11916.8\n"  "id1948  502.953953  173.237159  12468.3"  )  with open("bar.csv", "w") as f:  f.write(data1)  In order to parse this file into a ``DataFrame``, we simply need to supply the
column specifications to the ``read_fwf`` function along with the file name:  .. ipython:: python  # Column specifications are a list of half-intervals  colspecs = [(0, 6), (8, 20), (21, 33), (34, 43)]  df = pd.read_fwf("bar.csv", colspecs=colspecs, header=None, index_col=0)  df  Note how the parser automatically picks column names X.<column number> when
``header=None`` argument is specified. Alternatively, you can supply just the
column widths for contiguous columns:  .. ipython:: python  # Widths are a list of integers  widths = [6, 14, 13, 10]  df = pd.read_fwf("bar.csv", widths=widths, header=None)  df  The parser will take care of extra white spaces around the columns
so it's ok to have extra separation between the columns in the file.  By default, ``read_fwf`` will try to infer the file's ``colspecs`` by using the
first 100 rows of the file. It can do it only in cases when the columns are
aligned and correctly separated by the provided ``delimiter`` (default delimiter
is whitespace).  .. ipython:: python  df = pd.read_fwf("bar.csv", header=None, index_col=0)  df  ``read_fwf`` supports the ``dtype`` parameter for specifying the types of
parsed columns to be different from the inferred type.  .. ipython:: python  pd.read_fwf("bar.csv", header=None, index_col=0).dtypes  pd.read_fwf("bar.csv", header=None, dtype={2: "object"}).dtypes  .. ipython:: python  os.remove("bar.csv")  Indexes
'''  Files with an "implicit" index column
+++  Consider a file with one less entry in the header than the number of data
column:  .. ipython:: python  data = "A,B,C\n20090101,a,1,2\n20090102,b,3,4\n20090103,c,4,5"  print(data)  with open("foo.csv", "w") as f:  f.write(data)  In this special case, ``read_csv`` assumes that the first column is to be used
as the index of the ``DataFrame``:  .. ipython:: python  pd.read_csv("foo.csv")  Note that the dates weren't automatically parsed. In that case you would need

to do as before:  .. ipython:: python  df = pd.read_csv("foo.csv", parse_dates=True)  df.index  .. ipython:: python  os.remove("foo.csv")  Reading an index with a ``MultiIndex``
+++  .. _io.csv_multiindex:  Suppose you have data indexed by two columns:  .. ipython:: python  data = 'year,indiv,zit,xit\n1977,"A",1.2,.6\n1977,"B",1.5,.5'  print(data)  with open("mindex_ex.csv", mode="w") as f:  f.write(data)  The ``index_col`` argument to ``read_csv`` can take a list of
column numbers to turn multiple columns into a ``MultiIndex`` for the index of the
returned object:  .. ipython:: python  df = pd.read_csv("mindex_ex.csv", index_col=[0, 1])  df  df.loc[1977]  .. ipython:: python  os.remove("mindex_ex.csv")  .. _io.multi_index_columns:  Reading columns with a ``MultiIndex``
+++  By specifying list of row locations for the ``header`` argument, you
can read in a ``MultiIndex`` for the columns. Specifying non-consecutive
rows will skip the intervening rows.  .. ipython:: python  from pandas._testing import makeCustomDataframe as mkdf  df = mkdf(5, 3, r_idx_nlevels=2, c_idx_nlevels=4)  df.to_csv("mi.csv")  print(open("mi.csv").read())  pd.read_csv("mi.csv", header=[0, 1, 2, 3], index_col=[0, 1])  ``read_csv`` is also able to interpret a more common format
of multi-columns indices.  .. ipython:: python  data = ",a,a,a,b,c,c\n,q,r,s,t,u,v\none,1,2,3,4,5,6\ntwo,7,8,9,10,11,12"  print(data)  with open("mi2.csv", "w") as fh:  fh.write(data)  pd.read_csv("mi2.csv", header=[0, 1], index_col=0)  .. note::  If an ``index_col`` is not specified (e.g. you don't have an index, or wrote it  with ``df.to_csv(..., index=False)``, then any ``names`` on the columns index will  be *lost*.  .. ipython:: python  os.remove("mi.csv")  os.remove("mi2.csv")  .. _io.sniff:  Automatically "sniffing" the delimiter
'''  ``read_csv`` is capable of inferring delimited (not necessarily
comma-separated) files, as pandas uses the `python:csv.Sniffer`
class of the csv module. For this, you have to specify ``sep=None``.  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 4))  df.to_csv("tmp.csv", sep="|")  df.to_csv("tmp2.csv", sep=":")  pd.read_csv("tmp2.csv", sep=None, engine="python")  .. ipython:: python  os.remove("tmp2.csv")  .. _io.multiple_files:  Reading multiple files to create a single DataFrame
'''  It's best to use `~pandas.concat` to combine multiple files.
See the `cookbook<cookbook.csv.multiple_files>` for an example.  .. _io.chunking:  Iterating through files chunk by chunk
'''  Suppose you wish to iterate through a (potentially very large) file lazily
rather than reading the entire file into memory, such as the following:  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 4))  df.to_csv("tmp.csv", sep="|")  table = pd.read_csv("tmp.csv", sep="|")  table  By specifying a ``chunksize`` to ``read_csv``, the return
value will be an iterable object of type ``TextFileReader``:  .. ipython:: python  with pd.read_csv("tmp.csv", sep="|", chunksize=4) as reader:  reader  for chunk in reader:  print(chunk)  .. versionchanged:: 1.2  ``read_csv/json/sas`` return a context-manager when iterating through a file.  Specifying ``iterator=True`` will also return the ``TextFileReader`` object:  .. ipython:: python  with pd.read_csv("tmp.csv", sep="|", iterator=True) as reader:  reader.get_chunk(5)  .. ipython:: python  os.remove("tmp.csv")  Specifying the parser engine
'''  Pandas currently supports three engines, the C engine, the python engine, and an experimental
pyarrow engine (requires the ``pyarrow`` package). In general, the pyarrow engine is fastest
on larger workloads and is equivalent in speed to the C engine on most other workloads.
The python engine tends to be slower than the pyarrow and C engines on most workloads. However,

the pyarrow engine is much less robust than the C engine, which lacks a few features compared to the
Python engine.  Where possible, pandas uses the C parser (specified as ``engine='c'``), but it may fall
back to Python if C-unsupported options are specified.  Currently, options unsupported by the C and pyarrow engines include:  * ``sep`` other than a single character (e.g. regex separators)
* ``skipfooter``
* ``sep=None`` with ``delim_whitespace=False``  Specifying any of the above options will produce a ``ParserWarning`` unless the
python engine is selected explicitly using ``engine='python'``.  Options that are unsupported by the pyarrow engine which are not covered by the list above include:  * ``float_precision``
* ``chunksize``
* ``comment``
* ``nrows``
* ``thousands``
* ``memory_map``
* ``dialect``
* ``on_bad_lines``
* ``delim_whitespace``
* ``quoting``
* ``lineterminator``
* ``converters``
* ``decimal``
* ``iterator``
* ``dayfirst``
* ``infer_datetime_format``
* ``verbose``
* ``skipinitialspace``
* ``low_memory``  Specifying these options with ``engine='pyarrow'`` will raise a ``ValueError``.  .. _io.remote:  Reading/writing remote files
'''  You can pass in a URL to read or write remote files to many of pandas' IO
functions - the following example shows reading a CSV file:  .. code-block:: python  df = pd.read_csv("https://download.bls.gov/pub/time.series/cu/cu.item", sep="\t")  .. versionadded:: 1.3.0  A custom header can be sent alongside HTTP(s) requests by passing a dictionary
of header key value mappings to the ``storage_options`` keyword argument as shown below:  .. code-block:: python  headers = {"User-Agent": "pandas"}  df = pd.read_csv(  "https://download.bls.gov/pub/time.series/cu/cu.item",  sep="\t",  storage_options=headers  )  All URLs which are not local files or HTTP(s) are handled by
`fsspec`_, if installed, and its various filesystem implementations
(including Amazon S3, Google Cloud, SSH, FTP, webHDFS...).
Some of these implementations will require additional packages to be
installed, for example
S3 URLs require the `s3fs
<https://pypi.org/project/s3fs/>`_ library:  .. code-block:: python  df = pd.read_json("s3://pandas-test/adatafile.json")  When dealing with remote storage systems, you might need
extra configuration with environment variables or config files in
special locations. For example, to access data in your S3 bucket,
you will need to define credentials in one of the several ways listed in
the `S3Fs documentation
<https://s3fs.readthedocs.io/en/latest/#credentials>`_. The same is true
for several of the storage backends, and you should follow the links
at `fsimpl1`_ for implementations built into ``fsspec`` and `fsimpl2`_
for those not included in the main ``fsspec``
distribution.  You can also pass parameters directly to the backend driver. For example,
if you do *not* have S3 credentials, you can still access public data by
specifying an anonymous connection, such as  .. versionadded:: 1.2.0  .. code-block:: python  pd.read_csv(  "s3://ncei-wcsd-archive/data/processed/SH1305/18kHz/SaKe2013"  "-D20130523-T080854_to_SaKe2013-D20130523-T085643.csv",  storage_options={"anon": True},  )  ``fsspec`` also allows complex URLs, for accessing data in compressed
archives, local caching of files, and more. To locally cache the above
example, you would modify the call to  .. code-block:: python  pd.read_csv(  "simplecache://ncei-wcsd-archive/data/processed/SH1305/18kHz/"  "SaKe2013-D20130523-T080854_to_SaKe2013-D20130523-T085643.csv",  storage_options={"s3": {"anon": True}},  )  where we specify that the "anon" parameter is meant for the "s3" part of
the implementation, not to the caching implementation. Note that this caches to a temporary
directory for the duration of the session only, but you can also specify

a permanent store.  .. _fsspec: https://filesystem-spec.readthedocs.io/en/latest/
.. _fsimpl1: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
.. _fsimpl2: https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations  Writing out data
'''  .. _io.store_in_csv:  Writing to CSV format
+++  The ``Series`` and ``DataFrame`` objects have an instance method ``to_csv`` which
allows storing the contents of the object as a comma-separated-values file. The
function takes a number of arguments. Only the first is required.  * ``path_or_buf``: A string path to the file to write or a file object.  If a file object it must be opened with ``newline=''``
* ``sep`` : Field delimiter for the output file (default ",")
* ``na_rep``: A string representation of a missing value (default '')
* ``float_format``: Format string for floating point numbers
* ``columns``: Columns to write (default None)
* ``header``: Whether to write out the column names (default True)
* ``index``: whether to write row (index) names (default True)
* ``index_label``: Column label(s) for index column(s) if desired. If None  (default), and ``header`` and ``index`` are True, then the index names are  used. (A sequence should be given if the ``DataFrame`` uses MultiIndex).
* ``mode`` : Python write mode, default 'w'
* ``encoding``: a string representing the encoding to use if the contents are  non-ASCII, for Python versions prior to 3
* ``lineterminator``: Character sequence denoting line end (default ``os.linesep``)
* ``quoting``: Set quoting rules as in csv module (default csv.QUOTE_MINIMAL). Note that if you have set a ``float_format`` then floats are converted to strings and csv.QUOTE_NONNUMERIC will treat them as non-numeric
* ``quotechar``: Character used to quote fields (default '"')
* ``doublequote``: Control quoting of ``quotechar`` in fields (default True)
* ``escapechar``: Character used to escape ``sep`` and ``quotechar`` when  appropriate (default None)
* ``chunksize``: Number of rows to write at a time
* ``date_format``: Format string for datetime objects  Writing a formatted string
+++  .. _io.formatting:  The ``DataFrame`` object has an instance method ``to_string`` which allows control
over the string representation of the object. All arguments are optional:  * ``buf`` default None, for example a StringIO object
* ``columns`` default None, which columns to write
* ``col_space`` default None, minimum width of each column.
* ``na_rep`` default ``NaN``, representation of NA value
* ``formatters`` default None, a dictionary (by column) of functions each of  which takes a single argument and returns a formatted string
* ``float_format`` default None, a function which takes a single (float)  argument and returns a formatted string; to be applied to floats in the  ``DataFrame``.
* ``sparsify`` default True, set to False for a ``DataFrame`` with a hierarchical  index to print every MultiIndex key at each row.
* ``index_names`` default True, will print the names of the indices
* ``index`` default True, will print the index (ie, row labels)
* ``header`` default True, will print the column labels
* ``justify`` default ``left``, will print column headers left- or  right-justified  The ``Series`` object also has a ``to_string`` method, but with only the ``buf``,
``na_rep``, ``float_format`` arguments. There is also a ``length`` argument
which, if set to ``True``, will additionally output the length of the Series.  .. _io.json:

Read and write ``JSON`` format files and strings.  .. _io.json_writer:  Writing JSON
'''  A ``Series`` or ``DataFrame`` can be converted to a valid JSON string. Use ``to_json``
with optional parameters:  * ``path_or_buf`` : the pathname or buffer to write the output  This can be ``None`` in which case a JSON string is returned
* ``orient`` :  ``Series``:  * default is ``index``  * allowed values are {``split``, ``records``, ``index``}  ``DataFrame``:  * default is ``columns``  * allowed values are {``split``, ``records``, ``index``, ``columns``, ``values``, ``table``}  The format of the JSON string  .. csv-table::  20, 150  ;  ``split``; dict like {index -> [index], columns -> [columns], data -> [values]}  ``records``; list like [{column -> value}, ... , {column -> value}]  ``index``; dict like {index -> {column -> value}}  ``columns``; dict like {column -> {index -> value}}  ``values``; just the values array  ``table``; adhering to the JSON `Table Schema`_  * ``date_format`` : string, type of date conversion, 'epoch' for timestamp, 'iso' for ISO8601.
* ``double_precision`` : The number of decimal places to use when encoding floating point values, default 10.
* ``force_ascii`` : force encoded string to be ASCII, default True.
* ``date_unit`` : The time unit to encode to, governs timestamp and ISO8601 precision. One of 's', 'ms', 'us' or 'ns' for seconds, milliseconds, microseconds and nanoseconds respectively. Default 'ms'.
* ``default_handler`` : The handler to call if an object cannot otherwise be converted to a suitable format for JSON. Takes a single argument, which is the object to convert, and returns a serializable object.
* ``lines`` : If ``records`` orient, then will write each record per line as json.
* ``mode`` : string, writer mode when writing to path. 'w' for write, 'a' for append. Default 'w'  Note ``NaN``'s, ``NaT``'s and ``None`` will be converted to ``null`` and ``datetime`` objects will be converted based on the ``date_format`` and ``date_unit`` parameters.  .. ipython:: python  dfj = pd.DataFrame(np.random.randn(5, 2), columns=list("AB"))  json = dfj.to_json()  json  Orient options
+++  There are a number of different options for the format of the resulting JSON
file / string. Consider the following ``DataFrame`` and ``Series``:  .. ipython:: python  dfjo = pd.DataFrame(  dict(A=range(1, 4), B=range(4, 7), C=range(7, 10)),  columns=list("ABC"),  index=list("xyz"),  )  dfjo  sjo = pd.Series(dict(x=15, y=16, z=17), name="D")  sjo  **Column oriented** (the default for ``DataFrame``) serializes the data as
nested JSON objects with column labels acting as the primary index:  .. ipython:: python  dfjo.to_json(orient="columns")  # Not available for Series  **Index oriented** (the default for ``Series``) similar to column oriented
but the index labels are now primary:  .. ipython:: python  dfjo.to_json(orient="index")  sjo.to_json(orient="index")  **Record oriented** serializes the data to a JSON array of column -> value records,
index labels are not included. This is useful for passing ``DataFrame`` data to plotting
libraries, for example the JavaScript library ``d3.js``:  .. ipython:: python  dfjo.to_json(orient="records")  sjo.to_json(orient="records")  **Value oriented** is a bare-bones option which serializes to nested JSON arrays of
values only, column and index labels are not included:  .. ipython:: python  dfjo.to_json(orient="values")  # Not available for Series  **Split oriented** serializes to a JSON object containing separate entries for
values, index and columns. Name is also included for ``Series``:  .. ipython:: python  dfjo.to_json(orient="split")  sjo.to_json(orient="split")  **Table oriented** serializes to the JSON `Table Schema`_, allowing for the

preservation of metadata including but not limited to dtypes and index names.  .. note::  Any orient option that encodes to a JSON object will not preserve the ordering of  index and column labels during round-trip serialization. If you wish to preserve  label ordering use the ``split`` option as it uses ordered containers.  Date handling
+++  Writing in ISO date format:  .. ipython:: python  dfd = pd.DataFrame(np.random.randn(5, 2), columns=list("AB"))  dfd["date"] = pd.Timestamp("20130101")  dfd = dfd.sort_index(axis=1, ascending=False)  json = dfd.to_json(date_format="iso")  json  Writing in ISO date format, with microseconds:  .. ipython:: python  json = dfd.to_json(date_format="iso", date_unit="us")  json  Epoch timestamps, in seconds:  .. ipython:: python  json = dfd.to_json(date_format="epoch", date_unit="s")  json  Writing to a file, with a date index and a date column:  .. ipython:: python  dfj2 = dfj.copy()  dfj2["date"] = pd.Timestamp("20130101")  dfj2["ints"] = list(range(5))  dfj2["bools"] = True  dfj2.index = pd.date_range("20130101", periods=5)  dfj2.to_json("test.json")  with open("test.json") as fh:  print(fh.read())  Fallback behavior
+++  If the JSON serializer cannot handle the container contents directly it will
fall back in the following manner:  * if the dtype is unsupported (e.g. ``np.complex_``) then the ``default_handler``, if provided, will be called  for each value, otherwise an exception is raised.  * if an object is unsupported it will attempt the following:  * check if the object has defined a ``toDict`` method and call it.  A ``toDict`` method should return a ``dict`` which will then be JSON serialized.  * invoke the ``default_handler`` if one was provided.  * convert the object to a ``dict`` by traversing its contents. However this will often fail  with an ``OverflowError`` or give unexpected results.  In general the best approach for unsupported objects or dtypes is to provide a ``default_handler``.
For example:  .. code-block:: python  >>> DataFrame([1.0, 2.0, complex(1.0, 2.0)]).to_json()  # raises  RuntimeError: Unhandled numpy dtype 15  can be dealt with by specifying a simple ``default_handler``:  .. ipython:: python  pd.DataFrame([1.0, 2.0, complex(1.0, 2.0)]).to_json(default_handler=str)  .. _io.json_reader:  Reading JSON
'''  Reading a JSON string to pandas object can take a number of parameters.
The parser will try to parse a ``DataFrame`` if ``typ`` is not supplied or
is ``None``. To explicitly force ``Series`` parsing, pass ``typ=series``  * ``filepath_or_buffer`` : a **VALID** JSON string or file handle / StringIO. The string could be  a URL. Valid URL schemes include http, ftp, S3, and file. For file URLs, a host  is expected. For instance, a local file could be  file ://localhost/path/to/table.json
* ``typ``  : type of object to recover (series or frame), default 'frame'
* ``orient`` :  Series :  * default is ``index``  * allowed values are {``split``, ``records``, ``index``}  DataFrame  * default is ``columns``  * allowed values are {``split``, ``records``, ``index``, ``columns``, ``values``, ``table``}  The format of the JSON string  .. csv-table::  20, 150  ;  ``split``; dict like {index -> [index], columns -> [columns], data -> [values]}  ``records``; list like [{column -> value}, ... , {column -> value}]  ``index``; dict like {index -> {column -> value}}  ``columns``; dict like {column -> {index -> value}}  ``values``; just the values array  ``table``; adhering to the JSON `Table Schema`_  * ``dtype`` : if True, infer dtypes, if a dict of column to dtype, then use those, if ``False``, then don't infer dtypes at all, default is True, apply only to the data.
* ``convert_axes`` : boolean, try to convert the axes to the proper dtypes, default is ``True``
* ``convert_dates`` : a list of columns to parse for dates; If ``True``, then try to parse date-like columns, default is ``True``.

* ``keep_default_dates`` : boolean, default ``True``. If parsing dates, then parse the default date-like columns.
* ``precise_float`` : boolean, default ``False``. Set to enable usage of higher precision (strtod) function when decoding string to double values. Default (``False``) is to use fast but less precise builtin functionality.
* ``date_unit`` : string, the timestamp unit to detect if converting dates. Default  None. By default the timestamp precision will be detected, if this is not desired  then pass one of 's', 'ms', 'us' or 'ns' to force timestamp precision to  seconds, milliseconds, microseconds or nanoseconds respectively.
* ``lines`` : reads file as one json object per line.
* ``encoding`` : The encoding to use to decode py3 bytes.
* ``chunksize`` : when used in combination with ``lines=True``, return a JsonReader which reads in ``chunksize`` lines per iteration.
* ``engine``: Either ``"ujson"``, the built-in JSON parser, or ``"pyarrow"`` which dispatches to pyarrow's ``pyarrow.json.read_json``.  The ``"pyarrow"`` is only available when ``lines=True``  The parser will raise one of ``ValueError/TypeError/AssertionError`` if the JSON is not parseable.  If a non-default ``orient`` was used when encoding to JSON be sure to pass the same
option here so that decoding produces sensible results, see `Orient Options`_ for an
overview.  Data conversion
+++  The default of ``convert_axes=True``, ``dtype=True``, and ``convert_dates=True``
will try to parse the axes, and all of the data into appropriate types,
including dates. If you need to override specific dtypes, pass a dict to
``dtype``. ``convert_axes`` should only be set to ``False`` if you need to
preserve string-like numbers (e.g. '1', '2') in an axes.  .. note::  Large integer values may be converted to dates if ``convert_dates=True`` and the data and / or column labels appear 'date-like'. The exact threshold depends on the ``date_unit`` specified. 'date-like' means that the column label meets one of the following criteria:  * it ends with ``'_at'``  * it ends with ``'_time'``  * it begins with ``'timestamp'``  * it is ``'modified'``  * it is ``'date'``  .. warning::  When reading JSON data, automatic coercing into dtypes has some quirks:  * an index can be reconstructed in a different order from serialization, that is, the returned order is not guaranteed to be the same as before serialization  * a column that was ``float`` data will be converted to ``integer`` if it can be done safely, e.g. a column of ``1.``  * bool columns will be converted to ``integer`` on reconstruction  Thus there are times where you may want to specify specific dtypes via the ``dtype`` keyword argument.  Reading from a JSON string:  .. ipython:: python  pd.read_json(json)  Reading from a file:  .. ipython:: python  pd.read_json("test.json")  Don't convert any data (but still convert axes and dates):  .. ipython:: python  pd.read_json("test.json", dtype=object).dtypes  Specify dtypes for conversion:  .. ipython:: python  pd.read_json("test.json", dtype={"A": "float32", "bools": "int8"}).dtypes  Preserve string indices:  .. ipython:: python  si = pd.DataFrame(  np.zeros((4, 4)), columns=list(range(4)), index=[str(i) for i in range(4)]  )  si  si.index  si.columns  json = si.to_json()  sij = pd.read_json(json, convert_axes=False)  sij  sij.index  sij.columns  Dates written in nanoseconds need to be read back in nanoseconds:  .. ipython:: python  json = dfj2.to_json(date_unit="ns")  # Try to parse timestamps as milliseconds -> Won't Work  dfju = pd.read_json(json, date_unit="ms")  dfju  # Let pandas detect the correct precision  dfju = pd.read_json(json)  dfju  # Or specify that all timestamps are in nanoseconds  dfju = pd.read_json(json, date_unit="ns")  dfju  .. _io.json_normalize:  Normalization

'''  pandas provides a utility function to take a dict or list of dicts and *normalize* this semi-structured data
into a flat table.  .. ipython:: python  data = [  {"id": 1, "name": {"first": "Coleen", "last": "Volk"}},  {"name": {"given": "Mark", "family": "Regner"}},  {"id": 2, "name": "Faye Raker"},  ]  pd.json_normalize(data)  .. ipython:: python  data = [  {  "state": "Florida",  "shortname": "FL",  "info": {"governor": "Rick Scott"},  "county": [  {"name": "Dade", "population": 12345},  {"name": "Broward", "population": 4000},  {"name": "Palm Beach", "population": 6000},  ],  },  {  "state": "Ohio",  "shortname": "OH",  "info": {"governor": "John Kasich"},  "county": [  {"name": "Summit", "population": 1234},  {"name": "Cuyahoga", "population": 1337},  ],  },  ]  pd.json_normalize(data, "county", ["state", "shortname", ["info", "governor"]])  The max_level parameter provides more control over which level to end normalization.
With max_level=1 the following snippet normalizes until 1st nesting level of the provided dict.  .. ipython:: python  data = [  {  "CreatedBy": {"Name": "User001"},  "Lookup": {  "TextField": "Some text",  "UserField": {"Id": "ID001", "Name": "Name001"},  },  "Image": {"a": "b"},  }  ]  pd.json_normalize(data, max_level=1)  .. _io.jsonl:  Line delimited json
'''  pandas is able to read and write line-delimited json files that are common in data processing pipelines
using Hadoop or Spark.  For line-delimited json files, pandas can also return an iterator which reads in ``chunksize`` lines at a time. This can be useful for large files or to read from a stream.  .. ipython:: python  jsonl = """  {"a": 1, "b": 2}  {"a": 3, "b": 4}  """  df = pd.read_json(jsonl, lines=True)  df  df.to_json(orient="records", lines=True)  # reader is an iterator that returns ``chunksize`` lines each iteration  with pd.read_json(StringIO(jsonl), lines=True, chunksize=1) as reader:  reader  for chunk in reader:  print(chunk)  Line-limited json can also be read using the pyarrow reader by specifying ``engine="pyarrow"``.  .. ipython:: python  from io import BytesIO  df = pd.read_json(BytesIO(jsonl.encode()), lines=True, engine="pyarrow")  df  .. versionadded:: 2.0.0  .. _io.table_schema:  Table schema
'''  `Table Schema`_ is a spec for describing tabular datasets as a JSON
object. The JSON includes information on the field names, types, and
other attributes. You can use the orient ``table`` to build
a JSON string with two fields, ``schema`` and ``data``.  .. ipython:: python  df = pd.DataFrame(  {  "A": [1, 2, 3],  "B": ["a", "b", "c"],  "C": pd.date_range("2016-01-01", freq="d", periods=3),  },  index=pd.Index(range(3), name="idx"),  )  df  df.to_json(orient="table", date_format="iso")  The ``schema`` field contains the ``fields`` key, which itself contains
a list of column name to type pairs, including the ``Index`` or ``MultiIndex``
(see below for a list of types).
The ``schema`` field also contains a ``primaryKey`` field if the (Multi)index
is unique.  The second field, ``data``, contains the serialized data with the ``records``
orient.
The index is included, and any datetimes are ISO 8601 formatted, as required
by the Table Schema spec.  The full list of types supported are described in the Table Schema
spec. This table shows the mapping from pandas types:  === ===
pandas type  Table Schema type
=== ===
int64  integer
float64  number
bool  boolean
datetime64[ns]  datetime
timedelta64[ns] duration
categorical  any
object  str
=== ===  A few notes on the generated table schema:  * The ``schema`` object contains a ``pandas_version`` field. This contains  the version of pandas' dialect of the schema, and will be incremented  with each revision.

* All dates are converted to UTC when serializing. Even timezone naive values,  which are treated as UTC with an offset of 0.  .. ipython:: python  from pandas.io.json import build_table_schema  s = pd.Series(pd.date_range("2016", periods=4))  build_table_schema(s)  * datetimes with a timezone (before serializing), include an additional field  ``tz`` with the time zone name (e.g. ``'US/Central'``).  .. ipython:: python  s_tz = pd.Series(pd.date_range("2016", periods=12, tz="US/Central"))  build_table_schema(s_tz)  * Periods are converted to timestamps before serialization, and so have the  same behavior of being converted to UTC. In addition, periods will contain  and additional field ``freq`` with the period's frequency, e.g. ``'A-DEC'``.  .. ipython:: python  s_per = pd.Series(1, index=pd.period_range("2016", freq="A-DEC", periods=4))  build_table_schema(s_per)  * Categoricals use the ``any`` type and an ``enum`` constraint listing  the set of possible values. Additionally, an ``ordered`` field is included:  .. ipython:: python  s_cat = pd.Series(pd.Categorical(["a", "b", "a"]))  build_table_schema(s_cat)  * A ``primaryKey`` field, containing an array of labels, is included  *if the index is unique*:  .. ipython:: python  s_dupe = pd.Series([1, 2], index=[1, 1])  build_table_schema(s_dupe)  * The ``primaryKey`` behavior is the same with MultiIndexes, but in this  case the ``primaryKey`` is an array:  .. ipython:: python  s_multi = pd.Series(1, index=pd.MultiIndex.from_product([("a", "b"), (0, 1)]))  build_table_schema(s_multi)  * The default naming roughly follows these rules:  * For series, the ``object.name`` is used. If that's none, then the  name is ``values``  * For ``DataFrames``, the stringified version of the column name is used  * For ``Index`` (not ``MultiIndex``), ``index.name`` is used, with a  fallback to ``index`` if that is None.  * For ``MultiIndex``, ``mi.names`` is used. If any level has no name,  then ``level_<i>`` is used.  ``read_json`` also accepts ``orient='table'`` as an argument. This allows for
the preservation of metadata such as dtypes and index names in a
round-trippable manner.  .. ipython:: python  df = pd.DataFrame(  {  "foo": [1, 2, 3, 4],  "bar": ["a", "b", "c", "d"],  "baz": pd.date_range("2018-01-01", freq="d", periods=4),  "qux": pd.Categorical(["a", "b", "c", "c"]),  },  index=pd.Index(range(4), name="idx"),  )  df  df.dtypes  df.to_json("test.json", orient="table")  new_df = pd.read_json("test.json", orient="table")  new_df  new_df.dtypes  Please note that the literal string 'index' as the name of an `Index`
is not round-trippable, nor are any names beginning with ``'level_'`` within a
`MultiIndex`. These are used by default in `DataFrame.to_json` to
indicate missing values and the subsequent read cannot distinguish the intent.  .. ipython:: python  df.index.name = "index"  df.to_json("test.json", orient="table")  new_df = pd.read_json("test.json", orient="table")  print(new_df.index.name)  .. ipython:: python  os.remove("test.json")  When using ``orient='table'`` along with user-defined ``ExtensionArray``,
the generated schema will contain an additional ``extDtype`` key in the respective
``fields`` element. This extra key is not standard but does enable JSON roundtrips
for extension types (e.g. ``read_json(df.to_json(orient="table"), orient="table")``).  The ``extDtype`` key carries the name of the extension, if you have properly registered
the ``ExtensionDtype``, pandas will use said name to perform a lookup into the registry
and re-convert the serialized data into your custom dtype.  .. _Table Schema: https://specs.frictionlessdata.io/table-schema/

.. _io.read_html:  Reading HTML content
'''  .. warning::  We **highly encourage** you to read the `HTML Table Parsing gotchas <io.html.gotchas>`  below regarding the issues surrounding the BeautifulSoup4/html5lib/lxml parsers.  The top-level `~pandas.io.html.read_html` function can accept an HTML
string/file/URL and will parse HTML tables into list of pandas ``DataFrames``.
Let's look at a few examples.  .. note::  ``read_html`` returns a ``list`` of ``DataFrame`` objects, even if there is  only a single table contained in the HTML content.  Read a URL with no options:  .. code-block:: ipython  In [320]: "https://www.fdic.gov/resources/resolutions/bank-failures/failed-bank-list"  In [321]: pd.read_html(url)  Out[321]:  [  Bank NameBank  CityCity StateSt  ...  Acquiring InstitutionAI Closing DateClosing FundFund  0  Almena State Bank  Almena  KS  ...  Equity Bank  October 23, 2020  10538  1  First City Bank of Florida  Fort Walton Beach  FL  ...  United Fidelity Bank, fsb  October 16, 2020  10537  2  The First State Bank  Barboursville  WV  ...  MVB Bank, Inc.  April 3, 2020  10536  3  Ericson State Bank  Ericson  NE  ...  Farmers and Merchants Bank  February 14, 2020  10535  4  City National Bank of New Jersey  Newark  NJ  ...  Industrial Bank  November 1, 2019  10534  ..  ...  ...  ...  ...  ...  ...  ...  558  Superior Bank, FSB  Hinsdale  IL  ...  Superior Federal, FSB  July 27, 2001  6004  559  Malta National Bank  Malta  OH  ...  North Valley Bank  May 3, 2001  4648  560  First Alliance Bank & Trust Co.  Manchester  NH  ...  Southern New Hampshire Bank & Trust  February 2, 2001  4647  561  National State Bank of Metropolis  Metropolis  IL  ...  Banterra Bank of Marion  December 14, 2000  4646  562  Bank of Honolulu  Honolulu  HI  ...  Bank of the Orient  October 13, 2000  4645  [563 rows x 7 columns]]  .. note::  The data from the above URL changes every Monday so the resulting data above may be slightly different.  Read in the content of the file from the above URL and pass it to ``read_html``
as a string:  .. ipython:: python  html_str = """  <table>  <tr>  <th>A</th>  <th colspan="1">B</th>  <th rowspan="1">C</th>  </tr>  <tr>  <td>a</td>  <td>b</td>  <td>c</td>  </tr>  </table>  """  with open("tmp.html", "w") as f:  f.write(html_str)  df = pd.read_html("tmp.html")  df[0]  .. ipython:: python  os.remove("tmp.html")  You can even pass in an instance of ``StringIO`` if you so desire:  .. ipython:: python  dfs = pd.read_html(StringIO(html_str))  dfs[0]  .. note::  The following examples are not run by the IPython evaluator due to the fact  that having so many network-accessing functions slows down the documentation  build. If you spot an error or an example that doesn't run, please do not  hesitate to report it over on `pandas GitHub issues page  <https://github.com/pandas-dev/pandas/issues>`__.  Read a URL and match a table that contains specific text:  .. code-block:: python  match = "Metcalf Bank"  df_list = pd.read_html(url, match=match)  Specify a header row (by default ``<th>`` or ``<td>`` elements located within a
``<thead>`` are used to form the column index, if multiple rows are contained within
``<thead>`` then a MultiIndex is created); if specified, the header row is taken
from the data minus the parsed header elements (``<th>`` elements).  .. code-block:: python  dfs = pd.read_html(url, header=0)  Specify an index column:  .. code-block:: python  dfs = pd.read_html(url, index_col=0)  Specify a number of rows to skip:  .. code-block:: python  dfs = pd.read_html(url, skiprows=0)  Specify a number of rows to skip using a list (``range`` works

as well):  .. code-block:: python  dfs = pd.read_html(url, skiprows=range(2))  Specify an HTML attribute:  .. code-block:: python  dfs1 = pd.read_html(url, attrs={"id": "table"})  dfs2 = pd.read_html(url, attrs={"class": "sortable"})  print(np.array_equal(dfs1[0], dfs2[0]))  # Should be True  Specify values that should be converted to NaN:  .. code-block:: python  dfs = pd.read_html(url, na_values=["No Acquirer"])  Specify whether to keep the default set of NaN values:  .. code-block:: python  dfs = pd.read_html(url, keep_default_na=False)  Specify converters for columns. This is useful for numerical text data that has
leading zeros.  By default columns that are numerical are cast to numeric
types and the leading zeros are lost. To avoid this, we can convert these
columns to strings.  .. code-block:: python  url_mcc = "https://en.wikipedia.org/wiki/Mobile_country_code"  dfs = pd.read_html(  url_mcc,  match="Telekom Albania",  header=0,  converters={"MNC": str},  )  Use some combination of the above:  .. code-block:: python  dfs = pd.read_html(url, match="Metcalf Bank", index_col=0)  Read in pandas ``to_html`` output (with some loss of floating point precision):  .. code-block:: python  df = pd.DataFrame(np.random.randn(2, 2))  s = df.to_html(float_format="{0:.40g}".format)  dfin = pd.read_html(s, index_col=0)  The ``lxml`` backend will raise an error on a failed parse if that is the only
parser you provide. If you only have a single parser you can provide just a
string, but it is considered good practice to pass a list with one string if,
for example, the function expects a sequence of strings. You may use:  .. code-block:: python  dfs = pd.read_html(url, "Metcalf Bank", index_col=0, flavor=["lxml"])  Or you could pass ``flavor='lxml'`` without a list:  .. code-block:: python  dfs = pd.read_html(url, "Metcalf Bank", index_col=0, flavor="lxml")  However, if you have bs4 and html5lib installed and pass ``None`` or ``['lxml',
'bs4']`` then the parse will most likely succeed. Note that *as soon as a parse
succeeds, the function will return*.  .. code-block:: python  dfs = pd.read_html(url, "Metcalf Bank", index_col=0, flavor=["lxml", "bs4"])  Links can be extracted from cells along with the text using ``extract_links="all"``.  .. ipython:: python  html_table = """  <table>  <tr>  <th>GitHub</th>  </tr>  <tr>  <td><a href="https://github.com/pandas-dev/pandas">pandas</a></td>  </tr>  </table>  """  df = pd.read_html(  html_table,  extract_links="all"  )[0]  df  df[("GitHub", None)]  df[("GitHub", None)].str[1]  .. versionadded:: 1.5.0  .. _io.html:  Writing to HTML files
'''  ``DataFrame`` objects have an instance method ``to_html`` which renders the
contents of the ``DataFrame`` as an HTML table. The function arguments are as
in the method ``to_string`` described above.  .. note::  Not all of the possible options for ``DataFrame.to_html`` are shown here for  brevity's sake. See `~pandas.core.frame.DataFrame.to_html` for the  full set of options.  .. note::  In an HTML-rendering supported environment like a Jupyter Notebook, ``display(HTML(...))```  will render the raw HTML into the environment.  .. ipython:: python  from IPython.display import display, HTML  df = pd.DataFrame(np.random.randn(2, 2))  df  html = df.to_html()  print(html)  # raw html  display(HTML(html))  The ``columns`` argument will limit the columns shown:  .. ipython:: python  html = df.to_html(columns=[0])  print(html)  display(HTML(html))  ``float_format`` takes a Python callable to control the precision of floating
point values:  .. ipython:: python  html = df.to_html(float_format="{0:.10f}".format)  print(html)  display(HTML(html))  ``bold_rows`` will make the row labels bold by default, but you can turn that
off:  .. ipython:: python  html = df.to_html(bold_rows=False)  print(html)  display(HTML(html))  The ``classes`` argument provides the ability to give the resulting HTML
table CSS classes. Note that these classes are *appended* to the existing

``'dataframe'`` class.  .. ipython:: python  print(df.to_html(classes=["awesome_table_class", "even_more_awesome_class"]))  The ``render_links`` argument provides the ability to add hyperlinks to cells
that contain URLs.  .. ipython:: python  url_df = pd.DataFrame(  {  "name": ["Python", "pandas"],  "url": ["https://www.python.org/", "https://pandas.pydata.org"],  }  )  html = url_df.to_html(render_links=True)  print(html)  display(HTML(html))  Finally, the ``escape`` argument allows you to control whether the
"<", ">" and "&" characters escaped in the resulting HTML (by default it is
``True``). So to get the HTML without escaped characters pass ``escape=False``  .. ipython:: python  df = pd.DataFrame({"a": list("&<>"), "b": np.random.randn(3)})  Escaped:  .. ipython:: python  html = df.to_html()  print(html)  display(HTML(html))  Not escaped:  .. ipython:: python  html = df.to_html(escape=False)  print(html)  display(HTML(html))  .. note::  Some browsers may not show a difference in the rendering of the previous two  HTML tables.  .. _io.html.gotchas:  HTML Table Parsing Gotchas
'''  There are some versioning issues surrounding the libraries that are used to
parse HTML tables in the top-level pandas io function ``read_html``.  **Issues with** |lxml|_  * Benefits  * |lxml|_ is very fast.  * |lxml|_ requires Cython to install correctly.  * Drawbacks  * |lxml|_ does *not* make any guarantees about the results of its parse  *unless* it is given |svm|_.  * In light of the above, we have chosen to allow you, the user, to use the  |lxml|_ backend, but **this backend will use** |html5lib|_ if |lxml|_  fails to parse  * It is therefore *highly recommended* that you install both  |BeautifulSoup4|_ and |html5lib|_, so that you will still get a valid  result (provided everything else is valid) even if |lxml|_ fails.  **Issues with** |BeautifulSoup4|_ **using** |lxml|_ **as a backend**  * The above issues hold here as well since |BeautifulSoup4|_ is essentially  just a wrapper around a parser backend.  **Issues with** |BeautifulSoup4|_ **using** |html5lib|_ **as a backend**  * Benefits  * |html5lib|_ is far more lenient than |lxml|_ and consequently deals  with *real-life markup* in a much saner way rather than just, e.g.,  dropping an element without notifying you.  * |html5lib|_ *generates valid HTML5 markup from invalid markup  automatically*. This is extremely important for parsing HTML tables,  since it guarantees a valid document. However, that does NOT mean that  it is "correct", since the process of fixing markup does not have a  single definition.  * |html5lib|_ is pure Python and requires no additional build steps beyond  its own installation.  * Drawbacks  * The biggest drawback to using |html5lib|_ is that it is slow as  molasses.  However consider the fact that many tables on the web are not  big enough for the parsing algorithm runtime to matter. It is more  likely that the bottleneck will be in the process of reading the raw  text from the URL over the web, i.e., IO (input-output). For very large  tables, this might not be true.  .. |svm| replace:: **strictly valid markup**
.. _svm: https://validator.w3.org/docs/help.html#validation_basics  .. |html5lib| replace:: **html5lib**
.. _html5lib: https://github.com/html5lib/html5lib-python  .. |BeautifulSoup4| replace:: **BeautifulSoup4**
.. _BeautifulSoup4: https://www.crummy.com/software/BeautifulSoup  .. |lxml| replace:: **lxml**
.. _lxml: https://lxml.de  .. _io.latex:

LaTeX

.. versionadded:: 1.3.0  Currently there are no methods to read from LaTeX, only output methods.  Writing to LaTeX files
'''  .. note::  DataFrame *and* Styler objects currently have a ``to_latex`` method. We recommend  using the Styler.to_latex()_ method  over DataFrame.to_latex()_ due to the former's greater flexibility with  conditional styling, and the latter's possible future deprecation.  Review the documentation for Styler.to_latex_,
which gives examples of conditional styling and explains the operation of its keyword
arguments.  For simple application the following pattern is sufficient.  .. ipython:: python  df = pd.DataFrame([[1, 2], [3, 4]], index=["a", "b"], columns=["c", "d"])  print(df.style.to_latex())  To format values before output, chain the Styler.format_
method.  .. ipython:: python  print(df.style.format("€ {}").to_latex())

.. _io.read_xml:  Reading XML
'''  .. versionadded:: 1.3.0  The top-level `~pandas.io.xml.read_xml` function can accept an XML
string/file/URL and will parse nodes and attributes into a pandas ``DataFrame``.  .. note::  Since there is no standard XML structure where design types can vary in  many ways, ``read_xml`` works best with flatter, shallow versions. If  an XML document is deeply nested, use the ``stylesheet`` feature to  transform XML into a flatter version.  Let's look at a few examples.  Read an XML string:  .. ipython:: python  xml = """<?xml version="1.0" encoding="UTF-8"?>  <bookstore>  <book category="cooking">  <title lang="en">Everyday Italian</title>  <author>Giada De Laurentiis</author>  <year>2005</year>  <price>30.00</price>  </book>  <book category="children">  <title lang="en">Harry Potter</title>  <author>J K. Rowling</author>  <year>2005</year>  <price>29.99</price>  </book>  <book category="web">  <title lang="en">Learning XML</title>  <author>Erik T. Ray</author>  <year>2003</year>  <price>39.95</price>  </book>  </bookstore>"""  df = pd.read_xml(xml)  df  Read a URL with no options:  .. ipython:: python  df = pd.read_xml("https://www.w3schools.com/xml/books.xml")  df  Read in the content of the "books.xml" file and pass it to ``read_xml``
as a string:  .. ipython:: python  file_path = "books.xml"  with open(file_path, "w") as f:  f.write(xml)  with open(file_path, "r") as f:  df = pd.read_xml(f.read())  df  Read in the content of the "books.xml" as instance of ``StringIO`` or
``BytesIO`` and pass it to ``read_xml``:  .. ipython:: python  with open(file_path, "r") as f:  sio = StringIO(f.read())  df = pd.read_xml(sio)  df  .. ipython:: python  with open(file_path, "rb") as f:  bio = BytesIO(f.read())  df = pd.read_xml(bio)  df  Even read XML from AWS S3 buckets such as NIH NCBI PMC Article Datasets providing
Biomedical and Life Science Jorurnals:  .. ipython:: python  df = pd.read_xml(  "s3://pmc-oa-opendata/oa_comm/xml/all/PMC1236943.xml",  xpath=".//journal-meta",  )  df  With `lxml`_ as default ``parser``, you access the full-featured XML library
that extends Python's ElementTree API. One powerful tool is ability to query
nodes selectively or conditionally with more expressive XPath:  .. _lxml: https://lxml.de  .. ipython:: python  df = pd.read_xml(file_path, xpath="//book[year=2005]")  df  Specify only elements or only attributes to parse:  .. ipython:: python  df = pd.read_xml(file_path, elems_only=True)  df  .. ipython:: python  df = pd.read_xml(file_path, attrs_only=True)  df  .. ipython:: python  os.remove("books.xml")  XML documents can have namespaces with prefixes and default namespaces without
prefixes both of which are denoted with a special attribute ``xmlns``. In order
to parse by node under a namespace context, ``xpath`` must reference a prefix.  For example, below XML contains a namespace with prefix, ``doc``, and URI at
``https://example.com``. In order to parse ``doc:row`` nodes,
``namespaces`` must be used.  .. ipython:: python  xml = """<?xml version='1.0' encoding='utf-8'?>  <doc:data xmlns:doc="https://example.com">  <doc:row>  <doc:shape>square</doc:shape>  <doc:degrees>360</doc:degrees>  <doc:sides>4.0</doc:sides>  </doc:row>  <doc:row>  <doc:shape>circle</doc:shape>  <doc:degrees>360</doc:degrees>  <doc:sides/>  </doc:row>  <doc:row>  <doc:shape>triangle</doc:shape>  <doc:degrees>180</doc:degrees>  <doc:sides>3.0</doc:sides>  </doc:row>  </doc:data>"""  df = pd.read_xml(xml,  xpath="//doc:row",  namespaces={"doc": "https://example.com"})  df  Similarly, an XML document can have a default namespace without prefix. Failing
to assign a temporary prefix will return no nodes and raise a ``ValueError``.

But assigning *any* temporary name to correct URI allows parsing by nodes.  .. ipython:: python  xml = """<?xml version='1.0' encoding='utf-8'?>  <data xmlns="https://example.com">  <row>  <shape>square</shape>  <degrees>360</degrees>  <sides>4.0</sides>  </row>  <row>  <shape>circle</shape>  <degrees>360</degrees>  <sides/>  </row>  <row>  <shape>triangle</shape>  <degrees>180</degrees>  <sides>3.0</sides>  </row>  </data>"""  df = pd.read_xml(xml,  xpath="//pandas:row",  namespaces={"pandas": "https://example.com"})  df  However, if XPath does not reference node names such as default, ``/*``, then
``namespaces`` is not required.  .. note::  Since ``xpath`` identifies the parent of content to be parsed, only immediate  desendants which include child nodes or current attributes are parsed.  Therefore, ``read_xml`` will not parse the text of grandchildren or other  descendants and will not parse attributes of any descendant. To retrieve  lower level content, adjust xpath to lower level. For example,  .. ipython:: python  xml = """  <data>  <row>  <shape sides="4">square</shape>  <degrees>360</degrees>  </row>  <row>  <shape sides="0">circle</shape>  <degrees>360</degrees>  </row>  <row>  <shape sides="3">triangle</shape>  <degrees>180</degrees>  </row>  </data>"""  df = pd.read_xml(xml, xpath="./row")  df  shows the attribute ``sides`` on ``shape`` element was not parsed as  expected since this attribute resides on the child of ``row`` element  and not ``row`` element itself. In other words, ``sides`` attribute is a  grandchild level descendant of ``row`` element. However, the ``xpath``  targets ``row`` element which covers only its children and attributes.  With `lxml`_ as parser, you can flatten nested XML documents with an XSLT
script which also can be string/file/URL types. As background, `XSLT`_ is
a special-purpose language written in a special XML file that can transform
original XML documents into other XML, HTML, even text (CSV, JSON, etc.)
using an XSLT processor.  .. _lxml: https://lxml.de
.. _XSLT: https://www.w3.org/TR/xslt/  For example, consider this somewhat nested structure of Chicago "L" Rides
where station and rides elements encapsulate data in their own sections.
With below XSLT, ``lxml`` can transform original nested document into a flatter

output (as shown below for demonstration) for easier parse into ``DataFrame``:  .. ipython:: python  xml = """<?xml version='1.0' encoding='utf-8'?>  <response>  <row>  <station id="40850" name="Library"/>  <month>2020-09-01T000</month>  <rides>  <avg_weekday_rides>864.2</avg_weekday_rides>  <avg_saturday_rides>534</avg_saturday_rides>  <avg_sunday_holiday_rides>417.2</avg_sunday_holiday_rides>  </rides>  </row>  <row>  <station id="41700" name="Washington/Wabash"/>  <month>2020-09-01T000</month>  <rides>  <avg_weekday_rides>2707.4</avg_weekday_rides>  <avg_saturday_rides>1909.8</avg_saturday_rides>  <avg_sunday_holiday_rides>1438.6</avg_sunday_holiday_rides>  </rides>  </row>  <row>  <station id="40380" name="Clark/Lake"/>  <month>2020-09-01T000</month>  <rides>  <avg_weekday_rides>2949.6</avg_weekday_rides>  <avg_saturday_rides>1657</avg_saturday_rides>  <avg_sunday_holiday_rides>1453.8</avg_sunday_holiday_rides>  </rides>  </row>  </response>"""  xsl = """<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">  <xsl:output method="xml" omit-xml-declaration="no" indent="yes"/>  <xsl:strip-space elements="*"/>  <xsl:template match="/response">  <xsl:copy>  <xsl:apply-templates select="row"/>  </xsl:copy>  </xsl:template>  <xsl:template match="row">  <xsl:copy>  <station_id><xsl:value-of select="station/@id"/></station_id>  <station_name><xsl:value-of select="station/@name"/></station_name>  <xsl:copy-of select="month|rides/*"/>  </xsl:copy>  </xsl:template>  </xsl:stylesheet>"""  output = """<?xml version='1.0' encoding='utf-8'?>  <response>  <row>  <station_id>40850</station_id>  <station_name>Library</station_name>  <month>2020-09-01T000</month>  <avg_weekday_rides>864.2</avg_weekday_rides>  <avg_saturday_rides>534</avg_saturday_rides>  <avg_sunday_holiday_rides>417.2</avg_sunday_holiday_rides>  </row>  <row>  <station_id>41700</station_id>  <station_name>Washington/Wabash</station_name>  <month>2020-09-01T000</month>  <avg_weekday_rides>2707.4</avg_weekday_rides>  <avg_saturday_rides>1909.8</avg_saturday_rides>  <avg_sunday_holiday_rides>1438.6</avg_sunday_holiday_rides>  </row>  <row>  <station_id>40380</station_id>  <station_name>Clark/Lake</station_name>  <month>2020-09-01T000</month>  <avg_weekday_rides>2949.6</avg_weekday_rides>  <avg_saturday_rides>1657</avg_saturday_rides>  <avg_sunday_holiday_rides>1453.8</avg_sunday_holiday_rides>  </row>  </response>"""  df = pd.read_xml(xml, stylesheet=xsl)  df  For very large XML files that can range in hundreds of megabytes to gigabytes, `pandas.read_xml`
supports parsing such sizeable files using `lxml's iterparse`_ and `etree's iterparse`_
which are memory-efficient methods to iterate through an XML tree and extract specific elements and attributes.
without holding entire tree in memory.  .. versionadded:: 1.5.0  .. _`lxml's iterparse`: https://lxml.de/3.2/parsing.html#iterparse-and-iterwalk
.. _`etree's iterparse`: https://docs.python.org/3/library/xml.etree.elementtree.html#xml.etree.ElementTree.iterparse  To use this feature, you must pass a physical XML file path into ``read_xml`` and use the ``iterparse`` argument.
Files should not be compressed or point to online sources but stored on local disk. Also, ``iterparse`` should be
a dictionary where the key is the repeating nodes in document (which become the rows) and the value is a list of
any element or attribute that is a descendant (i.e., child, grandchild) of repeating node. Since XPath is not
used in this method, descendants do not need to share same relationship with one another. Below shows example

of reading in Wikipedia's very large (12 GB+) latest article data dump.  .. code-block:: ipython  In [1]: df = pd.read_xml(  ...  "/path/to/downloaded/enwikisource-latest-pages-articles.xml",  ...  iterparse = {"page": ["title", "ns", "id"]}  ...  )  ...  df  Out[2]:  title  ns  id  0  Gettysburg Address  0  21450  1  Main Page  0  42950  2  Declaration by United Nations  0  8435  3  Constitution of the United States of America  0  8435  4  Declaration of Independence (Israel)  0  17858  ...  ...  ...  ...  3578760  Page:Black cat 1897 07 v2 n10.pdf/17  104  219649  3578761  Page:Black cat 1897 07 v2 n10.pdf/43  104  219649  3578762  Page:Black cat 1897 07 v2 n10.pdf/44  104  219649  3578763  The History of Tom Jones, a Foundling/Book IX  0  12084291  3578764  Page:Shakespeare of Stratford (1926) Yale.djvu/91  104  21450  [3578765 rows x 3 columns]  .. _io.xml:  Writing XML
'''  .. versionadded:: 1.3.0  ``DataFrame`` objects have an instance method ``to_xml`` which renders the
contents of the ``DataFrame`` as an XML document.  .. note::  This method does not support special properties of XML including DTD,  CData, XSD schemas, processing instructions, comments, and others.  Only namespaces at the root level is supported. However, ``stylesheet``  allows design changes after initial output.  Let's look at a few examples.  Write an XML without options:  .. ipython:: python  geom_df = pd.DataFrame(  {  "shape": ["square", "circle", "triangle"],  "degrees": [360, 360, 180],  "sides": [4, np.nan, 3],  }  )  print(geom_df.to_xml())  Write an XML with new root and row name:  .. ipython:: python  print(geom_df.to_xml(root_name="geometry", row_name="objects"))  Write an attribute-centric XML:  .. ipython:: python  print(geom_df.to_xml(attr_cols=geom_df.columns.tolist()))  Write a mix of elements and attributes:  .. ipython:: python  print(  geom_df.to_xml(  index=False,  attr_cols=['shape'],  elem_cols=['degrees', 'sides'])  )  Any ``DataFrames`` with hierarchical columns will be flattened for XML element names
with levels delimited by underscores:  .. ipython:: python  ext_geom_df = pd.DataFrame(  {  "type": ["polygon", "other", "polygon"],  "shape": ["square", "circle", "triangle"],  "degrees": [360, 360, 180],  "sides": [4, np.nan, 3],  }  )  pvt_df = ext_geom_df.pivot_table(index='shape',  columns='type',  values=['degrees', 'sides'],  aggfunc='sum')  pvt_df  print(pvt_df.to_xml())  Write an XML with default namespace:  .. ipython:: python  print(geom_df.to_xml(namespaces={"": "https://example.com"}))  Write an XML with namespace prefix:  .. ipython:: python  print(  geom_df.to_xml(namespaces={"doc": "https://example.com"},  prefix="doc")  )  Write an XML without declaration or pretty print:  .. ipython:: python  print(  geom_df.to_xml(xml_declaration=False,  pretty_print=False)  )  Write an XML and transform with stylesheet:  .. ipython:: python  xsl = """<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">  <xsl:output method="xml" omit-xml-declaration="no" indent="yes"/>  <xsl:strip-space elements="*"/>  <xsl:template match="/data">  <geometry>  <xsl:apply-templates select="row"/>  </geometry>  </xsl:template>  <xsl:template match="row">  <object index="{index}">  <xsl:if test="shape!='circle'">  <xsl:attribute name="type">polygon</xsl:attribute>  </xsl:if>  <xsl:copy-of select="shape"/>  <property>  <xsl:copy-of select="degrees|sides"/>  </property>  </object>  </xsl:template>  </xsl:stylesheet>"""  print(geom_df.to_xml(stylesheet=xsl))  XML Final Notes

'''  * All XML documents adhere to `W3C specifications`_. Both ``etree`` and ``lxml``  parsers will fail to parse any markup document that is not well-formed or  follows XML syntax rules. Do be aware HTML is not an XML document unless it  follows XHTML specs. However, other popular markup types including KML, XAML,  RSS, MusicML, MathML are compliant `XML schemas`_.  * For above reason, if your application builds XML prior to pandas operations,  use appropriate DOM libraries like ``etree`` and ``lxml`` to build the necessary  document and not by string concatenation or regex adjustments. Always remember  XML is a *special* text file with markup rules.  * With very large XML files (several hundred MBs to GBs), XPath and XSLT  can become memory-intensive operations. Be sure to have enough available  RAM for reading and writing to large XML files (roughly about 5 times the  size of text).  * Because XSLT is a programming language, use it with caution since such scripts  can pose a security risk in your environment and can run large or infinite  recursive operations. Always test scripts on small fragments before full run.  * The `etree`_ parser supports all functionality of both ``read_xml`` and  ``to_xml`` except for complex XPath and any XSLT. Though limited in features,  ``etree`` is still a reliable and capable parser and tree builder. Its  performance may trail ``lxml`` to a certain degree for larger files but  relatively unnoticeable on small to medium size files.  .. _`W3C specifications`: https://www.w3.org/TR/xml/
.. _`XML schemas`: https://en.wikipedia.org/wiki/List_of_types_of_XML_schemas
.. _`etree`: https://docs.python.org/3/library/xml.etree.elementtree.html  .. _io.excel:

The `~pandas.read_excel` method can read Excel 2007+ (``.xlsx``) files
using the ``openpyxl`` Python module. Excel 2003 (``.xls``) files
can be read using ``xlrd``. Binary Excel (``.xlsb``)
files can be read using ``pyxlsb``.
The `~DataFrame.to_excel` instance method is used for
saving a ``DataFrame`` to Excel.  Generally the semantics are
similar to working with `csv<io.read_csv_table>` data.
See the `cookbook<cookbook.excel>` for some advanced strategies.  .. warning::  The xlrd_ package is now only for reading  old-style ``.xls`` files.  Before pandas 1.3.0, the default argument ``engine=None`` to `~pandas.read_excel`  would result in using the ``xlrd`` engine in many cases, including new  Excel 2007+ (``.xlsx``) files. pandas will now default to using the  openpyxl_ engine.  It is strongly encouraged to install ``openpyxl`` to read Excel 2007+  (``.xlsx``) files.  **Please do not report issues when using ``xlrd`` to read ``.xlsx`` files.**  This is no longer supported, switch to using ``openpyxl`` instead.  .. _io.excel_reader:  Reading Excel files
'''  In the most basic use-case, ``read_excel`` takes a path to an Excel
file, and the ``sheet_name`` indicating which sheet to parse.  .. code-block:: python  # Returns a DataFrame  pd.read_excel("path_to_file.xls", sheet_name="Sheet1")  .. _io.excel.excelfile_class:  ``ExcelFile`` class
+++  To facilitate working with multiple sheets from the same file, the ``ExcelFile``
class can be used to wrap the file and can be passed into ``read_excel``
There will be a performance benefit for reading multiple sheets as the file is
read into memory only once.  .. code-block:: python  xlsx = pd.ExcelFile("path_to_file.xls")  df = pd.read_excel(xlsx, "Sheet1")  The ``ExcelFile`` class can also be used as a context manager.  .. code-block:: python  with pd.ExcelFile("path_to_file.xls") as xls:  df1 = pd.read_excel(xls, "Sheet1")  df2 = pd.read_excel(xls, "Sheet2")  The ``sheet_names`` property will generate
a list of the sheet names in the file.  The primary use-case for an ``ExcelFile`` is parsing multiple sheets with
different parameters:  .. code-block:: python  data = {}  # For when Sheet1's format differs from Sheet2  with pd.ExcelFile("path_to_file.xls") as xls:  data["Sheet1"] = pd.read_excel(xls, "Sheet1", index_col=None, na_values=["NA"])  data["Sheet2"] = pd.read_excel(xls, "Sheet2", index_col=1)  Note that if the same parsing parameters are used for all sheets, a list
of sheet names can simply be passed to ``read_excel`` with no loss in performance.  .. code-block:: python  # using the ExcelFile class  data = {}  with pd.ExcelFile("path_to_file.xls") as xls:  data["Sheet1"] = pd.read_excel(xls, "Sheet1", index_col=None, na_values=["NA"])  data["Sheet2"] = pd.read_excel(xls, "Sheet2", index_col=None, na_values=["NA"])  # equivalent using the read_excel function  data = pd.read_excel(  "path_to_file.xls", ["Sheet1", "Sheet2"], index_col=None, na_values=["NA"]  )  ``ExcelFile`` can also be called with a ``xlrd.book.Book`` object
as a parameter. This allows the user to control how the excel file is read.
For example, sheets can be loaded on demand by calling ``xlrd.open_workbook()``
with ``on_demand=True``.  .. code-block:: python  import xlrd  xlrd_book = xlrd.open_workbook("path_to_file.xls", on_demand=True)  with pd.ExcelFile(xlrd_book) as xls:  df1 = pd.read_excel(xls, "Sheet1")  df2 = pd.read_excel(xls, "Sheet2")  .. _io.excel.specifying_sheets:  Specifying sheets
+++  .. note:: The second argument is ``sheet_name``, not to be confused with ``ExcelFile.sheet_names``.  .. note:: An ExcelFile's attribute ``sheet_names`` provides access to a list of sheets.  * The arguments ``sheet_name`` allows specifying the sheet or sheets to read.
* The default value for ``sheet_name`` is 0, indicating to read the first sheet
* Pass a string to refer to the name of a particular sheet in the workbook.
* Pass an integer to refer to the index of a sheet. Indices follow Python  convention, beginning at 0.

* Pass a list of either strings or integers, to return a dictionary of specified sheets.
* Pass a ``None`` to return a dictionary of all available sheets.  .. code-block:: python  # Returns a DataFrame  pd.read_excel("path_to_file.xls", "Sheet1", index_col=None, na_values=["NA"])  Using the sheet index:  .. code-block:: python  # Returns a DataFrame  pd.read_excel("path_to_file.xls", 0, index_col=None, na_values=["NA"])  Using all default values:  .. code-block:: python  # Returns a DataFrame  pd.read_excel("path_to_file.xls")  Using None to get all sheets:  .. code-block:: python  # Returns a dictionary of DataFrames  pd.read_excel("path_to_file.xls", sheet_name=None)  Using a list to get multiple sheets:  .. code-block:: python  # Returns the 1st and 4th sheet, as a dictionary of DataFrames.  pd.read_excel("path_to_file.xls", sheet_name=["Sheet1", 3])  ``read_excel`` can read more than one sheet, by setting ``sheet_name`` to either
a list of sheet names, a list of sheet positions, or ``None`` to read all sheets.
Sheets can be specified by sheet index or sheet name, using an integer or string,
respectively.  .. _io.excel.reading_multiindex:  Reading a ``MultiIndex``
+++  ``read_excel`` can read a ``MultiIndex`` index, by passing a list of columns to ``index_col``
and a ``MultiIndex`` column by passing a list of rows to ``header``.  If either the ``index``
or ``columns`` have serialized level names those will be read in as well by specifying
the rows/columns that make up the levels.  For example, to read in a ``MultiIndex`` index without names:  .. ipython:: python  df = pd.DataFrame(  {"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]},  index=pd.MultiIndex.from_product([["a", "b"], ["c", "d"]]),  )  df.to_excel("path_to_file.xlsx")  df = pd.read_excel("path_to_file.xlsx", index_col=[0, 1])  df  If the index has level names, they will parsed as well, using the same
parameters.  .. ipython:: python  df.index = df.index.set_names(["lvl1", "lvl2"])  df.to_excel("path_to_file.xlsx")  df = pd.read_excel("path_to_file.xlsx", index_col=[0, 1])  df  If the source file has both ``MultiIndex`` index and columns, lists specifying each
should be passed to ``index_col`` and ``header``:  .. ipython:: python  df.columns = pd.MultiIndex.from_product([["a"], ["b", "d"]], names=["c1", "c2"])  df.to_excel("path_to_file.xlsx")  df = pd.read_excel("path_to_file.xlsx", index_col=[0, 1], header=[0, 1])  df  .. ipython:: python  os.remove("path_to_file.xlsx")  Missing values in columns specified in ``index_col`` will be forward filled to
allow roundtripping with ``to_excel`` for ``merged_cells=True``. To avoid forward
filling the missing values use ``set_index`` after reading the data instead of
``index_col``.  Parsing specific columns
+++  It is often the case that users will insert columns to do temporary computations
in Excel and you may not want to read in those columns. ``read_excel`` takes
a ``usecols`` keyword to allow you to specify a subset of columns to parse.  .. versionchanged:: 1.0.0  Passing in an integer for ``usecols`` will no longer work. Please pass in a list
of ints from 0 to ``usecols`` inclusive instead.  You can specify a comma-delimited set of Excel columns and ranges as a string:  .. code-block:: python  pd.read_excel("path_to_file.xls", "Sheet1", usecols="A,C:E")  If ``usecols`` is a list of integers, then it is assumed to be the file column
indices to be parsed.  .. code-block:: python  pd.read_excel("path_to_file.xls", "Sheet1", usecols=[0, 2, 3])  Element order is ignored, so ``usecols=[0, 1]`` is the same as ``[1, 0]``.  If ``usecols`` is a list of strings, it is assumed that each string corresponds
to a column name provided either by the user in ``names`` or inferred from the

document header row(s). Those strings define which columns will be parsed:  .. code-block:: python  pd.read_excel("path_to_file.xls", "Sheet1", usecols=["foo", "bar"])  Element order is ignored, so ``usecols=['baz', 'joe']`` is the same as ``['joe', 'baz']``.  If ``usecols`` is callable, the callable function will be evaluated against
the column names, returning names where the callable function evaluates to ``True``.  .. code-block:: python  pd.read_excel("path_to_file.xls", "Sheet1", usecols=lambda x: x.isalpha())  Parsing dates
+++  Datetime-like values are normally automatically converted to the appropriate
dtype when reading the excel file. But if you have a column of strings that
*look* like dates (but are not actually formatted as dates in excel), you can
use the ``parse_dates`` keyword to parse those strings to datetimes:  .. code-block:: python  pd.read_excel("path_to_file.xls", "Sheet1", parse_dates=["date_strings"])  Cell converters
+++  It is possible to transform the contents of Excel cells via the ``converters``
option. For instance, to convert a column to boolean:  .. code-block:: python  pd.read_excel("path_to_file.xls", "Sheet1", converters={"MyBools": bool})  This options handles missing values and treats exceptions in the converters
as missing data. Transformations are applied cell by cell rather than to the
column as a whole, so the array dtype is not guaranteed. For instance, a
column of integers with missing values cannot be transformed to an array
with integer dtype, because NaN is strictly a float. You can manually mask
missing data to recover integer dtype:  .. code-block:: python  def cfun(x):  return int(x) if x else -1  pd.read_excel("path_to_file.xls", "Sheet1", converters={"MyInts": cfun})  Dtype specifications
+++  As an alternative to converters, the type for an entire column can
be specified using the ``dtype`` keyword, which takes a dictionary
mapping column names to types.  To interpret data with
no type inference, use the type ``str`` or ``object``.  .. code-block:: python  pd.read_excel("path_to_file.xls", dtype={"MyInts": "int64", "MyText": str})  .. _io.excel_writer:  Writing Excel files
'''  Writing Excel files to disk
+++  To write a ``DataFrame`` object to a sheet of an Excel file, you can use the
``to_excel`` instance method.  The arguments are largely the same as ``to_csv``
described above, the first argument being the name of the excel file, and the
optional second argument the name of the sheet to which the ``DataFrame`` should be
written. For example:  .. code-block:: python  df.to_excel("path_to_file.xlsx", sheet_name="Sheet1")  Files with a
``.xlsx`` extension will be written using ``xlsxwriter`` (if available) or
``openpyxl``.  The ``DataFrame`` will be written in a way that tries to mimic the REPL output.
The ``index_label`` will be placed in the second
row instead of the first. You can place it in the first row by setting the
``merge_cells`` option in ``to_excel()`` to ``False``:  .. code-block:: python  df.to_excel("path_to_file.xlsx", index_label="label", merge_cells=False)  In order to write separate ``DataFrames`` to separate sheets in a single Excel file,
one can pass an `~pandas.io.excel.ExcelWriter`.  .. code-block:: python  with pd.ExcelWriter("path_to_file.xlsx") as writer:  df1.to_excel(writer, sheet_name="Sheet1")  df2.to_excel(writer, sheet_name="Sheet2")  .. _io.excel_writing_buffer:  Writing Excel files to memory
+++  pandas supports writing Excel files to buffer-like objects such as ``StringIO`` or

``BytesIO`` using `~pandas.io.excel.ExcelWriter`.  .. code-block:: python  from io import BytesIO  bio = BytesIO()  # By setting the 'engine' in the ExcelWriter constructor.  writer = pd.ExcelWriter(bio, engine="xlsxwriter")  df.to_excel(writer, sheet_name="Sheet1")  # Save the workbook  writer.save()  # Seek to the beginning and read to copy the workbook to a variable in memory  bio.seek(0)  workbook = bio.read()  .. note::  ``engine`` is optional but recommended.  Setting the engine determines  the version of workbook produced. Setting ``engine='xlrd'`` will produce an  Excel 2003-format workbook (xls).  Using either ``'openpyxl'`` or  ``'xlsxwriter'`` will produce an Excel 2007-format workbook (xlsx). If  omitted, an Excel 2007-formatted workbook is produced.  .. _io.excel.writers:  Excel writer engines
'''  pandas chooses an Excel writer via two methods:  1. the ``engine`` keyword argument
2. the filename extension (via the default specified in config options)  By default, pandas uses the `XlsxWriter`_  for ``.xlsx``, `openpyxl`_
for ``.xlsm``. If you have multiple
engines installed, you can set the default engine through `setting the
config options <options>` ``io.excel.xlsx.writer`` and
``io.excel.xls.writer``. pandas will fall back on `openpyxl`_ for ``.xlsx``
files if `Xlsxwriter`_ is not available.  .. _XlsxWriter: https://xlsxwriter.readthedocs.io
.. _openpyxl: https://openpyxl.readthedocs.io/  To specify which writer you want to use, you can pass an engine keyword
argument to ``to_excel`` and to ``ExcelWriter``. The built-in engines are:  * ``openpyxl``: version 2.4 or higher is required
* ``xlsxwriter``  .. code-block:: python  # By setting the 'engine' in the DataFrame 'to_excel()' methods.  df.to_excel("path_to_file.xlsx", sheet_name="Sheet1", engine="xlsxwriter")  # By setting the 'engine' in the ExcelWriter constructor.  writer = pd.ExcelWriter("path_to_file.xlsx", engine="xlsxwriter")  # Or via pandas configuration.  from pandas import options  # noqa: E402  options.io.excel.xlsx.writer = "xlsxwriter"  df.to_excel("path_to_file.xlsx", sheet_name="Sheet1")  .. _io.excel.style:  Style and formatting
'''  The look and feel of Excel worksheets created from pandas can be modified using the following parameters on the ``DataFrame``'s ``to_excel`` method.  * ``float_format`` : Format string for floating point numbers (default ``None``).
* ``freeze_panes`` : A tuple of two integers representing the bottommost row and rightmost column to freeze. Each of these parameters is one-based, so (1, 1) will freeze the first row and first column (default ``None``).  Using the `Xlsxwriter`_ engine provides many options for controlling the
format of an Excel worksheet created with the ``to_excel`` method.  Excellent examples can be found in the
`Xlsxwriter`_ documentation here: https://xlsxwriter.readthedocs.io/working_with_pandas.html  .. _io.ods:

OpenDocument Spreadsheets

The io methods for `Excel files`_ also support reading and writing OpenDocument spreadsheets
using the odfpy_ module. The semantics and features for reading and writing
OpenDocument spreadsheets match what can be done for `Excel files`_ using
``engine='odf'``. The optional dependency 'odfpy' needs to be installed.  The `~pandas.read_excel` method can read OpenDocument spreadsheets  .. code-block:: python  # Returns a DataFrame  pd.read_excel("path_to_file.ods", engine="odf")  .. versionadded:: 1.1.0  Similarly, the `~pandas.to_excel` method can write OpenDocument spreadsheets  .. code-block:: python  # Writes DataFrame to a .ods file  df.to_excel("path_to_file.ods", engine="odf")  .. _io.xlsb:

Binary Excel (.xlsb) files

.. versionadded:: 1.0.0  The `~pandas.read_excel` method can also read binary Excel files
using the ``pyxlsb`` module. The semantics and features for reading
binary Excel files mostly match what can be done for `Excel files`_ using
``engine='pyxlsb'``. ``pyxlsb`` does not recognize datetime types
in files and will return floats instead.  .. code-block:: python  # Returns a DataFrame  pd.read_excel("path_to_file.xlsb", engine="pyxlsb")  .. note::  Currently pandas only supports *reading* binary Excel files. Writing  is not implemented.  .. _io.clipboard:

Clipboard

A handy way to grab data is to use the `~DataFrame.read_clipboard` method,
which takes the contents of the clipboard buffer and passes them to the
``read_csv`` method. For instance, you can copy the following text to the
clipboard (CTRL-C on many operating systems):  .. code-block:: console  A B C  x 1 4 p  y 2 5 q  z 3 6 r  And then import the data directly to a ``DataFrame`` by calling:  .. code-block:: python  >>> clipdf = pd.read_clipboard()  >>> clipdf  A B C  x 1 4 p  y 2 5 q  z 3 6 r  The ``to_clipboard`` method can be used to write the contents of a ``DataFrame`` to
the clipboard. Following which you can paste the clipboard contents into other
applications (CTRL-V on many operating systems). Here we illustrate writing a
``DataFrame`` into clipboard and reading it back.  .. code-block:: python  >>> df = pd.DataFrame(  ...  {"A": [1, 2, 3], "B": [4, 5, 6], "C": ["p", "q", "r"]}, index=["x", "y", "z"]  ... )  >>> df  A B C  x 1 4 p  y 2 5 q  z 3 6 r  >>> df.to_clipboard()  >>> pd.read_clipboard()  A B C  x 1 4 p  y 2 5 q  z 3 6 r  We can see that we got the same content back, which we had earlier written to the clipboard.  .. note::  You may need to install xclip or xsel (with PyQt5, PyQt4 or qtpy) on Linux to use these methods.  .. _io.pickle:

Pickling

All pandas objects are equipped with ``to_pickle`` methods which use Python's
``cPickle`` module to save data structures to disk using the pickle format.  .. ipython:: python  df  df.to_pickle("foo.pkl")  The ``read_pickle`` function in the ``pandas`` namespace can be used to load
any pickled pandas object (or any other pickled object) from file:  .. ipython:: python  pd.read_pickle("foo.pkl")  .. ipython:: python  os.remove("foo.pkl")  .. warning::  Loading pickled data received from untrusted sources can be unsafe.  See: https://docs.python.org/3/library/pickle.html  .. warning::  `read_pickle` is only guaranteed backwards compatible back to pandas version 0.20.3  .. _io.pickle.compression:  Compressed pickle files
'''  `read_pickle`, `DataFrame.to_pickle` and `Series.to_pickle` can read
and write compressed pickle files. The compression types of ``gzip``, ``bz2``, ``xz``, ``zstd`` are supported for reading and writing.
The ``zip`` file format only supports reading and must contain only one data file
to be read.  The compression type can be an explicit parameter or be inferred from the file extension.
If 'infer', then use ``gzip``, ``bz2``, ``zip``, ``xz``, ``zstd`` if filename ends in ``'.gz'``, ``'.bz2'``, ``'.zip'``,
``'.xz'``, or ``'.zst'``, respectively.  The compression parameter can also be a ``dict`` in order to pass options to the
compression protocol. It must have a ``'method'`` key set to the name
of the compression protocol, which must be one of
{``'zip'``, ``'gzip'``, ``'bz2'``, ``'xz'``, ``'zstd'``}. All other key-value pairs are passed to
the underlying compression library.  .. ipython:: python  df = pd.DataFrame(  {  "A": np.random.randn(1000),  "B": "foo",  "C": pd.date_range("20130101", periods=1000, freq="s"),  }  )  df  Using an explicit compression type:  .. ipython:: python  df.to_pickle("data.pkl.compress", compression="gzip")  rt = pd.read_pickle("data.pkl.compress", compression="gzip")  rt  Inferring compression type from the extension:  .. ipython:: python  df.to_pickle("data.pkl.xz", compression="infer")  rt = pd.read_pickle("data.pkl.xz", compression="infer")  rt  The default is to 'infer':  .. ipython:: python  df.to_pickle("data.pkl.gz")  rt = pd.read_pickle("data.pkl.gz")  rt  df["A"].to_pickle("s1.pkl.bz2")  rt = pd.read_pickle("s1.pkl.bz2")  rt  Passing options to the compression protocol in order to speed up compression:  .. ipython:: python  df.to_pickle("data.pkl.gz", compression={"method": "gzip", "compresslevel": 1})  .. ipython:: python  os.remove("data.pkl.compress")  os.remove("data.pkl.xz")  os.remove("data.pkl.gz")  os.remove("s1.pkl.bz2")  .. _io.msgpack:

msgpack

pandas support for ``msgpack`` has been removed in version 1.0.0. It is
recommended to use `pickle <io.pickle>` instead.  Alternatively, you can also the Arrow IPC serialization format for on-the-wire
transmission of pandas objects. For documentation on pyarrow, see
here_.  .. _io.hdf5:

``HDFStore`` is a dict-like object which reads and writes pandas using
the high performance HDF5 format using the excellent `PyTables
<https://www.pytables.org/>`__ library. See the `cookbook <cookbook.hdf>`
for some advanced strategies  .. warning::  pandas uses PyTables for reading and writing HDF5 files, which allows  serializing object-dtype data with pickle. Loading pickled data received from  untrusted sources can be unsafe.  See: https://docs.python.org/3/library/pickle.html for more.  .. ipython:: python  os.remove("store.h5")  .. ipython:: python  store = pd.HDFStore("store.h5")  print(store)  Objects can be written to the file just like adding key-value pairs to a
dict:  .. ipython:: python  index = pd.date_range("1/1/2000", periods=8)  s = pd.Series(np.random.randn(5), index=["a", "b", "c", "d", "e"])  df = pd.DataFrame(np.random.randn(8, 3), index=index, columns=["A", "B", "C"])  # store.put('s', s) is an equivalent method  store["s"] = s  store["df"] = df  store  In a current or later Python session, you can retrieve stored objects:  .. ipython:: python  # store.get('df') is an equivalent method  store["df"]  # dotted (attribute) access provides get as well  store.df  Deletion of the object specified by the key:  .. ipython:: python  # store.remove('df') is an equivalent method  del store["df"]  store  Closing a Store and using a context manager:  .. ipython:: python  store.close()  store  store.is_open  # Working with, and automatically closing the store using a context manager  with pd.HDFStore("store.h5") as store:  store.keys()  .. ipython:: python  store.close()  os.remove("store.h5")  Read/write API
'''  ``HDFStore`` supports a top-level API using  ``read_hdf`` for reading and ``to_hdf`` for writing,
similar to how ``read_csv`` and ``to_csv`` work.  .. ipython:: python  df_tl = pd.DataFrame({"A": list(range(5)), "B": list(range(5))})  df_tl.to_hdf("store_tl.h5", "table", append=True)  pd.read_hdf("store_tl.h5", "table", where=["index>2"])  .. ipython:: python  os.remove("store_tl.h5")  HDFStore will by default not drop rows that are all missing. This behavior can be changed by setting ``dropna=True``.  .. ipython:: python  df_with_missing = pd.DataFrame(  {  "col1": [0, np.nan, 2],  "col2": [1, np.nan, np.nan],  }  )  df_with_missing  df_with_missing.to_hdf("file.h5", "df_with_missing", format="table", mode="w")  pd.read_hdf("file.h5", "df_with_missing")  df_with_missing.to_hdf(  "file.h5", "df_with_missing", format="table", mode="w", dropna=True  )  pd.read_hdf("file.h5", "df_with_missing")  .. ipython:: python  os.remove("file.h5")  .. _io.hdf5-fixed:  Fixed format
'''  The examples above show storing using ``put``, which write the HDF5 to ``PyTables`` in a fixed array format, called
the ``fixed`` format. These types of stores are **not** appendable once written (though you can simply
remove them and rewrite). Nor are they **queryable**; they must be
retrieved in their entirety. They also do not support dataframes with non-unique column names.
The ``fixed`` format stores offer very fast writing and slightly faster reading than ``table`` stores.
This format is specified by default when using ``put`` or ``to_hdf`` or by ``format='fixed'`` or ``format='f'``.  .. warning::  A ``fixed`` format will raise a ``TypeError`` if you try to retrieve using a ``where``:  .. code-block:: python  >>> pd.DataFrame(np.random.randn(10, 2)).to_hdf("test_fixed.h5", "df")  >>> pd.read_hdf("test_fixed.h5", "df", where="index>5")  TypeError: cannot pass a where specification when reading a fixed format.  this store must be selected in its entirety  .. _io.hdf5-table:  Table format
'''  ``HDFStore`` supports another ``PyTables`` format on disk, the ``table``
format. Conceptually a ``table`` is shaped very much like a DataFrame,
with rows and columns. A ``table`` may be appended to in the same or
other sessions.  In addition, delete and query type operations are
supported. This format is specified by ``format='table'`` or ``format='t'``

to ``append`` or ``put`` or ``to_hdf``.  This format can be set as an option as well ``pd.set_option('io.hdf.default_format','table')`` to
enable ``put/append/to_hdf`` to by default store in the ``table`` format.  .. ipython:: python  os.remove("store.h5")  .. ipython:: python  store = pd.HDFStore("store.h5")  df1 = df[0:4]  df2 = df[4:]  # append data (creates a table automatically)  store.append("df", df1)  store.append("df", df2)  store  # select the entire object  store.select("df")  # the type of stored data  store.root.df._v_attrs.pandas_type  .. note::  You can also create a ``table`` by passing ``format='table'`` or ``format='t'`` to a ``put`` operation.  .. _io.hdf5-keys:  Hierarchical keys
'''  Keys to a store can be specified as a string. These can be in a
hierarchical path-name like format (e.g. ``foo/bar/bah``), which will
generate a hierarchy of sub-stores (or ``Groups`` in PyTables
parlance). Keys can be specified without the leading '/' and are **always**
absolute (e.g. 'foo' refers to '/foo'). Removal operations can remove
everything in the sub-store and **below**, so be *careful*.  .. ipython:: python  store.put("foo/bar/bah", df)  store.append("food/orange", df)  store.append("food/apple", df)  store  # a list of keys are returned  store.keys()  # remove all nodes under this level  store.remove("food")  store  You can walk through the group hierarchy using the ``walk`` method which
will yield a tuple for each group key along with the relative keys of its contents.  .. ipython:: python  for (path, subgroups, subkeys) in store.walk():  for subgroup in subgroups:  print("GROUP: {}/{}".format(path, subgroup))  for subkey in subkeys:  key = "/".join([path, subkey])  print("KEY: {}".format(key))  print(store.get(key))  .. warning::  Hierarchical keys cannot be retrieved as dotted (attribute) access as described above for items stored under the root node.  .. code-block:: ipython  In [8]: store.foo.bar.bah  AttributeError: 'HDFStore' object has no attribute 'foo'  # you can directly access the actual PyTables node but using the root node  In [9]: store.root.foo.bar.bah  Out[9]:  /foo/bar/bah (Group) ''  children := ['block0_items' (Array), 'block0_values' (Array), 'axis0' (Array), 'axis1' (Array)]  Instead, use explicit string based keys:  .. ipython:: python  store["foo/bar/bah"]  .. _io.hdf5-types:  Storing types
'''  Storing mixed types in a table
+++  Storing mixed-dtype data is supported. Strings are stored as a
fixed-width using the maximum size of the appended column. Subsequent attempts
at appending longer strings will raise a ``ValueError``.  Passing ``min_itemsize={`values`: size}`` as a parameter to append
will set a larger minimum for the string columns. Storing ``floats,
strings, ints, bools, datetime64`` are currently supported. For string
columns, passing ``nan_rep = 'nan'`` to append will change the default
nan representation on disk (which converts to/from ``np.nan``), this
defaults to ``nan``.  .. ipython:: python  df_mixed = pd.DataFrame(  {  "A": np.random.randn(8),  "B": np.random.randn(8),  "C": np.array(np.random.randn(8), dtype="float32"),  "string": "string",  "int": 1,  "bool": True,  "datetime64": pd.Timestamp("20010102"),  },  index=list(range(8)),  )  df_mixed.loc[df_mixed.index[3:5], ["A", "B", "string", "datetime64"]] = np.nan  store.append("df_mixed", df_mixed, min_itemsize={"values": 50})  df_mixed1 = store.select("df_mixed")  df_mixed1  df_mixed1.dtypes.value_counts()  # we have provided a minimum string column size  store.root.df_mixed.table  Storing MultiIndex DataFrames
+++  Storing MultiIndex ``DataFrames`` as tables is very similar to

storing/selecting from homogeneous index ``DataFrames``.  .. ipython:: python  index = pd.MultiIndex(  levels=[["foo", "bar", "baz", "qux"], ["one", "two", "three"]],  codes=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3], [0, 1, 2, 0, 1, 1, 2, 0, 1, 2]],  names=["foo", "bar"],  )  df_mi = pd.DataFrame(np.random.randn(10, 3), index=index, columns=["A", "B", "C"])  df_mi  store.append("df_mi", df_mi)  store.select("df_mi")  # the levels are automatically included as data columns  store.select("df_mi", "foo=bar")  .. note::  The ``index`` keyword is reserved and cannot be use as a level name.  .. _io.hdf5-query:  Querying
'''  Querying a table
+++  ``select`` and ``delete`` operations have an optional criterion that can
be specified to select/delete only a subset of the data. This allows one
to have a very large on-disk table and retrieve only a portion of the
data.  A query is specified using the ``Term`` class under the hood, as a boolean expression.  * ``index`` and ``columns`` are supported indexers of ``DataFrames``.
* if ``data_columns`` are specified, these can be used as additional indexers.
* level name in a MultiIndex, with default name  ``level_0``, ``level_1``, … if not provided.  Valid comparison operators are:  ``=, ==, !=, >, >=, <, <=``  Valid boolean expressions are combined with:  * ``|`` : or
* ``&`` : and
* ``(`` and ``)`` : for grouping  These rules are similar to how boolean expressions are used in pandas for indexing.  .. note::  - ``=`` will be automatically expanded to the comparison operator ``==``  - ``~`` is the not operator, but can only be used in very limited  circumstances  - If a list/tuple of expressions is passed they will be combined via ``&``  The following are valid expressions:  * ``'index >= date'``
* ``"columns = ['A', 'D']"``
* ``"columns in ['A', 'D']"``
* ``'columns = A'``
* ``'columns == A'``
* ``"~(columns = ['A', 'B'])"``
* ``'index > df.index[3] & string = "bar"'``
* ``'(index > df.index[3] & index <= df.index[6]) | string = "bar"'``
* ``"ts >= Timestamp('2012-02-01')"``
* ``"major_axis>=20130101"``  The ``indexers`` are on the left-hand side of the sub-expression:  ``columns``, ``major_axis``, ``ts``  The right-hand side of the sub-expression (after a comparison operator) can be:  * functions that will be evaluated, e.g. ``Timestamp('2012-02-01')``
* strings, e.g. ``"bar"``
* date-like, e.g. ``20130101``, or ``"20130101"``
* lists, e.g. ``"['A', 'B']"``
* variables that are defined in the local names space, e.g. ``date``  .. note::  Passing a string to a query by interpolating it into the query  expression is not recommended. Simply assign the string of interest to a  variable and use that variable in an expression. For example, do this  .. code-block:: python  string = "HolyMoly'"  store.select("df", "index == string")  instead of this  .. code-block:: ipython  string = "HolyMoly'"  store.select('df', f'index == {string}')  The latter will **not** work and will raise a ``SyntaxError``.Note that  there's a single quote followed by a double quote in the ``string``  variable.  If you *must* interpolate, use the ``'%r'`` format specifier  .. code-block:: python  store.select("df", "index == %r" % string)  which will quote ``string``.  Here are some examples:  .. ipython:: python  dfq = pd.DataFrame(  np.random.randn(10, 4),  columns=list("ABCD"),  index=pd.date_range("20130101", periods=10),  )  store.append("dfq", dfq, format="table", data_columns=True)  Use boolean expressions, with in-line function evaluation.  .. ipython:: python  store.select("dfq", "index>pd.Timestamp('20130104') & columns=['A', 'B']")  Use inline column reference.  .. ipython:: python  store.select("dfq", where="A>0 or C>0")  The ``columns`` keyword can be supplied to select a list of columns to be
returned, this is equivalent to passing a

``'columns=list_of_columns_to_filter'``:  .. ipython:: python  store.select("df", "columns=['A', 'B']")  ``start`` and ``stop`` parameters can be specified to limit the total search
space. These are in terms of the total number of rows in a table.  .. note::  ``select`` will raise a ``ValueError`` if the query expression has an unknown  variable reference. Usually this means that you are trying to select on a column  that is **not** a data_column.  ``select`` will raise a ``SyntaxError`` if the query expression is not valid.  .. _io.hdf5-timedelta:  Query timedelta64[ns]
+++  You can store and query using the ``timedelta64[ns]`` type. Terms can be
specified in the format: ``<float>(<unit>)``, where float may be signed (and fractional), and unit can be
``D,s,ms,us,ns`` for the timedelta. Here's an example:  .. ipython:: python  from datetime import timedelta  dftd = pd.DataFrame(  {  "A": pd.Timestamp("20130101"),  "B": [  pd.Timestamp("20130101") + timedelta(days=i, seconds=10)  for i in range(10)  ],  }  )  dftd["C"] = dftd["A"] - dftd["B"]  dftd  store.append("dftd", dftd, data_columns=True)  store.select("dftd", "C<'-3.5D'")  .. _io.query_multi:  Query MultiIndex
+++  Selecting from a ``MultiIndex`` can be achieved by using the name of the level.  .. ipython:: python  df_mi.index.names  store.select("df_mi", "foo=baz and bar=two")  If the ``MultiIndex`` levels names are ``None``, the levels are automatically made available via
the ``level_n`` keyword with ``n`` the level of the ``MultiIndex`` you want to select from.  .. ipython:: python  index = pd.MultiIndex(  levels=[["foo", "bar", "baz", "qux"], ["one", "two", "three"]],  codes=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3], [0, 1, 2, 0, 1, 1, 2, 0, 1, 2]],  )  df_mi_2 = pd.DataFrame(np.random.randn(10, 3), index=index, columns=["A", "B", "C"])  df_mi_2  store.append("df_mi_2", df_mi_2)  # the levels are automatically included as data columns with keyword level_n  store.select("df_mi_2", "level_0=foo and level_1=two")  Indexing
+++  You can create/modify an index for a table with ``create_table_index``
after data is already in the table (after and ``append/put``
operation). Creating a table index is **highly** encouraged. This will
speed your queries a great deal when you use a ``select`` with the
indexed dimension as the ``where``.  .. note::  Indexes are automagically created on the indexables  and any data columns you specify. This behavior can be turned off by passing  ``index=False`` to ``append``.  .. ipython:: python  # we have automagically already created an index (in the first section)  i = store.root.df.table.cols.index.index  i.optlevel, i.kind  # change an index by passing new parameters  store.create_table_index("df", optlevel=9, kind="full")  i = store.root.df.table.cols.index.index  i.optlevel, i.kind  Oftentimes when appending large amounts of data to a store, it is useful to turn off index creation for each append, then recreate at the end.  .. ipython:: python  df_1 = pd.DataFrame(np.random.randn(10, 2), columns=list("AB"))  df_2 = pd.DataFrame(np.random.randn(10, 2), columns=list("AB"))  st = pd.HDFStore("appends.h5", mode="w")  st.append("df", df_1, data_columns=["B"], index=False)  st.append("df", df_2, data_columns=["B"], index=False)  st.get_storer("df").table  Then create the index when finished appending.  .. ipython:: python  st.create_table_index("df", columns=["B"], optlevel=9, kind="full")  st.get_storer("df").table  st.close()  .. ipython:: python  os.remove("appends.h5")  See here_ for how to create a completely-sorted-index (CSI) on an existing store.  .. _io.hdf5-query-data-columns:  Query via data columns
+++  You can designate (and index) certain columns that you want to be able
to perform queries (other than the ``indexable`` columns, which you can
always query). For instance say you want to perform this common
operation, on-disk, and return just the frame that matches this

query. You can specify ``data_columns = True`` to force all columns to
be ``data_columns``.  .. ipython:: python  df_dc = df.copy()  df_dc["string"] = "foo"  df_dc.loc[df_dc.index[4:6], "string"] = np.nan  df_dc.loc[df_dc.index[7:9], "string"] = "bar"  df_dc["string2"] = "cool"  df_dc.loc[df_dc.index[1:3], ["B", "C"]] = 1.0  df_dc  # on-disk operations  store.append("df_dc", df_dc, data_columns=["B", "C", "string", "string2"])  store.select("df_dc", where="B > 0")  # getting creative  store.select("df_dc", "B > 0 & C > 0 & string == foo")  # this is in-memory version of this type of selection  df_dc[(df_dc.B > 0) & (df_dc.C > 0) & (df_dc.string == "foo")]  # we have automagically created this index and the B/C/string/string2  # columns are stored separately as ``PyTables`` columns  store.root.df_dc.table  There is some performance degradation by making lots of columns into
``data columns``, so it is up to the user to designate these. In addition,
you cannot change data columns (nor indexables) after the first
append/put operation (Of course you can simply read in the data and
create a new table!).  Iterator
+++  You can pass ``iterator=True`` or ``chunksize=number_in_a_chunk``
to ``select`` and ``select_as_multiple`` to return an iterator on the results.
The default is 50,000 rows returned in a chunk.  .. ipython:: python  for df in store.select("df", chunksize=3):  print(df)  .. note::  You can also use the iterator with ``read_hdf`` which will open, then  automatically close the store when finished iterating.  .. code-block:: python  for df in pd.read_hdf("store.h5", "df", chunksize=3):  print(df)  Note, that the chunksize keyword applies to the **source** rows. So if you
are doing a query, then the chunksize will subdivide the total rows in the table
and the query applied, returning an iterator on potentially unequal sized chunks.  Here is a recipe for generating a query and using it to create equal sized return
chunks.  .. ipython:: python  dfeq = pd.DataFrame({"number": np.arange(1, 11)})  dfeq  store.append("dfeq", dfeq, data_columns=["number"])  def chunks(l, n):  return [l[i: i + n] for i in range(0, len(l), n)]  evens = [2, 4, 6, 8, 10]  coordinates = store.select_as_coordinates("dfeq", "number=evens")  for c in chunks(coordinates, 2):  print(store.select("dfeq", where=c))  Advanced queries
+++  Select a single column
^^^  To retrieve a single indexable or data column, use the
method ``select_column``. This will, for example, enable you to get the index
very quickly. These return a ``Series`` of the result, indexed by the row number.
These do not currently accept the ``where`` selector.  .. ipython:: python  store.select_column("df_dc", "index")  store.select_column("df_dc", "string")  .. _io.hdf5-selecting_coordinates:  Selecting coordinates
^^^  Sometimes you want to get the coordinates (a.k.a the index locations) of your query. This returns an
``Index`` of the resulting locations. These coordinates can also be passed to subsequent
``where`` operations.  .. ipython:: python  df_coord = pd.DataFrame(  np.random.randn(1000, 2), index=pd.date_range("2000101", periods=1000)  )  store.append("df_coord", df_coord)  c = store.select_as_coordinates("df_coord", "index > 20020101")  c  store.select("df_coord", where=c)  .. _io.hdf5-where_mask:  Selecting using a where mask
^^^  Sometime your query can involve creating a list of rows to select. Usually this ``mask`` would
be a resulting ``index`` from an indexing operation. This example selects the months of

a datetimeindex which are 5.  .. ipython:: python  df_mask = pd.DataFrame(  np.random.randn(1000, 2), index=pd.date_range("2000101", periods=1000)  )  store.append("df_mask", df_mask)  c = store.select_column("df_mask", "index")  where = c[pd.DatetimeIndex(c).month == 5].index  store.select("df_mask", where=where)  Storer object
^^^  If you want to inspect the stored object, retrieve via
``get_storer``. You could use this programmatically to say get the number
of rows in an object.  .. ipython:: python  store.get_storer("df_dc").nrows  Multiple table queries
+++  The methods ``append_to_multiple`` and
``select_as_multiple`` can perform appending/selecting from
multiple tables at once. The idea is to have one table (call it the
selector table) that you index most/all of the columns, and perform your
queries. The other table(s) are data tables with an index matching the
selector table's index. You can then perform a very fast query
on the selector table, yet get lots of data back. This method is similar to
having a very wide table, but enables more efficient queries.  The ``append_to_multiple`` method splits a given single DataFrame
into multiple tables according to ``d``, a dictionary that maps the
table names to a list of 'columns' you want in that table. If ``None``
is used in place of a list, that table will have the remaining
unspecified columns of the given DataFrame. The argument ``selector``
defines which table is the selector table (which you can make queries from).
The argument ``dropna`` will drop rows from the input ``DataFrame`` to ensure
tables are synchronized.  This means that if a row for one of the tables
being written to is entirely ``np.NaN``, that row will be dropped from all tables.  If ``dropna`` is False, **THE USER IS RESPONSIBLE FOR SYNCHRONIZING THE TABLES**.
Remember that entirely ``np.Nan`` rows are not written to the HDFStore, so if
you choose to call ``dropna=False``, some tables may have more rows than others,
and therefore ``select_as_multiple`` may not work or it may return unexpected
results.  .. ipython:: python  df_mt = pd.DataFrame(  np.random.randn(8, 6),  index=pd.date_range("1/1/2000", periods=8),  columns=["A", "B", "C", "D", "E", "F"],  )  df_mt["foo"] = "bar"  df_mt.loc[df_mt.index[1], ("A", "B")] = np.nan  # you can also create the tables individually  store.append_to_multiple(  {"df1_mt": ["A", "B"], "df2_mt": None}, df_mt, selector="df1_mt"  )  store  # individual tables were created  store.select("df1_mt")  store.select("df2_mt")  # as a multiple  store.select_as_multiple(  ["df1_mt", "df2_mt"],  where=["A>0", "B>0"],  selector="df1_mt",  )  Delete from a table
'''  You can delete from a table selectively by specifying a ``where``. In
deleting rows, it is important to understand the ``PyTables`` deletes
rows by erasing the rows, then **moving** the following data. Thus
deleting can potentially be a very expensive operation depending on the
orientation of your data. To get optimal performance, it's
worthwhile to have the dimension you are deleting be the first of the
``indexables``.  Data is ordered (on the disk) in terms of the ``indexables``. Here's a
simple use case. You store panel-type data, with dates in the
``major_axis`` and ids in the ``minor_axis``. The data is then
interleaved like this:  * date_1  * id_1  * id_2  *  .  * id_n
* date_2  * id_1  *  .  * id_n  It should be clear that a delete operation on the ``major_axis`` will be
fairly quick, as one chunk is removed, then the following data moved. On
the other hand a delete operation on the ``minor_axis`` will be very
expensive. In this case it would almost certainly be faster to rewrite

the table using a ``where`` that selects all but the missing data.  .. warning::  Please note that HDF5 **DOES NOT RECLAIM SPACE** in the h5 files  automatically. Thus, repeatedly deleting (or removing nodes) and adding  again, **WILL TEND TO INCREASE THE FILE SIZE**.  To *repack and clean* the file, use `ptrepack <io.hdf5-ptrepack>`.  .. _io.hdf5-notes:  Notes & caveats
'''  Compression
+++  ``PyTables`` allows the stored data to be compressed. This applies to
all kinds of stores, not just tables. Two parameters are used to
control compression: ``complevel`` and ``complib``.  * ``complevel`` specifies if and how hard data is to be compressed.  ``complevel=0`` and ``complevel=None`` disables compression and  ``0<complevel<10`` enables compression.  * ``complib`` specifies which compression library to use.  If nothing is  specified the default library ``zlib`` is used. A  compression library usually optimizes for either good compression rates  or speed and the results will depend on the type of data. Which type of  compression to choose depends on your specific needs and data. The list  of supported compression libraries:  - zlib: The default compression library.  A classic in terms of compression, achieves good compression  rates but is somewhat slow.  - lzo: Fast  compression and decompression.  - bzip2: Good compression rates.  - blosc: Fast compression and  decompression.  Support for alternative blosc compressors:  - blosc:blosclz This is the  default compressor for ``blosc``  - `blosc:lz4  <https://fastcompression.blogspot.com/p/lz4.html>`_:  A compact, very popular and fast compressor.  - `blosc:lz4hc  <https://fastcompression.blogspot.com/p/lz4.html>`_:  A tweaked version of LZ4, produces better  compression ratios at the expense of speed.  - blosc  A popular compressor used in many places.  - blosc A classic;  somewhat slower than the previous ones, but  achieving better compression ratios.  - blosc An  extremely well balanced codec; it provides the best  compression ratios among the others above, and at  reasonably fast speed.  If ``complib`` is defined as something other than the listed libraries a  ``ValueError`` exception is issued.  .. note::  If the library specified with the ``complib`` option is missing on your platform,  compression defaults to ``zlib`` without further ado.  Enable compression for all objects within the file:  .. code-block:: python  store_compressed = pd.HDFStore(  "store_compressed.h5", complevel=9, complib="blosc:blosclz"  )  Or on-the-fly compression (this only applies to tables) in stores where compression is not enabled:  .. code-block:: python  store.append("df", df, complib="zlib", complevel=5)  .. _io.hdf5-ptrepack:  ptrepack
+++  ``PyTables`` offers better write performance when tables are compressed after
they are written, as opposed to turning on compression at the very
beginning. You can use the supplied ``PyTables`` utility
``ptrepack``. In addition, ``ptrepack`` can change compression levels
after the fact.  .. code-block:: console  ptrepack --chunkshape=auto --propindexes --complevel=9 --complib=blosc in.h5 out.h5  Furthermore ``ptrepack in.h5 out.h5`` will *repack* the file to allow
you to reuse previously deleted space. Alternatively, one can simply
remove the file and write again, or use the ``copy`` method.  .. _io.hdf5-caveats:  Caveats

+++  .. warning::  ``HDFStore`` is **not-threadsafe for writing**. The underlying  ``PyTables`` only supports concurrent reads (via threading or  processes). If you need reading and writing *at the same time*, you  need to serialize these operations in a single thread in a single  process. You will corrupt your data otherwise. See the (`2397`) for more information.  * If you use locks to manage write access between multiple processes, you  may want to use func:`~os.fsync` before releasing write locks. For  convenience you can use ``store.flush(fsync=True)`` to do this for you.
* Once a ``table`` is created columns (DataFrame)  are fixed; only exactly the same columns can be appended
* Be aware that timezones (e.g., ``pytz.timezone('US/Eastern')``)  are not necessarily equal across timezone versions.  So if data is  localized to a specific timezone in the HDFStore using one version  of a timezone library and that data is updated with another version, the data  will be converted to UTC since these timezones are not considered  equal.  Either use the same version of timezone library or use ``tz_convert`` with  the updated timezone definition.  .. warning::  ``PyTables`` will show a ``NaturalNameWarning`` if a column name  cannot be used as an attribute selector.  *Natural* identifiers contain only letters, numbers, and underscores,  and may not begin with a number.  Other identifiers cannot be used in a ``where`` clause  and are generally a bad idea.  .. _io.hdf5-data_types:  DataTypes
'''  ``HDFStore`` will map an object dtype to the ``PyTables`` underlying
dtype. This means the following types are known to work:  ===  ===
Type  Represents missing values
===  ===
floating : ``float64, float32, float16``  ``np.nan``
integer : ``int64, int32, int8, uint64,uint32, uint8``
boolean
``datetime64[ns]``  ``NaT``
``timedelta64[ns]``  ``NaT``
categorical : see the section below
object : ``strings``  ``np.nan``
===  ===  ``unicode`` columns are not supported, and **WILL FAIL**.  .. _io.hdf5-categorical:  Categorical data
+++  You can write data that contains ``category`` dtypes to a ``HDFStore``.
Queries work the same as if it was an object array. However, the ``category`` dtyped data is
stored in a more efficient manner.  .. ipython:: python  dfcat = pd.DataFrame(  {"A": pd.Series(list("aabbcdba")).astype("category"), "B": np.random.randn(8)}  )  dfcat  dfcat.dtypes  cstore = pd.HDFStore("cats.h5", mode="w")  cstore.append("dfcat", dfcat, format="table", data_columns=["A"])  result = cstore.select("dfcat", where="A in ['b', 'c']")  result  result.dtypes  .. ipython:: python  cstore.close()  os.remove("cats.h5")  String columns
+++  **min_itemsize**  The underlying implementation of ``HDFStore`` uses a fixed column width (itemsize) for string columns.
A string column itemsize is calculated as the maximum of the
length of data (for that column) that is passed to the ``HDFStore``, **in the first append**. Subsequent appends,
may introduce a string for a column **larger** than the column can hold, an Exception will be raised (otherwise you
could have a silent truncation of these columns, leading to loss of information). In the future we may relax this and
allow a user-specified truncation to occur.  Pass ``min_itemsize`` on the first table creation to a-priori specify the minimum length of a particular string column.
``min_itemsize`` can be an integer, or a dict mapping a column name to an integer. You can pass ``values`` as a key to

allow all *indexables* or *data_columns* to have this min_itemsize.  Passing a ``min_itemsize`` dict will cause all passed columns to be created as *data_columns* automatically.  .. note::  If you are not passing any ``data_columns``, then the ``min_itemsize`` will be the maximum of the length of any string passed  .. ipython:: python  dfs = pd.DataFrame({"A": "foo", "B": "bar"}, index=list(range(5)))  dfs  # A and B have a size of 30  store.append("dfs", dfs, min_itemsize=30)  store.get_storer("dfs").table  # A is created as a data_column with a size of 30  # B is size is calculated  store.append("dfs2", dfs, min_itemsize={"A": 30})  store.get_storer("dfs2").table  **nan_rep**  String columns will serialize a ``np.nan`` (a missing value) with the ``nan_rep`` string representation. This defaults to the string value ``nan``.
You could inadvertently turn an actual ``nan`` value into a missing value.  .. ipython:: python  dfss = pd.DataFrame({"A": ["foo", "bar", "nan"]})  dfss  store.append("dfss", dfss)  store.select("dfss")  # here you need to specify a different nan rep  store.append("dfss2", dfss, nan_rep="_nan_")  store.select("dfss2")  Performance
'''  * ``tables`` format come with a writing performance penalty as compared to  ``fixed`` stores. The benefit is the ability to append/delete and  query (potentially very large amounts of data).  Write times are  generally longer as compared with regular stores. Query times can  be quite fast, especially on an indexed axis.
* You can pass ``chunksize=<int>`` to ``append``, specifying the  write chunksize (default is 5000). This will significantly lower  your memory usage on writing.
* You can pass ``expectedrows=<int>`` to the first ``append``,  to set the TOTAL number of rows that ``PyTables`` will expect.  This will optimize read/write performance.
* Duplicate rows can be written to tables, but are filtered out in  selection (with the last items being selected; thus a table is  unique on major, minor pairs)
* A ``PerformanceWarning`` will be raised if you are attempting to  store types that will be pickled by PyTables (rather than stored as  endemic types). See  Here_  for more information and some solutions.  .. ipython:: python  store.close()  os.remove("store.h5")  .. _io.feather:

Feather

Feather provides binary columnar serialization for data frames. It is designed to make reading and writing data
frames efficient, and to make sharing data across data analysis languages easy.  Feather is designed to faithfully serialize and de-serialize DataFrames, supporting all of the pandas
dtypes, including extension dtypes such as categorical and datetime with tz.  Several caveats:  * The format will NOT write an ``Index``, or ``MultiIndex`` for the  ``DataFrame`` and will raise an error if a non-default one is provided. You  can ``.reset_index()`` to store the index or ``.reset_index(drop=True)`` to  ignore it.
* Duplicate column names and non-string columns names are not supported
* Actual Python objects in object dtype columns are not supported. These will  raise a helpful error message on an attempt at serialization.  See the Full Documentation_.  .. ipython:: python  df = pd.DataFrame(  {  "a": list("abc"),  "b": list(range(1, 4)),  "c": np.arange(3, 6).astype("u1"),  "d": np.arange(4.0, 7.0, dtype="float64"),  "e": [True, False, True],  "f": pd.Categorical(list("abc")),  "g": pd.date_range("20130101", periods=3),  "h": pd.date_range("20130101", periods=3, tz="US/Eastern"),  "i": pd.date_range("20130101", periods=3, freq="ns"),  }  )  df  df.dtypes  Write to a feather file.  .. ipython:: python  df.to_feather("example.feather")  Read from a feather file.  .. ipython:: python  result = pd.read_feather("example.feather")  result  # we preserve dtypes  result.dtypes  .. ipython:: python  os.remove("example.feather")  .. _io.parquet:

Apache Parquet_ provides a partitioned binary columnar serialization for data frames. It is designed to
make reading and writing data frames efficient, and to make sharing data across data analysis
languages easy. Parquet can use a variety of compression techniques to shrink the file size as much as possible
while still maintaining good read performance.  Parquet is designed to faithfully serialize and de-serialize ``DataFrame`` s, supporting all of the pandas
dtypes, including extension dtypes such as datetime with tz.  Several caveats.  * Duplicate column names and non-string columns names are not supported.
* The ``pyarrow`` engine always writes the index to the output, but ``fastparquet`` only writes non-default  indexes. This extra column can cause problems for non-pandas consumers that are not expecting it. You can  force including or omitting indexes with the ``index`` argument, regardless of the underlying engine.
* Index level names, if specified, must be strings.
* In the ``pyarrow`` engine, categorical dtypes for non-string types can be serialized to parquet, but will de-serialize as their primitive dtype.
* The ``pyarrow`` engine preserves the ``ordered`` flag of categorical dtypes with string types. ``fastparquet`` does not preserve the ``ordered`` flag.
* Non supported types include ``Interval`` and actual Python object types. These will raise a helpful error message  on an attempt at serialization. ``Period`` type is supported with pyarrow >= 0.16.0.
* The ``pyarrow`` engine preserves extension data types such as the nullable integer and string data  type (requiring pyarrow >= 0.16.0, and requiring the extension type to implement the needed protocols,  see the `extension types documentation <extending.extension.arrow>`).  You can specify an ``engine`` to direct the serialization. This can be one of ``pyarrow``, or ``fastparquet``, or ``auto``.
If the engine is NOT specified, then the ``pd.options.io.parquet.engine`` option is checked; if this is also ``auto``,
then ``pyarrow`` is tried, and falling back to ``fastparquet``.  See the documentation for pyarrow_ and fastparquet_.  .. note::  These engines are very similar and should read/write nearly identical parquet format files.  ``pyarrow>=8.0.0`` supports timedelta data, ``fastparquet>=0.1.4`` supports timezone aware datetimes.  These libraries differ by having different underlying dependencies (``fastparquet`` by using ``numba``, while ``pyarrow`` uses a c-library).  .. ipython:: python  df = pd.DataFrame(  {  "a": list("abc"),  "b": list(range(1, 4)),  "c": np.arange(3, 6).astype("u1"),  "d": np.arange(4.0, 7.0, dtype="float64"),  "e": [True, False, True],  "f": pd.date_range("20130101", periods=3),  "g": pd.date_range("20130101", periods=3, tz="US/Eastern"),  "h": pd.Categorical(list("abc")),  "i": pd.Categorical(list("abc"), ordered=True),  }  )  df  df.dtypes  Write to a parquet file.  .. ipython:: python  df.to_parquet("example_pa.parquet", engine="pyarrow")  df.to_parquet("example_fp.parquet", engine="fastparquet")  Read from a parquet file.  .. ipython:: python  result = pd.read_parquet("example_fp.parquet", engine="fastparquet")  result = pd.read_parquet("example_pa.parquet", engine="pyarrow")  result.dtypes  Read only certain columns of a parquet file.  .. ipython:: python  result = pd.read_parquet(  "example_fp.parquet",  engine="fastparquet",  columns=["a", "b"],  )  result = pd.read_parquet(  "example_pa.parquet",  engine="pyarrow",  columns=["a", "b"],  )  result.dtypes  .. ipython:: python  os.remove("example_pa.parquet")  os.remove("example_fp.parquet")  Handling indexes
'''  Serializing a ``DataFrame`` to parquet may include the implicit index as one or
more columns in the output file. Thus, this code:  .. ipython:: python  df = pd.DataFrame({"a": [1, 2], "b": [3, 4]})  df.to_parquet("test.parquet", engine="pyarrow")  creates a parquet file with *three* columns if you use ``pyarrow`` for serialization:

``a``, ``b``, and ``__index_level_0__``. If you're using ``fastparquet``, the
index may or may not
be written to the file.  This unexpected extra column causes some databases like Amazon Redshift to reject
the file, because that column doesn't exist in the target table.  If you want to omit a dataframe's indexes when writing, pass ``index=False`` to
`~pandas.DataFrame.to_parquet`:  .. ipython:: python  df.to_parquet("test.parquet", index=False)  This creates a parquet file with just the two expected columns, ``a`` and ``b``.
If your ``DataFrame`` has a custom index, you won't get it back when you load
this file into a ``DataFrame``.  Passing ``index=True`` will *always* write the index, even if that's not the
underlying engine's default behavior.  .. ipython:: python  os.remove("test.parquet")  Partitioning Parquet files
'''  Parquet supports partitioning of data based on the values of one or more columns.  .. ipython:: python  df = pd.DataFrame({"a": [0, 0, 1, 1], "b": [0, 1, 0, 1]})  df.to_parquet(path="test", engine="pyarrow", partition_cols=["a"], compression=None)  The ``path`` specifies the parent directory to which data will be saved.
The ``partition_cols`` are the column names by which the dataset will be partitioned.
Columns are partitioned in the order they are given. The partition splits are
determined by the unique values in the partition columns.
The above example creates a partitioned dataset that may look like:  .. code-block:: text  test  ├── a=0  │  ├── 0bac803e32dc42ae83fddfd029cbdebc.parquet  │  └──  ...  └── a=1  ├── e6ab24a4f45147b49b54a662f0c412a3.parquet  └── ...  .. ipython:: python  from shutil import rmtree  try:  rmtree("test")  except OSError:  pass  .. _io.orc:

ORC

.. versionadded:: 1.0.0  Similar to the parquet_ is a binary columnar serialization
for data frames. It is designed to make reading data frames efficient. pandas provides both the reader and the writer for the
ORC format, ~pandas.read_orc` and `~pandas.DataFrame.to_orc`. This requires the `pyarrow_ library.  .. warning::  * It is *highly recommended* to install pyarrow using conda due to some issues occurred by pyarrow.  * `~pandas.DataFrame.to_orc` requires pyarrow>=7.0.0.  * `~pandas.read_orc` and `~pandas.DataFrame.to_orc` are not supported on Windows yet, you can find valid environments on `install optional dependencies <install.warn_orc>`.  * For supported dtypes please refer to supported ORC features in Arrow_.  * Currently timezones in datetime columns are not preserved when a dataframe is converted into ORC files.  .. ipython:: python  df = pd.DataFrame(  {  "a": list("abc"),  "b": list(range(1, 4)),  "c": np.arange(4.0, 7.0, dtype="float64"),  "d": [True, False, True],  "e": pd.date_range("20130101", periods=3),  }  )  df  df.dtypes  Write to an orc file.  .. ipython:: python  df.to_orc("example_pa.orc", engine="pyarrow")  Read from an orc file.  .. ipython:: python  result = pd.read_orc("example_pa.orc")  result.dtypes  Read only certain columns of an orc file.  .. ipython:: python  result = pd.read_orc(  "example_pa.orc",  columns=["a", "b"],  )  result.dtypes  .. ipython:: python  os.remove("example_pa.orc")  .. _io.sql:

The `pandas.io.sql` module provides a collection of query wrappers to both
facilitate data retrieval and to reduce dependency on DB-specific API. Database abstraction
is provided by SQLAlchemy if installed. In addition you will need a driver library for
your database. Examples of such drivers are psycopg2_
for PostgreSQL or pymysql_ for MySQL.
For SQLite_ this is
included in Python's standard library by default.
You can find an overview of supported drivers for each SQL dialect in the
SQLAlchemy docs_.  If SQLAlchemy is not installed, you can use a `sqlite3.Connection` in place of
a SQLAlchemy engine, connection, or URI string.  See also some `cookbook examples <cookbook.sql>` for some advanced strategies.  The key functions are:  .. autosummary::  read_sql_table  read_sql_query  read_sql  DataFrame.to_sql  .. note::  The function `~pandas.read_sql` is a convenience wrapper around  `~pandas.read_sql_table` and `~pandas.read_sql_query` (and for  backward compatibility) and will delegate to specific function depending on  the provided input (database table name or sql query).  Table names do not need to be quoted if they have special characters.  In the following example, we use the SQlite_ SQL database
engine. You can use a temporary SQLite database where data are stored in
"memory".  To connect with SQLAlchemy you use the `create_engine` function to create an engine
object from database URI. You only need to create the engine once per database you are
connecting to.
For more information on `create_engine` and the URI formatting, see the examples
below and the SQLAlchemy documentation_  .. ipython:: python  from sqlalchemy import create_engine  # Create your engine.  engine = create_engine("sqlite:///")  If you want to manage your own connections you can pass one of those instead. The example below opens a
connection to the database using a Python context manager that automatically closes the connection after
the block has completed.
See the SQLAlchemy docs_
for an explanation of how the database connection is handled.  .. code-block:: python  with engine.connect() as conn, conn.begin():  data = pd.read_sql_table("data", conn)  .. warning::  When you open a connection to a database you are also responsible for closing it.  Side effects of leaving a connection open may include locking the database or  other breaking behaviour.  Writing DataFrames
'''  Assuming the following data is in a ``DataFrame`` ``data``, we can insert it into
the database using `~pandas.DataFrame.to_sql`.  +---+---+---+---+---+
| id  |  Date  | Col_1 | Col_2 | Col_3 |
+===+===+===+===+===+
| 26  | 2012-10-18 |  X  |  25.7 | True  |
| 42  | 2012-10-19 |  Y  | -12.4 | False |
| 63  | 2012-10-20 |  Z  |  5.73 | True  |
+---+---+---+---+---+  .. ipython:: python  import datetime  c = ["id", "Date", "Col_1", "Col_2", "Col_3"]  d = [  (26, datetime.datetime(2010, 10, 18), "X", 27.5, True),  (42, datetime.datetime(2010, 10, 19), "Y", -12.5, False),  (63, datetime.datetime(2010, 10, 20), "Z", 5.73, True),  ]  data = pd.DataFrame(d, columns=c)  data  data.to_sql("data", engine)  With some databases, writing large DataFrames can result in errors due to
packet size limitations being exceeded. This can be avoided by setting the
``chunksize`` parameter when calling ``to_sql``.  For example, the following
writes ``data`` to the database in batches of 1000 rows at a time:  .. ipython:: python  data.to_sql("data_chunked", engine, chunksize=1000)  SQL data types
+++  `~pandas.DataFrame.to_sql` will try to map your data to an appropriate
SQL data type based on the dtype of the data. When you have columns of dtype
``object``, pandas will try to infer the data type.  You can always override the default type by specifying the desired SQL type of
any of the columns by using the ``dtype`` argument. This argument needs a
dictionary mapping column names to SQLAlchemy types (or strings for the sqlite3
fallback mode).

For example, specifying to use the sqlalchemy ``String`` type instead of the
default ``Text`` type for string columns:  .. ipython:: python  from sqlalchemy.types import String  data.to_sql("data_dtype", engine, dtype={"Col_1": String})  .. note::  Due to the limited support for timedelta's in the different database  flavors, columns with type ``timedelta64`` will be written as integer  values as nanoseconds to the database and a warning will be raised.  .. note::  Columns of ``category`` dtype will be converted to the dense representation  as you would get with ``np.asarray(categorical)`` (e.g. for string categories  this gives an array of strings).  Because of this, reading the database table back in does **not** generate  a categorical.  .. _io.sql_datetime_data:  Datetime data types
'''  Using SQLAlchemy, `~pandas.DataFrame.to_sql` is capable of writing
datetime data that is timezone naive or timezone aware. However, the resulting
data stored in the database ultimately depends on the supported data type
for datetime data of the database system being used.  The following table lists supported data types for datetime data for some
common databases. Other database dialects may have different data types for
datetime data.  ===  ===  ===
Database  SQL Datetime Types  Timezone Support
===  ===  ===
SQLite  ``TEXT``  No
MySQL  ``TIMESTAMP`` or ``DATETIME``  No
PostgreSQL  ``TIMESTAMP`` or ``TIMESTAMP WITH TIME ZONE``  Yes
===  ===  ===  When writing timezone aware data to databases that do not support timezones,
the data will be written as timezone naive timestamps that are in local time
with respect to the timezone.  `~pandas.read_sql_table` is also capable of reading datetime data that is
timezone aware or naive. When reading ``TIMESTAMP WITH TIME ZONE`` types, pandas
will convert the data to UTC.  .. _io.sql.method:  Insertion method
+++  The parameter ``method`` controls the SQL insertion clause used.
Possible values are:  - ``None``: Uses standard SQL ``INSERT`` clause (one per row).
- ``'multi'``: Pass multiple values in a single ``INSERT`` clause.  It uses a *special* SQL syntax not supported by all backends.  This usually provides better performance for analytic databases  like *Presto* and *Redshift*, but has worse performance for  traditional SQL backend if the table contains many columns.  For more information check the SQLAlchemy `documentation  <https://docs.sqlalchemy.org/en/latest/core/dml.html#sqlalchemy.sql.expression.Insert.values.params.*args>`__.
- callable with signature ``(pd_table, conn, keys, data_iter)``:  This can be used to implement a more performant insertion method based on  specific backend dialect features.  Example of a callable using PostgreSQL `COPY clause
<https://www.postgresql.org/docs/current/sql-copy.html>`__::  # Alternative to_sql() *method* for DBs that support COPY FROM  import csv  from io import StringIO  def psql_insert_copy(table, conn, keys, data_iter):  """  Execute SQL statement inserting data

  table : pandas.io.sql.SQLTable  conn : sqlalchemy.engine.Engine or sqlalchemy.engine.Connection  keys : list of str  Column names  data_iter : Iterable that iterates the values to be inserted  """  # gets a DBAPI connection that can provide a cursor  dbapi_conn = conn.connection  with dbapi_conn.cursor() as cur:  s_buf = StringIO()  writer = csv.writer(s_buf)  writer.writerows(data_iter)  s_buf.seek(0)  columns = ', '.join(['"{}"'.format(k) for k in keys])  if table.schema:  table_name = '{}.{}'.format(table.schema, table.name)  else:  table_name = table.name  sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(  table_name, columns)  cur.copy_expert(sql=sql, file=s_buf)  Reading tables
'''  `~pandas.read_sql_table` will read a database table given the
table name and optionally a subset of columns to read.  .. note::  In order to use `~pandas.read_sql_table`, you **must** have the  SQLAlchemy optional dependency installed.  .. ipython:: python  pd.read_sql_table("data", engine)  .. note::  Note that pandas infers column dtypes from query outputs, and not by looking  up data types in the physical database schema. For example, assume ``userid``  is an integer column in a table. Then, intuitively, ``select userid ...`` will  return integer-valued series, while ``select cast(userid as text) ...`` will  return object-valued (str) series. Accordingly, if the query output is empty,  then all resulting columns will be returned as object-valued (since they are  most general). If you foresee that your query will sometimes generate an empty  result, you may want to explicitly typecast afterwards to ensure dtype  integrity.  You can also specify the name of the column as the ``DataFrame`` index,
and specify a subset of columns to be read.  .. ipython:: python  pd.read_sql_table("data", engine, index_col="id")  pd.read_sql_table("data", engine, columns=["Col_1", "Col_2"])  And you can explicitly force columns to be parsed as dates:  .. ipython:: python  pd.read_sql_table("data", engine, parse_dates=["Date"])  If needed you can explicitly specify a format string, or a dict of arguments
to pass to `pandas.to_datetime`:  .. code-block:: python  pd.read_sql_table("data", engine, parse_dates={"Date": "%Y-%m-%d"})  pd.read_sql_table(  "data",  engine,  parse_dates={"Date": {"format": "%Y-%m-%d %H:%M:%S"}},  )  You can check if a table exists using `~pandas.io.sql.has_table`  Schema support
'''  Reading from and writing to different schema's is supported through the ``schema``
keyword in the `~pandas.read_sql_table` and `~pandas.DataFrame.to_sql`
functions. Note however that this depends on the database flavor (sqlite does not
have schema's). For example:  .. code-block:: python  df.to_sql("table", engine, schema="other_schema")  pd.read_sql_table("table", engine, schema="other_schema")  Querying
'''  You can query using raw SQL in the `~pandas.read_sql_query` function.
In this case you must use the SQL variant appropriate for your database.
When using SQLAlchemy, you can also pass SQLAlchemy Expression language constructs,
which are database-agnostic.  .. ipython:: python  pd.read_sql_query("SELECT * FROM data", engine)  Of course, you can specify a more "complex" query.  .. ipython:: python  pd.read_sql_query("SELECT id, Col_1, Col_2 FROM data WHERE id = 42;", engine)  The `~pandas.read_sql_query` function supports a ``chunksize`` argument.
Specifying this will return an iterator through chunks of the query result:  .. ipython:: python  df = pd.DataFrame(np.random.randn(20, 3), columns=list("abc"))  df.to_sql("data_chunks", engine, index=False)  .. ipython:: python  for chunk in pd.read_sql_query("SELECT * FROM data_chunks", engine, chunksize=5):  print(chunk)  Engine connection examples
'''  To connect with SQLAlchemy you use the `create_engine` function to create an engine

connecting to.  .. code-block:: python  from sqlalchemy import create_engine  engine = create_engine("postgresql://scott:tiger@localhost:5432/mydatabase")  engine = create_engine("mysql+mysqldb://scott:tiger@localhost/foo")  engine = create_engine("oracle://scott:tiger@127.0.0.1:1521/sidname")  engine = create_engine("mssql+pyodbc://mydsn")  # sqlite://<nohostname>/<path>  # where <path> is relative:  engine = create_engine("sqlite:///foo.db")  # or absolute, starting with a slash:  engine = create_engine("sqlite:///absolute/path/to/foo.db")  For more information see the examples the SQLAlchemy documentation_  Advanced SQLAlchemy queries
'''  You can use SQLAlchemy constructs to describe your query.  Use `sqlalchemy.text` to specify query parameters in a backend-neutral way  .. ipython:: python  import sqlalchemy as sa  pd.read_sql(  sa.text("SELECT * FROM data where Col_1=:col1"), engine, params={"col1": "X"}  )  If you have an SQLAlchemy description of your database you can express where conditions using SQLAlchemy expressions  .. ipython:: python  metadata = sa.MetaData()  data_table = sa.Table(  "data",  metadata,  sa.Column("index", sa.Integer),  sa.Column("Date", sa.DateTime),  sa.Column("Col_1", sa.String),  sa.Column("Col_2", sa.Float),  sa.Column("Col_3", sa.Boolean),  )  pd.read_sql(sa.select(data_table).where(data_table.c.Col_3 is True), engine)  You can combine SQLAlchemy expressions with parameters passed to `read_sql` using `sqlalchemy.bindparam`  .. ipython:: python  import datetime as dt  expr = sa.select(data_table).where(data_table.c.Date > sa.bindparam("date"))  pd.read_sql(expr, engine, params={"date": dt.datetime(2010, 10, 18)})  Sqlite fallback
'''  The use of sqlite is supported without using SQLAlchemy.
This mode requires a Python database adapter which respect the `Python
DB-API <https://www.python.org/dev/peps/pep-0249/>`__.  You can create connections like so:  .. code-block:: python  import sqlite3  con = sqlite3.connect("")  And then issue the following queries:  .. code-block:: python  data.to_sql("data", con)  pd.read_sql_query("SELECT * FROM data", con)  .. _io.bigquery:

Google BigQuery

.. warning::  Starting in 0.20.0, pandas has split off Google BigQuery support into the  separate package ``pandas-gbq``. You can ``pip install pandas-gbq`` to get it.  The ``pandas-gbq`` package provides functionality to read/write from Google BigQuery.  pandas integrates with this external package. if ``pandas-gbq`` is installed, you can
use the pandas methods ``pd.read_gbq`` and ``DataFrame.to_gbq``, which will call the
respective functions from ``pandas-gbq``.  Full documentation can be found here_.  .. _io.stata:

.. _io.stata_writer:  Writing to stata format
'''  The method `~pandas.core.frame.DataFrame.to_stata` will write a DataFrame
into a .dta file. The format version of this file is always 115 (Stata 12).  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 2), columns=list("AB"))  df.to_stata("stata.dta")  *Stata* data files have limited data type support; only strings with
244 or fewer characters, ``int8``, ``int16``, ``int32``, ``float32``
and ``float64`` can be stored in ``.dta`` files.  Additionally,
*Stata* reserves certain values to represent missing data. Exporting a
non-missing value that is outside of the permitted range in Stata for
a particular data type will retype the variable to the next larger
size.  For example, ``int8`` values are restricted to lie between -127
and 100 in Stata, and so variables with values above 100 will trigger
a conversion to ``int16``. ``nan`` values in floating points data
types are stored as the basic missing data type (``.`` in *Stata*).  .. note::  It is not possible to export missing data values for integer data types.  The *Stata* writer gracefully handles other data types including ``int64``,
``bool``, ``uint8``, ``uint16``, ``uint32`` by casting to
the smallest supported type that can represent the data.  For example, data
with a type of ``uint8`` will be cast to ``int8`` if all values are less than
100 (the upper bound for non-missing ``int8`` data in *Stata*), or, if values are
outside of this range, the variable is cast to ``int16``.  .. warning::  Conversion from ``int64`` to ``float64`` may result in a loss of precision  if ``int64`` values are larger than 2**53.  .. warning::  `~pandas.io.stata.StataWriter` and  `~pandas.core.frame.DataFrame.to_stata` only support fixed width  strings containing up to 244 characters, a limitation imposed by the version  115 dta file format. Attempting to write *Stata* dta files with strings  longer than 244 characters raises a ``ValueError``.  .. _io.stata_reader:  Reading from Stata format
'''  The top-level function ``read_stata`` will read a dta file and return
either a ``DataFrame`` or a `~pandas.io.stata.StataReader` that can
be used to read the file incrementally.  .. ipython:: python  pd.read_stata("stata.dta")  Specifying a ``chunksize`` yields a
`~pandas.io.stata.StataReader` instance that can be used to
read ``chunksize`` lines from the file at a time.  The ``StataReader``
object can be used as an iterator.  .. ipython:: python  with pd.read_stata("stata.dta", chunksize=3) as reader:  for df in reader:  print(df.shape)  For more fine-grained control, use ``iterator=True`` and specify
``chunksize`` with each call to
`~pandas.io.stata.StataReader.read`.  .. ipython:: python  with pd.read_stata("stata.dta", iterator=True) as reader:  chunk1 = reader.read(5)  chunk2 = reader.read(5)  Currently the ``index`` is retrieved as a column.  The parameter ``convert_categoricals`` indicates whether value labels should be
read and used to create a ``Categorical`` variable from them. Value labels can
also be retrieved by the function ``value_labels``, which requires `~pandas.io.stata.StataReader.read`
to be called before use.  The parameter ``convert_missing`` indicates whether missing value
representations in Stata should be preserved.  If ``False`` (the default),
missing values are represented as ``np.nan``.  If ``True``, missing values are
represented using ``StataMissingValue`` objects, and columns containing missing
values will have ``object`` data type.  .. note::  `~pandas.read_stata` and  `~pandas.io.stata.StataReader` support .dta formats 113-115  (Stata 10-12), 117 (Stata 13), and 118 (Stata 14).  .. note::  Setting ``preserve_dtypes=False`` will upcast to the standard pandas data types:  ``int64`` for all integer types and ``float64`` for floating point data.  By default,  the Stata data types are preserved when importing.  .. ipython:: python  os.remove("stata.dta")  .. _io.stata-categorical:  Categorical data

+++  ``Categorical`` data can be exported to *Stata* data files as value labeled data.
The exported data consists of the underlying category codes as integer data values
and the categories as value labels.  *Stata* does not have an explicit equivalent
to a ``Categorical`` and information about *whether* the variable is ordered
is lost when exporting.  .. warning::  *Stata* only supports string value labels, and so ``str`` is called on the  categories when exporting data.  Exporting ``Categorical`` variables with  non-string categories produces a warning, and can result a loss of  information if the ``str`` representations of the categories are not unique.  Labeled data can similarly be imported from *Stata* data files as ``Categorical``
variables using the keyword argument ``convert_categoricals`` (``True`` by default).
The keyword argument ``order_categoricals`` (``True`` by default) determines
whether imported ``Categorical`` variables are ordered.  .. note::  When importing categorical data, the values of the variables in the *Stata*  data file are not preserved since ``Categorical`` variables always  use integer data types between ``-1`` and ``n-1`` where ``n`` is the number  of categories. If the original values in the *Stata* data file are required,  these can be imported by setting ``convert_categoricals=False``, which will  import original data (but not the variable labels). The original values can  be matched to the imported categorical data since there is a simple mapping  between the original *Stata* data values and the category codes of imported  Categorical variables: missing values are assigned code ``-1``, and the  smallest original value is assigned ``0``, the second smallest is assigned  ``1`` and so on until the largest original value is assigned the code ``n-1``.  .. note::  *Stata* supports partially labeled series. These series have value labels for  some but not all data values. Importing a partially labeled series will produce  a ``Categorical`` with string categories for the values that are labeled and  numeric categories for values with no label.  .. _io.sas:  .. _io.sas_reader:

SAS formats

The top-level function `read_sas` can read (but not write) SAS
XPORT (.xpt) and (since *v0.18.0*) SAS7BDAT (.sas7bdat) format files.  SAS files only contain two value types: ASCII text and floating point
values (usually 8 bytes but sometimes truncated).  For xport files,
there is no automatic type conversion to integers, dates, or
categoricals.  For SAS7BDAT files, the format codes may allow date
variables to be automatically converted to dates.  By default the
whole file is read and returned as a ``DataFrame``.  Specify a ``chunksize`` or use ``iterator=True`` to obtain reader
objects (``XportReader`` or ``SAS7BDATReader``) for incrementally
reading the file.  The reader objects also have attributes that
contain additional information about the file and its variables.  Read a SAS7BDAT file:  .. code-block:: python  df = pd.read_sas("sas_data.sas7bdat")  Obtain an iterator and read an XPORT file 100,000 lines at a time:  .. code-block:: python  def do_something(chunk):  pass  with pd.read_sas("sas_xport.xpt", chunk=1000) as rdr:  for chunk in rdr:  do_something(chunk)  The specification_ for the xport file format is available from the SAS
web site.  .. _specification: https://support.sas.com/content/dam/SAS/support/en/technical-papers/record-layout-of-a-sas-version-5-or-6-data-set-in-sas-transport-xport-format.pdf  No official documentation is available for the SAS7BDAT format.  .. _io.spss:  .. _io.spss_reader:

SPSS formats

The top-level function `read_spss` can read (but not write) SPSS
SAV (.sav) and  ZSAV (.zsav) format files.  SPSS files contain column names. By default the
whole file is read, categorical columns are converted into ``pd.Categorical``,
and a ``DataFrame`` with all columns is returned.  Specify the ``usecols`` parameter to obtain a subset of columns. Specify ``convert_categoricals=False``
to avoid converting categorical columns into ``pd.Categorical``.  Read an SPSS file:  .. code-block:: python  df = pd.read_spss("spss_data.sav")  Extract a subset of columns contained in ``usecols`` from an SPSS file and
avoid converting categorical columns into ``pd.Categorical``:  .. code-block:: python  df = pd.read_spss(  "spss_data.sav",  usecols=["foo", "bar"],  convert_categoricals=False,  )  More information about the SAV and ZSAV file formats is available here_.  .. _here: https://www.ibm.com/docs/en/spss-statistics/22.0.0  .. _io.other:

Other file formats

pandas itself only supports IO with a limited set of file formats that map
cleanly to its tabular data model. For reading and writing other file formats
into and from pandas, we recommend these packages from the broader community.  netCDF
'''  xarray_ provides data structures inspired by the pandas ``DataFrame`` for working
with multi-dimensional datasets, with a focus on the netCDF file format and
easy conversion to and from pandas.  .. _xarray: https://xarray.pydata.org/en/stable/  .. _io.perf:

This is an informal comparison of various IO methods, using pandas
0.24.2. Timings are machine dependent and small differences should be
ignored.  .. code-block:: ipython  In [1]: sz = 1000  In [2]: df = pd.DataFrame({'A': np.random.randn(sz), 'B': [1] * sz})  In [3]: df.info()  <class 'pandas.core.frame.DataFrame'>  RangeIndex: 1000 entries, 0 to 999  Data columns (total 2 columns):  A  1000 non-null float64  B  1000 non-null int64  dtypes: float64(1), int64(1)  memory usage: 15.3 MB  The following test functions will be used below to compare the performance of several IO methods:  .. code-block:: python  import numpy as np  import os  sz = 1000  df = pd.DataFrame({"A": np.random.randn(sz), "B": [1] * sz})  sz = 1000  np.random.seed(42)  df = pd.DataFrame({"A": np.random.randn(sz), "B": [1] * sz})  def test_sql_write(df):  if os.path.exists("test.sql"):  os.remove("test.sql")  sql_db = sqlite3.connect("test.sql")  df.to_sql(name="test_table", con=sql_db)  sql_db.close()  def test_sql_read():  sql_db = sqlite3.connect("test.sql")  pd.read_sql_query("select * from test_table", sql_db)  sql_db.close()  def test_hdf_fixed_write(df):  df.to_hdf("test_fixed.hdf", "test", mode="w")  def test_hdf_fixed_read():  pd.read_hdf("test_fixed.hdf", "test")  def test_hdf_fixed_write_compress(df):  df.to_hdf("test_fixed_compress.hdf", "test", mode="w", complib="blosc")  def test_hdf_fixed_read_compress():  pd.read_hdf("test_fixed_compress.hdf", "test")  def test_hdf_table_write(df):  df.to_hdf("test_table.hdf", "test", mode="w", format="table")  def test_hdf_table_read():  pd.read_hdf("test_table.hdf", "test")  def test_hdf_table_write_compress(df):  df.to_hdf(  "test_table_compress.hdf", "test", mode="w", complib="blosc", format="table"  )  def test_hdf_table_read_compress():  pd.read_hdf("test_table_compress.hdf", "test")  def test_csv_write(df):  df.to_csv("test.csv", mode="w")  def test_csv_read():  pd.read_csv("test.csv", index_col=0)  def test_feather_write(df):  df.to_feather("test.feather")  def test_feather_read():  pd.read_feather("test.feather")  def test_pickle_write(df):  df.to_pickle("test.pkl")  def test_pickle_read():  pd.read_pickle("test.pkl")  def test_pickle_write_compress(df):  df.to_pickle("test.pkl.compress", compression="xz")  def test_pickle_read_compress():  pd.read_pickle("test.pkl.compress", compression="xz")  def test_parquet_write(df):  df.to_parquet("test.parquet")  def test_parquet_read():  pd.read_parquet("test.parquet")  When writing, the top three functions in terms of speed are ``test_feather_write``, ``test_hdf_fixed_write`` and ``test_hdf_fixed_write_compress``.  .. code-block:: ipython  In [4]: %timeit test_sql_write(df)  3.29 s ± 43.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [5]: %timeit test_hdf_fixed_write(df)  19.4 ms ± 560 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [6]: %timeit test_hdf_fixed_write_compress(df)  19.6 ms ± 308 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [7]: %timeit test_hdf_table_write(df)  449 ms ± 5.61 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [8]: %timeit test_hdf_table_write_compress(df)  448 ms ± 11.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [9]: %timeit test_csv_write(df)  3.66 s ± 26.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [10]: %timeit test_feather_write(df)  9.75 ms ± 117 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  In [11]: %timeit test_pickle_write(df)  30.1 ms ± 229 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [12]: %timeit test_pickle_write_compress(df)  4.29 s ± 15.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [13]: %timeit test_parquet_write(df)  67.6 ms ± 706 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  When reading, the top three functions in terms of speed are ``test_feather_read``, ``test_pickle_read`` and

``test_hdf_fixed_read``.  .. code-block:: ipython  In [14]: %timeit test_sql_read()  1.77 s ± 17.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [15]: %timeit test_hdf_fixed_read()  19.4 ms ± 436 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [16]: %timeit test_hdf_fixed_read_compress()  19.5 ms ± 222 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [17]: %timeit test_hdf_table_read()  38.6 ms ± 857 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [18]: %timeit test_hdf_table_read_compress()  38.8 ms ± 1.49 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  In [19]: %timeit test_csv_read()  452 ms ± 9.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [20]: %timeit test_feather_read()  12.4 ms ± 99.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  In [21]: %timeit test_pickle_read()  18.4 ms ± 191 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  In [22]: %timeit test_pickle_read_compress()  915 ms ± 7.48 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  In [23]: %timeit test_parquet_read()  24.4 ms ± 146 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  The files ``test.pkl.compress``, ``test.parquet`` and ``test.feather`` took the least space on disk (in bytes).  .. code-block:: none  29519500 Oct 10 06:45 test.csv  16000248 Oct 10 06:45 test.feather  8281983  Oct 10 06:49 test.parquet  16000857 Oct 10 06:47 test.pkl  7552144  Oct 10 06:48 test.pkl.compress  34816000 Oct 10 06:42 test.sql  24009288 Oct 10 06:43 test_fixed.hdf  24009288 Oct 10 06:43 test_fixed_compress.hdf  24458940 Oct 10 06:44 test_table.hdf  24458940 Oct 10 06:44 test_table_compress.hdf

The `~pandas.concat` function (in the main pandas namespace) does all of
the heavy lifting of performing concatenation operations along an axis while
performing optional set logic (union or intersection) of the indexes (if any) on
the other axes. Note that I say "if any" because there is only a single possible
axis of concatenation for Series.  Before diving into all of the details of ``concat`` and what it can do, here is
a simple example:  .. ipython:: python  df1 = pd.DataFrame(  {  "A": ["A0", "A1", "A2", "A3"],  "B": ["B0", "B1", "B2", "B3"],  "C": ["C0", "C1", "C2", "C3"],  "D": ["D0", "D1", "D2", "D3"],  },  index=[0, 1, 2, 3],  )  df2 = pd.DataFrame(  {  "A": ["A4", "A5", "A6", "A7"],  "B": ["B4", "B5", "B6", "B7"],  "C": ["C4", "C5", "C6", "C7"],  "D": ["D4", "D5", "D6", "D7"],  },  index=[4, 5, 6, 7],  )  df3 = pd.DataFrame(  {  "A": ["A8", "A9", "A10", "A11"],  "B": ["B8", "B9", "B10", "B11"],  "C": ["C8", "C9", "C10", "C11"],  "D": ["D8", "D9", "D10", "D11"],  },  index=[8, 9, 10, 11],  )  frames = [df1, df2, df3]  result = pd.concat(frames)  .. ipython:: python  @savefig merging_concat_basic.png  p.plot(frames, result, labels=["df1", "df2", "df3"], vertical=True);  plt.close("all");  Like its sibling function on ndarrays, ``numpy.concatenate``, ``pandas.concat``
takes a list or dict of homogeneously-typed objects and concatenates them with
some configurable handling of "what to do with the other axes":  ::  pd.concat(  objs,  axis=0,  join="outer",  ignore_index=False,  keys=None,  levels=None,  names=None,  verify_integrity=False,  copy=True,  )  * ``objs`` : a sequence or mapping of Series or DataFrame objects. If a  dict is passed, the sorted keys will be used as the ``keys`` argument, unless  it is passed, in which case the values will be selected (see below). Any None  objects will be dropped silently unless they are all None in which case a  ValueError will be raised.
* ``axis`` : {0, 1, ...}, default 0. The axis to concatenate along.
* ``join`` : {'inner', 'outer'}, default 'outer'. How to handle indexes on  other axis(es). Outer for union and inner for intersection.
* ``ignore_index`` : boolean, default False. If True, do not use the index  values on the concatenation axis. The resulting axis will be labeled 0, ...,  n - 1. This is useful if you are concatenating objects where the  concatenation axis does not have meaningful indexing information. Note  the index values on the other axes are still respected in the join.
* ``keys`` : sequence, default None. Construct hierarchical index using the  passed keys as the outermost level. If multiple levels passed, should  contain tuples.
* ``levels`` : list of sequences, default None. Specific levels (unique values)  to use for constructing a MultiIndex. Otherwise they will be inferred from the  keys.
* ``names`` : list, default None. Names for the levels in the resulting  hierarchical index.
* ``verify_integrity`` : boolean, default False. Check whether the new  concatenated axis contains duplicates. This can be very expensive relative  to the actual data concatenation.
* ``copy`` : boolean, default True. If False, do not copy data unnecessarily.  Without a little bit of context many of these arguments don't make much sense.
Let's revisit the above example. Suppose we wanted to associate specific keys
with each of the pieces of the chopped up DataFrame. We can do this using the
``keys`` argument:  .. ipython:: python  result = pd.concat(frames, keys=["x", "y", "z"])  .. ipython:: python  @savefig merging_concat_keys.png  p.plot(frames, result, labels=["df1", "df2", "df3"], vertical=True)  plt.close("all");  As you can see (if you've read the rest of the documentation), the resulting
object's index has a `hierarchical index <advanced.hierarchical>`. This
means that we can now select out each chunk by key:  .. ipython:: python  result.loc["y"]  It's not a stretch to see how this can be very useful. More detail on this

functionality below.  .. note::  It is worth noting that `~pandas.concat` makes a full copy of the data, and that constantly  reusing this function can create a significant performance hit. If you need  to use the operation over several datasets, use a list comprehension.  ::  frames = [ process_your_file(f) for f in files ]  result = pd.concat(frames)  .. note::  When concatenating DataFrames with named axes, pandas will attempt to preserve  these index/column names whenever possible. In the case where all inputs share a  common name, this name will be assigned to the result. When the input names do  not all agree, the result will be unnamed. The same is true for `MultiIndex`,  but the logic is applied separately on a level-by-level basis.  Set logic on the other axes
~~~  When gluing together multiple DataFrames, you have a choice of how to handle
the other axes (other than the one being concatenated). This can be done in
the following two ways:  * Take the union of them all, ``join='outer'``. This is the default  option as it results in zero information loss.
* Take the intersection, ``join='inner'``.  Here is an example of each of these methods. First, the default ``join='outer'``
behavior:  .. ipython:: python  df4 = pd.DataFrame(  {  "B": ["B2", "B3", "B6", "B7"],  "D": ["D2", "D3", "D6", "D7"],  "F": ["F2", "F3", "F6", "F7"],  },  index=[2, 3, 6, 7],  )  result = pd.concat([df1, df4], axis=1)  .. ipython:: python  @savefig merging_concat_axis1.png  p.plot([df1, df4], result, labels=["df1", "df4"], vertical=False);  plt.close("all");  Here is the same thing with ``join='inner'``:  .. ipython:: python  result = pd.concat([df1, df4], axis=1, join="inner")  .. ipython:: python  @savefig merging_concat_axis1_inner.png  p.plot([df1, df4], result, labels=["df1", "df4"], vertical=False);  plt.close("all");  Lastly, suppose we just wanted to reuse the *exact index* from the original
DataFrame:  .. ipython:: python  result = pd.concat([df1, df4], axis=1).reindex(df1.index)  Similarly, we could index before the concatenation:  .. ipython:: python  pd.concat([df1, df4.reindex(df1.index)], axis=1)  .. ipython:: python  @savefig merging_concat_axis1_join_axes.png  p.plot([df1, df4], result, labels=["df1", "df4"], vertical=False);  plt.close("all");  .. _merging.ignore_index:  Ignoring indexes on the concatenation axis
~~~
For ``DataFrame`` objects which don't have a meaningful index, you may wish
to append them and ignore the fact that they may have overlapping indexes. To
do this, use the ``ignore_index`` argument:  .. ipython:: python  result = pd.concat([df1, df4], ignore_index=True, sort=False)  .. ipython:: python  @savefig merging_concat_ignore_index.png  p.plot([df1, df4], result, labels=["df1", "df4"], vertical=True);  plt.close("all");  .. _merging.mixed_ndims:  Concatenating with mixed ndims
~~~  You can concatenate a mix of ``Series`` and ``DataFrame`` objects. The
``Series`` will be transformed to ``DataFrame`` with the column name as

the name of the ``Series``.  .. ipython:: python  s1 = pd.Series(["X0", "X1", "X2", "X3"], name="X")  result = pd.concat([df1, s1], axis=1)  .. ipython:: python  @savefig merging_concat_mixed_ndim.png  p.plot([df1, s1], result, labels=["df1", "s1"], vertical=False);  plt.close("all");  .. note::  Since we're concatenating a ``Series`` to a ``DataFrame``, we could have  achieved the same result with `DataFrame.assign`. To concatenate an  arbitrary number of pandas objects (``DataFrame`` or ``Series``), use  ``concat``.  If unnamed ``Series`` are passed they will be numbered consecutively.  .. ipython:: python  s2 = pd.Series(["_0", "_1", "_2", "_3"])  result = pd.concat([df1, s2, s2, s2], axis=1)  .. ipython:: python  @savefig merging_concat_unnamed_series.png  p.plot([df1, s2], result, labels=["df1", "s2"], vertical=False);  plt.close("all");  Passing ``ignore_index=True`` will drop all name references.  .. ipython:: python  result = pd.concat([df1, s1], axis=1, ignore_index=True)  .. ipython:: python  @savefig merging_concat_series_ignore_index.png  p.plot([df1, s1], result, labels=["df1", "s1"], vertical=False);  plt.close("all");  More concatenating with group keys
~~~  A fairly common use of the ``keys`` argument is to override the column names
when creating a new ``DataFrame`` based on existing ``Series``.
Notice how the default behaviour consists on letting the resulting ``DataFrame``
inherit the parent ``Series``' name, when these existed.  .. ipython:: python  s3 = pd.Series([0, 1, 2, 3], name="foo")  s4 = pd.Series([0, 1, 2, 3])  s5 = pd.Series([0, 1, 4, 5])  pd.concat([s3, s4, s5], axis=1)  Through the ``keys`` argument we can override the existing column names.  .. ipython:: python  pd.concat([s3, s4, s5], axis=1, keys=["red", "blue", "yellow"])  Let's consider a variation of the very first example presented:  .. ipython:: python  result = pd.concat(frames, keys=["x", "y", "z"])  .. ipython:: python  @savefig merging_concat_group_keys2.png  p.plot(frames, result, labels=["df1", "df2", "df3"], vertical=True);  plt.close("all");  You can also pass a dict to ``concat`` in which case the dict keys will be used
for the ``keys`` argument (unless other keys are specified):  .. ipython:: python  pieces = {"x": df1, "y": df2, "z": df3}  result = pd.concat(pieces)  .. ipython:: python  @savefig merging_concat_dict.png  p.plot([df1, df2, df3], result, labels=["df1", "df2", "df3"], vertical=True);  plt.close("all");  .. ipython:: python  result = pd.concat(pieces, keys=["z", "y"])  .. ipython:: python  @savefig merging_concat_dict_keys.png  p.plot([df1, df2, df3], result, labels=["df1", "df2", "df3"], vertical=True);  plt.close("all");  The MultiIndex created has levels that are constructed from the passed keys and
the index of the ``DataFrame`` pieces:  .. ipython:: python  result.index.levels  If you wish to specify other levels (as will occasionally be the case), you can
do so using the ``levels`` argument:  .. ipython:: python  result = pd.concat(  pieces, keys=["x", "y", "z"], levels=[["z", "y", "x", "w"]], names=["group_key"]  )  .. ipython:: python  @savefig merging_concat_dict_keys_names.png  p.plot([df1, df2, df3], result, labels=["df1", "df2", "df3"], vertical=True);  plt.close("all");  .. ipython:: python  result.index.levels  This is fairly esoteric, but it is actually necessary for implementing things
like GroupBy where the order of a categorical variable is meaningful.  .. _merging.append.row:  Appending rows to a DataFrame
~~~  If you have a series that you want to append as a single row to a ``DataFrame``, you can convert the row into a

``DataFrame`` and use ``concat``  .. ipython:: python  s2 = pd.Series(["X0", "X1", "X2", "X3"], index=["A", "B", "C", "D"])  result = pd.concat([df1, s2.to_frame().T], ignore_index=True)  .. ipython:: python  @savefig merging_append_series_as_row.png  p.plot([df1, s2], result, labels=["df1", "s2"], vertical=True);  plt.close("all");  You should use ``ignore_index`` with this method to instruct DataFrame to
discard its index. If you wish to preserve the index, you should construct an
appropriately-indexed DataFrame and append or concatenate those objects.  .. _merging.join:

pandas has full-featured, **high performance** in-memory join operations
idiomatically very similar to relational databases like SQL. These methods
perform significantly better (in some cases well over an order of magnitude
better) than other open source implementations (like ``base::merge.data.frame``
in R). The reason for this is careful algorithmic design and the internal layout
of the data in ``DataFrame``.  See the `cookbook<cookbook.merge>` for some advanced strategies.  Users who are familiar with SQL but new to pandas might be interested in a
`comparison with SQL<compare_with_sql.join>`.  pandas provides a single function, `~pandas.merge`, as the entry point for
all standard database join operations between ``DataFrame`` or named ``Series`` objects:  ::  pd.merge(  left,  right,  how="inner",  on=None,  left_on=None,  right_on=None,  left_index=False,  right_index=False,  sort=True,  suffixes=("_x", "_y"),  copy=True,  indicator=False,  validate=None,  )  * ``left``: A DataFrame or named Series object.
* ``right``: Another DataFrame or named Series object.
* ``on``: Column or index level names to join on. Must be found in both the left  and right DataFrame and/or Series objects. If not passed and ``left_index`` and  ``right_index`` are ``False``, the intersection of the columns in the  DataFrames and/or Series will be inferred to be the join keys.
* ``left_on``: Columns or index levels from the left DataFrame or Series to use as  keys. Can either be column names, index level names, or arrays with length  equal to the length of the DataFrame or Series.
* ``right_on``: Columns or index levels from the right DataFrame or Series to use as  keys. Can either be column names, index level names, or arrays with length  equal to the length of the DataFrame or Series.
* ``left_index``: If ``True``, use the index (row labels) from the left  DataFrame or Series as its join key(s). In the case of a DataFrame or Series with a MultiIndex  (hierarchical), the number of levels must match the number of join keys  from the right DataFrame or Series.
* ``right_index``: Same usage as ``left_index`` for the right DataFrame or Series
* ``how``: One of ``'left'``, ``'right'``, ``'outer'``, ``'inner'``, ``'cross'``. Defaults  to ``inner``. See below for more detailed description of each method.
* ``sort``: Sort the result DataFrame by the join keys in lexicographical  order. Defaults to ``True``, setting to ``False`` will improve performance  substantially in many cases.
* ``suffixes``: A tuple of string suffixes to apply to overlapping  columns. Defaults to ``('_x', '_y')``.
* ``copy``: Always copy data (default ``True``) from the passed DataFrame or named Series  objects, even when reindexing is not necessary. Cannot be avoided in many  cases but may improve performance / memory usage. The cases where copying  can be avoided are somewhat pathological but this option is provided  nonetheless.

* ``indicator``: Add a column to the output DataFrame called ``_merge``  with information on the source of each row. ``_merge`` is Categorical-type  and takes on a value of ``left_only`` for observations whose merge key  only appears in ``'left'`` DataFrame or Series, ``right_only`` for observations whose  merge key only appears in ``'right'`` DataFrame or Series, and ``both`` if the  observation's merge key is found in both.  * ``validate`` : string, default None.  If specified, checks if merge is of specified type.  * "one_to_one" or "1:1": checks if merge keys are unique in both  left and right datasets.  * "one_to_many" or "1:m": checks if merge keys are unique in left  dataset.  * "many_to_one" or "m:1": checks if merge keys are unique in right  dataset.  * "many_to_many" or "m:m": allowed, but does not result in checks.  .. note::  Support for specifying index levels as the ``on``, ``left_on``, and  ``right_on`` parameters was added in version 0.23.0.  Support for merging named ``Series`` objects was added in version 0.24.0.  The return type will be the same as ``left``. If ``left`` is a ``DataFrame`` or named ``Series``
and ``right`` is a subclass of ``DataFrame``, the return type will still be ``DataFrame``.  ``merge`` is a function in the pandas namespace, and it is also available as a
``DataFrame`` instance method `~DataFrame.merge`, with the calling
``DataFrame`` being implicitly considered the left object in the join.  The related `~DataFrame.join` method, uses ``merge`` internally for the
index-on-index (by default) and column(s)-on-index join. If you are joining on
index only, you may wish to use ``DataFrame.join`` to save yourself some typing.  Brief primer on merge methods (relational algebra)
~~~  Experienced users of relational databases like SQL will be familiar with the
terminology used to describe join operations between two SQL-table like
structures (``DataFrame`` objects). There are several cases to consider which
are very important to understand:  * **one-to-one** joins: for example when joining two ``DataFrame`` objects on  their indexes (which must contain unique values).
* **many-to-one** joins: for example when joining an index (unique) to one or  more columns in a different ``DataFrame``.
* **many-to-many** joins: joining columns on columns.  .. note::  When joining columns on columns (potentially a many-to-many join), any  indexes on the passed ``DataFrame`` objects **will be discarded**.  It is worth spending some time understanding the result of the **many-to-many**
join case. In SQL / standard relational algebra, if a key combination appears
more than once in both tables, the resulting table will have the **Cartesian
product** of the associated data. Here is a very basic example with one unique
key combination:  .. ipython:: python  left = pd.DataFrame(  {  "key": ["K0", "K1", "K2", "K3"],  "A": ["A0", "A1", "A2", "A3"],  "B": ["B0", "B1", "B2", "B3"],  }  )  right = pd.DataFrame(  {  "key": ["K0", "K1", "K2", "K3"],  "C": ["C0", "C1", "C2", "C3"],  "D": ["D0", "D1", "D2", "D3"],  }  )  result = pd.merge(left, right, on="key")  .. ipython:: python  @savefig merging_merge_on_key.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  Here is a more complicated example with multiple join keys. Only the keys
appearing in ``left`` and ``right`` are present (the intersection), since

``how='inner'`` by default.  .. ipython:: python  left = pd.DataFrame(  {  "key1": ["K0", "K0", "K1", "K2"],  "key2": ["K0", "K1", "K0", "K1"],  "A": ["A0", "A1", "A2", "A3"],  "B": ["B0", "B1", "B2", "B3"],  }  )  right = pd.DataFrame(  {  "key1": ["K0", "K1", "K1", "K2"],  "key2": ["K0", "K0", "K0", "K0"],  "C": ["C0", "C1", "C2", "C3"],  "D": ["D0", "D1", "D2", "D3"],  }  )  result = pd.merge(left, right, on=["key1", "key2"])  .. ipython:: python  @savefig merging_merge_on_key_multiple.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  The ``how`` argument to ``merge`` specifies how to determine which keys are to
be included in the resulting table. If a key combination **does not appear** in
either the left or right tables, the values in the joined table will be
``NA``. Here is a summary of the ``how`` options and their SQL equivalent names:  .. csv-table::  "Merge method", "SQL Join Name", "Description"  20, 20, 60  ``left``, ``LEFT OUTER JOIN``, Use keys from left frame only  ``right``, ``RIGHT OUTER JOIN``, Use keys from right frame only  ``outer``, ``FULL OUTER JOIN``, Use union of keys from both frames  ``inner``, ``INNER JOIN``, Use intersection of keys from both frames  ``cross``, ``CROSS JOIN``, Create the cartesian product of rows of both frames  .. ipython:: python  result = pd.merge(left, right, how="left", on=["key1", "key2"])  .. ipython:: python  @savefig merging_merge_on_key_left.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = pd.merge(left, right, how="right", on=["key1", "key2"])  .. ipython:: python  @savefig merging_merge_on_key_right.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  .. ipython:: python  result = pd.merge(left, right, how="outer", on=["key1", "key2"])  .. ipython:: python  @savefig merging_merge_on_key_outer.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = pd.merge(left, right, how="inner", on=["key1", "key2"])  .. ipython:: python  @savefig merging_merge_on_key_inner.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = pd.merge(left, right, how="cross")  .. ipython:: python  @savefig merging_merge_cross.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  You can merge a mult-indexed Series and a DataFrame, if the names of
the MultiIndex correspond to the columns from the DataFrame. Transform
the Series to a DataFrame using `Series.reset_index` before merging,
as shown in the following example.  .. ipython:: python  df = pd.DataFrame({"Let": ["A", "B", "C"], "Num": [1, 2, 3]})  df  ser = pd.Series(  ["a", "b", "c", "d", "e", "f"],  index=pd.MultiIndex.from_arrays(  [["A", "B", "C"] * 2, [1, 2, 3, 4, 5, 6]], names=["Let", "Num"]  ),  )  ser  pd.merge(df, ser.reset_index(), on=["Let", "Num"])  Here is another example with duplicate join keys in DataFrames:  .. ipython:: python  left = pd.DataFrame({"A": [1, 2], "B": [2, 2]})  right = pd.DataFrame({"A": [4, 5, 6], "B": [2, 2, 2]})  result = pd.merge(left, right, on="B", how="outer")  .. ipython:: python  @savefig merging_merge_on_key_dup.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. warning::  Joining / merging on duplicate keys can cause a returned frame that is the multiplication of the row dimensions, which may result in memory overflow. It is the user' s responsibility to manage duplicate values in keys before joining large DataFrames.  .. _merging.validation:  Checking for duplicate keys
~~~  Users can use the ``validate`` argument to automatically check whether there
are unexpected duplicates in their merge keys. Key uniqueness is checked before

merge operations and so should protect against memory overflows. Checking key
uniqueness is also a good way to ensure user data structures are as expected.  In the following example, there are duplicate values of ``B`` in the right
``DataFrame``. As this is not a one-to-one merge -- as specified in the
``validate`` argument -- an exception will be raised.  .. ipython:: python  left = pd.DataFrame({"A": [1, 2], "B": [1, 2]})  right = pd.DataFrame({"A": [4, 5, 6], "B": [2, 2, 2]})  .. code-block:: ipython  In [53]: result = pd.merge(left, right, on="B", how="outer", validate="one_to_one")  ...  MergeError: Merge keys are not unique in right dataset; not a one-to-one merge  If the user is aware of the duplicates in the right ``DataFrame`` but wants to
ensure there are no duplicates in the left DataFrame, one can use the
``validate='one_to_many'`` argument instead, which will not raise an exception.  .. ipython:: python  pd.merge(left, right, on="B", how="outer", validate="one_to_many")  .. _merging.indicator:  The merge indicator
~~~  `~pandas.merge` accepts the argument ``indicator``. If ``True``, a
Categorical-type column called ``_merge`` will be added to the output object
that takes on values:  ===  ===  Observation Origin  ``_merge`` value  ===  ===  Merge key only in ``'left'`` frame  ``left_only``  Merge key only in ``'right'`` frame  ``right_only``  Merge key in both frames  ``both``  ===  ===  .. ipython:: python  df1 = pd.DataFrame({"col1": [0, 1], "col_left": ["a", "b"]})  df2 = pd.DataFrame({"col1": [1, 2, 2], "col_right": [2, 2, 2]})  pd.merge(df1, df2, on="col1", how="outer", indicator=True)  The ``indicator`` argument will also accept string arguments, in which case the indicator function will use the value of the passed string as the name for the indicator column.  .. ipython:: python  pd.merge(df1, df2, on="col1", how="outer", indicator="indicator_column")  .. _merging.dtypes:  Merge dtypes
~~~  Merging will preserve the dtype of the join keys.  .. ipython:: python  left = pd.DataFrame({"key": [1], "v1": [10]})  left  right = pd.DataFrame({"key": [1, 2], "v1": [20, 30]})  right  We are able to preserve the join keys:  .. ipython:: python  pd.merge(left, right, how="outer")  pd.merge(left, right, how="outer").dtypes  Of course if you have missing values that are introduced, then the
resulting dtype will be upcast.  .. ipython:: python  pd.merge(left, right, how="outer", on="key")  pd.merge(left, right, how="outer", on="key").dtypes  Merging will preserve ``category`` dtypes of the mergands. See also the section on `categoricals <categorical.merge>`.  The left frame.  .. ipython:: python  from pandas.api.types import CategoricalDtype  X = pd.Series(np.random.choice(["foo", "bar"], size=(10,)))  X = X.astype(CategoricalDtype(categories=["foo", "bar"]))  left = pd.DataFrame(  {"X": X, "Y": np.random.choice(["one", "two", "three"], size=(10,))}  )  left  left.dtypes  The right frame.  .. ipython:: python  right = pd.DataFrame(  {  "X": pd.Series(["foo", "bar"], dtype=CategoricalDtype(["foo", "bar"])),  "Z": [1, 2],  }  )  right  right.dtypes  The merged result:  .. ipython:: python  result = pd.merge(left, right, how="outer")  result  result.dtypes  .. note::  The category dtypes must be *exactly* the same, meaning the same categories and the ordered attribute.  Otherwise the result will coerce to the categories' dtype.  .. note::  Merging on ``category`` dtypes that are the same can be quite performant compared to ``object`` dtype merging.  .. _merging.join.index:  Joining on index
~~~  `DataFrame.join` is a convenient method for combining the columns of two
potentially differently-indexed ``DataFrames`` into a single result

``DataFrame``. Here is a very basic example:  .. ipython:: python  left = pd.DataFrame(  {"A": ["A0", "A1", "A2"], "B": ["B0", "B1", "B2"]}, index=["K0", "K1", "K2"]  )  right = pd.DataFrame(  {"C": ["C0", "C2", "C3"], "D": ["D0", "D2", "D3"]}, index=["K0", "K2", "K3"]  )  result = left.join(right)  .. ipython:: python  @savefig merging_join.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = left.join(right, how="outer")  .. ipython:: python  @savefig merging_join_outer.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  The same as above, but with ``how='inner'``.  .. ipython:: python  result = left.join(right, how="inner")  .. ipython:: python  @savefig merging_join_inner.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  The data alignment here is on the indexes (row labels). This same behavior can
be achieved using ``merge`` plus additional arguments instructing it to use the
indexes:  .. ipython:: python  result = pd.merge(left, right, left_index=True, right_index=True, how="outer")  .. ipython:: python  @savefig merging_merge_index_outer.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = pd.merge(left, right, left_index=True, right_index=True, how="inner")  .. ipython:: python  @savefig merging_merge_index_inner.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  Joining key columns on an index
~~~  `~DataFrame.join` takes an optional ``on`` argument which may be a column
or multiple column names, which specifies that the passed ``DataFrame`` is to be
aligned on that column in the ``DataFrame``. These two function calls are
completely equivalent:  ::  left.join(right, on=key_or_keys)  pd.merge(  left, right, left_on=key_or_keys, right_index=True, how="left", sort=False  )  Obviously you can choose whichever form you find more convenient. For
many-to-one joins (where one of the ``DataFrame``'s is already indexed by the
join key), using ``join`` may be more convenient. Here is a simple example:  .. ipython:: python  left = pd.DataFrame(  {  "A": ["A0", "A1", "A2", "A3"],  "B": ["B0", "B1", "B2", "B3"],  "key": ["K0", "K1", "K0", "K1"],  }  )  right = pd.DataFrame({"C": ["C0", "C1"], "D": ["D0", "D1"]}, index=["K0", "K1"])  result = left.join(right, on="key")  .. ipython:: python  @savefig merging_join_key_columns.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = pd.merge(  left, right, left_on="key", right_index=True, how="left", sort=False  )  .. ipython:: python  @savefig merging_merge_key_columns.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. _merging.multikey_join:  To join on multiple keys, the passed DataFrame must have a ``MultiIndex``:  .. ipython:: python  left = pd.DataFrame(  {  "A": ["A0", "A1", "A2", "A3"],  "B": ["B0", "B1", "B2", "B3"],  "key1": ["K0", "K0", "K1", "K2"],  "key2": ["K0", "K1", "K0", "K1"],  }  )  index = pd.MultiIndex.from_tuples(  [("K0", "K0"), ("K1", "K0"), ("K2", "K0"), ("K2", "K1")]  )  right = pd.DataFrame(  {"C": ["C0", "C1", "C2", "C3"], "D": ["D0", "D1", "D2", "D3"]}, index=index  )  Now this can be joined by passing the two key column names:  .. ipython:: python  result = left.join(right, on=["key1", "key2"])  .. ipython:: python  @savefig merging_join_multikeys.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. _merging.df_inner_join:  The default for ``DataFrame.join`` is to perform a left join (essentially a

"VLOOKUP" operation, for Excel users), which uses only the keys found in the
calling DataFrame. Other join types, for example inner join, can be just as
easily performed:  .. ipython:: python  result = left.join(right, on=["key1", "key2"], how="inner")  .. ipython:: python  @savefig merging_join_multikeys_inner.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  As you can see, this drops any rows where there was no match.  .. _merging.join_on_mi:  Joining a single Index to a MultiIndex
~~~  You can join a singly-indexed ``DataFrame`` with a level of a MultiIndexed ``DataFrame``.
The level will match on the name of the index of the singly-indexed frame against
a level name of the MultiIndexed frame.  ..  ipython:: python  left = pd.DataFrame(  {"A": ["A0", "A1", "A2"], "B": ["B0", "B1", "B2"]},  index=pd.Index(["K0", "K1", "K2"], name="key"),  )  index = pd.MultiIndex.from_tuples(  [("K0", "Y0"), ("K1", "Y1"), ("K2", "Y2"), ("K2", "Y3")],  names=["key", "Y"],  )  right = pd.DataFrame(  {"C": ["C0", "C1", "C2", "C3"], "D": ["D0", "D1", "D2", "D3"]},  index=index,  )  result = left.join(right, how="inner")  .. ipython:: python  @savefig merging_join_multiindex_inner.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  This is equivalent but less verbose and more memory efficient / faster than this.  ..  ipython:: python  result = pd.merge(  left.reset_index(), right.reset_index(), on=["key"], how="inner"  ).set_index(["key","Y"])  .. ipython:: python  @savefig merging_merge_multiindex_alternative.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. _merging.join_with_two_multi_indexes:  Joining with two MultiIndexes
~~~  This is supported in a limited way, provided that the index for the right
argument is completely used in the join, and is a subset of the indices in
the left argument, as in this example:  .. ipython:: python  leftindex = pd.MultiIndex.from_product(  [list("abc"), list("xy"), [1, 2]], names=["abc", "xy", "num"]  )  left = pd.DataFrame({"v1": range(12)}, index=leftindex)  left  rightindex = pd.MultiIndex.from_product(  [list("abc"), list("xy")], names=["abc", "xy"]  )  right = pd.DataFrame({"v2": [100 * i for i in range(1, 7)]}, index=rightindex)  right  left.join(right, on=["abc", "xy"], how="inner")  If that condition is not satisfied, a join with two multi-indexes can be
done using the following code.  .. ipython:: python  leftindex = pd.MultiIndex.from_tuples(  [("K0", "X0"), ("K0", "X1"), ("K1", "X2")], names=["key", "X"]  )  left = pd.DataFrame(  {"A": ["A0", "A1", "A2"], "B": ["B0", "B1", "B2"]}, index=leftindex  )  rightindex = pd.MultiIndex.from_tuples(  [("K0", "Y0"), ("K1", "Y1"), ("K2", "Y2"), ("K2", "Y3")], names=["key", "Y"]  )  right = pd.DataFrame(  {"C": ["C0", "C1", "C2", "C3"], "D": ["D0", "D1", "D2", "D3"]}, index=rightindex  )  result = pd.merge(  left.reset_index(), right.reset_index(), on=["key"], how="inner"  ).set_index(["key", "X", "Y"])  .. ipython:: python  @savefig merging_merge_two_multiindex.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. _merging.merge_on_columns_and_levels:  Merging on a combination of columns and index levels
~~~  Strings passed as the ``on``, ``left_on``, and ``right_on`` parameters
may refer to either column names or index level names.  This enables merging
``DataFrame`` instances on a combination of index levels and columns without

resetting indexes.  .. ipython:: python  left_index = pd.Index(["K0", "K0", "K1", "K2"], name="key1")  left = pd.DataFrame(  {  "A": ["A0", "A1", "A2", "A3"],  "B": ["B0", "B1", "B2", "B3"],  "key2": ["K0", "K1", "K0", "K1"],  },  index=left_index,  )  right_index = pd.Index(["K0", "K1", "K2", "K2"], name="key1")  right = pd.DataFrame(  {  "C": ["C0", "C1", "C2", "C3"],  "D": ["D0", "D1", "D2", "D3"],  "key2": ["K0", "K0", "K0", "K1"],  },  index=right_index,  )  result = left.merge(right, on=["key1", "key2"])  .. ipython:: python  @savefig merge_on_index_and_column.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. note::  When DataFrames are merged on a string that matches an index level in both  frames, the index level is preserved as an index level in the resulting  DataFrame.  .. note::  When DataFrames are merged using only some of the levels of a ``MultiIndex``,  the extra levels will be dropped from the resulting merge. In order to  preserve those levels, use ``reset_index`` on those level names to move  those levels to columns prior to doing the merge.  .. note::  If a string matches both a column name and an index level name, then a  warning is issued and the column takes precedence. This will result in an  ambiguity error in a future version.  Overlapping value columns
~~~  The merge ``suffixes`` argument takes a tuple of list of strings to append to
overlapping column names in the input ``DataFrame``\ s to disambiguate the result
columns:  .. ipython:: python  left = pd.DataFrame({"k": ["K0", "K1", "K2"], "v": [1, 2, 3]})  right = pd.DataFrame({"k": ["K0", "K0", "K3"], "v": [4, 5, 6]})  result = pd.merge(left, right, on="k")  .. ipython:: python  @savefig merging_merge_overlapped.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. ipython:: python  result = pd.merge(left, right, on="k", suffixes=("_l", "_r"))  .. ipython:: python  @savefig merging_merge_overlapped_suffix.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  `DataFrame.join` has ``lsuffix`` and ``rsuffix`` arguments which behave
similarly.  .. ipython:: python  left = left.set_index("k")  right = right.set_index("k")  result = left.join(right, lsuffix="_l", rsuffix="_r")  .. ipython:: python  @savefig merging_merge_overlapped_multi_suffix.png  p.plot([left, right], result, labels=["left", "right"], vertical=False);  plt.close("all");  .. _merging.multiple_join:  Joining multiple DataFrames
~~~  A list or tuple of ``DataFrames`` can also be passed to `~DataFrame.join`
to join them together on their indexes.  .. ipython:: python  right2 = pd.DataFrame({"v": [7, 8, 9]}, index=["K1", "K1", "K2"])  result = left.join([right, right2])  .. ipython:: python  @savefig merging_join_multi_df.png  p.plot(  [left, right, right2],  result,  labels=["left", "right", "right2"],  vertical=False,  );  plt.close("all");  .. _merging.combine_first.update:  Merging together values within Series or DataFrame columns
~~~  Another fairly common situation is to have two like-indexed (or similarly
indexed) ``Series`` or ``DataFrame`` objects and wanting to "patch" values in
one object from values for matching indices in the other. Here is an example:  .. ipython:: python  df1 = pd.DataFrame(  [[np.nan, 3.0, 5.0], [-4.6, np.nan, np.nan], [np.nan, 7.0, np.nan]]  )  df2 = pd.DataFrame([[-42.6, np.nan, -8.2], [-5.0, 1.6, 4]], index=[1, 2])  For this, use the `~DataFrame.combine_first` method:  .. ipython:: python  result = df1.combine_first(df2)  .. ipython:: python  @savefig merging_combine_first.png  p.plot([df1, df2], result, labels=["df1", "df2"], vertical=False);  plt.close("all");  Note that this method only takes values from the right ``DataFrame`` if they are

missing in the left ``DataFrame``. A related method, `~DataFrame.update`,
alters non-NA values in place:  .. ipython:: python  df1_copy = df1.copy()  .. ipython:: python  df1.update(df2)  .. ipython:: python  @savefig merging_update.png  p.plot([df1_copy, df2], df1, labels=["df1", "df2"], vertical=False);  plt.close("all");  .. _merging.time_series:

Timeseries friendly merging

.. _merging.merge_ordered:  Merging ordered data
~~~  A `merge_ordered` function allows combining time series and other
ordered data. In particular it has an optional ``fill_method`` keyword to
fill/interpolate missing data:  .. ipython:: python  left = pd.DataFrame(  {"k": ["K0", "K1", "K1", "K2"], "lv": [1, 2, 3, 4], "s": ["a", "b", "c", "d"]}  )  right = pd.DataFrame({"k": ["K1", "K2", "K4"], "rv": [1, 2, 3]})  pd.merge_ordered(left, right, fill_method="ffill", left_by="s")  .. _merging.merge_asof:  Merging asof
~~~  A `merge_asof` is similar to an ordered left-join except that we match on
nearest key rather than equal keys. For each row in the ``left`` ``DataFrame``,
we select the last row in the ``right`` ``DataFrame`` whose ``on`` key is less
than the left's key. Both DataFrames must be sorted by the key.  Optionally an asof merge can perform a group-wise merge. This matches the
``by`` key equally, in addition to the nearest match on the ``on`` key.  For example; we might have ``trades`` and ``quotes`` and we want to ``asof``
merge them.  .. ipython:: python  trades = pd.DataFrame(  {  "time": pd.to_datetime(  [  "20160525 1300.023",  "20160525 1300.038",  "20160525 1300.048",  "20160525 1300.048",  "20160525 1300.048",  ]  ),  "ticker": ["MSFT", "MSFT", "GOOG", "GOOG", "AAPL"],  "price": [51.95, 51.95, 720.77, 720.92, 98.00],  "quantity": [75, 155, 100, 100, 100],  },  columns=["time", "ticker", "price", "quantity"],  )  quotes = pd.DataFrame(  {  "time": pd.to_datetime(  [  "20160525 1300.023",  "20160525 1300.023",  "20160525 1300.030",  "20160525 1300.041",  "20160525 1300.048",  "20160525 1300.049",  "20160525 1300.072",  "20160525 1300.075",  ]  ),  "ticker": ["GOOG", "MSFT", "MSFT", "MSFT", "GOOG", "AAPL", "GOOG", "MSFT"],  "bid": [720.50, 51.95, 51.97, 51.99, 720.50, 97.99, 720.50, 52.01],  "ask": [720.93, 51.96, 51.98, 52.00, 720.93, 98.01, 720.88, 52.03],  },  columns=["time", "ticker", "bid", "ask"],  )  .. ipython:: python  trades  quotes  By default we are taking the asof of the quotes.  .. ipython:: python  pd.merge_asof(trades, quotes, on="time", by="ticker")  We only asof within ``2ms`` between the quote time and the trade time.  .. ipython:: python  pd.merge_asof(trades, quotes, on="time", by="ticker", tolerance=pd.Timedelta("2ms"))  We only asof within ``10ms`` between the quote time and the trade time and we
exclude exact matches on time. Note that though we exclude the exact matches
(of the quotes), prior quotes **do** propagate to that point in time.  .. ipython:: python  pd.merge_asof(  trades,  quotes,  on="time",  by="ticker",  tolerance=pd.Timedelta("10ms"),  allow_exact_matches=False,  )  .. _merging.compare:

Comparing objects

The `~Series.compare` and `~DataFrame.compare` methods allow you to
compare two DataFrame or Series, respectively, and summarize their differences.  This feature was added in `V1.1.0 <whatsnew_110.dataframe_or_series_comparing>`.  For example, you might want to compare two ``DataFrame`` and stack their differences
side by side.  .. ipython:: python  df = pd.DataFrame(  {  "col1": ["a", "a", "b", "b", "a"],  "col2": [1.0, 2.0, 3.0, np.nan, 5.0],  "col3": [1.0, 2.0, 3.0, 4.0, 5.0],  },  columns=["col1", "col2", "col3"],  )  df  .. ipython:: python  df2 = df.copy()  df2.loc[0, "col1"] = "c"  df2.loc[2, "col3"] = 4.0  df2  .. ipython:: python  df.compare(df2)  By default, if two corresponding values are equal, they will be shown as ``NaN``.
Furthermore, if all values in an entire row / column, the row / column will be
omitted from the result. The remaining differences will be aligned on columns.  If you wish, you may choose to stack the differences on rows.  .. ipython:: python  df.compare(df2, align_axis=0)  If you wish to keep all original rows and columns, set ``keep_shape`` argument
to ``True``.  .. ipython:: python  df.compare(df2, keep_shape=True)  You may also keep all the original values even if they are equal.  .. ipython:: python  df.compare(df2, keep_shape=True, keep_equal=True)

Integer dtypes and missing data

Because ``NaN`` is a float, a column of integers with even one missing values
is cast to floating-point dtype (see `gotchas.intna` for more). pandas
provides a nullable integer array, which can be used by explicitly requesting
the dtype:  .. ipython:: python  pd.Series([1, 2, np.nan, 4], dtype=pd.Int64Dtype())  Alternatively, the string alias ``dtype='Int64'`` (note the capital ``"I"``) can be
used.  See `integer_na` for more.

Datetimes

For datetime64[ns] types, ``NaT`` represents missing values. This is a pseudo-native
sentinel value that can be represented by NumPy in a singular dtype (datetime64[ns]).
pandas objects provide compatibility between ``NaT`` and ``NaN``.  .. ipython:: python  df2 = df.copy()  df2["timestamp"] = pd.Timestamp("20120101")  df2  df2.loc[["a", "c", "h"], ["one", "timestamp"]] = np.nan  df2  df2.dtypes.value_counts()  .. _missing.inserting:  Inserting missing data
~~~  You can insert missing values by simply assigning to containers. The
actual missing value used will be chosen based on the dtype.  For example, numeric containers will always use ``NaN`` regardless of
the missing value type chosen:  .. ipython:: python  s = pd.Series([1, 2, 3])  s.loc[0] = None  s  Likewise, datetime containers will always use ``NaT``.  For object containers, pandas will use the value given:  .. ipython:: python  s = pd.Series(["a", "b", "c"])  s.loc[0] = None  s.loc[1] = np.nan  s  .. _missing_data.calculations:  Calculations with missing data
~~~  Missing values propagate naturally through arithmetic operations between pandas
objects.  .. ipython:: python  df = df2.loc[:, ["one", "two", "three"]]  a = df2.loc[df2.index[:5], ["one", "two"]].fillna(method="pad")  b = df2.loc[df2.index[:5], ["one", "two", "three"]]  .. ipython:: python  a  b  a + b  The descriptive statistics and computational methods discussed in the
`data structure overview <basics.stats>` (and listed `here
<api.series.stats>` and `here <api.dataframe.stats>`) are all written to
account for missing data. For example:  * When summing data, NA (missing) values will be treated as zero.
* If the data are all NA, the result will be 0.
* Cumulative methods like `~DataFrame.cumsum` and `~DataFrame.cumprod` ignore NA values by default, but preserve them in the resulting arrays. To override this behaviour and include NA values, use ``skipna=False``.  .. ipython:: python  df  df["one"].sum()  df.mean(1)  df.cumsum()  df.cumsum(skipna=False)  .. _missing_data.numeric_sum:  Sum/prod of empties/nans
~~~  .. warning::  This behavior is now standard as of v0.22.0 and is consistent with the default in ``numpy``; previously sum/prod of all-NA or empty Series/DataFrames would return NaN.  See `v0.22.0 whatsnew <whatsnew_0220>` for more.  The sum of an empty or all-NA Series or column of a DataFrame is 0.  .. ipython:: python  pd.Series([np.nan]).sum()  pd.Series([], dtype="float64").sum()  The product of an empty or all-NA Series or column of a DataFrame is 1.  .. ipython:: python  pd.Series([np.nan]).prod()  pd.Series([], dtype="float64").prod()  NA values in GroupBy
~~~  NA groups in GroupBy are automatically excluded. This behavior is consistent
with R, for example:  .. ipython:: python  df  df.groupby("one").mean()  See the groupby section `here <groupby.missing>` for more information.

pandas objects are equipped with various data manipulation methods for dealing
with missing data.  .. _missing_data.fillna:  Filling missing values: fillna
~~~  `~DataFrame.fillna` can "fill in" NA values with non-NA data in a couple
of ways, which we illustrate:  **Replace NA with a scalar value**  .. ipython:: python  df2  df2.fillna(0)  df2["one"].fillna("missing")  **Fill gaps forward or backward**  Using the same filling arguments as `reindexing <basics.reindexing>`, we
can propagate non-NA values forward or backward:  .. ipython:: python  df  df.fillna(method="pad")  .. _missing_data.fillna.limit:  **Limit the amount of filling**  If we only want consecutive gaps filled up to a certain number of data points,
we can use the ``limit`` keyword:  .. ipython:: python  df.iloc[2:4, :] = np.nan  .. ipython:: python  df  df.fillna(method="pad", limit=1)  To remind you, these are the available filling methods:  .. csv-table::  "Method", "Action"  30, 50  pad / ffill, Fill values forward  bfill / backfill, Fill values backward  With time series data, using pad/ffill is extremely common so that the "last
known value" is available at every time point.  `~DataFrame.ffill` is equivalent to ``fillna(method='ffill')``
and `~DataFrame.bfill` is equivalent to ``fillna(method='bfill')``  .. _missing_data.PandasObject:  Filling with a PandasObject
~~~  You can also fillna using a dict or Series that is alignable. The labels of the dict or index of the Series
must match the columns of the frame you wish to fill. The
use case of this is to fill a DataFrame with the mean of that column.  .. ipython:: python  dff = pd.DataFrame(np.random.randn(10, 3), columns=list("ABC"))  dff.iloc[3:5, 0] = np.nan  dff.iloc[4:6, 1] = np.nan  dff.iloc[5:8, 2] = np.nan  dff  dff.fillna(dff.mean())  dff.fillna(dff.mean()["B":"C"])  Same result as above, but is aligning the 'fill' value which is
a Series in this case.  .. ipython:: python  dff.where(pd.notna(dff), dff.mean(), axis="columns")  .. _missing_data.dropna:  Dropping axis labels with missing data: dropna
~~~  You may wish to simply exclude labels from a data set which refer to missing
data. To do this, use `~DataFrame.dropna`:  .. ipython:: python  df["two"] = df["two"].fillna(0)  df["three"] = df["three"].fillna(0)  .. ipython:: python  df  df.dropna(axis=0)  df.dropna(axis=1)  df["one"].dropna()  An equivalent `~Series.dropna` is available for Series.
DataFrame.dropna has considerably more options than Series.dropna, which can be
examined `in the API <api.dataframe.missing>`.  .. _missing_data.interpolate:  Interpolation
~~~  Both Series and DataFrame objects have `~DataFrame.interpolate`
that, by default, performs linear interpolation at missing data points.  .. ipython:: python  np.random.seed(123456)  idx = pd.date_range("1/1/2000", periods=100, freq="BM")  ts = pd.Series(np.random.randn(100), index=idx)  ts[1:5] = np.nan  ts[20:30] = np.nan  ts[60:80] = np.nan  ts = ts.cumsum()  .. ipython:: python  ts  ts.count()  @savefig series_before_interpolate.png  ts.plot()  .. ipython:: python  ts.interpolate()  ts.interpolate().count()  @savefig series_interpolate.png  ts.interpolate().plot()  Index aware interpolation is available via the ``method`` keyword:  .. ipython:: python  ts2 = ts[[0, 1, 30, 60, 99]]  .. ipython:: python  ts2  ts2.interpolate()  ts2.interpolate(method="time")  For a floating-point index, use ``method='values'``:  .. ipython:: python  idx = [0.0, 1.0, 10.0]  ser = pd.Series([0.0, np.nan, 10.0], idx)  .. ipython:: python  ser  ser.interpolate()  ser.interpolate(method="values")  You can also interpolate with a DataFrame:  .. ipython:: python  df = pd.DataFrame(  {  "A": [1, 2.1, np.nan, 4.7, 5.6, 6.8],  "B": [0.25, np.nan, np.nan, 4, 12.2, 14.4],  }  )  df  df.interpolate()  The ``method`` argument gives access to fancier interpolation methods.
If you have scipy_ installed, you can pass the name of a 1-d interpolation routine to ``method``.

You'll want to consult the full scipy interpolation documentation_ and reference guide_ for details.
The appropriate interpolation method will depend on the type of data you are working with.  * If you are dealing with a time series that is growing at an increasing rate,  ``method='quadratic'`` may be appropriate.
* If you have values approximating a cumulative distribution function,  then ``method='pchip'`` should work well.
* To fill missing values with goal of smooth plotting, consider ``method='akima'``.  .. warning::  These methods require ``scipy``.  .. ipython:: python  df.interpolate(method="barycentric")  df.interpolate(method="pchip")  df.interpolate(method="akima")  When interpolating via a polynomial or spline approximation, you must also specify
the degree or order of the approximation:  .. ipython:: python  df.interpolate(method="spline", order=2)  df.interpolate(method="polynomial", order=2)  Compare several methods:  .. ipython:: python  np.random.seed(2)  ser = pd.Series(np.arange(1, 10.1, 0.25) ** 2 + np.random.randn(37))  missing = np.array([4, 13, 14, 15, 16, 17, 18, 20, 29])  ser[missing] = np.nan  methods = ["linear", "quadratic", "cubic"]  df = pd.DataFrame({m: ser.interpolate(method=m) for m in methods})  @savefig compare_interpolations.png  df.plot()  Another use case is interpolation at *new* values.
Suppose you have 100 observations from some distribution. And let's suppose
that you're particularly interested in what's happening around the middle.
You can mix pandas' ``reindex`` and ``interpolate`` methods to interpolate
at the new values.  .. ipython:: python  ser = pd.Series(np.sort(np.random.uniform(size=100)))  # interpolate at new_index  new_index = ser.index.union(pd.Index([49.25, 49.5, 49.75, 50.25, 50.5, 50.75]))  interp_s = ser.reindex(new_index).interpolate(method="pchip")  interp_s[49:51]  .. _scipy: https://scipy.org/
.. _documentation: https://docs.scipy.org/doc/scipy/reference/interpolate.html#univariate-interpolation
.. _guide: https://docs.scipy.org/doc/scipy/tutorial/interpolate.html  .. _missing_data.interp_limits:

Like other pandas fill methods, `~DataFrame.interpolate` accepts a ``limit`` keyword
argument. Use this argument to limit the number of consecutive ``NaN`` values
filled since the last valid observation:  .. ipython:: python  ser = pd.Series([np.nan, np.nan, 5, np.nan, np.nan, np.nan, 13, np.nan, np.nan])  ser  # fill all consecutive values in a forward direction  ser.interpolate()  # fill one consecutive value in a forward direction  ser.interpolate(limit=1)  By default, ``NaN`` values are filled in a ``forward`` direction. Use
``limit_direction`` parameter to fill ``backward`` or from ``both`` directions.  .. ipython:: python  # fill one consecutive value backwards  ser.interpolate(limit=1, limit_direction="backward")  # fill one consecutive value in both directions  ser.interpolate(limit=1, limit_direction="both")  # fill all consecutive values in both directions  ser.interpolate(limit_direction="both")  By default, ``NaN`` values are filled whether they are inside (surrounded by)
existing valid values, or outside existing valid values. The ``limit_area``
parameter restricts filling to either inside or outside values.  .. ipython:: python  # fill one consecutive inside value in both directions  ser.interpolate(limit_direction="both", limit_area="inside", limit=1)  # fill all consecutive outside values backward  ser.interpolate(limit_direction="backward", limit_area="outside")  # fill all consecutive outside values in both directions  ser.interpolate(limit_direction="both", limit_area="outside")  .. _missing_data.replace:  Replacing generic values
~~~
Often times we want to replace arbitrary values with other values.  `~Series.replace` in Series and `~DataFrame.replace` in DataFrame provides an efficient yet
flexible way to perform such replacements.  For a Series, you can replace a single value or a list of values by another
value:  .. ipython:: python  ser = pd.Series([0.0, 1.0, 2.0, 3.0, 4.0])  ser.replace(0, 5)  You can replace a list of values by a list of other values:  .. ipython:: python  ser.replace([0, 1, 2, 3, 4], [4, 3, 2, 1, 0])  You can also specify a mapping dict:  .. ipython:: python  ser.replace({0: 10, 1: 100})  For a DataFrame, you can specify individual values by column:  .. ipython:: python  df = pd.DataFrame({"a": [0, 1, 2, 3, 4], "b": [5, 6, 7, 8, 9]})  df.replace({"a": 0, "b": 5}, 100)  Instead of replacing with specified values, you can treat all given values as
missing and interpolate over them:  .. ipython:: python  ser.replace([1, 2, 3], method="pad")  .. _missing_data.replace_expression:  String/regular expression replacement
~~~  .. note::  Python strings prefixed with the ``r`` character such as ``r'hello world'``  are so-called "raw" strings. They have different semantics regarding  backslashes than strings without this prefix. Backslashes in raw strings  will be interpreted as an escaped backslash, e.g., ``r'\' == '\\'``. You  should `read about them  <https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals>`__  if this is unclear.  Replace the '.' with ``NaN`` (str -> str):  .. ipython:: python  d = {"a": list(range(4)), "b": list("ab.."), "c": ["a", "b", np.nan, "d"]}  df = pd.DataFrame(d)  df.replace(".", np.nan)  Now do it with a regular expression that removes surrounding whitespace
(regex -> regex):  .. ipython:: python  df.replace(r"\s*\.\s*", np.nan, regex=True)  Replace a few different values (list -> list):  .. ipython:: python  df.replace(["a", "."], ["b", np.nan])  list of regex -> list of regex:  .. ipython:: python  df.replace([r"\.", r"(a)"], ["dot", r"\1stuff"], regex=True)  Only search in column ``'b'`` (dict -> dict):  .. ipython:: python  df.replace({"b": "."}, {"b": np.nan})  Same as the previous example, but use a regular expression for

searching instead (dict of regex -> dict):  .. ipython:: python  df.replace({"b": r"\s*\.\s*"}, {"b": np.nan}, regex=True)  You can pass nested dictionaries of regular expressions that use ``regex=True``:  .. ipython:: python  df.replace({"b": {"b": r""}}, regex=True)  Alternatively, you can pass the nested dictionary like so:  .. ipython:: python  df.replace(regex={"b": {r"\s*\.\s*": np.nan}})  You can also use the group of a regular expression match when replacing (dict
of regex -> dict of regex), this works for lists as well.  .. ipython:: python  df.replace({"b": r"\s*(\.)\s*"}, {"b": r"\1ty"}, regex=True)  You can pass a list of regular expressions, of which those that match
will be replaced with a scalar (list of regex -> regex).  .. ipython:: python  df.replace([r"\s*\.\s*", r"a|b"], np.nan, regex=True)  All of the regular expression examples can also be passed with the
``to_replace`` argument as the ``regex`` argument. In this case the ``value``
argument must be passed explicitly by name or ``regex`` must be a nested
dictionary. The previous example, in this case, would then be:  .. ipython:: python  df.replace(regex=[r"\s*\.\s*", r"a|b"], value=np.nan)  This can be convenient if you do not want to pass ``regex=True`` every time you
want to use a regular expression.  .. note::  Anywhere in the above ``replace`` examples that you see a regular expression  a compiled regular expression is valid as well.  Numeric replacement
~~~  `~DataFrame.replace` is similar to `~DataFrame.fillna`.  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 2))  df[np.random.rand(df.shape[0]) > 0.5] = 1.5  df.replace(1.5, np.nan)  Replacing more than one value is possible by passing a list.  .. ipython:: python  df00 = df.iloc[0, 0]  df.replace([1.5, df00], [np.nan, "a"])  df[1].dtype

Missing data casting rules and indexing

While pandas supports storing arrays of integer and boolean type, these types
are not capable of storing missing data. Until we can switch to using a native
NA type in NumPy, we've established some "casting rules". When a reindexing
operation introduces missing data, the Series will be cast according to the
rules introduced in the table below.  .. csv-table::  "data type", "Cast to"  40, 40  integer, float  boolean, object  float, no cast  object, no cast  For example:  .. ipython:: python  s = pd.Series(np.random.randn(5), index=[0, 2, 4, 6, 7])  s > 0  (s > 0).dtype  crit = (s > 0).reindex(list(range(8)))  crit  crit.dtype  Ordinarily NumPy will complain if you try to use an object array (even if it
contains boolean values) instead of a boolean array to get or set values from
an ndarray (e.g. selecting values based on some criteria). If a boolean vector
contains NAs, an exception will be generated:  .. ipython:: python  reindexed = s.reindex(list(range(8))).fillna(0)  reindexed[crit]  However, these can be filled in using `~DataFrame.fillna` and it will work fine:  .. ipython:: python  reindexed[crit.fillna(False)]  reindexed[crit.fillna(True)]  pandas provides a nullable integer dtype, but you must explicitly request it
when creating the series or column. Notice that we use a capital "I" in
the ``dtype="Int64"``.  .. ipython:: python  s = pd.Series([0, 1, np.nan, 3, 4], dtype="Int64")  s  See `integer_na` for more.  .. _missing_data.NA:  Experimental ``NA`` scalar to denote missing values
~~~  .. warning::  Experimental: the behaviour of ``pd.NA`` can still change without warning.  .. versionadded:: 1.0.0  Starting from pandas 1.0, an experimental ``pd.NA`` value (singleton) is
available to represent scalar missing values. At this moment, it is used in
the nullable `integer <integer_na>`, boolean and
`dedicated string <text.types>` data types as the missing value indicator.  The goal of ``pd.NA`` is provide a "missing" indicator that can be used
consistently across data types (instead of ``np.nan``, ``None`` or ``pd.NaT``
depending on the data type).  For example, when having missing values in a Series with the nullable integer
dtype, it will use ``pd.NA``:  .. ipython:: python  s = pd.Series([1, 2, None], dtype="Int64")  s  s[2]  s[2] is pd.NA  Currently, pandas does not yet use those data types by default (when creating
a DataFrame or Series, or when reading in data), so you need to specify
the dtype explicitly.  An easy way to convert to those dtypes is explained
`here <missing_data.NA.conversion>`.

Propagation in arithmetic and comparison operations

In general, missing values *propagate* in operations involving ``pd.NA``. When
one of the operands is unknown, the outcome of the operation is also unknown.  For example, ``pd.NA`` propagates in arithmetic operations, similarly to
``np.nan``:  .. ipython:: python  pd.NA + 1  "a" * pd.NA  There are a few special cases when the result is known, even when one of the
operands is ``NA``.  .. ipython:: python  pd.NA ** 0  1 ** pd.NA  In equality and comparison operations, ``pd.NA`` also propagates. This deviates
from the behaviour of ``np.nan``, where comparisons with ``np.nan`` always
return ``False``.  .. ipython:: python  pd.NA == 1  pd.NA == pd.NA  pd.NA < 2.5  To check if a value is equal to ``pd.NA``, the `isna` function can be
used:  .. ipython:: python  pd.isna(pd.NA)  An exception on this basic propagation rule are *reductions* (such as the
mean or the minimum), where pandas defaults to skipping missing values. See
`above <missing_data.calculations>` for more.

Logical operations

For logical operations, ``pd.NA`` follows the rules of the
three-valued logic_ (or
*Kleene logic*, similarly to R, SQL and Julia). This logic means to only
propagate missing values when it is logically required.  For example, for the logical "or" operation (``|``), if one of the operands
is ``True``, we already know the result will be ``True``, regardless of the
other value (so regardless the missing value would be ``True`` or ``False``).
In this case, ``pd.NA`` does not propagate:  .. ipython:: python  True | False  True | pd.NA  pd.NA | True  On the other hand, if one of the operands is ``False``, the result depends
on the value of the other operand. Therefore, in this case ``pd.NA``
propagates:  .. ipython:: python  False | True  False | False  False | pd.NA  The behaviour of the logical "and" operation (``&``) can be derived using
similar logic (where now ``pd.NA`` will not propagate if one of the operands
is already ``False``):  .. ipython:: python  False & True  False & False  False & pd.NA  .. ipython:: python  True & True  True & False  True & pd.NA

``NA`` in a boolean context

Since the actual value of an NA is unknown, it is ambiguous to convert NA
to a boolean value. The following raises an error:  .. ipython:: python  bool(pd.NA)  This also means that ``pd.NA`` cannot be used in a context where it is
evaluated to a boolean, such as ``if condition: ...`` where ``condition`` can
potentially be ``pd.NA``. In such cases, `isna` can be used to check
for ``pd.NA`` or ``condition`` being ``pd.NA`` can be avoided, for example by
filling missing values beforehand.  A similar situation occurs when using Series or DataFrame objects in ``if``
statements, see `gotchas.truth`.

NumPy ufuncs

`pandas.NA` implements NumPy's ``__array_ufunc__`` protocol. Most ufuncs
work with ``NA``, and generally return ``NA``:  .. ipython:: python  np.log(pd.NA)  np.add(pd.NA, 1)  .. warning::  Currently, ufuncs involving an ndarray and ``NA`` will return an  object-dtype filled with NA values.  .. ipython:: python  a = np.array([1, 2, 3])  np.greater(a, pd.NA)  The return type here may change to return a different array type  in the future.  See `dsintro.numpy_interop` for more on ufuncs.  .. _missing_data.NA.conversion:

If you have a DataFrame or Series using traditional types that have missing data
represented using ``np.nan``, there are convenience methods
`~Series.convert_dtypes` in Series and `~DataFrame.convert_dtypes`
in DataFrame that can convert data to use the newer dtypes for integers, strings and
booleans listed `here <basics.dtypes>`. This is especially helpful after reading
in data sets when letting the readers such as `read_csv` and `read_excel`
infer default dtypes.  In this example, while the dtypes of all columns are changed, we show the results for
the first 10 columns.  .. ipython:: python  bb = pd.read_csv("data/baseball.csv", index_col="id")  bb[bb.columns[:10]].dtypes  .. ipython:: python  bbn = bb.convert_dtypes()  bbn[bbn.columns[:10]].dtypes

Overview
pandas has an options API configure and customize global behavior related to
`DataFrame` display, data behavior and more.  Options have a full "dotted-style", case-insensitive name (e.g. ``display.max_rows``).
You can get/set options directly as attributes of the top-level ``options`` attribute:  .. ipython:: python  import pandas as pd  pd.options.display.max_rows  pd.options.display.max_rows = 999  pd.options.display.max_rows  The API is composed of 5 relevant functions, available directly from the ``pandas``
namespace:  * `~pandas.get_option` / `~pandas.set_option` - get/set the value of a single option.
* `~pandas.reset_option` - reset one or more options to their default value.
* `~pandas.describe_option` - print the descriptions of one or more options.
* `~pandas.option_context` - execute a codeblock with a set of options  that revert to prior settings after execution.  .. note::  Developers can check out pandas/core/config_init.py for more information.  All of the functions above accept a regexp pattern (``re.search`` style) as an argument,
to match an unambiguous substring:  .. ipython:: python  pd.get_option("display.chop_threshold")  pd.set_option("display.chop_threshold", 2)  pd.get_option("display.chop_threshold")  pd.set_option("chop", 4)  pd.get_option("display.chop_threshold")  The following will **not work** because it matches multiple option names, e.g.
``display.max_colwidth``, ``display.max_rows``, ``display.max_columns``:  .. ipython:: python  pd.get_option("max")  .. warning::  Using this form of shorthand may cause your code to break if new options with similar names are added in future versions.  .. ipython:: python  pd.reset_option("all")  .. _options.available:

Available options

You can get a list of available options and their descriptions with `~pandas.describe_option`. When called
with no argument `~pandas.describe_option` will print out the descriptions for all available options.  .. ipython:: python  pd.describe_option()

Getting and setting options

As described above, `~pandas.get_option` and `~pandas.set_option`
are available from the pandas namespace.  To change an option, call
``set_option('option regex', new_value)``.  .. ipython:: python  pd.get_option("mode.sim_interactive")  pd.set_option("mode.sim_interactive", True)  pd.get_option("mode.sim_interactive")  .. note::  The option ``'mode.sim_interactive'`` is mostly used for debugging purposes.  You can use `~pandas.reset_option` to revert to a setting's default value  .. ipython:: python  pd.reset_option("display.max_rows")  .. ipython:: python  pd.get_option("display.max_rows")  pd.set_option("display.max_rows", 999)  pd.get_option("display.max_rows")  pd.reset_option("display.max_rows")  pd.get_option("display.max_rows")  It's also possible to reset multiple options at once (using a regex):  .. ipython:: python  pd.reset_option("^display")  `~pandas.option_context` context manager has been exposed through
the top-level API, allowing you to execute code with given option values. Option values
are restored automatically when you exit the ``with`` block:  .. ipython:: python  with pd.option_context("display.max_rows", 10, "display.max_columns", 5):  print(pd.get_option("display.max_rows"))  print(pd.get_option("display.max_columns"))  print(pd.get_option("display.max_rows"))  print(pd.get_option("display.max_columns"))

Setting startup options in Python/IPython environment

Using startup scripts for the Python/IPython environment to import pandas and set options makes working with pandas more efficient.
To do this, create a ``.py`` or ``.ipy`` script in the startup directory of the desired profile.
An example where the startup folder is in a default IPython profile can be found at:  .. code-block:: none  $IPYTHONDIR/profile_default/startup  More information can be found in the `IPython documentation
<https://ipython.org/ipython-doc/stable/interactive/tutorial.html#startup-files>`__.  An example startup script for pandas is displayed below:  .. code-block:: python  import pandas as pd  pd.set_option("display.max_rows", 999)  pd.set_option("display.precision", 5)  .. _options.frequently_used:

Frequently used options
The following is a demonstrates the more frequently used display options.  ``display.max_rows`` and ``display.max_columns`` sets the maximum number
of rows and columns displayed when a frame is pretty-printed. Truncated
lines are replaced by an ellipsis.  .. ipython:: python  df = pd.DataFrame(np.random.randn(7, 2))  pd.set_option("display.max_rows", 7)  df  pd.set_option("display.max_rows", 5)  df  pd.reset_option("display.max_rows")  Once the ``display.max_rows`` is exceeded, the ``display.min_rows`` options
determines how many rows are shown in the truncated repr.  .. ipython:: python  pd.set_option("display.max_rows", 8)  pd.set_option("display.min_rows", 4)  # below max_rows -> all rows shown  df = pd.DataFrame(np.random.randn(7, 2))  df  # above max_rows -> only min_rows (4) rows shown  df = pd.DataFrame(np.random.randn(9, 2))  df  pd.reset_option("display.max_rows")  pd.reset_option("display.min_rows")  ``display.expand_frame_repr`` allows for the representation of a
`DataFrame` to stretch across pages, wrapped over the all the columns.  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 10))  pd.set_option("expand_frame_repr", True)  df  pd.set_option("expand_frame_repr", False)  df  pd.reset_option("expand_frame_repr")  ``display.large_repr`` displays a `DataFrame` that exceed
``max_columns`` or ``max_rows`` as a truncated frame or summary.  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 10))  pd.set_option("display.max_rows", 5)  pd.set_option("large_repr", "truncate")  df  pd.set_option("large_repr", "info")  df  pd.reset_option("large_repr")  pd.reset_option("display.max_rows")  ``display.max_colwidth`` sets the maximum width of columns.  Cells
of this length or longer will be truncated with an ellipsis.  .. ipython:: python  df = pd.DataFrame(  np.array(  [  ["foo", "bar", "bim", "uncomfortably long string"],  ["horse", "cow", "banana", "apple"],  ]  )  )  pd.set_option("max_colwidth", 40)  df  pd.set_option("max_colwidth", 6)  df  pd.reset_option("max_colwidth")  ``display.max_info_columns`` sets a threshold for the number of columns
displayed when calling `~pandas.DataFrame.info`.  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 10))  pd.set_option("max_info_columns", 11)  df.info()  pd.set_option("max_info_columns", 5)  df.info()  pd.reset_option("max_info_columns")  ``display.max_info_rows``: `~pandas.DataFrame.info` will usually show null-counts for each column.
For a large `DataFrame`, this can be quite slow. ``max_info_rows`` and ``max_info_cols``
limit this null check to the specified rows and columns respectively. The `~pandas.DataFrame.info`
keyword argument ``show_counts=True`` will override this.  .. ipython:: python  df = pd.DataFrame(np.random.choice([0, 1, np.nan], size=(10, 10)))  df  pd.set_option("max_info_rows", 11)  df.info()  pd.set_option("max_info_rows", 5)  df.info()  pd.reset_option("max_info_rows")  ``display.precision`` sets the output display precision in terms of decimal places.  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 5))  pd.set_option("display.precision", 7)  df  pd.set_option("display.precision", 4)  df  ``display.chop_threshold`` sets the rounding threshold to zero when displaying a
`Series` or `DataFrame`. This setting does not change the
precision at which the number is stored.  .. ipython:: python  df = pd.DataFrame(np.random.randn(6, 6))  pd.set_option("chop_threshold", 0)  df  pd.set_option("chop_threshold", 0.5)  df  pd.reset_option("chop_threshold")  ``display.colheader_justify`` controls the justification of the headers.
The options are ``'right'``, and ``'left'``.  .. ipython:: python  df = pd.DataFrame(  np.array([np.random.randn(6), np.random.randint(1, 9, 6) * 0.1, np.zeros(6)]).T,  columns=["A", "B", "C"],  dtype="float",  )  pd.set_option("colheader_justify", "right")  df  pd.set_option("colheader_justify", "left")  df  pd.reset_option("colheader_justify")  .. _basics.console_output:

Number formatting

pandas also allows you to set how numbers are displayed in the console.
This option is not set through the ``set_options`` API.  Use the ``set_eng_float_format`` function
to alter the floating-point formatting of pandas objects to produce a particular
format.  .. ipython:: python  import numpy as np  pd.set_eng_float_format(accuracy=3, use_eng_prefix=True)  s = pd.Series(np.random.randn(5), index=["a", "b", "c", "d", "e"])  s / 1.0e3  s / 1.0e6  .. ipython:: python  pd.reset_option("^display")  Use `~pandas.DataFrame.round` to specifically control rounding of an individual `DataFrame`  .. _options.east_asian_width:

Unicode formatting

.. warning::  Enabling this option will affect the performance for printing of DataFrame and Series (about 2 times slower).  Use only when it is actually required.  Some East Asian countries use Unicode characters whose width corresponds to two Latin characters.
If a DataFrame or Series contains these characters, the default output mode may not align them properly.  .. ipython:: python  df = pd.DataFrame({"国籍": ["UK", "日本"], "名前": ["Alice", "しのぶ"]})  df  Enabling ``display.unicode.east_asian_width`` allows pandas to check each character's "East Asian Width" property.
These characters can be aligned properly by setting this option to ``True``. However, this will result in longer render
times than the standard ``len`` function.  .. ipython:: python  pd.set_option("display.unicode.east_asian_width", True)  df  In addition, Unicode characters whose width is "ambiguous" can either be 1 or 2 characters wide depending on the
terminal setting or encoding. The option ``display.unicode.ambiguous_as_wide`` can be used to handle the ambiguity.  By default, an "ambiguous" character's width, such as "¡" (inverted exclamation) in the example below, is taken to be 1.  .. ipython:: python  df = pd.DataFrame({"a": ["xxx", "¡¡"], "b": ["yyy", "¡¡"]})  df  Enabling ``display.unicode.ambiguous_as_wide`` makes pandas interpret these characters' widths to be 2.
(Note that this option will only be effective when ``display.unicode.east_asian_width`` is enabled.)  However, setting this option incorrectly for your terminal will cause these characters to be aligned incorrectly:  .. ipython:: python  pd.set_option("display.unicode.ambiguous_as_wide", True)  df  .. ipython:: python  pd.set_option("display.unicode.east_asian_width", False)  pd.set_option("display.unicode.ambiguous_as_wide", False)  .. _options.table_schema:

Table schema display

`DataFrame` and `Series` will publish a Table Schema representation
by default. This can be enabled globally with the
``display.html.table_schema`` option:  .. ipython:: python  pd.set_option("display.html.table_schema", True)  Only ``'display.max_rows'`` are serialized and published.  .. ipython:: python  pd.reset_option("display.html.table_schema")

Reshaping by pivoting DataFrame objects
  Data is often stored in so-called "stacked" or "record" format:  .. ipython:: python  import pandas._testing as tm  def unpivot(frame):  N, K = frame.shape  data = {  "value": frame.to_numpy().ravel("F"),  "variable": np.asarray(frame.columns).repeat(N),  "date": np.tile(np.asarray(frame.index), K),  }  return pd.DataFrame(data, columns=["date", "variable", "value"])  df = unpivot(tm.makeTimeDataFrame(3))  df  To select out everything for variable ``A`` we could do:  .. ipython:: python  filtered = df[df["variable"] == "A"]  filtered  But suppose we wish to do time series operations with the variables. A better
representation would be where the ``columns`` are the unique variables and an
``index`` of dates identifies individual observations. To reshape the data into
this form, we use the `DataFrame.pivot` method (also implemented as a
top level function `~pandas.pivot`):  .. ipython:: python  pivoted = df.pivot(index="date", columns="variable", values="value")  pivoted  If the ``values`` argument is omitted, and the input `DataFrame` has more than
one column of values which are not used as column or index inputs to `~DataFrame.pivot`,
then the resulting "pivoted" `DataFrame` will have `hierarchical columns
<advanced.hierarchical>` whose topmost level indicates the respective value
column:  .. ipython:: python  df["value2"] = df["value"] * 2  pivoted = df.pivot(index="date", columns="variable")  pivoted  You can then select subsets from the pivoted `DataFrame`:  .. ipython:: python  pivoted["value2"]  Note that this returns a view on the underlying data in the case where the data
are homogeneously-typed.  .. note::  `~pandas.pivot` will error with a ``ValueError: Index contains duplicate  entries, cannot reshape`` if the index/column pair is not unique. In this  case, consider using `~pandas.pivot_table` which is a generalization  of pivot that can handle duplicate values for one index/column pair.  .. _reshaping.stacking:

Reshaping by stacking and unstacking
  Closely related to the `~DataFrame.pivot` method are the related
`~DataFrame.stack` and `~DataFrame.unstack` methods available on
`Series` and `DataFrame`. These methods are designed to work together with
`MultiIndex` objects (see the section on `hierarchical indexing
<advanced.hierarchical>`). Here are essentially what these methods do:  * `~DataFrame.stack`: "pivot" a level of the (possibly hierarchical) column labels,  returning a `DataFrame` with an index with a new inner-most level of row  labels.
* `~DataFrame.unstack`: (inverse operation of `~DataFrame.stack`) "pivot" a level of the  (possibly hierarchical) row index to the column axis, producing a reshaped  `DataFrame` with a new inner-most level of column labels.  The clearest way to explain is by example. Let's take a prior example data set
from the hierarchical indexing section:  .. ipython:: python  tuples = list(  zip(  *[  ["bar", "bar", "baz", "baz", "foo", "foo", "qux", "qux"],  ["one", "two", "one", "two", "one", "two", "one", "two"],  ]  )  )  index = pd.MultiIndex.from_tuples(tuples, names=["first", "second"])  df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=["A", "B"])  df2 = df[:4]  df2  The `~DataFrame.stack` function "compresses" a level in the `DataFrame` columns to
produce either:  * A `Series`, in the case of a simple column Index.
* A `DataFrame`, in the case of a `MultiIndex` in the columns.  If the columns have a `MultiIndex`, you can choose which level to stack. The
stacked level becomes the new lowest level in a `MultiIndex` on the columns:  .. ipython:: python  stacked = df2.stack()  stacked  With a "stacked" `DataFrame` or `Series` (having a `MultiIndex` as the
``index``), the inverse operation of `~DataFrame.stack` is `~DataFrame.unstack`, which by default
unstacks the **last level**:  .. ipython:: python  stacked.unstack()  stacked.unstack(1)  stacked.unstack(0)  .. _reshaping.unstack_by_name:  If the indexes have names, you can use the level names instead of specifying
the level numbers:  .. ipython:: python  stacked.unstack("second")  Notice that the `~DataFrame.stack` and `~DataFrame.unstack` methods implicitly sort the index
levels involved. Hence a call to `~DataFrame.stack` and then `~DataFrame.unstack`, or vice versa,
will result in a **sorted** copy of the original `DataFrame` or `Series`:  .. ipython:: python  index = pd.MultiIndex.from_product([[2, 1], ["a", "b"]])  df = pd.DataFrame(np.random.randn(4), index=index, columns=["A"])  df  all(df.unstack().stack() == df.sort_index())  The above code will raise a ``TypeError`` if the call to `~DataFrame.sort_index` is
removed.  .. _reshaping.stack_multiple:  Multiple levels
~~~  You may also stack or unstack more than one level at a time by passing a list
of levels, in which case the end result is as if each level in the list were
processed individually.  .. ipython:: python  columns = pd.MultiIndex.from_tuples(  [  ("A", "cat", "long"),  ("B", "cat", "long"),  ("A", "dog", "short"),  ("B", "dog", "short"),  ],  names=["exp", "animal", "hair_length"],  )  df = pd.DataFrame(np.random.randn(4, 4), columns=columns)  df  df.stack(level=["animal", "hair_length"])  The list of levels can contain either level names or level numbers (but
not a mixture of the two).  .. ipython:: python  # df.stack(level=['animal', 'hair_length'])  # from above is equivalent to:  df.stack(level=[1, 2])  Missing data
~~~  These functions are intelligent about handling missing data and do not expect
each subgroup within the hierarchical index to have the same set of labels.
They also can handle the index being unsorted (but you can make it sorted by

calling `~DataFrame.sort_index`, of course). Here is a more complex example:  .. ipython:: python  columns = pd.MultiIndex.from_tuples(  [  ("A", "cat"),  ("B", "dog"),  ("B", "cat"),  ("A", "dog"),  ],  names=["exp", "animal"],  )  index = pd.MultiIndex.from_product(  [("bar", "baz", "foo", "qux"), ("one", "two")], names=["first", "second"]  )  df = pd.DataFrame(np.random.randn(8, 4), index=index, columns=columns)  df2 = df.iloc[[0, 1, 2, 4, 5, 7]]  df2  As mentioned above, `~DataFrame.stack` can be called with a ``level`` argument to select
which level in the columns to stack:  .. ipython:: python  df2.stack("exp")  df2.stack("animal")  Unstacking can result in missing values if subgroups do not have the same
set of labels.  By default, missing values will be replaced with the default
fill value for that data type, ``NaN`` for float, ``NaT`` for datetimelike,
etc.  For integer types, by default data will converted to float and missing
values will be set to ``NaN``.  .. ipython:: python  df3 = df.iloc[[0, 1, 4, 7], [1, 2]]  df3  df3.unstack()  Alternatively, unstack takes an optional ``fill_value`` argument, for specifying
the value of missing data.  .. ipython:: python  df3.unstack(fill_value=-1e9)  With a MultiIndex
~~~  Unstacking when the columns are a `MultiIndex` is also careful about doing
the right thing:  .. ipython:: python  df[:3].unstack(0)  df2.unstack(1)  .. _reshaping.melt:

Reshaping by melt
  The top-level `~pandas.melt` function and the corresponding `DataFrame.melt`
are useful to massage a `DataFrame` into a format where one or more columns
are *identifier variables*, while all other columns, considered *measured
variables*, are "unpivoted" to the row axis, leaving just two non-identifier
columns, "variable" and "value". The names of those columns can be customized
by supplying the ``var_name`` and ``value_name`` parameters.  For instance,  .. ipython:: python  cheese = pd.DataFrame(  {  "first": ["John", "Mary"],  "last": ["Doe", "Bo"],  "height": [5.5, 6.0],  "weight": [130, 150],  }  )  cheese  cheese.melt(id_vars=["first", "last"])  cheese.melt(id_vars=["first", "last"], var_name="quantity")  When transforming a DataFrame using `~pandas.melt`, the index will be ignored. The original index values can be kept around by setting the ``ignore_index`` parameter to ``False`` (default is ``True``). This will however duplicate them.  .. versionadded:: 1.1.0  .. ipython:: python  index = pd.MultiIndex.from_tuples([("person", "A"), ("person", "B")])  cheese = pd.DataFrame(  {  "first": ["John", "Mary"],  "last": ["Doe", "Bo"],  "height": [5.5, 6.0],  "weight": [130, 150],  },  index=index,  )  cheese  cheese.melt(id_vars=["first", "last"])  cheese.melt(id_vars=["first", "last"], ignore_index=False)  Another way to transform is to use the `~pandas.wide_to_long` panel data
convenience function. It is less flexible than `~pandas.melt`, but more
user-friendly.  .. ipython:: python  dft = pd.DataFrame(  {  "A1970": {0: "a", 1: "b", 2: "c"},  "A1980": {0: "d", 1: "e", 2: "f"},  "B1970": {0: 2.5, 1: 1.2, 2: 0.7},  "B1980": {0: 3.2, 1: 1.3, 2: 0.1},  "X": dict(zip(range(3), np.random.randn(3))),  }  )  dft["id"] = dft.index  dft  pd.wide_to_long(dft, ["A", "B"], i="id", j="year")  .. _reshaping.combine_with_groupby:

Combining with stats and GroupBy

It should be no shock that combining `~DataFrame.pivot` / `~DataFrame.stack` / `~DataFrame.unstack` with
GroupBy and the basic Series and DataFrame statistical functions can produce
some very expressive and fast data manipulations.  .. ipython:: python  df  df.stack().mean(1).unstack()  # same result, another way  df.groupby(level=1, axis=1).mean()  df.stack().groupby(level=1).mean()  df.mean().unstack(0)

Pivot tables

.. _reshaping.pivot:  While `~DataFrame.pivot` provides general purpose pivoting with various
data types (strings, numerics, etc.), pandas also provides `~pandas.pivot_table`
for pivoting with aggregation of numeric data.  The function `~pandas.pivot_table` can be used to create spreadsheet-style
pivot tables. See the `cookbook<cookbook.pivot>` for some advanced
strategies.  It takes a number of arguments:  * ``data``: a DataFrame object.
* ``values``: a column or a list of columns to aggregate.
* ``index``: a column, Grouper, array which has the same length as data, or list of them.  Keys to group by on the pivot table index. If an array is passed, it is being used as the same manner as column values.
* ``columns``: a column, Grouper, array which has the same length as data, or list of them.  Keys to group by on the pivot table column. If an array is passed, it is being used as the same manner as column values.
* ``aggfunc``: function to use for aggregation, defaulting to ``numpy.mean``.  Consider a data set like this:  .. ipython:: python  import datetime  df = pd.DataFrame(  {  "A": ["one", "one", "two", "three"] * 6,  "B": ["A", "B", "C"] * 8,  "C": ["foo", "foo", "foo", "bar", "bar", "bar"] * 4,  "D": np.random.randn(24),  "E": np.random.randn(24),  "F": [datetime.datetime(2013, i, 1) for i in range(1, 13)]  + [datetime.datetime(2013, i, 15) for i in range(1, 13)],  }  )  df  We can produce pivot tables from this data very easily:  .. ipython:: python  pd.pivot_table(df, values="D", index=["A", "B"], columns=["C"])  pd.pivot_table(df, values="D", index=["B"], columns=["A", "C"], aggfunc=np.sum)  pd.pivot_table(  df, values=["D", "E"],  index=["B"],  columns=["A", "C"],  aggfunc=np.sum,  )  The result object is a `DataFrame` having potentially hierarchical indexes on the
rows and columns. If the ``values`` column name is not given, the pivot table
will include all of the data in an additional level of hierarchy in the columns:  .. ipython:: python  pd.pivot_table(df[["A", "B", "C", "D", "E"]], index=["A", "B"], columns=["C"])  Also, you can use `Grouper` for ``index`` and ``columns`` keywords. For detail of `Grouper`, see `Grouping with a Grouper specification <groupby.specify>`.  .. ipython:: python  pd.pivot_table(df, values="D", index=pd.Grouper(freq="M", key="F"), columns="C")  You can render a nice output of the table omitting the missing values by
calling `~DataFrame.to_string` if you wish:  .. ipython:: python  table = pd.pivot_table(df, index=["A", "B"], columns=["C"], values=["D", "E"])  print(table.to_string(na_rep=""))  Note that `~DataFrame.pivot_table` is also available as an instance method on DataFrame,  i.e. `DataFrame.pivot_table`.  .. _reshaping.pivot.margins:  Adding margins
~~~  If you pass ``margins=True`` to `~DataFrame.pivot_table`, special ``All`` columns and
rows will be added with partial group aggregates across the categories on the
rows and columns:  .. ipython:: python  table = df.pivot_table(  index=["A", "B"],  columns="C",  values=["D", "E"],  margins=True,  aggfunc=np.std  )  table  Additionally, you can call `DataFrame.stack` to display a pivoted DataFrame
as having a multi-level index:  .. ipython:: python  table.stack()  .. _reshaping.crosstabulations:

Cross tabulations

Use `~pandas.crosstab` to compute a cross-tabulation of two (or more)
factors. By default `~pandas.crosstab` computes a frequency table of the factors
unless an array of values and an aggregation function are passed.  It takes a number of arguments  * ``index``: array-like, values to group by in the rows.
* ``columns``: array-like, values to group by in the columns.
* ``values``: array-like, optional, array of values to aggregate according to  the factors.
* ``aggfunc``: function, optional, If no values array is passed, computes a  frequency table.
* ``rownames``: sequence, default ``None``, must match number of row arrays passed.
* ``colnames``: sequence, default ``None``, if passed, must match number of column  arrays passed.
* ``margins``: boolean, default ``False``, Add row/column margins (subtotals)
* ``normalize``: boolean, {'all', 'index', 'columns'}, or {0,1}, default ``False``.  Normalize by dividing all values by the sum of values.  Any `Series` passed will have their name attributes used unless row or column
names for the cross-tabulation are specified  For example:  .. ipython:: python  foo, bar, dull, shiny, one, two = "foo", "bar", "dull", "shiny", "one", "two"  a = np.array([foo, foo, bar, bar, foo, foo], dtype=object)  b = np.array([one, one, two, one, two, one], dtype=object)  c = np.array([dull, dull, shiny, dull, dull, shiny], dtype=object)  pd.crosstab(a, [b, c], rownames=["a"], colnames=["b", "c"])  If `~pandas.crosstab` receives only two Series, it will provide a frequency table.  .. ipython:: python  df = pd.DataFrame(  {"A": [1, 2, 2, 2, 2], "B": [3, 3, 4, 4, 4], "C": [1, 1, np.nan, 1, 1]}  )  df  pd.crosstab(df["A"], df["B"])  `~pandas.crosstab` can also be implemented
to `Categorical` data.  .. ipython:: python  foo = pd.Categorical(["a", "b"], categories=["a", "b", "c"])  bar = pd.Categorical(["d", "e"], categories=["d", "e", "f"])  pd.crosstab(foo, bar)  If you want to include **all** of data categories even if the actual data does
not contain any instances of a particular category, you should set ``dropna=False``.  For example:  .. ipython:: python  pd.crosstab(foo, bar, dropna=False)  Normalization
~~~  Frequency tables can also be normalized to show percentages rather than counts
using the ``normalize`` argument:  .. ipython:: python  pd.crosstab(df["A"], df["B"], normalize=True)  ``normalize`` can also normalize values within each row or within each column:  .. ipython:: python  pd.crosstab(df["A"], df["B"], normalize="columns")  `~pandas.crosstab` can also be passed a third `Series` and an aggregation function
(``aggfunc``) that will be applied to the values of the third `Series` within
each group defined by the first two `Series`:  .. ipython:: python  pd.crosstab(df["A"], df["B"], values=df["C"], aggfunc=np.sum)  Adding margins
~~~  Finally, one can also add margins or normalize this output.  .. ipython:: python  pd.crosstab(  df["A"], df["B"], values=df["C"], aggfunc=np.sum, normalize=True, margins=True  )  .. _reshaping.tile:
.. _reshaping.tile.cut:

Tiling

The `~pandas.cut` function computes groupings for the values of the input
array and is often used to transform continuous variables to discrete or
categorical variables:  .. ipython:: python  ages = np.array([10, 15, 13, 12, 23, 25, 28, 59, 60])  pd.cut(ages, bins=3)  If the ``bins`` keyword is an integer, then equal-width bins are formed.
Alternatively we can specify custom bin-edges:  .. ipython:: python  c = pd.cut(ages, bins=[0, 18, 35, 70])  c  If the ``bins`` keyword is an `IntervalIndex`, then these will be
used to bin the passed data.::  pd.cut([25, 20, 50], bins=c.categories)  .. _reshaping.dummies:

Computing indicator / dummy variables

To convert a categorical variable into a "dummy" or "indicator" `DataFrame`,
for example a column in a `DataFrame` (a `Series`) which has ``k`` distinct
values, can derive a `DataFrame` containing ``k`` columns of 1s and 0s using
`~pandas.get_dummies`:  .. ipython:: python  df = pd.DataFrame({"key": list("bbacab"), "data1": range(6)})  pd.get_dummies(df["key"])  Sometimes it's useful to prefix the column names, for example when merging the result
with the original `DataFrame`:  .. ipython:: python  dummies = pd.get_dummies(df["key"], prefix="key")  dummies  df[["data1"]].join(dummies)  This function is often used along with discretization functions like `~pandas.cut`:  .. ipython:: python  values = np.random.randn(10)  values  bins = [0, 0.2, 0.4, 0.6, 0.8, 1]  pd.get_dummies(pd.cut(values, bins))  See also `Series.str.get_dummies <pandas.Series.str.get_dummies>`.  `get_dummies` also accepts a `DataFrame`. By default all categorical
variables (categorical in the statistical sense, those with ``object`` or
``categorical`` dtype) are encoded as dummy variables.  .. ipython:: python  df = pd.DataFrame({"A": ["a", "b", "a"], "B": ["c", "c", "b"], "C": [1, 2, 3]})  pd.get_dummies(df)  All non-object columns are included untouched in the output. You can control
the columns that are encoded with the ``columns`` keyword.  .. ipython:: python  pd.get_dummies(df, columns=["A"])  Notice that the ``B`` column is still included in the output, it just hasn't
been encoded. You can drop ``B`` before calling ``get_dummies`` if you don't
want to include it in the output.  As with the `Series` version, you can pass values for the ``prefix`` and
``prefix_sep``. By default the column name is used as the prefix, and ``_`` as
the prefix separator. You can specify ``prefix`` and ``prefix_sep`` in 3 ways:  * string: Use the same value for ``prefix`` or ``prefix_sep`` for each column  to be encoded.
* list: Must be the same length as the number of columns being encoded.
* dict: Mapping column name to prefix.  .. ipython:: python  simple = pd.get_dummies(df, prefix="new_prefix")  simple  from_list = pd.get_dummies(df, prefix=["from_A", "from_B"])  from_list  from_dict = pd.get_dummies(df, prefix={"B": "from_B", "A": "from_A"})  from_dict  Sometimes it will be useful to only keep k-1 levels of a categorical
variable to avoid collinearity when feeding the result to statistical models.
You can switch to this mode by turn on ``drop_first``.  .. ipython:: python  s = pd.Series(list("abcaa"))  pd.get_dummies(s)  pd.get_dummies(s, drop_first=True)  When a column contains only one level, it will be omitted in the result.  .. ipython:: python  df = pd.DataFrame({"A": list("aaa"), "B": list("ababc")})  pd.get_dummies(df)  pd.get_dummies(df, drop_first=True)  By default new columns will have ``np.uint8`` dtype.
To choose another dtype, use the ``dtype`` argument:  .. ipython:: python  df = pd.DataFrame({"A": list("abc"), "B": [1.1, 2.2, 3.3]})  pd.get_dummies(df, dtype=bool).dtypes  .. versionadded:: 1.5.0  To convert a "dummy" or "indicator" ``DataFrame``, into a categorical ``DataFrame``,
for example ``k`` columns of a ``DataFrame`` containing 1s and 0s can derive a
``DataFrame`` which has ``k`` distinct values using
`~pandas.from_dummies`:  .. ipython:: python  df = pd.DataFrame({"prefix_a": [0, 1, 0], "prefix_b": [1, 0, 1]})  df  pd.from_dummies(df, sep="_")  Dummy coded data only requires ``k - 1`` categories to be included, in this case
the ``k`` th category is the default category, implied by not being assigned any of
the other ``k - 1`` categories, can be passed via ``default_category``.  .. ipython:: python  df = pd.DataFrame({"prefix_a": [0, 1, 0]})  df  pd.from_dummies(df, sep="_", default_category="b")  .. _reshaping.factorize:

Factorizing values

To encode 1-d values as an enumerated type use `~pandas.factorize`:  .. ipython:: python  x = pd.Series(["A", "A", np.nan, "B", 3.14, np.inf])  x  labels, uniques = pd.factorize(x)  labels  uniques  Note that `~pandas.factorize` is similar to ``numpy.unique``, but differs in its
handling of NaN:  .. note::  The following ``numpy.unique`` will fail under Python 3 with a ``TypeError``  because of an ordering bug. See also  here_.  .. ipython:: python  ser = pd.Series(['A', 'A', np.nan, 'B', 3.14, np.inf])  pd.factorize(ser, sort=True)  np.unique(ser, return_inverse=True)[::-1]  .. note::  If you just want to handle one column as a categorical variable (like R's factor),  you can use  ``df["cat_col"] = pd.Categorical(df["col"])`` or  ``df["cat_col"] = df["col"].astype("category")``. For full docs on `~pandas.Categorical`,  see the `Categorical introduction <categorical>` and the  `API documentation <api.arrays.categorical>`.

Examples

In this section, we will review frequently asked questions and examples. The
column names and relevant column values are named to correspond with how this
DataFrame will be pivoted in the answers below.  .. ipython:: python  np.random.seed([3, 1415])  n = 20  cols = np.array(["key", "row", "item", "col"])  df = cols + pd.DataFrame(  (np.random.randint(5, size=(n, 4)) // [2, 1, 2, 1]).astype(str)  )  df.columns = cols  df = df.join(pd.DataFrame(np.random.rand(n, 2).round(2)).add_prefix("val"))  df  Pivoting with single aggregations
~~~  Suppose we wanted to pivot ``df`` such that the ``col`` values are columns,
``row`` values are the index, and the mean of ``val0`` are the values? In
particular, the resulting DataFrame should look like:  .. code-block:: text  col  col0  col1  col2  col3  col4  row  row0  0.77  0.605  NaN  0.860  0.65  row2  0.13  NaN  0.395  0.500  0.25  row3  NaN  0.310  NaN  0.545  NaN  row4  NaN  0.100  0.395  0.760  0.24  This solution uses `~pandas.pivot_table`. Also note that
``aggfunc='mean'`` is the default. It is included here to be explicit.  .. ipython:: python  df.pivot_table(values="val0", index="row", columns="col", aggfunc="mean")  Note that we can also replace the missing values by using the ``fill_value``
parameter.  .. ipython:: python  df.pivot_table(  values="val0",  index="row",  columns="col",  aggfunc="mean",  fill_value=0,  )  Also note that we can pass in other aggregation functions as well. For example,
we can also pass in ``sum``.  .. ipython:: python  df.pivot_table(  values="val0",  index="row",  columns="col",  aggfunc="sum",  fill_value=0,  )  Another aggregation we can do is calculate the frequency in which the columns
and rows occur together a.k.a. "cross tabulation". To do this, we can pass
``size`` to the ``aggfunc`` parameter.  .. ipython:: python  df.pivot_table(index="row", columns="col", fill_value=0, aggfunc="size")  Pivoting with multiple aggregations
~~~  We can also perform multiple aggregations. For example, to perform both a
``sum`` and ``mean``, we can pass in a list to the ``aggfunc`` argument.  .. ipython:: python  df.pivot_table(  values="val0",  index="row",  columns="col",  aggfunc=["mean", "sum"],  )  Note to aggregate over multiple value columns, we can pass in a list to the
``values`` parameter.  .. ipython:: python  df.pivot_table(  values=["val0", "val1"],  index="row",  columns="col",  aggfunc=["mean"],  )  Note to subdivide over multiple columns we can pass in a list to the
``columns`` parameter.  .. ipython:: python  df.pivot_table(  values=["val0"],  index="row",  columns=["item", "col"],  aggfunc=["mean"],  )  .. _reshaping.explode:

Exploding a list-like column

Sometimes the values in a column are list-like.  .. ipython:: python  keys = ["panda1", "panda2", "panda3"]  values = [["eats", "shoots"], ["shoots", "leaves"], ["eats", "leaves"]]  df = pd.DataFrame({"keys": keys, "values": values})  df  We can 'explode' the ``values`` column, transforming each list-like to a separate row, by using `~Series.explode`. This will replicate the index values from the original row:  .. ipython:: python  df["values"].explode()  You can also explode the column in the `DataFrame`.  .. ipython:: python  df.explode("values")  `Series.explode` will replace empty lists with ``np.nan`` and preserve scalar entries. The dtype of the resulting `Series` is always ``object``.  .. ipython:: python  s = pd.Series([[1, 2, 3], "foo", [], ["a", "b"]])  s  s.explode()  Here is a typical usecase. You have comma separated strings in a column and want to expand this.  .. ipython:: python  df = pd.DataFrame([{"var1": "a,b,c", "var2": 1}, {"var1": "d,e,f", "var2": 2}])  df  Creating a long form DataFrame is now straightforward using explode and chained operations  .. ipython:: python  df.assign(var1=df.var1.str.split(",")).explode("var1")

Load less data

Suppose our raw dataset on disk has many columns::  id_0  name_0  x_0  y_0  id_1  name_1  x_1  ...  name_8  x_8  y_8  id_9  name_9  x_9  y_9  timestamp  ...  2000-01-01 000  1015  Michael -0.399453  0.095427  994  Frank -0.176842  ...  Dan -0.315310  0.713892  1025  Victor -0.135779  0.346801  2000-01-01 000  969  Patricia  0.650773 -0.874275  1003  Laura  0.459153  ...  Ursula  0.913244 -0.630308  1047  Wendy -0.886285  0.035852  2000-01-01 000  1016  Victor -0.721465 -0.584710  1046  Michael  0.524994  ...  Ray -0.656593  0.692568  1064  Yvonne  0.070426  0.432047  2000-01-01 000  939  Alice -0.746004 -0.908008  996  Ingrid -0.414523  ...  Jerry -0.958994  0.608210  978  Wendy  0.855949 -0.648988  2000-01-01 000  1017  Dan  0.919451 -0.803504  1048  Jerry -0.569235  ...  Frank -0.577022 -0.409088  994  Bob -0.270132  0.335176  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  ...  2000-12-30 2300  999  Tim  0.162578  0.512817  973  Kevin -0.403352  ...  Tim -0.380415  0.008097  1041  Charlie  0.191477 -0.599519  2000-12-30 2300  970  Laura -0.433586 -0.600289  958  Oliver -0.966577  ...  Zelda  0.971274  0.402032  1038  Ursula  0.574016 -0.930992  2000-12-30 2300  1065  Edith  0.232211 -0.454540  971  Tim  0.158484  ...  Alice -0.222079 -0.919274  1022  Dan  0.031345 -0.657755  2000-12-30 2300  1019  Ingrid  0.322208 -0.615974  981  Hannah  0.607517  ...  Sarah -0.424440 -0.117274  990  George -0.375530  0.563312  2000-12-31 000  937  Ursula -0.906523  0.943178  1018  Alice -0.564513  ...  Jerry  0.236837  0.807650  985  Oliver  0.777642  0.783392  [525601 rows x 40 columns]  That can be generated by the following code snippet:  .. ipython:: python  import pandas as pd  import numpy as np  def make_timeseries(start="2000-01-01", end="2000-12-31", freq="1D", seed=None):  index = pd.date_range(start=start, end=end, freq=freq, name="timestamp")  n = len(index)  state = np.random.RandomState(seed)  columns = {  "name": state.choice(["Alice", "Bob", "Charlie"], size=n),  "id": state.poisson(1000, size=n),  "x": state.rand(n) * 2 - 1,  "y": state.rand(n) * 2 - 1,  }  df = pd.DataFrame(columns, index=index, columns=sorted(columns))  if df.index[-1] == end:  df = df.iloc[:-1]  return df  timeseries = [  make_timeseries(freq="1T", seed=i).rename(columns=lambda x: f"{x}_{i}")  for i in range(10)  ]  ts_wide = pd.concat(timeseries, axis=1)  ts_wide.to_parquet("timeseries_wide.parquet")  To load the columns we want, we have two options.
Option 1 loads in all the data and then filters to what we need.  .. ipython:: python  columns = ["id_0", "name_0", "x_0", "y_0"]  pd.read_parquet("timeseries_wide.parquet")[columns]  Option 2 only loads the columns we request.  .. ipython:: python  pd.read_parquet("timeseries_wide.parquet", columns=columns)  .. ipython:: python  import os  os.remove("timeseries_wide.parquet")  If we were to measure the memory usage of the two calls, we'd see that specifying
``columns`` uses about 1/10th the memory in this case.  With `pandas.read_csv`, you can specify ``usecols`` to limit the columns
read into memory. Not all file formats that can be read by pandas provide an option
to read a subset of columns.

Use efficient datatypes

The default pandas data types are not the most memory efficient. This is
especially true for text data columns with relatively few unique values (commonly
referred to as "low-cardinality" data). By using more efficient data types, you
can store larger datasets in memory.  .. ipython:: python  ts = make_timeseries(freq="30S", seed=0)  ts.to_parquet("timeseries.parquet")  ts = pd.read_parquet("timeseries.parquet")  ts  .. ipython:: python  os.remove("timeseries.parquet")  Now, let's inspect the data types and memory usage to see where we should focus our
attention.  .. ipython:: python  ts.dtypes  .. ipython:: python  ts.memory_usage(deep=True)  # memory usage in bytes  The ``name`` column is taking up much more memory than any other. It has just a
few unique values, so it's a good candidate for converting to a
`pandas.Categorical`. With a `pandas.Categorical`, we store each unique name once and use
space-efficient integers to know which specific name is used in each row.  .. ipython:: python  ts2 = ts.copy()  ts2["name"] = ts2["name"].astype("category")  ts2.memory_usage(deep=True)  We can go a bit further and downcast the numeric columns to their smallest types
using `pandas.to_numeric`.  .. ipython:: python  ts2["id"] = pd.to_numeric(ts2["id"], downcast="unsigned")  ts2[["x", "y"]] = ts2[["x", "y"]].apply(pd.to_numeric, downcast="float")  ts2.dtypes  .. ipython:: python  ts2.memory_usage(deep=True)  .. ipython:: python  reduction = ts2.memory_usage(deep=True).sum() / ts.memory_usage(deep=True).sum()  print(f"{reduction:0.2f}")  In all, we've reduced the in-memory footprint of this dataset to 1/5 of its
original size.  See `categorical` for more on `pandas.Categorical` and `basics.dtypes`
for an overview of all of pandas' dtypes.

Use chunking

Some workloads can be achieved with chunking: splitting a large problem like "convert this
directory of CSVs to parquet" into a bunch of small problems ("convert this individual CSV
file into a Parquet file. Now repeat that for each file in this directory."). As long as each chunk
fits in memory, you can work with datasets that are much larger than memory.  .. note::  Chunking works well when the operation you're performing requires zero or minimal  coordination between chunks. For more complicated workflows, you're better off  `using another library <scale.other_libraries>`.  Suppose we have an even larger "logical dataset" on disk that's a directory of parquet
files. Each file in the directory represents a different year of the entire dataset.  .. ipython:: python  import pathlib  N = 12  starts = [f"20{i:>02d}-01-01" for i in range(N)]  ends = [f"20{i:>02d}-12-13" for i in range(N)]  pathlib.Path("data/timeseries").mkdir(exist_ok=True)  for i, (start, end) in enumerate(zip(starts, ends)):  ts = make_timeseries(start=start, end=end, freq="1T", seed=i)  ts.to_parquet(f"data/timeseries/ts-{i:0>2d}.parquet")  ::  data  └── timeseries  ├── ts-00.parquet  ├── ts-01.parquet  ├── ts-02.parquet  ├── ts-03.parquet  ├── ts-04.parquet  ├── ts-05.parquet  ├── ts-06.parquet  ├── ts-07.parquet  ├── ts-08.parquet  ├── ts-09.parquet  ├── ts-10.parquet  └── ts-11.parquet  Now we'll implement an out-of-core `pandas.Series.value_counts`. The peak memory usage of this
workflow is the single largest chunk, plus a small series storing the unique value
counts up to this point. As long as each individual file fits in memory, this will
work for arbitrary-sized datasets.  .. ipython:: python  %%time  files = pathlib.Path("data/timeseries/").glob("ts*.parquet")  counts = pd.Series(dtype=int)  for path in files:  df = pd.read_parquet(path)  counts = counts.add(df["name"].value_counts(), fill_value=0)  counts.astype(int)  Some readers, like `pandas.read_csv`, offer parameters to control the
``chunksize`` when reading a single file.  Manually chunking is an OK option for workflows that don't
require too sophisticated of operations. Some operations, like `pandas.DataFrame.groupby`, are
much harder to do chunkwise. In these cases, you may be better switching to a
different library that implements these out-of-core algorithms for you.  .. _scale.other_libraries:

pandas is just one library offering a DataFrame API. Because of its popularity,
pandas' API has become something of a standard that other libraries implement.
The pandas documentation maintains a list of libraries implementing a DataFrame API
in `our ecosystem page <ecosystem.out-of-core>`.  For example, `Dask`_, a parallel computing library, has `dask.dataframe`_, a
pandas-like API for working with larger than memory datasets in parallel. Dask
can use multiple threads or processes on a single machine, or a cluster of
machines to process data in parallel.  We'll import ``dask.dataframe`` and notice that the API feels similar to pandas.
We can use Dask's ``read_parquet`` function, but provide a globstring of files to read in.  .. ipython:: python  import dask.dataframe as dd  ddf = dd.read_parquet("data/timeseries/ts*.parquet", engine="pyarrow")  ddf  Inspecting the ``ddf`` object, we see a few things  * There are familiar attributes like ``.columns`` and ``.dtypes``
* There are familiar methods like ``.groupby``, ``.sum``, etc.
* There are new attributes like ``.npartitions`` and ``.divisions``  The partitions and divisions are how Dask parallelizes computation. A **Dask**
DataFrame is made up of many pandas `pandas.DataFrame`. A single method call on a
Dask DataFrame ends up making many pandas method calls, and Dask knows how to
coordinate everything to get the result.  .. ipython:: python  ddf.columns  ddf.dtypes  ddf.npartitions  One major difference: the ``dask.dataframe`` API is *lazy*. If you look at the
repr above, you'll notice that the values aren't actually printed out; just the
column names and dtypes. That's because Dask hasn't actually read the data yet.
Rather than executing immediately, doing operations build up a **task graph**.  .. ipython:: python  ddf  ddf["name"]  ddf["name"].value_counts()  Each of these calls is instant because the result isn't being computed yet.
We're just building up a list of computation to do when someone needs the
result. Dask knows that the return type of a `pandas.Series.value_counts`
is a pandas `pandas.Series` with a certain dtype and a certain name. So the Dask version
returns a Dask Series with the same dtype and the same name.  To get the actual result you can call ``.compute()``.  .. ipython:: python  %time ddf["name"].value_counts().compute()  At that point, you get back the same thing you'd get with pandas, in this case
a concrete pandas `pandas.Series` with the count of each ``name``.  Calling ``.compute`` causes the full task graph to be executed. This includes
reading the data, selecting the columns, and doing the ``value_counts``. The
execution is done *in parallel* where possible, and Dask tries to keep the
overall memory footprint small. You can work with datasets that are much larger
than memory, as long as each partition (a regular pandas `pandas.DataFrame`) fits in memory.  By default, ``dask.dataframe`` operations use a threadpool to do operations in
parallel. We can also connect to a cluster to distribute the work on many
machines. In this case we'll connect to a local "cluster" made up of several
processes on this single machine.  .. code-block:: python  >>> from dask.distributed import Client, LocalCluster  >>> cluster = LocalCluster()  >>> client = Client(cluster)  >>> client  <Client: 'tcp://127.0.0.1:53349' processes=4 threads=8, memory=17.18 GB>  Once this ``client`` is created, all of Dask's computation will take place on
the cluster (which is just processes in this case).  Dask implements the most used parts of the pandas API. For example, we can do
a familiar groupby aggregation.  .. ipython:: python  %time ddf.groupby("name")[["x", "y"]].mean().compute().head()  The grouping and aggregation is done out-of-core and in parallel.  When Dask knows the ``divisions`` of a dataset, certain optimizations are
possible. When reading parquet datasets written by dask, the divisions will be
known automatically. In this case, since we created the parquet files manually,

we need to supply the divisions manually.  .. ipython:: python  N = 12  starts = [f"20{i:>02d}-01-01" for i in range(N)]  ends = [f"20{i:>02d}-12-13" for i in range(N)]  divisions = tuple(pd.to_datetime(starts)) + (pd.Timestamp(ends[-1]),)  ddf.divisions = divisions  ddf  Now we can do things like fast random access with ``.loc``.  .. ipython:: python  ddf.loc["2002-01-01 12:01":"2002-01-01 12:05"].compute()  Dask knows to just look in the 3rd partition for selecting values in 2002. It
doesn't need to look at any other data.  Many workflows involve a large amount of data and processing it in a way that
reduces the size to something that fits in memory. In this case, we'll resample
to daily frequency and take the mean. Once we've taken the mean, we know the
results will fit in memory, so we can safely call ``compute`` without running
out of memory. At that point it's just a regular pandas object.  .. ipython:: python  @savefig dask_resample.png  ddf[["x", "y"]].resample("1D").mean().cumsum().compute().plot()  .. ipython:: python  import shutil  shutil.rmtree("data/timeseries")  These Dask examples have all be done using multiple processes on a single
machine. Dask can be `deployed on a cluster
<https://docs.dask.org/en/latest/setup.html>`_ to scale up to even larger
datasets.  You see more dask examples at https://examples.dask.org.  .. _Dask: https://dask.org
.. _dask.dataframe: https://docs.dask.org/en/latest/dataframe.html

SparseArray

`arrays.SparseArray` is a `~pandas.api.extensions.ExtensionArray`
for storing an array of sparse values (see `basics.dtypes` for more
on extension arrays). It is a 1-dimensional ndarray-like object storing
only values distinct from the ``fill_value``:  .. ipython:: python  arr = np.random.randn(10)  arr[2:5] = np.nan  arr[7:8] = np.nan  sparr = pd.arrays.SparseArray(arr)  sparr  A sparse array can be converted to a regular (dense) ndarray with `numpy.asarray`  .. ipython:: python  np.asarray(sparr)  .. _sparse.dtype:

SparseDtype

The `SparseArray.dtype` property stores two pieces of information  1. The dtype of the non-sparse values
2. The scalar fill value  .. ipython:: python  sparr.dtype  A `SparseDtype` may be constructed by passing only a dtype  .. ipython:: python  pd.SparseDtype(np.dtype('datetime64[ns]'))  in which case a default fill value will be used (for NumPy dtypes this is often the
"missing" value for that dtype). To override this default an explicit fill value may be
passed instead  .. ipython:: python  pd.SparseDtype(np.dtype('datetime64[ns]'),  fill_value=pd.Timestamp('2017-01-01'))  Finally, the string alias ``'Sparse[dtype]'`` may be used to specify a sparse dtype
in many places  .. ipython:: python  pd.array([1, 0, 0, 2], dtype='Sparse[int]')  .. _sparse.accessor:

Sparse accessor

pandas provides a ``.sparse`` accessor, similar to ``.str`` for string data, ``.cat``
for categorical data, and ``.dt`` for datetime-like data. This namespace provides
attributes and methods that are specific to sparse data.  .. ipython:: python  s = pd.Series([0, 0, 1, 2], dtype="Sparse[int]")  s.sparse.density  s.sparse.fill_value  This accessor is available only on data with ``SparseDtype``, and on the `Series`
class itself for creating a Series with sparse data from a scipy COO matrix with.  A ``.sparse`` accessor has been added for `DataFrame` as well.
See `api.frame.sparse` for more.  .. _sparse.calculation:

Sparse calculation

You can apply NumPy ufuncs
to `arrays.SparseArray` and get a `arrays.SparseArray` as a result.  .. ipython:: python  arr = pd.arrays.SparseArray([1., np.nan, np.nan, -2., np.nan])  np.abs(arr)  The *ufunc* is also applied to ``fill_value``. This is needed to get
the correct dense result.  .. ipython:: python  arr = pd.arrays.SparseArray([1., -1, -1, -2., -1], fill_value=-1)  np.abs(arr)  np.abs(arr).to_dense()  .. _sparse.migration:

Migrating

.. note::  ``SparseSeries`` and ``SparseDataFrame`` were removed in pandas 1.0.0. This migration  guide is present to aid in migrating from previous versions.  In older versions of pandas, the ``SparseSeries`` and ``SparseDataFrame`` classes (documented below)
were the preferred way to work with sparse data. With the advent of extension arrays, these subclasses
are no longer needed. Their purpose is better served by using a regular Series or DataFrame with
sparse values instead.  .. note::  There's no performance or memory penalty to using a Series or DataFrame with sparse values,  rather than a SparseSeries or SparseDataFrame.  This section provides some guidance on migrating your code to the new style. As a reminder,
you can use the Python warnings module to control warnings. But we recommend modifying
your code, rather than ignoring the warning.  **Construction**  From an array-like, use the regular `Series` or
`DataFrame` constructors with `arrays.SparseArray` values.  .. code-block:: python  # Previous way  >>> pd.SparseDataFrame({"A": [0, 1]})  .. ipython:: python  # New way  pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])})  From a SciPy sparse matrix, use `DataFrame.sparse.from_spmatrix`,  .. code-block:: python  # Previous way  >>> from scipy import sparse  >>> mat = sparse.eye(3)  >>> df = pd.SparseDataFrame(mat, columns=['A', 'B', 'C'])  .. ipython:: python  # New way  from scipy import sparse  mat = sparse.eye(3)  df = pd.DataFrame.sparse.from_spmatrix(mat, columns=['A', 'B', 'C'])  df.dtypes  **Conversion**  From sparse to dense, use the ``.sparse`` accessors  .. ipython:: python  df.sparse.to_dense()  df.sparse.to_coo()  From dense to sparse, use `DataFrame.astype` with a `SparseDtype`.  .. ipython:: python  dense = pd.DataFrame({"A": [1, 0, 0, 1]})  dtype = pd.SparseDtype(int, fill_value=0)  dense.astype(dtype)  **Sparse Properties**  Sparse-specific properties, like ``density``, are available on the ``.sparse`` accessor.  .. ipython:: python  df.sparse.density  **General differences**  In a ``SparseDataFrame``, *all* columns were sparse. A `DataFrame` can have a mixture of
sparse and dense columns. As a consequence, assigning new columns to a `DataFrame` with sparse
values will not automatically convert the input to be sparse.  .. code-block:: python  # Previous Way  >>> df = pd.SparseDataFrame({"A": [0, 1]})  >>> df['B'] = [0, 0]  # implicitly becomes Sparse  >>> df['B'].dtype  Sparse[int64, nan]  Instead, you'll need to ensure that the values being assigned are sparse  .. ipython:: python  df = pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])})  df['B'] = [0, 0]  # remains dense  df['B'].dtype  df['B'] = pd.arrays.SparseArray([0, 0])  df['B'].dtype  The ``SparseDataFrame.default_kind`` and ``SparseDataFrame.default_fill_value`` attributes
have no replacement.  .. _sparse.scipysparse:

Interaction with *scipy.sparse*

Use `DataFrame.sparse.from_spmatrix` to create a `DataFrame` with sparse values from a sparse matrix.  .. ipython:: python  from scipy.sparse import csr_matrix  arr = np.random.random(size=(1000, 5))  arr[arr < .9] = 0  sp_arr = csr_matrix(arr)  sp_arr  sdf = pd.DataFrame.sparse.from_spmatrix(sp_arr)  sdf.head()  sdf.dtypes  All sparse formats are supported, but matrices that are not in `COOrdinate <scipy.sparse>` format will be converted, copying data as needed.
To convert back to sparse SciPy matrix in COO format, you can use the `DataFrame.sparse.to_coo` method:  .. ipython:: python  sdf.sparse.to_coo()  `Series.sparse.to_coo` is implemented for transforming a `Series` with sparse values indexed by a `MultiIndex` to a `scipy.sparse.coo_matrix`.  The method requires a `MultiIndex` with two or more levels.  .. ipython:: python  s = pd.Series([3.0, np.nan, 1.0, 3.0, np.nan, np.nan])  s.index = pd.MultiIndex.from_tuples(  [  (1, 2, "a", 0),  (1, 2, "a", 1),  (1, 1, "b", 0),  (1, 1, "b", 1),  (2, 1, "b", 0),  (2, 1, "b", 1),  ],  names=["A", "B", "C", "D"],  )  ss = s.astype('Sparse')  ss  In the example below, we transform the `Series` to a sparse representation of a 2-d array by specifying that the first and second ``MultiIndex`` levels define labels for the rows and the third and fourth levels define labels for the columns. We also specify that the column and row labels should be sorted in the final sparse representation.  .. ipython:: python  A, rows, columns = ss.sparse.to_coo(  row_levels=["A", "B"], column_levels=["C", "D"], sort_labels=True  )  A  A.todense()  rows  columns  Specifying different row and column labels (and not sorting them) yields a different sparse matrix:  .. ipython:: python  A, rows, columns = ss.sparse.to_coo(  row_levels=["A", "B", "C"], column_levels=["D"], sort_labels=False  )  A  A.todense()  rows  columns  A convenience method `Series.sparse.from_coo` is implemented for creating a `Series` with sparse values from a ``scipy.sparse.coo_matrix``.  .. ipython:: python  from scipy import sparse  A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])), shape=(3, 4))  A  A.todense()  The default behaviour (with ``dense_index=False``) simply returns a `Series` containing
only the non-null entries.  .. ipython:: python  ss = pd.Series.sparse.from_coo(A)  ss  Specifying ``dense_index=True`` will result in an index that is the Cartesian product of the
row and columns coordinates of the matrix. Note that this will consume a significant amount of memory
(relative to ``dense_index=False``) if the sparse matrix is large (and sparse) enough.  .. ipython:: python  ss_dense = pd.Series.sparse.from_coo(A, dense_index=True)  ss_dense

Text data types

.. versionadded:: 1.0.0  There are two ways to store text data in pandas:  1. ``object`` -dtype NumPy array.
2. `StringDtype` extension type.  We recommend using `StringDtype` to store text data.  Prior to pandas 1.0, ``object`` dtype was the only option. This was unfortunate
for many reasons:  1. You can accidentally store a *mixture* of strings and non-strings in an  ``object`` dtype array. It's better to have a dedicated dtype.
2. ``object`` dtype breaks dtype-specific operations like `DataFrame.select_dtypes`.  There isn't a clear way to select *just* text while excluding non-text  but still object-dtype columns.
3. When reading code, the contents of an ``object`` dtype array is less clear  than ``'string'``.  Currently, the performance of ``object`` dtype arrays of strings and
`arrays.StringArray` are about the same. We expect future enhancements
to significantly increase the performance and lower the memory overhead of
`~arrays.StringArray`.  .. warning::  ``StringArray`` is currently considered experimental. The implementation  and parts of the API may change without warning.  For backwards-compatibility, ``object`` dtype remains the default type we
infer a list of strings to  .. ipython:: python  pd.Series(["a", "b", "c"])  To explicitly request ``string`` dtype, specify the ``dtype``  .. ipython:: python  pd.Series(["a", "b", "c"], dtype="string")  pd.Series(["a", "b", "c"], dtype=pd.StringDtype())  Or ``astype`` after the ``Series`` or ``DataFrame`` is created  .. ipython:: python  s = pd.Series(["a", "b", "c"])  s  s.astype("string")  .. versionchanged:: 1.1.0  You can also use `StringDtype`/``"string"`` as the dtype on non-string data and
it will be converted to ``string`` dtype:  .. ipython:: python  s = pd.Series(["a", 2, np.nan], dtype="string")  s  type(s[1])  or convert from existing pandas data:  .. ipython:: python  s1 = pd.Series([1, 2, np.nan], dtype="Int64")  s1  s2 = s1.astype("string")  s2  type(s2[0])  .. _text.differences:  Behavior differences
^^^  These are places where the behavior of ``StringDtype`` objects differ from
``object`` dtype  l. For ``StringDtype``, `string accessor methods<api.series.str>`  that return **numeric** output will always return a nullable integer dtype,  rather than either int or float dtype, depending on the presence of NA values.  Methods returning **boolean** output will return a nullable boolean dtype.  .. ipython:: python  s = pd.Series(["a", None, "b"], dtype="string")  s  s.str.count("a")  s.dropna().str.count("a")  Both outputs are ``Int64`` dtype. Compare that with object-dtype  .. ipython:: python  s2 = pd.Series(["a", None, "b"], dtype="object")  s2.str.count("a")  s2.dropna().str.count("a")  When NA values are present, the output dtype is float64. Similarly for  methods returning boolean values.  .. ipython:: python  s.str.isdigit()  s.str.match("a")  2. Some string methods, like `Series.str.decode` are not available  on ``StringArray`` because ``StringArray`` only holds strings, not  bytes.
3. In comparison operations, `arrays.StringArray` and ``Series`` backed  by a ``StringArray`` will return an object with `BooleanDtype`,  rather than a ``bool`` dtype object. Missing values in a ``StringArray``  will propagate in comparison operations, rather than always comparing  unequal like `numpy.nan`.  Everything else that follows in the rest of this document applies equally to
``string`` and ``object`` dtype.  .. _text.string_methods:

String methods

Series and Index are equipped with a set of string processing methods
that make it easy to operate on each element of the array. Perhaps most
importantly, these methods exclude missing/NA values automatically. These are
accessed via the ``str`` attribute and generally have names matching
the equivalent (scalar) built-in string methods:  .. ipython:: python  s = pd.Series(  ["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"], dtype="string"  )  s.str.lower()  s.str.upper()  s.str.len()  .. ipython:: python  idx = pd.Index([" jack", "jill ", " jesse ", "frank"])  idx.str.strip()  idx.str.lstrip()  idx.str.rstrip()  The string methods on Index are especially useful for cleaning up or
transforming DataFrame columns. For instance, you may have columns with
leading or trailing whitespace:  .. ipython:: python  df = pd.DataFrame(  np.random.randn(3, 2), columns=[" Column A ", " Column B "], index=range(3)  )  df  Since ``df.columns`` is an Index object, we can use the ``.str`` accessor  .. ipython:: python  df.columns.str.strip()  df.columns.str.lower()  These string methods can then be used to clean up the columns as needed.
Here we are removing leading and trailing whitespaces, lower casing all names,
and replacing any remaining whitespaces with underscores:  .. ipython:: python  df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")  df  .. note::  If you have a ``Series`` where lots of elements are repeated  (i.e. the number of unique elements in the ``Series`` is a lot smaller than the length of the  ``Series``), it can be faster to convert the original ``Series`` to one of type  ``category`` and then use ``.str.<method>`` or ``.dt.<property>`` on that.  The performance difference comes from the fact that, for ``Series`` of type ``category``, the  string operations are done on the ``.categories`` and not on each element of the  ``Series``.  Please note that a ``Series`` of type ``category`` with string ``.categories`` has  some limitations in comparison to ``Series`` of type string (e.g. you can't add strings to  each other: ``s + " " + s`` won't work if ``s`` is a ``Series`` of type ``category``). Also,  ``.str`` methods which operate on elements of type ``list`` are not available on such a  ``Series``.  .. _text.warn_types:  .. warning::  Before v.0.25.0, the ``.str``-accessor did only the most rudimentary type checks. Starting with  v.0.25.0, the type of the Series is inferred and the allowed types (i.e. strings) are enforced more rigorously.  Generally speaking, the ``.str`` accessor is intended to work only on strings. With very few  exceptions, other uses are not supported, and may be disabled at a later point.  .. _text.split:

Splitting and replacing strings

Methods like ``split`` return a Series of lists:  .. ipython:: python  s2 = pd.Series(["a_b_c", "c_d_e", np.nan, "f_g_h"], dtype="string")  s2.str.split("_")  Elements in the split lists can be accessed using ``get`` or ``[]`` notation:  .. ipython:: python  s2.str.split("_").str.get(1)  s2.str.split("_").str[1]  It is easy to expand this to return a DataFrame using ``expand``.  .. ipython:: python  s2.str.split("_", expand=True)  When original ``Series`` has `StringDtype`, the output columns will all
be `StringDtype` as well.  It is also possible to limit the number of splits:  .. ipython:: python  s2.str.split("_", expand=True, n=1)  ``rsplit`` is similar to ``split`` except it works in the reverse direction,
i.e., from the end of the string to the beginning of the string:  .. ipython:: python  s2.str.rsplit("_", expand=True, n=1)  ``replace`` optionally uses `regular expressions
<https://docs.python.org/3/library/re.html>`__:  .. ipython:: python  s3 = pd.Series(  ["A", "B", "C", "Aaba", "Baca", "", np.nan, "CABA", "dog", "cat"],  dtype="string",  )  s3  s3.str.replace("^.a|dog", "XX-XX ", case=False, regex=True)  .. versionchanged:: 2.0  Single character pattern with ``regex=True`` will also be treated as regular expressions:  .. ipython:: python  s4 = pd.Series(["a.b", ".", "b", np.nan, ""], dtype="string")  s4  s4.str.replace(".", "a", regex=True)  If you want literal replacement of a string (equivalent to `str.replace`), you
can set the optional ``regex`` parameter to ``False``, rather than escaping each
character. In this case both ``pat`` and ``repl`` must be strings:  .. ipython:: python  dollars = pd.Series(["12", "-$10", "$10,000"], dtype="string")  # These lines are equivalent  dollars.str.replace(r"-\$", "-", regex=True)  dollars.str.replace("-$", "-", regex=False)  The ``replace`` method can also take a callable as replacement. It is called
on every ``pat`` using `re.sub`. The callable should expect one
positional argument (a regex object) and return a string.  .. ipython:: python  # Reverse every lowercase alphabetic word  pat = r"[a-z]+"  def repl(m):  return m.group(0)[::-1]  pd.Series(["foo 123", "bar baz", np.nan], dtype="string").str.replace(  pat, repl, regex=True  )  # Using regex groups  pat = r"(?P<one>\w+) (?P<two>\w+) (?P<three>\w+)"  def repl(m):  return m.group("two").swapcase()  pd.Series(["Foo Bar Baz", np.nan], dtype="string").str.replace(  pat, repl, regex=True  )  The ``replace`` method also accepts a compiled regular expression object
from `re.compile` as a pattern. All flags should be included in the
compiled regular expression object.  .. ipython:: python  import re  regex_pat = re.compile(r"^.a|dog", flags=re.IGNORECASE)  s3.str.replace(regex_pat, "XX-XX ", regex=True)  Including a ``flags`` argument when calling ``replace`` with a compiled
regular expression object will raise a ``ValueError``.  .. ipython::  @verbatim

In [1]: s3.str.replace(regex_pat, 'XX-XX ', flags=re.IGNORECASE)
  ValueError: case and flags cannot be set when pat is a compiled regex  ``removeprefix`` and ``removesuffix`` have the same effect as ``str.removeprefix`` and ``str.removesuffix`` added in Python 3.9
<https://docs.python.org/3/library/stdtypes.html#str.removeprefix>`__:  .. versionadded:: 1.4.0  .. ipython:: python  s = pd.Series(["str_foo", "str_bar", "no_prefix"])  s.str.removeprefix("str_")  s = pd.Series(["foo_str", "bar_str", "no_suffix"])  s.str.removesuffix("_str")  .. _text.concatenate:

Concatenation

There are several ways to concatenate a ``Series`` or ``Index``, either with itself or others, all based on `~Series.str.cat`,
resp. ``Index.str.cat``.  Concatenating a single Series into a string
^^^  The content of a ``Series`` (or ``Index``) can be concatenated:  .. ipython:: python  s = pd.Series(["a", "b", "c", "d"], dtype="string")  s.str.cat(sep=",")  If not specified, the keyword ``sep`` for the separator defaults to the empty string, ``sep=''``:  .. ipython:: python  s.str.cat()  By default, missing values are ignored. Using ``na_rep``, they can be given a representation:  .. ipython:: python  t = pd.Series(["a", "b", np.nan, "d"], dtype="string")  t.str.cat(sep=",")  t.str.cat(sep=",", na_rep="-")  Concatenating a Series and something list-like into a Series
^^^  The first argument to `~Series.str.cat` can be a list-like object, provided that it matches the length of the calling ``Series`` (or ``Index``).  .. ipython:: python  s.str.cat(["A", "B", "C", "D"])  Missing values on either side will result in missing values in the result as well, *unless* ``na_rep`` is specified:  .. ipython:: python  s.str.cat(t)  s.str.cat(t, na_rep="-")  Concatenating a Series and something array-like into a Series
^^^  The parameter ``others`` can also be two-dimensional. In this case, the number or rows must match the lengths of the calling ``Series`` (or ``Index``).  .. ipython:: python  d = pd.concat([t, s], axis=1)  s  d  s.str.cat(d, na_rep="-")  Concatenating a Series and an indexed object into a Series, with alignment
^^^  For concatenation with a ``Series`` or ``DataFrame``, it is possible to align the indexes before concatenation by setting
the ``join``-keyword.  .. ipython:: python  u = pd.Series(["b", "d", "a", "c"], index=[1, 3, 0, 2], dtype="string")  s  u  s.str.cat(u)  s.str.cat(u, join="left")  .. warning::  If the ``join`` keyword is not passed, the method `~Series.str.cat` will currently fall back to the behavior before version 0.23.0 (i.e. no alignment),  but a ``FutureWarning`` will be raised if any of the involved indexes differ, since this default will change to ``join='left'`` in a future version.  The usual options are available for ``join`` (one of ``'left', 'outer', 'inner', 'right'``).
In particular, alignment also means that the different lengths do not need to coincide anymore.  .. ipython:: python  v = pd.Series(["z", "a", "b", "d", "e"], index=[-1, 0, 1, 3, 4], dtype="string")  s  v  s.str.cat(v, join="left", na_rep="-")  s.str.cat(v, join="outer", na_rep="-")  The same alignment can be used when ``others`` is a ``DataFrame``:  .. ipython:: python  f = d.loc[[3, 2, 1, 0], :]  s  f  s.str.cat(f, join="left", na_rep="-")  Concatenating a Series and many objects into a Series
^^^  Several array-like items (specifically: ``Series``, ``Index``, and 1-dimensional variants of ``np.ndarray``)
can be combined in a list-like container (including iterators, ``dict``-views, etc.).  .. ipython:: python  s  u  s.str.cat([u, u.to_numpy()], join="left")  All elements without an index (e.g. ``np.ndarray``) within the passed list-like must match in length to the calling ``Series`` (or ``Index``),
but ``Series`` and ``Index`` may have arbitrary length (as long as alignment is not disabled with ``join=None``):  .. ipython:: python  v  s.str.cat([v, u, u.to_numpy()], join="outer", na_rep="-")  If using ``join='right'`` on a list-like of ``others`` that contains different indexes,
the union of these indexes will be used as the basis for the final concatenation:  .. ipython:: python  u.loc[[3]]  v.loc[[-1, 0]]  s.str.cat([u.loc[[3]], v.loc[[-1, 0]]], join="right", na_rep="-")

Indexing with ``.str``

.. _text.indexing:  You can use ``[]`` notation to directly index by position locations. If you index past the end
of the string, the result will be a ``NaN``.  .. ipython:: python  s = pd.Series(  ["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"], dtype="string"  )  s.str[0]  s.str[1]

.. _text.extract:  Extract first match in each subject (extract)
^^^  .. warning::  Before version 0.23, argument ``expand`` of the ``extract`` method defaulted to  ``False``. When ``expand=False``, ``expand`` returns a ``Series``, ``Index``, or  ``DataFrame``, depending on the subject and regular expression  pattern. When ``expand=True``, it always returns a ``DataFrame``,  which is more consistent and less confusing from the perspective of a user.  ``expand=True`` has been the default since version 0.23.0.  The ``extract`` method accepts a `regular expression
<https://docs.python.org/3/library/re.html>`__ with at least one
capture group.  Extracting a regular expression with more than one group returns a
DataFrame with one column per group.  .. ipython:: python  pd.Series(  ["a1", "b2", "c3"],  dtype="string",  ).str.extract(r"([ab])(\d)", expand=False)  Elements that do not match return a row filled with ``NaN``. Thus, a
Series of messy strings can be "converted" into a like-indexed Series
or DataFrame of cleaned-up or more useful strings, without
necessitating ``get()`` to access tuples or ``re.match`` objects. The
dtype of the result is always object, even if no match is found and
the result only contains ``NaN``.  Named groups like  .. ipython:: python  pd.Series(["a1", "b2", "c3"], dtype="string").str.extract(  r"(?P<letter>[ab])(?P<digit>\d)", expand=False  )  and optional groups like  .. ipython:: python  pd.Series(  ["a1", "b2", "3"],  dtype="string",  ).str.extract(r"([ab])?(\d)", expand=False)  can also be used. Note that any capture group names in the regular
expression will be used for column names; otherwise capture group
numbers will be used.  Extracting a regular expression with one group returns a ``DataFrame``
with one column if ``expand=True``.  .. ipython:: python  pd.Series(["a1", "b2", "c3"], dtype="string").str.extract(r"[ab](\d)", expand=True)  It returns a Series if ``expand=False``.  .. ipython:: python  pd.Series(["a1", "b2", "c3"], dtype="string").str.extract(r"[ab](\d)", expand=False)  Calling on an ``Index`` with a regex with exactly one capture group
returns a ``DataFrame`` with one column if ``expand=True``.  .. ipython:: python  s = pd.Series(["a1", "b2", "c3"], ["A11", "B22", "C33"], dtype="string")  s  s.index.str.extract("(?P<letter>[a-zA-Z])", expand=True)  It returns an ``Index`` if ``expand=False``.  .. ipython:: python  s.index.str.extract("(?P<letter>[a-zA-Z])", expand=False)  Calling on an ``Index`` with a regex with more than one capture group
returns a ``DataFrame`` if ``expand=True``.  .. ipython:: python  s.index.str.extract("(?P<letter>[a-zA-Z])([0-9]+)", expand=True)  It raises ``ValueError`` if ``expand=False``.  .. code-block:: python  >>> s.index.str.extract("(?P<letter>[a-zA-Z])([0-9]+)", expand=False)  ValueError: only one regex group is supported with Index  The table below summarizes the behavior of ``extract(expand=False)``
(input subject in first column, number of groups in regex in
first row)  +---+---+---+
|  | 1 group | >1 group  |
+---+---+---+
| Index  | Index  | ValueError |
| Series | Series  | DataFrame  |
+---+---+---+  Extract all matches in each subject (extractall)
^^^  .. _text.extractall:  Unlike ``extract`` (which returns only the first match),  .. ipython:: python  s = pd.Series(["a1a2", "b1", "c1"], index=["A", "B", "C"], dtype="string")  s  two_groups = "(?P<letter>[a-z])(?P<digit>[0-9])"  s.str.extract(two_groups, expand=True)  the ``extractall`` method returns every match. The result of
``extractall`` is always a ``DataFrame`` with a ``MultiIndex`` on its
rows. The last level of the ``MultiIndex`` is named ``match`` and
indicates the order in the subject.  .. ipython:: python  s.str.extractall(two_groups)  When each subject string in the Series has exactly one match,  .. ipython:: python  s = pd.Series(["a3", "b3", "c2"], dtype="string")  s  then ``extractall(pat).xs(0, level='match')`` gives the same result as

``extract(pat)``.  .. ipython:: python  extract_result = s.str.extract(two_groups, expand=True)  extract_result  extractall_result = s.str.extractall(two_groups)  extractall_result  extractall_result.xs(0, level="match")  ``Index`` also supports ``.str.extractall``. It returns a ``DataFrame`` which has the
same result as a ``Series.str.extractall`` with a default index (starts from 0).  .. ipython:: python  pd.Index(["a1a2", "b1", "c1"]).str.extractall(two_groups)  pd.Series(["a1a2", "b1", "c1"], dtype="string").str.extractall(two_groups)

Testing for strings that match or contain a pattern

You can check whether elements contain a pattern:  .. ipython:: python  pattern = r"[0-9][a-z]"  pd.Series(  ["1", "2", "3a", "3b", "03c", "4dx"],  dtype="string",  ).str.contains(pattern)  Or whether elements match a pattern:  .. ipython:: python  pd.Series(  ["1", "2", "3a", "3b", "03c", "4dx"],  dtype="string",  ).str.match(pattern)  .. versionadded:: 1.1.0  .. ipython:: python  pd.Series(  ["1", "2", "3a", "3b", "03c", "4dx"],  dtype="string",  ).str.fullmatch(pattern)  .. note::  The distinction between ``match``, ``fullmatch``, and ``contains`` is strictness:  ``fullmatch`` tests whether the entire string matches the regular expression;  ``match`` tests whether there is a match of the regular expression that begins  at the first character of the string; and ``contains`` tests whether there is  a match of the regular expression at any position within the string.  The corresponding functions in the ``re`` package for these three match modes are  re.fullmatch,  re.match, and  re.search,  respectively.  Methods like ``match``, ``fullmatch``, ``contains``, ``startswith``, and
``endswith`` take an extra ``na`` argument so missing values can be considered
True or False:  .. ipython:: python  s4 = pd.Series(  ["A", "B", "C", "Aaba", "Baca", np.nan, "CABA", "dog", "cat"], dtype="string"  )  s4.str.contains("A", na=False)  .. _text.indicator:

Creating indicator variables

You can extract dummy variables from string columns.
For example if they are separated by a ``'|'``:  .. ipython:: python  s = pd.Series(["a", "a|b", np.nan, "a|c"], dtype="string")  s.str.get_dummies(sep="|")  String ``Index`` also supports ``get_dummies`` which returns a ``MultiIndex``.  .. ipython:: python  idx = pd.Index(["a", "a|b", np.nan, "a|c"])  idx.str.get_dummies(sep="|")  See also `~pandas.get_dummies`.

Method summary

.. _text.summary:  .. csv-table::  "Method", "Description"  20, 80  ;  `~Series.str.cat`;Concatenate strings  `~Series.str.split`;Split strings on delimiter  `~Series.str.rsplit`;Split strings on delimiter working from the end of the string  `~Series.str.get`;Index into each element (retrieve i-th element)  `~Series.str.join`;Join strings in each element of the Series with passed separator  `~Series.str.get_dummies`;Split strings on the delimiter returning DataFrame of dummy variables  `~Series.str.contains`;Return boolean array if each string contains pattern/regex  `~Series.str.replace`;Replace occurrences of pattern/regex/string with some other string or the return value of a callable given the occurrence  `~Series.str.removeprefix`;Remove prefix from string, i.e. only remove if string starts with prefix.  `~Series.str.removesuffix`;Remove suffix from string, i.e. only remove if string ends with suffix.  `~Series.str.repeat`;Duplicate values (``s.str.repeat(3)`` equivalent to ``x * 3``)  `~Series.str.pad`;"Add whitespace to left, right, or both sides of strings"  `~Series.str.center`;Equivalent to ``str.center``  `~Series.str.ljust`;Equivalent to ``str.ljust``  `~Series.str.rjust`;Equivalent to ``str.rjust``  `~Series.str.zfill`;Equivalent to ``str.zfill``  `~Series.str.wrap`;Split long strings into lines with length less than a given width  `~Series.str.slice`;Slice each string in the Series  `~Series.str.slice_replace`;Replace slice in each string with passed value  `~Series.str.count`;Count occurrences of pattern  `~Series.str.startswith`;Equivalent to ``str.startswith(pat)`` for each element  `~Series.str.endswith`;Equivalent to ``str.endswith(pat)`` for each element  `~Series.str.findall`;Compute list of all occurrences of pattern/regex for each string  `~Series.str.match`;"Call ``re.match`` on each element, returning matched groups as list"  `~Series.str.extract`;"Call ``re.search`` on each element, returning DataFrame with one row for each element and one column for each regex capture group"  `~Series.str.extractall`;"Call ``re.findall`` on each element, returning DataFrame with one row for each match and one column for each regex capture group"  `~Series.str.len`;Compute string lengths  `~Series.str.strip`;Equivalent to ``str.strip``  `~Series.str.rstrip`;Equivalent to ``str.rstrip``  `~Series.str.lstrip`;Equivalent to ``str.lstrip``  `~Series.str.partition`;Equivalent to ``str.partition``  `~Series.str.rpartition`;Equivalent to ``str.rpartition``  `~Series.str.lower`;Equivalent to ``str.lower``  `~Series.str.casefold`;Equivalent to ``str.casefold``  `~Series.str.upper`;Equivalent to ``str.upper``  `~Series.str.find`;Equivalent to ``str.find``  `~Series.str.rfind`;Equivalent to ``str.rfind``  `~Series.str.index`;Equivalent to ``str.index``  `~Series.str.rindex`;Equivalent to ``str.rindex``  `~Series.str.capitalize`;Equivalent to ``str.capitalize``  `~Series.str.swapcase`;Equivalent to ``str.swapcase``  `~Series.str.normalize`;Return Unicode normal form. Equivalent to ``unicodedata.normalize``  `~Series.str.translate`;Equivalent to ``str.translate``  `~Series.str.isalnum`;Equivalent to ``str.isalnum``  `~Series.str.isalpha`;Equivalent to ``str.isalpha``  `~Series.str.isdigit`;Equivalent to ``str.isdigit``  `~Series.str.isspace`;Equivalent to ``str.isspace``  `~Series.str.islower`;Equivalent to ``str.islower``  `~Series.str.isupper`;Equivalent to ``str.isupper``  `~Series.str.istitle`;Equivalent to ``str.istitle``  `~Series.str.isnumeric`;Equivalent to ``str.isnumeric``  `~Series.str.isdecimal`;Equivalent to ``str.isdecimal``

Parsing

You can construct a ``Timedelta`` scalar through various arguments, including `ISO 8601 Duration`_ strings.  .. ipython:: python  import datetime  # strings  pd.Timedelta("1 days")  pd.Timedelta("1 days 000")  pd.Timedelta("1 days 2 hours")  pd.Timedelta("-1 days 2 min 3us")  # like datetime.timedelta  # note: these MUST be specified as keyword arguments  pd.Timedelta(days=1, seconds=1)  # integers with a unit  pd.Timedelta(1, unit="d")  # from a datetime.timedelta/np.timedelta64  pd.Timedelta(datetime.timedelta(days=1, seconds=1))  pd.Timedelta(np.timedelta64(1, "ms"))  # negative Timedeltas have this string repr  # to be more consistent with datetime.timedelta conventions  pd.Timedelta("-1us")  # a NaT  pd.Timedelta("nan")  pd.Timedelta("nat")  # ISO 8601 Duration strings  pd.Timedelta("P0DT0H1M0S")  pd.Timedelta("P0DT0H0M0.000123S")  `DateOffsets<timeseries.offsets>` (``Day, Hour, Minute, Second, Milli, Micro, Nano``) can also be used in construction.  .. ipython:: python  pd.Timedelta(pd.offsets.Second(2))  Further, operations among the scalars yield another scalar ``Timedelta``.  .. ipython:: python  pd.Timedelta(pd.offsets.Day(2)) + pd.Timedelta(pd.offsets.Second(2)) + pd.Timedelta(  "000.000123"  )  to_timedelta
~~~  Using the top-level ``pd.to_timedelta``, you can convert a scalar, array, list,
or Series from a recognized timedelta format / value into a ``Timedelta`` type.
It will construct Series if the input is a Series, a scalar if the input is
scalar-like, otherwise it will output a ``TimedeltaIndex``.  You can parse a single string to a Timedelta:  .. ipython:: python  pd.to_timedelta("1 days 0601.0003")  pd.to_timedelta("15.5us")  or a list/array of strings:  .. ipython:: python  pd.to_timedelta(["1 days 0601.0003", "15.5us", "nan"])  The ``unit`` keyword argument specifies the unit of the Timedelta if the input
is numeric:  .. ipython:: python  pd.to_timedelta(np.arange(5), unit="s")  pd.to_timedelta(np.arange(5), unit="d")  .. warning::  If a string or array of strings is passed as an input then the ``unit`` keyword  argument will be ignored. If a string without units is passed then the default  unit of nanoseconds is assumed.  .. _timedeltas.limitations:  Timedelta limitations
~~~  pandas represents ``Timedeltas`` in nanosecond resolution using
64 bit integers. As such, the 64 bit integer limits determine
the ``Timedelta`` limits.  .. ipython:: python  pd.Timedelta.min  pd.Timedelta.max  .. _timedeltas.operations:

You can operate on Series/DataFrames and construct ``timedelta64[ns]`` Series through
subtraction operations on ``datetime64[ns]`` Series, or ``Timestamps``.  .. ipython:: python  s = pd.Series(pd.date_range("2012-1-1", periods=3, freq="D"))  td = pd.Series([pd.Timedelta(days=i) for i in range(3)])  df = pd.DataFrame({"A": s, "B": td})  df  df["C"] = df["A"] + df["B"]  df  df.dtypes  s - s.max()  s - datetime.datetime(2011, 1, 1, 3, 5)  s + datetime.timedelta(minutes=5)  s + pd.offsets.Minute(5)  s + pd.offsets.Minute(5) + pd.offsets.Milli(5)  Operations with scalars from a ``timedelta64[ns]`` series:  .. ipython:: python  y = s - s[0]  y  Series of timedeltas with ``NaT`` values are supported:  .. ipython:: python  y = s - s.shift()  y  Elements can be set to ``NaT`` using ``np.nan`` analogously to datetimes:  .. ipython:: python  y[1] = np.nan  y  Operands can also appear in a reversed order (a singular object operated with a Series):  .. ipython:: python  s.max() - s  datetime.datetime(2011, 1, 1, 3, 5) - s  datetime.timedelta(minutes=5) + s  ``min, max`` and the corresponding ``idxmin, idxmax`` operations are supported on frames:  .. ipython:: python  A = s - pd.Timestamp("20120101") - pd.Timedelta("0005")  B = s - pd.Series(pd.date_range("2012-1-2", periods=3, freq="D"))  df = pd.DataFrame({"A": A, "B": B})  df  df.min()  df.min(axis=1)  df.idxmin()  df.idxmax()  ``min, max, idxmin, idxmax`` operations are supported on Series as well. A scalar result will be a ``Timedelta``.  .. ipython:: python  df.min().max()  df.min(axis=1).min()  df.min().idxmax()  df.min(axis=1).idxmin()  You can fillna on timedeltas, passing a timedelta to get a particular value.  .. ipython:: python  y.fillna(pd.Timedelta(0))  y.fillna(pd.Timedelta(10, unit="s"))  y.fillna(pd.Timedelta("-1 days, 0005"))  You can also negate, multiply and use ``abs`` on ``Timedeltas``:  .. ipython:: python  td1 = pd.Timedelta("-1 days 2 hours 3 seconds")  td1  -1 * td1  -td1  abs(td1)  .. _timedeltas.timedeltas_reductions:

Reductions

Numeric reduction operation for ``timedelta64[ns]`` will return ``Timedelta`` objects. As usual
``NaT`` are skipped during evaluation.  .. ipython:: python  y2 = pd.Series(  pd.to_timedelta(["-1 days +0005", "nat", "-1 days +0005", "1 days"])  )  y2  y2.mean()  y2.median()  y2.quantile(0.1)  y2.sum()  .. _timedeltas.timedeltas_convert:

Frequency conversion

Timedelta Series and ``TimedeltaIndex``, and ``Timedelta`` can be converted to other frequencies by astyping to a specific timedelta dtype.  .. ipython:: python  december = pd.Series(pd.date_range("20121201", periods=4))  january = pd.Series(pd.date_range("20130101", periods=4))  td = january - december  td[2] += datetime.timedelta(minutes=5, seconds=3)  td[3] = np.nan  td  # to seconds  td.astype("timedelta64[s]")  For timedelta64 resolutions other than the supported "s", "ms", "us", "ns",
an alternative is to divide by another timedelta object. Note that division by the NumPy scalar is true division, while astyping is equivalent of floor division.  .. ipython:: python  # to days  td / np.timedelta64(1, "D")  # to months (these are constant months)  td / np.timedelta64(1, "M")  Dividing or multiplying a ``timedelta64[ns]`` Series by an integer or integer Series
yields another ``timedelta64[ns]`` dtypes Series.  .. ipython:: python  td * -1  td * pd.Series([1, 2, 3, 4])  Rounded division (floor-division) of a ``timedelta64[ns]`` Series by a scalar
``Timedelta`` gives a series of integers.  .. ipython:: python  td // pd.Timedelta(days=3, hours=4)  pd.Timedelta(days=3, hours=4) // td  .. _timedeltas.mod_divmod:  The mod (%) and divmod operations are defined for ``Timedelta`` when operating with another timedelta-like or with a numeric argument.  .. ipython:: python  pd.Timedelta(hours=37) % datetime.timedelta(hours=2)  # divmod against a timedelta-like returns a pair (int, Timedelta)  divmod(datetime.timedelta(hours=2), pd.Timedelta(minutes=11))  # divmod against a numeric returns a pair (Timedelta, Timedelta)  divmod(pd.Timedelta(hours=25), 864000)

You can access various components of the ``Timedelta`` or ``TimedeltaIndex`` directly using the attributes ``days,seconds,microseconds,nanoseconds``. These are identical to the values returned by ``datetime.timedelta``, in that, for example, the ``.seconds`` attribute represents the number of seconds >= 0 and < 1 day. These are signed according to whether the ``Timedelta`` is signed.  These operations can also be directly accessed via the ``.dt`` property of the ``Series`` as well.  .. note::  Note that the attributes are NOT the displayed values of the ``Timedelta``. Use ``.components`` to retrieve the displayed values.  For a ``Series``:  .. ipython:: python  td.dt.days  td.dt.seconds  You can access the value of the fields for a scalar ``Timedelta`` directly.  .. ipython:: python  tds = pd.Timedelta("31 days 5 min 3 sec")  tds.days  tds.seconds  (-tds).seconds  You can use the ``.components`` property to access a reduced form of the timedelta. This returns a ``DataFrame`` indexed
similarly to the ``Series``. These are the *displayed* values of the ``Timedelta``.  .. ipython:: python  td.dt.components  td.dt.components.seconds  .. _timedeltas.isoformat:  You can convert a ``Timedelta`` to an `ISO 8601 Duration`_ string with the
``.isoformat`` method  .. ipython:: python  pd.Timedelta(  days=6, minutes=50, seconds=3, milliseconds=10, microseconds=10, nanoseconds=12  ).isoformat()  .. _ISO 8601 Duration: https://en.wikipedia.org/wiki/ISO_8601#Durations  .. _timedeltas.index:

To generate an index with time delta, you can use either the `TimedeltaIndex` or
the `timedelta_range` constructor.  Using ``TimedeltaIndex`` you can pass string-like, ``Timedelta``, ``timedelta``,
or ``np.timedelta64`` objects. Passing ``np.nan/pd.NaT/nat`` will represent missing values.  .. ipython:: python  pd.TimedeltaIndex(  [  "1 days",  "1 days, 0005",  np.timedelta64(2, "D"),  datetime.timedelta(days=2, seconds=2),  ]  )  The string 'infer' can be passed in order to set the frequency of the index as the
inferred frequency upon creation:  .. ipython:: python  pd.TimedeltaIndex(["0 days", "10 days", "20 days"], freq="infer")  Generating ranges of time deltas
~~~  Similar to `date_range`, you can construct regular ranges of a ``TimedeltaIndex``
using `timedelta_range`.  The default frequency for ``timedelta_range`` is
calendar day:  .. ipython:: python  pd.timedelta_range(start="1 days", periods=5)  Various combinations of ``start``, ``end``, and ``periods`` can be used with
``timedelta_range``:  .. ipython:: python  pd.timedelta_range(start="1 days", end="5 days")  pd.timedelta_range(end="10 days", periods=4)  The ``freq`` parameter can passed a variety of `frequency aliases <timeseries.offset_aliases>`:  .. ipython:: python  pd.timedelta_range(start="1 days", end="2 days", freq="30T")  pd.timedelta_range(start="1 days", periods=5, freq="2D5H")  Specifying ``start``, ``end``, and ``periods`` will generate a range of evenly spaced
timedeltas from ``start`` to ``end`` inclusively, with ``periods`` number of elements
in the resulting ``TimedeltaIndex``:  .. ipython:: python  pd.timedelta_range("0 days", "4 days", periods=5)  pd.timedelta_range("0 days", "4 days", periods=10)  Using the TimedeltaIndex
~~~  Similarly to other of the datetime-like indices, ``DatetimeIndex`` and ``PeriodIndex``, you can use
``TimedeltaIndex`` as the index of pandas objects.  .. ipython:: python  s = pd.Series(  np.arange(100),  index=pd.timedelta_range("1 days", periods=100, freq="h"),  )  s  Selections work similarly, with coercion on string-likes and slices:  .. ipython:: python  s["1 day":"2 day"]  s["1 day 0100"]  s[pd.Timedelta("1 day 1h")]  Furthermore you can use partial string selection and the range will be inferred:  .. ipython:: python  s["1 day":"1 day 5 hours"]  Operations
~~~  Finally, the combination of ``TimedeltaIndex`` with ``DatetimeIndex`` allow certain combination operations that are NaT preserving:  .. ipython:: python  tdi = pd.TimedeltaIndex(["1 days", pd.NaT, "2 days"])  tdi.to_list()  dti = pd.date_range("20130101", periods=3)  dti.to_list()  (dti + tdi).to_list()  (dti - tdi).to_list()  Conversions
~~~  Similarly to frequency conversion on a ``Series`` above, you can convert these indices to yield another Index.  .. ipython:: python  tdi / np.timedelta64(1, "s")  tdi.astype("timedelta64[s]")  Scalars type ops work as well. These can potentially return a *different* type of index.  .. ipython:: python  # adding or timedelta and date -> datelike  tdi + pd.Timestamp("20130101")  # subtraction of a date and a timedelta -> datelike  # note that trying to subtract a date from a Timedelta will raise an exception  (pd.Timestamp("20130101") - tdi).to_list()  # timedelta + timedelta -> timedelta  tdi + pd.Timedelta("10 days")  # division can result in a Timedelta if the divisor is an integer  tdi / 2  # or a float64 Index if the divisor is a Timedelta  tdi / tdi[0]  .. _timedeltas.resampling:

Similar to `timeseries resampling <timeseries.resampling>`, we can resample with a ``TimedeltaIndex``.  .. ipython:: python  s.resample("D").mean()

Overview

pandas captures 4 general time related concepts:  #. Date times: A specific date and time with timezone support. Similar to ``datetime.datetime`` from the standard library.
#. Time deltas: An absolute time duration. Similar to ``datetime.timedelta`` from the standard library.
#. Time spans: A span of time defined by a point in time and its associated frequency.
#. Date offsets: A relative time duration that respects calendar arithmetic. Similar to ``dateutil.relativedelta.relativedelta`` from the ``dateutil`` package.  ===  ===  ===  ===  ===
Concept  Scalar Class  Array Class  pandas Data Type  Primary Creation Method
===  ===  ===  ===  ===
Date times  ``Timestamp``  ``DatetimeIndex``  ``datetime64[ns]`` or ``datetime64[ns, tz]``  ``to_datetime`` or ``date_range``
Time deltas  ``Timedelta``  ``TimedeltaIndex``  ``timedelta64[ns]``  ``to_timedelta`` or ``timedelta_range``
Time spans  ``Period``  ``PeriodIndex``  ``period[freq]``  ``Period`` or ``period_range``
Date offsets  ``DateOffset``  ``None``  ``None``  ``DateOffset``
===  ===  ===  ===  ===  For time series data, it's conventional to represent the time component in the index of a `Series` or `DataFrame`
so manipulations can be performed with respect to the time element.  .. ipython:: python  pd.Series(range(3), index=pd.date_range("2000", freq="D", periods=3))  However, `Series` and `DataFrame` can directly also support the time component as data itself.  .. ipython:: python  pd.Series(pd.date_range("2000", freq="D", periods=3))  `Series` and `DataFrame` have extended data type support and functionality for ``datetime``, ``timedelta``
and ``Period`` data when passed into those constructors. ``DateOffset``
data however will be stored as ``object`` data.  .. ipython:: python  pd.Series(pd.period_range("1/1/2011", freq="M", periods=3))  pd.Series([pd.DateOffset(1), pd.DateOffset(2)])  pd.Series(pd.date_range("1/1/2011", freq="M", periods=3))  Lastly, pandas represents null date times, time deltas, and time spans as ``NaT`` which
is useful for representing missing or null date like values and behaves similar
as ``np.nan`` does for float data.  .. ipython:: python  pd.Timestamp(pd.NaT)  pd.Timedelta(pd.NaT)  pd.Period(pd.NaT)  # Equality acts as np.nan would  pd.NaT == pd.NaT  .. _timeseries.representation:

Timestamps vs. time spans

Timestamped data is the most basic type of time series data that associates
values with points in time. For pandas objects it means using the points in
time.  .. ipython:: python  import datetime  pd.Timestamp(datetime.datetime(2012, 5, 1))  pd.Timestamp("2012-05-01")  pd.Timestamp(2012, 5, 1)  However, in many cases it is more natural to associate things like change
variables with a time span instead. The span represented by ``Period`` can be
specified explicitly, or inferred from datetime string format.  For example:  .. ipython:: python  pd.Period("2011-01")  pd.Period("2012-05", freq="D")  `Timestamp` and `Period` can serve as an index. Lists of
``Timestamp`` and ``Period`` are automatically coerced to `DatetimeIndex`
and `PeriodIndex` respectively.  .. ipython:: python  dates = [  pd.Timestamp("2012-05-01"),  pd.Timestamp("2012-05-02"),  pd.Timestamp("2012-05-03"),  ]  ts = pd.Series(np.random.randn(3), dates)  type(ts.index)  ts.index  ts  periods = [pd.Period("2012-01"), pd.Period("2012-02"), pd.Period("2012-03")]  ts = pd.Series(np.random.randn(3), periods)  type(ts.index)  ts.index  ts  pandas allows you to capture both representations and
convert between them. Under the hood, pandas represents timestamps using
instances of ``Timestamp`` and sequences of timestamps using instances of
``DatetimeIndex``. For regular time spans, pandas uses ``Period`` objects for
scalar values and ``PeriodIndex`` for sequences of spans. Better support for
irregular intervals with arbitrary start and end points are forth-coming in
future releases.  .. _timeseries.converting:

To convert a `Series` or list-like object of date-like objects e.g. strings,
epochs, or a mixture, you can use the ``to_datetime`` function. When passed
a ``Series``, this returns a ``Series`` (with the same index), while a list-like
is converted to a ``DatetimeIndex``:  .. ipython:: python  pd.to_datetime(pd.Series(["Jul 31, 2009", "Jan 10, 2010", None]))  pd.to_datetime(["2005/11/23", "2010/12/31"])  If you use dates which start with the day first (i.e. European style),
you can pass the ``dayfirst`` flag:  .. ipython:: python  pd.to_datetime(["04-01-2012 10:00"], dayfirst=True)  pd.to_datetime(["04-14-2012 10:00"], dayfirst=True)  .. warning::  You see in the above example that ``dayfirst`` isn't strict. If a date  can't be parsed with the day being first it will be parsed as if  ``dayfirst`` were ``False`` and a warning will also be raised.  If you pass a single string to ``to_datetime``, it returns a single ``Timestamp``.
``Timestamp`` can also accept string input, but it doesn't accept string parsing
options like ``dayfirst`` or ``format``, so use ``to_datetime`` if these are required.  .. ipython:: python  pd.to_datetime("2010/11/12")  pd.Timestamp("2010/11/12")  You can also use the ``DatetimeIndex`` constructor directly:  .. ipython:: python  pd.DatetimeIndex(["2018-01-01", "2018-01-03", "2018-01-05"])  The string 'infer' can be passed in order to set the frequency of the index as the
inferred frequency upon creation:  .. ipython:: python  pd.DatetimeIndex(["2018-01-01", "2018-01-03", "2018-01-05"], freq="infer")  .. _timeseries.converting.format:  Providing a format argument
~~~  In addition to the required datetime string, a ``format`` argument can be passed to ensure specific parsing.
This could also potentially speed up the conversion considerably.  .. ipython:: python  pd.to_datetime("2010/11/12", format="%Y/%m/%d")  pd.to_datetime("12-11-2010 00:00", format="%d-%m-%Y %H:%M")  For more information on the choices available when specifying the ``format``
option, see the Python `datetime documentation`_.  .. _datetime documentation: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior  Assembling datetime from multiple DataFrame columns
~~~  You can also pass a ``DataFrame`` of integer or string columns to assemble into a ``Series`` of ``Timestamps``.  .. ipython:: python  df = pd.DataFrame(  {"year": [2015, 2016], "month": [2, 3], "day": [4, 5], "hour": [2, 3]}  )  pd.to_datetime(df)  You can pass only the columns that you need to assemble.  .. ipython:: python  pd.to_datetime(df[["year", "month", "day"]])  ``pd.to_datetime`` looks for standard designations of the datetime component in the column names, including:  * required: ``year``, ``month``, ``day``
* optional: ``hour``, ``minute``, ``second``, ``millisecond``, ``microsecond``, ``nanosecond``  Invalid data
~~~  The default behavior, ``errors='raise'``, is to raise when unparsable:  .. code-block:: ipython  In [2]: pd.to_datetime(['2009/07/31', 'asd'], errors='raise')  ValueError: Unknown datetime string format  Pass ``errors='ignore'`` to return the original input when unparsable:  .. ipython:: python  pd.to_datetime(["2009/07/31", "asd"], errors="ignore")  Pass ``errors='coerce'`` to convert unparsable data to ``NaT`` (not a time):  .. ipython:: python  pd.to_datetime(["2009/07/31", "asd"], errors="coerce")  .. _timeseries.converting.epoch:  Epoch timestamps
~~~  pandas supports converting integer or float epoch times to ``Timestamp`` and
``DatetimeIndex``. The default unit is nanoseconds, since that is how ``Timestamp``
objects are stored internally. However, epochs are often stored in another ``unit``
which can be specified. These are computed from the starting point specified by the

``origin`` parameter.  .. ipython:: python  pd.to_datetime(  [1349720105, 1349806505, 1349892905, 1349979305, 1350065705], unit="s"  )  pd.to_datetime(  [1349720105100, 1349720105200, 1349720105300, 1349720105400, 1349720105500],  unit="ms",  )  .. note::  The ``unit`` parameter does not use the same strings as the ``format`` parameter  that was discussed `above<timeseries.converting.format>`). The  available units are listed on the documentation for `pandas.to_datetime`.  .. versionchanged:: 1.0.0  Constructing a `Timestamp` or `DatetimeIndex` with an epoch timestamp
with the ``tz`` argument specified will raise a ValueError. If you have
epochs in wall time in another timezone, you can read the epochs
as timezone-naive timestamps and then localize to the appropriate timezone:  .. ipython:: python  pd.Timestamp(12623472000).tz_localize("US/Pacific")  pd.DatetimeIndex([12623472000]).tz_localize("US/Pacific")  .. note::  Epoch times will be rounded to the nearest nanosecond.  .. warning::  Conversion of float epoch times can lead to inaccurate and unexpected results.  `Python floats <python:tut-fp-issues>` have about 15 digits precision in  decimal. Rounding during conversion from float to high precision ``Timestamp`` is  unavoidable. The only way to achieve exact precision is to use a fixed-width  types (e.g. an int64).  .. ipython:: python  pd.to_datetime([1490195805.433, 1490195805.433502912], unit="s")  pd.to_datetime(1490195805433502912, unit="ns")  .. seealso::  `timeseries.origin`  .. _timeseries.converting.epoch_inverse:  From timestamps to epoch
~~~  To invert the operation from above, namely, to convert from a ``Timestamp`` to a 'unix' epoch:  .. ipython:: python  stamps = pd.date_range("2012-10-08 1805", periods=4, freq="D")  stamps  We subtract the epoch (midnight at January 1, 1970 UTC) and then floor divide by the
"unit" (1 second).  .. ipython:: python  (stamps - pd.Timestamp("1970-01-01")) // pd.Timedelta("1s")  .. _timeseries.origin:  Using the ``origin`` parameter
~~~  Using the ``origin`` parameter, one can specify an alternative starting point for creation
of a ``DatetimeIndex``. For example, to use 1960-01-01 as the starting date:  .. ipython:: python  pd.to_datetime([1, 2, 3], unit="D", origin=pd.Timestamp("1960-01-01"))  The default is set at ``origin='unix'``, which defaults to ``1970-01-01 000``.
Commonly called 'unix epoch' or POSIX time.  .. ipython:: python  pd.to_datetime([1, 2, 3], unit="D")  .. _timeseries.daterange:

Generating ranges of timestamps

To generate an index with timestamps, you can use either the ``DatetimeIndex`` or
``Index`` constructor and pass in a list of datetime objects:  .. ipython:: python  dates = [  datetime.datetime(2012, 5, 1),  datetime.datetime(2012, 5, 2),  datetime.datetime(2012, 5, 3),  ]  # Note the frequency information  index = pd.DatetimeIndex(dates)  index  # Automatically converted to DatetimeIndex  index = pd.Index(dates)  index  In practice this becomes very cumbersome because we often need a very long
index with a large number of timestamps. If we need timestamps on a regular
frequency, we can use the `date_range` and `bdate_range` functions
to create a ``DatetimeIndex``. The default frequency for ``date_range`` is a
**calendar day** while the default for ``bdate_range`` is a **business day**:  .. ipython:: python  start = datetime.datetime(2011, 1, 1)  end = datetime.datetime(2012, 1, 1)  index = pd.date_range(start, end)  index  index = pd.bdate_range(start, end)  index  Convenience functions like ``date_range`` and ``bdate_range`` can utilize a
variety of `frequency aliases <timeseries.offset_aliases>`:  .. ipython:: python  pd.date_range(start, periods=1000, freq="M")  pd.bdate_range(start, periods=250, freq="BQS")  ``date_range`` and ``bdate_range`` make it easy to generate a range of dates
using various combinations of parameters like ``start``, ``end``, ``periods``,
and ``freq``. The start and end dates are strictly inclusive, so dates outside
of those specified will not be generated:  .. ipython:: python  pd.date_range(start, end, freq="BM")  pd.date_range(start, end, freq="W")  pd.bdate_range(end=end, periods=20)  pd.bdate_range(start=start, periods=20)  Specifying ``start``, ``end``, and ``periods`` will generate a range of evenly spaced
dates from ``start`` to ``end`` inclusively, with ``periods`` number of elements in the
resulting ``DatetimeIndex``:  .. ipython:: python  pd.date_range("2018-01-01", "2018-01-05", periods=5)  pd.date_range("2018-01-01", "2018-01-05", periods=10)  .. _timeseries.custom-freq-ranges:  Custom frequency ranges
~~~  ``bdate_range`` can also generate a range of custom frequency dates by using
the ``weekmask`` and ``holidays`` parameters.  These parameters will only be
used if a custom frequency string is passed.  .. ipython:: python  weekmask = "Mon Wed Fri"  holidays = [datetime.datetime(2011, 1, 5), datetime.datetime(2011, 3, 14)]  pd.bdate_range(start, end, freq="C", weekmask=weekmask, holidays=holidays)  pd.bdate_range(start, end, freq="CBMS", weekmask=weekmask)  .. seealso::  `timeseries.custombusinessdays`  .. _timeseries.timestamp-limits:

Timestamp limitations

Since pandas represents timestamps in nanosecond resolution, the time span that
can be represented using a 64-bit integer is limited to approximately 584 years:  .. ipython:: python  pd.Timestamp.min  pd.Timestamp.max  .. seealso::  `timeseries.oob`  .. _timeseries.datetimeindex:

One of the main uses for ``DatetimeIndex`` is as an index for pandas objects.
The ``DatetimeIndex`` class contains many time series related optimizations:  * A large range of dates for various offsets are pre-computed and cached  under the hood in order to make generating subsequent date ranges very fast  (just have to grab a slice).
* Fast shifting using the ``shift`` method on pandas objects.
* Unioning of overlapping ``DatetimeIndex`` objects with the same frequency is  very fast (important for fast data alignment).
* Quick access to date fields via properties such as ``year``, ``month``, etc.
* Regularization functions like ``snap`` and very fast ``asof`` logic.  ``DatetimeIndex`` objects have all the basic functionality of regular ``Index``
objects, and a smorgasbord of advanced time series specific methods for easy
frequency processing.  .. seealso::  `Reindexing methods <basics.reindexing>`  .. note::  While pandas does not force you to have a sorted date index, some of these  methods may have unexpected or incorrect behavior if the dates are unsorted.  ``DatetimeIndex`` can be used like a regular index and offers all of its
intelligent functionality like selection, slicing, etc.  .. ipython:: python  rng = pd.date_range(start, end, freq="BM")  ts = pd.Series(np.random.randn(len(rng)), index=rng)  ts.index  ts[:5].index  ts[::2].index  .. _timeseries.partialindexing:  Partial string indexing
~~~  Dates and strings that parse to timestamps can be passed as indexing parameters:  .. ipython:: python  ts["1/31/2011"]  ts[datetime.datetime(2011, 12, 25):]  ts["10/31/2011":"12/31/2011"]  To provide convenience for accessing longer time series, you can also pass in
the year or year and month as strings:  .. ipython:: python  ts["2011"]  ts["2011-6"]  This type of slicing will work on a ``DataFrame`` with a ``DatetimeIndex`` as well. Since the
partial string selection is a form of label slicing, the endpoints **will be** included. This
would include matching times on an included date:  .. warning::  Indexing ``DataFrame`` rows with a *single* string with getitem (e.g. ``frame[dtstring]``)  is deprecated starting with pandas 1.2.0 (given the ambiguity whether it is indexing  the rows or selecting a column) and will be removed in a future version. The equivalent  with ``.loc`` (e.g. ``frame.loc[dtstring]``) is still supported.  .. ipython:: python  dft = pd.DataFrame(  np.random.randn(1000, 1),  columns=["A"],  index=pd.date_range("20130101", periods=1000, freq="T"),  )  dft  dft.loc["2013"]  This starts on the very first time in the month, and includes the last date and
time for the month:  .. ipython:: python  dft["2013-1":"2013-2"]  This specifies a stop time **that includes all of the times on the last day**:  .. ipython:: python  dft["2013-1":"2013-2-28"]  This specifies an **exact** stop time (and is not the same as the above):  .. ipython:: python  dft["2013-1":"2013-2-28 000"]  We are stopping on the included end-point as it is part of the index:  .. ipython:: python  dft["2013-1-15":"2013-1-15 1200"]  ``DatetimeIndex`` partial string indexing also works on a ``DataFrame`` with a ``MultiIndex``:  .. ipython:: python  dft2 = pd.DataFrame(  np.random.randn(20, 1),  columns=["A"],  index=pd.MultiIndex.from_product(  [pd.date_range("20130101", periods=10, freq="12H"), ["a", "b"]]  ),  )  dft2  dft2.loc["2013-01-05"]  idx = pd.IndexSlice  dft2 = dft2.swaplevel(0, 1).sort_index()  dft2.loc[idx[:, "2013-01-05"], :]  Slicing with string indexing also honors UTC offset.  .. ipython:: python  df = pd.DataFrame([0], index=pd.DatetimeIndex(["2019-01-01"], tz="US/Pacific"))  df  df["2019-01-01 1200+04:00":"2019-01-01 1300+04:00"]  .. _timeseries.slice_vs_exact_match:  Slice vs. exact match

~~~  The same string used as an indexing parameter can be treated either as a slice or as an exact match depending on the resolution of the index. If the string is less accurate than the index, it will be treated as a slice, otherwise as an exact match.  Consider a ``Series`` object with a minute resolution index:  .. ipython:: python  series_minute = pd.Series(  [1, 2, 3],  pd.DatetimeIndex(  ["2011-12-31 2300", "2012-01-01 000", "2012-01-01 000"]  ),  )  series_minute.index.resolution  A timestamp string less accurate than a minute gives a ``Series`` object.  .. ipython:: python  series_minute["2011-12-31 23"]  A timestamp string with minute resolution (or more accurate), gives a scalar instead, i.e. it is not casted to a slice.  .. ipython:: python  series_minute["2011-12-31 23:59"]  series_minute["2011-12-31 2300"]  If index resolution is second, then the minute-accurate timestamp gives a
``Series``.  .. ipython:: python  series_second = pd.Series(  [1, 2, 3],  pd.DatetimeIndex(  ["2011-12-31 2359", "2012-01-01 000", "2012-01-01 0001"]  ),  )  series_second.index.resolution  series_second["2011-12-31 23:59"]  If the timestamp string is treated as a slice, it can be used to index ``DataFrame`` with ``.loc[]`` as well.  .. ipython:: python  dft_minute = pd.DataFrame(  {"a": [1, 2, 3], "b": [4, 5, 6]}, index=series_minute.index  )  dft_minute.loc["2011-12-31 23"]  .. warning::  However, if the string is treated as an exact match, the selection in ``DataFrame``'s ``[]`` will be column-wise and not row-wise, see `Indexing Basics <indexing.basics>`. For example ``dft_minute['2011-12-31 23:59']`` will raise ``KeyError`` as ``'2012-12-31 23:59'`` has the same resolution as the index and there is no column with such name:  To *always* have unambiguous selection, whether the row is treated as a slice or a single selection, use ``.loc``.  .. ipython:: python  dft_minute.loc["2011-12-31 23:59"]  Note also that ``DatetimeIndex`` resolution cannot be less precise than day.  .. ipython:: python  series_monthly = pd.Series(  [1, 2, 3], pd.DatetimeIndex(["2011-12", "2012-01", "2012-02"])  )  series_monthly.index.resolution  series_monthly["2011-12"]  # returns Series  Exact indexing
~~~  As discussed in previous section, indexing a ``DatetimeIndex`` with a partial string depends on the "accuracy" of the period, in other words how specific the interval is in relation to the resolution of the index. In contrast, indexing with ``Timestamp`` or ``datetime`` objects is exact, because the objects have exact meaning. These also follow the semantics of *including both endpoints*.  These ``Timestamp`` and ``datetime`` objects have exact ``hours, minutes,`` and ``seconds``, even though they were not explicitly specified (they are ``0``).  .. ipython:: python  dft[datetime.datetime(2013, 1, 1): datetime.datetime(2013, 2, 28)]  With no defaults.  .. ipython:: python  dft[  datetime.datetime(2013, 1, 1, 10, 12, 0): datetime.datetime(  2013, 2, 28, 10, 12, 0  )  ]  Truncating & fancy indexing
~~~  A `~DataFrame.truncate` convenience function is provided that is similar
to slicing. Note that ``truncate`` assumes a 0 value for any unspecified date
component in a ``DatetimeIndex`` in contrast to slicing which returns any
partially matching dates:  .. ipython:: python  rng2 = pd.date_range("2011-01-01", "2012-01-01", freq="W")  ts2 = pd.Series(np.random.randn(len(rng2)), index=rng2)  ts2.truncate(before="2011-11", after="2011-12")  ts2["2011-11":"2011-12"]  Even complicated fancy indexing that breaks the ``DatetimeIndex`` frequency
regularity will result in a ``DatetimeIndex``, although frequency is lost:  .. ipython:: python  ts2[[0, 2, 6]].index  .. _timeseries.components:

Time/date components

There are several time/date properties that one can access from ``Timestamp`` or a collection of timestamps like a ``DatetimeIndex``.  .. csv-table::  "Property", "Description"  15, 65  year, "The year of the datetime"  month,"The month of the datetime"  day,"The days of the datetime"  hour,"The hour of the datetime"  minute,"The minutes of the datetime"  second,"The seconds of the datetime"  microsecond,"The microseconds of the datetime"  nanosecond,"The nanoseconds of the datetime"  date,"Returns datetime.date (does not contain timezone information)"  time,"Returns datetime.time (does not contain timezone information)"  timetz,"Returns datetime.time as local time with timezone information"  dayofyear,"The ordinal day of year"  day_of_year,"The ordinal day of year"  weekofyear,"The week ordinal of the year"  week,"The week ordinal of the year"  dayofweek,"The number of the day of the week with Monday=0, Sunday=6"  day_of_week,"The number of the day of the week with Monday=0, Sunday=6"  weekday,"The number of the day of the week with Monday=0, Sunday=6"  quarter,"Quarter of the date: Jan-Mar = 1, Apr-Jun = 2, etc."  days_in_month,"The number of days in the month of the datetime"  is_month_start,"Logical indicating if first day of month (defined by frequency)"  is_month_end,"Logical indicating if last day of month (defined by frequency)"  is_quarter_start,"Logical indicating if first day of quarter (defined by frequency)"  is_quarter_end,"Logical indicating if last day of quarter (defined by frequency)"  is_year_start,"Logical indicating if first day of year (defined by frequency)"  is_year_end,"Logical indicating if last day of year (defined by frequency)"  is_leap_year,"Logical indicating if the date belongs to a leap year"  Furthermore, if you have a ``Series`` with datetimelike values, then you can
access these properties via the ``.dt`` accessor, as detailed in the section
on `.dt accessors<basics.dt_accessors>`.  .. versionadded:: 1.1.0  You may obtain the year, week and day components of the ISO year from the ISO 8601 standard:  .. ipython:: python  idx = pd.date_range(start="2019-12-29", freq="D", periods=4)  idx.isocalendar()  idx.to_series().dt.isocalendar()  .. _timeseries.offsets:

In the preceding examples, frequency strings (e.g. ``'D'``) were used to specify
a frequency that defined:  * how the date times in `DatetimeIndex` were spaced when using `date_range`
* the frequency of a `Period` or `PeriodIndex`  These frequency strings map to a `DateOffset` object and its subclasses. A `DateOffset`
is similar to a `Timedelta` that represents a duration of time but follows specific calendar duration rules.
For example, a `Timedelta` day will always increment ``datetimes`` by 24 hours, while a `DateOffset` day
will increment ``datetimes`` to the same time the next day whether a day represents 23, 24 or 25 hours due to daylight
savings time. However, all `DateOffset` subclasses that are an hour or smaller
(``Hour``, ``Minute``, ``Second``, ``Milli``, ``Micro``, ``Nano``) behave like
`Timedelta` and respect absolute time.  The basic `DateOffset` acts similar to ``dateutil.relativedelta`` (`relativedelta documentation`_)
that shifts a date time by the corresponding calendar duration specified. The
arithmetic operator (``+``) can be used to perform the shift.  .. ipython:: python  # This particular day contains a day light savings time transition  ts = pd.Timestamp("2016-10-30 000", tz="Europe/Helsinki")  # Respects absolute time  ts + pd.Timedelta(days=1)  # Respects calendar time  ts + pd.DateOffset(days=1)  friday = pd.Timestamp("2018-01-05")  friday.day_name()  # Add 2 business days (Friday --> Tuesday)  two_business_days = 2 * pd.offsets.BDay()  friday + two_business_days  (friday + two_business_days).day_name()  Most ``DateOffsets`` have associated frequencies strings, or offset aliases, that can be passed

into ``freq`` keyword arguments. The available date offsets and associated frequency strings can be found below:  .. csv-table::  "Date Offset", "Frequency String", "Description"  15, 15, 65  `~pandas.tseries.offsets.DateOffset`, None, "Generic offset class, defaults to absolute 24 hours"  `~pandas.tseries.offsets.BDay` or `~pandas.tseries.offsets.BusinessDay`, ``'B'``,"business day (weekday)"  `~pandas.tseries.offsets.CDay` or `~pandas.tseries.offsets.CustomBusinessDay`, ``'C'``, "custom business day"  `~pandas.tseries.offsets.Week`, ``'W'``, "one week, optionally anchored on a day of the week"  `~pandas.tseries.offsets.WeekOfMonth`, ``'WOM'``, "the x-th day of the y-th week of each month"  `~pandas.tseries.offsets.LastWeekOfMonth`, ``'LWOM'``, "the x-th day of the last week of each month"  `~pandas.tseries.offsets.MonthEnd`, ``'M'``, "calendar month end"  `~pandas.tseries.offsets.MonthBegin`, ``'MS'``, "calendar month begin"  `~pandas.tseries.offsets.BMonthEnd` or `~pandas.tseries.offsets.BusinessMonthEnd`, ``'BM'``, "business month end"  `~pandas.tseries.offsets.BMonthBegin` or `~pandas.tseries.offsets.BusinessMonthBegin`, ``'BMS'``, "business month begin"  `~pandas.tseries.offsets.CBMonthEnd` or `~pandas.tseries.offsets.CustomBusinessMonthEnd`, ``'CBM'``, "custom business month end"  `~pandas.tseries.offsets.CBMonthBegin` or `~pandas.tseries.offsets.CustomBusinessMonthBegin`, ``'CBMS'``, "custom business month begin"  `~pandas.tseries.offsets.SemiMonthEnd`, ``'SM'``, "15th (or other day_of_month) and calendar month end"  `~pandas.tseries.offsets.SemiMonthBegin`, ``'SMS'``, "15th (or other day_of_month) and calendar month begin"  `~pandas.tseries.offsets.QuarterEnd`, ``'Q'``, "calendar quarter end"  `~pandas.tseries.offsets.QuarterBegin`, ``'QS'``, "calendar quarter begin"  `~pandas.tseries.offsets.BQuarterEnd`, ``'BQ``, "business quarter end"  `~pandas.tseries.offsets.BQuarterBegin`, ``'BQS'``, "business quarter begin"  `~pandas.tseries.offsets.FY5253Quarter`, ``'REQ'``, "retail (aka 52-53 week) quarter"  `~pandas.tseries.offsets.YearEnd`, ``'A'``, "calendar year end"  `~pandas.tseries.offsets.YearBegin`, ``'AS'`` or ``'BYS'``,"calendar year begin"  `~pandas.tseries.offsets.BYearEnd`, ``'BA'``, "business year end"  `~pandas.tseries.offsets.BYearBegin`, ``'BAS'``, "business year begin"  `~pandas.tseries.offsets.FY5253`, ``'RE'``, "retail (aka 52-53 week) year"  `~pandas.tseries.offsets.Easter`, None, "Easter holiday"  `~pandas.tseries.offsets.BusinessHour`, ``'BH'``, "business hour"  `~pandas.tseries.offsets.CustomBusinessHour`, ``'CBH'``, "custom business hour"  `~pandas.tseries.offsets.Day`, ``'D'``, "one absolute day"  `~pandas.tseries.offsets.Hour`, ``'H'``, "one hour"  `~pandas.tseries.offsets.Minute`, ``'T'`` or ``'min'``,"one minute"  `~pandas.tseries.offsets.Second`, ``'S'``, "one second"  `~pandas.tseries.offsets.Milli`, ``'L'`` or ``'ms'``, "one millisecond"  `~pandas.tseries.offsets.Micro`, ``'U'`` or ``'us'``, "one microsecond"  `~pandas.tseries.offsets.Nano`, ``'N'``, "one nanosecond"  ``DateOffsets`` additionally have `rollforward` and `rollback`
methods for moving a date forward or backward respectively to a valid offset
date relative to the offset. For example, business offsets will roll dates
that land on the weekends (Saturday and Sunday) forward to Monday since
business offsets operate on the weekdays.  .. ipython:: python  ts = pd.Timestamp("2018-01-06 000")  ts.day_name()  # BusinessHour's valid offset dates are Monday through Friday  offset = pd.offsets.BusinessHour(start="09:00")  # Bring the date to the closest offset date (Monday)  offset.rollforward(ts)  # Date is brought to the closest offset date first and then the hour is added  ts + offset  These operations preserve time (hour, minute, etc) information by default.
To reset time to midnight, use `normalize` before or after applying
the operation (depending on whether you want the time information included

in the operation).  .. ipython:: python  ts = pd.Timestamp("2014-01-01 09:00")  day = pd.offsets.Day()  day + ts  (day + ts).normalize()  ts = pd.Timestamp("2014-01-01 22:00")  hour = pd.offsets.Hour()  hour + ts  (hour + ts).normalize()  (hour + pd.Timestamp("2014-01-01 23:30")).normalize()  .. _relativedelta documentation: https://dateutil.readthedocs.io/en/stable/relativedelta.html  Parametric offsets
~~~  Some of the offsets can be "parameterized" when created to result in different
behaviors. For example, the ``Week`` offset for generating weekly data accepts a
``weekday`` parameter which results in the generated dates always lying on a
particular day of the week:  .. ipython:: python  d = datetime.datetime(2008, 8, 18, 9, 0)  d  d + pd.offsets.Week()  d + pd.offsets.Week(weekday=4)  (d + pd.offsets.Week(weekday=4)).weekday()  d - pd.offsets.Week()  The ``normalize`` option will be effective for addition and subtraction.  .. ipython:: python  d + pd.offsets.Week(normalize=True)  d - pd.offsets.Week(normalize=True)  Another example is parameterizing ``YearEnd`` with the specific ending month:  .. ipython:: python  d + pd.offsets.YearEnd()  d + pd.offsets.YearEnd(month=6)  .. _timeseries.offsetseries:  Using offsets with ``Series`` / ``DatetimeIndex``
~~~  Offsets can be used with either a ``Series`` or ``DatetimeIndex`` to
apply the offset to each element.  .. ipython:: python  rng = pd.date_range("2012-01-01", "2012-01-03")  s = pd.Series(rng)  rng  rng + pd.DateOffset(months=2)  s + pd.DateOffset(months=2)  s - pd.DateOffset(months=2)  If the offset class maps directly to a ``Timedelta`` (``Day``, ``Hour``,
``Minute``, ``Second``, ``Micro``, ``Milli``, ``Nano``) it can be
used exactly like a ``Timedelta`` - see the
`Timedelta section<timedeltas.operations>` for more examples.  .. ipython:: python  s - pd.offsets.Day(2)  td = s - pd.Series(pd.date_range("2011-12-29", "2011-12-31"))  td  td + pd.offsets.Minute(15)  Note that some offsets (such as ``BQuarterEnd``) do not have a
vectorized implementation.  They can still be used but may
calculate significantly slower and will show a ``PerformanceWarning``  .. ipython:: python  rng + pd.offsets.BQuarterEnd()  .. _timeseries.custombusinessdays:  Custom business days
~~~  The ``CDay`` or ``CustomBusinessDay`` class provides a parametric
``BusinessDay`` class which can be used to create customized business day
calendars which account for local holidays and local weekend conventions.  As an interesting example, let's look at Egypt where a Friday-Saturday weekend is observed.  .. ipython:: python  weekmask_egypt = "Sun Mon Tue Wed Thu"  # They also observe International Workers' Day so let's  # add that for a couple of years  holidays = [  "2012-05-01",  datetime.datetime(2013, 5, 1),  np.datetime64("2014-05-01"),  ]  bday_egypt = pd.offsets.CustomBusinessDay(  holidays=holidays,  weekmask=weekmask_egypt,  )  dt = datetime.datetime(2013, 4, 30)  dt + 2 * bday_egypt  Let's map to the weekday names:  .. ipython:: python  dts = pd.date_range(dt, periods=5, freq=bday_egypt)  pd.Series(dts.weekday, dts).map(pd.Series("Mon Tue Wed Thu Fri Sat Sun".split()))  Holiday calendars can be used to provide the list of holidays.  See the
`holiday calendar<timeseries.holiday>` section for more information.  .. ipython:: python  from pandas.tseries.holiday import USFederalHolidayCalendar  bday_us = pd.offsets.CustomBusinessDay(calendar=USFederalHolidayCalendar())  # Friday before MLK Day  dt = datetime.datetime(2014, 1, 17)  # Tuesday after MLK Day (Monday is skipped because it's a holiday)  dt + bday_us  Monthly offsets that respect a certain holiday calendar can be defined

in the usual way.  .. ipython:: python  bmth_us = pd.offsets.CustomBusinessMonthBegin(calendar=USFederalHolidayCalendar())  # Skip new years  dt = datetime.datetime(2013, 12, 17)  dt + bmth_us  # Define date index with custom offset  pd.date_range(start="20100101", end="20120101", freq=bmth_us)  .. note::  The frequency string 'C' is used to indicate that a CustomBusinessDay  DateOffset is used, it is important to note that since CustomBusinessDay is  a parameterised type, instances of CustomBusinessDay may differ and this is  not detectable from the 'C' frequency string. The user therefore needs to  ensure that the 'C' frequency string is used consistently within the user's  application.  .. _timeseries.businesshour:  Business hour
~~~  The ``BusinessHour`` class provides a business hour representation on ``BusinessDay``,
allowing to use specific start and end times.  By default, ``BusinessHour`` uses 9:00 - 17:00 as business hours.
Adding ``BusinessHour`` will increment ``Timestamp`` by hourly frequency.
If target ``Timestamp`` is out of business hours, move to the next business hour
then increment it. If the result exceeds the business hours end, the remaining
hours are added to the next business day.  .. ipython:: python  bh = pd.offsets.BusinessHour()  bh  # 2014-08-01 is Friday  pd.Timestamp("2014-08-01 10:00").weekday()  pd.Timestamp("2014-08-01 10:00") + bh  # Below example is the same as: pd.Timestamp('2014-08-01 09:00') + bh  pd.Timestamp("2014-08-01 08:00") + bh  # If the results is on the end time, move to the next business day  pd.Timestamp("2014-08-01 16:00") + bh  # Remainings are added to the next day  pd.Timestamp("2014-08-01 16:30") + bh  # Adding 2 business hours  pd.Timestamp("2014-08-01 10:00") + pd.offsets.BusinessHour(2)  # Subtracting 3 business hours  pd.Timestamp("2014-08-01 10:00") + pd.offsets.BusinessHour(-3)  You can also specify ``start`` and ``end`` time by keywords. The argument must
be a ``str`` with an ``hour:minute`` representation or a ``datetime.time``
instance. Specifying seconds, microseconds and nanoseconds as business hour
results in ``ValueError``.  .. ipython:: python  bh = pd.offsets.BusinessHour(start="11:00", end=datetime.time(20, 0))  bh  pd.Timestamp("2014-08-01 13:00") + bh  pd.Timestamp("2014-08-01 09:00") + bh  pd.Timestamp("2014-08-01 18:00") + bh  Passing ``start`` time later than ``end`` represents midnight business hour.
In this case, business hour exceeds midnight and overlap to the next day.
Valid business hours are distinguished by whether it started from valid ``BusinessDay``.  .. ipython:: python  bh = pd.offsets.BusinessHour(start="17:00", end="09:00")  bh  pd.Timestamp("2014-08-01 17:00") + bh  pd.Timestamp("2014-08-01 23:00") + bh  # Although 2014-08-02 is Saturday,  # it is valid because it starts from 08-01 (Friday).  pd.Timestamp("2014-08-02 04:00") + bh  # Although 2014-08-04 is Monday,  # it is out of business hours because it starts from 08-03 (Sunday).  pd.Timestamp("2014-08-04 04:00") + bh  Applying ``BusinessHour.rollforward`` and ``rollback`` to out of business hours results in
the next business hour start or previous day's end. Different from other offsets, ``BusinessHour.rollforward``
may output different results from ``apply`` by definition.  This is because one day's business hour end is equal to next day's business hour start. For example,
under the default business hours (9:00 - 17:00), there is no gap (0 minutes) between ``2014-08-01 17:00`` and

``2014-08-04 09:00``.  .. ipython:: python  # This adjusts a Timestamp to business hour edge  pd.offsets.BusinessHour().rollback(pd.Timestamp("2014-08-02 15:00"))  pd.offsets.BusinessHour().rollforward(pd.Timestamp("2014-08-02 15:00"))  # It is the same as BusinessHour() + pd.Timestamp('2014-08-01 17:00').  # And it is the same as BusinessHour() + pd.Timestamp('2014-08-04 09:00')  pd.offsets.BusinessHour() + pd.Timestamp("2014-08-02 15:00")  # BusinessDay results (for reference)  pd.offsets.BusinessHour().rollforward(pd.Timestamp("2014-08-02"))  # It is the same as BusinessDay() + pd.Timestamp('2014-08-01')  # The result is the same as rollworward because BusinessDay never overlap.  pd.offsets.BusinessHour() + pd.Timestamp("2014-08-02")  ``BusinessHour`` regards Saturday and Sunday as holidays. To use arbitrary
holidays, you can use ``CustomBusinessHour`` offset, as explained in the
following subsection.  .. _timeseries.custombusinesshour:  Custom business hour
~~~  The ``CustomBusinessHour`` is a mixture of ``BusinessHour`` and ``CustomBusinessDay`` which
allows you to specify arbitrary holidays. ``CustomBusinessHour`` works as the same
as ``BusinessHour`` except that it skips specified custom holidays.  .. ipython:: python  from pandas.tseries.holiday import USFederalHolidayCalendar  bhour_us = pd.offsets.CustomBusinessHour(calendar=USFederalHolidayCalendar())  # Friday before MLK Day  dt = datetime.datetime(2014, 1, 17, 15)  dt + bhour_us  # Tuesday after MLK Day (Monday is skipped because it's a holiday)  dt + bhour_us * 2  You can use keyword arguments supported by either ``BusinessHour`` and ``CustomBusinessDay``.  .. ipython:: python  bhour_mon = pd.offsets.CustomBusinessHour(start="10:00", weekmask="Tue Wed Thu Fri")  # Monday is skipped because it's a holiday, business hour starts from 10:00  dt + bhour_mon * 2  .. _timeseries.offset_aliases:  Offset aliases
~~~  A number of string aliases are given to useful common time series

frequencies. We will refer to these aliases as *offset aliases*.  .. csv-table::  "Alias", "Description"  15, 100  "B", "business day frequency"  "C", "custom business day frequency"  "D", "calendar day frequency"  "W", "weekly frequency"  "M", "month end frequency"  "SM", "semi-month end frequency (15th and end of month)"  "BM", "business month end frequency"  "CBM", "custom business month end frequency"  "MS", "month start frequency"  "SMS", "semi-month start frequency (1st and 15th)"  "BMS", "business month start frequency"  "CBMS", "custom business month start frequency"  "Q", "quarter end frequency"  "BQ", "business quarter end frequency"  "QS", "quarter start frequency"  "BQS", "business quarter start frequency"  "A, Y", "year end frequency"  "BA, BY", "business year end frequency"  "AS, YS", "year start frequency"  "BAS, BYS", "business year start frequency"  "BH", "business hour frequency"  "H", "hourly frequency"  "T, min", "minutely frequency"  "S", "secondly frequency"  "L, ms", "milliseconds"  "U, us", "microseconds"  "N", "nanoseconds"  .. note::  When using the offset aliases above, it should be noted that functions  such as `date_range`, `bdate_range`, will only return  timestamps that are in the interval defined by ``start_date`` and  ``end_date``. If the ``start_date`` does not correspond to the frequency,  the returned timestamps will start at the next valid timestamp, same for  ``end_date``, the returned timestamps will stop at the previous valid  timestamp.  For example, for the offset ``MS``, if the ``start_date`` is not the first  of the month, the returned timestamps will start with the first day of the  next month. If ``end_date`` is not the first day of a month, the last  returned timestamp will be the first day of the corresponding month.  .. ipython:: python  dates_lst_1 = pd.date_range("2020-01-06", "2020-04-03", freq="MS")  dates_lst_1  dates_lst_2 = pd.date_range("2020-01-01", "2020-04-01", freq="MS")  dates_lst_2  We can see in the above example `date_range` and  `bdate_range` will only return the valid timestamps between the  ``start_date`` and ``end_date``. If these are not valid timestamps for the  given frequency it will roll to the next value for ``start_date``  (respectively previous for the ``end_date``)  Combining aliases
~~~  As we have seen previously, the alias and the offset instance are fungible in
most functions:  .. ipython:: python  pd.date_range(start, periods=5, freq="B")  pd.date_range(start, periods=5, freq=pd.offsets.BDay())  You can combine together day and intraday offsets:  .. ipython:: python  pd.date_range(start, periods=10, freq="2h20min")  pd.date_range(start, periods=10, freq="1D10U")  Anchored offsets

~~~  For some frequencies you can specify an anchoring suffix:  .. csv-table::  "Alias", "Description"  15, 100  "W\-SUN", "weekly frequency (Sundays). Same as 'W'"  "W\-MON", "weekly frequency (Mondays)"  "W\-TUE", "weekly frequency (Tuesdays)"  "W\-WED", "weekly frequency (Wednesdays)"  "W\-THU", "weekly frequency (Thursdays)"  "W\-FRI", "weekly frequency (Fridays)"  "W\-SAT", "weekly frequency (Saturdays)"  "(B)Q(S)\-DEC", "quarterly frequency, year ends in December. Same as 'Q'"  "(B)Q(S)\-JAN", "quarterly frequency, year ends in January"  "(B)Q(S)\-FEB", "quarterly frequency, year ends in February"  "(B)Q(S)\-MAR", "quarterly frequency, year ends in March"  "(B)Q(S)\-APR", "quarterly frequency, year ends in April"  "(B)Q(S)\-MAY", "quarterly frequency, year ends in May"  "(B)Q(S)\-JUN", "quarterly frequency, year ends in June"  "(B)Q(S)\-JUL", "quarterly frequency, year ends in July"  "(B)Q(S)\-AUG", "quarterly frequency, year ends in August"  "(B)Q(S)\-SEP", "quarterly frequency, year ends in September"  "(B)Q(S)\-OCT", "quarterly frequency, year ends in October"  "(B)Q(S)\-NOV", "quarterly frequency, year ends in November"  "(B)A(S)\-DEC", "annual frequency, anchored end of December. Same as 'A'"  "(B)A(S)\-JAN", "annual frequency, anchored end of January"  "(B)A(S)\-FEB", "annual frequency, anchored end of February"  "(B)A(S)\-MAR", "annual frequency, anchored end of March"  "(B)A(S)\-APR", "annual frequency, anchored end of April"  "(B)A(S)\-MAY", "annual frequency, anchored end of May"  "(B)A(S)\-JUN", "annual frequency, anchored end of June"  "(B)A(S)\-JUL", "annual frequency, anchored end of July"  "(B)A(S)\-AUG", "annual frequency, anchored end of August"  "(B)A(S)\-SEP", "annual frequency, anchored end of September"  "(B)A(S)\-OCT", "annual frequency, anchored end of October"  "(B)A(S)\-NOV", "annual frequency, anchored end of November"  These can be used as arguments to ``date_range``, ``bdate_range``, constructors
for ``DatetimeIndex``, as well as various other timeseries-related functions
in pandas.  Anchored offset semantics
~~~  For those offsets that are anchored to the start or end of specific
frequency (``MonthEnd``, ``MonthBegin``, ``WeekEnd``, etc), the following
rules apply to rolling forward and backwards.  When ``n`` is not 0, if the given date is not on an anchor point, it snapped to the next(previous)
anchor point, and moved ``|n|-1`` additional steps forwards or backwards.  .. ipython:: python  pd.Timestamp("2014-01-02") + pd.offsets.MonthBegin(n=1)  pd.Timestamp("2014-01-02") + pd.offsets.MonthEnd(n=1)  pd.Timestamp("2014-01-02") - pd.offsets.MonthBegin(n=1)  pd.Timestamp("2014-01-02") - pd.offsets.MonthEnd(n=1)  pd.Timestamp("2014-01-02") + pd.offsets.MonthBegin(n=4)  pd.Timestamp("2014-01-02") - pd.offsets.MonthBegin(n=4)  If the given date *is* on an anchor point, it is moved ``|n|`` points forwards
or backwards.  .. ipython:: python  pd.Timestamp("2014-01-01") + pd.offsets.MonthBegin(n=1)  pd.Timestamp("2014-01-31") + pd.offsets.MonthEnd(n=1)  pd.Timestamp("2014-01-01") - pd.offsets.MonthBegin(n=1)  pd.Timestamp("2014-01-31") - pd.offsets.MonthEnd(n=1)  pd.Timestamp("2014-01-01") + pd.offsets.MonthBegin(n=4)  pd.Timestamp("2014-01-31") - pd.offsets.MonthBegin(n=4)  For the case when ``n=0``, the date is not moved if on an anchor point, otherwise
it is rolled forward to the next anchor point.  .. ipython:: python  pd.Timestamp("2014-01-02") + pd.offsets.MonthBegin(n=0)  pd.Timestamp("2014-01-02") + pd.offsets.MonthEnd(n=0)  pd.Timestamp("2014-01-01") + pd.offsets.MonthBegin(n=0)  pd.Timestamp("2014-01-31") + pd.offsets.MonthEnd(n=0)  .. _timeseries.holiday:  Holidays / holiday calendars
~~~  Holidays and calendars provide a simple way to define holiday rules to be used
with ``CustomBusinessDay`` or in other analysis that requires a predefined
set of holidays.  The ``AbstractHolidayCalendar`` class provides all the necessary
methods to return a list of holidays and only ``rules`` need to be defined

in a specific holiday calendar class. Furthermore, the ``start_date`` and ``end_date``
class attributes determine over what date range holidays are generated.  These
should be overwritten on the ``AbstractHolidayCalendar`` class to have the range
apply to all calendar subclasses.  ``USFederalHolidayCalendar`` is the
only calendar that exists and primarily serves as an example for developing
other calendars.  For holidays that occur on fixed dates (e.g., US Memorial Day or July 4th) an
observance rule determines when that holiday is observed if it falls on a weekend
or some other non-observed day.  Defined observance rules are:  .. csv-table::  "Rule", "Description"  15, 70  "nearest_workday", "move Saturday to Friday and Sunday to Monday"  "sunday_to_monday", "move Sunday to following Monday"  "next_monday_or_tuesday", "move Saturday to Monday and Sunday/Monday to Tuesday"  "previous_friday", move Saturday and Sunday to previous Friday"  "next_monday", "move Saturday and Sunday to following Monday"  An example of how holidays and holiday calendars are defined:  .. ipython:: python  from pandas.tseries.holiday import (  Holiday,  USMemorialDay,  AbstractHolidayCalendar,  nearest_workday,  MO,  )  class ExampleCalendar(AbstractHolidayCalendar):  rules = [  USMemorialDay,  Holiday("July 4th", month=7, day=4, observance=nearest_workday),  Holiday(  "Columbus Day",  month=10,  day=1,  offset=pd.DateOffset(weekday=MO(2)),  ),  ]  cal = ExampleCalendar()  cal.holidays(datetime.datetime(2012, 1, 1), datetime.datetime(2012, 12, 31))  **weekday=MO(2)** is same as **2 * Week(weekday=2)**  Using this calendar, creating an index or doing offset arithmetic skips weekends
and holidays (i.e., Memorial Day/July 4th).  For example, the below defines
a custom business day offset using the ``ExampleCalendar``.  Like any other offset,
it can be used to create a ``DatetimeIndex`` or added to ``datetime``
or ``Timestamp`` objects.  .. ipython:: python  pd.date_range(  start="7/1/2012", end="7/10/2012", freq=pd.offsets.CDay(calendar=cal)  ).to_pydatetime()  offset = pd.offsets.CustomBusinessDay(calendar=cal)  datetime.datetime(2012, 5, 25) + offset  datetime.datetime(2012, 7, 3) + offset  datetime.datetime(2012, 7, 3) + 2 * offset  datetime.datetime(2012, 7, 6) + offset  Ranges are defined by the ``start_date`` and ``end_date`` class attributes
of ``AbstractHolidayCalendar``.  The defaults are shown below.  .. ipython:: python  AbstractHolidayCalendar.start_date  AbstractHolidayCalendar.end_date  These dates can be overwritten by setting the attributes as
datetime/Timestamp/string.  .. ipython:: python  AbstractHolidayCalendar.start_date = datetime.datetime(2012, 1, 1)  AbstractHolidayCalendar.end_date = datetime.datetime(2012, 12, 31)  cal.holidays()  Every calendar class is accessible by name using the ``get_calendar`` function
which returns a holiday class instance.  Any imported calendar class will
automatically be available by this function.  Also, ``HolidayCalendarFactory``
provides an easy interface to create calendars that are combinations of calendars
or calendars with additional rules.  .. ipython:: python  from pandas.tseries.holiday import get_calendar, HolidayCalendarFactory, USLaborDay  cal = get_calendar("ExampleCalendar")  cal.rules  new_cal = HolidayCalendarFactory("NewExampleCalendar", cal, USLaborDay)  new_cal.rules  .. _timeseries.advanced_datetime:

Time Series-related instance methods

Shifting / lagging
~~~  One may want to *shift* or *lag* the values in a time series back and forward in
time. The method for this is `~Series.shift`, which is available on all of
the pandas objects.  .. ipython:: python  ts = pd.Series(range(len(rng)), index=rng)  ts = ts[:5]  ts.shift(1)  The ``shift`` method accepts an ``freq`` argument which can accept a
``DateOffset`` class or other ``timedelta``-like object or also an
`offset alias <timeseries.offset_aliases>`.  When ``freq`` is specified, ``shift`` method changes all the dates in the index
rather than changing the alignment of the data and the index:  .. ipython:: python  ts.shift(5, freq="D")  ts.shift(5, freq=pd.offsets.BDay())  ts.shift(5, freq="BM")  Note that with when ``freq`` is specified, the leading entry is no longer NaN
because the data is not being realigned.  Frequency conversion
~~~  The primary function for changing frequencies is the `~Series.asfreq`
method. For a ``DatetimeIndex``, this is basically just a thin, but convenient
wrapper around `~Series.reindex`  which generates a ``date_range`` and
calls ``reindex``.  .. ipython:: python  dr = pd.date_range("1/1/2010", periods=3, freq=3 * pd.offsets.BDay())  ts = pd.Series(np.random.randn(3), index=dr)  ts  ts.asfreq(pd.offsets.BDay())  ``asfreq`` provides a further convenience so you can specify an interpolation
method for any gaps that may appear after the frequency conversion.  .. ipython:: python  ts.asfreq(pd.offsets.BDay(), method="pad")  Filling forward / backward
~~~  Related to ``asfreq`` and ``reindex`` is `~Series.fillna`, which is
documented in the `missing data section <missing_data.fillna>`.  Converting to Python datetimes
~~~  ``DatetimeIndex`` can be converted to an array of Python native
class:`datetime.datetime` objects using the ``to_pydatetime`` method.  .. _timeseries.resampling:

pandas has a simple, powerful, and efficient functionality for performing
financial applications.  `~Series.resample` is a time-based groupby, followed by a reduction method
on each of its groups. See some `cookbook examples <cookbook.resample>` for
some advanced strategies.  The ``resample()`` method can be used directly from ``DataFrameGroupBy`` objects,
see the `groupby docs <groupby.transform.window_resample>`.  Basics
~~~  .. ipython:: python  rng = pd.date_range("1/1/2012", periods=100, freq="S")  ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)  ts.resample("5Min").sum()  The ``resample`` function is very flexible and allows you to specify many
different parameters to control the frequency conversion and resampling
operation.  Any function available via `dispatching <groupby.dispatch>` is available as
a method of the returned object, including ``sum``, ``mean``, ``std``, ``sem``,
``max``, ``min``, ``median``, ``first``, ``last``, ``ohlc``:  .. ipython:: python  ts.resample("5Min").mean()  ts.resample("5Min").ohlc()  ts.resample("5Min").max()  For downsampling, ``closed`` can be set to 'left' or 'right' to specify which
end of the interval is closed:  .. ipython:: python  ts.resample("5Min", closed="right").mean()  ts.resample("5Min", closed="left").mean()  Parameters like ``label`` are used to manipulate the resulting labels.
``label`` specifies whether the result is labeled with the beginning or
the end of the interval.  .. ipython:: python  ts.resample("5Min").mean()  # by default label='left'  ts.resample("5Min", label="left").mean()  .. warning::  The default values for ``label`` and ``closed`` is '**left**' for all  frequency offsets except for 'M', 'A', 'Q', 'BM', 'BA', 'BQ', and 'W'  which all have a default of 'right'.  This might unintendedly lead to looking ahead, where the value for a later  time is pulled back to a previous time as in the following example with  the `~pandas.tseries.offsets.BusinessDay` frequency:  .. ipython:: python  s = pd.date_range("2000-01-01", "2000-01-05").to_series()  s.iloc[2] = pd.NaT  s.dt.day_name()  # default: label='left', closed='left'  s.resample("B").last().dt.day_name()  Notice how the value for Sunday got pulled back to the previous Friday.  To get the behavior where the value for Sunday is pushed to Monday, use  instead  .. ipython:: python  s.resample("B", label="right", closed="right").last().dt.day_name()  The ``axis`` parameter can be set to 0 or 1 and allows you to resample the
specified axis for a ``DataFrame``.  ``kind`` can be set to 'timestamp' or 'period' to convert the resulting index
to/from timestamp and time span representations. By default ``resample``
retains the input representation.  ``convention`` can be set to 'start' or 'end' when resampling period data
(detail below). It specifies how low frequency periods are converted to higher
frequency periods.  Upsampling
~~~  For upsampling, you can specify a way to upsample and the ``limit`` parameter to interpolate over the gaps that are created:  .. ipython:: python  # from secondly to every 250 milliseconds  ts[:2].resample("250L").asfreq()  ts[:2].resample("250L").ffill()  ts[:2].resample("250L").ffill(limit=2)  Sparse resampling
~~~  Sparse timeseries are the ones where you have a lot fewer points relative
to the amount of time you are looking to resample. Naively upsampling a sparse
series can potentially generate lots of intermediate values. When you don't want
to use a method to fill these values, e.g. ``fill_method`` is ``None``, then
intermediate values will be filled with ``NaN``.  Since ``resample`` is a time-based groupby, the following is a method to efficiently

resample only the groups that are not all ``NaN``.  .. ipython:: python  rng = pd.date_range("2014-1-1", periods=100, freq="D") + pd.Timedelta("1s")  ts = pd.Series(range(100), index=rng)  If we want to resample to the full range of the series:  .. ipython:: python  ts.resample("3T").sum()  We can instead only resample those groups where we have points as follows:  .. ipython:: python  from functools import partial  from pandas.tseries.frequencies import to_offset  def round(t, freq):  # round a Timestamp to a specified freq  freq = to_offset(freq)  return pd.Timestamp((t.value // freq.delta.value) * freq.delta.value)  ts.groupby(partial(round, freq="3T")).sum()  .. _timeseries.aggregate:  Aggregation
~~~  Similar to the `aggregating API <basics.aggregate>`, `groupby API <groupby.aggregate>`, and the `window API <window.overview>`,
a ``Resampler`` can be selectively resampled.  Resampling a ``DataFrame``, the default will be to act on all columns with the same function.  .. ipython:: python  df = pd.DataFrame(  np.random.randn(1000, 3),  index=pd.date_range("1/1/2012", freq="S", periods=1000),  columns=["A", "B", "C"],  )  r = df.resample("3T")  r.mean()  We can select a specific column or columns using standard getitem.  .. ipython:: python  r["A"].mean()  r[["A", "B"]].mean()  You can pass a list or dict of functions to do aggregation with, outputting a ``DataFrame``:  .. ipython:: python  r["A"].agg([np.sum, np.mean, np.std])  On a resampled ``DataFrame``, you can pass a list of functions to apply to each
column, which produces an aggregated result with a hierarchical index:  .. ipython:: python  r.agg([np.sum, np.mean])  By passing a dict to ``aggregate`` you can apply a different aggregation to the
columns of a ``DataFrame``:  .. ipython:: python  r.agg({"A": np.sum, "B": lambda x: np.std(x, ddof=1)})  The function names can also be strings. In order for a string to be valid it
must be implemented on the resampled object:  .. ipython:: python  r.agg({"A": "sum", "B": "std"})  Furthermore, you can also specify multiple aggregation functions for each column separately.  .. ipython:: python  r.agg({"A": ["sum", "std"], "B": ["mean", "std"]})  If a ``DataFrame`` does not have a datetimelike index, but instead you want
to resample based on datetimelike column in the frame, it can passed to the
``on`` keyword.  .. ipython:: python  df = pd.DataFrame(  {"date": pd.date_range("2015-01-01", freq="W", periods=5), "a": np.arange(5)},  index=pd.MultiIndex.from_arrays(  [[1, 2, 3, 4, 5], pd.date_range("2015-01-01", freq="W", periods=5)],  names=["v", "d"],  ),  )  df  df.resample("M", on="date")[["a"]].sum()  Similarly, if you instead want to resample by a datetimelike
level of ``MultiIndex``, its name or location can be passed to the
``level`` keyword.  .. ipython:: python  df.resample("M", level="d")[["a"]].sum()  .. _timeseries.iterating-label:  Iterating through groups
~~~  With the ``Resampler`` object in hand, iterating through the grouped data is very
natural and functions similarly to func:`itertools.groupby`:  .. ipython:: python  small = pd.Series(  range(6),  index=pd.to_datetime(  [  "2017-01-01T000",  "2017-01-01T000",  "2017-01-01T000",  "2017-01-01T0100",  "2017-01-01T0300",  "2017-01-01T0300",  ]  ),  )  resampled = small.resample("H")  for name, group in resampled:  print("Group: ", name)  print("-" * 27)  print(group, end="\n\n")  See `groupby.iterating-label` or `Resampler.__iter__` for more.  .. _timeseries.adjust-the-start-of-the-bins:  Use ``origin`` or ``offset`` to adjust the start of the bins

~~~  .. versionadded:: 1.1.0  The bins of the grouping are adjusted based on the beginning of the day of the time series starting point. This works well with frequencies that are multiples of a day (like ``30D``) or that divide a day evenly (like ``90s`` or ``1min``). This can create inconsistencies with some frequencies that do not meet this criteria. To change this behavior you can specify a fixed Timestamp with the argument ``origin``.  For example:  .. ipython:: python  start, end = "2000-10-01 2300", "2000-10-02 000"  middle = "2000-10-02 000"  rng = pd.date_range(start, end, freq="7min")  ts = pd.Series(np.arange(len(rng)) * 3, index=rng)  ts  Here we can see that, when using ``origin`` with its default value (``'start_day'``), the result after ``'2000-10-02 000'`` are not identical depending on the start of time series:  .. ipython:: python  ts.resample("17min", origin="start_day").sum()  ts[middle:end].resample("17min", origin="start_day").sum()  Here we can see that, when setting ``origin`` to ``'epoch'``, the result after ``'2000-10-02 000'`` are identical depending on the start of time series:  .. ipython:: python  ts.resample("17min", origin="epoch").sum()  ts[middle:end].resample("17min", origin="epoch").sum()  If needed you can use a custom timestamp for ``origin``:  .. ipython:: python  ts.resample("17min", origin="2001-01-01").sum()  ts[middle:end].resample("17min", origin=pd.Timestamp("2001-01-01")).sum()  If needed you can just adjust the bins with an ``offset`` Timedelta that would be added to the default ``origin``.
Those two examples are equivalent for this time series:  .. ipython:: python  ts.resample("17min", origin="start").sum()  ts.resample("17min", offset="23h30min").sum()  Note the use of ``'start'`` for ``origin`` on the last example. In that case, ``origin`` will be set to the first value of the timeseries.  Backward resample
~~~  .. versionadded:: 1.3.0  Instead of adjusting the beginning of bins, sometimes we need to fix the end of the bins to make a backward resample with a given ``freq``. The backward resample sets ``closed`` to ``'right'`` by default since the last value should be considered as the edge point for the last bin.  We can set ``origin`` to ``'end'``. The value for a specific ``Timestamp`` index stands for the resample result from the current ``Timestamp`` minus ``freq`` to the current ``Timestamp`` with a right close.  .. ipython:: python  ts.resample('17min', origin='end').sum()  Besides, in contrast with the ``'start_day'`` option, ``end_day`` is supported. This will set the origin as the ceiling midnight of the largest ``Timestamp``.  .. ipython:: python  ts.resample('17min', origin='end_day').sum()  The above result uses ``2000-10-02 000`` as the last bin's right edge since the following computation.  .. ipython:: python  ceil_mid = rng.max().ceil('D')  freq = pd.offsets.Minute(17)  bin_res = ceil_mid - freq * ((ceil_mid - rng.max()) // freq)  bin_res  .. _timeseries.periods:

Regular intervals of time are represented by ``Period`` objects in pandas while
sequences of ``Period`` objects are collected in a ``PeriodIndex``, which can
be created with the convenience function ``period_range``.  Period
~~~  A ``Period`` represents a span of time (e.g., a day, a month, a quarter, etc).
You can specify the span via ``freq`` keyword using a frequency alias like below.
Because ``freq`` represents a span of ``Period``, it cannot be negative like "-3D".  .. ipython:: python  pd.Period("2012", freq="A-DEC")  pd.Period("2012-1-1", freq="D")  pd.Period("2012-1-1 19:00", freq="H")  pd.Period("2012-1-1 19:00", freq="5H")  Adding and subtracting integers from periods shifts the period by its own
frequency. Arithmetic is not allowed between ``Period`` with different ``freq`` (span).  .. ipython:: python  p = pd.Period("2012", freq="A-DEC")  p + 1  p - 3  p = pd.Period("2012-01", freq="2M")  p + 2  p - 1  p == pd.Period("2012-01", freq="3M")  If ``Period`` freq is daily or higher (``D``, ``H``, ``T``, ``S``, ``L``, ``U``, ``N``), ``offsets`` and ``timedelta``-like can be added if the result can have the same freq. Otherwise, ``ValueError`` will be raised.  .. ipython:: python  p = pd.Period("2014-07-01 09:00", freq="H")  p + pd.offsets.Hour(2)  p + datetime.timedelta(minutes=120)  p + np.timedelta64(7200, "s")  .. code-block:: ipython  In [1]: p + pd.offsets.Minute(5)  Traceback  ...  ValueError: Input has different freq from Period(freq=H)  If ``Period`` has other frequencies, only the same ``offsets`` can be added. Otherwise, ``ValueError`` will be raised.  .. ipython:: python  p = pd.Period("2014-07", freq="M")  p + pd.offsets.MonthEnd(3)  .. code-block:: ipython  In [1]: p + pd.offsets.MonthBegin(3)  Traceback  ...  ValueError: Input has different freq from Period(freq=M)  Taking the difference of ``Period`` instances with the same frequency will
return the number of frequency units between them:  .. ipython:: python  pd.Period("2012", freq="A-DEC") - pd.Period("2002", freq="A-DEC")  PeriodIndex and period_range
~~~
Regular sequences of ``Period`` objects can be collected in a ``PeriodIndex``,
which can be constructed using the ``period_range`` convenience function:  .. ipython:: python  prng = pd.period_range("1/1/2011", "1/1/2012", freq="M")  prng  The ``PeriodIndex`` constructor can also be used directly:  .. ipython:: python  pd.PeriodIndex(["2011-1", "2011-2", "2011-3"], freq="M")  Passing multiplied frequency outputs a sequence of ``Period`` which
has multiplied span.  .. ipython:: python  pd.period_range(start="2014-01", freq="3M", periods=4)  If ``start`` or ``end`` are ``Period`` objects, they will be used as anchor
endpoints for a ``PeriodIndex`` with frequency matching that of the
``PeriodIndex`` constructor.  .. ipython:: python  pd.period_range(  start=pd.Period("2017Q1", freq="Q"), end=pd.Period("2017Q2", freq="Q"), freq="M"  )  Just like ``DatetimeIndex``, a ``PeriodIndex`` can also be used to index pandas
objects:  .. ipython:: python  ps = pd.Series(np.random.randn(len(prng)), prng)  ps  ``PeriodIndex`` supports addition and subtraction with the same rule as ``Period``.  .. ipython:: python  idx = pd.period_range("2014-07-01 09:00", periods=5, freq="H")  idx  idx + pd.offsets.Hour(2)  idx = pd.period_range("2014-07", periods=5, freq="M")  idx  idx + pd.offsets.MonthEnd(3)  ``PeriodIndex`` has its own dtype named ``period``, refer to `Period Dtypes <timeseries.period_dtype>`.  .. _timeseries.period_dtype:  Period dtypes
~~~  ``PeriodIndex`` has a custom ``period`` dtype. This is a pandas extension
dtype similar to the `timezone aware dtype <timeseries.timezone_series>` (``datetime64[ns, tz]``).  The ``period`` dtype holds the ``freq`` attribute and is represented with

``period[freq]`` like ``period[D]`` or ``period[M]``, using `frequency strings <timeseries.offset_aliases>`.  .. ipython:: python  pi = pd.period_range("2016-01-01", periods=3, freq="M")  pi  pi.dtype  The ``period`` dtype can be used in ``.astype(...)``. It allows one to change the
``freq`` of a ``PeriodIndex`` like ``.asfreq()`` and convert a
``DatetimeIndex`` to ``PeriodIndex`` like ``to_period()``:  .. ipython:: python  # change monthly freq to daily freq  pi.astype("period[D]")  # convert to DatetimeIndex  pi.astype("datetime64[ns]")  # convert to PeriodIndex  dti = pd.date_range("2011-01-01", freq="M", periods=3)  dti  dti.astype("period[M]")  PeriodIndex partial string indexing
~~~  PeriodIndex now supports partial string slicing with non-monotonic indexes.  .. versionadded:: 1.1.0  You can pass in dates and strings to ``Series`` and ``DataFrame`` with ``PeriodIndex``, in the same manner as ``DatetimeIndex``. For details, refer to `DatetimeIndex Partial String Indexing <timeseries.partialindexing>`.  .. ipython:: python  ps["2011-01"]  ps[datetime.datetime(2011, 12, 25):]  ps["10/31/2011":"12/31/2011"]  Passing a string representing a lower frequency than ``PeriodIndex`` returns partial sliced data.  .. ipython:: python  ps["2011"]  dfp = pd.DataFrame(  np.random.randn(600, 1),  columns=["A"],  index=pd.period_range("2013-01-01 9:00", periods=600, freq="T"),  )  dfp  dfp.loc["2013-01-01 10H"]  As with ``DatetimeIndex``, the endpoints will be included in the result. The example below slices data starting from 10:00 to 11:59.  .. ipython:: python  dfp["2013-01-01 10H":"2013-01-01 11H"]  Frequency conversion and resampling with PeriodIndex
~~~
The frequency of ``Period`` and ``PeriodIndex`` can be converted via the ``asfreq``
method. Let's start with the fiscal year 2011, ending in December:  .. ipython:: python  p = pd.Period("2011", freq="A-DEC")  p  We can convert it to a monthly frequency. Using the ``how`` parameter, we can
specify whether to return the starting or ending month:  .. ipython:: python  p.asfreq("M", how="start")  p.asfreq("M", how="end")  The shorthands 's' and 'e' are provided for convenience:  .. ipython:: python  p.asfreq("M", "s")  p.asfreq("M", "e")  Converting to a "super-period" (e.g., annual frequency is a super-period of
quarterly frequency) automatically returns the super-period that includes the
input period:  .. ipython:: python  p = pd.Period("2011-12", freq="M")  p.asfreq("A-NOV")  Note that since we converted to an annual frequency that ends the year in
November, the monthly period of December 2011 is actually in the 2012 A-NOV
period.  .. _timeseries.quarterly:  Period conversions with anchored frequencies are particularly useful for
working with various quarterly data common to economics, business, and other
fields. Many organizations define quarters relative to the month in which their
fiscal year starts and ends. Thus, first quarter of 2011 could start in 2010 or
a few months into 2011. Via anchored frequencies, pandas works for all quarterly
frequencies ``Q-JAN`` through ``Q-DEC``.  ``Q-DEC`` define regular calendar quarters:  .. ipython:: python  p = pd.Period("2012Q1", freq="Q-DEC")  p.asfreq("D", "s")  p.asfreq("D", "e")  ``Q-MAR`` defines fiscal year end in March:  .. ipython:: python  p = pd.Period("2011Q4", freq="Q-MAR")  p.asfreq("D", "s")  p.asfreq("D", "e")  .. _timeseries.interchange:

Converting between representations

Timestamped data can be converted to PeriodIndex-ed data using ``to_period``
and vice-versa using ``to_timestamp``:  .. ipython:: python  rng = pd.date_range("1/1/2012", periods=5, freq="M")  ts = pd.Series(np.random.randn(len(rng)), index=rng)  ts  ps = ts.to_period()  ps  ps.to_timestamp()  Remember that 's' and 'e' can be used to return the timestamps at the start or
end of the period:  .. ipython:: python  ps.to_timestamp("D", how="s")  Converting between period and timestamp enables some convenient arithmetic
the quarter end:  .. ipython:: python  prng = pd.period_range("1990Q1", "2000Q4", freq="Q-NOV")  ts = pd.Series(np.random.randn(len(prng)), prng)  ts.index = (prng.asfreq("M", "e") + 1).asfreq("H", "s") + 9  ts.head()  .. _timeseries.oob:

Representing out-of-bounds spans

If you have data that is outside of the ``Timestamp`` bounds, see `Timestamp limitations <timeseries.timestamp-limits>`,
then you can use a ``PeriodIndex`` and/or ``Series`` of ``Periods`` to do computations.  .. ipython:: python  span = pd.period_range("1215-01-01", "1381-01-01", freq="D")  span  To convert from an ``int64`` based YYYMMDD representation.  .. ipython:: python  s = pd.Series([20121231, 20141130, 9991231])  s  def conv(x):  return pd.Period(year=x // 1000, month=x // 100 % 100, day=x % 100, freq="D")  s.apply(conv)  s.apply(conv)[2]  These can easily be converted to a ``PeriodIndex``:  .. ipython:: python  span = pd.PeriodIndex(s.apply(conv))  span  .. _timeseries.timezone:

pandas provides rich support for working with timestamps in different time
zones using the ``pytz`` and ``dateutil`` libraries or `datetime.timezone`
objects from the standard library.  Working with time zones
~~~  By default, pandas objects are time zone unaware:  .. ipython:: python  rng = pd.date_range("3/6/2012 00:00", periods=15, freq="D")  rng.tz is None  To localize these dates to a time zone (assign a particular time zone to a naive date),
you can use the ``tz_localize`` method or the ``tz`` keyword argument in
`date_range`, `Timestamp`, or `DatetimeIndex`.
You can either pass ``pytz`` or ``dateutil`` time zone objects or Olson time zone database strings.
Olson time zone strings will return ``pytz`` time zone objects by default.
To return ``dateutil`` time zone objects, append ``dateutil/`` before the string.  * In ``pytz`` you can find a list of common (and less common) time zones using  ``from pytz import common_timezones, all_timezones``.
* ``dateutil`` uses the OS time zones so there isn't a fixed list available. For  common zones, the names are the same as ``pytz``.  .. ipython:: python  import dateutil  # pytz  rng_pytz = pd.date_range("3/6/2012 00:00", periods=3, freq="D", tz="Europe/London")  rng_pytz.tz  # dateutil  rng_dateutil = pd.date_range("3/6/2012 00:00", periods=3, freq="D")  rng_dateutil = rng_dateutil.tz_localize("dateutil/Europe/London")  rng_dateutil.tz  # dateutil - utc special case  rng_utc = pd.date_range(  "3/6/2012 00:00",  periods=3,  freq="D",  tz=dateutil.tz.tzutc(),  )  rng_utc.tz  .. ipython:: python  # datetime.timezone  rng_utc = pd.date_range(  "3/6/2012 00:00",  periods=3,  freq="D",  tz=datetime.timezone.utc,  )  rng_utc.tz  Note that the ``UTC`` time zone is a special case in ``dateutil`` and should be constructed explicitly
as an instance of ``dateutil.tz.tzutc``. You can also construct other time
zones objects explicitly first.  .. ipython:: python  import pytz  # pytz  tz_pytz = pytz.timezone("Europe/London")  rng_pytz = pd.date_range("3/6/2012 00:00", periods=3, freq="D")  rng_pytz = rng_pytz.tz_localize(tz_pytz)  rng_pytz.tz == tz_pytz  # dateutil  tz_dateutil = dateutil.tz.gettz("Europe/London")  rng_dateutil = pd.date_range("3/6/2012 00:00", periods=3, freq="D", tz=tz_dateutil)  rng_dateutil.tz == tz_dateutil  To convert a time zone aware pandas object from one time zone to another,

you can use the ``tz_convert`` method.  .. ipython:: python  rng_pytz.tz_convert("US/Eastern")  .. note::  When using ``pytz`` time zones, `DatetimeIndex` will construct a different  time zone object than a `Timestamp` for the same time zone input. A `DatetimeIndex`  can hold a collection of `Timestamp` objects that may have different UTC offsets and cannot be  succinctly represented by one ``pytz`` time zone instance while one `Timestamp`  represents one point in time with a specific UTC offset.  .. ipython:: python  dti = pd.date_range("2019-01-01", periods=3, freq="D", tz="US/Pacific")  dti.tz  ts = pd.Timestamp("2019-01-01", tz="US/Pacific")  ts.tz  .. warning::  Be wary of conversions between libraries. For some time zones, ``pytz`` and ``dateutil`` have different  definitions of the zone. This is more of a problem for unusual time zones than for  'standard' zones like ``US/Eastern``.  .. warning::  Be aware that a time zone definition across versions of time zone libraries may not  be considered equal.  This may cause problems when working with stored data that  is localized using one version and operated on with a different version.  See `here<io.hdf5-notes>` for how to handle such a situation.  .. warning::  For ``pytz`` time zones, it is incorrect to pass a time zone object directly into  the ``datetime.datetime`` constructor  (e.g., ``datetime.datetime(2011, 1, 1, tzinfo=pytz.timezone('US/Eastern'))``.  Instead, the datetime needs to be localized using the ``localize`` method  on the ``pytz`` time zone object.  .. warning::  Be aware that for times in the future, correct conversion between time zones  (and UTC) cannot be guaranteed by any time zone library because a timezone's  offset from UTC may be changed by the respective government.  .. warning::  If you are using dates beyond 2038-01-18, due to current deficiencies  in the underlying libraries caused by the year 2038 problem, daylight saving time (DST) adjustments  to timezone aware dates will not be applied. If and when the underlying libraries are fixed,  the DST transitions will be applied.  For example, for two dates that are in British Summer Time (and so would normally be GMT+1), both the following asserts evaluate as true:  .. ipython:: python  d_2037 = "2037-03-31T010101"  d_2038 = "2038-03-31T010101"  DST = "Europe/London"  assert pd.Timestamp(d_2037, tz=DST) != pd.Timestamp(d_2037, tz="GMT")  assert pd.Timestamp(d_2038, tz=DST) == pd.Timestamp(d_2038, tz="GMT")  Under the hood, all timestamps are stored in UTC. Values from a time zone aware
`DatetimeIndex` or `Timestamp` will have their fields (day, hour, minute, etc.)
localized to the time zone. However, timestamps with the same UTC value are
still considered to be equal even if they are in different time zones:  .. ipython:: python  rng_eastern = rng_utc.tz_convert("US/Eastern")  rng_berlin = rng_utc.tz_convert("Europe/Berlin")  rng_eastern[2]  rng_berlin[2]  rng_eastern[2] == rng_berlin[2]  Operations between `Series` in different time zones will yield UTC
`Series`, aligning the data on the UTC timestamps:  .. ipython:: python  ts_utc = pd.Series(range(3), pd.date_range("20130101", periods=3, tz="UTC"))  eastern = ts_utc.tz_convert("US/Eastern")  berlin = ts_utc.tz_convert("Europe/Berlin")  result = eastern + berlin  result  result.index  To remove time zone information, use ``tz_localize(None)`` or ``tz_convert(None)``.
``tz_localize(None)`` will remove the time zone yielding the local time representation.
``tz_convert(None)`` will remove the time zone after converting to UTC time.  .. ipython:: python  didx = pd.date_range(start="2014-08-01 09:00", freq="H", periods=3, tz="US/Eastern")  didx  didx.tz_localize(None)  didx.tz_convert(None)  # tz_convert(None) is identical to tz_convert('UTC').tz_localize(None)  didx.tz_convert("UTC").tz_localize(None)  .. _timeseries.fold:  Fold
~~~  .. versionadded:: 1.1.0  For ambiguous times, pandas supports explicitly specifying the keyword-only fold argument.

Due to daylight saving time, one wall clock time can occur twice when shifting
from summer to winter time; fold describes whether the datetime-like corresponds
to the first (0) or the second time (1) the wall clock hits the ambiguous time.
Fold is supported only for constructing from naive ``datetime.datetime``
(see datetime documentation_ for details) or from `Timestamp`
or for constructing from components (see below). Only ``dateutil`` timezones are supported
(see dateutil documentation_
for ``dateutil`` methods that deal with ambiguous datetimes) as ``pytz``
timezones do not support fold (see pytz documentation_
for details on how ``pytz`` deals with ambiguous datetimes). To localize an ambiguous datetime
with ``pytz``, please use `Timestamp.tz_localize`. In general, we recommend to rely
on `Timestamp.tz_localize` when localizing ambiguous datetimes if you need direct
control over how they are handled.  .. ipython:: python  pd.Timestamp(  datetime.datetime(2019, 10, 27, 1, 30, 0, 0),  tz="dateutil/Europe/London",  fold=0,  )  pd.Timestamp(  year=2019,  month=10,  day=27,  hour=1,  minute=30,  tz="dateutil/Europe/London",  fold=1,  )  .. _timeseries.timezone_ambiguous:  Ambiguous times when localizing
~~~  ``tz_localize`` may not be able to determine the UTC offset of a timestamp
because daylight savings time (DST) in a local time zone causes some times to occur
twice within one day ("clocks fall back"). The following options are available:  * ``'raise'``: Raises a ``pytz.AmbiguousTimeError`` (the default behavior)
* ``'infer'``: Attempt to determine the correct offset base on the monotonicity of the timestamps
* ``'NaT'``: Replaces ambiguous times with ``NaT``
* ``bool``: ``True`` represents a DST time, ``False`` represents non-DST time. An array-like of ``bool`` values is supported for a sequence of times.  .. ipython:: python  rng_hourly = pd.DatetimeIndex(  ["11/06/2011 00:00", "11/06/2011 01:00", "11/06/2011 01:00", "11/06/2011 02:00"]  )  This will fail as there are ambiguous times (``'11/06/2011 01:00'``)  .. code-block:: ipython  In [2]: rng_hourly.tz_localize('US/Eastern')  AmbiguousTimeError: Cannot infer dst time from Timestamp('2011-11-06 0100'), try using the 'ambiguous' argument  Handle these ambiguous times by specifying the following.  .. ipython:: python  rng_hourly.tz_localize("US/Eastern", ambiguous="infer")  rng_hourly.tz_localize("US/Eastern", ambiguous="NaT")  rng_hourly.tz_localize("US/Eastern", ambiguous=[True, True, False, False])  .. _timeseries.timezone_nonexistent:  Nonexistent times when localizing
~~~  A DST transition may also shift the local time ahead by 1 hour creating nonexistent
local times ("clocks spring forward"). The behavior of localizing a timeseries with nonexistent times
can be controlled by the ``nonexistent`` argument. The following options are available:  * ``'raise'``: Raises a ``pytz.NonExistentTimeError`` (the default behavior)
* ``'NaT'``: Replaces nonexistent times with ``NaT``
* ``'shift_forward'``: Shifts nonexistent times forward to the closest real time
* ``'shift_backward'``: Shifts nonexistent times backward to the closest real time

* timedelta object: Shifts nonexistent times by the timedelta duration  .. ipython:: python  dti = pd.date_range(start="2015-03-29 0200", periods=3, freq="H")  # 2:30 is a nonexistent time  Localization of nonexistent times will raise an error by default.  .. code-block:: ipython  In [2]: dti.tz_localize('Europe/Warsaw')  NonExistentTimeError: 2015-03-29 0200  Transform nonexistent times to ``NaT`` or shift the times.  .. ipython:: python  dti  dti.tz_localize("Europe/Warsaw", nonexistent="shift_forward")  dti.tz_localize("Europe/Warsaw", nonexistent="shift_backward")  dti.tz_localize("Europe/Warsaw", nonexistent=pd.Timedelta(1, unit="H"))  dti.tz_localize("Europe/Warsaw", nonexistent="NaT")  .. _timeseries.timezone_series:  Time zone Series operations
~~~  A `Series` with time zone **naive** values is
represented with a dtype of ``datetime64[ns]``.  .. ipython:: python  s_naive = pd.Series(pd.date_range("20130101", periods=3))  s_naive  A `Series` with a time zone **aware** values is
represented with a dtype of ``datetime64[ns, tz]`` where ``tz`` is the time zone  .. ipython:: python  s_aware = pd.Series(pd.date_range("20130101", periods=3, tz="US/Eastern"))  s_aware  Both of these `Series` time zone information
can be manipulated via the ``.dt`` accessor, see `the dt accessor section <basics.dt_accessors>`.  For example, to localize and convert a naive stamp to time zone aware.  .. ipython:: python  s_naive.dt.tz_localize("UTC").dt.tz_convert("US/Eastern")  Time zone information can also be manipulated using the ``astype`` method.
This method can convert between different timezone-aware dtypes.  .. ipython:: python  # convert to a new time zone  s_aware.astype("datetime64[ns, CET]")  .. note::  Using `Series.to_numpy` on a ``Series``, returns a NumPy array of the data.  NumPy does not currently support time zones (even though it is *printing* in the local time zone!),  therefore an object array of Timestamps is returned for time zone aware data:  .. ipython:: python  s_naive.to_numpy()  s_aware.to_numpy()  By converting to an object array of Timestamps, it preserves the time zone  information. For example, when converting back to a Series:  .. ipython:: python  pd.Series(s_aware.to_numpy())  However, if you want an actual NumPy ``datetime64[ns]`` array (with the values  converted to UTC) instead of an array of objects, you can specify the  ``dtype`` argument:  .. ipython:: python  s_aware.to_numpy(dtype="datetime64[ns]")

Basic plotting: ``plot``

We will demonstrate the basics, see the `cookbook<cookbook.plotting>` for
some advanced strategies.  The ``plot`` method on Series and DataFrame is just a simple wrapper around
`plt.plot() <matplotlib.axes.Axes.plot>`:  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  ts = pd.Series(np.random.randn(1000), index=pd.date_range("1/1/2000", periods=1000))  ts = ts.cumsum()  @savefig series_plot_basic.png  ts.plot();  If the index consists of dates, it calls `gcf().autofmt_xdate() <matplotlib.figure.Figure.autofmt_xdate>`
to try to format the x-axis nicely as per above.  On DataFrame, `~DataFrame.plot` is a convenience to plot all of the columns with labels:  .. ipython:: python  plt.close("all")  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index, columns=list("ABCD"))  df = df.cumsum()  plt.figure();  @savefig frame_plot_basic.png  df.plot();  You can plot one column versus another using the ``x`` and ``y`` keywords in
`~DataFrame.plot`:  .. ipython:: python  plt.close("all")  plt.figure()  np.random.seed(123456)  .. ipython:: python  df3 = pd.DataFrame(np.random.randn(1000, 2), columns=["B", "C"]).cumsum()  df3["A"] = pd.Series(list(range(len(df)))  @savefig df_plot_xy.png  df3.plot(x="A", y="B");  .. note::  For more formatting and styling options, see  `formatting <visualization.formatting>` below.  .. ipython:: python  plt.close("all")  .. _visualization.other:

Plotting methods allow for a handful of plot styles other than the
default line plot. These methods can be provided as the ``kind``
keyword argument to `~DataFrame.plot`, and include:  * `'bar' <visualization.barplot>` or `'barh' <visualization.barplot>` for bar plots
* `'hist' <visualization.hist>` for histogram
* `'box' <visualization.box>` for boxplot
* `'kde' <visualization.kde>` or `'density' <visualization.kde>` for density plots
* `'area' <visualization.area_plot>` for area plots
* `'scatter' <visualization.scatter>` for scatter plots
* `'hexbin' <visualization.hexbin>` for hexagonal bin plots
* `'pie' <visualization.pie>` for pie plots  For example, a bar plot can be created the following way:  .. ipython:: python  plt.figure();  @savefig bar_plot_ex.png  df.iloc[5].plot(kind="bar");  You can also create these other plots using the methods ``DataFrame.plot.<kind>`` instead of providing the ``kind`` keyword argument. This makes it easier to discover plot methods and the specific arguments they use:  .. ipython::  In [14]: df = pd.DataFrame()  In [15]: df.plot.<TAB>  # noqa: E225, E999  df.plot.area  df.plot.barh  df.plot.density  df.plot.hist  df.plot.line  df.plot.scatter  df.plot.bar  df.plot.box  df.plot.hexbin  df.plot.kde  df.plot.pie  In addition to these ``kind`` s, there are the `DataFrame.hist() <visualization.hist>`,
and `DataFrame.boxplot() <visualization.box>` methods, which use a separate interface.  Finally, there are several `plotting functions <visualization.tools>` in ``pandas.plotting``
that take a `Series` or `DataFrame` as an argument. These
include:  * `Scatter Matrix <visualization.scatter_matrix>`
* `Andrews Curves <visualization.andrews_curves>`
* `Parallel Coordinates <visualization.parallel_coordinates>`
* `Lag Plot <visualization.lag>`
* `Autocorrelation Plot <visualization.autocorrelation>`
* `Bootstrap Plot <visualization.bootstrap>`
* `RadViz <visualization.radviz>`  Plots may also be adorned with `errorbars <visualization.errorbars>`
or `tables <visualization.table>`.  .. _visualization.barplot:  Bar plots
~~~  For labeled, non-time series data, you may wish to produce a bar plot:  .. ipython:: python  plt.figure();  @savefig bar_plot_ex.png  df.iloc[5].plot.bar();  plt.axhline(0, color="k");  Calling a DataFrame's `plot.bar() <DataFrame.plot.bar>` method produces a multiple
bar plot:  .. ipython:: python  plt.close("all")  plt.figure()  np.random.seed(123456)  .. ipython:: python  df2 = pd.DataFrame(np.random.rand(10, 4), columns=["a", "b", "c", "d"])  @savefig bar_plot_multi_ex.png  df2.plot.bar();  To produce a stacked bar plot, pass ``stacked=True``:  .. ipython:: python  plt.close("all")  plt.figure()  .. ipython:: python  @savefig bar_plot_stacked_ex.png  df2.plot.bar(stacked=True);  To get horizontal bar plots, use the ``barh`` method:  .. ipython:: python  plt.close("all")  plt.figure()  .. ipython:: python  @savefig barh_plot_stacked_ex.png  df2.plot.barh(stacked=True);  .. _visualization.hist:  Histograms
~~~  Histograms can be drawn by using the `DataFrame.plot.hist` and `Series.plot.hist` methods.  .. ipython:: python  df4 = pd.DataFrame(  {  "a": np.random.randn(1000) + 1,  "b": np.random.randn(1000),  "c": np.random.randn(1000) - 1,  },  columns=["a", "b", "c"],  )  plt.figure();  @savefig hist_new.png  df4.plot.hist(alpha=0.5);  .. ipython:: python  plt.close("all")  A histogram can be stacked using ``stacked=True``. Bin size can be changed
using the ``bins`` keyword.  .. ipython:: python  plt.figure();  @savefig hist_new_stacked.png  df4.plot.hist(stacked=True, bins=20);  .. ipython:: python  plt.close("all")  You can pass other keywords supported by matplotlib ``hist``. For example,
horizontal and cumulative histograms can be drawn by

``orientation='horizontal'`` and ``cumulative=True``.  .. ipython:: python  plt.figure();  @savefig hist_new_kwargs.png  df4["a"].plot.hist(orientation="horizontal", cumulative=True);  .. ipython:: python  plt.close("all")  See the `hist <matplotlib.axes.Axes.hist>` method and the
matplotlib hist documentation_ for more.  The existing interface ``DataFrame.hist`` to plot histogram still can be used.  .. ipython:: python  plt.figure();  @savefig hist_plot_ex.png  df["A"].diff().hist();  .. ipython:: python  plt.close("all")  `DataFrame.hist` plots the histograms of the columns on multiple
subplots:  .. ipython:: python  plt.figure();  @savefig frame_hist_ex.png  df.diff().hist(color="k", alpha=0.5, bins=50);  The ``by`` keyword can be specified to plot grouped histograms:  .. ipython:: python  plt.close("all")  plt.figure()  np.random.seed(123456)  .. ipython:: python  data = pd.Series(np.random.randn(1000))  @savefig grouped_hist.png  data.hist(by=np.random.randint(0, 4, 1000), figsize=(6, 4));  .. ipython:: python  plt.close("all")  np.random.seed(123456)  In addition, the ``by`` keyword can also be specified in `DataFrame.plot.hist`.  .. versionchanged:: 1.4.0  .. ipython:: python  data = pd.DataFrame(  {  "a": np.random.choice(["x", "y", "z"], 1000),  "b": np.random.choice(["e", "f", "g"], 1000),  "c": np.random.randn(1000),  "d": np.random.randn(1000) - 1,  },  )  @savefig grouped_hist_by.png  data.plot.hist(by=["a", "b"], figsize=(10, 5));  .. ipython:: python  plt.close("all")  .. _visualization.box:  Box plots
~~~  Boxplot can be drawn calling `Series.plot.box` and `DataFrame.plot.box`,
or `DataFrame.boxplot` to visualize the distribution of values within each column.  For instance, here is a boxplot representing five trials of 10 observations of
a uniform random variable on [0,1).  .. ipython:: python  plt.close("all")  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.rand(10, 5), columns=["A", "B", "C", "D", "E"])  @savefig box_plot_new.png  df.plot.box();  Boxplot can be colorized by passing ``color`` keyword. You can pass a ``dict``
whose keys are ``boxes``, ``whiskers``, ``medians`` and ``caps``.
If some keys are missing in the ``dict``, default colors are used
for the corresponding artists. Also, boxplot has ``sym`` keyword to specify fliers style.  When you pass other type of arguments via ``color`` keyword, it will be directly
passed to matplotlib for all the ``boxes``, ``whiskers``, ``medians`` and ``caps``
colorization.  The colors are applied to every boxes to be drawn. If you want
more complicated colorization, you can get each drawn artists by passing
`return_type <visualization.box.return>`.  .. ipython:: python  color = {  "boxes": "DarkGreen",  "whiskers": "DarkOrange",  "medians": "DarkBlue",  "caps": "Gray",  }  @savefig box_new_colorize.png  df.plot.box(color=color, sym="r+");  .. ipython:: python  plt.close("all")  Also, you can pass other keywords supported by matplotlib ``boxplot``.
For example, horizontal and custom-positioned boxplot can be drawn by
``vert=False`` and ``positions`` keywords.  .. ipython:: python  @savefig box_new_kwargs.png  df.plot.box(vert=False, positions=[1, 4, 5, 6, 8]);  See the `boxplot <matplotlib.axes.Axes.boxplot>` method and the
matplotlib boxplot documentation_ for more.  The existing interface ``DataFrame.boxplot`` to plot boxplot still can be used.  .. ipython:: python  plt.close("all")  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.rand(10, 5))  plt.figure();  @savefig box_plot_ex.png  bp = df.boxplot()  You can create a stratified boxplot using the ``by`` keyword argument to create

groupings.  For instance,  .. ipython:: python  plt.close("all")  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.rand(10, 2), columns=["Col1", "Col2"])  df["X"] = pd.Series(["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"])  plt.figure();  @savefig box_plot_ex2.png  bp = df.boxplot(by="X")  You can also pass a subset of columns to plot, as well as group by multiple
columns:  .. ipython:: python  plt.close("all")  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.rand(10, 3), columns=["Col1", "Col2", "Col3"])  df["X"] = pd.Series(["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"])  df["Y"] = pd.Series(["A", "B", "A", "B", "A", "B", "A", "B", "A", "B"])  plt.figure();  @savefig box_plot_ex3.png  bp = df.boxplot(column=["Col1", "Col2"], by=["X", "Y"])  .. ipython:: python  plt.close("all")  You could also create groupings with `DataFrame.plot.box`, for instance:  .. versionchanged:: 1.4.0  .. ipython:: python  plt.close("all")  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.rand(10, 3), columns=["Col1", "Col2", "Col3"])  df["X"] = pd.Series(["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"])  plt.figure();  @savefig box_plot_ex4.png  bp = df.plot.box(column=["Col1", "Col2"], by="X")  .. ipython:: python  plt.close("all")  .. _visualization.box.return:  In ``boxplot``, the return type can be controlled by the ``return_type``, keyword. The valid choices are ``{"axes", "dict", "both", None}``.
Faceting, created by ``DataFrame.boxplot`` with the ``by``
keyword, will affect the output type as well:  === === ===
``return_type``  Faceted Output type
=== === ===
``None``  No  axes
``None``  Yes  2-D ndarray of axes
``'axes'``  No  axes
``'axes'``  Yes  Series of axes
``'dict'``  No  dict of artists
``'dict'``  Yes  Series of dicts of artists
``'both'``  No  namedtuple
``'both'``  Yes  Series of namedtuples
=== === ===  ``Groupby.boxplot`` always returns a ``Series`` of ``return_type``.  .. ipython:: python  np.random.seed(1234)  df_box = pd.DataFrame(np.random.randn(50, 2))  df_box["g"] = np.random.choice(["A", "B"], size=50)  df_box.loc[df_box["g"] == "B", 1] += 3  @savefig boxplot_groupby.png  bp = df_box.boxplot(by="g")  .. ipython:: python  plt.close("all")  The subplots above are split by the numeric columns first, then the value of
the ``g`` column. Below the subplots are first split by the value of ``g``,
then by the numeric columns.  .. ipython:: python  @savefig groupby_boxplot_vis.png  bp = df_box.groupby("g").boxplot()  .. ipython:: python  plt.close("all")  .. _visualization.area_plot:  Area plot
~~~  You can create area plots with `Series.plot.area` and `DataFrame.plot.area`.
Area plots are stacked by default. To produce stacked area plot, each column must be either all positive or all negative values.  When input data contains ``NaN``, it will be automatically filled by 0. If you want to drop or fill by different values, use `dataframe.dropna` or `dataframe.fillna` before calling ``plot``.  .. ipython:: python  np.random.seed(123456)  plt.figure()  .. ipython:: python  df = pd.DataFrame(np.random.rand(10, 4), columns=["a", "b", "c", "d"])  @savefig area_plot_stacked.png  df.plot.area();  To produce an unstacked plot, pass ``stacked=False``. Alpha value is set to 0.5 unless otherwise specified:  .. ipython:: python  plt.close("all")  plt.figure()  .. ipython:: python  @savefig area_plot_unstacked.png  df.plot.area(stacked=False);  .. _visualization.scatter:  Scatter plot
~~~  Scatter plot can be drawn by using the `DataFrame.plot.scatter` method.
Scatter plot requires numeric columns for the x and y axes.

These can be specified by the ``x`` and ``y`` keywords.  .. ipython:: python  np.random.seed(123456)  plt.close("all")  plt.figure()  .. ipython:: python  df = pd.DataFrame(np.random.rand(50, 4), columns=["a", "b", "c", "d"])  df["species"] = pd.Categorical(  ["setosa"] * 20 + ["versicolor"] * 20 + ["virginica"] * 10  )  @savefig scatter_plot.png  df.plot.scatter(x="a", y="b");  To plot multiple column groups in a single axes, repeat ``plot`` method specifying target ``ax``.
It is recommended to specify ``color`` and ``label`` keywords to distinguish each groups.  .. ipython:: python  ax = df.plot.scatter(x="a", y="b", color="DarkBlue", label="Group 1")  @savefig scatter_plot_repeated.png  df.plot.scatter(x="c", y="d", color="DarkGreen", label="Group 2", ax=ax);  .. ipython:: python  plt.close("all")  The keyword ``c`` may be given as the name of a column to provide colors for
each point:  .. ipython:: python  @savefig scatter_plot_colored.png  df.plot.scatter(x="a", y="b", c="c", s=50);  .. ipython:: python  plt.close("all")  If a categorical column is passed to ``c``, then a discrete colorbar will be produced:  .. versionadded:: 1.3.0  .. ipython:: python  @savefig scatter_plot_categorical.png  df.plot.scatter(x="a", y="b", c="species", cmap="viridis", s=50);  .. ipython:: python  plt.close("all")  You can pass other keywords supported by matplotlib
`scatter <matplotlib.axes.Axes.scatter>`. The example  below shows a
bubble chart using a column of the ``DataFrame`` as the bubble size.  .. ipython:: python  @savefig scatter_plot_bubble.png  df.plot.scatter(x="a", y="b", s=df["c"] * 200);  .. ipython:: python  plt.close("all")  See the `scatter <matplotlib.axes.Axes.scatter>` method and the
matplotlib scatter documentation_ for more.  .. _visualization.hexbin:  Hexagonal bin plot
~~~  You can create hexagonal bin plots with `DataFrame.plot.hexbin`.
Hexbin plots can be a useful alternative to scatter plots if your data are
too dense to plot each point individually.  .. ipython:: python  plt.figure()  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.randn(1000, 2), columns=["a", "b"])  df["b"] = df["b"] + np.arange(1000)  @savefig hexbin_plot.png  df.plot.hexbin(x="a", y="b", gridsize=25);  A useful keyword argument is ``gridsize``; it controls the number of hexagons
in the x-direction, and defaults to 100. A larger ``gridsize`` means more, smaller
bins.  By default, a histogram of the counts around each ``(x, y)`` point is computed.
You can specify alternative aggregations by passing values to the ``C`` and
``reduce_C_function`` arguments. ``C`` specifies the value at each ``(x, y)`` point
and ``reduce_C_function`` is a function of one argument that reduces all the
values in a bin to a single number (e.g. ``mean``, ``max``, ``sum``, ``std``).  In this
example the positions are given by columns ``a`` and ``b``, while the value is
given by column ``z``. The bins are aggregated with NumPy's ``max`` function.  .. ipython:: python  plt.close("all")  plt.figure()  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.randn(1000, 2), columns=["a", "b"])  df["b"] = df["b"] + np.arange(1000)  df["z"] = np.random.uniform(0, 3, 1000)  @savefig hexbin_plot_agg.png  df.plot.hexbin(x="a", y="b", C="z", reduce_C_function=np.max, gridsize=25);  .. ipython:: python  plt.close("all")  See the `hexbin <matplotlib.axes.Axes.hexbin>` method and the
matplotlib hexbin documentation_ for more.  .. _visualization.pie:  Pie plot
~~~  You can create a pie plot with `DataFrame.plot.pie` or `Series.plot.pie`.
If your data includes any ``NaN``, they will be automatically filled with 0.

A ``ValueError`` will be raised if there are any negative values in your data.  .. ipython:: python  np.random.seed(123456)  plt.figure()  .. ipython:: python  series = pd.Series(3 * np.random.rand(4), index=["a", "b", "c", "d"], name="series")  @savefig series_pie_plot.png  series.plot.pie(figsize=(6, 6));  .. ipython:: python  plt.close("all")  For pie plots it's best to use square figures, i.e. a figure aspect ratio 1.
You can create the figure with equal width and height, or force the aspect ratio
to be equal after plotting by calling ``ax.set_aspect('equal')`` on the returned
``axes`` object.  Note that pie plot with `DataFrame` requires that you either specify a
target column by the ``y`` argument or ``subplots=True``. When ``y`` is
specified, pie plot of selected column will be drawn. If ``subplots=True`` is
specified, pie plots for each column are drawn as subplots. A legend will be
drawn in each pie plots by default; specify ``legend=False`` to hide it.  .. ipython:: python  np.random.seed(123456)  plt.figure()  .. ipython:: python  df = pd.DataFrame(  3 * np.random.rand(4, 2), index=["a", "b", "c", "d"], columns=["x", "y"]  )  @savefig df_pie_plot.png  df.plot.pie(subplots=True, figsize=(8, 4));  .. ipython:: python  plt.close("all")  You can use the ``labels`` and ``colors`` keywords to specify the labels and colors of each wedge.  .. warning::  Most pandas plots use the ``label`` and ``color`` arguments (note the lack of "s" on those).  To be consistent with `matplotlib.pyplot.pie` you must use ``labels`` and ``colors``.  If you want to hide wedge labels, specify ``labels=None``.
If ``fontsize`` is specified, the value will be applied to wedge labels.
Also, other keywords supported by `matplotlib.pyplot.pie` can be used.  .. ipython:: python  plt.figure()  .. ipython:: python  @savefig series_pie_plot_options.png  series.plot.pie(  labels=["AA", "BB", "CC", "DD"],  colors=["r", "g", "b", "c"],  autopct="%.2f",  fontsize=20,  figsize=(6, 6),  );  If you pass values whose sum total is less than 1.0 they will be rescaled so that they sum to 1.  .. ipython:: python  plt.close("all")  plt.figure()  .. ipython:: python  series = pd.Series([0.1] * 4, index=["a", "b", "c", "d"], name="series2")  @savefig series_pie_plot_semi.png  series.plot.pie(figsize=(6, 6));  See the matplotlib pie documentation_ for more.  .. ipython:: python  plt.close("all")  .. _visualization.missing_data:

Plotting with missing data

pandas tries to be pragmatic about plotting ``DataFrames`` or ``Series``
that contain missing data. Missing values are dropped, left out, or filled
depending on the plot type.  | Plot Type  | NaN Handling  |
+===+===+
| Line  | Leave gaps at NaNs  |  | Line (stacked) | Fill 0's  |  | Bar  | Fill 0's  |  | Scatter  | Drop NaNs  |  | Histogram  | Drop NaNs (column-wise)  |  | Box  | Drop NaNs (column-wise)  |  | Area  | Fill 0's  |  | KDE  | Drop NaNs (column-wise)  |  | Hexbin  | Drop NaNs  |  | Pie  | Fill 0's  |  If any of these defaults are not what you want, or if you want to be
explicit about how missing values are handled, consider using
`~pandas.DataFrame.fillna` or `~pandas.DataFrame.dropna`
before plotting.  .. _visualization.tools:

These functions can be imported from ``pandas.plotting``
and take a `Series` or `DataFrame` as an argument.  .. _visualization.scatter_matrix:  Scatter matrix plot
~~~  You can create a scatter plot matrix using the
``scatter_matrix`` method in ``pandas.plotting``:  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  from pandas.plotting import scatter_matrix  df = pd.DataFrame(np.random.randn(1000, 4), columns=["a", "b", "c", "d"])  @savefig scatter_matrix_kde.png  scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal="kde");  .. ipython:: python  plt.close("all")  .. _visualization.kde:  Density plot
~~~  You can create density plots using the `Series.plot.kde` and `DataFrame.plot.kde` methods.  .. ipython:: python  plt.figure()  np.random.seed(123456)  .. ipython:: python  ser = pd.Series(np.random.randn(1000))  @savefig kde_plot.png  ser.plot.kde();  .. ipython:: python  plt.close("all")  .. _visualization.andrews_curves:  Andrews curves
~~~  Andrews curves allow one to plot multivariate data as a large number
of curves that are created using the attributes of samples as coefficients
for Fourier series, see the Wikipedia entry_
for more information. By coloring these curves differently for each class
it is possible to visualize data clustering. Curves belonging to samples
of the same class will usually be closer together and form larger structures.  **Note**: The "Iris" dataset is available here_.  .. ipython:: python  from pandas.plotting import andrews_curves  data = pd.read_csv("data/iris.data")  plt.figure();  @savefig andrews_curves.png  andrews_curves(data, "Name");  .. _visualization.parallel_coordinates:  Parallel coordinates
~~~  Parallel coordinates is a plotting technique for plotting multivariate data,
see the Wikipedia entry_
for an introduction.
Parallel coordinates allows one to see clusters in data and to estimate other statistics visually.
Using parallel coordinates points are represented as connected line segments.
Each vertical line represents one attribute. One set of connected line segments
represents one data point. Points that tend to cluster will appear closer together.  .. ipython:: python  from pandas.plotting import parallel_coordinates  data = pd.read_csv("data/iris.data")  plt.figure();  @savefig parallel_coordinates.png  parallel_coordinates(data, "Name");  .. ipython:: python  plt.close("all")  .. _visualization.lag:  Lag plot
~~~  Lag plots are used to check if a data set or time series is random. Random
data should not exhibit any structure in the lag plot. Non-random structure
implies that the underlying data are not random. The ``lag`` argument may
be passed, and when ``lag=1`` the plot is essentially ``data[:-1]`` vs.
``data[1:]``.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  from pandas.plotting import lag_plot  plt.figure();  spacing = np.linspace(-99 * np.pi, 99 * np.pi, num=1000)  data = pd.Series(0.1 * np.random.rand(1000) + 0.9 * np.sin(spacing))  @savefig lag_plot.png  lag_plot(data);  .. ipython:: python  plt.close("all")  .. _visualization.autocorrelation:  Autocorrelation plot
~~~  Autocorrelation plots are often used for checking randomness in time series.
This is done by computing autocorrelations for data values at varying time lags.
If time series is random, such autocorrelations should be near zero for any and
all time-lag separations. If time series is non-random then one or more of the
autocorrelations will be significantly non-zero. The horizontal lines displayed
in the plot correspond to 95% and 99% confidence bands. The dashed line is 99%
confidence band. See the
Wikipedia entry_ for more about

autocorrelation plots.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  from pandas.plotting import autocorrelation_plot  plt.figure();  spacing = np.linspace(-9 * np.pi, 9 * np.pi, num=1000)  data = pd.Series(0.7 * np.random.rand(1000) + 0.3 * np.sin(spacing))  @savefig autocorrelation_plot.png  autocorrelation_plot(data);  .. ipython:: python  plt.close("all")  .. _visualization.bootstrap:  Bootstrap plot
~~~  Bootstrap plots are used to visually assess the uncertainty of a statistic, such
as mean, median, midrange, etc. A random subset of a specified size is selected
from a data set, the statistic in question is computed for this subset and the
process is repeated a specified number of times. Resulting plots and histograms
are what constitutes the bootstrap plot.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  from pandas.plotting import bootstrap_plot  data = pd.Series(np.random.rand(1000))  @savefig bootstrap_plot.png  bootstrap_plot(data, size=50, samples=500, color="grey");  .. ipython:: python  plt.close("all")  .. _visualization.radviz:  RadViz
~~~  RadViz is a way of visualizing multi-variate data. It is based on a simple
spring tension minimization algorithm. Basically you set up a bunch of points in
a plane. In our case they are equally spaced on a unit circle. Each point
represents a single attribute. You then pretend that each sample in the data set
is attached to each of these points by a spring, the stiffness of which is
proportional to the numerical value of that attribute (they are normalized to
unit interval). The point in the plane, where our sample settles to (where the
forces acting on our sample are at an equilibrium) is where a dot representing
our sample will be drawn. Depending on which class that sample belongs it will
be colored differently.
See the R package Radviz_
for more information.  **Note**: The "Iris" dataset is available here_.  .. ipython:: python  from pandas.plotting import radviz  data = pd.read_csv("data/iris.data")  plt.figure();  @savefig radviz.png  radviz(data, "Name");  .. ipython:: python  plt.close("all")  .. _visualization.formatting:

Setting the plot style
~~~  From version 1.5 and up, matplotlib offers a range of pre-configured plotting styles. Setting the
style can be used to easily give plots the general look that you want.
Setting the style is as easy as calling ``matplotlib.style.use(my_plot_style)`` before
creating your plot. For example you could write ``matplotlib.style.use('ggplot')`` for ggplot-style
plots.  You can see the various available style names at ``matplotlib.style.available`` and it's very
easy to try them out.  General plot style arguments
~~~  Most plotting methods have a set of keyword arguments that control the
layout and formatting of the returned plot:  .. ipython:: python  plt.figure();  @savefig series_plot_basic2.png  ts.plot(style="k--", label="Series");  .. ipython:: python  plt.close("all")  For each kind of plot (e.g. ``line``, ``bar``, ``scatter``) any additional arguments
keywords are passed along to the corresponding matplotlib function
(`ax.plot() <matplotlib.axes.Axes.plot>`,
`ax.bar() <matplotlib.axes.Axes.bar>`,
`ax.scatter() <matplotlib.axes.Axes.scatter>`). These can be used
to control additional styling, beyond what pandas provides.  Controlling the legend
~~~  You may set the ``legend`` argument to ``False`` to hide the legend, which is
shown by default.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index, columns=list("ABCD"))  df = df.cumsum()  @savefig frame_plot_basic_noleg.png  df.plot(legend=False);  .. ipython:: python  plt.close("all")  Controlling the labels
~~~  .. versionadded:: 1.1.0  You may set the ``xlabel`` and ``ylabel`` arguments to give the plot custom labels
for x and y axis. By default, pandas will pick up index name as xlabel, while leaving
it empty for ylabel.  .. ipython:: python  plt.figure();  .. ipython:: python  df.plot();  @savefig plot_xlabel_ylabel.png  df.plot(xlabel="new x", ylabel="new y");  .. ipython:: python  plt.close("all")  Scales
~~~  You may pass ``logy`` to get a log-scale Y axis.  .. ipython:: python  plt.figure()  np.random.seed(123456)  .. ipython:: python  ts = pd.Series(np.random.randn(1000), index=pd.date_range("1/1/2000", periods=1000))  ts = np.exp(ts.cumsum())  @savefig series_plot_logy.png  ts.plot(logy=True);  .. ipython:: python  plt.close("all")  See also the ``logx`` and ``loglog`` keyword arguments.  Plotting on a secondary y-axis
~~~  To plot data on a secondary y-axis, use the ``secondary_y`` keyword:  .. ipython:: python  plt.figure()  .. ipython:: python  df["A"].plot();  @savefig series_plot_secondary_y.png  df["B"].plot(secondary_y=True, style="g");  .. ipython:: python  plt.close("all")  To plot some columns in a ``DataFrame``, give the column names to the ``secondary_y``
keyword:  .. ipython:: python  plt.figure();  ax = df.plot(secondary_y=["A", "B"])  ax.set_ylabel("CD scale");  @savefig frame_plot_secondary_y.png  ax.right_ax.set_ylabel("AB scale");  .. ipython:: python  plt.close("all")  Note that the columns plotted on the secondary y-axis is automatically marked
with "(right)" in the legend. To turn off the automatic marking, use the
``mark_right=False`` keyword:  .. ipython:: python  plt.figure();  @savefig frame_plot_secondary_y_no_right.png  df.plot(secondary_y=["A", "B"], mark_right=False);  .. ipython:: python  plt.close("all")  .. _plotting.formatters:  Custom formatters for timeseries plots
~~~  .. versionchanged:: 1.0.0  pandas provides custom formatters for timeseries plots. These change the
formatting of the axis labels for dates and times. By default,
the custom formatters are applied only to plots created by pandas with
`DataFrame.plot` or `Series.plot`. To have them apply to all
plots, including those made by matplotlib, set the option
``pd.options.plotting.matplotlib.register_converters = True`` or use
`pandas.plotting.register_matplotlib_converters`.  Suppressing tick resolution adjustment
~~~  pandas includes automatic tick resolution adjustment for regular frequency

time-series data. For limited cases where pandas cannot infer the frequency
information (e.g., in an externally created ``twinx``), you can choose to
suppress this behavior for alignment purposes.  Here is the default behavior, notice how the x-axis tick labeling is performed:  .. ipython:: python  plt.figure();  @savefig ser_plot_suppress.png  df["A"].plot();  .. ipython:: python  plt.close("all")  Using the ``x_compat`` parameter, you can suppress this behavior:  .. ipython:: python  plt.figure();  @savefig ser_plot_suppress_parm.png  df["A"].plot(x_compat=True);  .. ipython:: python  plt.close("all")  If you have more than one plot that needs to be suppressed, the ``use`` method
in ``pandas.plotting.plot_params`` can be used in a ``with`` statement:  .. ipython:: python  plt.figure();  @savefig ser_plot_suppress_context.png  with pd.plotting.plot_params.use("x_compat", True):  df["A"].plot(color="r")  df["B"].plot(color="g")  df["C"].plot(color="b")  .. ipython:: python  plt.close("all")  Automatic date tick adjustment
~~~  ``TimedeltaIndex`` now uses the native matplotlib
tick locator methods, it is useful to call the automatic
date tick adjustment from matplotlib for figures whose ticklabels overlap.  See the `autofmt_xdate <matplotlib.figure.autofmt_xdate>` method and the
matplotlib documentation_ for more.  Subplots
~~~  Each ``Series`` in a ``DataFrame`` can be plotted on a different axis
with the ``subplots`` keyword:  .. ipython:: python  @savefig frame_plot_subplots.png  df.plot(subplots=True, figsize=(6, 6));  .. ipython:: python  plt.close("all")  Using layout and targeting multiple axes
~~~  The layout of subplots can be specified by the ``layout`` keyword. It can accept
``(rows, columns)``. The ``layout`` keyword can be used in
``hist`` and ``boxplot`` also. If the input is invalid, a ``ValueError`` will be raised.  The number of axes which can be contained by rows x columns specified by ``layout`` must be
larger than the number of required subplots. If layout can contain more axes than required,
blank axes are not drawn. Similar to a NumPy array's ``reshape`` method, you
can use ``-1`` for one dimension to automatically calculate the number of rows
or columns needed, given the other.  .. ipython:: python  @savefig frame_plot_subplots_layout.png  df.plot(subplots=True, layout=(2, 3), figsize=(6, 6), sharex=False);  .. ipython:: python  plt.close("all")  The above example is identical to using:  .. ipython:: python  df.plot(subplots=True, layout=(2, -1), figsize=(6, 6), sharex=False);  .. ipython:: python  plt.close("all")  The required number of columns (3) is inferred from the number of series to plot
and the given number of rows (2).  You can pass multiple axes created beforehand as list-like via ``ax`` keyword.
This allows more complicated layouts.
The passed axes must be the same number as the subplots being drawn.  When multiple axes are passed via the ``ax`` keyword, ``layout``, ``sharex`` and ``sharey`` keywords
don't affect to the output. You should explicitly pass ``sharex=False`` and ``sharey=False``,

otherwise you will see a warning.  .. ipython:: python  fig, axes = plt.subplots(4, 4, figsize=(9, 9))  plt.subplots_adjust(wspace=0.5, hspace=0.5)  target1 = [axes[0][0], axes[1][1], axes[2][2], axes[3][3]]  target2 = [axes[3][0], axes[2][1], axes[1][2], axes[0][3]]  df.plot(subplots=True, ax=target1, legend=False, sharex=False, sharey=False);  @savefig frame_plot_subplots_multi_ax.png  (-df).plot(subplots=True, ax=target2, legend=False, sharex=False, sharey=False);  .. ipython:: python  plt.close("all")  Another option is passing an ``ax`` argument to `Series.plot` to plot on a particular axis:  .. ipython:: python  np.random.seed(123456)  ts = pd.Series(np.random.randn(1000), index=pd.date_range("1/1/2000", periods=1000))  ts = ts.cumsum()  df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index, columns=list("ABCD"))  df = df.cumsum()  .. ipython:: python  plt.close("all")  .. ipython:: python  fig, axes = plt.subplots(nrows=2, ncols=2)  plt.subplots_adjust(wspace=0.2, hspace=0.5)  df["A"].plot(ax=axes[0, 0]);  axes[0, 0].set_title("A");  df["B"].plot(ax=axes[0, 1]);  axes[0, 1].set_title("B");  df["C"].plot(ax=axes[1, 0]);  axes[1, 0].set_title("C");  df["D"].plot(ax=axes[1, 1]);  @savefig series_plot_multi.png  axes[1, 1].set_title("D");  .. ipython:: python  plt.close("all")  .. _visualization.errorbars:  Plotting with error bars
~~~  Plotting with error bars is supported in `DataFrame.plot` and `Series.plot`.  Horizontal and vertical error bars can be supplied to the ``xerr`` and ``yerr`` keyword arguments to `~DataFrame.plot()`. The error values can be specified using a variety of formats:  * As a `DataFrame` or ``dict`` of errors with column names matching the ``columns`` attribute of the plotting `DataFrame` or matching the ``name`` attribute of the `Series`.
* As a ``str`` indicating which of the columns of plotting `DataFrame` contain the error values.
* As raw values (``list``, ``tuple``, or ``np.ndarray``). Must be the same length as the plotting `DataFrame`/`Series`.  Here is an example of one way to easily plot group means with standard deviations from the raw data.  .. ipython:: python  # Generate the data  ix3 = pd.MultiIndex.from_arrays(  [  ["a", "a", "a", "a", "a", "b", "b", "b", "b", "b"],  ["foo", "foo", "foo", "bar", "bar", "foo", "foo", "bar", "bar", "bar"],  ],  names=["letter", "word"],  )  df3 = pd.DataFrame(  {  "data1": [9, 3, 2, 4, 3, 2, 4, 6, 3, 2],  "data2": [9, 6, 5, 7, 5, 4, 5, 6, 5, 1],  },  index=ix3,  )  # Group by index labels and take the means and standard deviations  # for each group  gp3 = df3.groupby(level=("letter", "word"))  means = gp3.mean()  errors = gp3.std()  means  errors  # Plot  fig, ax = plt.subplots()  @savefig errorbar_example.png  means.plot.bar(yerr=errors, ax=ax, capsize=4, rot=0);  .. ipython:: python  plt.close("all")  Asymmetrical error bars are also supported, however raw error values must be provided in this case. For a ``N`` length `Series`, a ``2xN`` array should be provided indicating lower and upper (or left and right) errors. For a ``MxN`` `DataFrame`, asymmetrical errors should be in a ``Mx2xN`` array.  Here is an example of one way to plot the min/max range using asymmetrical error bars.  .. ipython:: python  mins = gp3.min()  maxs = gp3.max()  # errors should be positive, and defined in the order of lower, upper  errors = [[means[c] - mins[c], maxs[c] - means[c]] for c in df3.columns]  # Plot  fig, ax = plt.subplots()  @savefig errorbar_asymmetrical_example.png  means.plot.bar(yerr=errors, ax=ax, capsize=4, rot=0);  .. ipython:: python  plt.close("all")  .. _visualization.table:  Plotting tables

~~~  Plotting with matplotlib table is now supported in  `DataFrame.plot` and `Series.plot` with a ``table`` keyword. The ``table`` keyword can accept ``bool``, `DataFrame` or `Series`. The simple way to draw a table is to specify ``table=True``. Data will be transposed to meet matplotlib's default layout.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  fig, ax = plt.subplots(1, 1, figsize=(7, 6.5))  df = pd.DataFrame(np.random.rand(5, 3), columns=["a", "b", "c"])  ax.xaxis.tick_top()  # Display x-axis ticks on top.  @savefig line_plot_table_true.png  df.plot(table=True, ax=ax);  .. ipython:: python  plt.close("all")  Also, you can pass a different `DataFrame` or `Series` to the
``table`` keyword. The data will be drawn as displayed in print method
(not transposed automatically). If required, it should be transposed manually
as seen in the example below.  .. ipython:: python  fig, ax = plt.subplots(1, 1, figsize=(7, 6.75))  ax.xaxis.tick_top()  # Display x-axis ticks on top.  @savefig line_plot_table_data.png  df.plot(table=np.round(df.T, 2), ax=ax);  .. ipython:: python  plt.close("all")  There also exists a helper function ``pandas.plotting.table``, which creates a
table from `DataFrame` or `Series`, and adds it to an
``matplotlib.Axes`` instance. This function can accept keywords which the
matplotlib table_ has.  .. ipython:: python  from pandas.plotting import table  fig, ax = plt.subplots(1, 1)  table(ax, np.round(df.describe(), 2), loc="upper right", colWidths=[0.2, 0.2, 0.2]);  @savefig line_plot_table_describe.png  df.plot(ax=ax, ylim=(0, 2), legend=None);  .. ipython:: python  plt.close("all")  **Note**: You can get table instances on the axes using `axes.tables`` property for further decorations. See the `matplotlib table documentation_ for more.  .. _visualization.colormaps:  Colormaps
~~~  A potential issue when plotting a large number of columns is that it can be
difficult to distinguish some series due to repetition in the default colors. To
remedy this, ``DataFrame`` plotting supports the use of the ``colormap`` argument,
which accepts either a Matplotlib colormap_
or a string that is a name of a colormap registered with Matplotlib. A
visualization of the default matplotlib colormaps is available `here
<https://matplotlib.org/stable/gallery/color/colormap_reference.html>`__.  As matplotlib does not directly support colormaps for line-based plots, the
colors are selected based on an even spacing determined by the number of columns
in the ``DataFrame``. There is no consideration made for background color, so some
colormaps will produce lines that are not easily visible.  To use the cubehelix colormap, we can pass ``colormap='cubehelix'``.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  df = pd.DataFrame(np.random.randn(1000, 10), index=ts.index)  df = df.cumsum()  plt.figure();  @savefig cubehelix.png  df.plot(colormap="cubehelix");  .. ipython:: python  plt.close("all")  Alternatively, we can pass the colormap itself:  .. ipython:: python  from matplotlib import cm  plt.figure();  @savefig cubehelix_cm.png  df.plot(colormap=cm.cubehelix);  .. ipython:: python  plt.close("all")  Colormaps can also be used other plot types, like bar charts:  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  dd = pd.DataFrame(np.random.randn(10, 10)).applymap(abs)  dd = dd.cumsum()  plt.figure();  @savefig greens.png  dd.plot.bar(colormap="Greens");  .. ipython:: python  plt.close("all")  Parallel coordinates charts:  .. ipython:: python  plt.figure();  @savefig parallel_gist_rainbow.png  parallel_coordinates(data, "Name", colormap="gist_rainbow");  .. ipython:: python  plt.close("all")  Andrews curves charts:  .. ipython:: python  plt.figure();  @savefig andrews_curve_winter.png  andrews_curves(data, "Name", colormap="winter");  .. ipython:: python  plt.close("all")

Plotting directly with Matplotlib

In some situations it may still be preferable or necessary to prepare plots
directly with matplotlib, for instance when a certain type of plot or
customization is not (yet) supported by pandas. ``Series`` and ``DataFrame``
objects behave like arrays and can therefore be passed directly to
matplotlib functions without explicit casts.  pandas also automatically registers formatters and locators that recognize date
indices, thereby extending date and time support to practically all plot types
available in matplotlib. Although this formatting does not provide the same
level of refinement you would get when plotting via pandas, it can be faster
when plotting a large number of points.  .. ipython:: python  np.random.seed(123456)  .. ipython:: python  price = pd.Series(  np.random.randn(150).cumsum(),  index=pd.date_range("2000-1-1", periods=150, freq="B"),  )  ma = price.rolling(20).mean()  mstd = price.rolling(20).std()  plt.figure();  plt.plot(price.index, price, "k");  plt.plot(ma.index, ma, "b");  @savefig bollinger.png  plt.fill_between(mstd.index, ma - 2 * mstd, ma + 2 * mstd, color="b", alpha=0.2);  .. ipython:: python  plt.close("all")

Starting in version 0.25, pandas can be extended with third-party plotting backends. The
one based on Matplotlib.  This can be done by passing 'backend.module' as the argument ``backend`` in ``plot``
function. For example:  .. code-block:: python  >>> Series([1, 2, 3]).plot(backend="backend.module")  Alternatively, you can also set this option globally, do you don't need to specify
the keyword in each ``plot`` call. For example:  .. code-block:: python  >>> pd.set_option("plotting.backend", "backend.module")  >>> pd.Series([1, 2, 3]).plot()  Or:  .. code-block:: python  >>> pd.options.plotting.backend = "backend.module"  >>> pd.Series([1, 2, 3]).plot()  This would be more or less equivalent to:  .. code-block:: python  >>> import backend.module  >>> backend.module.plot(pd.Series([1, 2, 3]))  The backend module can then use other visualization tools (Bokeh, Altair, hvplot,...)
to generate the plots. Some libraries implementing a backend for pandas are listed
on the ecosystem `ecosystem.visualization` page.  Developers guide can be found at
https://pandas.pydata.org/docs/dev/development/extending.html#plotting-backends

pandas supports 4 types of windowing operations:  #. Rolling window: Generic fixed or variable sliding window over the values.
#. Weighted window: Weighted, non-rectangular window supplied by the ``scipy.signal`` library.
#. Expanding window: Accumulating window over the values.
#. Exponentially Weighted window: Accumulating and exponentially weighted window over the values.  ===  ===  ===  ===  ===  ===  ===
Concept  Method  Returned Object  Supports time-based windows  Supports chained groupby  Supports table method  Supports online operations
===  ===  ===  ===  ===  ===  ===
Rolling window  ``rolling``  ``Rolling``  Yes  Yes  Yes (as of version 1.3)  No
Weighted window  ``rolling``  ``Window``  No  No  No  No
Expanding window  ``expanding``  ``Expanding``  No  Yes  Yes (as of version 1.3)  No
Exponentially Weighted window  ``ewm``  ``ExponentialMovingWindow``  No  Yes (as of version 1.2)  No  Yes (as of version 1.3)
===  ===  ===  ===  ===  ===  ===  As noted above, some operations support specifying a window based on a time offset:  .. ipython:: python  s = pd.Series(range(5), index=pd.date_range('2020-01-01', periods=5, freq='1D'))  s.rolling(window='2D').sum()  Additionally, some methods support chaining a ``groupby`` operation with a windowing operation
which will first group the data by the specified keys and then perform a windowing operation per group.  .. ipython:: python  df = pd.DataFrame({'A': ['a', 'b', 'a', 'b', 'a'], 'B': range(5)})  df.groupby('A').expanding().sum()  .. note::  Windowing operations currently only support numeric data (integer and float)  and will always return ``float64`` values.  .. warning::  Some windowing aggregation, ``mean``, ``sum``, ``var`` and ``std`` methods may suffer from numerical  imprecision due to the underlying windowing algorithms accumulating sums. When values differ  with magnitude `1/np.finfo(np.double).eps` this results in truncation. It must be  noted, that large values may have an impact on windows, which do not include these values. `Kahan summation  <https://en.wikipedia.org/wiki/Kahan_summation_algorithm>`__ is used  to compute the rolling sums to preserve accuracy as much as possible.  .. versionadded:: 1.3.0  Some windowing operations also support the ``method='table'`` option in the constructor which
performs the windowing operation over an entire `DataFrame` instead of a single column or row at a time.
This can provide a useful performance benefit for a `DataFrame` with many columns or rows
(with the corresponding ``axis`` argument) or the ability to utilize other columns during the windowing
operation. The ``method='table'`` option can only be used if ``engine='numba'`` is specified
in the corresponding method call.  For example, a weighted mean_ calculation can
be calculated with `~Rolling.apply` by specifying a separate column of weights.  .. ipython:: python  def weighted_mean(x):  arr = np.ones((1, x.shape[1]))  arr[:, :2] = (x[:, :2] * x[:, 2]).sum(axis=0) / x[:, 2].sum()  return arr  df = pd.DataFrame([[1, 2, 0.6], [2, 3, 0.4], [3, 4, 0.2], [4, 5, 0.7]])  df.rolling(2, method="table", min_periods=0).apply(weighted_mean, raw=True, engine="numba")  # noqa:E501  .. versionadded:: 1.3  Some windowing operations also support an ``online`` method after constructing a windowing object
which returns a new object that supports passing in new `DataFrame` or `Series` objects
to continue the windowing calculation with the new values (i.e. online calculations).  The methods on this new windowing objects must call the aggregation method first to "prime" the initial
state of the online calculation. Then, new `DataFrame` or `Series` objects can be passed in

the ``update`` argument to continue the windowing calculation.  .. ipython:: python  df = pd.DataFrame([[1, 2, 0.6], [2, 3, 0.4], [3, 4, 0.2], [4, 5, 0.7]])  df.ewm(0.5).mean()  .. ipython:: python  online_ewm = df.head(2).ewm(0.5).online()  online_ewm.mean()  online_ewm.mean(update=df.tail(1))  All windowing operations support a ``min_periods`` argument that dictates the minimum amount of
non-``np.nan`` values a window must have; otherwise, the resulting value is ``np.nan``.
``min_periods`` defaults to 1 for time-based windows and ``window`` for fixed windows  .. ipython:: python  s = pd.Series([np.nan, 1, 2, np.nan, np.nan, 3])  s.rolling(window=3, min_periods=1).sum()  s.rolling(window=3, min_periods=2).sum()  # Equivalent to min_periods=3  s.rolling(window=3, min_periods=None).sum()  Additionally, all windowing operations supports the ``aggregate`` method for returning a result
of multiple aggregations applied to a window.  .. ipython:: python  df = pd.DataFrame({"A": range(5), "B": range(10, 15)})  df.expanding().agg([np.sum, np.mean, np.std])  .. _window.generic:

Generic rolling windows support specifying windows as a fixed number of observations or variable
number of observations based on an offset. If a time based offset is provided, the corresponding
time based index must be monotonic.  .. ipython:: python  times = ['2020-01-01', '2020-01-03', '2020-01-04', '2020-01-05', '2020-01-29']  s = pd.Series(range(5), index=pd.DatetimeIndex(times))  s  # Window with 2 observations  s.rolling(window=2).sum()  # Window with 2 days worth of observations  s.rolling(window='2D').sum()  For all supported aggregation functions, see `api.functions_rolling`.  .. _window.center:  Centering windows
~~~  By default the labels are set to the right edge of the window, but a
``center`` keyword is available so the labels can be set at the center.  .. ipython:: python  s = pd.Series(range(10))  s.rolling(window=5).mean()  s.rolling(window=5, center=True).mean()  This can also be applied to datetime-like indices.  .. versionadded:: 1.3.0  .. ipython:: python  df = pd.DataFrame(  {"A": [0, 1, 2, 3, 4]}, index=pd.date_range("2020", periods=5, freq="1D")  )  df  df.rolling("2D", center=False).mean()  df.rolling("2D", center=True).mean()  .. _window.endpoints:  Rolling window endpoints
~~~  The inclusion of the interval endpoints in rolling window calculations can be specified with the ``closed``
parameter:  ===  ===
Value  Behavior
===  ===
``'right'``  close right endpoint
``'left'``  close left endpoint
``'both'``  close both endpoints
``'neither'``  open endpoints
===  ===  For example, having the right endpoint open is useful in many problems that require that there is no contamination
from present information back to past information. This allows the rolling window to compute statistics
"up to that point in time", but not including that point in time.  .. ipython:: python  df = pd.DataFrame(  {"x": 1},  index=[  pd.Timestamp("20130101 0901"),  pd.Timestamp("20130101 0902"),  pd.Timestamp("20130101 0903"),  pd.Timestamp("20130101 0904"),  pd.Timestamp("20130101 0906"),  ],  )  df["right"] = df.rolling("2s", closed="right").x.sum()  # default  df["both"] = df.rolling("2s", closed="both").x.sum()  df["left"] = df.rolling("2s", closed="left").x.sum()  df["neither"] = df.rolling("2s", closed="neither").x.sum()  df  .. _window.custom_rolling_window:  Custom window rolling
~~~  .. versionadded:: 1.0  In addition to accepting an integer or offset as a ``window`` argument, ``rolling`` also accepts
a ``BaseIndexer`` subclass that allows a user to define a custom method for calculating window bounds.
The ``BaseIndexer`` subclass will need to define a ``get_window_bounds`` method that returns
a tuple of two arrays, the first being the starting indices of the windows and second being the
ending indices of the windows. Additionally, ``num_values``, ``min_periods``, ``center``, ``closed``
and will automatically be passed to ``get_window_bounds`` and the defined method must
always accept these arguments.  For example, if we have the following `DataFrame`  .. ipython:: python  use_expanding = [True, False, True, False, True]  use_expanding  df = pd.DataFrame({"values": range(5)})  df  and we want to use an expanding window where ``use_expanding`` is ``True`` otherwise a window of size

1, we can create the following ``BaseIndexer`` subclass:  .. code-block:: ipython  In [2]: from pandas.api.indexers import BaseIndexer  In [3]: class CustomIndexer(BaseIndexer):  ...:  def get_window_bounds(self, num_values, min_periods, center, closed):  ...:  start = np.empty(num_values, dtype=np.int64)  ...:  end = np.empty(num_values, dtype=np.int64)  ...:  for i in range(num_values):  ...:  if self.use_expanding[i]:  ...:  start[i] = 0  ...:  end[i] = i + 1  ...:  else:  ...:  start[i] = i  ...:  end[i] = i + self.window_size  ...:  return start, end  In [4]: indexer = CustomIndexer(window_size=1, use_expanding=use_expanding)  In [5]: df.rolling(indexer).sum()  Out[5]:  values  0  0.0  1  1.0  2  3.0  3  3.0  4  10.0  You can view other examples of `BaseIndexer`` subclasses `here_  .. versionadded:: 1.1  One subclass of note within those examples is the ``VariableOffsetWindowIndexer`` that allows
rolling operations over a non-fixed offset like a ``BusinessDay``.  .. ipython:: python  from pandas.api.indexers import VariableOffsetWindowIndexer  df = pd.DataFrame(range(10), index=pd.date_range("2020", periods=10))  offset = pd.offsets.BDay(1)  indexer = VariableOffsetWindowIndexer(index=df.index, offset=offset)  df  df.rolling(indexer).sum()  For some problems knowledge of the future is available for analysis. For example, this occurs when
each data point is a full time series read from an experiment, and the task is to extract underlying
conditions. In these cases it can be useful to perform forward-looking rolling window computations.
`FixedForwardWindowIndexer <pandas.api.indexers.FixedForwardWindowIndexer>` class is available for this purpose.
This `BaseIndexer <pandas.api.indexers.BaseIndexer>` subclass implements a closed fixed-width
forward-looking rolling window, and we can use it as follows:  .. ipython:: python  from pandas.api.indexers import FixedForwardWindowIndexer  indexer = FixedForwardWindowIndexer(window_size=2)  df.rolling(indexer, min_periods=1).sum()  We can also achieve this by using slicing, applying rolling aggregation, and then flipping the result as shown in example below:  .. ipython:: python  df = pd.DataFrame(  data=[  [pd.Timestamp("2018-01-01 000"), 100],  [pd.Timestamp("2018-01-01 0001"), 101],  [pd.Timestamp("2018-01-01 0003"), 103],  [pd.Timestamp("2018-01-01 0004"), 111],  ],  columns=["time", "value"],  ).set_index("time")  df  reversed_df = df[::-1].rolling("2s").sum()[::-1]  reversed_df  .. _window.rolling_apply:  Rolling apply
~~~  The `~Rolling.apply` function takes an extra ``func`` argument and performs
generic rolling computations. The ``func`` argument should be a single function
that produces a single value from an ndarray input. ``raw`` specifies whether
the windows are cast as `Series` objects (``raw=False``) or ndarray objects (``raw=True``).  .. ipython:: python  def mad(x):  return np.fabs(x - x.mean()).mean()  s = pd.Series(range(10))  s.rolling(window=4).apply(mad, raw=True)  .. _window.numba_engine:  Numba engine
~~~  .. versionadded:: 1.0  Additionally, ~Rolling.apply` can leverage `Numba_
if installed as an optional dependency. The apply aggregation can be executed using Numba by specifying
``engine='numba'`` and ``engine_kwargs`` arguments (``raw`` must also be set to ``True``).
See `enhancing performance with Numba <enhancingperf.numba>` for general usage of the arguments and performance considerations.  Numba will be applied in potentially two routines:  #. If `func`` is a standard Python function, the engine will `JIT_ the passed function. ``func`` can also be a JITed function in which case the engine will not JIT the function again.
#. The engine will JIT the for loop where the apply function is applied to each window.  The ``engine_kwargs`` argument is a dictionary of keyword arguments that will be passed into the
numba.jit decorator_.
These keyword arguments will be applied to *both* the passed function (if a standard Python function)

and the apply for loop over each window.  .. versionadded:: 1.3.0  ``mean``, ``median``, ``max``, ``min``, and ``sum`` also support the ``engine`` and ``engine_kwargs`` arguments.  .. _window.cov_corr:  Binary window functions
~~~  `~Rolling.cov` and `~Rolling.corr` can compute moving window statistics about
two `Series` or any combination of `DataFrame`/`Series` or
`DataFrame`/`DataFrame`. Here is the behavior in each case:  * two `Series`: compute the statistic for the pairing.
* `DataFrame`/`Series`: compute the statistics for each column of the DataFrame  with the passed Series, thus returning a DataFrame.
* `DataFrame`/`DataFrame`: by default compute the statistic for matching column  names, returning a DataFrame. If the keyword argument ``pairwise=True`` is  passed then computes the statistic for each pair of columns, returning a `DataFrame` with a  `MultiIndex` whose values are the dates in question (see `the next section  <window.corr_pairwise>`).  For example:  .. ipython:: python  df = pd.DataFrame(  np.random.randn(10, 4),  index=pd.date_range("2020-01-01", periods=10),  columns=["A", "B", "C", "D"],  )  df = df.cumsum()  df2 = df[:4]  df2.rolling(window=2).corr(df2["B"])  .. _window.corr_pairwise:  Computing rolling pairwise covariances and correlations
~~~  In financial data analysis and other fields it's common to compute covariance
and correlation matrices for a collection of time series. Often one is also
interested in moving-window covariance and correlation matrices. This can be
done by passing the ``pairwise`` keyword argument, which in the case of
`DataFrame` inputs will yield a MultiIndexed `DataFrame` whose ``index`` are the dates in
question. In the case of a single DataFrame argument the ``pairwise`` argument
can even be omitted:  .. note::  Missing values are ignored and each entry is computed using the pairwise  complete observations.  Assuming the missing data are missing at random this results in an estimate  for the covariance matrix which is unbiased. However, for many applications  this estimate may not be acceptable because the estimated covariance matrix  is not guaranteed to be positive semi-definite. This could lead to  estimated correlations having absolute values which are greater than one,  and/or a non-invertible covariance matrix. See `Estimation of covariance  matrices <https://en.wikipedia.org/w/index.php?title=Estimation_of_covariance_matrices>`_  for more details.  .. ipython:: python  covs = (  df[["B", "C", "D"]]  .rolling(window=4)  .cov(df[["A", "B", "C"]], pairwise=True)  )  covs  .. _window.weighted:

Weighted window

The ``win_type`` argument in ``.rolling`` generates a weighted windows that are commonly used in filtering
and spectral estimation. ``win_type`` must be string that corresponds to a `scipy.signal window function
<https://docs.scipy.org/doc/scipy/reference/signal.windows.html#module-scipy.signal.windows>`__.
Scipy must be installed in order to use these windows, and supplementary arguments
that the Scipy window methods take must be specified in the aggregation function.  .. ipython:: python  s = pd.Series(range(10))  s.rolling(window=5).mean()  s.rolling(window=5, win_type="triang").mean()  # Supplementary Scipy arguments passed in the aggregation function  s.rolling(window=5, win_type="gaussian").mean(std=0.1)  For all supported aggregation functions, see `api.functions_window`.  .. _window.expanding:

Expanding window

An expanding window yields the value of an aggregation statistic with all the data available up to that
point in time. Since these calculations are a special case of rolling statistics,
they are implemented in pandas such that the following two calls are equivalent:  .. ipython:: python  df = pd.DataFrame(range(5))  df.rolling(window=len(df), min_periods=1).mean()  df.expanding(min_periods=1).mean()  For all supported aggregation functions, see `api.functions_expanding`.  .. _window.exponentially_weighted:

An exponentially weighted window is similar to an expanding window but with each prior point
being exponentially weighted down relative to the current point.  In general, a weighted moving average is calculated as  .. math::  y_t = \frac{\sum_{i=0}^t w_i x_{t-i}}{\sum_{i=0}^t w_i},  where `x_t` is the input, `y_t` is the result and the `w_i`
are the weights.  For all supported aggregation functions, see `api.functions_ewm`.  The EW functions support two variants of exponential weights.
The default, ``adjust=True``, uses the weights `w_i = (1 - \alpha)^i`
which gives  .. math::  y_t = \frac{x_t + (1 - \alpha)x_{t-1} + (1 - \alpha)^2 x_{t-2} + ...  + (1 - \alpha)^t x_{0}}{1 + (1 - \alpha) + (1 - \alpha)^2 + ...  + (1 - \alpha)^t}  When ``adjust=False`` is specified, moving averages are calculated as  .. math::  y_0 &= x_0 \\  y_t &= (1 - \alpha) y_{t-1} + \alpha x_t,  which is equivalent to using weights  .. math::  w_i = \begin{cases}  \alpha (1 - \alpha)^i & \text{if } i < t \\  (1 - \alpha)^i  & \text{if } i = t.  \end{cases}  .. note::  These equations are sometimes written in terms of `\alpha' = 1 - \alpha`, e.g.  .. math::  y_t = \alpha' y_{t-1} + (1 - \alpha') x_t.  The difference between the above two variants arises because we are
dealing with series which have finite history. Consider a series of infinite
history, with ``adjust=True``:  .. math::  y_t = \frac{x_t + (1 - \alpha)x_{t-1} + (1 - \alpha)^2 x_{t-2} + ...}  {1 + (1 - \alpha) + (1 - \alpha)^2 + ...}  Noting that the denominator is a geometric series with initial term equal to 1
and a ratio of `1 - \alpha` we have  .. math::  y_t &= \frac{x_t + (1 - \alpha)x_{t-1} + (1 - \alpha)^2 x_{t-2} + ...}  {\frac{1}{1 - (1 - \alpha)}}\\  &= [x_t + (1 - \alpha)x_{t-1} + (1 - \alpha)^2 x_{t-2} + ...] \alpha \\  &= \alpha x_t + [(1-\alpha)x_{t-1} + (1 - \alpha)^2 x_{t-2} + ...]\alpha \\  &= \alpha x_t + (1 - \alpha)[x_{t-1} + (1 - \alpha) x_{t-2} + ...]\alpha\\  &= \alpha x_t + (1 - \alpha) y_{t-1}  which is the same expression as ``adjust=False`` above and therefore
shows the equivalence of the two variants for infinite series.
When ``adjust=False``, we have `y_0 = x_0` and
`y_t = \alpha x_t + (1 - \alpha) y_{t-1}`.
Therefore, there is an assumption that `x_0` is not an ordinary value
but rather an exponentially weighted moment of the infinite series up to that
point.  One must have `0 < \alpha \leq 1`, and while it is possible to pass
`\alpha` directly, it's often easier to think about either the
**span**, **center  of mass (com)** or **half-life** of an EW moment:  .. math::  \alpha =  \begin{cases}  \frac{2}{s + 1},  & \text{for span}\ s \geq 1\\  \frac{1}{1 + c},  & \text{for center of mass}\ c \geq 0\\  1 - \exp^{\frac{\log 0.5}{h}}, & \text{for half-life}\ h > 0  \end{cases}  One must specify precisely one of **span**, **center of mass**, **half-life**
and **alpha** to the EW functions:  * **Span** corresponds to what is commonly called an "N-day EW moving average".
* **Center of mass** has a more physical interpretation and can be thought of  in terms of span: `c = (s - 1) / 2`.
* **Half-life** is the period of time for the exponential weight to reduce to  one half.
* **Alpha** specifies the smoothing factor directly.  .. versionadded:: 1.1.0  You can also specify ``halflife`` in terms of a timedelta convertible unit to specify the amount of
time it takes for an observation to decay to half its value when also specifying a sequence
of ``times``.  .. ipython:: python  df = pd.DataFrame({"B": [0, 1, 2, np.nan, 4]})  df  times = ["2020-01-01", "2020-01-03", "2020-01-10", "2020-01-15", "2020-01-17"]  df.ewm(halflife="4 days", times=pd.DatetimeIndex(times)).mean()  The following formula is used to compute exponentially weighted mean with an input vector of times:  .. math::  y_t = \frac{\sum_{i=0}^t 0.5^\frac{t_{t} - t_{i}}{\lambda} x_{t-i}}{\sum_{i=0}^t 0.5^\frac{t_{t} - t_{i}}{\lambda}},  ExponentialMovingWindow also has an ``ignore_na`` argument, which determines how

intermediate null values affect the calculation of the weights.
When ``ignore_na=False`` (the default), weights are calculated based on absolute
positions, so that intermediate null values affect the result.
When ``ignore_na=True``,
weights are calculated by ignoring intermediate null values.
For example, assuming ``adjust=True``, if ``ignore_na=False``, the weighted
average of ``3, NaN, 5`` would be calculated as  .. math::  \frac{(1-\alpha)^2 \cdot 3 + 1 \cdot 5}{(1-\alpha)^2 + 1}.  Whereas if ``ignore_na=True``, the weighted average would be calculated as  .. math::  \frac{(1-\alpha) \cdot 3 + 1 \cdot 5}{(1-\alpha) + 1}.  The `~Ewm.var`, `~Ewm.std`, and `~Ewm.cov` functions have a ``bias`` argument,
specifying whether the result should contain biased or unbiased statistics.
For example, if ``bias=True``, ``ewmvar(x)`` is calculated as
``ewmvar(x) = ewma(x**2) - ewma(x)**2``;
whereas if ``bias=False`` (the default), the biased variance statistics
are scaled by debiasing factors  .. math::  \frac{\left(\sum_{i=0}^t w_i\right)^2}{\left(\sum_{i=0}^t w_i\right)^2 - \sum_{i=0}^t w_i^2}.  (For `w_i = 1`, this reduces to the usual `N / (N - 1)` factor,
with `N = t + 1`.)
See Weighted Sample Variance_
on Wikipedia for further details.

Version 1.5

.. toctree::  2  v1.5.4  v1.5.3  v1.5.2  v1.5.1  v1.5.0

Version 1.4

.. toctree::  2  v1.4.4  v1.4.3  v1.4.2  v1.4.1  v1.4.0

Version 1.3

.. toctree::  2  v1.3.5  v1.3.4  v1.3.3  v1.3.2  v1.3.1  v1.3.0

Version 1.2

.. toctree::  2  v1.2.5  v1.2.4  v1.2.3  v1.2.2  v1.2.1  v1.2.0

Version 1.1

.. toctree::  2  v1.1.5  v1.1.4  v1.1.3  v1.1.2  v1.1.1  v1.1.0

Version 1.0

.. toctree::  2  v1.0.5  v1.0.4  v1.0.3  v1.0.2  v1.0.1  v1.0.0

Version 0.25

.. toctree::  2  v0.25.3  v0.25.2  v0.25.1  v0.25.0

Version 0.23

.. toctree::  2  v0.23.4  v0.23.3  v0.23.2  v0.23.1  v0.23.0

{{ header }}  This is a major release from 0.9.1 and includes many new features and
enhancements along with a large number of bug fixes. There are also a number of
important API changes that long-time pandas users should pay close attention
to.  File parsing new features
~~~  The delimited file parsing engine (the guts of ``read_csv`` and ``read_table``)
has been rewritten from the ground up and now uses a fraction the amount of
memory while parsing, while being 40% or more faster in most use cases (in some
cases much faster).  There are also many new features:  - Much-improved Unicode handling via the ``encoding`` option.
- Column filtering (``usecols``)
- Dtype specification (``dtype`` argument)
- Ability to specify strings to be recognized as True/False
- Ability to yield NumPy record arrays (``as_recarray``)
- High performance ``delim_whitespace`` option
- Decimal format (e.g. European format) specification
- Easier CSV dialect options: ``escapechar``, ``lineterminator``,  ``quotechar``, etc.
- More robust handling of many exceptional kinds of files observed in the wild  API changes
~~~  **Deprecated DataFrame BINOP TimeSeries special case behavior**  The default behavior of binary operations between a DataFrame and a Series has
always been to align on the DataFrame's columns and broadcast down the rows,
**except** in the special case that the DataFrame contains time series. Since
there are now method for each binary operator enabling you to specify how you
want to broadcast, we are phasing out this special case (Zen of Python:
*Special cases aren't special enough to break the rules*). Here's what I'm
talking about:  .. ipython:: python  import pandas as pd  df = pd.DataFrame(np.random.randn(6, 4), index=pd.date_range("1/1/2000", periods=6))  df  # deprecated now  df - df[0]  # Change your code to  df.sub(df[0], axis=0)  # align on axis 0 (rows)  You will get a deprecation warning in the 0.10.x series, and the deprecated
functionality will be removed in 0.11 or later.  **Altered resample default behavior**  The default time series ``resample`` binning behavior of daily ``D`` and
*higher* frequencies has been changed to ``closed='left', label='left'``. Lower
nfrequencies are unaffected. The prior defaults were causing a great deal of
confusion for users, especially resampling data to daily frequency (which

labeled the aggregated group with the end of the interval: the next day).  .. code-block:: ipython  In [1]: dates = pd.date_range('1/1/2000', '1/5/2000', freq='4h')  In [2]: series = pd.Series(np.arange(len(dates)), index=dates)  In [3]: series  Out[3]:  2000-01-01 000  0  2000-01-01 0400  1  2000-01-01 0800  2  2000-01-01 1200  3  2000-01-01 1600  4  2000-01-01 2000  5  2000-01-02 000  6  2000-01-02 0400  7  2000-01-02 0800  8  2000-01-02 1200  9  2000-01-02 1600  10  2000-01-02 2000  11  2000-01-03 000  12  2000-01-03 0400  13  2000-01-03 0800  14  2000-01-03 1200  15  2000-01-03 1600  16  2000-01-03 2000  17  2000-01-04 000  18  2000-01-04 0400  19  2000-01-04 0800  20  2000-01-04 1200  21  2000-01-04 1600  22  2000-01-04 2000  23  2000-01-05 000  24  Freq: 4H, dtype: int64  In [4]: series.resample('D', how='sum')  Out[4]:  2000-01-01  15  2000-01-02  51  2000-01-03  87  2000-01-04  123  2000-01-05  24  Freq: D, dtype: int64  In [5]: # old behavior  In [6]: series.resample('D', how='sum', closed='right', label='right')  Out[6]:  2000-01-01  0  2000-01-02  21  2000-01-03  57  2000-01-04  93  2000-01-05  129  Freq: D, dtype: int64  - Infinity and negative infinity are no longer treated as NA by ``isnull`` and  ``notnull``. That they ever were was a relic of early pandas. This behavior  can be re-enabled globally by the ``mode.use_inf_as_null`` option:  .. code-block:: ipython  In [6]: s = pd.Series([1.5, np.inf, 3.4, -np.inf])  In [7]: pd.isnull(s)  Out[7]:  0  False  1  False  2  False  3  False  Length: 4, dtype: bool  In [8]: s.fillna(0)  Out[8]:  0  1.5000  1  inf  2  3.4000  3  -inf  Length: 4, dtype: float64  In [9]: pd.set_option('use_inf_as_null', True)  In [10]: pd.isnull(s)  Out[10]:  0  False  1  True  2  False  3  True  Length: 4, dtype: bool  In [11]: s.fillna(0)  Out[11]:  0  1.5  1  0.0  2  3.4  3  0.0  Length: 4, dtype: float64  In [12]: pd.reset_option('use_inf_as_null')  - Methods with the ``inplace`` option now all return ``None`` instead of the  calling object. E.g. code written like ``df = df.fillna(0, inplace=True)``  may stop working. To fix, simply delete the unnecessary variable assignment.  - ``pandas.merge`` no longer sorts the group keys (``sort=False``) by  default. This was done for performance reasons: the group-key sorting is  often one of the more expensive parts of the computation and is often  unnecessary.  - The default column names for a file with no header have been changed to the  integers ``0`` through ``N - 1``. This is to create consistency with the  DataFrame constructor with no columns specified. The v0.9.0 behavior (names  ``X0``, ``X1``, ...) can be reproduced by specifying ``prefix='X'``:  .. ipython:: python  import io  data = """  a,b,c  1,Yes,2  3,No,4  """  print(data)  pd.read_csv(io.StringIO(data), header=None)  pd.read_csv(io.StringIO(data), header=None, prefix="X")  - Values like ``'Yes'`` and ``'No'`` are not interpreted as boolean by default,  though this can be controlled by new ``true_values`` and ``false_values``  arguments:  .. code-block:: ipython  In [4]: print(data)  a,b,c  1,Yes,2  3,No,4  In [5]: pd.read_csv(io.StringIO(data))  Out[5]:  a  b  c  0  1  Yes  2  1  3  No  4  In [6]: pd.read_csv(io.StringIO(data), true_values=["Yes"], false_values=["No"])  Out[6]:  a  b  c  0  1  True  2  1  3  False  4  - The file parsers will not recognize non-string values arising from a  converter function as NA if passed in the ``na_values`` argument. It's better  to do post-processing using the ``replace`` function instead.  - Calling ``fillna`` on Series or DataFrame with no arguments is no longer  valid code. You must either specify a fill value or an interpolation method:  .. ipython:: python  s = pd.Series([np.nan, 1.0, 2.0, np.nan, 4])  s  s.fillna(0)  s.fillna(method="pad")  Convenience methods ``ffill`` and  ``bfill`` have been added:  .. ipython:: python  s.ffill()  - ``Series.apply`` will now operate on a returned value from the applied  function, that is itself a series, and possibly upcast the result to a  DataFrame  .. ipython:: python  def f(x):  return pd.Series([x, x ** 2], index=["x", "x^2"])  s = pd.Series(np.random.rand(5))  s  s.apply(f)  - New API functions for working with pandas options (`2097`):  - ``get_option`` / ``set_option`` - get/set the value of an option. Partial  names are accepted.  - ``reset_option`` - reset one or more options to  their default value. Partial names are accepted.  - ``describe_option`` -  print a description of one or more options. When called with no  arguments. print all registered options.  Note: ``set_printoptions``/ ``reset_printoptions`` are now deprecated (but  functioning), the print options now live under "display.XYZ". For example:  .. ipython:: python  pd.get_option("display.max_rows")  - to_string() methods now always return unicode strings  (`2224`).  New features

~~~  Wide DataFrame printing
~~~  Instead of printing the summary information, pandas now splits the string
representation across multiple rows by default:  .. ipython:: python  wide_frame = pd.DataFrame(np.random.randn(5, 16))  wide_frame  The old behavior of printing out summary information can be achieved via the
'expand_frame_repr' print option:  .. ipython:: python  pd.set_option("expand_frame_repr", False)  wide_frame  .. ipython:: python  pd.reset_option("expand_frame_repr")  The width of each line can be changed via 'line_width' (80 by default):  .. code-block:: python  pd.set_option("line_width", 40)  wide_frame  Updated PyTables support

~~~  `Docs <io.hdf5>` for PyTables ``Table`` format & several enhancements to the api. Here is a taste of what to expect.  .. code-block:: ipython  In [41]: store = pd.HDFStore('store.h5')  In [42]: df = pd.DataFrame(np.random.randn(8, 3),  ...:  index=pd.date_range('1/1/2000', periods=8),  ...:  columns=['A', 'B', 'C'])  In [43]: df  Out[43]:  A  B  C  2000-01-01 -2.036047  0.000830 -0.955697  2000-01-02 -0.898872 -0.725411  0.059904  2000-01-03 -0.449644  1.082900 -1.221265  2000-01-04  0.361078  1.330704  0.855932  2000-01-05 -1.216718  1.48887  0.018993  2000-01-06 -0.877046  0.045976  0.437274  2000-01-07 -0.567182 -0.888657 -0.556383  2000-01-08  0.655457  1.117949 -2.782376  [8 rows x 3 columns]  # appending data frames  In [44]: df1 = df[0:4]  In [45]: df2 = df[4:]  In [46]: store.append('df', df1)  In [47]: store.append('df', df2)  In [48]: store  Out[48]:  <class 'pandas.io.pytables.HDFStore'>  File path: store.h5  /df  frame_table  (typ->appendable,nrows->8,ncols->3,indexers->[index])  # selecting the entire store  In [49]: store.select('df')  Out[49]:  A  B  C  2000-01-01 -2.036047  0.000830 -0.955697  2000-01-02 -0.898872 -0.725411  0.059904  2000-01-03 -0.449644  1.082900 -1.221265  2000-01-04  0.361078  1.330704  0.855932  2000-01-05 -1.216718  1.48887  0.018993  2000-01-06 -0.877046  0.045976  0.437274  2000-01-07 -0.567182 -0.888657 -0.556383  2000-01-08  0.655457  1.117949 -2.782376  [8 rows x 3 columns]  .. code-block:: ipython  In [50]: wp = pd.Panel(np.random.randn(2, 5, 4), items=['Item1', 'Item2'],  ...:  major_axis=pd.date_range('1/1/2000', periods=5),  ...:  minor_axis=['A', 'B', 'C', 'D'])  In [51]: wp  Out[51]:  <class 'pandas.core.panel.Panel'>  Dimensions: 2 (items) x 5 (major_axis) x 4 (minor_axis)  Items axis: Item1 to Item2  Major_axis axis: 2000-01-01 000 to 2000-01-05 000  Minor_axis axis: A to D  # storing a panel  In [52]: store.append('wp', wp)  # selecting via A QUERY  In [53]: store.select('wp', [pd.Term('major_axis>2000102'),  ...:  pd.Term('minor_axis', '=', ['A', 'B'])])  ...:  Out[53]:  <class 'pandas.core.panel.Panel'>  Dimensions: 2 (items) x 3 (major_axis) x 2 (minor_axis)  Items axis: Item1 to Item2  Major_axis axis: 2000-01-03 000 to 2000-01-05 000  Minor_axis axis: A to B  # removing data from tables  In [54]: store.remove('wp', pd.Term('major_axis>2000103'))  Out[54]: 8  In [55]: store.select('wp')  Out[55]:  <class 'pandas.core.panel.Panel'>  Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)  Items axis: Item1 to Item2  Major_axis axis: 2000-01-01 000 to 2000-01-03 000  Minor_axis axis: A to D  # deleting a store  In [56]: del store['df']  In [57]: store  Out[57]:  <class 'pandas.io.pytables.HDFStore'>  File path: store.h5  /wp  wide_table  (typ->appendable,nrows->12,ncols->2,indexers->[major_axis,minor_axis])  **Enhancements**  - added ability to hierarchical keys  .. code-block:: ipython  In [58]: store.put('foo/bar/bah', df)  In [59]: store.append('food/orange', df)  In [60]: store.append('food/apple', df)  In [61]: store  Out[61]:  <class 'pandas.io.pytables.HDFStore'>  File path: store.h5  /foo/bar/bah  frame  (shape->[8,3])  /food/apple  frame_table  (typ->appendable,nrows->8,ncols->3,indexers->[index])  /food/orange  frame_table  (typ->appendable,nrows->8,ncols->3,indexers->[index])  /wp  wide_table  (typ->appendable,nrows->12,ncols->2,indexers->[major_axis,minor_axis])  # remove all nodes under this level  In [62]: store.remove('food')  In [63]: store  Out[63]:  <class 'pandas.io.pytables.HDFStore'>  File path: store.h5  /foo/bar/bah  frame  (shape->[8,3])  /wp  wide_table  (typ->appendable,nrows->12,ncols->2,indexers->[major_axis,minor_axis])  - added mixed-dtype support!  .. code-block:: ipython  In [64]: df['string'] = 'string'  In [65]: df['int'] = 1  In [66]: store.append('df', df)  In [67]: df1 = store.select('df')  In [68]: df1  Out[68]:  A  B  C  string  int  2000-01-01 -2.036047  0.000830 -0.955697  string  1  2000-01-02 -0.898872 -0.725411  0.059904  string  1  2000-01-03 -0.449644  1.082900 -1.221265  string  1  2000-01-04  0.361078  1.330704  0.855932  string  1  2000-01-05 -1.216718  1.48887  0.018993  string  1  2000-01-06 -0.877046  0.045976  0.437274  string  1  2000-01-07 -0.567182 -0.888657 -0.556383  string  1  2000-01-08  0.655457  1.117949 -2.782376  string  1  [8 rows x 5 columns]  In [69]: df1.get_dtype_counts()  Out[69]:  float64  3  int64  1  object  1  dtype: int64  - performance improvements on table writing

- support for arbitrarily indexed dimensions
- ``SparseSeries`` now has a ``density`` property (`2384`)
- enable ``Series.str.strip/lstrip/rstrip`` methods to take an input argument  to strip arbitrary characters (`2411`)
- implement ``value_vars`` in ``melt`` to limit values to certain columns  and add ``melt`` to pandas namespace (`2412`)  **Bug Fixes**  - added ``Term`` method of specifying where conditions (`1996`).
- ``del store['df']`` now call ``store.remove('df')`` for store deletion
- deleting of consecutive rows is much faster than before
- ``min_itemsize`` parameter can be specified in table creation to force a  minimum size for indexing columns (the previous implementation would set the  column size based on the first append)
- indexing support via ``create_table_index`` (requires PyTables >= 2.3)  (`698`).
- appending on a store would fail if the table was not first created via ``put``
- fixed issue with missing attributes after loading a pickled dataframe (GH2431)
- minor change to select and remove: require a table ONLY if where is also  provided (and not None)  **Compatibility**  0.10 of ``HDFStore`` is backwards compatible for reading tables created in a prior version of pandas,
however, query terms using the prior (undocumented) methodology are unsupported. You must read in the entire
file and write it out using the new format to take advantage of the updates.  N dimensional panels (experimental)
~~~  Adding experimental support for Panel4D and factory functions to create n-dimensional named panels.
Here is a taste of what to expect.  .. code-block:: ipython  In [58]: p4d = Panel4D(np.random.randn(2, 2, 5, 4),  ...:  labels=['Label1','Label2'],  ...:  items=['Item1', 'Item2'],  ...:  major_axis=date_range('1/1/2000', periods=5),  ...:  minor_axis=['A', 'B', 'C', 'D'])  ...:  In [59]: p4d  Out[59]:  <class 'pandas.core.panelnd.Panel4D'>  Dimensions: 2 (labels) x 2 (items) x 5 (major_axis) x 4 (minor_axis)  Labels axis: Label1 to Label2  Items axis: Item1 to Item2  Major_axis axis: 2000-01-01 000 to 2000-01-05 000  Minor_axis axis: A to D  See the `full release notes
<release>` or issue tracker
on GitHub for a complete list.  .. _whatsnew_0.10.0.contributors:  Contributors
~~~  .. contributors:: v0.9.0..v0.10.0

{{ header }}  This is a minor release from 0.10.0 and includes new features, enhancements,
and bug fixes. In particular, there is substantial new HDFStore functionality
contributed by Jeff Reback.  An undesired API breakage with functions taking the ``inplace`` option has been
reverted and deprecation warnings added.  API changes
~~~  - Functions taking an ``inplace`` option return the calling object as before. A  deprecation message has been added
- Groupby aggregations Max/Min no longer exclude non-numeric data (`2700`)
- Resampling an empty DataFrame now returns an empty DataFrame instead of  raising an exception (`2640`)
- The file reader will now raise an exception when NA values are found in an  explicitly specified integer column instead of converting the column to float  (`2631`)
- DatetimeIndex.unique now returns a DatetimeIndex with the same name and
- timezone instead of an array (`2563`)  New features
~~~  - MySQL support for database (contribution from Dan Allan)  HDFStore
~~~  You may need to upgrade your existing data files. Please visit the
**compatibility** section in the main docs.  .. ipython:: python  import os  os.remove("store.h5")  You can designate (and index) certain columns that you want to be able to
perform queries on a table, by passing a list to ``data_columns``  .. ipython:: python  store = pd.HDFStore("store.h5")  df = pd.DataFrame(  np.random.randn(8, 3),  index=pd.date_range("1/1/2000", periods=8),  columns=["A", "B", "C"],  )  df["string"] = "foo"  df.loc[df.index[4:6], "string"] = np.nan  df.loc[df.index[7:9], "string"] = "bar"  df["string2"] = "cool"  df  # on-disk operations  store.append("df", df, data_columns=["B", "C", "string", "string2"])  store.select("df", "B>0 and string=='foo'")  # this is in-memory version of this type of selection  df[(df.B > 0) & (df.string == "foo")]  Retrieving unique values in an indexable or data column.  .. code-block:: python  # note that this is deprecated as of 0.14.0  # can be replicated by: store.select_column('df','index').unique()  store.unique("df", "index")  store.unique("df", "string")  You can now store ``datetime64`` in data columns  .. ipython:: python  df_mixed = df.copy()  df_mixed["datetime64"] = pd.Timestamp("20010102")  df_mixed.loc[df_mixed.index[3:4], ["A", "B"]] = np.nan  store.append("df_mixed", df_mixed)  df_mixed1 = store.select("df_mixed")  df_mixed1  df_mixed1.dtypes.value_counts()  You can pass ``columns`` keyword to select to filter a list of the return
columns, this is equivalent to passing a

``Term('columns',list_of_columns_to_filter)``  .. ipython:: python  store.select("df", columns=["A", "B"])  ``HDFStore`` now serializes MultiIndex dataframes when appending tables.  .. code-block:: ipython  In [19]: index = pd.MultiIndex(levels=[['foo', 'bar', 'baz', 'qux'],  ...:  ['one', 'two', 'three']],  ...:  labels=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3],  ...:  [0, 1, 2, 0, 1, 1, 2, 0, 1, 2]],  ...:  names=['foo', 'bar'])  ...:  In [20]: df = pd.DataFrame(np.random.randn(10, 3), index=index,  ...:  columns=['A', 'B', 'C'])  ...:  In [21]: df  Out[21]:  A  B  C  foo bar  foo one  -0.116619  0.295575 -1.047704  two  1.640556  1.905836  2.772115  three  0.088787 -1.144197 -0.633372  bar one  0.925372 -0.006438 -0.820408  two  -0.600874 -1.039266  0.824758  baz two  -0.824095 -0.337730 -0.927764  three -0.840123  0.248505 -0.109250  qux one  0.431977 -0.460710  0.336505  two  -3.207595 -1.535854  0.409769  three -0.673145 -0.741113 -0.110891  In [22]: store.append('mi', df)  In [23]: store.select('mi')  Out[23]:  A  B  C  foo bar  foo one  -0.116619  0.295575 -1.047704  two  1.640556  1.905836  2.772115  three  0.088787 -1.144197 -0.633372  bar one  0.925372 -0.006438 -0.820408  two  -0.600874 -1.039266  0.824758  baz two  -0.824095 -0.337730 -0.927764  three -0.840123  0.248505 -0.109250  qux one  0.431977 -0.460710  0.336505  two  -3.207595 -1.535854  0.409769  three -0.673145 -0.741113 -0.110891  # the levels are automatically included as data columns  In [24]: store.select('mi', "foo='bar'")  Out[24]:  A  B  C  foo bar  bar one  0.925372 -0.006438 -0.820408  two -0.600874 -1.039266  0.824758  Multi-table creation via ``append_to_multiple`` and selection via
``select_as_multiple`` can create/select from multiple tables and return a
combined result, by using ``where`` on a selector table.  .. ipython:: python  df_mt = pd.DataFrame(  np.random.randn(8, 6),  index=pd.date_range("1/1/2000", periods=8),  columns=["A", "B", "C", "D", "E", "F"],  )  df_mt["foo"] = "bar"  # you can also create the tables individually  store.append_to_multiple(  {"df1_mt": ["A", "B"], "df2_mt": None}, df_mt, selector="df1_mt"  )  store  # individual tables were created  store.select("df1_mt")  store.select("df2_mt")  # as a multiple  store.select_as_multiple(  ["df1_mt", "df2_mt"], where=["A>0", "B>0"], selector="df1_mt"  )  .. ipython:: python  store.close()  os.remove("store.h5")  **Enhancements**  - ``HDFStore`` now can read native PyTables table format tables  - You can pass ``nan_rep = 'my_nan_rep'`` to append, to change the default nan  representation on disk (which converts to/from ``np.nan``), this defaults to  ``nan``.  - You can pass ``index`` to ``append``. This defaults to ``True``. This will  automagically create indices on the *indexables* and *data columns* of the  table  - You can pass ``chunksize=an integer`` to ``append``, to change the writing  chunksize (default is 5000). This will significantly lower your memory usage  on writing.  - You can pass ``expectedrows=an integer`` to the first ``append``, to set the  TOTAL number of expected rows that ``PyTables`` will expected. This will  optimize read/write performance.  - ``Select`` now supports passing ``start`` and ``stop`` to provide selection  space limiting in selection.  - Greatly improved ISO8601 (e.g., yyy-mm-dd) date parsing for file parsers (`2698`)
- Allow ``DataFrame.merge`` to handle combinatorial sizes too large for 64-bit  integer (`2690`)
- Series now has unary negation (-series) and inversion (~series) operators (`2686`)
- DataFrame.plot now includes a ``logx`` parameter to change the x-axis to log scale (`2327`)
- Series arithmetic operators can now handle constant and ndarray input (`2574`)
- ExcelFile now takes a ``kind`` argument to specify the file type (`2613`)

- A faster implementation for Series.str methods (`2602`)  **Bug Fixes**  - ``HDFStore`` tables can now store ``float32`` types correctly (cannot be  mixed with ``float64`` however)
- Fixed Google Analytics prefix when specifying request segment (`2713`).
- Function to reset Google Analytics token store so users can recover from  improperly setup client secrets (`2687`).
- Fixed groupby bug resulting in segfault when passing in MultiIndex (`2706`)
- Fixed bug where passing a Series with datetime64 values into ``to_datetime``  results in bogus output values (`2699`)
- Fixed bug in ``pattern in HDFStore`` expressions when pattern is not a valid  regex (`2694`)
- Fixed performance issues while aggregating boolean data (`2692`)
- When given a boolean mask key and a Series of new values, Series __setitem__  will now align the incoming values with the original Series (`2686`)
- Fixed MemoryError caused by performing counting sort on sorting MultiIndex  levels with a very large number of combinatorial values (`2684`)
- Fixed bug that causes plotting to fail when the index is a DatetimeIndex with  a fixed-offset timezone (`2683`)
- Corrected business day subtraction logic when the offset is more than 5 bdays  and the starting date is on a weekend (`2680`)
- Fixed C file parser behavior when the file has more columns than data  (`2668`)
- Fixed file reader bug that misaligned columns with data in the presence of an  implicit column and a specified ``usecols`` value
- DataFrames with numerical or datetime indices are now sorted prior to  plotting (`2609`)
- Fixed DataFrame.from_records error when passed columns, index, but empty  records (`2633`)
- Several bug fixed for Series operations when dtype is datetime64 (`2689`,  `2629`, `2626`)  See the `full release notes
on GitHub for a complete list.  .. _whatsnew_0.10.1.contributors:  Contributors
~~~  .. contributors:: v0.10.0..v0.10.1

{{ header }}  This is a major release from 0.10.1 and includes many new features and
enhancements along with a large number of bug fixes. The methods of Selecting
Data have had quite a number of additions, and Dtype support is now full-fledged.
There are also a number of important API changes that long-time pandas users should
pay close attention to.  There is a new section in the documentation, `10 Minutes to Pandas <10min>`,
primarily geared to new users.  There is a new section in the documentation, `Cookbook <cookbook>`, a collection
of useful recipes in pandas (and that we want contributions!).  There are several libraries that are now `Recommended Dependencies <install.recommended_dependencies>`  Selection choices
~~~  Starting in 0.11.0, object selection has had a number of user-requested additions in
order to support more explicit location based indexing. pandas now supports
three types of multi-axis indexing.  - ``.loc`` is strictly label based, will raise ``KeyError`` when the items are not found, allowed inputs are:  - A single label, e.g. ``5`` or ``'a'``, (note that ``5`` is interpreted as a *label* of the index. This use is **not** an integer position along the index)  - A list or array of labels ``['a', 'b', 'c']``  - A slice object with labels ``'a':'f'``, (note that contrary to usual python slices, **both** the start and the stop are included!)  - A boolean array  See more at `Selection by Label <indexing.label>`  - ``.iloc`` is strictly integer position based (from ``0`` to ``length-1`` of the axis), will raise ``IndexError`` when the requested indices are out of bounds. Allowed inputs are:  - An integer e.g. ``5``  - A list or array of integers ``[4, 3, 0]``  - A slice object with ints ``1:7``  - A boolean array  See more at `Selection by Position <indexing.integer>`  - ``.ix`` supports mixed integer and label based access. It is primarily label based, but will fallback to integer positional access. ``.ix`` is the most general and will support  any of the inputs to ``.loc`` and ``.iloc``, as well as support for floating point label schemes. ``.ix`` is especially useful when dealing with mixed positional and label  based hierarchical indexes.  As using integer slices with ``.ix`` have different behavior depending on whether the slice  is interpreted as position based or label based, it's usually better to be  explicit and use ``.iloc`` or ``.loc``.  See more at `Advanced Indexing <advanced>` and `Advanced Hierarchical <advanced.advanced_hierarchical>`.  Selection deprecations
~~~  Starting in version 0.11.0, these methods *may* be deprecated in future versions.  - ``irow``
- ``icol``
- ``iget_value``  See the section `Selection by Position <indexing.integer>` for substitutes.  Dtypes
~~~  Numeric dtypes will propagate and can coexist in DataFrames. If a dtype is passed (either directly via the ``dtype`` keyword, a passed ``ndarray``, or a passed ``Series``, then it will be preserved in DataFrame operations. Furthermore, different numeric dtypes will **NOT** be combined. The following example will give you a taste.  .. ipython:: python  df1 = pd.DataFrame(np.random.randn(8, 1), columns=['A'], dtype='float32')  df1  df1.dtypes  df2 = pd.DataFrame({'A': pd.Series(np.random.randn(8), dtype='float16'),  'B': pd.Series(np.random.randn(8)),  'C': pd.Series(range(8), dtype='uint8')})  df2  df2.dtypes  # here you get some upcasting  df3 = df1.reindex_like(df2).fillna(value=0.0) + df2  df3  df3.dtypes  Dtype conversion

~~~  This is lower-common-denominator upcasting, meaning you get the dtype which can accommodate all of the types  .. ipython:: python  df3.values.dtype  Conversion  .. ipython:: python  df3.astype('float32').dtypes  Mixed conversion  .. code-block:: ipython  In [12]: df3['D'] = '1.'  In [13]: df3['E'] = '1'  In [14]: df3.convert_objects(convert_numeric=True).dtypes  Out[14]:  A  float32  B  float64  C  float64  D  float64  E  int64  dtype: object  # same, but specific dtype conversion  In [15]: df3['D'] = df3['D'].astype('float16')  In [16]: df3['E'] = df3['E'].astype('int32')  In [17]: df3.dtypes  Out[17]:  A  float32  B  float64  C  float64  D  float16  E  int32  dtype: object  Forcing date coercion (and setting ``NaT`` when not datelike)  .. code-block:: ipython  In [18]: import datetime  In [19]: s = pd.Series([datetime.datetime(2001, 1, 1, 0, 0), 'foo', 1.0, 1,  ...:  pd.Timestamp('20010104'), '20010105'], dtype='O')  ...:  In [20]: s.convert_objects(convert_dates='coerce')  Out[20]:  0  2001-01-01  1  NaT  2  NaT  3  NaT  4  2001-01-04  5  2001-01-05  dtype: datetime64[ns]  Dtype gotchas
~~~  **Platform gotchas**  Starting in 0.11.0, construction of DataFrame/Series will use default dtypes of ``int64`` and ``float64``,
*regardless of platform*. This is not an apparent change from earlier versions of pandas. If you specify
dtypes, they *WILL* be respected, however (`2837`)  The following will all result in ``int64`` dtypes  .. code-block:: ipython  In [21]: pd.DataFrame([1, 2], columns=['a']).dtypes  Out[21]:  a  int64  dtype: object  In [22]: pd.DataFrame({'a': [1, 2]}).dtypes  Out[22]:  a  int64  dtype: object  In [23]: pd.DataFrame({'a': 1}, index=range(2)).dtypes  Out[23]:  a  int64  dtype: object  Keep in mind that ``DataFrame(np.array([1,2]))`` **WILL** result in ``int32`` on 32-bit platforms!  **Upcasting gotchas**  Performing indexing operations on integer type data can easily upcast the data.
The dtype of the input data will be preserved in cases where ``nans`` are not introduced.  .. code-block:: ipython  In [24]: dfi = df3.astype('int32')  In [25]: dfi['D'] = dfi['D'].astype('int64')  In [26]: dfi  Out[26]:  A  B  C  D  E  0  0  0  0  1  1  1 -2  0  1  1  1  2 -2  0  2  1  1  3  0 -1  3  1  1  4  1  0  4  1  1  5  0  0  5  1  1  6  0 -1  6  1  1  7  0  0  7  1  1  In [27]: dfi.dtypes  Out[27]:  A  int32  B  int32  C  int32  D  int64  E  int32  dtype: object  In [28]: casted = dfi[dfi > 0]  In [29]: casted  Out[29]:  A  B  C  D  E  0  NaN NaN  NaN  1  1  1  NaN NaN  1.0  1  1  2  NaN NaN  2.0  1  1  3  NaN NaN  3.0  1  1  4  1.0 NaN  4.0  1  1  5  NaN NaN  5.0  1  1  6  NaN NaN  6.0  1  1  7  NaN NaN  7.0  1  1  In [30]: casted.dtypes  Out[30]:  A  float64  B  float64  C  float64  D  int64  E  int32  dtype: object  While float dtypes are unchanged.  .. code-block:: ipython  In [31]: df4 = df3.copy()  In [32]: df4['A'] = df4['A'].astype('float32')  In [33]: df4.dtypes  Out[33]:  A  float32  B  float64  C  float64  D  float16  E  int32  dtype: object  In [34]: casted = df4[df4 > 0]  In [35]: casted  Out[35]:  A  B  C  D  E  0  NaN  NaN  NaN  1.0  1  1  NaN  0.567020  1.0  1.0  1  2  NaN  0.276232  2.0  1.0  1  3  NaN  NaN  3.0  1.0  1  4  1.933792  NaN  4.0  1.0  1  5  NaN  0.113648  5.0  1.0  1  6  NaN  NaN  6.0  1.0  1  7  NaN  0.524988  7.0  1.0  1  In [36]: casted.dtypes  Out[36]:  A  float32  B  float64  C  float64  D  float16  E  int32  dtype: object  Datetimes conversion
~~~  Datetime64[ns] columns in a DataFrame (or a Series) allow the use of ``np.nan`` to indicate a nan value,
in addition to the traditional ``NaT``, or not-a-time. This allows convenient nan setting in a generic way.
Furthermore ``datetime64[ns]`` columns are created by default, when passed datetimelike objects (*this change was introduced in 0.10.1*)

(`2809`, `2810`)  .. ipython:: python  df = pd.DataFrame(np.random.randn(6, 2), pd.date_range('20010102', periods=6),  columns=['A', ' B'])  df['timestamp'] = pd.Timestamp('20010103')  df  # datetime64[ns] out of the box  df.dtypes.value_counts()  # use the traditional nan, which is mapped to NaT internally  df.loc[df.index[2:4], ['A', 'timestamp']] = np.nan  df  Astype conversion on ``datetime64[ns]`` to ``object``, implicitly converts ``NaT`` to ``np.nan``  .. ipython:: python  import datetime  s = pd.Series([datetime.datetime(2001, 1, 2, 0, 0) for i in range(3)])  s.dtype  s[1] = np.nan  s  s.dtype  s = s.astype('O')  s  s.dtype  API changes
~~~  - Added to_series() method to indices, to facilitate the creation of indexers  (`3275`)  - ``HDFStore``  - added the method ``select_column`` to select a single column from a table as a Series.  - deprecated the ``unique`` method, can be replicated by ``select_column(key,column).unique()``  - ``min_itemsize`` parameter to ``append`` will now automatically create data_columns for passed keys  Enhancements

~~~  - Improved performance of df.to_csv() by up to 10x in some cases. (`3059`)  - Numexpr is now a `Recommended Dependencies <install.recommended_dependencies>`, to accelerate certain  types of numerical and boolean operations  - Bottleneck is now a `Recommended Dependencies <install.recommended_dependencies>`, to accelerate certain  types of ``nan`` operations  - ``HDFStore``  - support ``read_hdf/to_hdf`` API similar to ``read_csv/to_csv``  .. ipython:: python  df = pd.DataFrame({'A': range(5), 'B': range(5)})  df.to_hdf('store.h5', 'table', append=True)  pd.read_hdf('store.h5', 'table', where=['index > 2'])  .. ipython:: python  import os  os.remove('store.h5')  - provide dotted attribute access to ``get`` from stores, e.g. ``store.df == store['df']``  - new keywords ``iterator=boolean``, and ``chunksize=number_in_a_chunk`` are  provided to support iteration on ``select`` and ``select_as_multiple`` (`3076`)  - You can now select timestamps from an *unordered* timeseries similarly to an *ordered* timeseries (`2437`)  - You can now select with a string from a DataFrame with a datelike index, in a similar way to a Series (`3070`)  .. ipython:: python  idx = pd.date_range("2001-10-1", periods=5, freq='M')  ts = pd.Series(np.random.rand(len(idx)), index=idx)  ts['2001']  df = pd.DataFrame({'A': ts})  df['2001']  - ``Squeeze`` to possibly remove length 1 dimensions from an object.  .. code-block:: python  >>> p = pd.Panel(np.random.randn(3, 4, 4), items=['ItemA', 'ItemB', 'ItemC'],  ...  major_axis=pd.date_range('20010102', periods=4),  ...  minor_axis=['A', 'B', 'C', 'D'])  >>> p  <class 'pandas.core.panel.Panel'>  Dimensions: 3 (items) x 4 (major_axis) x 4 (minor_axis)  Items axis: ItemA to ItemC  Major_axis axis: 2001-01-02 000 to 2001-01-05 000  Minor_axis axis: A to D  >>> p.reindex(items=['ItemA']).squeeze()  A  B  C  D  2001-01-02  0.926089 -2.026458  0.501277 -0.204683  2001-01-03 -0.076524  1.081161  1.141361  0.479243  2001-01-04  0.641817 -0.185352  1.824568  0.809152  2001-01-05  0.575237  0.669934  1.398014 -0.399338  >>> p.reindex(items=['ItemA'], minor=['B']).squeeze()  2001-01-02  -2.026458  2001-01-03  1.081161  2001-01-04  -0.185352  2001-01-05  0.669934  Freq: D, Name: B, dtype: float64  - In ``pd.io.data.Options``,  + Fix bug when trying to fetch data for the current month when already  past expiry.  + Now using lxml to scrape html instead of BeautifulSoup (lxml was faster).  + New instance variables for calls and puts are automatically created  when a method that creates them is called. This works for current month  where the instance variables are simply ``calls`` and ``puts``. Also  works for future expiry months and save the instance variable as  ``callsMMYY`` or ``putsMMYY``, where ``MMYY`` are, respectively, the  month and year of the option's expiry.  + ``Options.get_near_stock_price`` now allows the user to specify the  month for which to get relevant options data.  + ``Options.get_forward_data`` now has optional kwargs ``near`` and  ``above_below``. This allows the user to specify if they would like to  only return forward looking data for options near the current stock  price. This just obtains the data from Options.get_near_stock_price  instead of Options.get_xxx_data() (`2758`).  - Cursor coordinate information is now displayed in time-series plots.  - added option ``display.max_seq_items`` to control the number of  elements printed per sequence pprinting it.  (`2979`)  - added option ``display.chop_threshold`` to control display of small numerical  values. (`2739`)  - added option ``display.max_info_rows`` to prevent verbose_info from being  calculated for frames above 1M rows (configurable). (`2807`, `2918`)  - value_counts() now accepts a "normalize" argument, for normalized  histograms. (`2710`).  - DataFrame.from_records now accepts not only dicts but any instance of  the collections.Mapping ABC.  - added option ``display.mpl_style`` providing a sleeker visual style  for plots. Based on https://gist.github.com/huyng/816622 (`3075`).  - Treat boolean values as integers (values 1 and 0) for numeric  operations. (`2641`)  - to_html() now accepts an optional "escape" argument to control reserved  HTML character escaping (enabled by default) and escapes ``&``, in addition  to ``<`` and ``>``.  (`2919`)  See the `full release notes

on GitHub for a complete list.  .. _whatsnew_0.11.0.contributors:  Contributors
~~~  .. contributors:: v0.10.1..v0.11.0

{{ header }}  This is a major release from 0.11.0 and includes several new features and
enhancements along with a large number of bug fixes.  Highlights include a consistent I/O API naming scheme, routines to read html,
write MultiIndexes to csv files, read & write STATA data files, read & write JSON format
files, Python 3 support for ``HDFStore``, filtering of groupby expressions via ``filter``, and a
revamped ``replace`` routine that accepts regular expressions.  API changes

~~~  - The I/O API is now much more consistent with a set of top level ``reader`` functions  accessed like ``pd.read_csv()`` that generally return a ``pandas`` object.  * ``read_csv``  * ``read_excel``  * ``read_hdf``  * ``read_sql``  * ``read_json``  * ``read_html``  * ``read_stata``  * ``read_clipboard``  The corresponding ``writer`` functions are object methods that are accessed like ``df.to_csv()``  * ``to_csv``  * ``to_excel``  * ``to_hdf``  * ``to_sql``  * ``to_json``  * ``to_html``  * ``to_stata``  * ``to_clipboard``  - Fix modulo and integer division on Series,DataFrames to act similarly to ``float`` dtypes to return  ``np.nan`` or ``np.inf`` as appropriate (`3590`). This correct a numpy bug that treats ``integer``  and ``float`` dtypes differently.  .. ipython:: python  p = pd.DataFrame({"first": [4, 5, 8], "second": [0, 0, 3]})  p % 0  p % p  p / p  p / 0  - Add ``squeeze`` keyword to ``groupby`` to allow reduction from  DataFrame -> Series if groups are unique. This is a Regression from 0.10.1.  We are reverting back to the prior behavior. This means groupby will return the  same shaped objects whether the groups are unique or not. Revert this issue (`2893`)  with (`3596`).  .. code-block:: ipython  In [2]: df2 = pd.DataFrame([{"val1": 1, "val2": 20},  ...:  {"val1": 1, "val2": 19},  ...:  {"val1": 1, "val2": 27},  ...:  {"val1": 1, "val2": 12}])  In [3]: def func(dataf):  ...:  return dataf["val2"] - dataf["val2"].mean()  ...:  In [4]: # squeezing the result frame to a series (because we have unique groups)  ...: df2.groupby("val1", squeeze=True).apply(func)  Out[4]:  0  0.5  1  -0.5  2  7.5  3  -7.5  Name: 1, dtype: float64  In [5]: # no squeezing (the default, and behavior in 0.10.1)  ...: df2.groupby("val1").apply(func)  Out[5]:  val2  0  1  2  3  val1  1  0.5 -0.5  7.5 -7.5  - Raise on ``iloc`` when boolean indexing with a label based indexer mask  e.g. a boolean Series, even with integer labels, will raise. Since ``iloc``  is purely positional based, the labels on the Series are not alignable (`3631`)  This case is rarely used, and there are plenty of alternatives. This preserves the  ``iloc`` API to be *purely* positional based.  .. ipython:: python  df = pd.DataFrame(range(5), index=list("ABCDE"), columns=["a"])  mask = df.a % 2 == 0  mask  # this is what you should use  df.loc[mask]  # this will work as well  df.iloc[mask.values]  ``df.iloc[mask]`` will raise a ``ValueError``  - The ``raise_on_error`` argument to plotting functions is removed. Instead,  plotting functions raise a ``TypeError`` when the ``dtype`` of the object  is ``object`` to remind you to avoid ``object`` arrays whenever possible  and thus you should cast to an appropriate numeric dtype if you need to  plot something.  - Add ``colormap`` keyword to DataFrame plotting methods. Accepts either a  matplotlib colormap object (ie, matplotlib.cm.jet) or a string name of such  an object (ie, 'jet'). The colormap is sampled to select the color for each  column. Please see `visualization.colormaps` for more information.  (`3860`)  - ``DataFrame.interpolate()`` is now deprecated. Please use  ``DataFrame.fillna()`` and ``DataFrame.replace()`` instead. (`3582`,  `3675`, `3676`)  - the ``method`` and ``axis`` arguments of ``DataFrame.replace()`` are  deprecated  - ``DataFrame.replace`` 's ``infer_types`` parameter is removed and now  performs conversion by default. (`3907`)  - Add the keyword ``allow_duplicates`` to ``DataFrame.insert`` to allow a duplicate column  to be inserted if ``True``, default is ``False`` (same as prior to 0.12) (`3679`)  - Implement ``__nonzero__`` for ``NDFrame`` objects (`3691`, `3696`)  - IO api  - added top-level function ``read_excel`` to replace the following,  The original API is deprecated and will be removed in a future version  .. code-block:: python  from pandas.io.parsers import ExcelFile  xls = ExcelFile("path_to_file.xls")  xls.parse("Sheet1", index_col=None, na_values=["NA"])  With  .. code-block:: python  import pandas as pd  pd.read_excel("path_to_file.xls", "Sheet1", index_col=None, na_values=["NA"])  - added top-level function ``read_sql`` that is equivalent to the following  .. code-block:: python  from pandas.io.sql import read_frame  read_frame(...)  - ``DataFrame.to_html`` and ``DataFrame.to_latex`` now accept a path for  their first argument (`3702`)  - Do not allow astypes on ``datetime64[ns]`` except to ``object``, and  ``timedelta64[ns]`` to ``object/int`` (`3425`)  - The behavior of ``datetime64`` dtypes has changed with respect to certain  so-called reduction operations (`3726`). The following operations now  raise a ``TypeError`` when performed on a ``Series`` and return an *empty*  ``Series`` when performed on a ``DataFrame`` similar to performing these  operations on, for example, a

operations (`3726`). The following operations now  raise a ``TypeError`` when performed on a ``Series`` and return an *empty*  ``Series`` when performed on a ``DataFrame`` similar to performing these  operations on, for example, a ``DataFrame`` of ``slice`` objects:  - sum, prod, mean, std, var, skew, kurt, corr, and cov  - ``read_html`` now defaults to ``None`` when reading, and falls back on  ``bs4`` + ``html5lib`` when lxml fails to parse. a list of parsers to try  until success is also valid  - The internal ``pandas`` class hierarchy has changed (slightly). The  previous ``PandasObject`` now is called ``PandasContainer`` and a new  ``PandasObject`` has become the base class for ``PandasContainer`` as well  as ``Index``, ``Categorical``, ``GroupBy``, ``SparseList``, and  ``SparseArray`` (+ their base classes). Currently, ``PandasObject``  provides string methods (from ``StringMixin``). (`4090`, `4092`)  - New ``StringMixin`` that, given a ``__unicode__`` method, gets python 2 and  python 3 compatible string methods (``__str__``, ``__bytes__``, and  ``__repr__``). Plus string safety throughout. Now employed in many places  throughout the pandas library. (`4090`, `4092`)  IO enhancements

~~~  - ``pd.read_html()`` can now parse HTML strings, files or urls and return  DataFrames, courtesy of @cpcloud. (`3477`, `3605`, `3606`, `3616`).  It works with a *single* parser backend: BeautifulSoup4 + html5lib `See the docs<io.html>`  You can use ``pd.read_html()`` to read the output from ``DataFrame.to_html()`` like so  .. ipython:: python  df = pd.DataFrame({"a": range(3), "b": list("abc")})  print(df)  html = df.to_html()  alist = pd.read_html(html, index_col=0)  print(df == alist[0])  Note that ``alist`` here is a Python ``list`` so ``pd.read_html()`` and  ``DataFrame.to_html()`` are not inverses.  - ``pd.read_html()`` no longer performs hard conversion of date strings  (`3656`).  .. warning::  You may have to install an older version of BeautifulSoup4,  `See the installation docs<install.optional_dependencies>`  - Added module for reading and writing Stata files: ``pandas.io.stata`` (`1512`)  accessible via ``read_stata`` top-level function for reading,  and ``to_stata`` DataFrame method for writing, `See the docs<io.stata>`  - Added module for reading and writing json format files: ``pandas.io.json``  accessible via ``read_json`` top-level function for reading,  and ``to_json`` DataFrame method for writing, `See the docs<io.json>`  various issues (`1226`, `3804`, `3876`, `3867`, `1305`)  - ``MultiIndex`` column support for reading and writing csv format files  - The ``header`` option in ``read_csv`` now accepts a  list of the rows from which to read the index.  - The option, ``tupleize_cols`` can now be specified in both ``to_csv`` and  ``read_csv``, to provide compatibility for the pre 0.12 behavior of  writing and reading ``MultIndex`` columns via a list of tuples. The default in  0.12 is to write lists of tuples and *not* interpret list of tuples as a  ``MultiIndex`` column.  Note: The default behavior in 0.12 remains unchanged from prior versions, but starting with 0.13,  the default *to* write and read ``MultiIndex`` columns will be in the new  format. (`3571`, `1651`, `3141`)  - If an ``index_col`` is not specified (e.g. you don't have an index, or wrote it  with ``df.to_csv(..., index=False``), then any ``names`` on the columns index will  be *lost*.  .. ipython:: python  from pandas._testing import makeCustomDataframe as mkdf  df = mkdf(5, 3, r_idx_nlevels=2, c_idx_nlevels=4)  df.to_csv("mi.csv")  print(open("mi.csv").read())  pd.read_csv("mi.csv", header=[0, 1, 2, 3], index_col=[0, 1])  .. ipython:: python  import os  os.remove("mi.csv")  - Support for ``HDFStore`` (via ``PyTables 3.0.0``) on Python3  - Iterator support via ``read_hdf`` that automatically opens and closes the  store when iteration is finished. This is only for *tables*  .. code-block:: ipython  In [25]: path = 'store_iterator.h5'  In [26]: pd.DataFrame(np.random.randn(10, 2)).to_hdf(path, 'df', table=True)  In [27]: for df in pd.read_hdf(path, 'df', chunksize=3):  ...:  print(df)  ...:  0  1  0  0.713216 -0.778461  1 -0.661062  0.862877  2  0.344342  0.149565  0  1  3 -0.626968 -0.875772  4 -0.930687 -0.218983  5  0.949965 -0.442354  0  1  6 -0.402985  1.111358  7 -0.241527 -0.670477  8  0.049355  0.632633  0  1  9 -1.502767 -1.225492  - ``read_csv`` will now throw a more informative error message when a file  contains no columns, e.g., all newline characters  Other enhancements

~~~  - ``DataFrame.replace()`` now allows regular expressions on contained  ``Series`` with object dtype. See the examples section in the regular docs  `Replacing via String Expression <missing_data.replace_expression>`  For example you can do  .. ipython:: python  df = pd.DataFrame({"a": list("ab.."), "b": [1, 2, 3, 4]})  df.replace(regex=r"\s*\.\s*", value=np.nan)  to replace all occurrences of the string ``'.'`` with zero or more  instances of surrounding white space with ``NaN``.  Regular string replacement still works as expected. For example, you can do  .. ipython:: python  df.replace(".", np.nan)  to replace all occurrences of the string ``'.'`` with ``NaN``.  - ``pd.melt()`` now accepts the optional parameters ``var_name`` and ``value_name``  to specify custom column names of the returned DataFrame.  - ``pd.set_option()`` now allows N option, value pairs (`3667`).  Let's say that we had an option ``'a.b'`` and another option ``'b.c'``.  We can set them at the same time:  .. code-block:: ipython  In [31]: pd.get_option('a.b')  Out[31]: 2  In [32]: pd.get_option('b.c')  Out[32]: 3  In [33]: pd.set_option('a.b', 1, 'b.c', 4)  In [34]: pd.get_option('a.b')  Out[34]: 1  In [35]: pd.get_option('b.c')  Out[35]: 4  - The ``filter`` method for group objects returns a subset of the original  object. Suppose we want to take only elements that belong to groups with a  group sum greater than 2.  .. ipython:: python  sf = pd.Series([1, 1, 2, 3, 3, 3])  sf.groupby(sf).filter(lambda x: x.sum() > 2)  The argument of ``filter`` must a function that, applied to the group as a  whole, returns ``True`` or ``False``.  Another useful operation is filtering out elements that belong to groups  with only a couple members.  .. ipython:: python  dff = pd.DataFrame({"A": np.arange(8), "B": list("aabbbcc")})  dff.groupby("B").filter(lambda x: len(x) > 2)  Alternatively, instead of dropping the offending groups, we can return a  like-indexed objects where the groups that do not pass the filter are  filled with NaNs.  .. ipython:: python  dff.groupby("B").filter(lambda x: len(x) > 2, dropna=False)  - Series and DataFrame hist methods now take a ``figsize`` argument (`3834`)  - DatetimeIndexes no longer try to convert mixed-integer indexes during join  operations (`3877`)  - Timestamp.min and Timestamp.max now represent valid Timestamp instances instead  of the default datetime.min and datetime.max (respectively), thanks @SleepingPills  - ``read_html`` now raises when no tables are found and BeautifulSoup==4.2.0  is detected (`4214`)  Experimental features
~~~  - Added experimental ``CustomBusinessDay`` class to support ``DateOffsets``  with custom holiday calendars and custom weekmasks. (`2301`)  .. note::  This uses the ``numpy.busdaycalendar`` API introduced in Numpy 1.7 and  therefore requires Numpy 1.7.0 or newer.  .. ipython:: python  from pandas.tseries.offsets import CustomBusinessDay  from datetime import datetime  # As an interesting example, let's look at Egypt where  # a Friday-Saturday weekend is observed.  weekmask_egypt = "Sun Mon Tue Wed Thu"  # They also observe International Workers' Day so let's  # add that for a couple of years  holidays = ["2012-05-01", datetime(2013, 5, 1), np.datetime64("2014-05-01")]  bday_egypt = CustomBusinessDay(holidays=holidays, weekmask=weekmask_egypt)  dt = datetime(2013, 4, 30)  print(dt + 2 * bday_egypt)  dts = pd.date_range(dt, periods=5, freq=bday_egypt)  print(pd.Series(dts.weekday, dts).map(pd.Series("Mon Tue Wed Thu Fri Sat Sun".split()))  Bug fixes

~~~  - Plotting functions now raise a ``TypeError`` before trying to plot anything  if the associated objects have a dtype of ``object`` (`1818`,  `3572`, `3911`, `3912`), but they will try to convert object arrays to  numeric arrays if possible so that you can still plot, for example, an  object array with floats. This happens before any drawing takes place which  eliminates any spurious plots from showing up.  - ``fillna`` methods now raise a ``TypeError`` if the ``value`` parameter is  a list or tuple.  - ``Series.str`` now supports iteration (`3638`). You can iterate over the  individual elements of each string in the ``Series``. Each iteration yields  a ``Series`` with either a single character at each index of the original  ``Series`` or ``NaN``. For example,  .. code-block:: ipython  In [38]: strs = "go", "bow", "joe", "slow"  In [32]: ds = pd.Series(strs)  In [33]: for s in ds.str:  ...:  print(s)  0  g  1  b  2  j  3  s  dtype: object  0  o  1  o  2  o  3  l  dtype: object  0  NaN  1  w  2  e  3  o  dtype: object  0  NaN  1  NaN  2  NaN  3  w  dtype: object  In [41]: s  Out[41]:  0  NaN  1  NaN  2  NaN  3  w  dtype: object  In [42]: s.dropna().values.item() == "w"  Out[42]: True  The last element yielded by the iterator will be a ``Series`` containing  the last element of the longest string in the ``Series`` with all other  elements being ``NaN``. Here since ``'slow'`` is the longest string  and there are no other strings with the same length ``'w'`` is the only  non-null string in the yielded ``Series``.  - ``HDFStore``  - will retain index attributes (freq,tz,name) on recreation (`3499`)  - will warn with a ``AttributeConflictWarning`` if you are attempting to append  an index with a different frequency than the existing, or attempting  to append an index with a different name than the existing  - support datelike columns with a timezone as data_columns (`2852`)  - Non-unique index support clarified (`3468`).  - Fix assigning a new index to a duplicate index in a DataFrame would fail (`3468`)  - Fix construction of a DataFrame with a duplicate index  - ref_locs support to allow duplicative indices across dtypes,  allows iget support to always find the index (even across dtypes) (`2194`)  - applymap on a DataFrame with a non-unique index now works  (removed warning) (`2786`), and fix (`3230`)  - Fix to_csv to handle non-unique columns (`3495`)  - Duplicate indexes with getitem will return items in the correct order (`3455`, `3457`)  and handle missing elements like unique indices (`3561`)  - Duplicate indexes with and empty DataFrame.from_records will return a correct frame (`3562`)  - Concat to produce a non-unique columns when duplicates are across dtypes is fixed (`3602`)  - Allow insert/delete to non-unique columns (`3679`)  - Non-unique indexing with a slice via ``loc`` and friends fixed (`3659`)  - Allow insert/delete to non-unique columns (`3679`)  - Extend ``reindex`` to correctly deal with non-unique indices (`3679`)  - ``DataFrame.itertuples()`` now works with frames with duplicate column  names (`3873`)  - Bug in non-unique indexing via ``iloc`` (`4017`); added ``takeable`` argument to  ``reindex`` for location-based taking  - Allow non-unique indexing in series via ``.ix/.loc`` and ``__getitem__`` (`4246`)  - Fixed non-unique indexing memory allocation issue with ``.ix/.loc`` (`4280`)  - ``DataFrame.from_records`` did not accept empty recarrays (`3682`)  - ``read_html`` now correctly skips tests (`3741`)  - Fixed a bug where ``DataFrame.replace`` with a compiled regular expression  in the ``to_replace`` argument wasn't working (`3907`)  - Improved ``network`` test decorator to catch ``IOError`` (and therefore  ``URLError`` as well). Added ``with_connectivity_check`` decorator to allow  explicitly checking a website as a proxy for seeing if there is network  connectivity. Plus, new ``optional_args`` decorator factory for decorators.  (`3910`, `3914`)  - Fixed testing issue where too many sockets where open thus leading to a  connection reset issue (`3982`, `3985`, `4028`,  `4054`)  - Fixed failing tests in test_yahoo, test_google where symbols were not  retrieved but were being accessed (`3982`, `3985`,  `4028`, `4054`)  - ``Series.hist`` will now take the figure from the current environment if  one is not passed  - Fixed bug where a 1xN DataFrame would barf on a 1xN mask (`4071`)  - Fixed running of ``tox`` under python3 where the pickle import was getting  rewritten in an incompatible way (`4062`, `4063`)  - Fixed bug where sharex and sharey were not being passed to grouped_hist  (`4089`)  - Fixed bug in ``DataFrame.replace`` where a nested dict wasn't being  iterated over when regex=False (`4115`)  - Fixed bug in the parsing of microseconds when using the ``format``  argument in ``to_datetime`` (`4152`)  - Fixed bug in ``PandasAutoDateLocator`` where

where a nested dict wasn't being  iterated over when regex=False (`4115`)  - Fixed bug in the parsing of microseconds when using the ``format``  argument in ``to_datetime`` (`4152`)  - Fixed bug in ``PandasAutoDateLocator`` where ``invert_xaxis`` triggered  incorrectly ``MilliSecondLocator``  (`3990`)  - Fixed bug in plotting that wasn't raising on invalid colormap for  matplotlib 1.1.1 (`4215`)  - Fixed the legend displaying in ``DataFrame.plot(kind='kde')`` (`4216`)  - Fixed bug where Index slices weren't carrying the name attribute  (`4226`)  - Fixed bug in initializing ``DatetimeIndex`` with an array of strings  in a certain time zone (`4229`)  - Fixed bug where html5lib wasn't being properly skipped (`4265`)  - Fixed bug where get_data_famafrench wasn't using the correct file edges  (`4281`)  See the `full release notes

on GitHub for a complete list.  .. _whatsnew_0.12.0.contributors:  Contributors
~~~  .. contributors:: v0.11.0..v0.12.0

{{ header }}  This is a major release from 0.12.0 and includes a number of API changes, several new features and
enhancements along with a large number of bug fixes.  Highlights include:  - support for a new index type ``Float64Index``, and other Indexing enhancements
- ``HDFStore`` has a new string based syntax for query specification
- support for new methods of interpolation
- updated ``timedelta`` operations
- a new string manipulation method ``extract``
- Nanosecond support for Offsets
- ``isin`` for DataFrames  Several experimental features are added, including:  - new ``eval/query`` methods for expression evaluation
- support for ``msgpack`` serialization
- an i/o interface to Google's ``BigQuery``  Their are several new or updated docs sections including:  - `Comparison with SQL<compare_with_sql>`, which should be useful for those familiar with SQL but still learning pandas.
- `Comparison with R<compare_with_r>`, idiom translations from R to pandas.
- `Enhancing Performance<enhancingperf>`, ways to enhance pandas performance with ``eval/query``.  .. warning::  In 0.13.0 ``Series`` has internally been refactored to no longer sub-class ``ndarray``  but instead subclass ``NDFrame``, similar to the rest of the pandas containers. This should be  a transparent change with only very limited API implications. See `Internal Refactoring<whatsnew_0130.refactoring>`  API changes
~~~  - ``read_excel`` now supports an integer in its ``sheetname`` argument giving  the index of the sheet to read in (`4301`).
- Text parser now treats anything that reads like inf ("inf", "Inf", "-Inf",  "iNf", etc.) as infinity. (`4220`, `4219`), affecting  ``read_table``, ``read_csv``, etc.
- ``pandas`` now is Python 2/3 compatible without the need for 2to3 thanks to  @jtratner. As a result, pandas now uses iterators more extensively. This  also led to the introduction of substantive parts of the Benjamin  Peterson's ``six`` library into compat. (`4384`, `4375`,  `4372`)
- ``pandas.util.compat`` and ``pandas.util.py3compat`` have been merged into  ``pandas.compat``. ``pandas.compat`` now includes many functions allowing  2/3 compatibility. It contains both list and iterator versions of range,  filter, map and zip, plus other necessary elements for Python 3  compatibility. ``lmap``, ``lzip``, ``lrange`` and ``lfilter`` all produce  lists instead of iterators, for compatibility with ``numpy``, subscripting  and ``pandas`` constructors.(`4384`, `4375`, `4372`)
- ``Series.get`` with negative indexers now returns the same as ``[]`` (`4390`)
- Changes to how ``Index`` and ``MultiIndex`` handle metadata (``levels``,  ``labels``, and ``names``) (`4039`):  .. code-block:: python  # previously, you would have set levels or labels directly  >>> pd.index.levels = [[1, 2, 3, 4], [1, 2, 4, 4]]  # now, you use the set_levels or set_labels methods  >>> index = pd.index.set_levels([[1, 2, 3, 4], [1, 2, 4, 4]])  # similarly, for names, you can rename the object  # but setting names is not deprecated  >>> index = pd.index.set_names(["bob", "cranberry"])  # and all methods take an inplace kwarg - but return None  >>> pd.index.set_names(["bob", "cranberry"], inplace=True)  - **All** division with ``NDFrame`` objects is now *truedivision*, regardless  of the future import. This means that operating on pandas objects will by default  use *floating point* division, and return a floating point dtype.  You can use ``//`` and ``floordiv`` to do integer division.  Integer division  .. code-block:: ipython  In [3]: arr = np.array([1, 2, 3, 4])  In [4]: arr2 = np.array([5, 3, 2, 1])  In [5]: arr / arr2  Out[5]: array([0, 0, 1, 4])  In [6]: pd.Series(arr) // pd.Series(arr2)  Out[6]:  0  0  1  0  2  1  3  4  dtype: int64  True Division  .. code-block:: ipython  In [7]: pd.Series(arr) / pd.Series(arr2)  # no future import required  Out[7]:  0  0.2000  1  0.6667  2  1.5000  3  4.000  dtype: float64  - Infer and downcast dtype if ``downcast='infer'`` is passed to ``fillna/ffill/bfill`` (`4604`)

- ``__nonzero__`` for all NDFrame objects, will now raise a ``ValueError``, this reverts back to (`1073`, `4633`)  behavior. See `gotchas<gotchas.truth>` for a more detailed discussion.  This prevents doing boolean comparison on *entire* pandas objects, which is inherently ambiguous. These all will raise a ``ValueError``.  .. code-block:: python  >>> df = pd.DataFrame({'A': np.random.randn(10),  ...  'B': np.random.randn(10),  ...  'C': pd.date_range('20130101', periods=10)  ...  })  ...  >>> if df:  ...  pass  ...  Traceback (most recent call last):  ...  ValueError: The truth value of a DataFrame is ambiguous.  Use a.empty,  a.bool(), a.item(), a.any() or a.all().  >>> df1 = df  >>> df2 = df  >>> df1 and df2  Traceback (most recent call last):  ...  ValueError: The truth value of a DataFrame is ambiguous.  Use a.empty,  a.bool(), a.item(), a.any() or a.all().  >>> d = [1, 2, 3]  >>> s1 = pd.Series(d)  >>> s2 = pd.Series(d)  >>> s1 and s2  Traceback (most recent call last):  ...  ValueError: The truth value of a DataFrame is ambiguous.  Use a.empty,  a.bool(), a.item(), a.any() or a.all().  Added the ``.bool()`` method to ``NDFrame`` objects to facilitate evaluating of single-element boolean Series:  .. ipython:: python  pd.Series([True]).bool()  pd.Series([False]).bool()  pd.DataFrame([[True]]).bool()  pd.DataFrame([[False]]).bool()  - All non-Index NDFrames (``Series``, ``DataFrame``, ``Panel``, ``Panel4D``,  ``SparsePanel``, etc.), now support the entire set of arithmetic operators  and arithmetic flex methods (add, sub, mul, etc.). ``SparsePanel`` does not  support ``pow`` or ``mod`` with non-scalars. (`3765`)
- ``Series`` and ``DataFrame`` now have a ``mode()`` method to calculate the  statistical mode(s) by axis/Series. (`5367`)  - Chained assignment will now by default warn if the user is assigning to a copy. This can be changed  with the option ``mode.chained_assignment``, allowed options are ``raise/warn/None``. See `the docs<indexing.view_versus_copy>`.  .. ipython:: python  dfc = pd.DataFrame({'A': ['aaa', 'bbb', 'ccc'], 'B': [1, 2, 3]})  pd.set_option('chained_assignment', 'warn')  The following warning / exception will show if this is attempted.  .. ipython:: python  dfc.loc[0]['A'] = 111  ::  Traceback (most recent call last)  ...  SettingWithCopyWarning:  A value is trying to be set on a copy of a slice from a DataFrame.  Try using .loc[row_index,col_indexer] = value instead  Here is the correct method of assignment.  .. ipython:: python  dfc.loc[0, 'A'] = 11  dfc  - ``Panel.reindex`` has the following call signature ``Panel.reindex(items=None, major_axis=None, minor_axis=None, **kwargs)``  to conform with other ``NDFrame`` objects. See `Internal Refactoring<whatsnew_0130.refactoring>` for more information.  - ``Series.argmin`` and ``Series.argmax`` are now aliased to ``Series.idxmin`` and ``Series.idxmax``. These return the *index* of the  min or max element respectively. Prior to 0.13.0 these would return the position of the min / max element. (`6214`)  Prior version deprecations/changes
~~~  These were announced changes in 0.12 or prior that are taking effect as of 0.13.0  - Remove deprecated ``Factor`` (`3650`)
- Remove deprecated ``set_printoptions/reset_printoptions`` (`3046`)
- Remove deprecated ``_verbose_info`` (`3215`)
- Remove deprecated ``read_clipboard/to_clipboard/ExcelFile/ExcelWriter`` from ``pandas.io.parsers`` (`3717`)  These are available as functions in the main pandas namespace (e.g. ``pd.read_clipboard``)
- default for ``tupleize_cols`` is now ``False`` for both ``to_csv`` and ``read_csv``. Fair warning in 0.12 (`3604`)
- default for ``display.max_seq_len`` is now 100 rather than ``None``. This activates  truncated display ("...") of long sequences in various places. (`3391`)  Deprecations
~~~  Deprecated in 0.13.0  - deprecated ``iterkv``, which will be removed in a future release (this was  an alias of iteritems used to bypass ``2to3``'s changes).  (`4384`, `4375`, `4372`)

- deprecated the string method ``match``, whose role is now performed more  idiomatically by ``extract``. In a future release, the default behavior  of ``match`` will change to become analogous to ``contains``, which returns  a boolean indexer. (Their  distinction is strictness: ``match`` relies on ``re.match`` while  ``contains`` relies on ``re.search``.) In this release, the deprecated  behavior is the default, but the new behavior is available through the  keyword argument ``as_indexer=True``.  Indexing API changes
~~~  Prior to 0.13, it was impossible to use a label indexer (``.loc/.ix``) to set a value that
was not contained in the index of a particular axis. (`2578`). See `the docs<indexing.basics.partial_setting>`  In the ``Series`` case this is effectively an appending operation  .. ipython:: python  s = pd.Series([1, 2, 3])  s  s[5] = 5.  s  .. ipython:: python  dfi = pd.DataFrame(np.arange(6).reshape(3, 2),  columns=['A', 'B'])  dfi  This would previously ``KeyError``  .. ipython:: python  dfi.loc[:, 'C'] = dfi.loc[:, 'A']  dfi  This is like an ``append`` operation.  .. ipython:: python  dfi.loc[3] = 5  dfi  A Panel setting operation on an arbitrary axis aligns the input to the Panel  .. code-block:: ipython  In [20]: p = pd.Panel(np.arange(16).reshape(2, 4, 2),  ...:  items=['Item1', 'Item2'],  ...:  major_axis=pd.date_range('2001/1/12', periods=4),  ...:  minor_axis=['A', 'B'], dtype='float64')  ...:  In [21]: p  Out[21]:  <class 'pandas.core.panel.Panel'>  Dimensions: 2 (items) x 4 (major_axis) x 2 (minor_axis)  Items axis: Item1 to Item2  Major_axis axis: 2001-01-12 000 to 2001-01-15 000  Minor_axis axis: A to B  In [22]: p.loc[:, :, 'C'] = pd.Series([30, 32], index=p.items)  In [23]: p  Out[23]:  <class 'pandas.core.panel.Panel'>  Dimensions: 2 (items) x 4 (major_axis) x 3 (minor_axis)  Items axis: Item1 to Item2  Major_axis axis: 2001-01-12 000 to 2001-01-15 000  Minor_axis axis: A to C  In [24]: p.loc[:, :, 'C']  Out[24]:  Item1  Item2  2001-01-12  30.0  32.0  2001-01-13  30.0  32.0  2001-01-14  30.0  32.0  2001-01-15  30.0  32.0  Float64Index API change
~~~  - Added a new index type, ``Float64Index``. This will be automatically created when passing floating values in index creation.  This enables a pure label-based slicing paradigm that makes ``[],ix,loc`` for scalar indexing and slicing work exactly the  same. (`263`)  Construction is by default for floating type values.  .. ipython:: python  index = pd.Index([1.5, 2, 3, 4.5, 5])  index  s = pd.Series(range(5), index=index)  s  Scalar selection for ``[],.ix,.loc`` will always be label based. An integer will match an equal float index (e.g. ``3`` is equivalent to ``3.0``)  .. ipython:: python  s[3]  s.loc[3]  The only positional indexing is via ``iloc``  .. ipython:: python  s.iloc[3]  A scalar index that is not found will raise ``KeyError``  Slicing is ALWAYS on the values of the index, for ``[],ix,loc`` and ALWAYS positional with ``iloc``  .. ipython:: python  s[2:4]  s.loc[2:4]  s.iloc[2:4]  In float indexes, slicing using floats are allowed  .. ipython:: python  s[2.1:4.6]  s.loc[2.1:4.6]  - Indexing on other index types are preserved (and positional fallback for ``[],ix``), with the exception, that floating point slicing  on indexes on non ``Float64Index`` will now raise a ``TypeError``.  .. code-block:: ipython  In [1]: pd.Series(range(5))[3.5]  TypeError: the label [3.5] is not a proper indexer for this index type (Int64Index)  In [1]: pd.Series(range(5))[3.5:4.5]  TypeError: the slice start [3.5] is not a proper indexer for this index type (Int64Index)  Using a scalar float indexer will be deprecated in a future version, but is allowed for now.  .. code-block:: ipython  In [3]: pd.Series(range(5))[3.0]  Out[3]: 3  HDFStore API changes

~~~  - Query Format Changes. A much more string-like query format is now supported. See `the docs<io.hdf5-query>`.  .. ipython:: python  path = 'test.h5'  dfq = pd.DataFrame(np.random.randn(10, 4),  columns=list('ABCD'),  index=pd.date_range('20130101', periods=10))  dfq.to_hdf(path, 'dfq', format='table', data_columns=True)  Use boolean expressions, with in-line function evaluation.  .. ipython:: python  pd.read_hdf(path, 'dfq',  where="index>Timestamp('20130104') & columns=['A', 'B']")  Use an inline column reference  .. ipython:: python  pd.read_hdf(path, 'dfq',  where="A>0 or C>0")  .. ipython:: python  import os  os.remove(path)  - the ``format`` keyword now replaces the ``table`` keyword; allowed values are ``fixed(f)`` or ``table(t)``  the same defaults as prior < 0.13.0 remain, e.g. ``put`` implies ``fixed`` format and ``append`` implies  ``table`` format. This default format can be set as an option by setting ``io.hdf.default_format``.  .. ipython:: python  path = 'test.h5'  df = pd.DataFrame(np.random.randn(10, 2))  df.to_hdf(path, 'df_table', format='table')  df.to_hdf(path, 'df_table2', append=True)  df.to_hdf(path, 'df_fixed')  with pd.HDFStore(path) as store:  print(store)  .. ipython:: python  import os  os.remove(path)  - Significant table writing performance improvements
- handle a passed ``Series`` in table format (`4330`)
- can now serialize a ``timedelta64[ns]`` dtype in a table (`3577`), See `the docs<io.hdf5-timedelta>`.
- added an ``is_open`` property to indicate if the underlying file handle is_open;  a closed store will now report 'CLOSED' when viewing the store (rather than raising an error)  (`4409`)
- a close of a ``HDFStore`` now will close that instance of the ``HDFStore``  but will only close the actual file if the ref count (by ``PyTables``) w.r.t. all of the open handles  are 0. Essentially you have a local instance of ``HDFStore`` referenced by a variable. Once you  close it, it will report closed. Other references (to the same file) will continue to operate  until they themselves are closed. Performing an action on a closed file will raise  ``ClosedFileError``  .. ipython:: python  path = 'test.h5'  df = pd.DataFrame(np.random.randn(10, 2))  store1 = pd.HDFStore(path)  store2 = pd.HDFStore(path)  store1.append('df', df)  store2.append('df2', df)  store1  store2  store1.close()  store2  store2.close()  store2  .. ipython:: python  import os  os.remove(path)  - removed the ``_quiet`` attribute, replace by a ``DuplicateWarning`` if retrieving  duplicate rows from a table (`4367`)
- removed the ``warn`` argument from ``open``. Instead a ``PossibleDataLossError`` exception will  be raised if you try to use ``mode='w'`` with an OPEN file handle (`4367`)
- allow a passed locations array or mask as a ``where`` condition (`4467`).  See `the docs<io.hdf5-where_mask>` for an example.
- add the keyword ``dropna=True`` to ``append`` to change whether ALL nan rows are not written  to the store (default is ``True``, ALL nan rows are NOT written), also settable  via the option ``io.hdf.dropna_table`` (`4625`)
- pass through store creation arguments; can be used to support in-memory stores  DataFrame repr changes
~~~  The HTML and plain text representations of `DataFrame` now show
a truncated view of the table once it exceeds a certain size, rather
than switching to the short info view (`4886`, `5550`).
This makes the representation more consistent as small DataFrames get
larger.  Truncated HTML representation of a DataFrame  To get the info view, call `DataFrame.info`. If you prefer the
info view as the repr for large DataFrames, you can set this by running
``set_option('display.large_repr', 'info')``.  Enhancements
~~~  - ``df.to_clipboard()`` learned a new ``excel`` keyword that let's you  paste df data directly into excel (enabled by default). (`5070`).
- ``read_html`` now raises a ``URLError`` instead of catching and raising a  ``ValueError`` (`4303`, `4305`)

- Added a test for ``read_clipboard()`` and ``to_clipboard()`` (`4282`)
- Clipboard functionality now works with PySide (`4282`)
- Added a more informative error message when plot arguments contain  overlapping color and style arguments (`4402`)

- ``to_dict`` now takes ``records`` as a possible out type.  Returns an array  of column-keyed dictionaries. (`4936`)  - ``NaN`` handing in get_dummies (`4446`) with ``dummy_na``  .. ipython:: python  # previously, nan was erroneously counted as 2 here  # now it is not counted at all  pd.get_dummies([1, 2, np.nan])  # unless requested  pd.get_dummies([1, 2, np.nan], dummy_na=True)  - ``timedelta64[ns]`` operations. See `the docs<timedeltas.timedeltas_convert>`.  .. warning::  Most of these operations require ``numpy >= 1.7``  Using the new top-level ``to_timedelta``, you can convert a scalar or array from the standard  timedelta format (produced by ``to_csv``) into a timedelta type (``np.timedelta64`` in ``nanoseconds``).  .. ipython:: python  pd.to_timedelta('1 days 0601.0003')  pd.to_timedelta('15.5us')  pd.to_timedelta(['1 days 0601.0003', '15.5us', 'nan'])  pd.to_timedelta(np.arange(5), unit='s')  pd.to_timedelta(np.arange(5), unit='d')  A Series of dtype ``timedelta64[ns]`` can now be divided by another  ``timedelta64[ns]`` object, or astyped to yield a ``float64`` dtyped Series. This  is frequency conversion. See `the docs<timedeltas.timedeltas_convert>` for the docs.  .. ipython:: python  import datetime  td = pd.Series(pd.date_range('20130101', periods=4)) - pd.Series(  pd.date_range('20121201', periods=4))  td[2] += np.timedelta64(datetime.timedelta(minutes=5, seconds=3))  td[3] = np.nan  td  # to days  td / np.timedelta64(1, 'D')  td.astype('timedelta64[D]')  # to seconds  td / np.timedelta64(1, 's')  td.astype('timedelta64[s]')  Dividing or multiplying a ``timedelta64[ns]`` Series by an integer or integer Series  .. ipython:: python  td * -1  td * pd.Series([1, 2, 3, 4])  Absolute ``DateOffset`` objects can act equivalently to ``timedeltas``  .. ipython:: python  from pandas import offsets  td + offsets.Minute(5) + offsets.Milli(5)  Fillna is now supported for timedeltas  .. ipython:: python  td.fillna(pd.Timedelta(0))  td.fillna(datetime.timedelta(days=1, seconds=5))  You can do numeric reduction operations on timedeltas.  .. ipython:: python  td.mean()  td.quantile(.1)  - ``plot(kind='kde')`` now accepts the optional parameters ``bw_method`` and  ``ind``, passed to scipy.stats.gaussian_kde() (for scipy >= 0.11.0) to set  the bandwidth, and to gkde.evaluate() to specify the indices at which it  is evaluated, respectively. See scipy docs. (`4298`)  - DataFrame constructor now accepts a numpy masked record array (`3478`)  - The new vectorized string method ``extract`` return regular expression  matches more conveniently.  .. ipython:: python  pd.Series(['a1', 'b2', 'c3']).str.extract('[ab](\\d)')  Elements that do not match return ``NaN``. Extracting a regular expression  with more than one group returns a DataFrame with one column per group.  .. ipython:: python  pd.Series(['a1', 'b2', 'c3']).str.extract('([ab])(\\d)')  Elements that do not match return a row of ``NaN``.  Thus, a Series of messy strings can be *converted* into a  like-indexed Series or DataFrame of cleaned-up or more useful strings,  without necessitating ``get()`` to access tuples or ``re.match`` objects.  Named groups like  .. ipython:: python  pd.Series(['a1', 'b2', 'c3']).str.extract(  '(?P<letter>[ab])(?P<digit>\\d)')  and optional groups can also be used.  .. ipython:: python  pd.Series(['a1', 'b2', '3']).str.extract(  '(?P<letter>[ab])?(?P<digit>\\d)')  - ``read_stata`` now accepts Stata 13 format (`4291`)  - ``read_fwf`` now infers the column specifications from the first 100 rows of  the file if the data has correctly separated and properly aligned columns  using the delimiter provided to the function (`4488`).  - support for nanosecond times as an offset  .. warning::  These operations require ``numpy >= 1.7``  Period conversions in the range of seconds and below were reworked and extended  up to nanoseconds. Periods in the nanosecond range are now available.  .. ipython:: python  pd.date_range('2013-01-01', periods=5, freq='5N')  or with frequency as offset  .. ipython:: python  pd.date_range('2013-01-01', periods=5, freq=pd.offsets.Nano(5))  Timestamps can be modified in the nanosecond range  .. ipython:: python  t = pd.Timestamp('20130101 0902')  t + pd.tseries.offsets.Nano(123)  - A new method, ``isin`` for DataFrames, which plays nicely with boolean indexing. The argument to ``isin``, what we're comparing the DataFrame to, can be a DataFrame, Series, dict, or array of values. See `the docs<indexing.basics.indexing_isin>` for more.  To get the rows where any of the conditions are met:  .. ipython:: python  dfi = pd.DataFrame({'A': [1, 2, 3, 4], 'B': ['a', 'b', 'f', 'n']})  dfi  other = pd.DataFrame({'A': [1, 3, 3, 7], 'B':

docs<indexing.basics.indexing_isin>` for more.  To get the rows where any of the conditions are met:  .. ipython:: python  dfi = pd.DataFrame({'A': [1, 2, 3, 4], 'B': ['a', 'b', 'f', 'n']})  dfi  other = pd.DataFrame({'A': [1, 3, 3, 7], 'B': ['e', 'f', 'f', 'e']})  mask = dfi.isin(other)  mask  dfi[mask.any(axis=1)]  - ``Series`` now supports a ``to_frame`` method to convert it to a single-column DataFrame (`5164`)  - All R datasets listed here http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/00Index.html can now be loaded into pandas objects  .. code-block:: python  # note that pandas.rpy was deprecated in v0.16.0  import pandas.rpy.common as com  com.load_data('Titanic')  - ``tz_localize`` can infer a fall daylight savings transition based on the structure  of the unlocalized data (`4230`), see `the docs<timeseries.timezone>`  - ``DatetimeIndex`` is now in the API documentation, see `the docs<api.datetimeindex>`  - `~pandas.io.json.json_normalize` is a new method to allow you to create a flat table  from semi-structured JSON data. See `the docs<io.json_normalize>` (`1067`)  - Added PySide support for the qtpandas DataFrameModel and DataFrameWidget.  - Python csv parser now supports usecols (`4335`)  - Frequencies gained several new offsets:  * ``LastWeekOfMonth`` (`4637`)  * ``FY5253``, and ``FY5253Quarter`` (`4511`)  - DataFrame has a new ``interpolate`` method, similar to Series (`4434`, `1892`)  .. ipython:: python  df = pd.DataFrame({'A': [1, 2.1, np.nan, 4.7, 5.6, 6.8],  'B': [.25, np.nan, np.nan, 4, 12.2, 14.4]})  df.interpolate()  Additionally, the ``method`` argument to ``interpolate`` has been expanded  to include ``'nearest', 'zero', 'slinear', 'quadratic', 'cubic',  'barycentric', 'krogh', 'piecewise_polynomial', 'pchip', 'polynomial', 'spline'``  The new methods require scipy_. Consult the Scipy reference guide_ and documentation_ for more information  about when the various methods are appropriate. See `the docs<missing_data.interpolate>`.  Interpolate now also accepts a ``limit`` keyword argument.  This works similar to ``fillna``'s limit:  .. ipython:: python  ser = pd.Series([1, 3, np.nan, np.nan, np.nan, 11])  ser.interpolate(limit=2)  - Added ``wide_to_long`` panel data convenience function. See `the docs<reshaping.melt>`.  .. ipython:: python  np.random.seed(123)  df = pd.DataFrame({"A1970" : {0 : "a", 1 : "b", 2 : "c"},  "A1980" : {0 : "d", 1 : "e", 2 : "f"},  "B1970" : {0 : 2.5, 1 : 1.2, 2 : .7},  "B1980" : {0 : 3.2, 1 : 1.3, 2 : .1},  "X"  : dict(zip(range(3), np.random.randn(3)))  })  df["id"] = df.index  df  pd.wide_to_long(df, ["A", "B"], i="id", j="year")  .. _scipy: http://www.scipy.org

.. _documentation: http://docs.scipy.org/doc/scipy/reference/interpolate.html#univariate-interpolation
.. _guide: https://docs.scipy.org/doc/scipy/tutorial/interpolate.html  - ``to_csv`` now takes a ``date_format`` keyword argument that specifies how  output datetime objects should be formatted. Datetimes encountered in the  index, columns, and values will all have this formatting applied. (`4313`)
- ``DataFrame.plot`` will scatter plot x versus y by passing ``kind='scatter'`` (`2215`)
- Added support for Google Analytics v3 API segment IDs that also supports v2 IDs. (`5271`)  .. _whatsnew_0130.experimental:  Experimental

~~~  - The new `~pandas.eval` function implements expression evaluation using  ``numexpr`` behind the scenes. This results in large speedups for  complicated expressions involving large DataFrames/Series. For example,  .. ipython:: python  nrows, ncols = 2000, 100  df1, df2, df3, df4 = [pd.DataFrame(np.random.randn(nrows, ncols))  for _ in range(4)]  .. ipython:: python  # eval with NumExpr backend  %timeit pd.eval('df1 + df2 + df3 + df4')  .. ipython:: python  # pure Python evaluation  %timeit df1 + df2 + df3 + df4  For more details, see the `the docs<enhancingperf.eval>`  - Similar to ``pandas.eval``, `~pandas.DataFrame` has a new  ``DataFrame.eval`` method that evaluates an expression in the context of  the ``DataFrame``. For example,  .. ipython:: python  try:  del a  # noqa: F821  except NameError:  pass  try:  del b  # noqa: F821  except NameError:  pass  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 2), columns=['a', 'b'])  df.eval('a + b')  - `~pandas.DataFrame.query` method has been added that allows  you to select elements of a ``DataFrame`` using a natural query syntax  nearly identical to Python syntax. For example,  .. ipython:: python  try:  del a  # noqa: F821  except NameError:  pass  try:  del b  # noqa: F821  except NameError:  pass  try:  del c  # noqa: F821  except NameError:  pass  .. ipython:: python  n = 20  df = pd.DataFrame(np.random.randint(n, size=(n, 3)), columns=['a', 'b', 'c'])  df.query('a < b < c')  selects all the rows of ``df`` where ``a < b < c`` evaluates to ``True``.  For more details see the `the docs<indexing.query>`.  - ``pd.read_msgpack()`` and ``pd.to_msgpack()`` are now a supported method of serialization  of arbitrary pandas (and python objects) in a lightweight portable binary format. See `the docs<io.msgpack>`  .. warning::  Since this is an EXPERIMENTAL LIBRARY, the storage format may not be stable until a future release.  .. code-block:: python  df = pd.DataFrame(np.random.rand(5, 2), columns=list('AB'))  df.to_msgpack('foo.msg')  pd.read_msgpack('foo.msg')  s = pd.Series(np.random.rand(5), index=pd.date_range('20130101', periods=5))  pd.to_msgpack('foo.msg', df, s)  pd.read_msgpack('foo.msg')  You can pass ``iterator=True`` to iterator over the unpacked results  .. code-block:: python  for o in pd.read_msgpack('foo.msg', iterator=True):  print(o)  .. ipython:: python  os.remove('foo.msg')  - ``pandas.io.gbq`` provides a simple way to extract from, and load data into,  Google's BigQuery Data Sets by way of pandas DataFrames. BigQuery is a high  performance SQL-like database service, useful for performing ad-hoc queries  against extremely large datasets. `See the docs <io.bigquery>`  .. code-block:: python  from pandas.io import gbq  # A query to select the average monthly temperatures in the  # in the year 2000 across the USA. The dataset,  # publicata:samples.gsod, is available on all BigQuery accounts,  # and is based on NOAA gsod data.  query = """SELECT station_number as STATION,  month as MONTH, AVG(mean_temp) as MEAN_TEMP  FROM publicdata:samples.gsod  WHERE YEAR = 2000  GROUP BY STATION, MONTH  ORDER BY STATION, MONTH ASC"""  # Fetch the result set for this query  # Your Google BigQuery Project ID  # To find this, see your dashboard:  # https://console.developers.google.com/iam-admin/projects?authuser=0  projectid = 'xxx'  df = gbq.read_gbq(query, project_id=projectid)  # Use pandas to process and reshape the dataset  df2 = df.pivot(index='STATION', columns='MONTH', values='MEAN_TEMP')  df3 = pd.concat([df2.min(), df2.mean(), df2.max()],  axis=1, keys=["Min Tem", "Mean Temp", "Max Temp"])  The resulting DataFrame is::  > df3  Min Tem  Mean Temp  Max Temp  MONTH  1  -53.336667  39.827892  89.770968  2  -49.837500  43.685219  93.437932  3  -77.926087  48.708355  96.09998  4  -82.892858  55.070087  97.317240  5  -92.378261  61.428117  102.042856  6  -77.703334  65.85888  102.9000  7  -87.821428  68.169663  106.510714  8  -89.431999  68.614215  105.5000  9  -86.61112  63.436935  107.142856  10  -78.209677  56.880838  92.10333  11  -50.125000  48.861228  94.996428  12  -50.332258  42.286879  94.396774  .. warning::  To use this module, you will need a BigQuery account. See  <https://cloud.google.com/products/big-query> for details.  As of 10/10/13, there is a bug in Google's API preventing result sets  from being larger than 100,000 rows. A patch is scheduled for the week of  10/14/13.  .. _whatsnew_0130.refactoring:  Internal refactoring

~~~  In 0.13.0 there is a major refactor primarily to subclass ``Series`` from
``NDFrame``, which is the base class currently for ``DataFrame`` and ``Panel``,
to unify methods and behaviors. Series formerly subclassed directly from
``ndarray``. (`4080`, `3862`, `816`)  .. warning::  There are two potential incompatibilities from < 0.13.0  - Using certain numpy functions would previously return a ``Series`` if passed a ``Series``  as an argument. This seems only to affect ``np.ones_like``, ``np.empty_like``,  ``np.diff`` and ``np.where``. These now return ``ndarrays``.  .. ipython:: python  s = pd.Series([1, 2, 3, 4])  Numpy Usage  .. ipython:: python  np.ones_like(s)  np.diff(s)  np.where(s > 1, s, np.nan)  Pandonic Usage  .. ipython:: python  pd.Series(1, index=s.index)  s.diff()  s.where(s > 1)  - Passing a ``Series`` directly to a cython function expecting an ``ndarray`` type will no  long work directly, you must pass ``Series.values``, See `Enhancing Performance<enhancingperf.ndarray>`  - ``Series(0.5)`` would previously return the scalar ``0.5``, instead this will return a 1-element ``Series``  - This change breaks ``rpy2<=2.3.8``. an Issue has been opened against rpy2 and a workaround  is detailed in `5698`. Thanks @JanSchulz.  - Pickle compatibility is preserved for pickles created prior to 0.13. These must be unpickled with ``pd.read_pickle``, see `Pickling<io.pickle>`.  - Refactor of series.py/frame.py/panel.py to move common code to generic.py  - added ``_setup_axes`` to created generic NDFrame structures  - moved methods  - ``from_axes,_wrap_array,axes,ix,loc,iloc,shape,empty,swapaxes,transpose,pop``  - ``__iter__,keys,__contains__,__len__,__neg__,__invert__``  - ``convert_objects,as_blocks,as_matrix,values``  - ``__getstate__,__setstate__`` (compat remains in frame/panel)  - ``__getattr__,__setattr__``  - ``_indexed_same,reindex_like,align,where,mask``  - ``fillna,replace`` (``Series`` replace is now consistent with ``DataFrame``)  - ``filter`` (also added axis argument to selectively filter on a different axis)  - ``reindex,reindex_axis,take``  - ``truncate`` (moved to become part of ``NDFrame``)  - These are API changes which make ``Panel`` more consistent with ``DataFrame``  - ``swapaxes`` on a ``Panel`` with the same axes specified now return a copy  - support attribute access for setting  - filter supports the same API as the original ``DataFrame`` filter  - Reindex called with no arguments will now return a copy of the input object  - ``TimeSeries`` is now an alias for ``Series``. the property ``is_time_series``  can be used to distinguish (if desired)  - Refactor of Sparse objects to use BlockManager  - Created a new block type in internals, ``SparseBlock``, which can hold multi-dtypes  and is non-consolidatable. ``SparseSeries`` and ``SparseDataFrame`` now inherit  more methods from there hierarchy (Series/DataFrame), and no longer inherit  from ``SparseArray`` (which instead is the object of the ``SparseBlock``)  - Sparse suite now supports integration with non-sparse data. Non-float sparse  data is supportable (partially implemented)  - Operations on sparse structures within DataFrames should preserve sparseness,  merging type operations will convert to dense (and back to sparse), so might  be somewhat inefficient  - enable setitem on ``SparseSeries`` for boolean/integer/slices  - ``SparsePanels`` implementation is unchanged (e.g. not using BlockManager, needs work)  - added ``ftypes`` method to Series/DataFrame, similar to ``dtypes``, but indicates  if the underlying is sparse/dense (as well as the dtype)
- All ``NDFrame`` objects can now use ``__finalize__()`` to specify various  values to propagate to new objects from an existing one (e.g. ``name`` in ``Series`` will  follow more automatically now)
- Internal type checking is now done via a suite of generated classes, allowing ``isinstance(value, klass)``  without having to directly import the klass, courtesy of @jtratner

- Bug in Series update where the parent frame is not updating its cache based on  changes (`4080`) or types (`3217`), fillna (`3386`)
- Indexing with dtype conversions fixed (`4463`, `4204`)
- Refactor ``Series.reindex`` to core/generic.py (`4604`, `4618`), allow ``method=`` in reindexing  on a Series to work
- ``Series.copy`` no longer accepts the ``order`` parameter and is now consistent with ``NDFrame`` copy
- Refactor ``rename`` methods to core/generic.py; fixes ``Series.rename`` for (`4605`), and adds ``rename``  with the same signature for ``Panel``
- Refactor ``clip`` methods to core/generic.py (`4798`)
- Refactor of ``_get_numeric_data/_get_bool_data`` to core/generic.py, allowing Series/Panel functionality
- ``Series`` (for index) / ``Panel`` (for items) now allow attribute access to its elements  (`1903`)  .. ipython:: python  s = pd.Series([1, 2, 3], index=list('abc'))  s.b  s.a = 5  s  .. _release.bug_fixes-0.13.0:  Bug fixes
~~~  - ``HDFStore``  - raising an invalid ``TypeError`` rather than ``ValueError`` when  appending with a different block ordering (`4096`)  - ``read_hdf`` was not respecting as passed ``mode`` (`4504`)  - appending a 0-len table will work correctly (`4273`)  - ``to_hdf`` was raising when passing both arguments ``append`` and  ``table`` (`4584`)  - reading from a store with duplicate columns across dtypes would raise  (`4767`)  - Fixed a bug where ``ValueError`` wasn't correctly raised when column  names weren't strings (`4956`)  - A zero length series written in Fixed format not deserializing properly.  (`4708`)  - Fixed decoding perf issue on pyt3 (`5441`)  - Validate levels in a MultiIndex before storing (`5527`)  - Correctly handle ``data_columns`` with a Panel (`5717`)
- Fixed bug in tslib.tz_convert(vals, tz1, tz2): it could raise IndexError  exception while trying to access trans[pos + 1] (`4496`)
- The ``by`` argument now works correctly with the ``layout`` argument  (`4102`, `4014`) in ``*.hist`` plotting methods
- Fixed bug in ``PeriodIndex.map`` where using ``str`` would return the str  representation of the index (`4136`)
- Fixed test failure ``test_time_series_plot_color_with_empty_kwargs`` when  using custom matplotlib default colors (`4345`)
- Fix running of stata IO tests. Now uses temporary files to write  (`4353`)
- Fixed an issue where ``DataFrame.sum`` was slower than ``DataFrame.mean``  for integer valued frames (`4365`)
- ``read_html`` tests now work with Python 2.6 (`4351`)
- Fixed bug where ``network`` testing was throwing ``NameError`` because a  local variable was undefined (`4381`)
- In ``to_json``, raise if a passed ``orient`` would cause loss of data  because of a duplicate index (`4359`)
- In ``to_json``, fix date handling so milliseconds are the default timestamp  as the docstring says (`4362`).
- ``as_index`` is no longer ignored when doing groupby apply (`4648`,  `3417`)
- JSON NaT handling fixed, NaTs are now serialized to ``null`` (`4498`)
- Fixed JSON handling of escapable characters in JSON object keys  (`4593`)
- Fixed passing ``keep_default_na=False`` when ``na_values=None``  (`4318`)
- Fixed bug with ``values`` raising an error on a DataFrame with duplicate  columns and mixed dtypes, surfaced in (`4377`)
- Fixed bug with duplicate columns and type conversion in ``read_json`` when  ``orient='split'`` (`4377`)
- Fixed JSON bug where locales with decimal separators other than '.' threw  exceptions when encoding / decoding certain values. (`4918`)
- Fix ``.iat`` indexing with a ``PeriodIndex`` (`4390`)
- Fixed an issue where ``PeriodIndex`` joining with self was returning a new  instance rather than the same instance (`4379`); also adds a test  for this for the other index types
- Fixed a bug with all the dtypes being converted to object when using the  CSV cparser with the usecols parameter (`3192`)

- Fix an issue in merging blocks where the resulting DataFrame had partially  set _ref_locs (`4403`)
- Fixed an issue where hist subplots were being overwritten when they were  called using the top level matplotlib API (`4408`)
- Fixed a bug where calling ``Series.astype(str)`` would truncate the string  (`4405`, `4437`)
- Fixed a py3 compat issue where bytes were being repr'd as tuples  (`4455`)
- Fixed Panel attribute naming conflict if item is named 'a'  (`3440`)
- Fixed an issue where duplicate indexes were raising when plotting  (`4486`)
- Fixed an issue where cumsum and cumprod didn't work with bool dtypes  (`4170`, `4440`)
- Fixed Panel slicing issued in ``xs`` that was returning an incorrect dimmed  object (`4016`)
- Fix resampling bug where custom reduce function not used if only one group  (`3849`, `4494`)
- Fixed Panel assignment with a transposed frame (`3830`)
- Raise on set indexing with a Panel and a Panel as a value which needs  alignment (`3777`)
- frozenset objects now raise in the ``Series`` constructor (`4482`,  `4480`)
- Fixed issue with sorting a duplicate MultiIndex that has multiple dtypes  (`4516`)
- Fixed bug in ``DataFrame.set_values`` which was causing name attributes to  be lost when expanding the index. (`3742`, `4039`)
- Fixed issue where individual ``names``, ``levels`` and ``labels`` could be  set on ``MultiIndex`` without validation (`3714`, `4039`)
- Fixed (`3334`) in pivot_table. Margins did not compute if values is  the index.
- Fix bug in having a rhs of ``np.timedelta64`` or ``np.offsets.DateOffset``  when operating with datetimes (`4532`)
- Fix arithmetic with series/datetimeindex and ``np.timedelta64`` not working  the same (`4134`) and buggy timedelta in NumPy 1.6 (`4135`)
- Fix bug in ``pd.read_clipboard`` on windows with PY3 (`4561`); not  decoding properly
- ``tslib.get_period_field()`` and ``tslib.get_period_field_arr()`` now raise  if code argument out of range (`4519`, `4520`)
- Fix boolean indexing on an empty series loses index names (`4235`),  infer_dtype works with empty arrays.
- Fix reindexing with multiple axes; if an axes match was not replacing the  current axes, leading to a possible lazy frequency inference issue  (`3317`)
- Fixed issue where ``DataFrame.apply`` was reraising exceptions incorrectly  (causing the original stack trace to be truncated).
- Fix selection with ``ix/loc`` and non_unique selectors (`4619`)
- Fix assignment with iloc/loc involving a dtype change in an existing column  (`4312`, `5702`) have internal setitem_with_indexer in core/indexing  to use Block.setitem
- Fixed bug where thousands operator was not handled correctly for floating  point numbers in csv_import (`4322`)
- Fix an issue with CacheableOffset not properly being used by many  DateOffset; this prevented the DateOffset from being cached (`4609`)
- Fix boolean comparison with a DataFrame on the lhs, and a list/tuple on the  rhs (`4576`)
- Fix error/dtype conversion with setitem of ``None`` on ``Series/DataFrame``  (`4667`)
- Fix decoding based on a passed in non-default encoding in ``pd.read_stata``  (`4626`)
- Fix ``DataFrame.from_records`` with a plain-vanilla ``ndarray``.  (`4727`)
- Fix some inconsistencies with ``Index.rename`` and ``MultiIndex.rename``,  etc. (`4718`, `4628`)
- Bug in using ``iloc/loc`` with a cross-sectional and duplicate indices  (`4726`)
- Bug with using ``QUOTE_NONE`` with ``to_csv`` causing ``Exception``.  (`4328`)
- Bug with Series indexing not raising an error when the right-hand-side has  an incorrect length (`2702`)
- Bug in MultiIndexing with a partial string selection as one part of a  MultIndex (`4758`)
- Bug with reindexing on the index with a non-unique index will now raise  ``ValueError`` (`4746`)
- Bug in setting with ``loc/ix`` a single indexer with a MultiIndex axis and  a NumPy array, related to (`3777`)

- Bug in concatenation with duplicate columns across dtypes not merging with  axis=0 (`4771`, `4975`)
- Bug in ``iloc`` with a slice index failing (`4771`)
- Incorrect error message with no colspecs or width in ``read_fwf``.  (`4774`)
- Fix bugs in indexing in a Series with a duplicate index (`4548`,  `4550`)
- Fixed bug with reading compressed files with ``read_fwf`` in Python 3.  (`3963`)
- Fixed an issue with a duplicate index and assignment with a dtype change  (`4686`)
- Fixed bug with reading compressed files in as ``bytes`` rather than ``str``  in Python 3. Simplifies bytes-producing file-handling in Python 3  (`3963`, `4785`).
- Fixed an issue related to ticklocs/ticklabels with log scale bar plots  across different versions of matplotlib (`4789`)
- Suppressed DeprecationWarning associated with internal calls issued by  repr() (`4391`)
- Fixed an issue with a duplicate index and duplicate selector with ``.loc``  (`4825`)
- Fixed an issue with ``DataFrame.sort_index`` where, when sorting by a  single column and passing a list for ``ascending``, the argument for  ``ascending`` was being interpreted as ``True`` (`4839`,  `4846`)
- Fixed ``Panel.tshift`` not working. Added ``freq`` support to ``Panel.shift``  (`4853`)
- Fix an issue in TextFileReader w/ Python engine (i.e. PythonParser)  with thousands != "," (`4596`)
- Bug in getitem with a duplicate index when using where (`4879`)
- Fix Type inference code coerces float column into datetime (`4601`)
- Fixed ``_ensure_numeric`` does not check for complex numbers  (`4902`)
- Fixed a bug in ``Series.hist`` where two figures were being created when  the ``by`` argument was passed (`4112`, `4113`).
- Fixed a bug in ``convert_objects`` for > 2 ndims (`4937`)
- Fixed a bug in DataFrame/Panel cache insertion and subsequent indexing  (`4939`, `5424`)
- Fixed string methods for ``FrozenNDArray`` and ``FrozenList``  (`4929`)
- Fixed a bug with setting invalid or out-of-range values in indexing  enlargement scenarios (`4940`)
- Tests for fillna on empty Series (`4346`), thanks @immerrr
- Fixed ``copy()`` to shallow copy axes/indices as well and thereby keep  separate metadata. (`4202`, `4830`)
- Fixed skiprows option in Python parser for read_csv (`4382`)
- Fixed bug preventing ``cut`` from working with ``np.inf`` levels without  explicitly passing labels (`3415`)
- Fixed wrong check for overlapping in ``DatetimeIndex.union``  (`4564`)
- Fixed conflict between thousands separator and date parser in csv_parser  (`4678`)
- Fix appending when dtypes are not the same (error showing mixing  float/np.datetime64) (`4993`)
- Fix repr for DateOffset. No longer show duplicate entries in kwds.  Removed unused offset fields. (`4638`)
- Fixed wrong index name during read_csv if using usecols. Applies to c  parser only. (`4201`)
- ``Timestamp`` objects can now appear in the left hand side of a comparison  operation with a ``Series`` or ``DataFrame`` object (`4982`).
- Fix a bug when indexing with ``np.nan`` via ``iloc/loc`` (`5016`)
- Fixed a bug where low memory c parser could create different types in  different chunks of the same file. Now coerces to numerical type or raises  warning. (`3866`)
- Fix a bug where reshaping a ``Series`` to its own shape raised  ``TypeError`` (`4554`) and other reshaping issues.
- Bug in setting with ``ix/loc`` and a mixed int/string index (`4544`)
- Make sure series-series boolean comparisons are label based (`4947`)
- Bug in multi-level indexing with a Timestamp partial indexer  (`4294`)
- Tests/fix for MultiIndex construction of an all-nan frame (`4078`)
- Fixed a bug where `~pandas.read_html` wasn't correctly inferring  values of tables with commas (`5029`)
- Fixed a bug where `~pandas.read_html` wasn't providing a stable  ordering of returned tables (`4770`, `5029`).
- Fixed a bug where `~pandas.read_html` was incorrectly parsing when  passed ``index_col=0`` (`5066`).

- Fixed a bug where `~pandas.read_html` was incorrectly inferring the  type of headers (`5048`).
- Fixed a bug where ``DatetimeIndex`` joins with ``PeriodIndex`` caused a  stack overflow (`3899`).
- Fixed a bug where ``groupby`` objects didn't allow plots (`5102`).
- Fixed a bug where ``groupby`` objects weren't tab-completing column names  (`5102`).
- Fixed a bug where ``groupby.plot()`` and friends were duplicating figures  multiple times (`5102`).
- Provide automatic conversion of ``object`` dtypes on fillna, related  (`5103`)
- Fixed a bug where default options were being overwritten in the option  parser cleaning (`5121`).
- Treat a list/ndarray identically for ``iloc`` indexing with list-like  (`5006`)
- Fix ``MultiIndex.get_level_values()`` with missing values (`5074`)
- Fix bound checking for Timestamp() with datetime64 input (`4065`)
- Fix a bug where ``TestReadHtml`` wasn't calling the correct ``read_html()``  function (`5150`).
- Fix a bug with ``NDFrame.replace()`` which made replacement appear as  though it was (incorrectly) using regular expressions (`5143`).
- Fix better error message for to_datetime (`4928`)
- Made sure different locales are tested on travis-ci (`4918`). Also  adds a couple of utilities for getting locales and setting locales with a  context manager.
- Fixed segfault on ``isnull(MultiIndex)`` (now raises an error instead)  (`5123`, `5125`)
- Allow duplicate indices when performing operations that align  (`5185`, `5639`)
- Compound dtypes in a constructor raise ``NotImplementedError``  (`5191`)
- Bug in comparing duplicate frames (`4421`) related
- Bug in describe on duplicate frames
- Bug in ``to_datetime`` with a format and ``coerce=True`` not raising  (`5195`)
- Bug in ``loc`` setting with multiple indexers and a rhs of a Series that  needs broadcasting (`5206`)
- Fixed bug where inplace setting of levels or labels on ``MultiIndex`` would  not clear cached ``values`` property and therefore return wrong ``values``.  (`5215`)
- Fixed bug where filtering a grouped DataFrame or Series did not maintain  the original ordering (`4621`).
- Fixed ``Period`` with a business date freq to always roll-forward if on a  non-business date. (`5203`)
- Fixed bug in Excel writers where frames with duplicate column names weren't  written correctly. (`5235`)
- Fixed issue with ``drop`` and a non-unique index on Series (`5248`)
- Fixed segfault in C parser caused by passing more names than columns in  the file. (`5156`)
- Fix ``Series.isin`` with date/time-like dtypes (`5021`)
- C and Python Parser can now handle the more common MultiIndex column  format which doesn't have a row for index names (`4702`)
- Bug when trying to use an out-of-bounds date as an object dtype  (`5312`)
- Bug when trying to display an embedded PandasObject (`5324`)
- Allows operating of Timestamps to return a datetime if the result is out-of-bounds  related (`5312`)
- Fix return value/type signature of ``initObjToJSON()`` to be compatible  with numpy's ``import_array()`` (`5334`, `5326`)
- Bug when renaming then set_index on a DataFrame (`5344`)
- Test suite no longer leaves around temporary files when testing graphics. (`5347`)  (thanks for catching this @yarikoptic!)
- Fixed html tests on win32. (`4580`)
- Make sure that ``head/tail`` are ``iloc`` based, (`5370`)
- Fixed bug for ``PeriodIndex`` string representation if there are 1 or 2  elements. (`5372`)
- The GroupBy methods ``transform`` and ``filter`` can be used on Series  and DataFrames that have repeated (non-unique) indices. (`4620`)
- Fix empty series not printing name in repr (`4651`)
- Make tests create temp files in temp directory by default. (`5419`)
- ``pd.to_timedelta`` of a scalar returns a scalar (`5410`)
- ``pd.to_timedelta`` accepts ``NaN`` and ``NaT``, returning ``NaT`` instead of raising (`5437`)
- performance improvements in ``isnull`` on larger size pandas objects

- Fixed various setitem with 1d ndarray that does not have a matching  length to the indexer (`5508`)
- Bug in getitem with a MultiIndex and ``iloc`` (`5528`)
- Bug in delitem on a Series (`5542`)
- Bug fix in apply when using custom function and objects are not mutated (`5545`)
- Bug in selecting from a non-unique index with ``loc`` (`5553`)
- Bug in groupby returning non-consistent types when user function returns a ``None``, (`5592`)
- Work around regression in numpy 1.7.0 which erroneously raises IndexError from ``ndarray.item`` (`5666`)
- Bug in repeated indexing of object with resultant non-unique index (`5678`)
- Bug in fillna with Series and a passed series/dict (`5703`)
- Bug in groupby transform with a datetime-like grouper (`5712`)
- Bug in MultiIndex selection in PY3 when using certain keys (`5725`)
- Row-wise concat of differing dtypes failing in certain cases (`5754`)  .. _whatsnew_0.13.0.contributors:  Contributors
~~~  .. contributors:: v0.12.0..v0.13.0

{{ header }}  This is a minor release from 0.13.0 and includes a small number of API changes, several new features,
enhancements, and performance improvements along with a large number of bug fixes. We recommend that all
users upgrade to this version.  Highlights include:  - Added ``infer_datetime_format`` keyword to ``read_csv/to_datetime`` to allow speedups for homogeneously formatted datetimes.
- Will intelligently limit display precision for datetime/timedelta formats.
- Enhanced Panel `~pandas.Panel.apply` method.
- Suggested tutorials in new `Tutorials<tutorials>` section.
- Our pandas ecosystem is growing, We now feature related projects in a new `Pandas Ecosystem<ecosystem>` section.
- Much work has been taking place on improving the docs, and a new `Contributing<contributing>` section has been added.
- Even though it may only be of interest to devs, we <3 our new CI status page: ScatterCI_.  .. warning::  0.13.1 fixes a bug that was caused by a combination of having numpy < 1.8, and doing  chained assignment on a string-like array. Please review `the docs<indexing.view_versus_copy>`,  chained indexing can have unexpected results and should generally be avoided.  This would previously segfault:  .. ipython:: python  df = pd.DataFrame({"A": np.array(["foo", "bar", "bah", "foo", "bar"])})  df["A"].iloc[0] = np.nan  df  The recommended way to do this type of assignment is:  .. ipython:: python  df = pd.DataFrame({"A": np.array(["foo", "bar", "bah", "foo", "bar"])})  df.loc[0, "A"] = np.nan  df  Output formatting enhancements
~~~  - df.info() view now display dtype info per column (`5682`)  - df.info() now honors the option ``max_info_rows``, to disable null counts for large frames (`5974`)  .. ipython:: python  max_info_rows = pd.get_option("max_info_rows")  df = pd.DataFrame(  {  "A": np.random.randn(10),  "B": np.random.randn(10),  "C": pd.date_range("20130101", periods=10),  }  )  df.iloc[3:6, [0, 2]] = np.nan  .. ipython:: python  # set to not display the null counts  pd.set_option("max_info_rows", 0)  df.info()  .. ipython:: python  # this is the default (same as in 0.13.0)  pd.set_option("max_info_rows", max_info_rows)  df.info()  - Add ``show_dimensions`` display option for the new DataFrame repr to control whether the dimensions print.  .. ipython:: python  df = pd.DataFrame([[1, 2], [3, 4]])  pd.set_option("show_dimensions", False)  df  pd.set_option("show_dimensions", True)  df  - The ``ArrayFormatter`` for ``datetime`` and ``timedelta64`` now intelligently  limit precision based on the values in the array (`3401`)  Previously output might look like:  ..  code-block:: text  age  today  diff  0 2001-01-01 000 2013-04-19 000 4491 days, 000  1 2004-06-01 000 2013-04-19 000 3244 days, 000  Now the output looks like:  .. ipython:: python  df = pd.DataFrame(  [pd.Timestamp("20010101"), pd.Timestamp("20040601")], columns=["age"]  )  df["today"] = pd.Timestamp("20130419")  df["diff"] = df["today"] - df["age"]  df  API changes

~~~  - Add ``-NaN`` and ``-nan`` to the default set of NA values (`5952`).  See `NA Values <io.na_values>`.  - Added ``Series.str.get_dummies`` vectorized string method (`6021`), to extract  dummy/indicator variables for separated string columns:  .. ipython:: python  s = pd.Series(["a", "a|b", np.nan, "a|c"])  s.str.get_dummies(sep="|")  - Added the ``NDFrame.equals()`` method to compare if two NDFrames are  equal have equal axes, dtypes, and values. Added the  ``array_equivalent`` function to compare if two ndarrays are  equal. NaNs in identical locations are treated as  equal. (`5283`) See also `the docs<basics.equals>` for a motivating example.  .. code-block:: python  df = pd.DataFrame({"col": ["foo", 0, np.nan]})  df2 = pd.DataFrame({"col": [np.nan, 0, "foo"]}, index=[2, 1, 0])  df.equals(df2)  df.equals(df2.sort_index())  - ``DataFrame.apply`` will use the ``reduce`` argument to determine whether a  ``Series`` or a ``DataFrame`` should be returned when the ``DataFrame`` is  empty (`6007`).  Previously, calling ``DataFrame.apply`` an empty ``DataFrame`` would return  either a ``DataFrame`` if there were no columns, or the function being  applied would be called with an empty ``Series`` to guess whether a  ``Series`` or ``DataFrame`` should be returned:  .. code-block:: ipython  In [32]: def applied_func(col):  ...:  print("Apply function being called with: ", col)  ...:  return col.sum()  ...:  In [33]: empty = DataFrame(columns=['a', 'b'])  In [34]: empty.apply(applied_func)  Apply function being called with:  Series([], Length: 0, dtype: float64)  Out[34]:  a  NaN  b  NaN  Length: 2, dtype: float64  Now, when ``apply`` is called on an empty ``DataFrame``: if the ``reduce``  argument is ``True`` a ``Series`` will returned, if it is ``False`` a  ``DataFrame`` will be returned, and if it is ``None`` (the default) the  function being applied will be called with an empty series to try and guess  the return type.  .. code-block:: ipython  In [35]: empty.apply(applied_func, reduce=True)  Out[35]:  a  NaN  b  NaN  Length: 2, dtype: float64  In [36]: empty.apply(applied_func, reduce=False)  Out[36]:  Empty DataFrame  Columns: [a, b]  Index: []  [0 rows x 2 columns]  Prior version deprecations/changes
~~~  There are no announced changes in 0.13 or prior that are taking effect as of 0.13.1  Deprecations
~~~  There are no deprecations of prior behavior in 0.13.1  Enhancements

~~~  - ``pd.read_csv`` and ``pd.to_datetime`` learned a new ``infer_datetime_format`` keyword which greatly  improves parsing perf in many cases. Thanks to @lexual for suggesting and @danbirken  for rapidly implementing. (`5490`, `6021`)  If ``parse_dates`` is enabled and this flag is set, pandas will attempt to  infer the format of the datetime strings in the columns, and if it can  be inferred, switch to a faster method of parsing them.  In some cases  this can increase the parsing speed by ~5-10x.  .. code-block:: python  # Try to infer the format for the index column  df = pd.read_csv(  "foo.csv", index_col=0, parse_dates=True, infer_datetime_format=True  )  - ``date_format`` and ``datetime_format`` keywords can now be specified when writing to ``excel``  files (`4133`)  - ``MultiIndex.from_product`` convenience function for creating a MultiIndex from  the cartesian product of a set of iterables (`6055`):  .. ipython:: python  shades = ["light", "dark"]  colors = ["red", "green", "blue"]  pd.MultiIndex.from_product([shades, colors], names=["shade", "color"])  - Panel `~pandas.Panel.apply` will work on non-ufuncs. See `the docs<basics.apply>`.  .. code-block:: ipython  In [28]: import pandas._testing as tm  In [29]: panel = tm.makePanel(5)  In [30]: panel  Out[30]:  <class 'pandas.core.panel.Panel'>  Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)  Items axis: ItemA to ItemC  Major_axis axis: 2000-01-03 000 to 2000-01-07 000  Minor_axis axis: A to D  In [31]: panel['ItemA']  Out[31]:  A  B  C  D  2000-01-03 -0.673690  0.577046 -1.344312 -1.469388  2000-01-04  0.113648 -1.715002  0.844885  0.357021  2000-01-05 -1.478427 -1.039268  1.075770 -0.674600  2000-01-06  0.524988 -0.370647 -0.109050 -1.776904  2000-01-07  0.404705 -1.157892  1.643563 -0.968914  [5 rows x 4 columns]  Specifying an ``apply`` that operates on a Series (to return a single element)  .. code-block:: ipython  In [32]: panel.apply(lambda x: x.dtype, axis='items')  Out[32]:  A  B  C  D  2000-01-03  float64  float64  float64  float64  2000-01-04  float64  float64  float64  float64  2000-01-05  float64  float64  float64  float64  2000-01-06  float64  float64  float64  float64  2000-01-07  float64  float64  float64  float64  [5 rows x 4 columns]  A similar reduction type operation  .. code-block:: ipython  In [33]: panel.apply(lambda x: x.sum(), axis='major_axis')  Out[33]:  ItemA  ItemB  ItemC  A -1.108775 -1.090118 -2.984435  B -3.705764  0.409204  1.866240  C  2.110856  2.960500 -0.974967  D -4.532785  0.303202 -3.685193  [4 rows x 3 columns]  This is equivalent to  .. code-block:: ipython  In [34]: panel.sum('major_axis')  Out[34]:  ItemA  ItemB  ItemC  A -1.108775 -1.090118 -2.984435  B -3.705764  0.409204  1.866240  C  2.110856  2.960500 -0.974967  D -4.532785  0.303202 -3.685193  [4 rows x 3 columns]  A transformation operation that returns a Panel, but is computing  the z-score across the major_axis  .. code-block:: ipython  In [35]: result = panel.apply(lambda x: (x - x.mean()) / x.std(),  ...:  axis='major_axis')  ...:  In [36]: result  Out[36]:  <class 'pandas.core.panel.Panel'>  Dimensions: 3 (items) x 5 (major_axis) x 4 (minor_axis)  Items axis: ItemA to ItemC  Major_axis axis: 2000-01-03 000 to 2000-01-07 000  Minor_axis axis: A to D  In [37]: result['ItemA']  # noqa E999  Out[37]:  A  B  C  D  2000-01-03 -0.535778  1.500802 -1.506416 -0.681456  2000-01-04  0.397628 -1.108752  0.360481  1.529895  2000-01-05 -1.489811 -0.339412  0.557374  0.280845  2000-01-06  0.885279  0.421830 -0.453013 -1.053785  2000-01-07  0.742682 -0.474468  1.041575 -0.075499  [5 rows x 4 columns]  - Panel `~pandas.Panel.apply` operating on cross-sectional slabs. (`1148`)  .. code-block:: ipython  In [38]: def f(x):  ...:  return ((x.T - x.mean(1)) / x.std(1)).T  ...:  In [39]: result = panel.apply(f, axis=['items', 'major_axis'])  In [40]: result  Out[40]:  <class 'pandas.core.panel.Panel'>  Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)  Items axis: A to D  Major_axis axis: 2000-01-03 000 to 2000-01-07 000  Minor_axis axis: ItemA to ItemC  In [41]: result.loc[:, :, 'ItemA']  Out[41]:  A  B  C  D  2000-01-03  0.012922 -0.030874 -0.629546 -0.757034  2000-01-04  0.392053 -1.071665  0.163228  0.548188  2000-01-05 -1.093650 -0.640898  0.385734 -1.154310  2000-01-06  1.005446 -1.154593 -0.595615 -0.809185  2000-01-07  0.783051 -0.198053  0.919339 -1.052721  [5 rows x 4 columns]  This is equivalent to the following  .. code-block:: ipython  In [42]: result = pd.Panel({ax: f(panel.loc[:, :, ax]) for ax in panel.minor_axis})  In [43]: result  Out[43]:  <class 'pandas.core.panel.Panel'>  Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)  Items axis: A to D  Major_axis axis: 2000-01-03 000 to 2000-01-07 000  Minor_axis axis: ItemA to ItemC  In [44]: result.loc[:, :, 'ItemA']  Out[44]:  A

'pandas.core.panel.Panel'>  Dimensions: 4 (items) x 5 (major_axis) x 3 (minor_axis)  Items axis: A to D  Major_axis axis: 2000-01-03 000 to 2000-01-07 000  Minor_axis axis: ItemA to ItemC  In [44]: result.loc[:, :, 'ItemA']  Out[44]:  A  B  C  D  2000-01-03  0.012922 -0.030874 -0.629546 -0.757034  2000-01-04  0.392053 -1.071665  0.163228  0.548188  2000-01-05 -1.093650 -0.640898  0.385734 -1.154310  2000-01-06  1.005446 -1.154593 -0.595615 -0.809185  2000-01-07  0.783051 -0.198053  0.919339 -1.052721  [5 rows x 4 columns]  Performance

~~~  Performance improvements for 0.13.1  - Series datetime/timedelta binary operations (`5801`)
- DataFrame ``count/dropna`` for ``axis=1``
- Series.str.contains now has a ``regex=False`` keyword which can be faster for plain (non-regex) string patterns. (`5879`)
- Series.str.extract (`5944`)
- ``dtypes/ftypes`` methods (`5968`)
- indexing with object dtypes (`5968`)
- ``DataFrame.apply`` (`6013`)
- Regression in JSON IO (`5765`)
- Index construction from Series (`6150`)  Experimental
~~~  There are no experimental changes in 0.13.1  .. _release.bug_fixes-0.13.1:  Bug fixes
~~~  - Bug in ``io.wb.get_countries`` not including all countries (`6008`)
- Bug in Series replace with timestamp dict (`5797`)
- read_csv/read_table now respects the ``prefix`` kwarg (`5732`).
- Bug in selection with missing values via ``.ix`` from a duplicate indexed DataFrame failing (`5835`)
- Fix issue of boolean comparison on empty DataFrames (`5808`)
- Bug in isnull handling ``NaT`` in an object array (`5443`)
- Bug in ``to_datetime`` when passed a ``np.nan`` or integer datelike and a format string (`5863`)
- Bug in groupby dtype conversion with datetimelike (`5869`)
- Regression in handling of empty Series as indexers to Series  (`5877`)
- Bug in internal caching, related to (`5727`)
- Testing bug in reading JSON/msgpack from a non-filepath on windows under py3 (`5874`)
- Bug when assigning to .ix[tuple(...)] (`5896`)
- Bug in fully reindexing a Panel (`5905`)
- Bug in idxmin/max with object dtypes (`5914`)
- Bug in ``BusinessDay`` when adding n days to a date not on offset when n>5 and n%5==0 (`5890`)
- Bug in assigning to chained series with a series via ix (`5928`)
- Bug in creating an empty DataFrame, copying, then assigning (`5932`)
- Bug in DataFrame.tail with empty frame (`5846`)
- Bug in propagating metadata on ``resample`` (`5862`)
- Fixed string-representation of ``NaT`` to be "NaT" (`5708`)
- Fixed string-representation for Timestamp to show nanoseconds if present (`5912`)
- ``pd.match`` not returning passed sentinel
- ``Panel.to_frame()`` no longer fails when ``major_axis`` is a  ``MultiIndex`` (`5402`).
- Bug in ``pd.read_msgpack`` with inferring a ``DateTimeIndex`` frequency  incorrectly (`5947`)
- Fixed ``to_datetime`` for array with both Tz-aware datetimes and ``NaT``'s  (`5961`)
- Bug in rolling skew/kurtosis when passed a Series with bad data (`5749`)
- Bug in scipy ``interpolate`` methods with a datetime index (`5975`)
- Bug in NaT comparison if a mixed datetime/np.datetime64 with NaT were passed (`5968`)
- Fixed bug with ``pd.concat`` losing dtype information if all inputs are empty (`5742`)
- Recent changes in IPython cause warnings to be emitted when using previous versions  of pandas in QTConsole, now fixed. If you're using an older version and  need to suppress the warnings, see (`5922`).
- Bug in merging ``timedelta`` dtypes (`5695`)
- Bug in plotting.scatter_matrix function. Wrong alignment among diagonal  and off-diagonal plots, see (`5497`).
- Regression in Series with a MultiIndex via ix (`6018`)
- Bug in Series.xs with a MultiIndex (`6018`)
- Bug in Series construction of mixed type with datelike and an integer (which should result in  object type and not automatic conversion) (`6028`)
- Possible segfault when chained indexing with an object array under NumPy 1.7.1 (`6026`, `6056`)
- Bug in setting using fancy indexing a single element with a non-scalar (e.g. a list),  (`6043`)
- ``to_sql`` did not respect ``if_exists`` (`4110` `4304`)
- Regression in ``.get(None)`` indexing from 0.12 (`5652`)
- Subtle ``iloc`` indexing bug, surfaced in (`6059`)
- Bug with insert of strings into DatetimeIndex (`5818`)
- Fixed unicode bug in to_html/HTML repr (`6098`)
- Fixed missing arg validation in get_options_data (`6105`)
- Bug in assignment with duplicate columns in a frame where the locations  are a slice (e.g. next to each other) (`6120`)
- Bug in propagating _ref_locs during construction of a DataFrame with dups  index/columns (`6121`)

- Bug in ``DataFrame.apply`` when using mixed datelike reductions (`6125`)
- Bug in ``DataFrame.append`` when appending a row with different columns (`6129`)
- Bug in DataFrame construction with recarray and non-ns datetime dtype (`6140`)
- Bug in ``.loc`` setitem indexing with a dataframe on rhs, multiple item setting, and  a datetimelike (`6152`)
- Fixed a bug in ``query``/``eval`` during lexicographic string comparisons (`6155`).
- Fixed a bug in ``query`` where the index of a single-element ``Series`` was  being thrown away (`6148`).
- Bug in ``HDFStore`` on appending a dataframe with MultiIndexed columns to  an existing table (`6167`)
- Consistency with dtypes in setting an empty DataFrame (`6171`)
- Bug in selecting on a MultiIndex ``HDFStore`` even in the presence of under  specified column spec (`6169`)
- Bug in ``nanops.var`` with ``ddof=1`` and 1 elements would sometimes return ``inf``  rather than ``nan`` on some platforms (`6136`)
- Bug in Series and DataFrame bar plots ignoring the ``use_index`` keyword (`6209`)
- Bug in groupby with mixed str/int under python3 fixed; ``argsort`` was failing (`6212`)  .. _whatsnew_0.13.1.contributors:  Contributors
~~~  .. contributors:: v0.13.0..v0.13.1

{{ header }}  This is a major release from 0.13.1 and includes a small number of API changes, several new features,
users upgrade to this version.  - Highlights include:  - Officially support Python 3.4  - SQL interfaces updated to use ``sqlalchemy``, See `Here<whatsnew_0140.sql>`.  - Display interface changes, See `Here<whatsnew_0140.display>`  - MultiIndexing Using Slicers, See `Here<whatsnew_0140.slicers>`.  - Ability to join a singly-indexed DataFrame with a MultiIndexed DataFrame, see `Here <merging.join_on_mi>`  - More consistency in groupby results and more flexible groupby specifications, See `Here<whatsnew_0140.groupby>`  - Holiday calendars are now supported in ``CustomBusinessDay``, see `Here <timeseries.holiday>`  - Several improvements in plotting functions, including: hexbin, area and pie plots, see `Here<whatsnew_0140.plotting>`.  - Performance doc section on I/O operations, See `Here <io.perf>`  - `Other Enhancements <whatsnew_0140.enhancements>`  - `API Changes <whatsnew_0140.api>`  - `Text Parsing API Changes <whatsnew_0140.parsing>`  - `Groupby API Changes <whatsnew_0140.groupby>`  - `Performance Improvements <whatsnew_0140.performance>`  - `Prior Deprecations <whatsnew_0140.prior_deprecations>`  - `Deprecations <whatsnew_0140.deprecations>`  - `Known Issues <whatsnew_0140.knownissues>`  - `Bug Fixes <whatsnew_0140.bug_fixes>`  .. warning::  In 0.14.0 all ``NDFrame`` based containers have undergone significant internal refactoring. Before that each block of  homogeneous data had its own labels and extra care was necessary to keep those in sync with the parent container's labels.  This should not have any visible user/API behavior changes (`6745`)  .. _whatsnew_0140.api:  API changes
~~~  - ``read_excel`` uses 0 as the default sheet (`6573`)
- ``iloc`` will now accept out-of-bounds indexers for slices, e.g. a value that exceeds the length of the object being  indexed. These will be excluded. This will make pandas conform more with python/numpy indexing of out-of-bounds  values. A single indexer that is out-of-bounds and drops the dimensions of the object will still raise  ``IndexError`` (`6296`, `6299`). This could result in an empty axis (e.g. an empty DataFrame being returned)  .. ipython:: python  dfl = pd.DataFrame(np.random.randn(5, 2), columns=list('AB'))  dfl  dfl.iloc[:, 2:3]  dfl.iloc[:, 1:3]  dfl.iloc[4:6]  These are out-of-bounds selections  .. code-block:: python  >>> dfl.iloc[[4, 5, 6]]  IndexError: positional indexers are out-of-bounds  >>> dfl.iloc[:, 4]  IndexError: single positional indexer is out-of-bounds  - Slicing with negative start, stop & step values handles corner cases better (`6531`):  - ``df.iloc[:-len(df)]`` is now empty  - ``df.iloc[len(df)::-1]`` now enumerates all elements in reverse  - The `DataFrame.interpolate` keyword ``downcast`` default has been changed from ``infer`` to  ``None``. This is to preserve the original dtype unless explicitly requested otherwise (`6290`).
- When converting a dataframe to HTML it used to return ``Empty DataFrame``. This special case has  been removed, instead a header with the column names is returned (`6062`).

- ``Series`` and ``Index`` now internally share more common operations, e.g. ``factorize(),nunique(),value_counts()`` are  now supported on ``Index`` types as well. The ``Series.weekday`` property from is removed  from Series for API consistency. Using a ``DatetimeIndex/PeriodIndex`` method on a Series will now raise a ``TypeError``.  (`4551`, `4056`, `5519`, `6380`, `7206`).  - Add ``is_month_start``, ``is_month_end``, ``is_quarter_start``, ``is_quarter_end``, ``is_year_start``, ``is_year_end`` accessors for ``DateTimeIndex`` / ``Timestamp`` which return a boolean array of whether the timestamp(s) are at the start/end of the month/quarter/year defined by the frequency of the ``DateTimeIndex`` / ``Timestamp`` (`4565`, `6998`)  - Local variable usage has changed in  `pandas.eval`/`DataFrame.eval`/`DataFrame.query`  (`5987`). For the `~pandas.DataFrame` methods, two things have  changed  - Column names are now given precedence over locals  - Local variables must be referred to explicitly. This means that even if  you have a local variable that is *not* a column you must still refer to  it with the ``'@'`` prefix.  - You can have an expression like ``df.query('@a < a')`` with no complaints  from ``pandas`` about ambiguity of the name ``a``.  - The top-level `pandas.eval` function does not allow you use the  ``'@'`` prefix and provides you with an error message telling you so.  - ``NameResolutionError`` was removed because it isn't necessary anymore.  - Define and document the order of column vs index names in query/eval (`6676`)
- ``concat`` will now concatenate mixed Series and DataFrames using the Series name  or numbering columns as needed (`2385`). See `the docs <merging.mixed_ndims>`

- Slicing and advanced/boolean indexing operations on ``Index`` classes as well  as `Index.delete` and `Index.drop` methods will no longer change the type of the  resulting index (`6440`, `7040`)  .. ipython:: python  i = pd.Index([1, 2, 3, 'a', 'b', 'c'])  i[[0, 1, 2]]  i.drop(['a', 'b', 'c'])  Previously, the above operation would return ``Int64Index``.  If you'd like  to do this manually, use `Index.astype`  .. ipython:: python  i[[0, 1, 2]].astype(np.int_)  - ``set_index`` no longer converts MultiIndexes to an Index of tuples. For example,  the old behavior returned an Index in this case (`6459`):  .. ipython:: python  np.random.seed(1234)  from itertools import product  tuples = list(product(('a', 'b'), ('c', 'd')))  mi = pd.MultiIndex.from_tuples(tuples)  df_multi = pd.DataFrame(np.random.randn(4, 2), index=mi)  tuple_ind = pd.Index(tuples, tupleize_cols=False)  df_multi.index  .. ipython:: python  # Old behavior, casted MultiIndex to an Index  tuple_ind  df_multi.set_index(tuple_ind)  # New behavior  mi  df_multi.set_index(mi)  This also applies when passing multiple indices to ``set_index``:  .. ipython:: python  @suppress  df_multi.index = tuple_ind  # Old output, 2-level MultiIndex of tuples  df_multi.set_index([df_multi.index, df_multi.index])  @suppress  df_multi.index = mi  # New output, 4-level MultiIndex  df_multi.set_index([df_multi.index, df_multi.index])  - ``pairwise`` keyword was added to the statistical moment functions  ``rolling_cov``, ``rolling_corr``, ``ewmcov``, ``ewmcorr``,  ``expanding_cov``, ``expanding_corr`` to allow the calculation of moving  window covariance and correlation matrices (`4950`). See  `Computing rolling pairwise covariances and correlations  <window.corr_pairwise>` in the docs.  .. code-block:: ipython  In [1]: df = pd.DataFrame(np.random.randn(10, 4), columns=list('ABCD'))  In [4]: covs = pd.rolling_cov(df[['A', 'B', 'C']],  ...:  df[['B', 'C', 'D']],  ...:  5,  ...:  pairwise=True)  In [5]: covs[df.index[-1]]  Out[5]:  B  C  D  A  0.035310  0.326593 -0.505430  B  0.137748 -0.006888 -0.005383  C -0.006888  0.861040  0.020762  - ``Series.iteritems()`` is now lazy (returns an iterator rather than a list). This was the documented behavior prior to 0.14. (`6760`)  - Added ``nunique`` and ``value_counts`` functions to ``Index`` for counting unique elements. (`6734`)
- ``stack`` and ``unstack`` now raise a ``ValueError`` when the ``level`` keyword refers  to a non-unique item in the ``Index`` (previously raised a ``KeyError``). (`6738`)
- drop unused order argument from ``Series.sort``; args now are in the same order as ``Series.order``;  add ``na_position`` arg to conform to ``Series.order`` (`6847`)
- default sorting algorithm for ``Series.order`` is now ``quicksort``, to conform with ``Series.sort``  (and numpy defaults)
- add ``inplace`` keyword to ``Series.order/sort`` to make them inverses (`6859`)
- ``DataFrame.sort`` now places NaNs at the beginning or end of the sort according to the ``na_position`` parameter. (`3917`)
- accept ``TextFileReader`` in ``concat``, which was affecting a common user idiom (`6583`), this was a regression  from 0.13.1
- Added ``factorize`` functions to ``Index`` and ``Series`` to get indexer and unique values (`7090`)
- ``describe`` on a DataFrame with a mix of Timestamp and string like objects returns a different Index (`7088`).  Previously the index was unintentionally sorted.

- Arithmetic operations with **only** ``bool`` dtypes now give a warning indicating  that they are evaluated in Python space for ``+``, ``-``,  and ``*`` operations and raise for all others (`7011`, `6762`,  `7015`, `7210`)  .. code-block:: python  >>> x = pd.Series(np.random.rand(10) > 0.5)  >>> y = True  >>> x + y  # warning generated: should do x | y instead  UserWarning: evaluating in Python space because the '+' operator is not  supported by numexpr for the bool dtype, use '|' instead  >>> x / y  # this raises because it doesn't make sense  NotImplementedError: operator '/' not implemented for bool dtypes  - In ``HDFStore``, ``select_as_multiple`` will always raise a ``KeyError``, when a key or the selector is not found (`6177`)
- ``df['col'] = value`` and ``df.loc[:,'col'] = value`` are now completely equivalent;  previously the ``.loc`` would not necessarily coerce the dtype of the resultant series (`6149`)
- ``dtypes`` and ``ftypes`` now return a series with ``dtype=object`` on empty containers (`5740`)
- ``df.to_csv`` will now return a string of the CSV data if neither a target path nor a buffer is provided  (`6061`)
- ``pd.infer_freq()`` will now raise a ``TypeError`` if given an invalid ``Series/Index``  type (`6407`, `6463`)
- A tuple passed to ``DataFame.sort_index`` will be interpreted as the levels of  the index, rather than requiring a list of tuple (`4370`)
- all offset operations now return ``Timestamp`` types (rather than datetime), Business/Week frequencies were incorrect (`4069`)
- ``to_excel`` now converts ``np.inf`` into a string representation,  customizable by the ``inf_rep`` keyword argument (Excel has no native inf  representation) (`6782`)
- Replace ``pandas.compat.scipy.scoreatpercentile`` with ``numpy.percentile`` (`6810`)
- ``.quantile`` on a ``datetime[ns]`` series now returns ``Timestamp`` instead  of ``np.datetime64`` objects (`6810`)
- change ``AssertionError`` to ``TypeError`` for invalid types passed to ``concat`` (`6583`)
- Raise a ``TypeError`` when ``DataFrame`` is passed an iterator as the  ``data`` argument (`5357`)  .. _whatsnew_0140.display:  Display changes
~~~  - The default way of printing large DataFrames has changed. DataFrames  exceeding ``max_rows`` and/or ``max_columns`` are now displayed in a  centrally truncated view, consistent with the printing of a  `pandas.Series` (`5603`).  In previous versions, a DataFrame was truncated once the dimension  constraints were reached and an ellipse (...) signaled that part of  the data was cut off.  The previous look of truncate.  In the current version, large DataFrames are centrally truncated,  showing a preview of head and tail in both dimensions.  The new look.  - allow option ``'truncate'`` for ``display.show_dimensions`` to only show the dimensions if the  frame is truncated (`6547`).  The default for ``display.show_dimensions`` will now be ``truncate``. This is consistent with  how Series display length.  .. ipython:: python  dfd = pd.DataFrame(np.arange(25).reshape(-1, 5),  index=[0, 1, 2, 3, 4],  columns=[0, 1, 2, 3, 4])  # show dimensions since this is truncated  with pd.option_context('display.max_rows', 2, 'display.max_columns', 2,  'display.show_dimensions', 'truncate'):  print(dfd)  # will not show dimensions since it is not truncated  with pd.option_context('display.max_rows', 10, 'display.max_columns', 40,  'display.show_dimensions', 'truncate'):  print(dfd)  - Regression in the display of a MultiIndexed Series with ``display.max_rows`` is less than the  length of the series (`7101`)
- Fixed a bug in the HTML repr of a truncated Series or DataFrame not showing the class name with the  ``large_repr`` set to 'info' (`7105`)

- The ``verbose`` keyword in ``DataFrame.info()``, which controls whether to shorten the ``info``  representation, is now ``None`` by default. This will follow the global setting in  ``display.max_info_columns``. The global setting can be overridden with ``verbose=True`` or  ``verbose=False``.
- Fixed a bug with the ``info`` repr not honoring the ``display.max_info_columns`` setting (`6939`)
- Offset/freq info now in Timestamp __repr__ (`4553`)  .. _whatsnew_0140.parsing:  Text parsing API changes
~~~  `read_csv`/`read_table` will now be noisier w.r.t invalid options rather than falling back to the ``PythonParser``.  - Raise ``ValueError`` when ``sep`` specified with  ``delim_whitespace=True`` in `read_csv`/`read_table`  (`6607`)
- Raise ``ValueError`` when ``engine='c'`` specified with unsupported  options in `read_csv`/`read_table` (`6607`)
- Raise ``ValueError`` when fallback to python parser causes options to be  ignored (`6607`)
- Produce `~pandas.io.parsers.ParserWarning` on fallback to python  parser when no options are ignored (`6607`)
- Translate ``sep='\s+'`` to ``delim_whitespace=True`` in  `read_csv`/`read_table` if no other C-unsupported options  specified (`6607`)  .. _whatsnew_0140.groupby:  GroupBy API changes
~~~  More consistent behavior for some groupby methods:  - groupby ``head`` and ``tail`` now act more like ``filter`` rather than an aggregation:  .. ipython:: python  df = pd.DataFrame([[1, 2], [1, 4], [5, 6]], columns=['A', 'B'])  g = df.groupby('A')  g.head(1)  # filters DataFrame  g.apply(lambda x: x.head(1))  # used to simply fall-through  - groupby head and tail respect column selection:  .. ipython:: python  g[['B']].head(1)  - groupby ``nth`` now reduces by default; filtering can be achieved by passing ``as_index=False``. With an optional ``dropna`` argument to ignore  NaN. See `the docs <groupby.nth>`.  Reducing  .. ipython:: python  df = pd.DataFrame([[1, np.nan], [1, 4], [5, 6]], columns=['A', 'B'])  g = df.groupby('A')  g.nth(0)  # this is equivalent to g.first()  g.nth(0, dropna='any')  # this is equivalent to g.last()  g.nth(-1, dropna='any')  Filtering  .. ipython:: python  gf = df.groupby('A', as_index=False)  gf.nth(0)  gf.nth(0, dropna='any')  - groupby will now not return the grouped column for non-cython functions (`5610`, `5614`, `6732`),  as its already the index  .. ipython:: python  df = pd.DataFrame([[1, np.nan], [1, 4], [5, 6], [5, 8]], columns=['A', 'B'])  g = df.groupby('A')  g.count()  g.describe()  - passing ``as_index`` will leave the grouped column in-place (this is not change in 0.14.0)  .. ipython:: python  df = pd.DataFrame([[1, np.nan], [1, 4], [5, 6], [5, 8]], columns=['A', 'B'])  g = df.groupby('A', as_index=False)  g.count()  g.describe()  - Allow specification of a more complex groupby via ``pd.Grouper``, such as grouping  by a Time and a string field simultaneously. See `the docs <groupby.specify>`. (`3794`)  - Better propagation/preservation of Series names when performing groupby  operations:  - ``SeriesGroupBy.agg`` will ensure that the name attribute of the original  series is propagated to the result (`6265`).  - If the function provided to ``GroupBy.apply`` returns a named series, the  name of the series will be kept as the name of the column index of the  DataFrame returned by ``GroupBy.apply`` (`6124`).  This facilitates  ``DataFrame.stack`` operations where the name of the column index is used as  the name of the inserted column containing the pivoted data.  .. _whatsnew_0140.sql:  SQL
~~~  The SQL reading and writing functions now support more database flavors
through SQLAlchemy (`2717`, `4163`, `5950`, `6292`).
All databases supported by SQLAlchemy can be used, such
as PostgreSQL, MySQL, Oracle, Microsoft SQL server (see documentation of
SQLAlchemy on `included dialects

<https://sqlalchemy.readthedocs.io/en/latest/dialects/index.html>`_).  The functionality of providing DBAPI connection objects will only be supported
for sqlite3 in the future. The ``'mysql'`` flavor is deprecated.  The new functions `~pandas.read_sql_query` and `~pandas.read_sql_table`
are introduced. The function `~pandas.read_sql` is kept as a convenience
wrapper around the other two and will delegate to specific function depending on
the provided input (database table name or sql query).  In practice, you have to provide a SQLAlchemy ``engine`` to the sql functions.
To connect with SQLAlchemy you use the `create_engine` function to create an engine
connecting to. For an in-memory sqlite database:  .. ipython:: python  from sqlalchemy import create_engine  # Create your connection.  engine = create_engine('sqlite:///')  This ``engine`` can then be used to write or read data to/from this database:  .. ipython:: python  df = pd.DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'c']})  df.to_sql('db_table', engine, index=False)  You can read data from a database by specifying the table name:  .. ipython:: python  pd.read_sql_table('db_table', engine)  or by specifying a sql query:  .. ipython:: python  pd.read_sql_query('SELECT * FROM db_table', engine)  Some other enhancements to the sql functions include:  - support for writing the index. This can be controlled with the ``index``  keyword (default is True).
- specify the column label to use when writing the index with ``index_label``.
- specify string columns to parse as datetimes with the ``parse_dates``  keyword in `~pandas.read_sql_query` and `~pandas.read_sql_table`.  .. warning::  Some of the existing functions or function aliases have been deprecated  and will be removed in future versions. This includes: ``tquery``, ``uquery``,  ``read_frame``, ``frame_query``, ``write_frame``.  .. warning::  The support for the 'mysql' flavor when using DBAPI connection objects has been deprecated.  MySQL will be further supported with SQLAlchemy engines (`6900`).  .. _whatsnew_0140.slicers:  Multi-indexing using slicers
~~~  In 0.14.0 we added a new way to slice MultiIndexed objects.
You can slice a MultiIndex by providing multiple indexers.  You can provide any of the selectors as if you are indexing by label, see `Selection by Label <indexing.label>`,
*deeper* levels, they will be implied as ``slice(None)``.  As usual, **both sides** of the slicers are included as this is label indexing.  See `the docs<advanced.mi_slicers>`

See also issues (`6134`, `4036`, `3057`, `2598`, `5641`, `7106`)  .. warning::  You should specify all axes in the ``.loc`` specifier, meaning the indexer for the **index** and  for the **columns**. Their are some ambiguous cases where the passed indexer could be mis-interpreted  as indexing *both* axes, rather than into say the MuliIndex for the rows.  You should do this:  .. code-block:: python  >>> df.loc[(slice('A1', 'A3'), ...), :]  # noqa: E901  rather than this:  .. code-block:: python  >>> df.loc[(slice('A1', 'A3'), ...)]  # noqa: E901  .. warning::  You will need to make sure that the selection axes are fully lexsorted!  .. ipython:: python  def mklbl(prefix, n):  return ["%s%s" % (prefix, i) for i in range(n)]  index = pd.MultiIndex.from_product([mklbl('A', 4),  mklbl('B', 2),  mklbl('C', 4),  mklbl('D', 2)])  columns = pd.MultiIndex.from_tuples([('a', 'foo'), ('a', 'bar'),  ('b', 'foo'), ('b', 'bah')],  names=['lvl0', 'lvl1'])  df = pd.DataFrame(np.arange(len(index) * len(columns)).reshape((len(index),  len(columns))),  index=index,  columns=columns).sort_index().sort_index(axis=1)  df  Basic MultiIndex slicing using slices, lists, and labels.  .. ipython:: python  df.loc[(slice('A1', 'A3'), slice(None), ['C1', 'C3']), :]  You can use a ``pd.IndexSlice`` to shortcut the creation of these slices  .. ipython:: python  idx = pd.IndexSlice  df.loc[idx[:, :, ['C1', 'C3']], idx[:, 'foo']]  It is possible to perform quite complicated selections using this method on multiple
axes at the same time.  .. ipython:: python  df.loc['A1', (slice(None), 'foo')]  df.loc[idx[:, :, ['C1', 'C3']], idx[:, 'foo']]  Using a boolean indexer you can provide selection related to the *values*.  .. ipython:: python  mask = df[('a', 'foo')] > 200  df.loc[idx[mask, :, ['C1', 'C3']], idx[:, 'foo']]  You can also specify the ``axis`` argument to ``.loc`` to interpret the passed
slicers on a single axis.  .. ipython:: python  df.loc(axis=0)[:, :, ['C1', 'C3']]  Furthermore you can *set* the values using these methods  .. ipython:: python  df2 = df.copy()  df2.loc(axis=0)[:, :, ['C1', 'C3']] = -10  df2  You can use a right-hand-side of an alignable object as well.  .. ipython:: python  df2 = df.copy()  df2.loc[idx[:, :, ['C1', 'C3']], :] = df2 * 1000  df2  .. _whatsnew_0140.plotting:  Plotting
~~~  - Hexagonal bin plots from ``DataFrame.plot`` with ``kind='hexbin'`` (`5478`), See `the docs<visualization.hexbin>`.
- ``DataFrame.plot`` and ``Series.plot`` now supports area plot with specifying ``kind='area'`` (`6656`), See `the docs<visualization.area_plot>`
- Pie plots from ``Series.plot`` and ``DataFrame.plot`` with ``kind='pie'`` (`6976`), See `the docs<visualization.pie>`.
- Plotting with Error Bars is now supported in the ``.plot`` method of ``DataFrame`` and ``Series`` objects (`3796`, `6834`), See `the docs<visualization.errorbars>`.
- ``DataFrame.plot`` and ``Series.plot`` now support a ``table`` keyword for plotting ``matplotlib.Table``, See `the docs<visualization.table>`. The ``table`` keyword can receive the following values.  - ``False``: Do nothing (default).  - ``True``: Draw a table using the ``DataFrame`` or ``Series`` called ``plot`` method. Data will be transposed to meet matplotlib's default layout.  - ``DataFrame`` or ``Series``: Draw matplotlib.table using the passed data. The data will be drawn as displayed in print method (not transposed automatically).  Also, helper function ``pandas.tools.plotting.table`` is added to create a table from ``DataFrame`` and ``Series``, and add it to an ``matplotlib.Axes``.  - ``plot(legend='reverse')`` will now reverse the order of legend labels for  most plot kinds. (`6014`)

- Line plot and area plot can be stacked by ``stacked=True`` (`6656`)  - Following keywords are now acceptable for `DataFrame.plot` with ``kind='bar'`` and ``kind='barh'``:  - ``width``: Specify the bar width. In previous versions, static value 0.5 was passed to matplotlib and it cannot be overwritten. (`6604`)  - ``align``: Specify the bar alignment. Default is ``center`` (different from matplotlib). In previous versions, pandas passes ``align='edge'`` to matplotlib and adjust the location to ``center`` by itself, and it results ``align`` keyword is not applied as expected. (`4525`)  - ``position``: Specify relative alignments for bar plot layout. From 0 (left/bottom-end) to 1(right/top-end). Default is 0.5 (center). (`6604`)  Because of the default ``align`` value changes, coordinates of bar plots are now located on integer values (0.0, 1.0, 2.0 ...). This is intended to make bar plot be located on the same coordinates as line plot. However, bar plot may differs unexpectedly when you manually adjust the bar location or drawing area, such as using ``set_xlim``, ``set_ylim``, etc. In this cases, please modify your script to meet with new coordinates.  - The `parallel_coordinates` function now takes argument ``color``  instead of ``colors``. A ``FutureWarning`` is raised to alert that  the old ``colors`` argument will not be supported in a future release. (`6956`)  - The `parallel_coordinates` and `andrews_curves` functions now take  positional argument ``frame`` instead of ``data``. A ``FutureWarning`` is  raised if the old ``data`` argument is used by name. (`6956`)  - `DataFrame.boxplot` now supports ``layout`` keyword (`6769`)
- `DataFrame.boxplot` has a new keyword argument, ``return_type``. It accepts ``'dict'``,  ``'axes'``, or ``'both'``, in which case a namedtuple with the matplotlib  axes and a dict of matplotlib Lines is returned.  .. _whatsnew_0140.prior_deprecations:  Prior version deprecations/changes
~~~  There are prior version deprecations that are taking effect as of 0.14.0.  - Remove `DateRange` in favor of `DatetimeIndex` (`6816`)
- Remove ``column`` keyword from ``DataFrame.sort`` (`4370`)
- Remove ``precision`` keyword from `set_eng_float_format` (`395`)
- Remove ``force_unicode`` keyword from `DataFrame.to_string`,  `DataFrame.to_latex`, and `DataFrame.to_html`; these function  encode in unicode by default (`2224`, `2225`)
- Remove ``nanRep`` keyword from `DataFrame.to_csv` and  `DataFrame.to_string` (`275`)
- Remove ``unique`` keyword from `HDFStore.select_column` (`3256`)
- Remove ``inferTimeRule`` keyword from `Timestamp.offset` (`391`)
- Remove ``name`` keyword from `get_data_yahoo` and  get_data_google` ( `commit b921d1a_ )
- Remove ``offset`` keyword from `DatetimeIndex` constructor  ( commit 3136390_ )
- Remove ``time_rule`` from several rolling-moment statistical functions, such  as `rolling_sum` (`1042`)
- Removed neg ``-`` boolean operations on numpy arrays in favor of inv ``~``, as this is going to  be deprecated in numpy 1.9 (`6960`)  .. _whatsnew_0140.deprecations:  Deprecations

~~~  - The `pivot_table`/`DataFrame.pivot_table` and `crosstab` functions  now take arguments ``index`` and ``columns`` instead of ``rows`` and ``cols``.  A  ``FutureWarning`` is raised to alert that the old ``rows`` and ``cols`` arguments  will not be supported in a future release (`5505`)  - The `DataFrame.drop_duplicates` and `DataFrame.duplicated` methods  now take argument ``subset`` instead of ``cols`` to better align with  `DataFrame.dropna`.  A ``FutureWarning`` is raised to alert that the old  ``cols`` arguments will not be supported in a future release (`6680`)  - The `DataFrame.to_csv` and `DataFrame.to_excel` functions  now takes argument ``columns`` instead of ``cols``.  A  ``FutureWarning`` is raised to alert that the old ``cols`` arguments  will not be supported in a future release (`6645`)  - Indexers will warn ``FutureWarning`` when used with a scalar indexer and  a non-floating point Index (`4892`, `6960`)  .. code-block:: ipython  # non-floating point indexes can only be indexed by integers / labels  In [1]: pd.Series(1, np.arange(5))[3.0]  pandas/core/index.py FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point  Out[1]: 1  In [2]: pd.Series(1, np.arange(5)).iloc[3.0]  pandas/core/index.py FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point  Out[2]: 1  In [3]: pd.Series(1, np.arange(5)).iloc[3.0:4]  pandas/core/index.py FutureWarning: slice indexers when using iloc should be integers and not floating point  Out[3]:  3  1  dtype: int64  # these are Float64Indexes, so integer or floating point is acceptable  In [4]: pd.Series(1, np.arange(5.))[3]  Out[4]: 1  In [5]: pd.Series(1, np.arange(5.))[3.0]  Out[6]: 1  - Numpy 1.9 compat w.r.t. deprecation warnings (`6960`)  - `Panel.shift` now has a function signature that matches `DataFrame.shift`.  The old positional argument ``lags`` has been changed to a keyword argument  ``periods`` with a default value of 1. A ``FutureWarning`` is raised if the  old argument ``lags`` is used by name. (`6910`)
- The ``order`` keyword argument of `factorize` will be removed. (`6926`).  - Remove the ``copy`` keyword from `DataFrame.xs`, `Panel.major_xs`, `Panel.minor_xs`. A view will be  returned if possible, otherwise a copy will be made. Previously the user could think that ``copy=False`` would  ALWAYS return a view. (`6894`)  - The `parallel_coordinates` function now takes argument ``color``  instead of ``colors``. A ``FutureWarning`` is raised to alert that  the old ``colors`` argument will not be supported in a future release. (`6956`)  - The `parallel_coordinates` and `andrews_curves` functions now take  positional argument ``frame`` instead of ``data``. A ``FutureWarning`` is  raised if the old ``data`` argument is used by name. (`6956`)  - The support for the 'mysql' flavor when using DBAPI connection objects has been deprecated.  MySQL will be further supported with SQLAlchemy engines (`6900`).  - The following ``io.sql`` functions have been deprecated: ``tquery``, ``uquery``, ``read_frame``, ``frame_query``, ``write_frame``.  - The ``percentile_width`` keyword argument in `~DataFrame.describe` has been deprecated.  Use the ``percentiles`` keyword instead, which takes a list of percentiles to display. The  default output is unchanged.  - The default return type of `boxplot` will change from a dict to a matplotlib Axes  in a future release. You can use the future behavior now by passing ``return_type='axes'``  to boxplot.  .. _whatsnew_0140.knownissues:  Known issues
~~~  - OpenPyXL 2.0.0 breaks backwards compatibility (`7169`)  .. _whatsnew_0140.enhancements:  Enhancements

~~~  - DataFrame and Series will create a MultiIndex object if passed a tuples dict, See `the docs<basics.dataframe.from_dict_of_tuples>` (`3323`)  .. ipython:: python  pd.Series({('a', 'b'): 1, ('a', 'a'): 0,  ('a', 'c'): 2, ('b', 'a'): 3, ('b', 'b'): 4})  pd.DataFrame({('a', 'b'): {('A', 'B'): 1, ('A', 'C'): 2},  ('a', 'a'): {('A', 'C'): 3, ('A', 'B'): 4},  ('a', 'c'): {('A', 'B'): 5, ('A', 'C'): 6},  ('b', 'a'): {('A', 'C'): 7, ('A', 'B'): 8},  ('b', 'b'): {('A', 'D'): 9, ('A', 'B'): 10}})  - Added the ``sym_diff`` method to ``Index`` (`5543`)
- ``DataFrame.to_latex`` now takes a longtable keyword, which if True will return a table in a longtable environment. (`6617`)
- Add option to turn off escaping in ``DataFrame.to_latex`` (`6472`)
- ``pd.read_clipboard`` will, if the keyword ``sep`` is unspecified, try to detect data copied from a spreadsheet  and parse accordingly. (`6223`)
- Joining a singly-indexed DataFrame with a MultiIndexed DataFrame (`3662`)  See `the docs<merging.join_on_mi>`. Joining MultiIndex DataFrames on both the left and right is not yet supported ATM.  .. ipython:: python  household = pd.DataFrame({'household_id': [1, 2, 3],  'male': [0, 1, 0],  'wealth': [196087.3, 316478.7, 294750]  },  columns=['household_id', 'male', 'wealth']  ).set_index('household_id')  household  portfolio = pd.DataFrame({'household_id': [1, 2, 2, 3, 3, 3, 4],  'asset_id': ["nl000301109",  "nl000289783",  "gb00b03mlx29",  "gb00b03mlx29",  "lu0197800237",  "nl000289965",  np.nan],  'name': ["ABN Amro",  "Robeco",  "Royal Dutch Shell",  "Royal Dutch Shell",  "AAB Eastern Europe Equity Fund",  "Postbank BioTech Fonds",  np.nan],  'share': [1.0, 0.4, 0.6, 0.15, 0.6, 0.25, 1.0]  },  columns=['household_id', 'asset_id', 'name', 'share']  ).set_index(['household_id', 'asset_id'])  portfolio  household.join(portfolio, how='inner')  - ``quotechar``, ``doublequote``, and ``escapechar`` can now be specified when  using ``DataFrame.to_csv`` (`5414`, `4528`)
- Partially sort by only the specified levels of a MultiIndex with the  ``sort_remaining`` boolean kwarg. (`3984`)
- Added ``to_julian_date`` to ``TimeStamp`` and ``DatetimeIndex``.  The Julian  Date is used primarily in astronomy and represents the number of days from  noon, January 1, 4713 BC.  Because nanoseconds are used to define the time  in pandas the actual range of dates that you can use is 1678 AD to 2262 AD. (`4041`)
- ``DataFrame.to_stata`` will now check data for compatibility with Stata data types  and will upcast when needed.  When it is not possible to losslessly upcast, a warning  is issued (`6327`)
- ``DataFrame.to_stata`` and ``StataWriter`` will accept keyword arguments time_stamp  and data_label which allow the time stamp and dataset label to be set when creating a  file. (`6545`)
- ``pandas.io.gbq`` now handles reading unicode strings properly. (`5940`)
- `Holidays Calendars<timeseries.holiday>` are now available and can be used with the ``CustomBusinessDay`` offset (`6719`)
- ``Float64Index`` is now backed by a ``float64`` dtype ndarray instead of an  ``object`` dtype array (`6471`).
- Implemented ``Panel.pct_change`` (`6904`)
- Added ``how`` option to rolling-moment functions to dictate how to handle resampling; `rolling_max` defaults to max,  `rolling_min` defaults to min, and all others default to mean (`6297`)
- ``CustomBusinessMonthBegin`` and ``CustomBusinessMonthEnd`` are now available (`6866`)
- `Series.quantile` and `DataFrame.quantile` now accept an array of  quantiles.
- `~DataFrame.describe` now accepts an array of percentiles to include in the summary statistics (`4196`)

- ``pivot_table`` can now accept ``Grouper`` by ``index`` and ``columns`` keywords (`6913`)  .. ipython:: python  import datetime  df = pd.DataFrame({  'Branch': 'A A A A A B'.split(),  'Buyer': 'Carl Mark Carl Carl Joe Joe'.split(),  'Quantity': [1, 3, 5, 1, 8, 1],  'Date': [datetime.datetime(2013, 11, 1, 13, 0),  datetime.datetime(2013, 9, 1, 13, 5),  datetime.datetime(2013, 10, 1, 20, 0),  datetime.datetime(2013, 10, 2, 10, 0),  datetime.datetime(2013, 11, 1, 20, 0),  datetime.datetime(2013, 10, 2, 10, 0)],  'PayDay': [datetime.datetime(2013, 10, 4, 0, 0),  datetime.datetime(2013, 10, 15, 13, 5),  datetime.datetime(2013, 9, 5, 20, 0),  datetime.datetime(2013, 11, 2, 10, 0),  datetime.datetime(2013, 10, 7, 20, 0),  datetime.datetime(2013, 9, 5, 10, 0)]})  df  df.pivot_table(values='Quantity',  index=pd.Grouper(freq='M', key='Date'),  columns=pd.Grouper(freq='M', key='PayDay'),  aggfunc=np.sum)  - Arrays of strings can be wrapped to a specified width (``str.wrap``) (`6999`)
- Add `~Series.nsmallest` and `Series.nlargest` methods to Series, See `the docs <basics.nsorted>` (`3960`)  - ``PeriodIndex`` fully supports partial string indexing like ``DatetimeIndex`` (`7043`)  .. ipython:: python  prng = pd.period_range('2013-01-01 09:00', periods=100, freq='H')  ps = pd.Series(np.random.randn(len(prng)), index=prng)  ps  ps['2013-01-02']  - ``read_excel`` can now read milliseconds in Excel dates and times with xlrd >= 0.9.3. (`5945`)
- ``pd.stats.moments.rolling_var`` now uses Welford's method for increased numerical stability (`6817`)
- pd.expanding_apply and pd.rolling_apply now take args and kwargs that are passed on to  the func (`6289`)
- ``DataFrame.rank()`` now has a percentage rank option (`5971`)
- ``Series.rank()`` now has a percentage rank option (`5971`)
- ``Series.rank()`` and ``DataFrame.rank()`` now accept ``method='dense'`` for ranks without gaps (`6514`)
- Support passing ``encoding`` with xlwt (`3710`)
- Refactor Block classes removing ``Block.items`` attributes to avoid duplication  in item handling (`6745`, `6988`).
- Testing statements updated to use specialized asserts (`6175`)  .. _whatsnew_0140.performance:  Performance
~~~  - Performance improvement when converting ``DatetimeIndex`` to floating ordinals  using ``DatetimeConverter`` (`6636`)
- Performance improvement for  ``DataFrame.shift`` (`5609`)
- Performance improvement in indexing into a MultiIndexed Series (`5567`)
- Performance improvements in single-dtyped indexing (`6484`)
- Improve performance of DataFrame construction with certain offsets, by removing faulty caching  (e.g. MonthEnd,BusinessMonthEnd), (`6479`)
- Improve performance of ``CustomBusinessDay`` (`6584`)
- improve performance of slice indexing on Series with string keys (`6341`, `6372`)
- Performance improvement for ``DataFrame.from_records`` when reading a  specified number of rows from an iterable (`6700`)
- Performance improvements in timedelta conversions for integer dtypes (`6754`)
- Improved performance of compatible pickles (`6899`)
- Improve performance in certain reindexing operations by optimizing ``take_2d`` (`6749`)
- ``GroupBy.count()`` is now implemented in Cython and is much faster for large  numbers of groups (`7016`).  Experimental
~~~  There are no experimental changes in 0.14.0  .. _whatsnew_0140.bug_fixes:  Bug fixes
~~~  - Bug in Series ValueError when index doesn't match data (`6532`)
- Prevent segfault due to MultiIndex not being supported in HDFStore table  format (`1848`)
- Bug in ``pd.DataFrame.sort_index`` where mergesort wasn't stable when ``ascending=False`` (`6399`)
- Bug in ``pd.tseries.frequencies.to_offset`` when argument has leading zeros (`6391`)
- Bug in version string gen. for dev versions with shallow clones / install from tarball (`6127`)

- Inconsistent tz parsing ``Timestamp`` / ``to_datetime`` for current year (`5958`)
- Indexing bugs with reordered indexes (`6252`, `6254`)
- Bug in ``.xs`` with a Series multiindex (`6258`, `5684`)
- Bug in conversion of a string types to a DatetimeIndex with a specified frequency (`6273`, `6274`)
- Bug in ``eval`` where type-promotion failed for large expressions (`6205`)
- Bug in interpolate with ``inplace=True`` (`6281`)
- ``HDFStore.remove`` now handles start and stop (`6177`)
- ``HDFStore.select_as_multiple`` handles start and stop the same way as ``select`` (`6177`)
- ``HDFStore.select_as_coordinates`` and ``select_column`` works with a ``where`` clause that results in filters (`6177`)
- Regression in join of non_unique_indexes (`6329`)
- Issue with groupby ``agg`` with a single function and a mixed-type frame (`6337`)
- Bug in ``DataFrame.replace()`` when passing a non- ``bool``  ``to_replace`` argument (`6332`)
- Raise when trying to align on different levels of a MultiIndex assignment (`3738`)
- Bug in setting complex dtypes via boolean indexing (`6345`)
- Bug in TimeGrouper/resample when presented with a non-monotonic DatetimeIndex that would return invalid results. (`4161`)
- Bug in index name propagation in TimeGrouper/resample (`4161`)
- TimeGrouper has a more compatible API to the rest of the groupers (e.g. ``groups`` was missing) (`3881`)
- Bug in multiple grouping with a TimeGrouper depending on target column order (`6764`)
- Bug in ``pd.eval`` when parsing strings with possible tokens like ``'&'``  (`6351`)
- Bug correctly handle placements of ``-inf`` in Panels when dividing by integer 0 (`6178`)
- ``DataFrame.shift`` with ``axis=1`` was raising (`6371`)
- Disabled clipboard tests until release time (run locally with ``nosetests -A disabled``) (`6048`).
- Bug in ``DataFrame.replace()`` when passing a nested ``dict`` that contained  keys not in the values to be replaced (`6342`)
- ``str.match`` ignored the na flag (`6609`).
- Bug in take with duplicate columns that were not consolidated (`6240`)
- Bug in interpolate changing dtypes (`6290`)
- Bug in ``Series.get``, was using a buggy access method (`6383`)
- Bug in hdfstore queries of the form ``where=[('date', '>=', datetime(2013,1,1)), ('date', '<=', datetime(2014,1,1))]`` (`6313`)
- Bug in ``DataFrame.dropna`` with duplicate indices (`6355`)
- Regression in chained getitem indexing with embedded list-like from 0.12 (`6394`)
- ``Float64Index`` with nans not comparing correctly (`6401`)
- ``eval``/``query`` expressions with strings containing the ``@`` character  will now work (`6366`).
- Bug in ``Series.reindex`` when specifying a ``method`` with some nan values was inconsistent (noted on a resample) (`6418`)
- Bug in `DataFrame.replace` where nested dicts were erroneously  depending on the order of dictionary keys and values (`5338`).
- Performance issue in concatenating with empty objects (`3259`)
- Clarify sorting of ``sym_diff`` on ``Index`` objects with ``NaN`` values (`6444`)
- Regression in ``MultiIndex.from_product`` with a ``DatetimeIndex`` as input (`6439`)
- Bug in ``str.extract`` when passed a non-default index (`6348`)
- Bug in ``str.split`` when passed ``pat=None`` and ``n=1`` (`6466`)
- Bug in ``io.data.DataReader`` when passed ``"F-F_Momentum_Factor"`` and ``data_source="famafrench"`` (`6460`)
- Bug in ``sum`` of a ``timedelta64[ns]`` series (`6462`)
- Bug in ``resample`` with a timezone and certain offsets (`6397`)
- Bug in ``iat/iloc`` with duplicate indices on a Series (`6493`)
- Bug in ``read_html`` where nan's were incorrectly being used to indicate  missing values in text. Should use the empty string for consistency with the  rest of pandas (`5129`).
- Bug in ``read_html`` tests where redirected invalid URLs would make one test  fail (`6445`).

- Bug in multi-axis indexing using ``.loc`` on non-unique indices (`6504`)
- Bug that caused _ref_locs corruption when slice indexing across columns axis of a DataFrame (`6525`)
- Regression from 0.13 in the treatment of numpy ``datetime64`` non-ns dtypes in Series creation (`6529`)
- ``.names`` attribute of MultiIndexes passed to ``set_index`` are now preserved (`6459`).
- Bug in setitem with a duplicate index and an alignable rhs (`6541`)
- Bug in setitem with ``.loc`` on mixed integer Indexes (`6546`)
- Bug in ``pd.read_stata`` which would use the wrong data types and missing values (`6327`)
- Bug in ``DataFrame.to_stata`` that lead to data loss in certain cases, and could be exported using the  wrong data types and missing values (`6335`)
- ``StataWriter`` replaces missing values in string columns by empty string (`6802`)
- Inconsistent types in ``Timestamp`` addition/subtraction (`6543`)
- Bug in preserving frequency across Timestamp addition/subtraction (`4547`)
- Bug in empty list lookup caused ``IndexError`` exceptions (`6536`, `6551`)
- ``Series.quantile`` raising on an ``object`` dtype (`6555`)
- Bug in ``.xs`` with a ``nan`` in level when dropped (`6574`)
- Bug in fillna with ``method='bfill/ffill'`` and ``datetime64[ns]`` dtype (`6587`)
- Bug in sql writing with mixed dtypes possibly leading to data loss (`6509`)
- Bug in ``Series.pop`` (`6600`)
- Bug in ``iloc`` indexing when positional indexer matched ``Int64Index`` of the corresponding axis and no reordering happened (`6612`)
- Bug in ``fillna`` with ``limit`` and ``value`` specified
- Bug in ``DataFrame.to_stata`` when columns have non-string names (`4558`)
- Bug in compat with ``np.compress``, surfaced in (`6658`)
- Bug in binary operations with a rhs of a Series not aligning (`6681`)
- Bug in ``DataFrame.to_stata`` which incorrectly handles nan values and ignores ``with_index`` keyword argument (`6685`)
- Bug in resample with extra bins when using an evenly divisible frequency (`4076`)
- Bug in consistency of groupby aggregation when passing a custom function (`6715`)
- Bug in resample when ``how=None`` resample freq is the same as the axis frequency (`5955`)
- Bug in downcasting inference with empty arrays (`6733`)
- Bug in ``obj.blocks`` on sparse containers dropping all but the last items of same for dtype (`6748`)
- Bug in unpickling ``NaT (NaTType)`` (`4606`)
- Bug in ``DataFrame.replace()`` where regex meta characters were being treated  as regex even when ``regex=False`` (`6777`).
- Bug in timedelta ops on 32-bit platforms (`6808`)
- Bug in setting a tz-aware index directly via ``.index`` (`6785`)
- Bug in expressions.py where numexpr would try to evaluate arithmetic ops  (`6762`).
- Bug in Makefile where it didn't remove Cython generated C files with ``make  clean`` (`6768`)
- Bug with numpy < 1.7.2 when reading long strings from ``HDFStore`` (`6166`)
- Bug in ``DataFrame._reduce`` where non bool-like (0/1) integers were being  converted into bools. (`6806`)
- Regression from 0.13 with ``fillna`` and a Series on datetime-like (`6344`)
- Bug in adding ``np.timedelta64`` to ``DatetimeIndex`` with timezone outputs incorrect results (`6818`)
- Bug in ``DataFrame.replace()`` where changing a dtype through replacement  would only replace the first occurrence of a value (`6689`)
- Better error message when passing a frequency of 'MS' in ``Period`` construction (GH5332)
- Bug in ``Series.__unicode__`` when ``max_rows=None`` and the Series has more than 1000 rows. (`6863`)
- Bug in ``groupby.get_group`` where a datelike wasn't always accepted (`5267`)
- Bug in ``groupBy.get_group`` created by ``TimeGrouper`` raises ``AttributeError`` (`6914`)
- Bug in ``DatetimeIndex.tz_localize`` and ``DatetimeIndex.tz_convert`` converting ``NaT`` incorrectly (`5546`)
- Bug in arithmetic operations affecting ``NaT`` (`6873`)

- Bug in ``Series.str.extract`` where the resulting ``Series`` from a single  group match wasn't renamed to the group name
- Bug in ``DataFrame.to_csv`` where setting ``index=False`` ignored the  ``header`` kwarg (`6186`)
- Bug in ``DataFrame.plot`` and ``Series.plot``, where the legend behave inconsistently when plotting to the same axes repeatedly (`6678`)
- Internal tests for patching ``__finalize__`` / bug in merge not finalizing (`6923`, `6927`)
- accept ``TextFileReader`` in ``concat``, which was affecting a common user idiom (`6583`)
- Bug in C parser with leading white space (`3374`)
- Bug in C parser with ``delim_whitespace=True`` and ``\r``-delimited lines
- Bug in python parser with explicit MultiIndex in row following column header (`6893`)
- Bug in ``Series.rank`` and ``DataFrame.rank`` that caused small floats (<1e-13) to all receive the same rank (`6886`)
- Bug in ``DataFrame.apply`` with functions that used ``*args`` or ``**kwargs`` and returned  an empty result (`6952`)
- Bug in sum/mean on 32-bit platforms on overflows (`6915`)
- Moved ``Panel.shift`` to ``NDFrame.slice_shift`` and fixed to respect multiple dtypes. (`6959`)
- Bug in enabling ``subplots=True`` in ``DataFrame.plot`` only has single column raises ``TypeError``, and ``Series.plot`` raises ``AttributeError`` (`6951`)
- Bug in ``DataFrame.plot`` draws unnecessary axes when enabling ``subplots`` and ``kind=scatter`` (`6951`)
- Bug in ``read_csv`` from a filesystem with non-utf-8 encoding (`6807`)
- Bug in ``iloc`` when setting / aligning (`6766`)
- Bug causing UnicodeEncodeError when get_dummies called with unicode values and a prefix (`6885`)
- Bug in timeseries-with-frequency plot cursor display (`5453`)
- Bug surfaced in ``groupby.plot`` when using a ``Float64Index`` (`7025`)
- Stopped tests from failing if options data isn't able to be downloaded from Yahoo (`7034`)
- Bug in ``parallel_coordinates`` and ``radviz`` where reordering of class column  caused possible color/class mismatch (`6956`)
- Bug in ``radviz`` and ``andrews_curves`` where multiple values of 'color'  were being passed to plotting method (`6956`)
- Bug in ``Float64Index.isin()`` where containing ``nan`` s would make indices  claim that they contained all the things (`7066`).
- Bug in ``DataFrame.boxplot`` where it failed to use the axis passed as the ``ax`` argument (`3578`)
- Bug in the ``XlsxWriter`` and ``XlwtWriter`` implementations that resulted in datetime columns being formatted without the time (`7075`)  were being passed to plotting method
- `read_fwf` treats ``None`` in ``colspec`` like regular python slices. It now reads from the beginning  or until the end of the line when ``colspec`` contains a ``None`` (previously raised a ``TypeError``)
- Bug in cache coherence with chained indexing and slicing; add ``_is_view`` property to ``NDFrame`` to correctly predict  views; mark ``is_copy`` on ``xs`` only if its an actual copy (and not a view) (`7084`)
- Bug in DatetimeIndex creation from string ndarray with ``dayfirst=True`` (`5917`)
- Bug in ``MultiIndex.from_arrays`` created from ``DatetimeIndex`` doesn't preserve ``freq`` and ``tz`` (`7090`)
- Bug in ``unstack`` raises ``ValueError`` when ``MultiIndex`` contains ``PeriodIndex`` (`4342`)
- Bug in ``boxplot`` and ``hist`` draws unnecessary axes (`6769`)
- Regression in ``groupby.nth()`` for out-of-bounds indexers (`6621`)
- Bug in ``quantile`` with datetime values (`6965`)
- Bug in ``Dataframe.set_index``, ``reindex`` and ``pivot`` don't preserve ``DatetimeIndex`` and ``PeriodIndex`` attributes (`3950`, `5878`, `6631`)
- Bug in ``MultiIndex.get_level_values`` doesn't preserve ``DatetimeIndex`` and ``PeriodIndex`` attributes (`7092`)
- Bug in ``Groupby`` doesn't preserve ``tz`` (`3950`)
- Bug in ``PeriodIndex`` partial string slicing (`6716`)

- Bug in the HTML repr of a truncated Series or DataFrame not showing the class name with the ``large_repr`` set to 'info'  (`7105`)
- Bug in ``DatetimeIndex`` specifying ``freq`` raises ``ValueError`` when passed value is too short (`7098`)
- Bug ``PeriodIndex`` string slicing with out of bounds values (`5407`)
- Fixed a memory error in the hashtable implementation/factorizer on resizing of large tables (`7157`)
- Bug in ``isnull`` when applied to 0-dimensional object arrays (`7176`)
- Bug in ``query``/``eval`` where global constants were not looked up correctly  (`7178`)
- Bug in recognizing out-of-bounds positional list indexers with ``iloc`` and a multi-axis tuple indexer (`7189`)
- Bug in setitem with a single value, MultiIndex and integer indices (`7190`, `7218`)
- Bug in expressions evaluation with reversed ops, showing in series-dataframe ops (`7198`, `7192`)
- Bug in multi-axis indexing with > 2 ndim and a MultiIndex (`7199`)
- Fix a bug where invalid eval/query operations would blow the stack (`5198`)  .. _whatsnew_0.14.0.contributors:  Contributors
~~~  .. contributors:: v0.13.1..v0.14.0

{{ header }}  This is a minor release from 0.14.0 and includes a small number of API changes, several new features,
users upgrade to this version.  - Highlights include:  - New methods `~pandas.DataFrame.select_dtypes` to select columns  based on the dtype and `~pandas.Series.sem` to calculate the  standard error of the mean.  - Support for dateutil timezones (see `docs <timeseries.timezone>`).  - Support for ignoring full line comments in the `~pandas.read_csv`  text parser.  - New documentation section on `Options and Settings <options>`.  - Lots of bug fixes.  - `Enhancements <whatsnew_0141.enhancements>`
- `API Changes <whatsnew_0141.api>`
- `Performance Improvements <whatsnew_0141.performance>`
- `Experimental Changes <whatsnew_0141.experimental>`
- `Bug Fixes <whatsnew_0141.bug_fixes>`  .. _whatsnew_0141.api:  API changes
~~~  - Openpyxl now raises a ValueError on construction of the openpyxl writer  instead of warning on pandas import (`7284`).  - For ``StringMethods.extract``, when no match is found, the result - only  containing ``NaN`` values - now also has ``dtype=object`` instead of  ``float`` (`7242`)  - ``Period`` objects no longer raise a ``TypeError`` when compared using ``==``  with another object that *isn't* a ``Period``. Instead  when comparing a ``Period`` with another object using ``==`` if the other  object isn't a ``Period`` ``False`` is returned. (`7376`)  - Previously, the behaviour on resetting the time or not in  ``offsets.apply``, ``rollforward`` and ``rollback`` operations differed  between offsets. With the support of the ``normalize`` keyword for all offsets(see  below) with a default value of False (preserve time), the behaviour changed for certain  offsets (BusinessMonthBegin, MonthEnd, BusinessMonthEnd, CustomBusinessMonthEnd,  BusinessYearBegin, LastWeekOfMonth, FY5253Quarter, LastWeekOfMonth, Easter):  .. code-block:: ipython  In [6]: from pandas.tseries import offsets  In [7]: d = pd.Timestamp('2014-01-01 09:00')  # old behaviour < 0.14.1  In [8]: d + offsets.MonthEnd()  Out[8]: pd.Timestamp('2014-01-31 000')  Starting from 0.14.1 all offsets preserve time by default. The old  behaviour can be obtained with ``normalize=True``  .. ipython:: python  import pandas.tseries.offsets as offsets  d = pd.Timestamp("2014-01-01 09:00")  .. ipython:: python  # new behaviour  d + offsets.MonthEnd()  d + offsets.MonthEnd(normalize=True)  Note that for the other offsets the default behaviour did not change.  - Add back ``#N/A N/A`` as a default NA value in text parsing, (regression from 0.12) (`5521`)
- Raise a ``TypeError`` on inplace-setting with a ``.where`` and a non ``np.nan`` value as this is inconsistent  with a set-item expression like ``df[mask] = None`` (`7656`)  .. _whatsnew_0141.enhancements:  Enhancements
~~~  - Add ``dropna`` argument to ``value_counts`` and ``nunique`` (`5569`).
- Add `~pandas.DataFrame.select_dtypes` method to allow selection of  columns based on dtype (`7316`). See `the docs <basics.selectdtypes>`.
- All ``offsets`` supports the ``normalize`` keyword to specify whether  ``offsets.apply``, ``rollforward`` and ``rollback`` resets the time (hour,  minute, etc) or not (default ``False``, preserves time) (`7156`):  .. code-block:: python  import pandas.tseries.offsets as offsets  day = offsets.Day()  day.apply(pd.Timestamp("2014-01-01 09:00"))  day = offsets.Day(normalize=True)  day.apply(pd.Timestamp("2014-01-01 09:00"))  - ``PeriodIndex`` is represented as the same format as ``DatetimeIndex`` (`7601`)
- ``StringMethods`` now work on empty Series (`7242`)
- The file parsers ``read_csv`` and ``read_table`` now ignore line comments provided by  the parameter ``comment``, which accepts only a single character for the C reader.  In particular, they allow for comments before file data begins (`2685`)

- Add ``NotImplementedError`` for simultaneous use of ``chunksize`` and ``nrows``  for read_csv() (`6774`).
- Tests for basic reading of public S3 buckets now exist (`7281`).
- ``read_html`` now sports an ``encoding`` argument that is passed to the  underlying parser library. You can use this to read non-ascii encoded web  pages (`7323`).
- ``read_excel`` now supports reading from URLs in the same way  that ``read_csv`` does.  (`6809`)
- Support for dateutil timezones, which can now be used in the same way as  pytz timezones across pandas. (`4688`)  .. ipython:: python  rng = pd.date_range(  "3/6/2012 00:00", periods=10, freq="D", tz="dateutil/Europe/London"  )  rng.tz  See `the docs <timeseries.timezone>`.  - Implemented ``sem`` (standard error of the mean) operation for ``Series``,  ``DataFrame``, ``Panel``, and ``Groupby`` (`6897`)
- Add ``nlargest`` and ``nsmallest`` to the ``Series`` ``groupby`` allowlist,  which means you can now use these methods on a ``SeriesGroupBy`` object  (`7053`).
- All offsets ``apply``, ``rollforward`` and ``rollback`` can now handle ``np.datetime64``, previously results in ``ApplyTypeError`` (`7452`)
- ``Period`` and ``PeriodIndex`` can contain ``NaT`` in its values (`7485`)
- Support pickling ``Series``, ``DataFrame`` and ``Panel`` objects with  non-unique labels along *item* axis (``index``, ``columns`` and ``items``  respectively) (`7370`).
- Improved inference of datetime/timedelta with mixed null objects. Regression from 0.13.1 in interpretation of an object Index  with all null elements (`7431`)  .. _whatsnew_0141.performance:  Performance
~~~
- Improvements in dtype inference for numeric operations involving yielding performance gains for dtypes: ``int64``, ``timedelta64``, ``datetime64`` (`7223`)
- Improvements in Series.transform for significant performance gains (`6496`)
- Improvements in DataFrame.transform with ufuncs and built-in grouper functions for significant performance gains (`7383`)
- Regression in groupby aggregation of datetime64 dtypes (`7555`)
- Improvements in ``MultiIndex.from_product`` for large iterables (`7627`)  .. _whatsnew_0141.experimental:  Experimental
~~~  - ``pandas.io.data.Options`` has a new method, ``get_all_data`` method, and now consistently returns a  MultiIndexed ``DataFrame`` (`5602`)
- ``io.gbq.read_gbq`` and ``io.gbq.to_gbq`` were refactored to remove the  dependency on the Google ``bq.py`` command line client. This submodule  now uses ``httplib2`` and the Google ``apiclient`` and ``oauth2client`` API client  libraries which should be more stable and, therefore, reliable than  ``bq.py``. See `the docs <io.bigquery>`. (`6937`).  .. _whatsnew_0141.bug_fixes:  Bug fixes
~~~
- Bug in ``DataFrame.where`` with a symmetric shaped frame and a passed other of a DataFrame (`7506`)
- Bug in Panel indexing with a MultiIndex axis (`7516`)
- Regression in datetimelike slice indexing with a duplicated index and non-exact end-points (`7523`)
- Bug in setitem with list-of-lists and single vs mixed types (`7551`:)
- Bug in time ops with non-aligned Series (`7500`)
- Bug in timedelta inference when assigning an incomplete Series (`7592`)
- Bug in groupby ``.nth`` with a Series and integer-like column name (`7559`)
- Bug in ``Series.get`` with a boolean accessor (`7407`)
- Bug in ``value_counts`` where ``NaT`` did not qualify as missing (``NaN``) (`7423`)
- Bug in ``to_timedelta`` that accepted invalid units and misinterpreted 'm/h' (`7611`, `6423`)
- Bug in line plot doesn't set correct ``xlim`` if ``secondary_y=True`` (`7459`)
- Bug in grouped ``hist`` and ``scatter`` plots use old ``figsize`` default (`7394`)
- Bug in plotting subplots with ``DataFrame.plot``, ``hist`` clears passed ``ax`` even if the number of subplots is one (`7391`).
- Bug in plotting subplots with ``DataFrame.boxplot`` with ``by`` kw raises ``ValueError`` if the number of subplots exceeds 1 (`7391`).
- Bug in subplots displays ``ticklabels`` and ``labels`` in different rule (`5897`)

- Bug in ``Panel.apply`` with a MultiIndex as an axis (`7469`)
- Bug in ``DatetimeIndex.insert`` doesn't preserve ``name`` and ``tz`` (`7299`)
- Bug in ``DatetimeIndex.asobject`` doesn't preserve ``name`` (`7299`)
- Bug in MultiIndex slicing with datetimelike ranges (strings and Timestamps), (`7429`)
- Bug in ``Index.min`` and ``max`` doesn't handle ``nan`` and ``NaT`` properly (`7261`)
- Bug in ``PeriodIndex.min/max`` results in ``int`` (`7609`)
- Bug in ``resample`` where ``fill_method`` was ignored if you passed ``how`` (`2073`)
- Bug in ``TimeGrouper`` doesn't exclude column specified by ``key`` (`7227`)
- Bug in ``DataFrame`` and ``Series`` bar and barh plot raises ``TypeError`` when ``bottom``  and ``left`` keyword is specified (`7226`)
- Bug in ``DataFrame.hist`` raises ``TypeError`` when it contains non numeric column (`7277`)
- Bug in ``Index.delete`` does not preserve ``name`` and ``freq`` attributes (`7302`)
- Bug in ``DataFrame.query()``/``eval`` where local string variables with the @  sign were being treated as temporaries attempting to be deleted  (`7300`).
- Bug in ``Float64Index`` which didn't allow duplicates (`7149`).
- Bug in ``DataFrame.replace()`` where truthy values were being replaced  (`7140`).
- Bug in ``StringMethods.extract()`` where a single match group Series  would use the matcher's name instead of the group name (`7313`).
- Bug in ``isnull()`` when ``mode.use_inf_as_null == True`` where isnull  wouldn't test ``True`` when it encountered an ``inf``/``-inf``  (`7315`).
- Bug in inferred_freq results in None for eastern hemisphere timezones (`7310`)
- Bug in ``Easter`` returns incorrect date when offset is negative (`7195`)
- Bug in broadcasting with ``.div``, integer dtypes and divide-by-zero (`7325`)
- Bug in ``CustomBusinessDay.apply`` raises ``NameError`` when ``np.datetime64`` object is passed (`7196`)
- Bug in ``MultiIndex.append``, ``concat`` and ``pivot_table`` don't preserve timezone (`6606`)
- Bug in ``.loc`` with a list of indexers on a single-multi index level (that is not nested) (`7349`)
- Bug in ``Series.map`` when mapping a dict with tuple keys of different lengths (`7333`)
- Bug all ``StringMethods`` now work on empty Series (`7242`)
- Fix delegation of ``read_sql`` to ``read_sql_query`` when query does not contain 'select' (`7324`).
- Bug where a string column name assignment to a ``DataFrame`` with a  ``Float64Index`` raised a ``TypeError`` during a call to ``np.isnan``  (`7366`).
- Bug where ``NDFrame.replace()`` didn't correctly replace objects with  ``Period`` values (`7379`).
- Bug in ``.ix`` getitem should always return a Series (`7150`)
- Bug in MultiIndex slicing with incomplete indexers (`7399`)
- Bug in MultiIndex slicing with a step in a sliced level (`7400`)
- Bug where negative indexers in ``DatetimeIndex`` were not correctly sliced  (`7408`)
- Bug where ``NaT`` wasn't repr'd correctly in a ``MultiIndex`` (`7406`,  `7409`).
- Bug where bool objects were converted to ``nan`` in ``convert_objects``  (`7416`).
- Bug in ``quantile`` ignoring the axis keyword argument (`7306`)
- Bug where ``nanops._maybe_null_out`` doesn't work with complex numbers  (`7353`)
- Bug in several ``nanops`` functions when ``axis==0`` for  1-dimensional ``nan`` arrays (`7354`)
- Bug where ``nanops.nanmedian`` doesn't work when ``axis==None``  (`7352`)
- Bug where ``nanops._has_infs`` doesn't work with many dtypes  (`7357`)
- Bug in ``StataReader.data`` where reading a 0-observation dta failed (`7369`)
- Bug in ``StataReader`` when reading Stata 13 (117) files containing fixed width strings (`7360`)
- Bug in ``StataWriter`` where encoding was ignored (`7286`)
- Bug in ``DatetimeIndex`` comparison doesn't handle ``NaT`` properly (`7529`)
- Bug in passing input with ``tzinfo`` to some offsets ``apply``, ``rollforward`` or ``rollback`` resets ``tzinfo`` or raises ``ValueError`` (`7465`)

- Bug in ``DatetimeIndex.to_period``, ``PeriodIndex.asobject``, ``PeriodIndex.to_timestamp`` doesn't preserve ``name`` (`7485`)
- Bug in ``DatetimeIndex.to_period`` and ``PeriodIndex.to_timestamp`` handle ``NaT`` incorrectly (`7228`)
- Bug in ``offsets.apply``, ``rollforward`` and ``rollback`` may return normal ``datetime`` (`7502`)
- Bug in ``resample`` raises ``ValueError`` when target contains ``NaT`` (`7227`)
- Bug in ``Timestamp.tz_localize`` resets ``nanosecond`` info (`7534`)
- Bug in ``DatetimeIndex.asobject`` raises ``ValueError`` when it contains ``NaT`` (`7539`)
- Bug in ``Timestamp.__new__`` doesn't preserve nanosecond properly (`7610`)
- Bug in ``Index.astype(float)`` where it would return an ``object`` dtype  ``Index`` (`7464`).
- Bug in ``DataFrame.reset_index`` loses ``tz`` (`3950`)
- Bug in ``DatetimeIndex.freqstr`` raises ``AttributeError`` when ``freq`` is ``None`` (`7606`)
- Bug in ``GroupBy.size`` created by ``TimeGrouper`` raises ``AttributeError`` (`7453`)
- Bug in single column bar plot is misaligned (`7498`).
- Bug in area plot with tz-aware time series raises ``ValueError`` (`7471`)
- Bug in non-monotonic ``Index.union`` may preserve ``name`` incorrectly (`7458`)
- Bug in ``DatetimeIndex.intersection`` doesn't preserve timezone (`4690`)
- Bug in ``rolling_var`` where a window larger than the array would raise an error(`7297`)
- Bug with last plotted timeseries dictating ``xlim`` (`2960`)
- Bug with ``secondary_y`` axis not being considered for timeseries ``xlim`` (`3490`)
- Bug in ``Float64Index`` assignment with a non scalar indexer (`7586`)
- Bug in ``pandas.core.strings.str_contains`` does not properly match in a case insensitive fashion when ``regex=False`` and ``case=False`` (`7505`)
- Bug in ``expanding_cov``, ``expanding_corr``, ``rolling_cov``, and ``rolling_corr`` for two arguments with mismatched index  (`7512`)
- Bug in ``to_sql`` taking the boolean column as text column (`7678`)
- Bug in grouped ``hist`` doesn't handle ``rot`` kw and ``sharex`` kw properly (`7234`)
- Bug in ``.loc`` performing fallback integer indexing with ``object`` dtype indices (`7496`)
- Bug (regression) in ``PeriodIndex`` constructor when passed ``Series`` objects (`7701`).  .. _whatsnew_0.14.1.contributors:  Contributors
~~~  .. contributors:: v0.14.0..v0.14.1

{{ header }}  This is a major release from 0.14.1 and includes a small number of API changes, several new features,
users upgrade to this version.  .. warning::  pandas >= 0.15.0 will no longer support compatibility with NumPy versions <  1.7.0. If you want to use the latest versions of pandas, please upgrade to  NumPy >= 1.7.0 (`7711`)  - Highlights include:  - The ``Categorical`` type was integrated as a first-class pandas type, see `here <whatsnew_0150.cat>`  - New scalar type ``Timedelta``, and a new index type ``TimedeltaIndex``, see `here <whatsnew_0150.timedeltaindex>`  - New datetimelike properties accessor ``.dt`` for Series, see `Datetimelike Properties <whatsnew_0150.dt>`  - New DataFrame default display for ``df.info()`` to include memory usage, see `Memory Usage <whatsnew_0150.memory>`  - ``read_csv`` will now by default ignore blank lines when parsing, see `here <whatsnew_0150.blanklines>`  - API change in using Indexes in set operations, see `here <whatsnew_0150.index_set_ops>`  - Enhancements in the handling of timezones, see `here <whatsnew_0150.tz>`  - A lot of improvements to the rolling and expanding moment functions, see `here <whatsnew_0150.roll>`  - Internal refactoring of the ``Index`` class to no longer sub-class ``ndarray``, see `Internal Refactoring <whatsnew_0150.refactoring>`  - dropping support for ``PyTables`` less than version 3.0.0, and ``numexpr`` less than version 2.1 (`7990`)  - Split indexing documentation into `Indexing and Selecting Data <indexing>` and `MultiIndex / Advanced Indexing <advanced>`  - Split out string methods documentation into `Working with Text Data <text>`  - Check the `API Changes <whatsnew_0150.api>` and `deprecations <whatsnew_0150.deprecations>` before updating  - `Other Enhancements <whatsnew_0150.enhancements>`  - `Performance Improvements <whatsnew_0150.performance>`  - `Bug Fixes <whatsnew_0150.bug_fixes>`  .. warning::  In 0.15.0 ``Index`` has internally been refactored to no longer sub-class ``ndarray``  but instead subclass ``PandasObject``, similarly to the rest of the pandas objects. This change allows very easy sub-classing and creation of new index types. This should be  a transparent change with only very limited API implications (See the `Internal Refactoring <whatsnew_0150.refactoring>`)  .. warning::  The refactoring in `~pandas.Categorical` changed the two argument constructor from  "codes/labels and levels" to "values and levels (now called 'categories')". This can lead to subtle bugs. If you use  `~pandas.Categorical` directly, please audit your code before updating to this pandas  version and change it to use the `~pandas.Categorical.from_codes` constructor. See more on ``Categorical`` `here <whatsnew_0150.cat>`  New features
~~~  .. _whatsnew_0150.cat:  Categoricals in Series/DataFrame
^^^  `~pandas.Categorical` can now be included in ``Series`` and ``DataFrames`` and gained new
methods to manipulate. Thanks to Jan Schulz for much of this API/implementation. (`3943`, `5313`, `5314`,
`7444`, `7839`, `7848`, `7864`, `7914`, `7768`, `8006`, `3678`,
`8075`, `8076`, `8143`, `8453`, `8518`).  For full docs, see the `categorical introduction <categorical>` and the

`API documentation <api.arrays.categorical>`.  .. ipython:: python  df = pd.DataFrame({"id": [1, 2, 3, 4, 5, 6],  "raw_grade": ['a', 'b', 'b', 'a', 'a', 'e']})  df["grade"] = df["raw_grade"].astype("category")  df["grade"]  # Rename the categories  df["grade"] = df["grade"].cat.rename_categories(["very good", "good", "very bad"])  # Reorder the categories and simultaneously add the missing categories  df["grade"] = df["grade"].cat.set_categories(["very bad", "bad",  "medium", "good", "very good"])  df["grade"]  df.sort_values("grade")  df.groupby("grade").size()  - ``pandas.core.group_agg`` and ``pandas.core.factor_agg`` were removed. As an alternative, construct  a dataframe and use ``df.groupby(<group>).agg(<func>)``.  - Supplying "codes/labels and levels" to the `~pandas.Categorical` constructor is not  supported anymore. Supplying two arguments to the constructor is now interpreted as  "values and levels (now called 'categories')". Please change your code to use the `~pandas.Categorical.from_codes`  constructor.  - The ``Categorical.labels`` attribute was renamed to ``Categorical.codes`` and is read  only. If you want to manipulate codes, please use one of the  `API methods on Categoricals <api.arrays.categorical>`.  - The ``Categorical.levels`` attribute is renamed to ``Categorical.categories``.  .. _whatsnew_0150.timedeltaindex:  TimedeltaIndex/scalar
^^^  We introduce a new scalar type ``Timedelta``, which is a subclass of ``datetime.timedelta``, and behaves in a similar manner,
but allows compatibility with ``np.timedelta64`` types as well as a host of custom representation, parsing, and attributes.
This type is very similar to how ``Timestamp`` works for ``datetimes``. It is a nice-API box for the type. See the `docs <timedeltas.timedeltas>`.

(`3009`, `4533`, `8209`, `8187`, `8190`, `7869`, `7661`, `8345`, `8471`)  .. warning::  ``Timedelta`` scalars (and ``TimedeltaIndex``) component fields are *not the same* as the component fields on a ``datetime.timedelta`` object. For example, ``.seconds`` on a ``datetime.timedelta`` object returns the total number of seconds combined between ``hours``, ``minutes`` and ``seconds``. In contrast, the pandas ``Timedelta`` breaks out hours, minutes, microseconds and nanoseconds separately.  .. code-block:: ipython  # Timedelta accessor  In [9]: tds = pd.Timedelta('31 days 5 min 3 sec')  In [10]: tds.minutes  Out[10]: 5L  In [11]: tds.seconds  Out[11]: 3L  # datetime.timedelta accessor  # this is 5 minutes * 60 + 3 seconds  In [12]: tds.to_pytimedelta().seconds  Out[12]: 303  **Note**: this is no longer true starting from v0.16.0, where full  compatibility with ``datetime.timedelta`` is introduced. See the  `0.16.0 whatsnew entry <whatsnew_0160.api_breaking.timedelta>`  .. warning::  Prior to 0.15.0 ``pd.to_timedelta`` would return a ``Series`` for list-like/Series input, and a ``np.timedelta64`` for scalar input.  It will now return a ``TimedeltaIndex`` for list-like input, ``Series`` for Series input, and ``Timedelta`` for scalar input.  The arguments to ``pd.to_timedelta`` are now ``(arg,unit='ns',box=True,coerce=False)``, previously were ``(arg,box=True,unit='ns')`` as these are more logical.  Construct a scalar  .. ipython:: python  pd.Timedelta('1 days 0601.0003')  pd.Timedelta('15.5us')  pd.Timedelta('1 hour 15.5us')  # negative Timedeltas have this string repr  # to be more consistent with datetime.timedelta conventions  pd.Timedelta('-1us')  # a NaT  pd.Timedelta('nan')  Access fields for a ``Timedelta``  .. ipython:: python  td = pd.Timedelta('1 hour 3m 15.5us')  td.seconds  td.microseconds  td.nanoseconds  Construct a ``TimedeltaIndex``  .. ipython:: python  import datetime  .. ipython:: python  pd.TimedeltaIndex(['1 days', '1 days, 0005',  np.timedelta64(2, 'D'),  datetime.timedelta(days=2, seconds=2)])  Constructing a ``TimedeltaIndex`` with a regular range  .. ipython:: python  pd.timedelta_range('1 days', periods=5, freq='D')  pd.timedelta_range(start='1 days', end='2 days', freq='30T')  You can now use a ``TimedeltaIndex`` as the index of a pandas object  .. ipython:: python  s = pd.Series(np.arange(5),  index=pd.timedelta_range('1 days', periods=5, freq='s'))  s  You can select with partial string selections  .. ipython:: python  s['1 day 0002']  s['1 day':'1 day 0002']  Finally, the combination of ``TimedeltaIndex`` with ``DatetimeIndex`` allow certain combination operations that are ``NaT`` preserving:  .. ipython:: python  tdi = pd.TimedeltaIndex(['1 days', pd.NaT, '2 days'])  tdi.tolist()  dti = pd.date_range('20130101', periods=3)  dti.tolist()  (dti + tdi).tolist()  (dti - tdi).tolist()  - iteration of a ``Series`` e.g. ``list(Series(...))`` of ``timedelta64[ns]`` would prior to v0.15.0 return ``np.timedelta64`` for each element. These will now be wrapped in ``Timedelta``.  .. _whatsnew_0150.memory:  Memory usage

^^^  Implemented methods to find memory usage of a DataFrame. See the `FAQ <df-memory-usage>` for more. (`6852`).  A new display option ``display.memory_usage`` (see `options`) sets the default behavior of the ``memory_usage`` argument in the ``df.info()`` method. By default ``display.memory_usage`` is ``True``.  .. ipython:: python  dtypes = ['int64', 'float64', 'datetime64[ns]', 'timedelta64[ns]',  'complex128', 'object', 'bool']  n = 5000  data = {t: np.random.randint(100, size=n).astype(t) for t in dtypes}  df = pd.DataFrame(data)  df['categorical'] = df['object'].astype('category')  df.info()  Additionally `~pandas.DataFrame.memory_usage` is an available method for a dataframe object which returns the memory usage of each column.  .. ipython:: python  df.memory_usage(index=True)  .. _whatsnew_0150.dt:  Series.dt accessor
^^^  ``Series`` has gained an accessor to succinctly return datetime like properties for the *values* of the Series, if its a datetime/period like Series. (`7207`)
This will return a Series, indexed like the existing Series. See the `docs <basics.dt_accessors>`  .. ipython:: python  # datetime  s = pd.Series(pd.date_range('20130101 0912', periods=4))  s  s.dt.hour  s.dt.second  s.dt.day  s.dt.freq  This enables nice expressions like this:  .. ipython:: python  s[s.dt.day == 2]  You can easily produce tz aware transformations:  .. ipython:: python  stz = s.dt.tz_localize('US/Eastern')  stz  stz.dt.tz  You can also chain these types of operations:  .. ipython:: python  s.dt.tz_localize('UTC').dt.tz_convert('US/Eastern')  The ``.dt`` accessor works for period and timedelta dtypes.  .. ipython:: python  # period  s = pd.Series(pd.period_range('20130101', periods=4, freq='D'))  s  s.dt.year  s.dt.day  .. ipython:: python  # timedelta  s = pd.Series(pd.timedelta_range('1 day 0005', periods=4, freq='s'))  s  s.dt.days  s.dt.seconds  s.dt.components  .. _whatsnew_0150.tz:  Timezone handling improvements
^^^  - ``tz_localize(None)`` for tz-aware ``Timestamp`` and ``DatetimeIndex`` now removes timezone holding local time,  previously this resulted in ``Exception`` or ``TypeError`` (`7812`)  .. ipython:: python  ts = pd.Timestamp('2014-08-01 09:00', tz='US/Eastern')  ts  ts.tz_localize(None)  didx = pd.date_range(start='2014-08-01 09:00', freq='H',  periods=10, tz='US/Eastern')  didx  didx.tz_localize(None)  - ``tz_localize`` now accepts the ``ambiguous`` keyword which allows for passing an array of bools  indicating whether the date belongs in DST or not, 'NaT' for setting transition times to NaT,  'infer' for inferring DST/non-DST, and 'raise' (default) for an ``AmbiguousTimeError`` to be raised. See `the docs<timeseries.timezone_ambiguous>` for more details (`7943`)  - ``DataFrame.tz_localize`` and ``DataFrame.tz_convert`` now accepts an optional ``level`` argument  for localizing a specific level of a MultiIndex (`7846`)  - ``Timestamp.tz_localize`` and ``Timestamp.tz_convert`` now raise ``TypeError`` in error cases, rather than ``Exception`` (`8025`)  - a timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone (rather than being a naive ``datetime64[ns]``) as ``object`` dtype (`8411`)  - ``Timestamp.__repr__`` displays ``dateutil.tz.tzoffset`` info (`7907`)  .. _whatsnew_0150.roll:  Rolling/expanding moments improvements

^^^  - `rolling_min`, `rolling_max`, `rolling_cov`, and `rolling_corr`  now return objects with all ``NaN`` when ``len(arg) < min_periods <= window`` rather  than raising. (This makes all rolling functions consistent in this behavior). (`7766`)  Prior to 0.15.0  .. ipython:: python  s = pd.Series([10, 11, 12, 13])  .. code-block:: ipython  In [15]: pd.rolling_min(s, window=10, min_periods=5)  ValueError: min_periods (5) must be <= window (4)  New behavior  .. code-block:: ipython  In [4]: pd.rolling_min(s, window=10, min_periods=5)  Out[4]:  0  NaN  1  NaN  2  NaN  3  NaN  dtype: float64  - `rolling_max`, `rolling_min`, `rolling_sum`, `rolling_mean`, `rolling_median`,  `rolling_std`, `rolling_var`, `rolling_skew`, `rolling_kurt`, `rolling_quantile`,  `rolling_cov`, `rolling_corr`, `rolling_corr_pairwise`,  `rolling_window`, and `rolling_apply` with ``center=True`` previously would return a result of the same  structure as the input ``arg`` with ``NaN`` in the final ``(window-1)/2`` entries.  Now the final ``(window-1)/2`` entries of the result are calculated as if the input ``arg`` were followed  by ``(window-1)/2`` ``NaN`` values (or with shrinking windows, in the case of `rolling_apply`).  (`7925`, `8269`)  Prior behavior (note final value is ``NaN``):  .. code-block:: ipython  In [7]: pd.rolling_sum(Series(range(4)), window=3, min_periods=0, center=True)  Out[7]:  0  1  1  3  2  6  3  NaN  dtype: float64  New behavior (note final value is ``5 = sum([2, 3, NaN])``):  .. code-block:: ipython  In [7]: pd.rolling_sum(pd.Series(range(4)), window=3,  ...:  min_periods=0, center=True)  Out[7]:  0  1  1  3  2  6  3  5  dtype: float64  - `rolling_window` now normalizes the weights properly in rolling mean mode (`mean=True`) so that  the calculated weighted means (e.g. 'triang', 'gaussian') are distributed about the same means as those  calculated without weighting (i.e. 'boxcar'). See `the note on normalization <window.weighted>` for further details. (`7618`)  .. ipython:: python  s = pd.Series([10.5, 8.8, 11.4, 9.7, 9.3])  Behavior prior to 0.15.0:  .. code-block:: ipython  In [39]: pd.rolling_window(s, window=3, win_type='triang', center=True)  Out[39]:  0  NaN  1  6.58333  2  6.88333  3  6.68333  4  NaN  dtype: float64  New behavior  .. code-block:: ipython  In [10]: pd.rolling_window(s, window=3, win_type='triang', center=True)  Out[10]:  0  NaN  1  9.875  2  10.325  3  10.025  4  NaN  dtype: float64  - Removed ``center`` argument from all `expanding_ <expanding_apply>` functions (see `list <api.functions_expanding>`),  as the results produced when ``center=True`` did not make much sense. (`7925`)  - Added optional ``ddof`` argument to `expanding_cov` and `rolling_cov`.  The default value of ``1`` is backwards-compatible. (`8279`)  - Documented the ``ddof`` argument to `expanding_var`, `expanding_std`,  `rolling_var`, and `rolling_std`. These functions' support of a  ``ddof`` argument (with a default value of ``1``) was previously undocumented. (`8064`)  - `ewma`, `ewmstd`, `ewmvol`, `ewmvar`, `ewmcov`, and `ewmcorr`  now interpret ``min_periods`` in the same manner that the `rolling_*()` and `expanding_*()` functions do:  a given result entry will be ``NaN`` if the (expanding, in this case) window does not contain  at least ``min_periods`` values. The previous behavior was to set to ``NaN`` the ``min_periods`` entries  starting with the first non- ``NaN`` value. (`7977`)  Prior behavior (note values start at index ``2``, which is ``min_periods`` after index ``0``  (the index of the first non-empty value)):  .. ipython:: python  s  = pd.Series([1, None, None, None, 2, 3])  .. code-block:: ipython  In [51]: pd.ewma(s, com=3., min_periods=2)  Out[51]:  0  NaN  1  NaN  2  1.000  3  1.000  4  1.571429  5  2.189189  dtype: float64  New behavior (note values start at index ``4``, the location of the 2nd (since ``min_periods=2``) non-empty value):  .. code-block:: ipython  In [2]: pd.ewma(s, com=3., min_periods=2)  Out[2]:  0  NaN  1  NaN  2  NaN  3  NaN  4  1.759644  5  2.383784  dtype: float64  - `ewmstd`, `ewmvol`, `ewmvar`, `ewmcov`, and `ewmcorr`  now have an optional ``adjust`` argument, just like `ewma` does,  affecting how the weights are calculated.  The default value of ``adjust`` is ``True``, which is backwards-compatible.  See `Exponentially weighted moment functions <window.exponentially_weighted>` for details. (`7911`)  - `ewma`, `ewmstd`, `ewmvol`, `ewmvar`, `ewmcov`, and `ewmcorr`  now have an optional ``ignore_na`` argument.  When ``ignore_na=False`` (the default), missing values are taken into account in the weights calculation.  When ``ignore_na=True`` (which reproduces the pre-0.15.0 behavior), missing values are ignored in the weights calculation.  (`7543`)  .. code-block:: ipython

missing values are taken into account in the weights calculation.  When ``ignore_na=True`` (which reproduces the pre-0.15.0 behavior), missing values are ignored in the weights calculation.  (`7543`)  .. code-block:: ipython  In [7]: pd.ewma(pd.Series([None, 1., 8.]), com=2.)  Out[7]:  0  NaN  1  1.0  2  5.2  dtype: float64  In [8]: pd.ewma(pd.Series([1., None, 8.]), com=2.,  ...:  ignore_na=True)  # pre-0.15.0 behavior  Out[8]:  0  1.0  1  1.0  2  5.2  dtype: float64  In [9]: pd.ewma(pd.Series([1., None, 8.]), com=2.,  ...:  ignore_na=False)  # new default  Out[9]:  0  1.000  1  1.000  2  5.846154  dtype: float64  .. warning::  By default (``ignore_na=False``) the `ewm*()` functions' weights calculation  in the presence of missing values is different than in pre-0.15.0 versions.  To reproduce the pre-0.15.0 calculation of weights in the presence of missing values  one must specify explicitly ``ignore_na=True``.  - Bug in `expanding_cov`, `expanding_corr`, `rolling_cov`, `rolling_cor`, `ewmcov`, and `ewmcorr`  returning results with columns sorted by name and producing an error for non-unique columns;  now handles non-unique columns and returns columns in original order  (except for the case of two DataFrames with ``pairwise=False``, where behavior is unchanged) (`7542`)

- Bug in `rolling_count` and `expanding_*()` functions unnecessarily producing error message for zero-length data (`8056`)
- Bug in `rolling_apply` and `expanding_apply` interpreting ``min_periods=0`` as ``min_periods=1`` (`8080`)
- Bug in `expanding_std` and `expanding_var` for a single value producing a confusing error message (`7900`)
- Bug in `rolling_std` and `rolling_var` for a single value producing ``0`` rather than ``NaN`` (`7900`)  - Bug in `ewmstd`, `ewmvol`, `ewmvar`, and `ewmcov`  calculation of de-biasing factors when ``bias=False`` (the default).  Previously an incorrect constant factor was used, based on ``adjust=True``, ``ignore_na=True``,  and an infinite number of observations.  Now a different factor is used for each entry, based on the actual weights  (analogous to the usual ``N/(N-1)`` factor).  In particular, for a single point a value of ``NaN`` is returned when ``bias=False``,  whereas previously a value of (approximately) ``0`` was returned.  For example, consider the following pre-0.15.0 results for ``ewmvar(..., bias=False)``,  and the corresponding debiasing factors:  .. ipython:: python  s = pd.Series([1., 2., 0., 4.])  .. code-block:: ipython  In [89]: pd.ewmvar(s, com=2., bias=False)  Out[89]:  0  -2.775558e-16  1  3.000e-01  2  9.556787e-01  3  3.585799e+00  dtype: float64  In [90]: pd.ewmvar(s, com=2., bias=False) / pd.ewmvar(s, com=2., bias=True)  Out[90]:  0  1.25  1  1.25  2  1.25  3  1.25  dtype: float64  Note that entry ``0`` is approximately 0, and the debiasing factors are a constant 1.25.  By comparison, the following 0.15.0 results have a ``NaN`` for entry ``0``,  and the debiasing factors are decreasing (towards 1.25):  .. code-block:: ipython  In [14]: pd.ewmvar(s, com=2., bias=False)  Out[14]:  0  NaN  1  0.5000  2  1.210526  3  4.089069  dtype: float64  In [15]: pd.ewmvar(s, com=2., bias=False) / pd.ewmvar(s, com=2., bias=True)  Out[15]:  0  NaN  1  2.08333  2  1.58333  3  1.425439  dtype: float64  See `Exponentially weighted moment functions <window.exponentially_weighted>` for details. (`7912`)  .. _whatsnew_0150.sql:  Improvements in the SQL IO module
^^^  - Added support for a ``chunksize`` parameter to ``to_sql`` function. This allows DataFrame to be written in chunks and avoid packet-size overflow errors (`8062`).
- Added support for a ``chunksize`` parameter to ``read_sql`` function. Specifying this argument will return an iterator through chunks of the query result (`2908`).
- Added support for writing ``datetime.date`` and ``datetime.time`` object columns with ``to_sql`` (`6932`).
- Added support for specifying a ``schema`` to read from/write to with ``read_sql_table`` and ``to_sql`` (`7441`, `7952`).  For example:  .. code-block:: python  df.to_sql('table', engine, schema='other_schema')  # noqa F821  pd.read_sql_table('table', engine, schema='other_schema')  # noqa F821  - Added support for writing ``NaN`` values with ``to_sql`` (`2754`).
- Added support for writing datetime64 columns with ``to_sql`` for all database flavors (`7103`).  .. _whatsnew_0150.api:  Backwards incompatible API changes
~~~  .. _whatsnew_0150.api_breaking:  Breaking changes
^^^  API changes related to ``Categorical`` (see `here <whatsnew_0150.cat>`
for more details):  - The ``Categorical`` constructor with two arguments changed from  "codes/labels and levels" to "values and levels (now called 'categories')".  This can lead to subtle bugs. If you use `~pandas.Categorical` directly,  please audit your code by changing it to use the `~pandas.Categorical.from_codes`  constructor.  An old function call like (prior to 0.15.0):  .. code-block:: python  pd.Categorical([0,1,0,2,1], levels=['a', 'b', 'c'])  will have to adapted to the following to keep the same behaviour:  .. code-block:: ipython  In [2]: pd.Categorical.from_codes([0,1,0,2,1], categories=['a', 'b', 'c'])  Out[2]:  [a, b, a, c, b]  Categories (3, object): [a, b, c]  API changes related to the introduction of the ``Timedelta`` scalar (see

`above <whatsnew_0150.timedeltaindex>` for more details):  - Prior to 0.15.0 `to_timedelta` would return a ``Series`` for list-like/Series input,  and a ``np.timedelta64`` for scalar input. It will now return a ``TimedeltaIndex`` for  list-like input, ``Series`` for Series input, and ``Timedelta`` for scalar input.  For API changes related to the rolling and expanding functions, see detailed overview `above <whatsnew_0150.roll>`.  Other notable API changes:  - Consistency when indexing with ``.loc`` and a list-like indexer when no values are found.  .. ipython:: python  df = pd.DataFrame([['a'], ['b']], index=[1, 2])  df  In prior versions there was a difference in these two constructs:  - ``df.loc[[3]]`` would return a frame reindexed by 3 (with all ``np.nan`` values)  - ``df.loc[[3],:]`` would raise ``KeyError``.  Both will now raise a ``KeyError``. The rule is that *at least 1* indexer must be found when using a list-like and ``.loc`` (`7999`)  Furthermore in prior versions these were also different:  - ``df.loc[[1,3]]`` would return a frame reindexed by [1,3]  - ``df.loc[[1,3],:]`` would raise ``KeyError``.  Both will now return a frame reindex by [1,3]. E.g.  .. code-block:: ipython  In [3]: df.loc[[1, 3]]  Out[3]:  0  1  a  3  NaN  In [4]: df.loc[[1, 3], :]  Out[4]:  0  1  a  3  NaN  This can also be seen in multi-axis indexing with a ``Panel``.  .. code-block:: python  >>> p = pd.Panel(np.arange(2 * 3 * 4).reshape(2, 3, 4),  ...  items=['ItemA', 'ItemB'],  ...  major_axis=[1, 2, 3],  ...  minor_axis=['A', 'B', 'C', 'D'])  >>> p  <class 'pandas.core.panel.Panel'>  Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)  Items axis: ItemA to ItemB  Major_axis axis: 1 to 3  Minor_axis axis: A to D  The following would raise ``KeyError`` prior to 0.15.0:  .. code-block:: ipython  In [5]:  Out[5]:  ItemA  ItemD  1  3  NaN  2  7  NaN  3  11  NaN  Furthermore, ``.loc`` will raise If no values are found in a MultiIndex with a list-like indexer:  .. ipython:: python  s = pd.Series(np.arange(3, dtype='int64'),  index=pd.MultiIndex.from_product([['A'],  ['foo', 'bar', 'baz']],  names=['one', 'two'])  ).sort_index()  s  try:  s.loc[['D']]  except KeyError as e:  print("KeyError: " + str(e))  - Assigning values to ``None`` now considers the dtype when choosing an 'empty' value (`7941`).  Previously, assigning to ``None`` in numeric containers changed the  dtype to object (or errored, depending on the call). It now uses  ``NaN``:  .. ipython:: python  s = pd.Series([1, 2, 3])  s.loc[0] = None  s  ``NaT`` is now used similarly for datetime containers.  For object containers, we now preserve ``None`` values (previously these  were converted to ``NaN`` values).  .. ipython:: python  s = pd.Series(["a", "b", "c"])  s.loc[0] = None  s  To insert a ``NaN``, you must explicitly use ``np.nan``. See the `docs <missing.inserting>`.  - In prior versions, updating a pandas object inplace would not reflect in other python references to this object. (`8511`, `5104`)  .. ipython:: python  s = pd.Series([1, 2, 3])  s2 = s  s += 1.5  Behavior prior to v0.15.0  .. code-block:: ipython  # the original object  In [5]: s  Out[5]:  0  2.5  1  3.5  2  4.5  dtype: float64  # a reference to the original object  In [7]: s2  Out[7]:  0  1  1  2  2  3  dtype: int64  This is now the correct behavior  .. ipython:: python  # the original object  s  # a reference to the original object  s2  .. _whatsnew_0150.blanklines:  - Made both the C-based and Python engines for ``read_csv`` and ``read_table`` ignore empty lines in input as well as  white space-filled lines, as long as ``sep`` is not white space. This is an API change  that can be controlled by the keyword parameter ``skip_blank_lines``.  See `the docs <io.skiplines>` (`4466`)  - A timeseries/index localized to UTC when inserted into a Series/DataFrame will preserve the UTC timezone  and inserted as ``object`` dtype rather than being converted to a naive ``datetime64[ns]`` (`8411`).  - Bug in passing a ``DatetimeIndex`` with a timezone that was not being retained in DataFrame construction from a dict (`7822`)  In prior versions this would drop the timezone, now it retains the timezone,  but gives a column of ``object`` dtype:  .. ipython:: python  i = pd.date_range('1/1/2011', periods=3, freq='10s', tz='US/Eastern')  i  df = pd.DataFrame({'a': i})  df  df.dtypes  Previously this would have yielded a column of ``datetime64`` dtype, but without timezone info.  The behaviour of assigning a column to an existing dataframe as ``df['a'] = i``  remains unchanged (this already returned an  ``object`` column with a timezone).  - When passing multiple levels to `~pandas.DataFrame.stack()`, it will now raise a ``ValueError`` when the  levels aren't all level names or all level numbers (`7660`). See  `Reshaping by stacking and unstacking <reshaping.stack_multiple>`.  - Raise a

levels to `~pandas.DataFrame.stack()`, it will now raise a ``ValueError`` when the  levels aren't all level names or all level numbers (`7660`). See  `Reshaping by stacking and unstacking <reshaping.stack_multiple>`.  - Raise a ``ValueError`` in ``df.to_hdf`` with 'fixed' format, if ``df`` has non-unique columns as the resulting file will be broken (`7761`)  - ``SettingWithCopy`` raise/warnings (according to the option ``mode.chained_assignment``) will now be issued when setting a value on a sliced mixed-dtype DataFrame using chained-assignment. (`7845`, `7950`)  .. code-block:: python  In [1]: df = pd.DataFrame(np.arange(0, 9), columns=['count'])  In [2]: df['group'] = 'b'  In [3]: df.iloc[0:5]['group'] = 'a'  /usr/local/bin/ipython SettingWithCopyWarning:  A value is trying to be set on a copy of a slice from a DataFrame.  Try using .loc[row_indexer,col_indexer] = value instead  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy  - ``merge``, ``DataFrame.merge``, and ``ordered_merge`` now return the same type  as the ``left`` argument (`7737`).  - Previously an enlargement with a mixed-dtype frame would act unlike ``.append`` which will preserve dtypes (related `2578`, `8176`):  .. ipython:: python  df = pd.DataFrame([[True, 1], [False, 2]],  columns=["female", "fitness"])  df  df.dtypes  # dtypes are now preserved  df.loc[2] = df.loc[1]  df  df.dtypes  - ``Series.to_csv()`` now returns a string when ``path=None``, matching the behaviour of ``DataFrame.to_csv()`` (`8215`).  - ``read_hdf`` now raises ``IOError`` when a file that doesn't exist is passed in. Previously, a new, empty file was created, and a ``KeyError`` raised (`7715`).  - ``DataFrame.info()`` now ends its output with a newline character (`8114`)

- Concatenating no objects will now raise a ``ValueError`` rather than a bare ``Exception``.
- Merge errors will now be sub-classes of ``ValueError`` rather than raw ``Exception`` (`8501`)
- ``DataFrame.plot`` and ``Series.plot`` keywords are now have consistent orders (`8037`)  .. _whatsnew_0150.refactoring:  Internal refactoring
^^^  In 0.15.0 ``Index`` has internally been refactored to no longer sub-class ``ndarray``
but instead subclass ``PandasObject``, similarly to the rest of the pandas objects. This
change allows very easy sub-classing and creation of new index types. This should be
a transparent change with only very limited API implications (`5080`, `7439`, `7796`, `8024`, `8367`, `7997`, `8522`):  - you may need to unpickle pandas version < 0.15.0 pickles using ``pd.read_pickle`` rather than ``pickle.load``. See `pickle docs <io.pickle>`
- when plotting with a ``PeriodIndex``, the matplotlib internal axes will now be arrays of ``Period`` rather than a ``PeriodIndex`` (this is similar to how a ``DatetimeIndex`` passes arrays of ``datetimes`` now)
- MultiIndexes will now raise similarly to other pandas objects w.r.t. truth testing, see `here <gotchas.truth>` (`7897`).
- When plotting a DatetimeIndex directly with matplotlib's ``plot`` function,  the axis labels will no longer be formatted as dates but as integers (the  internal representation of a ``datetime64``). **UPDATE** This is fixed  in 0.15.1, see `here <whatsnew_0151.datetime64_plotting>`.  .. _whatsnew_0150.deprecations:  Deprecations
^^^  - The attributes ``Categorical`` ``labels`` and ``levels`` attributes are  deprecated and renamed to ``codes`` and ``categories``.
- The ``outtype`` argument to ``pd.DataFrame.to_dict`` has been deprecated in favor of ``orient``. (`7840`)
- The ``convert_dummies`` method has been deprecated in favor of  ``get_dummies`` (`8140`)
- The ``infer_dst`` argument in ``tz_localize`` will be deprecated in favor of  ``ambiguous`` to allow for more flexibility in dealing with DST transitions.  Replace ``infer_dst=True`` with ``ambiguous='infer'`` for the same behavior (`7943`).  See `the docs<timeseries.timezone_ambiguous>` for more details.
- The top-level ``pd.value_range`` has been deprecated and can be replaced by ``.describe()`` (`8481`)  .. _whatsnew_0150.index_set_ops:  - The ``Index`` set operations ``+`` and ``-`` were deprecated in order to provide these for numeric type operations on certain index types. ``+`` can be replaced by ``.union()`` or ``|``, and ``-`` by ``.difference()``. Further the method name ``Index.diff()`` is deprecated and can be replaced by ``Index.difference()`` (`8226`)  .. code-block:: python  # +  pd.Index(['a', 'b', 'c']) + pd.Index(['b', 'c', 'd'])  # should be replaced by  pd.Index(['a', 'b', 'c']).union(pd.Index(['b', 'c', 'd']))  .. code-block:: python  # -  pd.Index(['a', 'b', 'c']) - pd.Index(['b', 'c', 'd'])  # should be replaced by  pd.Index(['a', 'b', 'c']).difference(pd.Index(['b', 'c', 'd']))  - The ``infer_types`` argument to `~pandas.read_html` now has no  effect and is deprecated (`7762`, `7032`).  .. _whatsnew_0150.prior_deprecations:  Removal of prior version deprecations/changes
^^^  - Remove ``DataFrame.delevel`` method in favor of ``DataFrame.reset_index``  .. _whatsnew_0150.enhancements:  Enhancements
~~~  Enhancements in the importing/exporting of Stata files:  - Added support for bool, uint8, uint16 and uint32 data types in ``to_stata`` (`7097`, `7365`)
- Added conversion option when importing Stata files (`8527`)
- ``DataFrame.to_stata`` and ``StataWriter`` check string length for  compatibility with limitations imposed in dta files where fixed-width  strings must contain 244 or fewer characters.  Attempting to write Stata  dta files with strings longer than 244 characters raises a ``ValueError``. (`7858`)

- ``read_stata`` and ``StataReader`` can import missing data information into a  ``DataFrame`` by setting the argument ``convert_missing`` to ``True``. When  using this options, missing values are returned as ``StataMissingValue``  objects and columns containing missing values have ``object`` data type. (`8045`)  Enhancements in the plotting functions:  - Added ``layout`` keyword to ``DataFrame.plot``. You can pass a tuple of ``(rows, columns)``, one of which can be ``-1`` to automatically infer (`6667`, `8071`).
- Allow to pass multiple axes to ``DataFrame.plot``, ``hist`` and ``boxplot`` (`5353`, `6970`, `7069`)
- Added support for ``c``, ``colormap`` and ``colorbar`` arguments for ``DataFrame.plot`` with ``kind='scatter'`` (`7780`)
- Histogram from ``DataFrame.plot`` with ``kind='hist'`` (`7809`), See `the docs<visualization.hist>`.
- Boxplot from ``DataFrame.plot`` with ``kind='box'`` (`7998`), See `the docs<visualization.box>`.  Other:  - ``read_csv`` now has a keyword parameter ``float_precision`` which specifies which floating-point converter the C engine should use during parsing, see `here <io.float_precision>` (`8002`, `8044`)  - Added ``searchsorted`` method to ``Series`` objects (`7447`)  - `describe` on mixed-types DataFrames is more flexible. Type-based column filtering is now possible via the ``include``/``exclude`` arguments.  See the `docs <basics.describe>` (`8164`).  .. ipython:: python  df = pd.DataFrame({'catA': ['foo', 'foo', 'bar'] * 8,  'catB': ['a', 'b', 'c', 'd'] * 6,  'numC': np.arange(24),  'numD': np.arange(24.) + .5})  df.describe(include=["object"])  df.describe(include=["number", "object"], exclude=["float"])  Requesting all columns is possible with the shorthand 'all'  .. ipython:: python  df.describe(include='all')  Without those arguments, ``describe`` will behave as before, including only numerical columns or, if none are, only categorical columns. See also the `docs <basics.describe>`  - Added ``split`` as an option to the ``orient`` argument in ``pd.DataFrame.to_dict``. (`7840`)  - The ``get_dummies`` method can now be used on DataFrames. By default only  categorical columns are encoded as 0's and 1's, while other columns are  left untouched.  .. ipython:: python  df = pd.DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'],  'C': [1, 2, 3]})  pd.get_dummies(df)  - ``PeriodIndex`` supports ``resolution`` as the same as ``DatetimeIndex`` (`7708`)
- ``pandas.tseries.holiday`` has added support for additional holidays and ways to observe holidays (`7070`)
- ``pandas.tseries.holiday.Holiday`` now supports a list of offsets in Python3 (`7070`)
- ``pandas.tseries.holiday.Holiday`` now supports a days_of_week parameter (`7070`)

- ``GroupBy.nth()`` now supports selecting multiple nth values (`7910`)  .. ipython:: python  business_dates = pd.date_range(start='4/1/2014', end='6/30/2014', freq='B')  df = pd.DataFrame(1, index=business_dates, columns=['a', 'b'])  # get the first, 4th, and last date index for each month  df.groupby([df.index.year, df.index.month]).nth([0, 3, -1])  - ``Period`` and ``PeriodIndex`` supports addition/subtraction with ``timedelta``-likes (`7966`)  If ``Period`` freq is ``D``, ``H``, ``T``, ``S``, ``L``, ``U``, ``N``, ``Timedelta``-like can be added if the result can have same freq. Otherwise, only the same ``offsets`` can be added.  .. ipython:: python  idx = pd.period_range('2014-07-01 09:00', periods=5, freq='H')  idx  idx + pd.offsets.Hour(2)  idx + pd.Timedelta('120m')  idx = pd.period_range('2014-07', periods=5, freq='M')  idx  idx + pd.offsets.MonthEnd(3)  - Added experimental compatibility with ``openpyxl`` for versions >= 2.0. The ``DataFrame.to_excel``  method ``engine`` keyword now recognizes ``openpyxl1`` and ``openpyxl2``  which will explicitly require openpyxl v1 and v2 respectively, failing if  the requested version is not available. The ``openpyxl`` engine is a now a  meta-engine that automatically uses whichever version of openpyxl is  installed. (`7177`)  - ``DataFrame.fillna`` can now accept a ``DataFrame`` as a fill value (`8377`)  - Passing multiple levels to `~pandas.DataFrame.stack()` will now work when multiple level  numbers are passed (`7660`). See  `Reshaping by stacking and unstacking <reshaping.stack_multiple>`.  - `set_names`, `set_labels`, and `set_levels` methods now take an optional ``level`` keyword argument to all modification of specific level(s) of a MultiIndex. Additionally `set_names` now accepts a scalar string value when operating on an ``Index`` or on a specific level of a ``MultiIndex`` (`7792`)  .. ipython:: python  idx = pd.MultiIndex.from_product([['a'], range(3), list("pqr")],  names=['foo', 'bar', 'baz'])  idx.set_names('qux', level=0)  idx.set_names(['qux', 'corge'], level=[0, 1])  idx.set_levels(['a', 'b', 'c'], level='bar')  idx.set_levels([['a', 'b', 'c'], [1, 2, 3]], level=[1, 2])  - ``Index.isin`` now supports a ``level`` argument to specify which index level  to use for membership tests (`7892`, `7890`)  .. code-block:: ipython  In [1]: idx = pd.MultiIndex.from_product([[0, 1], ['a', 'b', 'c']])  In [2]: idx.values  Out[2]: array([(0, 'a'), (0, 'b'), (0, 'c'), (1, 'a'), (1, 'b'), (1, 'c')], dtype=object)  In [3]: idx.isin(['a', 'c', 'e'], level=1)  Out[3]: array([ True, False,  True,  True, False,  True], dtype=bool)  - ``Index`` now supports ``duplicated`` and ``drop_duplicates``. (`4060`)  .. ipython:: python  idx = pd.Index([1, 2, 3, 4, 1, 2])  idx  idx.duplicated()  idx.drop_duplicates()  - add ``copy=True`` argument to ``pd.concat`` to enable pass through of complete blocks (`8252`)  - Added support for numpy 1.8+ data types (``bool_``, ``int_``, ``float_``, ``string_``) for conversion to R dataframe  (`8400`)  .. _whatsnew_0150.performance:  Performance
~~~  - Performance improvements in ``DatetimeIndex.__iter__`` to allow faster iteration (`7683`)
- Performance improvements in ``Period`` creation (and ``PeriodIndex`` setitem) (`5155`)
- Improvements in Series.transform for significant performance gains (revised) (`6496`)
- Performance improvements in ``StataReader`` when reading large files (`8040`, `8073`)
- Performance improvements in ``StataWriter`` when writing large files (`8079`)
- Performance and memory usage improvements in multi-key ``groupby`` (`8128`)
- Performance improvements in groupby ``.agg`` and ``.apply`` where builtins max/min were not mapped to numpy/cythonized versions (`7722`)
- Performance improvement in writing to sql (``to_sql``) of up to 50% (`8208`).

- Performance benchmarking of groupby for large value of ngroups (`6787`)
- Performance improvement in ``CustomBusinessDay``, ``CustomBusinessMonth`` (`8236`)
- Performance improvement for ``MultiIndex.values`` for multi-level indexes containing datetimes (`8543`)  .. _whatsnew_0150.bug_fixes:  Bug fixes
~~~  - Bug in pivot_table, when using margins and a dict aggfunc (`8349`)
- Bug in ``read_csv`` where ``squeeze=True`` would return a view (`8217`)
- Bug in checking of table name in ``read_sql`` in certain cases (`7826`).
- Bug in ``DataFrame.groupby`` where ``Grouper`` does not recognize level when frequency is specified (`7885`)
- Bug in multiindexes dtypes getting mixed up when DataFrame is saved to SQL table (`8021`)
- Bug in ``Series`` 0-division with a float and integer operand dtypes  (`7785`)
- Bug in ``Series.astype("unicode")`` not calling ``unicode`` on the values correctly (`7758`)
- Bug in ``DataFrame.as_matrix()`` with mixed ``datetime64[ns]`` and ``timedelta64[ns]`` dtypes (`7778`)
- Bug in ``HDFStore.select_column()`` not preserving UTC timezone info when selecting a ``DatetimeIndex`` (`777`)
- Bug in ``to_datetime`` when ``format='%Y%m%d'`` and ``coerce=True`` are specified, where previously an object array was returned (rather than  a coerced time-series with ``NaT``), (`7930`)
- Bug in ``DatetimeIndex`` and ``PeriodIndex`` in-place addition and subtraction cause different result from normal one (`6527`)
- Bug in adding and subtracting ``PeriodIndex`` with ``PeriodIndex`` raise ``TypeError`` (`7741`)
- Bug in ``combine_first`` with ``PeriodIndex`` data raises ``TypeError`` (`3367`)
- Bug in MultiIndex slicing with missing indexers (`7866`)
- Bug in MultiIndex slicing with various edge cases (`8132`)
- Regression in MultiIndex indexing with a non-scalar type object (`7914`)
- Bug in ``Timestamp`` comparisons with ``==`` and ``int64`` dtype (`8058`)
- Bug in pickles contains ``DateOffset`` may raise ``AttributeError`` when ``normalize`` attribute is referred internally (`7748`)
- Bug in ``Panel`` when using ``major_xs`` and ``copy=False`` is passed (deprecation warning fails because of missing ``warnings``) (`8152`).
- Bug in pickle deserialization that failed for pre-0.14.1 containers with dup items trying to avoid ambiguity  when matching block and manager items, when there's only one block there's no ambiguity (`7794`)
- Bug in putting a ``PeriodIndex`` into a ``Series`` would convert to ``int64`` dtype, rather than ``object`` of ``Periods`` (`7932`)
- Bug in ``HDFStore`` iteration when passing a where (`8014`)
- Bug in ``DataFrameGroupby.transform`` when transforming with a passed non-sorted key (`8046`, `8430`)
- Bug in repeated timeseries line and area plot may result in ``ValueError`` or incorrect kind (`7733`)
- Bug in inference in a ``MultiIndex`` with ``datetime.date`` inputs (`7888`)
- Bug in ``get`` where an ``IndexError`` would not cause the default value to be returned (`7725`)
- Bug in ``offsets.apply``, ``rollforward`` and ``rollback`` may reset nanosecond (`7697`)
- Bug in ``offsets.apply``, ``rollforward`` and ``rollback`` may raise ``AttributeError`` if ``Timestamp`` has ``dateutil`` tzinfo (`7697`)
- Bug in sorting a MultiIndex frame with a ``Float64Index`` (`8017`)
- Bug in inconsistent panel setitem with a rhs of a ``DataFrame`` for alignment (`7763`)
- Bug in ``is_superperiod`` and ``is_subperiod`` cannot handle higher frequencies than ``S`` (`7760`, `7772`, `7803`)
- Bug in 32-bit platforms with ``Series.shift`` (`8129`)
- Bug in ``PeriodIndex.unique`` returns int64 ``np.ndarray`` (`7540`)
- Bug in ``groupby.apply`` with a non-affecting mutation in the function (`8467`)
- Bug in ``DataFrame.reset_index`` which has ``MultiIndex`` contains ``PeriodIndex`` or ``DatetimeIndex`` with tz raises ``ValueError`` (`7746`, `7793`)
- Bug in ``DataFrame.plot`` with ``subplots=True`` may draw unnecessary minor xticks and yticks (`7801`)

- Bug in ``StataReader`` which did not read variable labels in 117 files due to difference between Stata documentation and implementation (`7816`)
- Bug in ``StataReader`` where strings were always converted to 244 characters-fixed width irrespective of underlying string size (`7858`)
- Bug in ``DataFrame.plot`` and ``Series.plot`` may ignore ``rot`` and ``fontsize`` keywords (`7844`)
- Bug in ``DatetimeIndex.value_counts`` doesn't preserve tz  (`7735`)
- Bug in ``PeriodIndex.value_counts`` results in ``Int64Index`` (`7735`)
- Bug in ``DataFrame.join`` when doing left join on index and there are multiple matches (`5391`)
- Bug in ``GroupBy.transform()`` where int groups with a transform that  didn't preserve the index were incorrectly truncated (`7972`).
- Bug in ``groupby`` where callable objects without name attributes would take the wrong path,  and produce a ``DataFrame`` instead of a ``Series`` (`7929`)
- Bug in ``groupby`` error message when a DataFrame grouping column is duplicated (`7511`)
- Bug in ``read_html`` where the ``infer_types`` argument forced coercion of  date-likes incorrectly (`7762`, `7032`).
- Bug in ``Series.str.cat`` with an index which was filtered as to not include the first item (`7857`)
- Bug in ``Timestamp`` cannot parse ``nanosecond`` from string (`7878`)
- Bug in ``Timestamp`` with string offset and ``tz`` results incorrect (`7833`)
- Bug in ``tslib.tz_convert`` and ``tslib.tz_convert_single`` may return different results (`7798`)
- Bug in ``DatetimeIndex.intersection`` of non-overlapping timestamps with tz raises ``IndexError`` (`7880`)
- Bug in alignment with TimeOps and non-unique indexes (`8363`)
- Bug in ``GroupBy.filter()`` where fast path vs. slow path made the filter  return a non scalar value that appeared valid but wasn't (`7870`).
- Bug in ``date_range()``/``DatetimeIndex()`` when the timezone was inferred from input dates yet incorrect  times were returned when crossing DST boundaries (`7835`, `7901`).
- Bug in ``to_excel()`` where a negative sign was being prepended to positive infinity and was absent for negative infinity (`7949`)
- Bug in area plot draws legend with incorrect ``alpha`` when ``stacked=True`` (`8027`)
- ``Period`` and ``PeriodIndex`` addition/subtraction with ``np.timedelta64`` results in incorrect internal representations (`7740`)
- Bug in ``Holiday`` with no offset or observance (`7987`)
- Bug in ``DataFrame.to_latex`` formatting when columns or index is a ``MultiIndex`` (`7982`).
- Bug in ``DateOffset`` around Daylight Savings Time produces unexpected results (`5175`).
- Bug in ``DataFrame.shift`` where empty columns would throw ``ZeroDivisionError`` on numpy 1.7 (`8019`)
- Bug in installation where ``html_encoding/*.html`` wasn't installed and  therefore some tests were not running correctly (`7927`).
- Bug in ``read_html`` where ``bytes`` objects were not tested for in  ``_read`` (`7927`).
- Bug in ``DataFrame.stack()`` when one of the column levels was a datelike (`8039`)
- Bug in broadcasting numpy scalars with ``DataFrame`` (`8116`)
- Bug in ``pivot_table`` performed with nameless ``index`` and ``columns`` raises ``KeyError`` (`8103`)
- Bug in ``DataFrame.plot(kind='scatter')`` draws points and errorbars with different colors when the color is specified by ``c`` keyword (`8081`)
- Bug in ``Float64Index`` where ``iat`` and ``at`` were not testing and were  failing (`8092`).
- Bug in ``DataFrame.boxplot()`` where y-limits were not set correctly when  producing multiple axes (`7528`, `5517`).
- Bug in ``read_csv`` where line comments were not handled correctly given  a custom line terminator or ``delim_whitespace=True`` (`8122`).
- Bug in ``read_html`` where empty tables caused a ``StopIteration`` (`7575`)
- Bug in casting when setting a column in a same-dtype block (`7704`)
- Bug in accessing groups from a ``GroupBy`` when the original grouper  was a tuple (`8121`).

- Bug in ``.at`` that would accept integer indexers on a non-integer index and do fallback (`7814`)
- Bug with kde plot and NaNs (`8182`)
- Bug in ``GroupBy.count`` with float32 data type were nan values were not excluded (`8169`).
- Bug with stacked barplots and NaNs (`8175`).
- Bug in resample with non evenly divisible offsets (e.g. '7s') (`8371`)
- Bug in interpolation methods with the ``limit`` keyword when no values needed interpolating (`7173`).
- Bug where ``col_space`` was ignored in ``DataFrame.to_string()`` when ``header=False`` (`8230`).
- Bug with ``DatetimeIndex.asof`` incorrectly matching partial strings and returning the wrong date (`8245`).
- Bug in plotting methods modifying the global matplotlib rcParams (`8242`).
- Bug in ``DataFrame.__setitem__`` that caused errors when setting a dataframe column to a sparse array (`8131`)
- Bug where ``Dataframe.boxplot()`` failed when entire column was empty (`8181`).
- Bug with messed variables in ``radviz`` visualization (`8199`).
- Bug in ``to_clipboard`` that would clip long column data (`8305`)
- Bug in ``DataFrame`` terminal display: Setting max_column/max_rows to zero did not trigger auto-resizing of dfs to fit terminal width/height (`7180`).
- Bug in OLS where running with "cluster" and "nw_lags" parameters did not work correctly, but also did not throw an error  (`5884`).
- Bug in ``DataFrame.dropna`` that interpreted non-existent columns in the subset argument as the 'last column' (`8303`)
- Bug in ``Index.intersection`` on non-monotonic non-unique indexes (`8362`).
- Bug in masked series assignment where mismatching types would break alignment (`8387`)
- Bug in ``NDFrame.equals`` gives false negatives with dtype=object (`8437`)
- Bug in assignment with indexer where type diversity would break alignment (`8258`)
- Bug in ``NDFrame.loc`` indexing when row/column names were lost when target was a list/ndarray (`6552`)
- Regression in ``NDFrame.loc`` indexing when rows/columns were converted to Float64Index if target was an empty list/ndarray (`7774`)
- Bug in ``Series`` that allows it to be indexed by a ``DataFrame`` which has unexpected results.  Such indexing is no longer permitted (`8444`)
- Bug in item assignment of a ``DataFrame`` with MultiIndex columns where right-hand-side columns were not aligned (`7655`)
- Suppress FutureWarning generated by NumPy when comparing object arrays containing NaN for equality (`7065`)
- Bug in ``DataFrame.eval()`` where the dtype of the ``not`` operator (``~``)  was not correctly inferred as ``bool``.  .. _whatsnew_0.15.0.contributors:  Contributors
~~~  .. contributors:: v0.14.1..v0.15.0

{{ header }}  This is a minor bug-fix release from 0.15.0 and includes a small number of API changes, several new features,
users upgrade to this version.  - `Enhancements <whatsnew_0151.enhancements>`
- `API Changes <whatsnew_0151.api>`
- `Bug Fixes <whatsnew_0151.bug_fixes>`  .. _whatsnew_0151.api:  API changes

~~~  - ``s.dt.hour`` and other ``.dt`` accessors will now return ``np.nan`` for missing values (rather than previously -1), (`8689`)  .. ipython:: python  s = pd.Series(pd.date_range("20130101", periods=5, freq="D"))  s.iloc[2] = np.nan  s  previous behavior:  .. code-block:: ipython  In [6]: s.dt.hour  Out[6]:  0  0  1  0  2  -1  3  0  4  0  dtype: int64  current behavior:  .. ipython:: python  s.dt.hour  - ``groupby`` with ``as_index=False`` will not add erroneous extra columns to  result (`8582`):  .. ipython:: python  np.random.seed(2718281)  df = pd.DataFrame(np.random.randint(0, 100, (10, 2)), columns=["jim", "joe"])  df.head()  ts = pd.Series(5 * np.random.randint(0, 3, 10))  previous behavior:  .. code-block:: ipython  In [4]: df.groupby(ts, as_index=False).max()  Out[4]:  NaN  jim  joe  0  0  72  83  1  5  77  84  2  10  96  65  current behavior:  .. ipython:: python  df.groupby(ts, as_index=False).max()  - ``groupby`` will not erroneously exclude columns if the column name conflicts  with the grouper name (`8112`):  .. ipython:: python  df = pd.DataFrame({"jim": range(5), "joe": range(5, 10)})  df  gr = df.groupby(df["jim"] < 2)  previous behavior (excludes 1st column from output):  .. code-block:: ipython  In [4]: gr.apply(sum)  Out[4]:  joe  jim  False  24  True  11  current behavior:  .. ipython:: python  gr.apply(sum)  - Support for slicing with monotonic decreasing indexes, even if ``start`` or ``stop`` is  not found in the index (`7860`):  .. ipython:: python  s = pd.Series(["a", "b", "c", "d"], [4, 3, 2, 1])  s  previous behavior:  .. code-block:: ipython  In [8]: s.loc[3.5:1.5]  KeyError: 3.5  current behavior:  .. ipython:: python  s.loc[3.5:1.5]  - ``io.data.Options`` has been fixed for a change in the format of the Yahoo Options page (`8612`), (`8741`)  .. note::  As a result of a change in Yahoo's option page layout, when an expiry date is given,  ``Options`` methods now return data for a single expiry date.  Previously, methods returned all  data for the selected month.  The ``month`` and ``year`` parameters have been undeprecated and can be used to get all  options data for a given month.  If an expiry date that is not valid is given, data for the next expiry after the given  date is returned.  Option data frames are now saved on the instance as ``callsYYMMDD`` or ``putsYYMMDD``.  Previously  they were saved as ``callsMMYY`` and ``putsMMYY``.  The next expiry is saved as ``calls`` and ``puts``.  New features:  - The expiry parameter can now be a single date or a list-like object containing dates.  - A new property ``expiry_dates`` was added, which returns all available expiry dates.  Current behavior:  .. code-block:: ipython  In [17]: from pandas.io.data import Options  In [18]: aapl = Options('aapl', 'yahoo')  In [19]: aapl.get_call_data().iloc[0:5, 0:1]  Out[19]:  Last  Strike Expiry  Type Symbol  80  2014-11-14 call AAPL141114C0008000  29.05  84  2014-11-14 call AAPL141114C00084000  24.80  85  2014-11-14 call AAPL141114C00085000  24.05  86  2014-11-14 call AAPL141114C00086000  22.76  87  2014-11-14 call AAPL141114C00087000  21.74  In [20]: aapl.expiry_dates  Out[20]:  [datetime.date(2014, 11, 14),  datetime.date(2014, 11, 22),  datetime.date(2014, 11, 28),  datetime.date(2014, 12, 5),  datetime.date(2014, 12, 12),  datetime.date(2014, 12, 20),  datetime.date(2015, 1, 17),  datetime.date(2015, 2, 20),  datetime.date(2015, 4, 17),  datetime.date(2015, 7, 17),  datetime.date(2016, 1, 15),  datetime.date(2017, 1, 20)]  In [21]: aapl.get_near_stock_price(expiry=aapl.expiry_dates[0:3]).iloc[0:5, 0:1]  Out[21]:  Last  Strike Expiry  Type Symbol  109  2014-11-22 call AAPL141122C00109000  1.48  2014-11-28 call AAPL141128C00109000  1.79  110  2014-11-14 call AAPL141114C0011000  0.55  2014-11-22 call AAPL141122C0011000  1.02  2014-11-28 call AAPL141128C0011000  1.32  .. _whatsnew_0151.datetime64_plotting:  - pandas now also registers the ``datetime64`` dtype in matplotlib's units registry  to plot such values as datetimes. This is activated once pandas is imported. In  previous versions, plotting an array of ``datetime64`` values will have resulted  in plotted integer values. To keep the previous behaviour, you can do  ``del matplotlib.units.registry[np.datetime64]`` (`8614`).  .. _whatsnew_0151.enhancements:  Enhancements

~~~  - ``concat`` permits a wider variety of iterables of pandas objects to be  passed as the first parameter (`8645`):  .. ipython:: python  from collections import deque  df1 = pd.DataFrame([1, 2, 3])  df2 = pd.DataFrame([4, 5, 6])  previous behavior:  .. code-block:: ipython  In [7]: pd.concat(deque((df1, df2)))  TypeError: first argument must be a list-like of pandas objects, you passed an object of type "deque"  current behavior:  .. ipython:: python  pd.concat(deque((df1, df2)))  - Represent ``MultiIndex`` labels with a dtype that utilizes memory based on the level size. In prior versions, the memory usage was a constant 8 bytes per element in each level. In addition, in prior versions, the *reported* memory usage was incorrect as it didn't show the usage for the memory occupied by the underling data array. (`8456`)  .. ipython:: python  dfi = pd.DataFrame(  1, index=pd.MultiIndex.from_product([["a"], range(1000)]), columns=["A"]  )  previous behavior:  .. code-block:: ipython  # this was underreported in prior versions  In [1]: dfi.memory_usage(index=True)  Out[1]:  Index  8000 # took about 24008 bytes in < 0.15.1  A  8000  dtype: int64  current behavior:  .. ipython:: python  dfi.memory_usage(index=True)  - Added Index properties ``is_monotonic_increasing`` and ``is_monotonic_decreasing`` (`8680`).  - Added option to select columns when importing Stata files (`7935`)  - Qualify memory usage in ``DataFrame.info()`` by adding ``+`` if it is a lower bound (`8578`)  - Raise errors in certain aggregation cases where an argument such as ``numeric_only`` is not handled (`8592`).  - Added support for 3-character ISO and non-standard country codes in `io.wb.download()` (`8482`)  - World Bank data requests now will warn/raise based  on an ``errors`` argument, as well as a list of hard-coded country codes and  the World Bank's JSON response.  In prior versions, the error messages  didn't look at the World Bank's JSON response.  Problem-inducing input were  simply dropped prior to the request. The issue was that many good countries  were cropped in the hard-coded approach.  All countries will work now, but  some bad countries will raise exceptions because some edge cases break the  entire response. (`8482`)  - Added option to ``Series.str.split()`` to return a ``DataFrame`` rather than a ``Series`` (`8428`)  - Added option to ``df.info(null_counts=None|True|False)`` to override the default display options and force showing of the null-counts (`8701`)  .. _whatsnew_0151.bug_fixes:  Bug fixes
~~~  - Bug in unpickling of a ``CustomBusinessDay`` object (`8591`)
- Bug in coercing ``Categorical`` to a records array, e.g. ``df.to_records()`` (`8626`)
- Bug in ``Categorical`` not created properly with ``Series.to_frame()`` (`8626`)
- Bug in coercing in astype of a ``Categorical`` of a passed ``pd.Categorical`` (this now raises ``TypeError`` correctly), (`8626`)
- Bug in ``cut``/``qcut`` when using ``Series`` and ``retbins=True`` (`8589`)
- Bug in writing Categorical columns to an SQL database with ``to_sql`` (`8624`).
- Bug in comparing ``Categorical`` of datetime raising when being compared to a scalar datetime (`8687`)
- Bug in selecting from a ``Categorical`` with ``.iloc`` (`8623`)
- Bug in groupby-transform with a Categorical (`8623`)
- Bug in duplicated/drop_duplicates with a Categorical (`8623`)
- Bug in ``Categorical`` reflected comparison operator raising if the first argument was a numpy array scalar (e.g. np.int64) (`8658`)
- Bug in Panel indexing with a list-like (`8710`)
- Compat issue is ``DataFrame.dtypes`` when ``options.mode.use_inf_as_null`` is True (`8722`)
- Bug in ``read_csv``, ``dialect`` parameter would not take a string (`8703`)
- Bug in slicing a MultiIndex level with an empty-list (`8737`)
- Bug in numeric index operations of add/sub with Float/Index Index with numpy arrays (`8608`)
- Bug in setitem with empty indexer and unwanted coercion of dtypes (`8669`)

- Bug in ix/loc block splitting on setitem (manifests with integer-like dtypes, e.g. datetime64) (`8607`)
- Bug when doing label based indexing with integers not found in the index for  non-unique but monotonic indexes (`8680`).
- Bug when indexing a Float64Index with ``np.nan`` on numpy 1.7 (`8980`).
- Fix ``shape`` attribute for ``MultiIndex`` (`8609`)
- Bug in ``GroupBy`` where a name conflict between the grouper and columns  would break ``groupby`` operations (`7115`, `8112`)
- Fixed a bug where plotting a column ``y`` and specifying a label would mutate the index name of the original DataFrame (`8494`)
- Fix regression in plotting of a DatetimeIndex directly with matplotlib (`8614`).
- Bug in ``date_range`` where partially-specified dates would incorporate current date (`6961`)
- Bug in Setting by indexer to a scalar value with a mixed-dtype ``Panel4d`` was failing (`8702`)
- Bug where ``DataReader``'s would fail if one of the symbols passed was invalid.  Now returns data for valid symbols and np.nan for invalid (`8494`)
- Bug in ``get_quote_yahoo`` that wouldn't allow non-float return values (`5229`).  .. _whatsnew_0.15.1.contributors:  Contributors
~~~  .. contributors:: v0.15.0..v0.15.1

{{ header }}  This is a minor release from 0.15.1 and includes a large number of bug fixes
along with several new features, enhancements, and performance improvements.
A small number of API changes were necessary to fix existing bugs.
We recommend that all users upgrade to this version.  - `Enhancements <whatsnew_0152.enhancements>`
- `API Changes <whatsnew_0152.api>`
- `Performance Improvements <whatsnew_0152.performance>`
- `Bug Fixes <whatsnew_0152.bug_fixes>`  .. _whatsnew_0152.api:  API changes
~~~  - Indexing in ``MultiIndex`` beyond lex-sort depth is now supported, though  a lexically sorted index will have a better performance. (`2646`)  .. ipython:: python  df = pd.DataFrame({'jim':[0, 0, 1, 1],  'joe':['x', 'x', 'z', 'y'],  'jolie':np.random.rand(4)}).set_index(['jim', 'joe'])  df  df.index.lexsort_depth  # in prior versions this would raise a KeyError  # will now show a PerformanceWarning  df.loc[(1, 'z')]  # lexically sorting  df2 = df.sort_index()  df2  df2.index.lexsort_depth  df2.loc[(1,'z')]  - Bug in unique of Series with ``category`` dtype, which returned all categories regardless  whether they were "used" or not (see `8559` for the discussion).  Previous behaviour was to return all categories:  .. code-block:: ipython  In [3]: cat = pd.Categorical(['a', 'b', 'a'], categories=['a', 'b', 'c'])  In [4]: cat  Out[4]:  [a, b, a]  Categories (3, object): [a < b < c]  In [5]: cat.unique()  Out[5]: array(['a', 'b', 'c'], dtype=object)  Now, only the categories that do effectively occur in the array are returned:  .. ipython:: python  cat = pd.Categorical(['a', 'b', 'a'], categories=['a', 'b', 'c'])  cat.unique()  - ``Series.all`` and ``Series.any`` now support the ``level`` and ``skipna`` parameters. ``Series.all``, ``Series.any``, ``Index.all``, and ``Index.any`` no longer support the ``out`` and ``keepdims`` parameters, which existed for compatibility with ndarray. Various index types no longer support the ``all`` and ``any`` aggregation functions and will now raise ``TypeError``. (`8302`).  - Allow equality comparisons of Series with a categorical dtype and object dtype; previously these would raise ``TypeError`` (`8938`)  - Bug in ``NDFrame``: conflicting attribute/column names now behave consistently between getting and setting. Previously, when both a column and attribute named ``y`` existed, ``data.y`` would return the attribute, while ``data.y = z`` would update the column (`8994`)  .. ipython:: python  data = pd.DataFrame({'x': [1, 2, 3]})  data.y = 2  data['y'] = [2, 4, 6]  data  # this assignment was inconsistent  data.y = 5  Old behavior:  .. code-block:: ipython  In [6]: data.y  Out[6]: 2  In [7]: data['y'].values  Out[7]: array([5, 5, 5])  New behavior:  .. ipython:: python  data.y  data['y'].values  - ``Timestamp('now')`` is now equivalent to ``Timestamp.now()`` in that it returns the local time rather than UTC. Also, ``Timestamp('today')`` is now equivalent to ``Timestamp.today()`` and both have ``tz`` as a possible argument. (`9000`)  - Fix negative step support for label-based slices (`8753`)  Old behavior:  .. code-block:: ipython  In [1]: s = pd.Series(np.arange(3), ['a', 'b', 'c'])  Out[1]:  a  0  b  1  c  2  dtype: int64  In [2]: s.loc['c':'a':-1]  Out[2]:  c  2  dtype: int64  New behavior:  .. ipython:: python  s = pd.Series(np.arange(3), ['a', 'b', 'c'])  s.loc['c':'a':-1]  .. _whatsnew_0152.enhancements:  Enhancements
~~~  ``Categorical`` enhancements:  - Added ability to export Categorical data to Stata (`8633`).  See `here <io.stata-categorical>` for limitations of categorical variables exported to Stata data files.
- Added flag ``order_categoricals`` to ``StataReader`` and ``read_stata`` to select whether to order imported categorical data (`8836`).  See `here <io.stata-categorical>` for more information on importing categorical variables from Stata data files.

- Added ability to export Categorical data to/from HDF5 (`7621`). Queries work the same as if it was an object array. However, the ``category`` dtyped data is stored in a more efficient manner. See `here <io.hdf5-categorical>` for an example and caveats w.r.t. prior versions of pandas.
- Added support for ``searchsorted()`` on ``Categorical`` class (`8420`).  Other enhancements:  - Added the ability to specify the SQL type of columns when writing a DataFrame  to a database (`8778`).  For example, specifying to use the sqlalchemy ``String`` type instead of the  default ``Text`` type for string columns:  .. code-block:: python  from sqlalchemy.types import String  data.to_sql('data_dtype', engine, dtype={'Col_1': String})  # noqa F821  - ``Series.all`` and ``Series.any`` now support the ``level`` and ``skipna`` parameters (`8302`):  .. code-block:: python  >>> s = pd.Series([False, True, False], index=[0, 0, 1])  >>> s.any(level=0)  0  True  1  False  dtype: bool  - ``Panel`` now supports the ``all`` and ``any`` aggregation functions. (`8302`):  .. code-block:: python  >>> p = pd.Panel(np.random.rand(2, 5, 4) > 0.1)  >>> p.all()  0  1  2  3  0  True  True  True  True  1  True  False  True  True  2  True  True  True  True  3  False  True  False  True  4  True  True  True  True  - Added support for ``utcfromtimestamp()``, ``fromtimestamp()``, and ``combine()`` on ``Timestamp`` class (`5351`).
- Added Google Analytics (pandas.io.ga`) basic documentation (`8835`). See `here_.
- ``Timedelta`` arithmetic returns ``NotImplemented`` in unknown cases, allowing extensions by custom classes (`8813`).
- ``Timedelta`` now supports arithmetic with ``numpy.ndarray`` objects of the appropriate dtype (numpy 1.8 or newer only) (`8884`).
- Added ``Timedelta.to_timedelta64()`` method to the public API (`8884`).
- Added ``gbq.generate_bq_schema()`` function to the gbq module (`8325`).
- ``Series`` now works with map objects the same way as generators (`8909`).
- Added context manager to ``HDFStore`` for automatic closing (`8791`).
- ``to_datetime`` gains an ``exact`` keyword to allow for a format to not require an exact match for a provided format string (if its ``False``). ``exact`` defaults to ``True`` (meaning that exact matching is still the default)  (`8904`)
- Added ``axvlines`` boolean option to parallel_coordinates plot function, determines whether vertical lines will be printed, default is True
- Added ability to read table footers to read_html (`8552`)
- ``to_sql`` now infers data types of non-NA values for columns that contain NA values and have dtype ``object`` (`8778`).  .. _whatsnew_0152.performance:  Performance
~~~  - Reduce memory usage when skiprows is an integer in read_csv (`8681`)
- Performance boost for ``to_datetime`` conversions with a passed ``format=``, and the ``exact=False`` (`8904`)  .. _whatsnew_0152.bug_fixes:  Bug fixes
~~~  - Bug in concat of Series with ``category`` dtype which were coercing to ``object``. (`8641`)
- Bug in Timestamp-Timestamp not returning a Timedelta type and datelike-datelike ops with timezones (`8865`)
- Made consistent a timezone mismatch exception (either tz operated with None or incompatible timezone), will now return ``TypeError`` rather than ``ValueError`` (a couple of edge cases only), (`8865`)
- Bug in using a ``pd.Grouper(key=...)`` with no level/axis or level only (`8795`, `8866`)
- Report a ``TypeError`` when invalid/no parameters are passed in a groupby (`8015`)
- Bug in packaging pandas with ``py2app/cx_Freeze`` (`8602`, `8831`)
- Bug in ``groupby`` signatures that didn't include \*args or \*\*kwargs (`8733`).
- ``io.data.Options`` now raises ``RemoteDataError`` when no expiry dates are available from Yahoo and when it receives no data from Yahoo (`8761`), (`8783`).
- Unclear error message in csv parsing when passing dtype and names and the parsed data is a different data type (`8833`)
- Bug in slicing a MultiIndex with an empty list and at least one boolean indexer (`8781`)

- ``io.data.Options`` now raises ``RemoteDataError`` when no expiry dates are available from Yahoo (`8761`).
- ``Timedelta`` kwargs may now be numpy ints and floats (`8757`).
- Fixed several outstanding bugs for ``Timedelta`` arithmetic and comparisons (`8813`, `5963`, `5436`).
- ``sql_schema`` now generates dialect appropriate ``CREATE TABLE`` statements (`8697`)
- ``slice`` string method now takes step into account (`8754`)
- Bug in ``BlockManager`` where setting values with different type would break block integrity (`8850`)
- Bug in ``DatetimeIndex`` when using ``time`` object as key (`8667`)
- Bug in ``merge`` where ``how='left'`` and ``sort=False`` would not preserve left frame order (`7331`)
- Bug in ``MultiIndex.reindex`` where reindexing at level would not reorder labels (`4088`)
- Bug in certain operations with dateutil timezones, manifesting with dateutil 2.3 (`8639`)
- Regression in DatetimeIndex iteration with a Fixed/Local offset timezone (`8890`)
- Bug in ``to_datetime`` when parsing a nanoseconds using the ``%f`` format (`8989`)
- Fix: The font size was only set on x axis if vertical or the y axis if horizontal. (`8765`)
- Fixed division by 0 when reading big csv files in python 3 (`8621`)
- Bug in outputting a MultiIndex with ``to_html,index=False`` which would add an extra column (`8452`)
- Imported categorical variables from Stata files retain the ordinal information in the underlying data (`8836`).
- Defined ``.size`` attribute across ``NDFrame`` objects to provide compat with numpy >= 1.9.1; buggy with ``np.array_split`` (`8846`)
- Skip testing of histogram plots for matplotlib <= 1.2 (`8648`).
- Bug where ``get_data_google`` returned object dtypes (`3995`)
- Bug in ``DataFrame.stack(..., dropna=False)`` when the DataFrame's ``columns`` is a ``MultiIndex``  whose ``labels`` do not reference all its ``levels``. (`8844`)
- Bug in that Option context applied on ``__enter__`` (`8514`)
- Bug in resample that causes a ValueError when resampling across multiple days  and the last offset is not calculated from the start of the range (`8683`)
- Bug where ``DataFrame.plot(kind='scatter')`` fails when checking if an np.array is in the DataFrame (`8852`)
- Bug in ``pd.infer_freq/DataFrame.inferred_freq`` that prevented proper sub-daily frequency inference when the index contained DST days (`8772`).
- Bug where index name was still used when plotting a series with ``use_index=False`` (`8558`).
- Bugs when trying to stack multiple columns, when some (or all) of the level names are numbers (`8584`).
- Bug in ``MultiIndex`` where ``__contains__`` returns wrong result if index is not lexically sorted or unique (`7724`)
- BUG CSV: fix problem with trailing white space in skipped rows, (`8679`), (`8661`), (`8983`)
- Regression in ``Timestamp`` does not parse 'Z' zone designator for UTC (`8771`)
- Bug in ``StataWriter`` the produces writes strings with 244 characters irrespective of actual size (`8969`)
- Fixed ValueError raised by cummin/cummax when datetime64 Series contains NaT. (`8965`)
- Bug in DataReader returns object dtype if there are missing values (`8980`)
- Bug in plotting if sharex was enabled and index was a timeseries, would show labels on multiple axes (`3964`).
- Bug where passing a unit to the TimedeltaIndex constructor applied the to nano-second conversion twice. (`9011`).
- Bug in plotting of a period-like array (`9012`)  .. _whatsnew_0.15.2.contributors:  Contributors
~~~  .. contributors:: v0.15.1..v0.15.2

{{ header }}  This is a major release from 0.15.2 and includes a small number of API changes, several new features,
users upgrade to this version.  Highlights include:  - ``DataFrame.assign`` method, see `here <whatsnew_0160.enhancements.assign>`
- ``Series.to_coo/from_coo`` methods to interact with ``scipy.sparse``, see `here <whatsnew_0160.enhancements.sparse>`
- Backwards incompatible change to ``Timedelta`` to conform the ``.seconds`` attribute with ``datetime.timedelta``, see `here <whatsnew_0160.api_breaking.timedelta>`
- Changes to the ``.loc`` slicing API to conform with the behavior of ``.ix`` see `here <whatsnew_0160.api_breaking.indexing>`
- Changes to the default for ordering in the ``Categorical`` constructor, see `here <whatsnew_0160.api_breaking.categorical>`
-  Enhancement to the ``.str`` accessor to make string operations easier, see `here <whatsnew_0160.enhancements.string>`
- The ``pandas.tools.rplot``, ``pandas.sandbox.qtpandas`` and ``pandas.rpy``  modules are deprecated. We refer users to external packages like  seaborn,  pandas-qt and  rpy2 for similar or equivalent  functionality, see `here <whatsnew_0160.deprecations>`  Check the `API Changes <whatsnew_0160.api>` and `deprecations <whatsnew_0160.deprecations>` before updating.  .. contents:: What's new in v0.16.0  none  .. _whatsnew_0160.enhancements:  New features
~~~  .. _whatsnew_0160.enhancements.assign:  DataFrame assign
^^^  Inspired by `dplyr's
<https://dplyr.tidyverse.org/articles/dplyr.html#mutating-operations>`__ ``mutate`` verb, DataFrame has a new
`~pandas.DataFrame.assign` method.
The function signature for ``assign`` is simply ``**kwargs``. The keys
to be inserted (for example, a ``Series`` or NumPy array), or a function
of one argument to be called on the ``DataFrame``. The new values are inserted,
and the entire DataFrame (with all original and new columns) is returned.  .. ipython:: python  iris = pd.read_csv('data/iris.data')  iris.head()  iris.assign(sepal_ratio=iris['SepalWidth'] / iris['SepalLength']).head()  Above was an example of inserting a precomputed value. We can also pass in
a function to be evaluated.  .. ipython:: python  iris.assign(sepal_ratio=lambda x: (x['SepalWidth']  / x['SepalLength'])).head()  The power of ``assign`` comes when used in chains of operations. For example,
we can limit the DataFrame to just those with a Sepal Length greater than 5,
calculate the ratio, and plot  .. ipython:: python  iris = pd.read_csv('data/iris.data')  (iris.query('SepalLength > 5')  .assign(SepalRatio=lambda x: x.SepalWidth / x.SepalLength,  PetalRatio=lambda x: x.PetalWidth / x.PetalLength)  .plot(kind='scatter', x='SepalRatio', y='PetalRatio'))  50 %  See the `documentation <dsintro.chained_assignment>` for more. (`9229`)  .. _whatsnew_0160.enhancements.sparse:  Interaction with scipy.sparse
^^^  Added `SparseSeries.to_coo` and `SparseSeries.from_coo` methods (`8048`) for converting to and from ``scipy.sparse.coo_matrix`` instances (see `here <sparse.scipysparse>`). For example, given a SparseSeries with MultiIndex we can convert to a ``scipy.sparse.coo_matrix`` by specifying the row and column labels as index levels:  .. code-block:: python  s = pd.Series([3.0, np.nan, 1.0, 3.0, np.nan, np.nan])  s.index = pd.MultiIndex.from_tuples([(1, 2, 'a', 0),  (1, 2, 'a', 1),  (1, 1, 'b', 0),  (1, 1, 'b', 1),  (2, 1, 'b', 0),  (2, 1, 'b', 1)],  names=['A', 'B', 'C', 'D'])  s  # SparseSeries  ss = s.to_sparse()  ss  A, rows, columns = ss.to_coo(row_levels=['A', 'B'],  column_levels=['C', 'D'],  sort_labels=False)  A  A.todense()  rows  columns  The from_coo method is a convenience method for creating a ``SparseSeries``

from a ``scipy.sparse.coo_matrix``:  .. code-block:: python  from scipy import sparse  A = sparse.coo_matrix(([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])),  shape=(3, 4))  A  A.todense()  ss = pd.SparseSeries.from_coo(A)  ss  .. _whatsnew_0160.enhancements.string:  String methods enhancements
^^^  - Following new methods are accessible via ``.str`` accessor to apply the function to each values. This is intended to make it more consistent with standard methods on strings. (`9282`, `9352`, `9386`, `9387`, `9439`)  ===  ===  ===  ===  ===  ..  ..  Methods  ..  ..  ===  ===  ===  ===  ===  ``isalnum()``  ``isalpha()``  ``isdigit()``  ``isdigit()``  ``isspace()``  ``islower()``  ``isupper()``  ``istitle()``  ``isnumeric()``  ``isdecimal()``  ``find()``  ``rfind()``  ``ljust()``  ``rjust()``  ``zfill()``  ===  ===  ===  ===  ===  .. ipython:: python  s = pd.Series(['abcd', '3456', 'EFGH'])  s.str.isalpha()  s.str.find('ab')  - `Series.str.pad` and `Series.str.center` now accept ``fillchar`` option to specify filling character (`9352`)  .. ipython:: python  s = pd.Series(['12', '300', '25'])  s.str.pad(5, fillchar='_')  - Added `Series.str.slice_replace`, which previously raised ``NotImplementedError`` (`888`)  .. ipython:: python  s = pd.Series(['ABCD', 'EFGH', 'IJK'])  s.str.slice_replace(1, 3, 'X')  # replaced with empty char  s.str.slice_replace(0, 1)  .. _whatsnew_0160.enhancements.other:  Other enhancements
^^^  - Reindex now supports ``method='nearest'`` for frames or series with a monotonic increasing or decreasing index (`9258`):  .. ipython:: python  df = pd.DataFrame({'x': range(5)})  df.reindex([0.2, 1.8, 3.5], method='nearest')  This method is also exposed by the lower level ``Index.get_indexer`` and ``Index.get_loc`` methods.  - The ``read_excel()`` function's `sheetname <io.excel.specifying_sheets>` argument now accepts a list and ``None``, to get multiple or all sheets respectively.  If more than one sheet is specified, a dictionary is returned. (`9450`)  .. code-block:: python  # Returns the 1st and 4th sheet, as a dictionary of DataFrames.  pd.read_excel('path_to_file.xls', sheetname=['Sheet1', 3])  - Allow Stata files to be read incrementally with an iterator; support for long strings in Stata files. See the docs `here<io.stata_reader>` (`9493`:).
- Paths beginning with ~ will now be expanded to begin with the user's home directory (`9066`)
- Added time interval selection in ``get_data_yahoo`` (`9071`)
- Added ``Timestamp.to_datetime64()`` to complement ``Timedelta.to_timedelta64()`` (`9255`)
- ``tseries.frequencies.to_offset()`` now accepts ``Timedelta`` as input (`9064`)
- Lag parameter was added to the autocorrelation method of ``Series``, defaults to lag-1 autocorrelation (`9192`)
- ``Timedelta`` will now accept ``nanoseconds`` keyword in constructor (`9273`)
- SQL code now safely escapes table and column names (`8986`)
- Added auto-complete for ``Series.str.<tab>``, ``Series.dt.<tab>`` and ``Series.cat.<tab>`` (`9322`)
- ``Index.get_indexer`` now supports ``method='pad'`` and ``method='backfill'`` even for any target array, not just monotonic targets. These methods also work for monotonic decreasing as well as monotonic increasing indexes (`9258`).
- ``Index.asof`` now works on all index types (`9258`).
- A ``verbose`` argument has been augmented in ``io.read_excel()``, defaults to False. Set to True to print sheet names as they are parsed. (`9450`)
- Added ``days_in_month`` (compatibility alias ``daysinmonth``) property to ``Timestamp``, ``DatetimeIndex``, ``Period``, ``PeriodIndex``, and ``Series.dt`` (`9572`)
- Added ``decimal`` option in ``to_csv`` to provide formatting for non-'.' decimal separators (`781`)
- Added ``normalize`` option for ``Timestamp`` to normalized to midnight (`8794`)
- Added example for ``DataFrame`` import to R using HDF5 file and ``rhdf5``  library. See the documentation for more  (`9636`).  .. _whatsnew_0160.api:  Backwards incompatible API changes

~~~  .. _whatsnew_0160.api_breaking:  .. _whatsnew_0160.api_breaking.timedelta:  Changes in timedelta
^^^  In v0.15.0 a new scalar type ``Timedelta`` was introduced, that is a
sub-class of ``datetime.timedelta``. Mentioned `here <whatsnew_0150.timedeltaindex>` was a notice of an API change w.r.t. the ``.seconds`` accessor. The intent was to provide a user-friendly set of accessors that give the 'natural' value for that unit, e.g. if you had a ``Timedelta('1 day, 1012')``, then ``.seconds`` would return 12. However, this is at odds with the definition of ``datetime.timedelta``, which defines ``.seconds`` as ``10 * 3600 + 11 * 60 + 12 == 36672``.  So in v0.16.0, we are restoring the API to match that of ``datetime.timedelta``. Further, the component values are still available through the ``.components`` accessor. This affects the ``.seconds`` and ``.microseconds`` accessors, and removes the ``.hours``, ``.minutes``, ``.milliseconds`` accessors. These changes affect ``TimedeltaIndex`` and the Series ``.dt`` accessor as well. (`9185`, `9139`)  Previous behavior  .. code-block:: ipython  In [2]: t = pd.Timedelta('1 day, 1012.100123')  In [3]: t.days  Out[3]: 1  In [4]: t.seconds  Out[4]: 12  In [5]: t.microseconds  Out[5]: 123  New behavior  .. ipython:: python  t = pd.Timedelta('1 day, 1012.100123')  t.days  t.seconds  t.microseconds  Using ``.components`` allows the full component access  .. ipython:: python  t.components  t.components.seconds  .. _whatsnew_0160.api_breaking.indexing:  Indexing changes
^^^  The behavior of a small sub-set of edge cases for using ``.loc`` have changed (`8613`). Furthermore we have improved the content of the error messages that are raised:  - Slicing with ``.loc`` where the start and/or stop bound is not found in the index is now allowed; this previously would raise a ``KeyError``. This makes the behavior the same as ``.ix`` in this case. This change is only for slicing, not when indexing with a single label.  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 4),  columns=list('ABCD'),  index=pd.date_range('20130101', periods=5))  df  s = pd.Series(range(5), [-2, -1, 1, 2, 3])  s  Previous behavior  .. code-block:: ipython  In [4]: df.loc['2013-01-02':'2013-01-10']  KeyError: 'stop bound [2013-01-10] is not in the [index]'  In [6]: s.loc[-10:3]  KeyError: 'start bound [-10] is not the [index]'  New behavior  .. ipython:: python  df.loc['2013-01-02':'2013-01-10']  s.loc[-10:3]  - Allow slicing with float-like values on an integer index for ``.ix``. Previously this was only enabled for ``.loc``:  Previous behavior  .. code-block:: ipython  In [8]: s.ix[-1.0:2]  TypeError: the slice start value [-1.0] is not a proper indexer for this index type (Int64Index)  New behavior  .. code-block:: python  In [2]: s.ix[-1.0:2]  Out[2]:  -1  1  1  2  2  3  dtype: int64  - Provide a useful exception for indexing with an invalid type for that index when using ``.loc``. For example trying to use ``.loc`` on an index of type ``DatetimeIndex`` or ``PeriodIndex`` or ``TimedeltaIndex``, with an integer (or a float).  Previous behavior  .. code-block:: python  In [4]: df.loc[2:3]  KeyError: 'start bound [2] is not the [index]'  New behavior  .. code-block:: ipython  In [4]: df.loc[2:3]  TypeError: Cannot do slice indexing on <class 'pandas.tseries.index.DatetimeIndex'> with <type 'int'> keys  .. _whatsnew_0160.api_breaking.categorical:  Categorical changes

^^^  In prior versions, ``Categoricals`` that had an unspecified ordering (meaning no ``ordered`` keyword was passed) were defaulted as ``ordered`` Categoricals. Going forward, the ``ordered`` keyword in the ``Categorical`` constructor will default to ``False``. Ordering must now be explicit.  Furthermore, previously you *could* change the ``ordered`` attribute of a Categorical by just setting the attribute, e.g. ``cat.ordered=True``; This is now deprecated and you should use ``cat.as_ordered()`` or ``cat.as_unordered()``. These will by default return a **new** object and not modify the existing object. (`9347`, `9190`)  Previous behavior  .. code-block:: ipython  In [3]: s = pd.Series([0, 1, 2], dtype='category')  In [4]: s  Out[4]:  0  0  1  1  2  2  dtype: category  Categories (3, int64): [0 < 1 < 2]  In [5]: s.cat.ordered  Out[5]: True  In [6]: s.cat.ordered = False  In [7]: s  Out[7]:  0  0  1  1  2  2  dtype: category  Categories (3, int64): [0, 1, 2]  New behavior  .. ipython:: python  s = pd.Series([0, 1, 2], dtype='category')  s  s.cat.ordered  s = s.cat.as_ordered()  s  s.cat.ordered  # you can set in the constructor of the Categorical  s = pd.Series(pd.Categorical([0, 1, 2], ordered=True))  s  s.cat.ordered  For ease of creation of series of categorical data, we have added the ability to pass keywords when calling ``.astype()``. These are passed directly to the constructor.  .. code-block:: python  In [54]: s = pd.Series(["a", "b", "c", "a"]).astype('category', ordered=True)  In [55]: s  Out[55]:  0  a  1  b  2  c  3  a  dtype: category  Categories (3, object): [a < b < c]  In [56]: s = (pd.Series(["a", "b", "c", "a"])  ...:  .astype('category', categories=list('abcdef'), ordered=False))  In [57]: s  Out[57]:  0  a  1  b  2  c  3  a  dtype: category  Categories (6, object): [a, b, c, d, e, f]  .. _whatsnew_0160.api_breaking.other:  Other API changes
^^^  - ``Index.duplicated`` now returns ``np.array(dtype=bool)`` rather than ``Index(dtype=object)`` containing ``bool`` values. (`8875`)
- ``DataFrame.to_json`` now returns accurate type serialisation for each column for frames of mixed dtype (`9037`)  Previously data was coerced to a common dtype before serialisation, which for  example resulted in integers being serialised to floats:  .. code-block:: ipython  In [2]: pd.DataFrame({'i': [1,2], 'f': [3.0, 4.2]}).to_json()  Out[2]: '{"f":{"0":3.0,"1":4.2},"i":{"0":1.0,"1":2.0}}'  Now each column is serialised using its correct dtype:  .. code-block:: ipython  In [2]:  pd.DataFrame({'i': [1,2], 'f': [3.0, 4.2]}).to_json()  Out[2]: '{"f":{"0":3.0,"1":4.2},"i":{"0":1,"1":2}}'  - ``DatetimeIndex``, ``PeriodIndex`` and ``TimedeltaIndex.summary`` now output the same format. (`9116`)

- ``TimedeltaIndex.freqstr`` now output the same string format as ``DatetimeIndex``. (`9116`)  - Bar and horizontal bar plots no longer add a dashed line along the info axis. The prior style can be achieved with matplotlib's ``axhline`` or ``axvline`` methods (`9088`).  - ``Series`` accessors ``.dt``, ``.cat`` and ``.str`` now raise ``AttributeError`` instead of ``TypeError`` if the series does not contain the appropriate type of data (`9617`). This follows Python's built-in exception hierarchy more closely and ensures that tests like ``hasattr(s, 'cat')`` are consistent on both Python 2 and 3.  - ``Series`` now supports bitwise operation for integral types (`9016`). Previously even if the input dtypes were integral, the output dtype was coerced to ``bool``.  Previous behavior  .. code-block:: ipython  In [2]: pd.Series([0, 1, 2, 3], list('abcd')) | pd.Series([4, 4, 4, 4], list('abcd'))  Out[2]:  a  True  b  True  c  True  d  True  dtype: bool  New behavior. If the input dtypes are integral, the output dtype is also integral and the output  values are the result of the bitwise operation.  .. code-block:: ipython  In [2]: pd.Series([0, 1, 2, 3], list('abcd')) | pd.Series([4, 4, 4, 4], list('abcd'))  Out[2]:  a  4  b  5  c  6  d  7  dtype: int64  - During division involving a ``Series`` or ``DataFrame``, ``0/0`` and ``0//0`` now give ``np.nan`` instead of ``np.inf``. (`9144`, `8445`)  Previous behavior  .. code-block:: ipython  In [2]: p = pd.Series([0, 1])  In [3]: p / 0  Out[3]:  0  inf  1  inf  dtype: float64  In [4]: p // 0  Out[4]:  0  inf  1  inf  dtype: float64  New behavior  .. ipython:: python  p = pd.Series([0, 1])  p / 0  p // 0  - ``Series.values_counts`` and ``Series.describe`` for categorical data will now put ``NaN`` entries at the end. (`9443`)
- ``Series.describe`` for categorical data will now give counts and frequencies of 0, not ``NaN``, for unused categories (`9443`)  - Due to a bug fix, looking up a partial string label with ``DatetimeIndex.asof`` now includes values that match the string, even if they are after the start of the partial string label (`9258`).  Old behavior:  .. code-block:: ipython  In [4]: pd.to_datetime(['2000-01-31', '2000-02-28']).asof('2000-02')  Out[4]: Timestamp('2000-01-31 000')  Fixed behavior:  .. ipython:: python  pd.to_datetime(['2000-01-31', '2000-02-28']).asof('2000-02')  To reproduce the old behavior, simply add more precision to the label (e.g., use ``2000-02-01`` instead of ``2000-02``).  .. _whatsnew_0160.deprecations:  Deprecations
^^^  - The ``rplot`` trellis plotting interface is deprecated and will be removed  in a future version. We refer to external packages like  seaborn for similar  but more refined functionality (`3445`).  The documentation includes some examples how to convert your existing code  from `rplot`` to seaborn `here_.  - The ``pandas.sandbox.qtpandas`` interface is deprecated and will be removed in a future version.  We refer users to the external package pandas-qt. (`9615`)  - The ``pandas.rpy`` interface is deprecated and will be removed in a future version.  Similar functionality can be accessed through the rpy2 project (`9602`)  - Adding ``DatetimeIndex/PeriodIndex`` to another ``DatetimeIndex/PeriodIndex`` is being deprecated as a set-operation. This will be changed to a ``TypeError`` in a future version. ``.union()`` should be used for the union set operation. (`9094`)
- Subtracting ``DatetimeIndex/PeriodIndex`` from another ``DatetimeIndex/PeriodIndex`` is being deprecated as a set-operation. This will be changed to an actual numeric subtraction yielding a ``TimeDeltaIndex`` in a future version. ``.difference()`` should be used for the differencing set operation. (`9094`)  .. _whatsnew_0160.prior_deprecations:  Removal of prior version deprecations/changes
^^^  - ``DataFrame.pivot_table`` and ``crosstab``'s ``rows`` and ``cols`` keyword arguments were removed in favor  of ``index`` and ``columns`` (`6581`)

- ``DataFrame.to_excel`` and ``DataFrame.to_csv`` ``cols`` keyword argument was removed in favor of ``columns`` (`6581`)
- Removed ``convert_dummies`` in favor of ``get_dummies`` (`6581`)
- Removed ``value_range`` in favor of ``describe`` (`6581`)  .. _whatsnew_0160.performance:  Performance improvements
~~~  - Fixed a performance regression for ``.loc`` indexing with an array or list-like (`9126`:).
- ``DataFrame.to_json`` 30x performance improvement for mixed dtype frames. (`9037`)
- Performance improvements in ``MultiIndex.duplicated`` by working with labels instead of values (`9125`)
- Improved the speed of ``nunique`` by calling ``unique`` instead of ``value_counts`` (`9129`, `7771`)
- Performance improvement of up to 10x in ``DataFrame.count`` and ``DataFrame.dropna`` by taking advantage of homogeneous/heterogeneous dtypes appropriately (`9136`)
- Performance improvement of up to 20x in ``DataFrame.count`` when using a ``MultiIndex`` and the ``level`` keyword argument  (`9163`)
- Performance and memory usage improvements in ``merge`` when key space exceeds ``int64`` bounds (`9151`)
- Performance improvements in multi-key ``groupby`` (`9429`)
- Performance improvements in ``MultiIndex.sortlevel`` (`9445`)
- Performance and memory usage improvements in ``DataFrame.duplicated`` (`9398`)
- Cythonized ``Period`` (`9440`)
- Decreased memory usage on ``to_hdf`` (`9648`)  .. _whatsnew_0160.bug_fixes:  Bug fixes
~~~  - Changed ``.to_html`` to remove leading/trailing spaces in table body (`4987`)
- Fixed issue using ``read_csv`` on s3 with Python 3 (`9452`)
- Fixed compatibility issue in ``DatetimeIndex`` affecting architectures where ``numpy.int_`` defaults to ``numpy.int32`` (`8943`)
- Bug in Panel indexing with an object-like (`9140`)
- Bug in the returned ``Series.dt.components`` index was reset to the default index (`9247`)
- Bug in ``Categorical.__getitem__/__setitem__`` with listlike input getting incorrect results from indexer coercion (`9469`)
- Bug in partial setting with a DatetimeIndex (`9478`)
- Bug in groupby for integer and datetime64 columns when applying an aggregator that caused the value to be  changed when the number was sufficiently large (`9311`, `6620`)
- Fixed bug in ``to_sql`` when mapping a ``Timestamp`` object column (datetime  column with timezone info) to the appropriate sqlalchemy type (`9085`).
- Fixed bug in ``to_sql`` ``dtype`` argument not accepting an instantiated  SQLAlchemy type  (`9083`).
- Bug in ``.loc`` partial setting with a ``np.datetime64`` (`9516`)
- Incorrect dtypes inferred on datetimelike looking ``Series`` & on ``.xs`` slices (`9477`)
- Items in ``Categorical.unique()`` (and ``s.unique()`` if ``s`` is of dtype ``category``) now appear in the order in which they are originally found, not in sorted order (`9331`). This is now consistent with the behavior for other dtypes in pandas.
- Fixed bug on big endian platforms which produced incorrect results in ``StataReader`` (`8688`).
- Bug in ``MultiIndex.has_duplicates`` when having many levels causes an indexer overflow (`9075`, `5873`)
- Bug in ``pivot`` and ``unstack`` where ``nan`` values would break index alignment (`4862`, `7401`, `7403`, `7405`, `7466`, `9497`)
- Bug in left ``join`` on MultiIndex with ``sort=True`` or null values (`9210`).
- Bug in ``MultiIndex`` where inserting new keys would fail (`9250`).
- Bug in ``groupby`` when key space exceeds ``int64`` bounds (`9096`).
- Bug in ``unstack`` with ``TimedeltaIndex`` or ``DatetimeIndex`` and nulls (`9491`).
- Bug in ``rank`` where comparing floats with tolerance will cause inconsistent behaviour (`8365`).
- Fixed character encoding bug in ``read_stata`` and ``StataReader`` when loading data from a URL (`9231`).
- Bug in adding ``offsets.Nano`` to other offsets raises ``TypeError`` (`9284`)

- Bug in ``DatetimeIndex`` iteration, related to (`8890`), fixed in (`9100`)
- Bugs in ``resample`` around DST transitions. This required fixing offset classes so they behave correctly on DST transitions. (`5172`, `8744`, `8653`, `9173`, `9468`).
- Bug in binary operator method (eg ``.mul()``) alignment with integer levels (`9463`).
- Bug in boxplot, scatter and hexbin plot may show an unnecessary warning (`8877`)
- Bug in subplot with ``layout`` kw may show unnecessary warning (`9464`)
- Bug in using grouper functions that need passed through arguments (e.g. axis), when using wrapped function (e.g. ``fillna``), (`9221`)
- ``DataFrame`` now properly supports simultaneous ``copy`` and ``dtype`` arguments in constructor (`9099`)
- Bug in ``read_csv`` when using skiprows on a file with CR line endings with the c engine. (`9079`)
- ``isnull`` now detects ``NaT`` in ``PeriodIndex`` (`9129`)
- Bug in groupby ``.nth()`` with a multiple column groupby (`8979`)
- Bug in ``DataFrame.where`` and ``Series.where`` coerce numerics to string incorrectly (`9280`)
- Bug in ``DataFrame.where`` and ``Series.where`` raise ``ValueError`` when string list-like is passed. (`9280`)
- Accessing ``Series.str`` methods on with non-string values now raises ``TypeError`` instead of producing incorrect results (`9184`)
- Bug in ``DatetimeIndex.__contains__`` when index has duplicates and is not monotonic increasing (`9512`)
- Fixed division by zero error for ``Series.kurt()`` when all values are equal (`9197`)
- Fixed issue in the ``xlsxwriter`` engine where it added a default 'General' format to cells if no other format was applied. This prevented other row or column formatting being applied. (`9167`)
- Fixes issue with ``index_col=False`` when ``usecols`` is also specified in ``read_csv``. (`9082`)
- Bug where ``wide_to_long`` would modify the input stub names list (`9204`)
- Bug in ``to_sql`` not storing float64 values using double precision. (`9009`)
- ``SparseSeries`` and ``SparsePanel`` now accept zero argument constructors (same as their non-sparse counterparts) (`9272`).
- Regression in merging ``Categorical`` and ``object`` dtypes (`9426`)
- Bug in ``read_csv`` with buffer overflows with certain malformed input files (`9205`)
- Bug in groupby MultiIndex with missing pair (`9049`, `9344`)
- Fixed bug in ``Series.groupby`` where grouping on ``MultiIndex`` levels would ignore the sort argument (`9444`)
- Fix bug in ``DataFrame.Groupby`` where ``sort=False`` is ignored in the case of Categorical columns. (`8868`)
- Fixed bug with reading CSV files from Amazon S3 on python 3 raising a TypeError (`9452`)
- Bug in the Google BigQuery reader where the 'jobComplete' key may be present but False in the query results (`8728`)
- Bug in ``Series.values_counts`` with excluding ``NaN`` for categorical type ``Series`` with ``dropna=True`` (`9443`)
- Fixed missing numeric_only option for ``DataFrame.std/var/sem`` (`9201`)
- Support constructing ``Panel`` or ``Panel4D`` with scalar data (`8285`)

- ``Series`` text representation disconnected from ``max_rows``/``max_columns`` (`7508`).  \  - ``Series`` number formatting inconsistent when truncated (`8532`).  Previous behavior  .. code-block:: python  In [2]: pd.options.display.max_rows = 10  In [3]: s = pd.Series([1,1,1,1,1,1,1,1,1,1,0.999,1,1]*10)  In [4]: s  Out[4]:  0  1  1  1  2  1  ...  127  0.999  128  1.000  129  1.000  Length: 130, dtype: float64  New behavior  .. code-block:: python  0  1.000  1  1.000  2  1.000  3  1.000  4  1.000  ...  125  1.000  126  1.000  127  0.999  128  1.000  129  1.000  dtype: float64  - A Spurious ``SettingWithCopy`` Warning was generated when setting a new item in a frame in some cases (`8730`)  The following would previously report a ``SettingWithCopy`` Warning.  .. ipython:: python  df1 = pd.DataFrame({'x': pd.Series(['a', 'b', 'c']),  'y': pd.Series(['d', 'e', 'f'])})  df2 = df1[['x']]  df2['y'] = ['g', 'h', 'i']  .. _whatsnew_0.16.0.contributors:  Contributors
~~~  .. contributors:: v0.15.2..v0.16.0

{{ header }}  This is a minor bug-fix release from 0.16.0 and includes a large number of
bug fixes along several new features, enhancements, and performance improvements.
We recommend that all users upgrade to this version.  Highlights include:  - Support for a ``CategoricalIndex``, a category based index, see `here <whatsnew_0161.enhancements.categoricalindex>`
- New section on how-to-contribute to *pandas*, see `here <contributing>`
- Revised "Merge, join, and concatenate" documentation, including graphical examples to make it easier to understand each operations, see `here <merging>`
- New method ``sample`` for drawing random samples from Series, DataFrames and Panels. See `here <whatsnew_0161.enhancements.sample>`
- The default ``Index`` printing has changed to a more uniform format, see `here <whatsnew_0161.index_repr>`
- ``BusinessHour`` datetime-offset is now supported, see `here <timeseries.businesshour>`  -  Further enhancement to the ``.str`` accessor to make string operations easier, see `here <whatsnew_0161.enhancements.string>`  .. contents:: What's new in v0.16.1  none  .. _whatsnew_0161.enhancements:  .. warning::  In pandas 0.17.0, the sub-package ``pandas.io.data`` will be removed in favor of a separately installable package (`8961`).  Enhancements
~~~  .. _whatsnew_0161.enhancements.categoricalindex:  CategoricalIndex
^^^  We introduce a ``CategoricalIndex``, a new type of index object that is useful for supporting
indexing with duplicates. This is a container around a ``Categorical`` (introduced in v0.15.0)
and allows efficient indexing and storage of an index with a large number of duplicated elements. Prior to 0.16.1,
setting the index of a ``DataFrame/Series`` with a ``category`` dtype would convert this to regular object-based ``Index``.  .. code-block:: ipython  In [1]: df = pd.DataFrame({'A': np.arange(6),  ...:  'B': pd.Series(list('aabbca'))  ...:  .astype('category', categories=list('cab'))  ...:  })  ...:  In [2]: df  Out[2]:  A  B  0  0  a  1  1  a  2  2  b  3  3  b  4  4  c  5  5  a  In [3]: df.dtypes  Out[3]:  A  int64  B  category  dtype: object  In [4]: df.B.cat.categories  Out[4]: Index(['c', 'a', 'b'], dtype='object')  setting the index, will create a ``CategoricalIndex``  .. code-block:: ipython  In [5]: df2 = df.set_index('B')  In [6]: df2.index  Out[6]: CategoricalIndex(['a', 'a', 'b', 'b', 'c', 'a'], categories=['c', 'a', 'b'], ordered=False, name='B', dtype='category')  indexing with ``__getitem__/.iloc/.loc/.ix`` works similarly to an Index with duplicates.
The indexers MUST be in the category or the operation will raise.  .. code-block:: ipython  In [7]: df2.loc['a']  Out[7]:  A  B  a  0  a  1  a  5  and preserves the ``CategoricalIndex``  .. code-block:: ipython  In [8]: df2.loc['a'].index  Out[8]: CategoricalIndex(['a', 'a', 'a'], categories=['c', 'a', 'b'], ordered=False, name='B', dtype='category')  sorting will order by the order of the categories  .. code-block:: ipython  In [9]: df2.sort_index()  Out[9]:  A  B  c  4  a  0  a  1  a  5  b  2  b  3  groupby operations on the index will preserve the index nature as well  .. code-block:: ipython  In [10]: df2.groupby(level=0).sum()  Out[10]:  A  B  c  4  a  6  b  5  In [11]: df2.groupby(level=0).sum().index  Out[11]: CategoricalIndex(['c', 'a', 'b'], categories=['c', 'a', 'b'], ordered=False, name='B', dtype='category')  reindexing operations, will return a resulting index based on the type of the passed
indexer, meaning that passing a list will return a plain-old-``Index``; indexing with
of the PASSED ``Categorical`` dtype. This allows one to arbitrarily index these even with

values NOT in the categories, similarly to how you can reindex ANY pandas index.  .. code-block:: ipython  In [12]: df2.reindex(['a', 'e'])  Out[12]:  A  B  a  0.0  a  1.0  a  5.0  e  NaN  In [13]: df2.reindex(['a', 'e']).index  Out[13]: pd.Index(['a', 'a', 'a', 'e'], dtype='object', name='B')  In [14]: df2.reindex(pd.Categorical(['a', 'e'], categories=list('abcde')))  Out[14]:  A  B  a  0.0  a  1.0  a  5.0  e  NaN  In [15]: df2.reindex(pd.Categorical(['a', 'e'], categories=list('abcde'))).index  Out[15]: pd.CategoricalIndex(['a', 'a', 'a', 'e'],  categories=['a', 'b', 'c', 'd', 'e'],  ordered=False, name='B',  dtype='category')  See the `documentation <advanced.categoricalindex>` for more. (`7629`, `10038`, `10039`)  .. _whatsnew_0161.enhancements.sample:  Sample
^^^  Series, DataFrames, and Panels now have a new method: `~pandas.DataFrame.sample`.
The method accepts a specific number of rows or columns to return, or a fraction of the
total number or rows or columns. It also has options for sampling with or without replacement,
for passing in a column for weights for non-uniform sampling, and for setting seed values to
facilitate replication. (`2419`)  .. ipython:: python  example_series = pd.Series([0, 1, 2, 3, 4, 5])  # When no arguments are passed, returns 1  example_series.sample()  # One may specify either a number of rows:  example_series.sample(n=3)  # Or a fraction of the rows:  example_series.sample(frac=0.5)  # weights are accepted.  example_weights = [0, 0, 0.2, 0.2, 0.2, 0.4]  example_series.sample(n=3, weights=example_weights)  # weights will also be normalized if they do not sum to one,  # and missing values will be treated as zeros.  example_weights2 = [0.5, 0, 0, 0, None, np.nan]  example_series.sample(n=1, weights=example_weights2)  When applied to a DataFrame, one may pass the name of a column to specify sampling weights
when sampling from rows.  .. ipython:: python  df = pd.DataFrame({"col1": [9, 8, 7, 6], "weight_column": [0.5, 0.4, 0.1, 0]})  df.sample(n=3, weights="weight_column")  .. _whatsnew_0161.enhancements.string:  String methods enhancements
^^^  `Continuing from v0.16.0 <whatsnew_0160.enhancements.string>`, the following

enhancements make string operations easier and more consistent with standard python string operations.  - Added ``StringMethods`` (``.str`` accessor) to ``Index`` (`9068`)  The ``.str`` accessor is now available for both ``Series`` and ``Index``.  .. ipython:: python  idx = pd.Index([" jack", "jill ", " jesse ", "frank"])  idx.str.strip()  One special case for the ``.str`` accessor on ``Index`` is that if a string method returns ``bool``, the ``.str`` accessor  will return a ``np.array`` instead of a boolean ``Index`` (`8875`). This enables the following expression  to work naturally:  .. ipython:: python  idx = pd.Index(["a1", "a2", "b1", "b2"])  s = pd.Series(range(4), index=idx)  s  idx.str.startswith("a")  s[s.index.str.startswith("a")]  - The following new methods are accessible via ``.str`` accessor to apply the function to each values. (`9766`, `9773`, `10031`, `10045`, `10052`)  ===  ===  ===  ===  ===  ..  ..  Methods  ..  ..  ===  ===  ===  ===  ===  ``capitalize()``  ``swapcase()``  ``normalize()``  ``partition()``  ``rpartition()``  ``index()``  ``rindex()``  ``translate()``  ===  ===  ===  ===  ===  - ``split`` now takes ``expand`` keyword to specify whether to expand dimensionality. ``return_type`` is deprecated. (`9847`)  .. ipython:: python  s = pd.Series(["a,b", "a,c", "b,c"])  # return Series  s.str.split(",")  # return DataFrame  s.str.split(",", expand=True)  idx = pd.Index(["a,b", "a,c", "b,c"])  # return Index  idx.str.split(",")  # return MultiIndex  idx.str.split(",", expand=True)  - Improved ``extract`` and ``get_dummies`` methods for ``Index.str`` (`9980`)  .. _whatsnew_0161.enhancements.other:  Other enhancements
^^^  - ``BusinessHour`` offset is now supported, which represents business hours starting from 09:00 - 17:00 on ``BusinessDay`` by default. See `Here <timeseries.businesshour>` for details. (`7905`)  .. ipython:: python  pd.Timestamp("2014-08-01 09:00") + pd.tseries.offsets.BusinessHour()  pd.Timestamp("2014-08-01 07:00") + pd.tseries.offsets.BusinessHour()  pd.Timestamp("2014-08-01 16:30") + pd.tseries.offsets.BusinessHour()  - ``DataFrame.diff`` now takes an ``axis`` parameter that determines the direction of differencing (`9727`)  - Allow ``clip``, ``clip_lower``, and ``clip_upper`` to accept array-like arguments as thresholds (This is a regression from 0.11.0). These methods now have an ``axis`` parameter which determines how the Series or DataFrame will be aligned with the threshold(s). (`6966`)  - ``DataFrame.mask()`` and ``Series.mask()`` now support same keywords as ``where`` (`8801`)  - ``drop`` function can now accept ``errors`` keyword to suppress ``ValueError`` raised when any of label does not exist in the target data. (`6736`)  .. ipython:: python  df = pd.DataFrame(np.random.randn(3, 3), columns=["A", "B", "C"])  df.drop(["A", "X"], axis=1, errors="ignore")  - Add support for separating years and quarters using dashes, for  example 2014-Q1.  (`9688`)  - Allow conversion of values with dtype ``datetime64`` or ``timedelta64`` to strings using ``astype(str)`` (`9757`)
- ``get_dummies`` function now accepts ``sparse`` keyword.  If set to ``True``, the return ``DataFrame`` is sparse, e.g. ``SparseDataFrame``. (`8823`)
- ``Period`` now accepts ``datetime64`` as value input. (`9054`)  - Allow timedelta string conversion when leading zero is missing from time definition, ie ``000`` vs ``000``. (`9570`)
- Allow ``Panel.shift`` with ``axis='items'`` (`9890`)  - Trying to write an excel file now raises ``NotImplementedError`` if the ``DataFrame`` has a ``MultiIndex`` instead of writing a broken Excel file. (`9794`)
- Allow ``Categorical.add_categories`` to accept ``Series`` or ``np.array``. (`9927`)  - Add/delete ``str/dt/cat`` accessors dynamically from ``__dir__``. (`9910`)

- Add ``normalize`` as a ``dt`` accessor method. (`10047`)  - ``DataFrame`` and ``Series`` now have ``_constructor_expanddim`` property as overridable constructor for one higher dimensionality data. This should be used only when it is really needed, see `here <extending.subclassing-pandas>`  - ``pd.lib.infer_dtype`` now returns ``'bytes'`` in Python 3 where appropriate. (`10032`)  .. _whatsnew_0161.api:  API changes
~~~  - When passing in an ax to ``df.plot( ..., ax=ax)``, the ``sharex`` kwarg will now default to ``False``.  The result is that the visibility of xlabels and xticklabels will not anymore be changed. You  have to do that by yourself for the right axes in your figure or set ``sharex=True`` explicitly  (but this changes the visible for all axes in the figure, not only the one which is passed in!).  If pandas creates the subplots itself (e.g. no passed in ``ax`` kwarg), then the  default is still ``sharex=True`` and the visibility changes are applied.  - `~pandas.DataFrame.assign` now inserts new columns in alphabetical order. Previously  the order was arbitrary. (`9777`)  - By default, ``read_csv`` and ``read_table`` will now try to infer the compression type based on the file extension. Set ``compression=None`` to restore the previous behavior (no decompression). (`9770`)  .. _whatsnew_0161.deprecations:  Deprecations
^^^  - ``Series.str.split``'s ``return_type`` keyword was removed in favor of ``expand`` (`9847`)  .. _whatsnew_0161.index_repr:  Index representation
~~~  The string representation of ``Index`` and its sub-classes have now been unified. These will show a single-line display if there are few values; a wrapped multi-line display for a lot of values (but less than ``display.max_seq_items``; if lots of items (> ``display.max_seq_items``) will show a truncated display (the head and tail of the data). The formatting for ``MultiIndex`` is unchanged (a multi-line wrapped display). The display width responds to the option ``display.max_seq_items``, which is defaulted to 100. (`6482`)  Previous behavior  .. code-block:: ipython  In [2]: pd.Index(range(4), name='foo')  Out[2]: Int64Index([0, 1, 2, 3], dtype='int64')  In [3]: pd.Index(range(104), name='foo')  Out[3]: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, ...], dtype='int64')  In [4]: pd.date_range('20130101', periods=4, name='foo', tz='US/Eastern')  Out[4]:  <class 'pandas.tseries.index.DatetimeIndex'>  [2013-01-01 000-05:00, ..., 2013-01-04 000-05:00]  Length: 4, Freq: D, Timezone: US/Eastern  In [5]: pd.date_range('20130101', periods=104, name='foo', tz='US/Eastern')  Out[5]:  <class 'pandas.tseries.index.DatetimeIndex'>  [2013-01-01 000-05:00, ..., 2013-04-14 000-04:00]  Length: 104, Freq: D, Timezone: US/Eastern  New behavior  .. ipython:: python  pd.set_option("display.width", 80)  pd.Index(range(4), name="foo")  pd.Index(range(30), name="foo")  pd.Index(range(104), name="foo")  pd.CategoricalIndex(["a", "bb", "ccc", "ddd"], ordered=True, name="foobar")  pd.CategoricalIndex(["a", "bb", "ccc", "ddd"] * 10, ordered=True, name="foobar")  pd.CategoricalIndex(["a", "bb", "ccc", "ddd"] * 100, ordered=True, name="foobar")  pd.date_range("20130101", periods=4, name="foo", tz="US/Eastern")  pd.date_range("20130101", periods=25, freq="D")  pd.date_range("20130101", periods=104, name="foo", tz="US/Eastern")  .. _whatsnew_0161.performance:  Performance improvements

~~~  - Improved csv write performance with mixed dtypes, including datetimes by up to 5x (`9940`)
- Improved csv write performance generally by 2x (`9940`)
- Improved the performance of ``pd.lib.max_len_string_array`` by 5-7x (`10024`)  .. _whatsnew_0161.bug_fixes:  Bug fixes
~~~  - Bug where labels did not appear properly in the legend of ``DataFrame.plot()``, passing ``label=`` arguments works, and Series indices are no longer mutated. (`9542`)
- Bug in json serialization causing a segfault when a frame had zero length. (`9805`)
- Bug in ``read_csv`` where missing trailing delimiters would cause segfault. (`5664`)
- Bug in retaining index name on appending (`9862`)
- Bug in ``scatter_matrix`` draws unexpected axis ticklabels (`5662`)
- Fixed bug in ``StataWriter`` resulting in changes to input ``DataFrame`` upon save (`9795`).
- Bug in ``transform`` causing length mismatch when null entries were present and a fast aggregator was being used (`9697`)
- Bug in ``equals`` causing false negatives when block order differed (`9330`)
- Bug in grouping with multiple ``pd.Grouper`` where one is non-time based (`10063`)
- Bug in ``read_sql_table`` error when reading postgres table with timezone (`7139`)
- Bug in ``DataFrame`` slicing may not retain metadata (`9776`)
- Bug where ``TimdeltaIndex`` were not properly serialized in fixed ``HDFStore`` (`9635`)
- Bug with ``TimedeltaIndex`` constructor ignoring ``name`` when given another ``TimedeltaIndex`` as data (`10025`).
- Bug in ``DataFrameFormatter._get_formatted_index`` with not applying ``max_colwidth`` to the ``DataFrame`` index (`7856`)
- Bug in ``.loc`` with a read-only ndarray data source (`10043`)
- Bug in ``groupby.apply()`` that would raise if a passed user defined function either returned only ``None`` (for all input). (`9685`)
- Always use temporary files in pytables tests (`9992`)
- Bug in plotting continuously using ``secondary_y`` may not show legend properly. (`9610`, `9779`)
- Bug in ``DataFrame.plot(kind="hist")`` results in ``TypeError`` when ``DataFrame`` contains non-numeric columns  (`9853`)
- Bug where repeated plotting of ``DataFrame`` with a ``DatetimeIndex`` may raise ``TypeError`` (`9852`)
- Bug in ``setup.py`` that would allow an incompat cython version to build (`9827`)
- Bug in plotting ``secondary_y`` incorrectly attaches ``right_ax`` property to secondary axes specifying itself recursively. (`9861`)
- Bug in ``Series.quantile`` on empty Series of type ``Datetime`` or ``Timedelta`` (`9675`)
- Bug in ``where`` causing incorrect results when upcasting was required (`9731`)
- Bug in ``FloatArrayFormatter`` where decision boundary for displaying "small" floats in decimal format is off by one order of magnitude for a given display.precision (`9764`)
- Fixed bug where ``DataFrame.plot()`` raised an error when both ``color`` and ``style`` keywords were passed and there was no color symbol in the style strings (`9671`)
- Not showing a ``DeprecationWarning`` on combining list-likes with an ``Index`` (`10083`)
- Bug in ``read_csv`` and ``read_table`` when using ``skip_rows`` parameter if blank lines are present. (`9832`)
- Bug in ``read_csv()`` interprets ``index_col=True`` as ``1`` (`9798`)
- Bug in index equality comparisons using ``==`` failing on Index/MultiIndex type incompatibility (`9785`)
- Bug in which ``SparseDataFrame`` could not take ``nan`` as a column name (`8822`)
- Bug in ``to_msgpack`` and ``read_msgpack`` zlib and blosc compression support (`9783`)
- Bug ``GroupBy.size`` doesn't attach index name properly if grouped by ``TimeGrouper`` (`9925`)
- Bug causing an exception in slice assignments because ``length_of_indexer`` returns wrong results (`9995`)
- Bug in csv parser causing lines with initial white space plus one non-space character to be skipped. (`9710`)
- Bug in C csv parser causing spurious NaNs when data started with newline followed by white space. (`10022`)
- Bug causing elements with a null group to spill into the final group when grouping by a ``Categorical`` (`9603`)

- Bug where .iloc and .loc behavior is not consistent on empty dataframes (`9964`)
- Bug in invalid attribute access on a ``TimedeltaIndex`` incorrectly raised ``ValueError`` instead of ``AttributeError`` (`9680`)
- Bug in unequal comparisons between categorical data and a scalar, which was not in the categories (e.g. ``Series(Categorical(list("abc"), ordered=True)) > "d"``. This returned ``False`` for all elements, but now raises a ``TypeError``. Equality comparisons also now return ``False`` for ``==`` and ``True`` for ``!=``. (`9848`)
- Bug in DataFrame ``__setitem__`` when right hand side is a dictionary (`9874`)
- Bug in ``where`` when dtype is ``datetime64/timedelta64``, but dtype of other is not (`9804`)
- Bug in ``MultiIndex.sortlevel()`` results in unicode level name breaks (`9856`)
- Bug in which ``groupby.transform`` incorrectly enforced output dtypes to match input dtypes. (`9807`)
- Bug in ``DataFrame`` constructor when ``columns`` parameter is set, and ``data`` is an empty list (`9939`)
- Bug in bar plot with ``log=True`` raises ``TypeError`` if all values are less than 1 (`9905`)
- Bug in horizontal bar plot ignores ``log=True`` (`9905`)
- Bug in PyTables queries that did not return proper results using the index (`8265`, `9676`)
- Bug where dividing a dataframe containing values of type ``Decimal`` by another ``Decimal`` would raise. (`9787`)
- Bug where using DataFrames asfreq would remove the name of the index. (`9885`)
- Bug causing extra index point when resample BM/BQ (`9756`)
- Changed caching in ``AbstractHolidayCalendar`` to be at the instance level rather than at the class level as the latter can result in unexpected behaviour. (`9552`)
- Fixed latex output for MultiIndexed dataframes (`9778`)
- Bug causing an exception when setting an empty range using ``DataFrame.loc`` (`9596`)
- Bug in hiding ticklabels with subplots and shared axes when adding a new plot to an existing grid of axes (`9158`)
- Bug in ``transform`` and ``filter`` when grouping on a categorical variable (`9921`)
- Bug in ``transform`` when groups are equal in number and dtype to the input index (`9700`)
- Google BigQuery connector now imports dependencies on a per-method basis.(`9713`)
- Updated BigQuery connector to no longer use deprecated ``oauth2client.tools.run()`` (`8327`)
- Bug in subclassed ``DataFrame``. It may not return the correct class, when slicing or subsetting it. (`9632`)
- Bug in ``.median()`` where non-float null values are not handled correctly (`10040`)
- Bug in Series.fillna() where it raises if a numerically convertible string is given (`10092`)  .. _whatsnew_0.16.1.contributors:  Contributors
~~~  .. contributors:: v0.16.0..v0.16.1

Version 0.16.2 (June 12, 2015)

{{ header }}  This is a minor bug-fix release from 0.16.1 and includes a large number of
bug fixes along some new features (`~DataFrame.pipe` method), enhancements, and performance improvements.  We recommend that all users upgrade to this version.  Highlights include:  - A new ``pipe`` method, see `here <whatsnew_0162.enhancements.pipe>`
- Documentation on how to use numba_ with *pandas*, see `here <enhancingperf.numba>`  .. contents:: What's new in v0.16.2  none  .. _numba: http://numba.pydata.org  .. _whatsnew_0162.enhancements:  New features
~~~  .. _whatsnew_0162.enhancements.pipe:  Pipe
^^^  We've introduced a new method `DataFrame.pipe`. As suggested by the name, ``pipe``
should be used to pipe data through a chain of function calls.
The goal is to avoid confusing nested function calls like  .. code-block:: python  # df is a DataFrame  # f, g, and h are functions that take and return DataFrames  f(g(h(df), arg1=1), arg2=2, arg3=3)  # noqa F821  The logic flows from inside out, and function names are separated from their keyword arguments.
This can be rewritten as  .. code-block:: python  (  df.pipe(h)  # noqa F821  .pipe(g, arg1=1)  # noqa F821  .pipe(f, arg2=2, arg3=3)  # noqa F821  )  Now both the code and the logic flow from top to bottom. Keyword arguments are next to
their functions. Overall the code is much more readable.  In the example above, the functions ``f``, ``g``, and ``h`` each expected the DataFrame as the first positional argument.
When the function you wish to apply takes its data anywhere other than the first argument, pass a tuple
of ``(function, keyword)`` indicating where the DataFrame should flow. For example:  .. code-block:: ipython  In [1]: import statsmodels.formula.api as sm  In [2]: bb = pd.read_csv("data/baseball.csv", index_col="id")  # sm.ols takes (formula, data)  In [3]: (  ...:  bb.query("h > 0")  ...:  .assign(ln_h=lambda df: np.log(df.h))  ...:  .pipe((sm.ols, "data"), "hr ~ ln_h + year + g + C(lg)")  ...:  .fit()  ...:  .summary()  ...: )  ...:  Out[3]:  <class 'statsmodels.iolib.summary.Summary'>  """

  Dep. Variable:  hr  R-squared:  0.685  Model:  OLS  Adj. R-squared:  0.665  Method:  Least Squares  F-statistic:  34.28  Date:  Tue, 22 Nov 2022  Prob (F-statistic):  3.48e-15  Time:  0523  Log-Likelihood:  -205.92  No. Observations:  68  AIC:  421.8  Df Residuals:  63  BIC:  432.9  Df Model:  4

  Notes:  [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.  [2] The condition number is large, 1.49e+07. This might indicate that there are  strong multicollinearity or other numerical problems.  """  The pipe method is inspired by unix pipes, which stream text through
processes. More recently dplyr_ and magrittr_ have introduced the
popular ``(%>%)`` pipe operator for R_.  See the `documentation <basics.pipe>` for more. (`10129`)  .. _dplyr: https://github.com/tidyverse/dplyr
.. _magrittr: https://github.com/smbache/magrittr
.. _R: http://www.r-project.org  .. _whatsnew_0162.enhancements.other:  Other enhancements
^^^  - Added ``rsplit`` to Index/Series StringMethods (`10303`)  - Removed the hard-coded size limits on the ``DataFrame`` HTML representation  in the IPython notebook, and leave this to IPython itself (only for IPython  v3.0 or greater). This eliminates the duplicate scroll bars that appeared in  the notebook with large frames (`10231`).  Note that the notebook has a ``toggle output scrolling`` feature to limit the  display of very large frames (by clicking left of the output).  You can also configure the way DataFrames are displayed using the pandas  options, see here `here <options.frequently_used>`.  - ``axis`` parameter of ``DataFrame.quantile`` now accepts also ``index`` and ``column``. (`9543`)  .. _whatsnew_0162.api:  API changes
~~~  - ``Holiday`` now raises ``NotImplementedError`` if both ``offset`` and ``observance`` are used in the constructor instead of returning an incorrect result (`10217`).  .. _whatsnew_0162.performance:  Performance improvements
~~~  - Improved ``Series.resample`` performance with ``dtype=datetime64[ns]`` (`7754`)
- Increase performance of ``str.split`` when ``expand=True`` (`10081`)  .. _whatsnew_0162.bug_fixes:  Bug fixes
~~~  - Bug in ``Series.hist`` raises an error when a one row ``Series`` was given (`10214`)
- Bug where ``HDFStore.select`` modifies the passed columns list (`7212`)
- Bug in ``Categorical`` repr with ``display.width`` of ``None`` in Python 3 (`10087`)
- Bug in ``to_json`` with certain orients and a ``CategoricalIndex`` would segfault (`10317`)
- Bug where some of the nan functions do not have consistent return dtypes (`10251`)
- Bug in ``DataFrame.quantile`` on checking that a valid axis was passed (`9543`)
- Bug in ``groupby.apply`` aggregation for ``Categorical`` not preserving categories (`10138`)
- Bug in ``to_csv`` where ``date_format`` is ignored if the ``datetime`` is fractional (`10209`)
- Bug in ``DataFrame.to_json`` with mixed data types (`10289`)
- Bug in cache updating when consolidating (`10264`)
- Bug in ``mean()`` where integer dtypes can overflow (`10172`)
- Bug where ``Panel.from_dict`` does not set dtype when specified (`10058`)
- Bug in ``Index.union`` raises ``AttributeError`` when passing array-likes. (`10149`)
- Bug in ``Timestamp``'s' ``microsecond``, ``quarter``, ``dayofyear``, ``week`` and ``daysinmonth`` properties return ``np.int`` type, not built-in ``int``. (`10050`)
- Bug in ``NaT`` raises ``AttributeError`` when accessing to ``daysinmonth``, ``dayofweek`` properties. (`10096`)
- Bug in Index repr when using the ``max_seq_items=None`` setting (`10182`).
- Bug in getting timezone data with ``dateutil`` on various platforms ( `9059`, `8639`, `9663`, `10121`)
- Bug in displaying datetimes with mixed frequencies; display 'ms' datetimes to the proper precision. (`10170`)
- Bug in ``setitem`` where type promotion is applied to the entire block (`10280`)
- Bug in ``Series`` arithmetic methods may incorrectly hold names (`10068`)
- Bug in ``GroupBy.get_group`` when grouping on multiple keys, one of which is categorical. (`10132`)
- Bug in ``DatetimeIndex`` and ``TimedeltaIndex`` names are lost after timedelta arithmetic ( `9926`)
- Bug in ``DataFrame`` construction from nested ``dict`` with ``datetime64`` (`10160`)

- Bug in ``Series`` construction from ``dict`` with ``datetime64`` keys (`9456`)
- Bug in ``Series.plot(label="LABEL")`` not correctly setting the label (`10119`)
- Bug in ``plot`` not defaulting to matplotlib ``axes.grid`` setting (`9792`)
- Bug causing strings containing an exponent, but no decimal to be parsed as ``int`` instead of ``float`` in ``engine='python'`` for the ``read_csv`` parser (`9565`)
- Bug in ``Series.align`` resets ``name`` when ``fill_value`` is specified (`10067`)
- Bug in ``read_csv`` causing index name not to be set on an empty DataFrame (`10184`)
- Bug in ``SparseSeries.abs`` resets ``name`` (`10241`)
- Bug in ``TimedeltaIndex`` slicing may reset freq (`10292`)
- Bug in ``GroupBy.get_group`` raises ``ValueError`` when group key contains ``NaT`` (`6992`)
- Bug in ``SparseSeries`` constructor ignores input data name (`10258`)
- Bug in ``Categorical.remove_categories`` causing a ``ValueError`` when removing the ``NaN`` category if underlying dtype is floating-point (`10156`)
- Bug where infer_freq infers time rule (WOM-5XXX) unsupported by to_offset (`9425`)
- Bug in ``DataFrame.to_hdf()`` where table format would raise a seemingly unrelated error for invalid (non-string) column names. This is now explicitly forbidden. (`9057`)
- Bug to handle masking empty ``DataFrame`` (`10126`).
- Bug where MySQL interface could not handle numeric table/column names (`10255`)
- Bug in ``read_csv`` with a ``date_parser`` that returned a ``datetime64`` array of other time resolution than ``[ns]`` (`10245`)
- Bug in ``Panel.apply`` when the result has ndim=0 (`10332`)
- Bug in ``read_hdf`` where ``auto_close`` could not be passed (`9327`).
- Bug in ``read_hdf`` where open stores could not be used (`10330`).
- Bug in adding empty ``DataFrames``, now results in a ``DataFrame`` that ``.equals`` an empty ``DataFrame`` (`10181`).
- Bug in ``to_hdf`` and ``HDFStore`` which did not check that complib choices were valid (`4582`, `8874`).  .. _whatsnew_0.16.2.contributors:  Contributors
~~~  .. contributors:: v0.16.1..v0.16.2

{{ header }}  This is a major release from 0.16.2 and includes a small number of API changes, several new features,
users upgrade to this version.  .. warning::  pandas >= 0.17.0 will no longer support compatibility with Python version 3.2 (`9118`)  .. warning::  The ``pandas.io.data`` package is deprecated and will be replaced by the  pandas-datareader package.  This will allow the data modules to be independently updated to your pandas  installation. The API for ``pandas-datareader v0.1.1`` is exactly the same  as in ``pandas v0.17.0`` (`8961`, `10861`).  After installing pandas-datareader, you can easily change your imports:  .. code-block:: python  from pandas.io import data, wb  becomes  .. code-block:: python  from pandas_datareader import data, wb  Highlights include:  - Release the Global Interpreter Lock (GIL) on some cython operations, see `here <whatsnew_0170.gil>`
- Plotting methods are now available as attributes of the ``.plot`` accessor, see `here <whatsnew_0170.plot>`
- The sorting API has been revamped to remove some long-time inconsistencies, see `here <whatsnew_0170.api_breaking.sorting>`
- Support for a ``datetime64[ns]`` with timezones as a first-class dtype, see `here <whatsnew_0170.tz>`
- The default for ``to_datetime`` will now be to ``raise`` when presented with unparsable formats,  previously this would return the original input. Also, date parse  functions now return consistent results. See `here <whatsnew_0170.api_breaking.to_datetime>`
- The default for ``dropna`` in ``HDFStore`` has changed to ``False``, to store by default all rows even  if they are all ``NaN``, see `here <whatsnew_0170.api_breaking.hdf_dropna>`
- Datetime accessor (``dt``) now supports ``Series.dt.strftime`` to generate formatted strings for datetime-likes, and ``Series.dt.total_seconds`` to generate each duration of the timedelta in seconds. See `here <whatsnew_0170.strftime>`
- ``Period`` and ``PeriodIndex`` can handle multiplied freq like ``3D``, which corresponding to 3 days span. See `here <whatsnew_0170.periodfreq>`
- Development installed versions of pandas will now have ``PEP440`` compliant version strings (`9518`)
- Development support for benchmarking with the Air Speed Velocity library (`8361`)
- Support for reading SAS xport files, see `here <whatsnew_0170.enhancements.sas_xport>`
- Documentation comparing SAS to *pandas*, see `here <compare_with_sas>`
- Removal of the automatic TimeSeries broadcasting, deprecated since 0.8.0, see `here <whatsnew_0170.prior_deprecations>`
- Display format with plain text can optionally align with Unicode East Asian Width, see `here <whatsnew_0170.east_asian_width>`
- Compatibility with Python 3.5 (`11097`)
- Compatibility with matplotlib 1.5.0 (`111`)  Check the `API Changes <whatsnew_0170.api>` and `deprecations <whatsnew_0170.deprecations>` before updating.  .. contents:: What's new in v0.17.0  none  .. _whatsnew_0170.enhancements:  New features
~~~  .. _whatsnew_0170.tz:  Datetime with TZ
^^^  We are adding an implementation that natively supports datetime with timezones. A ``Series`` or a ``DataFrame`` column previously
*could* be assigned a datetime with timezones, and would work as an ``object`` dtype. This had performance issues with a large

number rows. See the `docs <timeseries.timezone_series>` for more details. (`8260`, `10763`, `11034`).  The new implementation allows for having a single-timezone across all rows, with operations in a performant manner.  .. ipython:: python  df = pd.DataFrame(  {  "A": pd.date_range("20130101", periods=3),  "B": pd.date_range("20130101", periods=3, tz="US/Eastern"),  "C": pd.date_range("20130101", periods=3, tz="CET"),  }  )  df  df.dtypes  .. ipython:: python  df.B  df.B.dt.tz_localize(None)  This uses a new-dtype representation as well, that is very similar in look-and-feel to its numpy cousin ``datetime64[ns]``  .. ipython:: python  df["B"].dtype  type(df["B"].dtype)  .. note::  There is a slightly different string repr for the underlying ``DatetimeIndex`` as a result of the dtype changes, but  functionally these are the same.  Previous behavior:  .. code-block:: ipython  In [1]: pd.date_range('20130101', periods=3, tz='US/Eastern')  Out[1]: DatetimeIndex(['2013-01-01 000-05:00', '2013-01-02 000-05:00',  '2013-01-03 000-05:00'],  dtype='datetime64[ns]', freq='D', tz='US/Eastern')  In [2]: pd.date_range('20130101', periods=3, tz='US/Eastern').dtype  Out[2]: dtype('<M8[ns]')  New behavior:  .. ipython:: python  pd.date_range("20130101", periods=3, tz="US/Eastern")  pd.date_range("20130101", periods=3, tz="US/Eastern").dtype  .. _whatsnew_0170.gil:  Releasing the GIL
^^^  We are releasing the global-interpreter-lock (GIL) on some cython operations.
This will allow other threads to run simultaneously during computation, potentially allowing performance improvements
from multi-threading. Notably ``groupby``, ``nsmallest``, ``value_counts`` and some indexing operations benefit from this. (`8882`)  For example the groupby expression in the following code will have the GIL released during the factorization step, e.g. ``df.groupby('key')``
as well as the ``.sum()`` operation.  .. code-block:: python  N = 1000  ngroups = 10  df = DataFrame(  {"key": np.random.randint(0, ngroups, size=N), "data": np.random.randn(N)}  )  df.groupby("key")["data"].sum()  Releasing of the GIL could benefit an application that uses threads for user interactions (e.g. QT_), or performing multi-threaded computations. A nice example of a library that can handle these types of computation-in-parallel is the dask_ library.  .. _dask: https://dask.readthedocs.io/en/latest/
.. _QT: https://wiki.python.org/moin/PyQt  .. _whatsnew_0170.plot:  Plot submethods
^^^  The Series and DataFrame ``.plot()`` method allows for customizing `plot types<visualization.other>` by supplying the ``kind`` keyword arguments. Unfortunately, many of these kinds of plots use different required and optional keyword arguments, which makes it difficult to discover what any given plot kind uses out of the dozens of possible arguments.  To alleviate this issue, we have added a new, optional plotting interface, which exposes each kind of plot as a method of the ``.plot`` attribute. Instead of writing ``series.plot(kind=<kind>, ...)``, you can now also use ``series.plot.<kind>(...)``:  .. ipython::  In [13]: df = pd.DataFrame(np.random.rand(10, 2), columns=['a', 'b'])  In [14]: df.plot.bar()  As a result of this change, these methods are now all discoverable via tab-completion:  .. ipython::  In [15]: df.plot.<TAB>  # noqa: E225, E999  df.plot.area  df.plot.barh  df.plot.density  df.plot.hist  df.plot.line  df.plot.scatter  df.plot.bar  df.plot.box  df.plot.hexbin  df.plot.kde  df.plot.pie  Each method signature only includes relevant arguments. Currently, these are limited to required arguments, but in the future these will include optional arguments, as well. For an overview, see the new `api.dataframe.plotting` API documentation.  .. _whatsnew_0170.strftime:  Additional methods for ``dt`` accessor
^^^  Series.dt.strftime

"""  We are now supporting a ``Series.dt.strftime`` method for datetime-likes to generate a formatted string (`10110`). Examples:  .. ipython:: python  # DatetimeIndex  s = pd.Series(pd.date_range("20130101", periods=4))  s  s.dt.strftime("%Y/%m/%d")  .. ipython:: python  # PeriodIndex  s = pd.Series(pd.period_range("20130101", periods=4))  s  s.dt.strftime("%Y/%m/%d")  The string format is as the python standard library and details can be found here  Series.dt.total_seconds
"""  ``pd.Series`` of type ``timedelta64`` has new method ``.dt.total_seconds()`` returning the duration of the timedelta in seconds (`10817`)  .. ipython:: python  # TimedeltaIndex  s = pd.Series(pd.timedelta_range("1 minutes", periods=4))  s  s.dt.total_seconds()  .. _whatsnew_0170.periodfreq:  Period frequency enhancement
^^^  ``Period``, ``PeriodIndex`` and ``period_range`` can now accept multiplied freq. Also, ``Period.freq`` and ``PeriodIndex.freq`` are now stored as a ``DateOffset`` instance like ``DatetimeIndex``, and not as ``str`` (`7811`)  A multiplied freq represents a span of corresponding length. The example below creates a period of 3 days. Addition and subtraction will shift the period by its span.  .. ipython:: python  p = pd.Period("2015-08-01", freq="3D")  p  p + 1  p - 2  p.to_timestamp()  p.to_timestamp(how="E")  You can use the multiplied freq in ``PeriodIndex`` and ``period_range``.  .. ipython:: python  idx = pd.period_range("2015-08-01", periods=4, freq="2D")  idx  idx + 1  .. _whatsnew_0170.enhancements.sas_xport:  Support for SAS XPORT files
^^^  `~pandas.io.read_sas` provides support for reading *SAS XPORT* format files. (`4052`).  .. code-block:: python  df = pd.read_sas("sas_xport.xpt")  It is also possible to obtain an iterator and read an XPORT file
incrementally.  .. code-block:: python  for df in pd.read_sas("sas_xport.xpt", chunksize=1000):  do_something(df)  See the `docs <io.sas>` for more details.  .. _whatsnew_0170.matheval:  Support for math functions in .eval()
^^^  `~pandas.eval` now supports calling math functions (`4893`)  .. code-block:: python  df = pd.DataFrame({"a": np.random.randn(10)})  df.eval("b = sin(a)")  The support math functions are ``sin``, ``cos``, ``exp``, ``log``, ``expm1``, ``log1p``,
``sqrt``, ``sinh``, ``cosh``, ``tanh``, ``arcsin``, ``arccos``, ``arctan``, ``arccosh``,
``arcsinh``, ``arctanh``, ``abs`` and ``arctan2``.  These functions map to the intrinsics for the ``NumExpr`` engine.  For the Python
engine, they are mapped to ``NumPy`` calls.  Changes to Excel with ``MultiIndex``
^^^  In version 0.16.2 a ``DataFrame`` with ``MultiIndex`` columns could not be written to Excel via ``to_excel``.
That functionality has been added (`10564`), along with updating  ``read_excel`` so that the data can
be read back with, no loss of information, by specifying which columns/rows make up the ``MultiIndex``
in the ``header`` and ``index_col`` parameters (`4679`)  See the `documentation <io.excel>` for more details.  .. ipython:: python  df = pd.DataFrame(  [[1, 2, 3, 4], [5, 6, 7, 8]],  columns=pd.MultiIndex.from_product(  [["foo", "bar"], ["a", "b"]], names=["col1", "col2"]  ),  index=pd.MultiIndex.from_product([["j"], ["l", "k"]], names=["i1", "i2"]),  )  df  df.to_excel("test.xlsx")  df = pd.read_excel("test.xlsx", header=[0, 1], index_col=[0, 1])  df  .. ipython:: python  import os  os.remove("test.xlsx")  Previously, it was necessary to specify the ``has_index_names`` argument in ``read_excel``,
if the serialized data had index names.  For version 0.17.0 the output format of ``to_excel``
has been changed to make this keyword unnecessary - the change is shown below.  **Old**  **New**  .. warning::  Excel files saved in version 0.16.2 or prior that had index names will still able to be read in,  but the ``has_index_names`` argument must specified to ``True``.  .. _whatsnew_0170.gbq:  Google BigQuery enhancements
^^^

^^^
- Added ability to automatically create a table/dataset using the `pandas.io.gbq.to_gbq` function if the destination table/dataset does not exist. (`8325`, `11121`).
- Added ability to replace an existing table and schema when calling the pandas.io.gbq.to_gbq` function via the ``if_exists`` argument. See the `docs_ for more details (`8325`).
- ``InvalidColumnOrder`` and ``InvalidPageToken`` in the gbq module will raise ``ValueError`` instead of ``IOError``.
- The ``generate_bq_schema()`` function is now deprecated and will be removed in a future version (`11121`)
- The gbq module will now support Python 3 (`11094`).  .. _whatsnew_0170.east_asian_width:  Display alignment with Unicode East Asian width
^^^  .. warning::  Enabling this option will affect the performance for printing of ``DataFrame`` and ``Series`` (about 2 times slower).  Use only when it is actually required.  Some East Asian countries use Unicode characters its width is corresponding to 2 alphabets. If a ``DataFrame`` or ``Series`` contains these characters, the default output cannot be aligned properly. The following options are added to enable precise handling for these characters.  - ``display.unicode.east_asian_width``: Whether to use the Unicode East Asian Width to calculate the display text width. (`2612`)
- ``display.unicode.ambiguous_as_wide``: Whether to handle Unicode characters belong to Ambiguous as Wide. (`11102`)  .. ipython:: python  df = pd.DataFrame({u"国籍": ["UK", u"日本"], u"名前": ["Alice", u"しのぶ"]})  df  .. ipython:: python  pd.set_option("display.unicode.east_asian_width", True)  df  For further details, see `here <options.east_asian_width>`  .. ipython:: python  pd.set_option("display.unicode.east_asian_width", False)  .. _whatsnew_0170.enhancements.other:  Other enhancements
^^^  - Support for ``openpyxl`` >= 2.2. The API for style support is now stable (`10125`)

- ``merge`` now accepts the argument ``indicator`` which adds a Categorical-type column (by default called ``_merge``) to the output object that takes on the values (`8790`)  ===  ===  Observation Origin  ``_merge`` value  ===  ===  Merge key only in ``'left'`` frame  ``left_only``  Merge key only in ``'right'`` frame  ``right_only``  Merge key in both frames  ``both``  ===  ===  .. ipython:: python  df1 = pd.DataFrame({"col1": [0, 1], "col_left": ["a", "b"]})  df2 = pd.DataFrame({"col1": [1, 2, 2], "col_right": [2, 2, 2]})  pd.merge(df1, df2, on="col1", how="outer", indicator=True)  For more, see the `updated docs <merging.indicator>`  - ``pd.to_numeric`` is a new function to coerce strings to numbers (possibly with coercion) (`11133`)  - ``pd.merge`` will now allow duplicate column names if they are not merged upon (`10639`).  - ``pd.pivot`` will now allow passing index as ``None`` (`3962`).  - ``pd.concat`` will now use existing Series names if provided (`10698`).  .. ipython:: python  foo = pd.Series([1, 2], name="foo")  bar = pd.Series([1, 2])  baz = pd.Series([4, 5])  Previous behavior:  .. code-block:: ipython  In [1]: pd.concat([foo, bar, baz], axis=1)  Out[1]:  0  1  2  0  1  1  4  1  2  2  5  New behavior:  .. ipython:: python  pd.concat([foo, bar, baz], axis=1)  - ``DataFrame`` has gained the ``nlargest`` and ``nsmallest`` methods (`10393`)  - Add a ``limit_direction`` keyword argument that works with ``limit`` to enable ``interpolate`` to fill ``NaN`` values forward, backward, or both (`9218`, `10420`, `1115`)  .. ipython:: python  ser = pd.Series([np.nan, np.nan, 5, np.nan, np.nan, np.nan, 13])  ser.interpolate(limit=1, limit_direction="both")  - Added a ``DataFrame.round`` method to round the values to a variable number of decimal places (`10568`).  .. ipython:: python  df = pd.DataFrame(  np.random.random([3, 3]),  columns=["A", "B", "C"],  index=["first", "second", "third"],  )  df  df.round(2)  df.round({"A": 0, "C": 2})  - ``drop_duplicates`` and ``duplicated`` now accept a ``keep`` keyword to target first, last, and all duplicates. The ``take_last`` keyword is deprecated, see `here <whatsnew_0170.deprecations>` (`6511`, `8505`)  .. ipython:: python  s = pd.Series(["A", "B", "C", "A", "B", "D"])  s.drop_duplicates()  s.drop_duplicates(keep="last")  s.drop_duplicates(keep=False)  - Reindex now has a ``tolerance`` argument that allows for finer control of `basics.limits_on_reindex_fill` (`10411`):  .. ipython:: python  df = pd.DataFrame({"x": range(5), "t": pd.date_range("2000-01-01", periods=5)})  df.reindex([0.1, 1.9, 3.5], method="nearest", tolerance=0.2)  When used on a ``DatetimeIndex``, ``TimedeltaIndex`` or ``PeriodIndex``, ``tolerance`` will coerced into a ``Timedelta`` if possible. This allows you to specify tolerance with a string:  .. ipython:: python  df = df.set_index("t")  df.reindex(pd.to_datetime(["1999-12-31"]), method="nearest", tolerance="1 day")  ``tolerance`` is also exposed by the lower level ``Index.get_indexer`` and ``Index.get_loc`` methods.  - Added functionality to use the ``base`` argument when resampling a ``TimeDeltaIndex`` (`10530`)  - ``DatetimeIndex`` can be instantiated using strings contains ``NaT`` (`7599`)  - ``to_datetime`` can now accept the ``yearfirst`` keyword (`7599`)  - ``pandas.tseries.offsets`` larger than the ``Day`` offset can now be used with a ``Series`` for addition/subtraction (`10699`).  See the `docs <timeseries.offsetseries>` for more details.  - ``pd.Timedelta.total_seconds()`` now returns Timedelta duration to ns precision (previously microsecond precision) (`10939`)  - ``PeriodIndex`` now supports arithmetic with ``np.ndarray`` (`10638`)  - Support pickling of ``Period`` objects (`10439`)  - ``.as_blocks`` will now take a ``copy`` optional argument to return a copy of the data, default is to copy (no change in behavior from prior versions), (`9607`)  - ``regex`` argument to ``DataFrame.filter`` now handles numeric column names instead of raising ``ValueError`` (`10384`).  - Enable reading gzip compressed files via URL, either by explicitly setting the compression parameter or by inferring from the presence of the HTTP Content-Encoding header in the response (`8685`)  - Enable writing Excel files in `memory <io.excel_writing_buffer>` using StringIO/BytesIO (`7074`)  - Enable serialization of lists and dicts to strings in ``ExcelWriter`` (`8188`)  - SQL io functions now accept a SQLAlchemy connectable. (`7877`)  - ``pd.read_sql`` and ``to_sql`` can accept database URI as ``con`` parameter (`10214`)  - ``read_sql_table`` will now allow reading from views (`10750`).  - Enable writing complex values to ``HDFStores`` when using the ``table`` format

and ``to_sql`` can accept database URI as ``con`` parameter (`10214`)  - ``read_sql_table`` will now allow reading from views (`10750`).  - Enable writing complex values to ``HDFStores`` when using the ``table`` format (`10447`)  - Enable ``pd.read_hdf`` to be used without specifying a key when the HDF file contains a single dataset (`10443`)  - ``pd.read_stata`` will now read Stata 118 type files. (`9882`)  - ``msgpack`` submodule has been updated to 0.4.6 with backward compatibility (`10581`)  - ``DataFrame.to_dict`` now accepts ``orient='index'`` keyword argument (`10844`).  - ``DataFrame.apply`` will return a Series of dicts if the passed function returns a dict and ``reduce=True`` (`8735`).  - Allow passing ``kwargs`` to the interpolation methods (`10378`).  - Improved error message when concatenating an empty iterable of ``Dataframe`` objects (`9157`)  - ``pd.read_csv`` can now read bz2-compressed files incrementally, and the C parser can read bz2-compressed files from AWS S3 (`11070`, `11072`).  - In ``pd.read_csv``, recognize ``s3n://`` and ``s3a://`` URLs as designating S3 file storage (`11070`, `11071`).  - Read CSV files from AWS S3 incrementally, instead of first downloading the entire file. (Full file download still required for compressed files in Python 2.)  (`11070`, `11073`)  - ``pd.read_csv`` is now able to infer compression type for files read from AWS S3 storage (`11070`, `11074`).  .. _whatsnew_0170.api:  .. _whatsnew_0170.api_breaking:  Backwards incompatible API changes

~~~  .. _whatsnew_0170.api_breaking.sorting:  Changes to sorting API
^^^  The sorting API has had some longtime inconsistencies. (`9816`, `8239`).  Here is a summary of the API **PRIOR** to 0.17.0:  - ``Series.sort`` is **INPLACE** while ``DataFrame.sort`` returns a new object.
- ``Series.order`` returns a new object
- It was possible to use ``Series/DataFrame.sort_index`` to sort by **values** by passing the ``by`` keyword.
- ``Series/DataFrame.sortlevel`` worked only on a ``MultiIndex`` for sorting by index.  To address these issues, we have revamped the API:  - We have introduced a new method, `DataFrame.sort_values`, which is the merger of ``DataFrame.sort()``, ``Series.sort()``,  and ``Series.order()``, to handle sorting of **values**.
- The existing methods ``Series.sort()``, ``Series.order()``, and ``DataFrame.sort()`` have been deprecated and will be removed in a  future version.
- The ``by`` argument of ``DataFrame.sort_index()`` has been deprecated and will be removed in a future version.
- The existing method ``.sort_index()`` will gain the ``level`` keyword to enable level sorting.  We now have two distinct and non-overlapping methods of sorting. A ``*`` marks items that
will show a ``FutureWarning``.  To sort by the **values**:  ===  ===
Previous  Replacement
===  ===
\* ``Series.order()``  ``Series.sort_values()``
\* ``Series.sort()``  ``Series.sort_values(inplace=True)``
\* ``DataFrame.sort(columns=...)``  ``DataFrame.sort_values(by=...)``
===  ===  To sort by the **index**:  ===  ===
===  ===
``Series.sort_index()``  ``Series.sort_index()``
``Series.sortlevel(level=...)``  ``Series.sort_index(level=...``)
``DataFrame.sort_index()``  ``DataFrame.sort_index()``
``DataFrame.sortlevel(level=...)``  ``DataFrame.sort_index(level=...)``
\* ``DataFrame.sort()``  ``DataFrame.sort_index()``
===  ===  We have also deprecated and changed similar methods in two Series-like classes, ``Index`` and ``Categorical``.  ===  ===
===  ===
\* ``Index.order()``  ``Index.sort_values()``
\* ``Categorical.order()``  ``Categorical.sort_values()``
===  ===  .. _whatsnew_0170.api_breaking.to_datetime:  Changes to to_datetime and to_timedelta
^^^  Error handling
"""  The default for ``pd.to_datetime`` error handling has changed to ``errors='raise'``.
In prior versions it was ``errors='ignore'``. Furthermore, the ``coerce`` argument
has been deprecated in favor of ``errors='coerce'``. This means that invalid parsing
will raise rather that return the original input as in previous versions. (`10636`)  Previous behavior:  .. code-block:: ipython  In [2]: pd.to_datetime(['2009-07-31', 'asd'])  Out[2]: array(['2009-07-31', 'asd'], dtype=object)  New behavior:  .. code-block:: ipython  In [3]: pd.to_datetime(['2009-07-31', 'asd'])  ValueError: Unknown string format  Of course you can coerce this as well.  .. ipython:: python  pd.to_datetime(["2009-07-31", "asd"], errors="coerce")  To keep the previous behavior, you can use ``errors='ignore'``:  .. ipython:: python  pd.to_datetime(["2009-07-31", "asd"], errors="ignore")  Furthermore, ``pd.to_timedelta`` has gained a similar API, of ``errors='raise'|'ignore'|'coerce'``, and the ``coerce`` keyword
has been deprecated in favor of ``errors='coerce'``.  Consistent parsing
"""  The string parsing of ``to_datetime``, ``Timestamp`` and ``DatetimeIndex`` has
been made consistent. (`7599`)  Prior to v0.17.0, ``Timestamp`` and ``to_datetime`` may parse year-only datetime-string incorrectly using today's date, otherwise ``DatetimeIndex``
uses the beginning of the year. ``Timestamp`` and ``to_datetime`` may raise ``ValueError`` in some types of datetime-string which ``DatetimeIndex``

can parse, such as a quarterly string.  Previous behavior:  .. code-block:: ipython  In [1]: pd.Timestamp('2012Q2')  Traceback  ...  ValueError: Unable to parse 2012Q2  # Results in today's date.  In [2]: pd.Timestamp('2014')  Out [2]: 2014-08-12 000  v0.17.0 can parse them as below. It works on ``DatetimeIndex`` also.  New behavior:  .. ipython:: python  pd.Timestamp("2012Q2")  pd.Timestamp("2014")  pd.DatetimeIndex(["2012Q2", "2014"])  .. note::  If you want to perform calculations based on today's date, use ``Timestamp.now()`` and ``pandas.tseries.offsets``.  .. ipython:: python  import pandas.tseries.offsets as offsets  pd.Timestamp.now()  pd.Timestamp.now() + offsets.DateOffset(years=1)  Changes to Index comparisons
^^^  Operator equal on ``Index`` should behavior similarly to ``Series`` (`9947`, `10637`)  Starting in v0.17.0, comparing ``Index`` objects of different lengths will raise
a ``ValueError``. This is to be consistent with the behavior of ``Series``.  Previous behavior:  .. code-block:: ipython  In [2]: pd.Index([1, 2, 3]) == pd.Index([1, 4, 5])  Out[2]: array([ True, False, False], dtype=bool)  In [3]: pd.Index([1, 2, 3]) == pd.Index([2])  Out[3]: array([False,  True, False], dtype=bool)  In [4]: pd.Index([1, 2, 3]) == pd.Index([1, 2])  Out[4]: False  New behavior:  .. code-block:: ipython  In [8]: pd.Index([1, 2, 3]) == pd.Index([1, 4, 5])  Out[8]: array([ True, False, False], dtype=bool)  In [9]: pd.Index([1, 2, 3]) == pd.Index([2])  ValueError: Lengths must match to compare  In [10]: pd.Index([1, 2, 3]) == pd.Index([1, 2])  ValueError: Lengths must match to compare  Note that this is different from the ``numpy`` behavior where a comparison can
be broadcast:  .. ipython:: python  np.array([1, 2, 3]) == np.array([1])  or it can return False if broadcasting can not be done:  .. ipython:: python  np.array([1, 2, 3]) == np.array([1, 2])  Changes to boolean comparisons vs. None
^^^  Boolean comparisons of a ``Series`` vs ``None`` will now be equivalent to comparing with ``np.nan``, rather than raise ``TypeError``. (`1079`).  .. ipython:: python  s = pd.Series(range(3))  s.iloc[1] = None  s  Previous behavior:  .. code-block:: ipython  In [5]: s == None  TypeError: Could not compare <type 'NoneType'> type with Series  New behavior:  .. ipython:: python  s == None  Usually you simply want to know which values are null.  .. ipython:: python  s.isnull()  .. warning::  You generally will want to use ``isnull/notnull`` for these types of comparisons, as ``isnull/notnull`` tells you which elements are null. One has to be  mindful that ``nan's`` don't compare equal, but ``None's`` do. Note that pandas/numpy uses the fact that ``np.nan != np.nan``, and treats ``None`` like ``np.nan``.  .. ipython:: python  None == None  np.nan == np.nan  .. _whatsnew_0170.api_breaking.hdf_dropna:  HDFStore dropna behavior
^^^  The default behavior for HDFStore write functions with ``format='table'`` is now to keep rows that are all missing. Previously, the behavior was to drop rows that were all missing save the index. The previous behavior can be replicated using the ``dropna=True`` option. (`9382`)  Previous behavior:  .. ipython:: python  df_with_missing = pd.DataFrame(  {"col1": [0, np.nan, 2], "col2": [1, np.nan, np.nan]}  )  df_with_missing  .. code-block:: ipython  In [27]:  df_with_missing.to_hdf('file.h5',  'df_with_missing',  format='table',  mode='w')  In [28]: pd.read_hdf('file.h5', 'df_with_missing')  Out [28]:  col1  col2  0  0  1  2  2  NaN  New behavior:  .. ipython:: python  df_with_missing.to_hdf("file.h5", "df_with_missing", format="table", mode="w")  pd.read_hdf("file.h5", "df_with_missing")  .. ipython:: python  import os  os.remove("file.h5")  See the `docs <io.hdf5>` for more details.  .. _whatsnew_0170.api_breaking.display_precision:  Changes to ``display.precision`` option

^^^  The ``display.precision`` option has been clarified to refer to decimal places (`10451`).  Earlier versions of pandas would format floating point numbers to have one less decimal place than the value in
``display.precision``.  .. code-block:: ipython  In [1]: pd.set_option('display.precision', 2)  In [2]: pd.DataFrame({'x': [123.456789]})  Out[2]:  x  0  123.5  If interpreting precision as "significant figures" this did work for scientific notation but that same interpretation
did not work for values with standard formatting. It was also out of step with how numpy handles formatting.  Going forward the value of ``display.precision`` will directly control the number of places after the decimal, for
regular formatting as well as scientific notation, similar to how numpy's ``precision`` print option works.  .. ipython:: python  pd.set_option("display.precision", 2)  pd.DataFrame({"x": [123.456789]})  To preserve output behavior with prior versions the default value of ``display.precision`` has been reduced to ``6``
from ``7``.  .. ipython:: python  pd.set_option("display.precision", 6)  .. _whatsnew_0170.api_breaking.categorical_unique:  Changes to ``Categorical.unique``
^^^  ``Categorical.unique`` now returns new ``Categoricals`` with ``categories`` and ``codes`` that are unique, rather than returning ``np.array`` (`10508`)  - unordered category: values and categories are sorted by appearance order.
- ordered category: values are sorted by appearance order, categories keep existing order.  .. ipython:: python  cat = pd.Categorical(["C", "A", "B", "C"], categories=["A", "B", "C"], ordered=True)  cat  cat.unique()  cat = pd.Categorical(["C", "A", "B", "C"], categories=["A", "B", "C"])  cat  cat.unique()  Changes to ``bool`` passed as ``header`` in parsers
^^^  In earlier versions of pandas, if a bool was passed the ``header`` argument of
``read_csv``, ``read_excel``, or ``read_html`` it was implicitly converted to
an integer, resulting in ``header=0`` for ``False`` and ``header=1`` for ``True``
(`6113`)  A ``bool`` input to ``header`` will now raise a ``TypeError``  .. code-block:: ipython  In [29]: df = pd.read_csv('data.csv', header=False)  TypeError: Passing a bool to header is invalid. Use header=None for no header or  header=int or list-like of ints to specify the row(s) making up the column names  .. _whatsnew_0170.api_breaking.other:  Other API changes
^^^  - Line and kde plot with ``subplots=True`` now uses default colors, not all black. Specify ``color='k'`` to draw all lines in black (`9894`)
- Calling the ``.value_counts()`` method on a Series with a ``categorical`` dtype now returns a Series with a ``CategoricalIndex`` (`10704`)
- The metadata properties of subclasses of pandas objects will now be serialized (`10553`).
- ``groupby`` using ``Categorical`` follows the same rule as ``Categorical.unique`` described above  (`10508`)
- When constructing ``DataFrame`` with an array of ``complex64`` dtype previously meant the corresponding column  was automatically promoted to the ``complex128`` dtype. pandas will now preserve the itemsize of the input for complex data (`10952`)
- some numeric reduction operators would return ``ValueError``, rather than ``TypeError`` on object types that includes strings and numbers (`11131`)
- Passing currently unsupported ``chunksize`` argument to ``read_excel`` or ``ExcelFile.parse`` will now raise ``NotImplementedError`` (`8011`)
- Allow an ``ExcelFile`` object to be passed into ``read_excel`` (`11198`)
- ``DatetimeIndex.union`` does not infer ``freq`` if ``self`` and the input have ``None`` as ``freq`` (`11086`)

- ``NaT``'s methods now either raise ``ValueError``, or return ``np.nan`` or ``NaT`` (`9513`)  ===  ===  Behavior  Methods  ===  ===  return ``np.nan``  ``weekday``, ``isoweekday``  return ``NaT``  ``date``, ``now``, ``replace``, ``to_datetime``, ``today``  return ``np.datetime64('NaT')``  ``to_datetime64`` (unchanged)  raise ``ValueError``  All other public methods (names not beginning with underscores)  ===  ===  .. _whatsnew_0170.deprecations:  Deprecations
^^^  - For ``Series`` the following indexing functions are deprecated (`10177`).  ===  ===  Deprecated Function  Replacement  ===  ===  ``.irow(i)``  ``.iloc[i]`` or ``.iat[i]``  ``.iget(i)``  ``.iloc[i]`` or ``.iat[i]``  ``.iget_value(i)``  ``.iloc[i]`` or ``.iat[i]``  ===  ===  - For ``DataFrame`` the following indexing functions are deprecated (`10177`).  ===  ===  Deprecated Function  Replacement  ===  ===  ``.irow(i)``  ``.iloc[i]``  ``.iget_value(i, j)``  ``.iloc[i, j]`` or ``.iat[i, j]``  ``.icol(j)``  ``.iloc[:, j]``  ===  ===  .. note:: These indexing function have been deprecated in the documentation since 0.11.0.  - ``Categorical.name`` was deprecated to make ``Categorical`` more ``numpy.ndarray`` like. Use ``Series(cat, name="whatever")`` instead (`10482`).
- Setting missing values (NaN) in a ``Categorical``'s ``categories`` will issue a warning (`10748`). You can still have missing values in the ``values``.
- ``drop_duplicates`` and ``duplicated``'s ``take_last`` keyword was deprecated in favor of ``keep``. (`6511`, `8505`)
- ``Series.nsmallest`` and ``nlargest``'s ``take_last`` keyword was deprecated in favor of ``keep``. (`10792`)
- ``DataFrame.combineAdd`` and ``DataFrame.combineMult`` are deprecated. They  can easily be replaced by using the ``add`` and ``mul`` methods:  ``DataFrame.add(other, fill_value=0)`` and ``DataFrame.mul(other, fill_value=1.)``  (`10735`).
- ``TimeSeries`` deprecated in favor of ``Series`` (note that this has been an alias since 0.13.0), (`10890`)
- ``SparsePanel`` deprecated and will be removed in a future version (`11157`).
- ``Series.is_time_series`` deprecated in favor of ``Series.index.is_all_dates`` (`11135`)
- Legacy offsets (like ``'A@JAN'``) are deprecated (note that this has been alias since 0.8.0) (`10878`)
- ``WidePanel`` deprecated in favor of ``Panel``, ``LongPanel`` in favor of ``DataFrame`` (note these have been aliases since < 0.11.0), (`10892`)
- ``DataFrame.convert_objects`` has been deprecated in favor of type-specific functions ``pd.to_datetime``, ``pd.to_timestamp`` and ``pd.to_numeric`` (new in 0.17.0) (`11133`).  .. _whatsnew_0170.prior_deprecations:  Removal of prior version deprecations/changes
^^^  - Removal of ``na_last`` parameters from ``Series.order()`` and ``Series.sort()``, in favor of ``na_position``. (`5231`)
- Remove of ``percentile_width`` from ``.describe()``, in favor of ``percentiles``. (`7088`)
- Removal of ``colSpace`` parameter from ``DataFrame.to_string()``, in favor of ``col_space``, circa 0.8.0 version.
- Removal of automatic time-series broadcasting (`2304`)  .. ipython:: python  np.random.seed(1234)  df = pd.DataFrame(  np.random.randn(5, 2),  columns=list("AB"),  index=pd.date_range("2013-01-01", periods=5),  )  df  Previously  .. code-block:: ipython  In [3]: df + df.A  FutureWarning: TimeSeries broadcasting along DataFrame index by default is deprecated.  Please use DataFrame.<op> to explicitly broadcast arithmetic operations along the index  Out[3]:  A  B  2013-01-01  0.942870 -0.719541  2013-01-02  2.865414  1.120055  2013-01-03 -1.441177  0.166574  2013-01-04  1.719177  0.223065  2013-01-05  0.031393 -2.226989  Current  .. ipython:: python  df.add(df.A, axis="index")  - Remove ``table`` keyword in ``HDFStore.put/append``, in favor of using ``format=`` (`4645`)

- Remove ``kind`` in ``read_excel/ExcelFile`` as its unused (`4712`)
- Remove ``infer_type`` keyword from ``pd.read_html`` as its unused (`4770`, `7032`)
- Remove ``offset`` and ``timeRule`` keywords from ``Series.tshift/shift``, in favor of ``freq`` (`4853`, `4864`)
- Remove ``pd.load/pd.save`` aliases in favor of ``pd.to_pickle/pd.read_pickle`` (`3787`)  .. _whatsnew_0170.performance:  Performance improvements
~~~  - Development support for benchmarking with the Air Speed Velocity library (`8361`)
- Added vbench benchmarks for alternative ExcelWriter engines and reading Excel files (`7171`)
- Performance improvements in ``Categorical.value_counts`` (`10804`)
- Performance improvements in ``SeriesGroupBy.nunique`` and ``SeriesGroupBy.value_counts`` and ``SeriesGroupby.transform`` (`10820`, `11077`)
- Performance improvements in ``DataFrame.drop_duplicates`` with integer dtypes (`10917`)
- Performance improvements in ``DataFrame.duplicated`` with wide frames. (`10161`, `11180`)
- 4x improvement in ``timedelta`` string parsing (`6755`, `10426`)
- 8x improvement in ``timedelta64`` and ``datetime64`` ops (`6755`)
- Significantly improved performance of indexing ``MultiIndex`` with slicers (`10287`)
- 8x improvement in ``iloc`` using list-like input (`10791`)
- Improved performance of ``Series.isin`` for datetimelike/integer Series (`10287`)
- 20x improvement in ``concat`` of Categoricals when categories are identical (`10587`)
- Improved performance of ``to_datetime`` when specified format string is ISO8601 (`10178`)
- 2x improvement of ``Series.value_counts`` for float dtype (`10821`)
- Enable ``infer_datetime_format`` in ``to_datetime`` when date components do not have 0 padding (`11142`)
- Regression from 0.16.1 in constructing ``DataFrame`` from nested dictionary (`11084`)
- Performance improvements in addition/subtraction operations for ``DateOffset`` with ``Series`` or ``DatetimeIndex``  (`10744`, `11205`)  .. _whatsnew_0170.bug_fixes:  Bug fixes
~~~  - Bug in incorrect computation of ``.mean()`` on ``timedelta64[ns]`` because of overflow (`9442`)
- Bug in  ``.isin`` on older numpies (`11232`)
- Bug in ``DataFrame.to_html(index=False)`` renders unnecessary ``name`` row (`10344`)
- Bug in ``DataFrame.to_latex()`` the ``column_format`` argument could not be passed (`9402`)
- Bug in ``DatetimeIndex`` when localizing with ``NaT`` (`10477`)
- Bug in ``Series.dt`` ops in preserving meta-data (`10477`)
- Bug in preserving ``NaT`` when passed in an otherwise invalid ``to_datetime`` construction (`10477`)
- Bug in ``DataFrame.apply`` when function returns categorical series. (`9573`)
- Bug in ``to_datetime`` with invalid dates and formats supplied (`10154`)
- Bug in ``Index.drop_duplicates`` dropping name(s) (`10115`)
- Bug in ``Series.quantile`` dropping name (`10881`)
- Bug in ``pd.Series`` when setting a value on an empty ``Series`` whose index has a frequency. (`10193`)
- Bug in ``pd.Series.interpolate`` with invalid ``order`` keyword values. (`10633`)
- Bug in ``DataFrame.plot`` raises ``ValueError`` when color name is specified by multiple characters (`10387`)
- Bug in ``Index`` construction with a mixed list of tuples (`10697`)
- Bug in ``DataFrame.reset_index`` when index contains ``NaT``. (`10388`)
- Bug in ``ExcelReader`` when worksheet is empty (`6403`)
- Bug in ``BinGrouper.group_info`` where returned values are not compatible with base class (`10914`)
- Bug in clearing the cache on ``DataFrame.pop`` and a subsequent inplace op (`10912`)
- Bug in indexing with a mixed-integer ``Index`` causing an ``ImportError`` (`10610`)
- Bug in ``Series.count`` when index has nulls (`10946`)
- Bug in pickling of a non-regular freq ``DatetimeIndex`` (`11002`)
- Bug causing ``DataFrame.where`` to not respect the ``axis`` parameter when the frame has a symmetric shape. (`9736`)
- Bug in ``Table.select_column`` where name is not preserved (`10392`)
- Bug in ``offsets.generate_range`` where ``start`` and ``end`` have finer precision than ``offset`` (`9907`)

- Bug in ``pd.rolling_*`` where ``Series.name`` would be lost in the output (`10565`)
- Bug in ``stack`` when index or columns are not unique. (`10417`)
- Bug in setting a ``Panel`` when an axis has a MultiIndex (`10360`)
- Bug in ``USFederalHolidayCalendar`` where ``USMemorialDay`` and ``USMartinLutherKingJr`` were incorrect (`10278` and `9760` )
- Bug in ``.sample()`` where returned object, if set, gives unnecessary ``SettingWithCopyWarning`` (`10738`)
- Bug in ``.sample()`` where weights passed as ``Series`` were not aligned along axis before being treated positionally, potentially causing problems if weight indices were not aligned with sampled object. (`10738`)  - Regression fixed in (`9311`, `6620`, `9345`), where groupby with a datetime-like converting to float with certain aggregators (`10979`)  - Bug in ``DataFrame.interpolate`` with ``axis=1`` and ``inplace=True`` (`10395`)
- Bug in ``io.sql.get_schema`` when specifying multiple columns as primary  key (`10385`).  - Bug in ``groupby(sort=False)`` with datetime-like ``Categorical`` raises ``ValueError`` (`10505`)
- Bug in ``groupby(axis=1)`` with ``filter()`` throws ``IndexError`` (`11041`)
- Bug in ``test_categorical`` on big-endian builds (`10425`)
- Bug in ``Series.shift`` and ``DataFrame.shift`` not supporting categorical data (`9416`)
- Bug in ``Series.map`` using categorical ``Series`` raises ``AttributeError`` (`10324`)
- Bug in ``MultiIndex.get_level_values`` including ``Categorical`` raises ``AttributeError`` (`10460`)
- Bug in ``pd.get_dummies`` with ``sparse=True`` not returning ``SparseDataFrame`` (`10531`)
- Bug in ``Index`` subtypes (such as ``PeriodIndex``) not returning their own type for ``.drop`` and ``.insert`` methods (`10620`)
- Bug in ``algos.outer_join_indexer`` when ``right`` array is empty (`10618`)  - Bug in ``filter`` (regression from 0.16.0) and ``transform`` when grouping on multiple keys, one of which is datetime-like (`10114`)  - Bug in ``to_datetime`` and ``to_timedelta`` causing ``Index`` name to be lost (`10875`)
- Bug in ``len(DataFrame.groupby)`` causing ``IndexError`` when there's a column containing only NaNs (`11016`)  - Bug that caused segfault when resampling an empty Series (`10228`)
- Bug in ``DatetimeIndex`` and ``PeriodIndex.value_counts`` resets name from its result, but retains in result's ``Index``. (`10150`)
- Bug in ``pd.eval`` using ``numexpr`` engine coerces 1 element numpy array to scalar (`10546`)
- Bug in ``pd.concat`` with ``axis=0`` when column is of dtype ``category`` (`10177`)
- Bug in ``read_msgpack`` where input type is not always checked (`10369`, `10630`)
- Bug in ``pd.read_csv`` with kwargs ``index_col=False``, ``index_col=['a', 'b']`` or ``dtype``  (`10413`, `10467`, `10577`)
- Bug in ``Series.from_csv`` with ``header`` kwarg not setting the ``Series.name`` or the ``Series.index.name`` (`10483`)
- Bug in ``groupby.var`` which caused variance to be inaccurate for small float values (`10448`)
- Bug in ``Series.plot(kind='hist')`` Y Label not informative (`10485`)
- Bug in ``read_csv`` when using a converter which generates a ``uint8`` type (`9266`)  - Bug causes memory leak in time-series line and area plot (`9003`)  - Bug when setting a ``Panel`` sliced along the major or minor axes when the right-hand side is a ``DataFrame`` (`11014`)
- Bug that returns ``None`` and does not raise ``NotImplementedError`` when operator functions (e.g. ``.add``) of ``Panel`` are not implemented (`7692`)  - Bug in line and kde plot cannot accept multiple colors when ``subplots=True`` (`9894`)
- Bug in ``DataFrame.plot`` raises ``ValueError`` when color name is specified by multiple characters (`10387`)  - Bug in left and right ``align`` of ``Series`` with ``MultiIndex`` may be inverted (`10665`)

- Bug in left and right ``join`` of with ``MultiIndex`` may be inverted (`10741`)  - Bug in ``read_stata`` when reading a file with a different order set in ``columns`` (`10757`)
- Bug in ``Categorical`` may not representing properly when category contains ``tz`` or ``Period`` (`10713`)
- Bug in ``Categorical.__iter__`` may not returning correct ``datetime`` and ``Period`` (`10713`)
- Bug in indexing with a ``PeriodIndex`` on an object with a ``PeriodIndex`` (`4125`)
- Bug in ``read_csv`` with ``engine='c'``: EOF preceded by a comment, blank line, etc. was not handled correctly (`10728`, `10548`)  - Reading "famafrench" data via ``DataReader`` results in HTTP 404 error because of the website url is changed (`10591`).
- Bug in ``read_msgpack`` where DataFrame to decode has duplicate column names (`9618`)
- Bug in ``io.common.get_filepath_or_buffer`` which caused reading of valid S3 files to fail if the bucket also contained keys for which the user does not have read permission (`10604`)
- Bug in vectorised setting of timestamp columns with python ``datetime.date`` and numpy ``datetime64`` (`10408`, `10412`)
- Bug in ``Index.take`` may add unnecessary ``freq`` attribute (`10791`)
- Bug in ``merge`` with empty ``DataFrame`` may raise ``IndexError`` (`10824`)
- Bug in ``to_latex`` where unexpected keyword argument for some documented arguments (`10888`)
- Bug in indexing of large ``DataFrame`` where ``IndexError`` is uncaught (`10645` and `10692`)
- Bug in ``read_csv`` when using the ``nrows`` or ``chunksize`` parameters if file contains only a header line (`9535`)
- Bug in serialization of ``category`` types in HDF5 in presence of alternate encodings. (`10366`)
- Bug in ``pd.DataFrame`` when constructing an empty DataFrame with a string dtype (`9428`)
- Bug in ``pd.DataFrame.diff`` when DataFrame is not consolidated (`10907`)
- Bug in ``pd.unique`` for arrays with the ``datetime64`` or ``timedelta64`` dtype that meant an array with object dtype was returned instead the original dtype (`9431`)
- Bug in ``Timedelta`` raising error when slicing from 0s (`10583`)
- Bug in ``DatetimeIndex.take`` and ``TimedeltaIndex.take`` may not raise ``IndexError`` against invalid index (`10295`)
- Bug in ``Series([np.nan]).astype('M8[ms]')``, which now returns ``Series([pd.NaT])`` (`10747`)
- Bug in ``PeriodIndex.order`` reset freq (`10295`)
- Bug in ``date_range`` when ``freq`` divides ``end`` as nanos (`10885`)
- Bug in ``iloc`` allowing memory outside bounds of a Series to be accessed with negative integers (`10779`)
- Bug in ``read_msgpack`` where encoding is not respected (`10581`)
- Bug preventing access to the first index when using ``iloc`` with a list containing the appropriate negative integer (`10547`, `10779`)
- Bug in ``TimedeltaIndex`` formatter causing error while trying to save ``DataFrame`` with ``TimedeltaIndex`` using ``to_csv`` (`10833`)
- Bug in ``DataFrame.where`` when handling Series slicing (`10218`, `9558`)
- Bug where ``pd.read_gbq`` throws ``ValueError`` when Bigquery returns zero rows (`10273`)
- Bug in ``to_json`` which was causing segmentation fault when serializing 0-rank ndarray (`9576`)
- Bug in plotting functions may raise ``IndexError`` when plotted on ``GridSpec`` (`10819`)
- Bug in plot result may show unnecessary minor ticklabels (`10657`)
- Bug in ``groupby`` incorrect computation for aggregation on ``DataFrame`` with ``NaT`` (E.g ``first``, ``last``, ``min``). (`10590`, `11010`)
- Bug when constructing ``DataFrame`` where passing a dictionary with only scalar values and specifying columns did not raise an error (`10856`)
- Bug in ``.var()`` causing roundoff errors for highly similar values (`10242`)
- Bug in ``DataFrame.plot(subplots=True)`` with duplicated columns outputs incorrect result (`10962`)
- Bug in ``Index`` arithmetic may result in incorrect class (`10638`)
- Bug in ``date_range`` results in empty if freq is negative annually, quarterly and monthly (`11018`)
- Bug in ``DatetimeIndex`` cannot infer negative freq (`11018`)

- Remove use of some deprecated numpy comparison operations, mainly in tests. (`10569`)
- Bug in ``Index`` dtype may not applied properly (`11017`)
- Bug in ``io.gbq`` when testing for minimum google api client version (`10652`)
- Bug in ``DataFrame`` construction from nested ``dict`` with ``timedelta`` keys (`11129`)
- Bug in ``.fillna`` against may raise ``TypeError`` when data contains datetime dtype (`7095`, `11153`)
- Bug in ``.groupby`` when number of keys to group by is same as length of index (`11185`)
- Bug in ``convert_objects`` where converted values might not be returned if all null and ``coerce`` (`9589`)
- Bug in ``convert_objects`` where ``copy`` keyword was not respected (`9589`)  .. _whatsnew_0.17.0.contributors:  Contributors
~~~  .. contributors:: v0.16.2..v0.17.0

{{ header }}  .. note::  We are proud to announce that *pandas* has become a sponsored project of the (`NumFOCUS organization`_). This will help ensure the success of development of *pandas* as a world-class open-source project.  .. _numfocus organization: http://www.numfocus.org/blog/numfocus-announces-new-fiscally-sponsored-project-pandas  This is a minor bug-fix release from 0.17.0 and includes a large number of
We recommend that all users upgrade to this version.  Highlights include:  - Support for Conditional HTML Formatting, see `here <whatsnew_0171.style>`
- Releasing the GIL on the csv reader & other ops, see `here <whatsnew_0171.performance>`
- Fixed regression in ``DataFrame.drop_duplicates`` from 0.16.2, causing incorrect results on integer values (`11376`)  .. contents:: What's new in v0.17.1  none  New features
~~~  .. _whatsnew_0171.style:  Conditional HTML formatting
^^^  .. warning::  This is a new feature and is under active development.  We'll be adding features an  possibly making breaking changes in future  releases. Feedback is welcome in `11610`  We've added *experimental* support for conditional HTML formatting:
the visual styling of a DataFrame based on the data.
The styling is accomplished with HTML and CSS.
Accesses the styler class with the `pandas.DataFrame.style`, attribute,
an instance of `~pandas.core.style.Styler` with your data attached.  Here's a quick example:  .. ipython:: python  np.random.seed(123)  df = pd.DataFrame(np.random.randn(10, 5), columns=list("abcde"))  html = df.style.background_gradient(cmap="viridis", low=0.5)  We can render the HTML to get the following table.  .. raw:: html  whatsnew_0171_html_table.html  `~pandas.core.style.Styler` interacts nicely with the Jupyter Notebook.
See the `documentation </user_guide/style.ipynb>` for more.  .. _whatsnew_0171.enhancements:  Enhancements
~~~  - ``DatetimeIndex`` now supports conversion to strings with ``astype(str)`` (`10442`)
- Support for ``compression`` (gzip/bz2) in `pandas.DataFrame.to_csv` (`7615`)
- ``pd.read_*`` functions can now also accept `python:pathlib.Path`, or `py:py._path.local.LocalPath`  objects for the ``filepath_or_buffer`` argument. (`11033`)  - The ``DataFrame`` and ``Series`` functions ``.to_csv()``, ``.to_html()`` and ``.to_latex()`` can now handle paths beginning with tildes (e.g. ``~/Documents/``) (`11438`)
- ``DataFrame`` now uses the fields of a ``namedtuple`` as columns, if columns are not supplied (`11181`)
- ``DataFrame.itertuples()`` now returns ``namedtuple`` objects, when possible. (`11269`, `11625`)
- Added ``axvlines_kwds`` to parallel coordinates plot (`10709`)
- Option to ``.info()`` and ``.memory_usage()`` to provide for deep introspection of memory consumption. Note that this can be expensive to compute and therefore is an optional parameter. (`11595`)  .. ipython:: python  df = pd.DataFrame({"A": ["foo"] * 1000})  # noqa: F821  df["B"] = df["A"].astype("category")  # shows the '+' as we have object dtypes  df.info()  # we have an accurate memory assessment (but can be expensive to compute this)  df.info(memory_usage="deep")  - ``Index`` now has a ``fillna`` method (`10089`)  .. ipython:: python  pd.Index([1, np.nan, 3]).fillna(2)  - Series of type ``category`` now make ``.str.<...>`` and ``.dt.<...>`` accessor methods / properties available, if the categories are of that type. (`10661`)  .. ipython:: python  s = pd.Series(list("aabb")).astype("category")  s  s.str.contains("a")  date = pd.Series(pd.date_range("1/1/2015", periods=5)).astype("category")  date  date.dt.day  - ``pivot_table`` now has a ``margins_name`` argument so you can use something other than the default of 'All' (`3335`)
- Implement export of ``datetime64[ns, tz]`` dtypes with a fixed HDF5 store (`11411`)
- Pretty printing sets (e.g. in DataFrame cells) now uses set literal syntax (``{x, y}``) instead of  Legacy Python syntax (``set([x, y])``) (`11215`)

- Improve the error message in `pandas.io.gbq.to_gbq` when a streaming insert fails (`11285`)  and when the DataFrame does not match the schema of the destination table (`11359`)  .. _whatsnew_0171.api:  API changes
~~~  - raise ``NotImplementedError`` in ``Index.shift`` for non-supported index types (`8038`)
- ``min`` and ``max`` reductions on ``datetime64`` and ``timedelta64`` dtyped series now  result in ``NaT`` and not ``nan`` (`11245`).
- Indexing with a null key will raise a ``TypeError``, instead of a ``ValueError`` (`11356`)
- ``Series.ptp`` will now ignore missing values by default (`11163`)  .. _whatsnew_0171.deprecations:  Deprecations
^^^  - The ``pandas.io.ga`` module which implements ``google-analytics`` support is deprecated and will be removed in a future version (`11308`)
- Deprecate the ``engine`` keyword in ``.to_csv()``, which will be removed in a future version (`11274`)  .. _whatsnew_0171.performance:  Performance improvements
~~~  - Checking monotonic-ness before sorting on an index (`11080`)
- ``Series.dropna`` performance improvement when its dtype can't contain ``NaN`` (`11159`)
- Release the GIL on most datetime field operations (e.g. ``DatetimeIndex.year``, ``Series.dt.year``), normalization, and conversion to and from ``Period``, ``DatetimeIndex.to_period`` and ``PeriodIndex.to_timestamp`` (`11263`)
- Release the GIL on some rolling algos: ``rolling_median``, ``rolling_mean``, ``rolling_max``, ``rolling_min``, ``rolling_var``, ``rolling_kurt``, ``rolling_skew`` (`11450`)
- Release the GIL when reading and parsing text files in ``read_csv``, ``read_table`` (`11272`)
- Improved performance of ``rolling_median`` (`11450`)
- Improved performance of ``to_excel`` (`11352`)
- Performance bug in repr of ``Categorical`` categories, which was rendering the strings before chopping them for display (`11305`)
- Performance improvement in ``Categorical.remove_unused_categories``, (`11643`).
- Improved performance of ``Series`` constructor with no data and ``DatetimeIndex`` (`11433`)
- Improved performance of ``shift``, ``cumprod``, and ``cumsum`` with groupby (`4095`)  .. _whatsnew_0171.bug_fixes:  Bug fixes
~~~  - ``SparseArray.__iter__()`` now does not cause ``PendingDeprecationWarning`` in Python 3.5 (`11622`)
- Regression from 0.16.2 for output formatting of long floats/nan, restored in (`11302`)
- ``Series.sort_index()`` now correctly handles the ``inplace`` option (`11402`)
- Incorrectly distributed .c file in the build on ``PyPi`` when reading a csv of floats and passing ``na_values=<a scalar>`` would show an exception (`11374`)
- Bug in ``.to_latex()`` output broken when the index has a name (`10660`)
- Bug in ``HDFStore.append`` with strings whose encoded length exceeded the max unencoded length (`11234`)
- Bug in merging ``datetime64[ns, tz]`` dtypes (`11405`)
- Bug in ``HDFStore.select`` when comparing with a numpy scalar in a where clause (`11283`)
- Bug in using ``DataFrame.ix`` with a MultiIndex indexer (`11372`)
- Bug in ``date_range`` with ambiguous endpoints (`11626`)
- Prevent adding new attributes to the accessors ``.str``, ``.dt`` and ``.cat``. Retrieving such  a value was not possible, so error out on setting it. (`10673`)
- Bug in tz-conversions with an ambiguous time and ``.dt`` accessors (`11295`)
- Bug in output formatting when using an index of ambiguous times (`11619`)
- Bug in comparisons of Series vs list-likes (`11339`)
- Bug in ``DataFrame.replace`` with a ``datetime64[ns, tz]`` and a non-compat to_replace (`11326`, `11153`)
- Bug in ``isnull`` where ``numpy.datetime64('NaT')`` in a ``numpy.array`` was not determined to be null(`11206`)
- Bug in list-like indexing with a mixed-integer Index (`11320`)
- Bug in ``pivot_table`` with ``margins=True`` when indexes are of ``Categorical`` dtype (`10993`)
- Bug in ``DataFrame.plot`` cannot use hex strings colors (`10299`)

- Regression in ``DataFrame.drop_duplicates`` from 0.16.2, causing incorrect results on integer values (`11376`)
- Bug in ``pd.eval`` where unary ops in a list error (`11235`)
- Bug in ``squeeze()`` with zero length arrays (`11230`, `8999`)
- Bug in ``describe()`` dropping column names for hierarchical indexes (`11517`)
- Bug in ``DataFrame.pct_change()`` not propagating ``axis`` keyword on ``.fillna`` method (`11150`)
- Bug in ``.to_csv()`` when a mix of integer and string column names are passed as the ``columns`` parameter (`11637`)
- Bug in indexing with a ``range``, (`11652`)
- Bug in inference of numpy scalars and preserving dtype when setting columns (`11638`)
- Bug in ``to_sql`` using unicode column names giving UnicodeEncodeError with (`11431`).
- Fix regression in setting of ``xticks`` in ``plot`` (`11529`).
- Bug in ``holiday.dates`` where observance rules could not be applied to holiday and doc enhancement (`11477`, `11533`)
- Fix plotting issues when having plain ``Axes`` instances instead of ``SubplotAxes`` (`11520`, `11556`).
- Bug in ``DataFrame.to_latex()`` produces an extra rule when ``header=False`` (`7124`)
- Bug in ``df.groupby(...).apply(func)`` when a func returns a ``Series`` containing a new datetimelike column (`11324`)
- Bug in ``pandas.json`` when file to load is big (`11344`)
- Bugs in ``to_excel`` with duplicate columns (`11007`, `10982`, `10970`)
- Fixed a bug that prevented the construction of an empty series of dtype ``datetime64[ns, tz]`` (`11245`).
- Bug in ``read_excel`` with MultiIndex containing integers (`11317`)
- Bug in ``to_excel`` with openpyxl 2.2+ and merging (`11408`)
- Bug in ``DataFrame.to_dict()`` produces a ``np.datetime64`` object instead of ``Timestamp`` when only datetime is present in data (`11327`)
- Bug in ``DataFrame.corr()`` raises exception when computes Kendall correlation for DataFrames with boolean and not boolean columns (`11560`)
- Bug in the link-time error caused by C ``inline`` functions on FreeBSD 10+ (with ``clang``) (`10510`)
- Bug in ``DataFrame.to_csv`` in passing through arguments for formatting ``MultiIndexes``, including ``date_format`` (`7791`)
- Bug in ``DataFrame.join()`` with ``how='right'`` producing a ``TypeError`` (`11519`)
- Bug in ``Series.quantile`` with empty list results has ``Index`` with ``object`` dtype (`11588`)
- Bug in ``pd.merge`` results in empty ``Int64Index`` rather than ``Index(dtype=object)`` when the merge result is empty (`11588`)
- Bug in ``Categorical.remove_unused_categories`` when having ``NaN`` values (`11599`)
- Bug in ``DataFrame.to_sparse()`` loses column names for MultiIndexes (`11600`)
- Bug in ``DataFrame.round()`` with non-unique column index producing a Fatal Python error (`11611`)
- Bug in ``DataFrame.round()`` with ``decimals`` being a non-unique indexed Series producing extra columns (`11618`)  .. _whatsnew_0.17.1.contributors:  Contributors
~~~  .. contributors:: v0.17.0..v0.17.1

{{ header }}  This is a major release from 0.17.1 and includes a small number of API changes, several new features,
users upgrade to this version.  .. warning::  pandas >= 0.18.0 no longer supports compatibility with Python version 2.6  and 3.3 (`7718`, `11273`)  .. warning::  ``numexpr`` version 2.4.4 will now show a warning and not be used as a computation back-end for pandas because of some buggy behavior. This does not affect other versions (>= 2.1 and >= 2.4.6). (`12489`)  Highlights include:  - Moving and expanding window functions are now methods on Series and DataFrame,  similar to ``.groupby``, see `here <whatsnew_0180.enhancements.moments>`.
- Adding support for a ``RangeIndex`` as a specialized form of the ``Int64Index``  for memory savings, see `here <whatsnew_0180.enhancements.rangeindex>`.
- API breaking change to the ``.resample`` method to make it more ``.groupby``  like, see `here <whatsnew_0180.breaking.resample>`.
- Removal of support for positional indexing with floats, which was deprecated  since 0.14.0. This will now raise a ``TypeError``, see `here <whatsnew_0180.float_indexers>`.
- The ``.to_xarray()`` function has been added for compatibility with the  xarray package_, see `here <whatsnew_0180.enhancements.xarray>`.
- The ``read_sas`` function has been enhanced to read ``sas7bdat`` files, see `here <whatsnew_0180.enhancements.sas>`.
- Addition of the `.str.extractall() method <whatsnew_0180.enhancements.extract>`,  and API changes to the `.str.extract() method <whatsnew_0180.enhancements.extract>`  and `.str.cat() method <whatsnew_0180.enhancements.strcat>`.
- ``pd.test()`` top-level nose test runner is available (`4327`).  Check the `API Changes <whatsnew_0180.api_breaking>` and `deprecations <whatsnew_0180.deprecations>` before updating.  .. contents:: What's new in v0.18.0  none  .. _whatsnew_0180.enhancements:  New features
~~~  .. _whatsnew_0180.enhancements.moments:  Window functions are now methods
^^^  Window functions have been refactored to be methods on ``Series/DataFrame`` objects, rather than top-level functions, which are now deprecated. This allows these window-type functions, to have a similar API to that of ``.groupby``. See the full documentation `here <window.overview>` (`11603`, `12373`)  .. ipython:: python  np.random.seed(1234)  df = pd.DataFrame({'A': range(10), 'B': np.random.randn(10)})  df  Previous behavior:  .. code-block:: ipython  In [8]: pd.rolling_mean(df, window=3)  FutureWarning: pd.rolling_mean is deprecated for DataFrame and will be removed in a future version, replace with  DataFrame.rolling(window=3,center=False).mean()  Out[8]:  A  B  0 NaN  NaN  1 NaN  NaN  2  1  0.237722  3  2 -0.023640  4  3  0.133155  5  4 -0.048693  6  5  0.342054  7  6  0.370076  8  7  0.079587  9  8 -0.954504  New behavior:  .. ipython:: python  r = df.rolling(window=3)  These show a descriptive repr  .. ipython:: python  r
with tab-completion of available methods and properties.  .. code-block:: ipython  In [9]: r.<TAB>  # noqa E225, E999  r.A  r.agg  r.apply  r.count  r.exclusions  r.max  r.median  r.name  r.skew  r.sum  r.B  r.aggregate  r.corr  r.cov  r.kurt  r.mean  r.min  r.quantile  r.std  r.var  The methods operate on the ``Rolling`` object itself  .. ipython:: python  r.mean()  They provide getitem accessors  .. ipython:: python  r['A'].mean()  And multiple aggregations  .. ipython:: python  r.agg({'A': ['mean', 'std'],  'B': ['mean', 'std']})  .. _whatsnew_0180.enhancements.rename:  Changes to rename
^^^  ``Series.rename`` and ``NDFrame.rename_axis`` can now take a scalar or list-like

argument for altering the Series or axis *name*, in addition to their old behaviors of altering labels. (`9494`, `11965`)  .. ipython:: python  s = pd.Series(np.random.randn(5))  s.rename('newname')  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 2))  (df.rename_axis("indexname")  .rename_axis("columns_name", axis="columns"))  The new functionality works well in method chains. Previously these methods only accepted functions or dicts mapping a *label* to a new label.
This continues to work as before for function or dict-like values.  .. _whatsnew_0180.enhancements.rangeindex:  Range Index
^^^  A ``RangeIndex`` has been added to the ``Int64Index`` sub-classes to support a memory saving alternative for common use cases. This has a similar implementation to the python ``range`` object (``xrange`` in python 2), in that it only stores the start, stop, and step values for the index. It will transparently interact with the user API, converting to ``Int64Index`` if needed.  This will now be the default constructed index for ``NDFrame`` objects, rather than previous an ``Int64Index``. (`939`, `12070`, `12071`, `12109`, `12888`)  Previous behavior:  .. code-block:: ipython  In [3]: s = pd.Series(range(1000))  In [4]: s.index  Out[4]:  Int64Index([  0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  ...  990, 991, 992, 993, 994, 995, 996, 997, 998, 999], dtype='int64', length=1000)  In [6]: s.index.nbytes  Out[6]: 8000  New behavior:  .. ipython:: python  s = pd.Series(range(1000))  s.index  s.index.nbytes  .. _whatsnew_0180.enhancements.extract:  Changes to str.extract
^^^  The `.str.extract <text.extract>` method takes a regular
expression with capture groups, finds the first match in each subject
string, and returns the contents of the capture groups
(`11386`).  In v0.18.0, the ``expand`` argument was added to
``extract``.  - ``expand=False``: it returns a ``Series``, ``Index``, or ``DataFrame``, depending on the subject and regular expression pattern (same behavior as pre-0.18.0).
- ``expand=True``: it always returns a ``DataFrame``, which is more consistent and less confusing from the perspective of a user.  Currently the default is ``expand=None`` which gives a ``FutureWarning`` and uses ``expand=False``. To avoid this warning, please explicitly specify ``expand``.  .. code-block:: ipython  In [1]: pd.Series(['a1', 'b2', 'c3']).str.extract(r'[ab](\d)', expand=None)  FutureWarning: currently extract(expand=None) means expand=False (return Index/Series/DataFrame)  but in a future version of pandas this will be changed to expand=True (return DataFrame)  Out[1]:  0  1  1  2  2  NaN  dtype: object  Extracting a regular expression with one group returns a Series if
``expand=False``.  .. ipython:: python  pd.Series(['a1', 'b2', 'c3']).str.extract(r'[ab](\d)', expand=False)  It returns a ``DataFrame`` with one column if ``expand=True``.  .. ipython:: python  pd.Series(['a1', 'b2', 'c3']).str.extract(r'[ab](\d)', expand=True)  Calling on an ``Index`` with a regex with exactly one capture group
returns an ``Index`` if ``expand=False``.  .. ipython:: python  s = pd.Series(["a1", "b2", "c3"], ["A11", "B22", "C33"])  s.index  s.index.str.extract("(?P<letter>[a-zA-Z])", expand=False)  It returns a ``DataFrame`` with one column if ``expand=True``.  .. ipython:: python  s.index.str.extract("(?P<letter>[a-zA-Z])", expand=True)  Calling on an ``Index`` with a regex with more than one capture group
raises ``ValueError`` if ``expand=False``.  .. code-block:: python  >>> s.index.str.extract("(?P<letter>[a-zA-Z])([0-9]+)", expand=False)  ValueError: only one regex group is supported with Index  It returns a ``DataFrame`` if ``expand=True``.  .. ipython:: python  s.index.str.extract("(?P<letter>[a-zA-Z])([0-9]+)", expand=True)  In summary, ``extract(expand=True)`` always returns a ``DataFrame``
with a row for every subject string, and a column for every capture

group.  .. _whatsnew_0180.enhancements.extractall:  Addition of str.extractall
^^^  The `.str.extractall <text.extractall>` method was added
(`11386`).  Unlike ``extract``, which returns only the first
match.  .. ipython:: python  s = pd.Series(["a1a2", "b1", "c1"], ["A", "B", "C"])  s  s.str.extract(r"(?P<letter>[ab])(?P<digit>\d)", expand=False)  The ``extractall`` method returns all matches.  .. ipython:: python  s.str.extractall(r"(?P<letter>[ab])(?P<digit>\d)")  .. _whatsnew_0180.enhancements.strcat:  Changes to str.cat
^^^  The method ``.str.cat()`` concatenates the members of a ``Series``. Before, if ``NaN`` values were present in the Series, calling ``.str.cat()`` on it would return ``NaN``, unlike the rest of the ``Series.str.*`` API. This behavior has been amended to ignore ``NaN`` values by default. (`11435`).  A new, friendlier ``ValueError`` is added to protect against the mistake of supplying the ``sep`` as an arg, rather than as a kwarg. (`11334`).  .. ipython:: python  pd.Series(['a', 'b', np.nan, 'c']).str.cat(sep=' ')  pd.Series(['a', 'b', np.nan, 'c']).str.cat(sep=' ', na_rep='?')  .. code-block:: ipython  In [2]: pd.Series(['a', 'b', np.nan, 'c']).str.cat(' ')  ValueError: Did you mean to supply a ``sep`` keyword?  .. _whatsnew_0180.enhancements.rounding:  Datetimelike rounding
^^^  ``DatetimeIndex``, ``Timestamp``, ``TimedeltaIndex``, ``Timedelta`` have gained the ``.round()``, ``.floor()`` and ``.ceil()`` method for datetimelike rounding, flooring and ceiling. (`4314`, `11963`)  Naive datetimes  .. ipython:: python  dr = pd.date_range('20130101 0956.1234', periods=3)  dr  dr.round('s')  # Timestamp scalar  dr[0]  dr[0].round('10s')  Tz-aware are rounded, floored and ceiled in local times  .. ipython:: python  dr = dr.tz_localize('US/Eastern')  dr  dr.round('s')  Timedeltas  .. ipython:: python  t = pd.timedelta_range('1 days 2 hr 13 min 45 us', periods=3, freq='d')  t  t.round('10min')  # Timedelta scalar  t[0]  t[0].round('2h')  In addition, ``.round()``, ``.floor()`` and ``.ceil()`` will be available through the ``.dt`` accessor of ``Series``.  .. ipython:: python  s = pd.Series(dr)  s  s.dt.round('D')  Formatting of integers in FloatIndex
^^^  Integers in ``FloatIndex``, e.g. 1., are now formatted with a decimal point and a ``0`` digit, e.g. ``1.0`` (`11713`)
This change not only affects the display to the console, but also the output of IO methods like ``.to_csv`` or ``.to_html``.  Previous behavior:  .. code-block:: ipython  In [2]: s = pd.Series([1, 2, 3], index=np.arange(3.))  In [3]: s  Out[3]:  0  1  1  2  2  3  dtype: int64  In [4]: s.index  Out[4]: Float64Index([0.0, 1.0, 2.0], dtype='float64')  In [5]: print(s.to_csv(path=None))  0,1  1,2  2,3  New behavior:  .. ipython:: python  s = pd.Series([1, 2, 3], index=np.arange(3.))  s  s.index  print(s.to_csv(path_or_buf=None, header=False))  Changes to dtype assignment behaviors

^^^  When a DataFrame's slice is updated with a new slice of the same dtype, the dtype of the DataFrame will now remain the same. (`10503`)  Previous behavior:  .. code-block:: ipython  In [5]: df = pd.DataFrame({'a': [0, 1, 1],  'b': pd.Series([100, 200, 300], dtype='uint32')})  In [7]: df.dtypes  Out[7]:  a  int64  b  uint32  dtype: object  In [8]: ix = df['a'] == 1  In [9]: df.loc[ix, 'b'] = df.loc[ix, 'b']  In [11]: df.dtypes  Out[11]:  a  int64  b  int64  dtype: object  New behavior:  .. ipython:: python  df = pd.DataFrame({'a': [0, 1, 1],  'b': pd.Series([100, 200, 300], dtype='uint32')})  df.dtypes  ix = df['a'] == 1  df.loc[ix, 'b'] = df.loc[ix, 'b']  df.dtypes  When a DataFrame's integer slice is partially updated with a new slice of floats that could potentially be down-casted to integer without losing precision, the dtype of the slice will be set to float instead of integer.  Previous behavior:  .. code-block:: ipython  In [4]: df = pd.DataFrame(np.array(range(1,10)).reshape(3,3),  columns=list('abc'),  index=[[4,4,8], [8,10,12]])  In [5]: df  Out[5]:  a  b  c  4 8  1  2  3  10  4  5  6  8 12  7  8  9  In [7]: df.ix[4, 'c'] = np.array([0., 1.])  In [8]: df  Out[8]:  a  b  c  4 8  1  2  0  10  4  5  1  8 12  7  8  9  New behavior:  .. ipython:: python  df = pd.DataFrame(np.array(range(1,10)).reshape(3,3),  columns=list('abc'),  index=[[4,4,8], [8,10,12]])  df  df.loc[4, 'c'] = np.array([0., 1.])  df  .. _whatsnew_0180.enhancements.xarray:  Method to_xarray
^^^  In a future version of pandas, we will be deprecating ``Panel`` and other > 2 ndim objects. In order to provide for continuity,
all ``NDFrame`` objects have gained the ``.to_xarray()`` method in order to convert to ``xarray`` objects, which has
a pandas-like interface for > 2 ndim. (`11972`)  See the xarray full-documentation here_.  .. code-block:: ipython  In [1]: p = Panel(np.arange(2*3*4).reshape(2,3,4))  In [2]: p.to_xarray()  Out[2]:  <xarray.DataArray (items: 2, major_axis: 3, minor_axis: 4)>  array([[[ 0,  1,  2,  3],  [ 4,  5,  6,  7],  [ 8,  9, 10, 11]],  [[12, 13, 14, 15],  [16, 17, 18, 19],  [20, 21, 22, 23]]])  Coordinates:  * items  (items) int64 0 1  * major_axis  (major_axis) int64 0 1 2  * minor_axis  (minor_axis) int64 0 1 2 3  Latex representation
^^^  ``DataFrame`` has gained a ``._repr_latex_()`` method in order to allow for conversion to latex in a ipython/jupyter notebook using nbconvert. (`11778`)  Note that this must be activated by setting the option ``pd.display.latex.repr=True`` (`12182`)  For example, if you have a jupyter notebook you plan to convert to latex using nbconvert, place the statement ``pd.display.latex.repr=True`` in the first cell to have the contained DataFrame output also stored as latex.  The options ``display.latex.escape`` and ``display.latex.longtable`` have also been added to the configuration and are used automatically by the ``to_latex``
method. See the `available options docs <options.available>` for more info.  .. _whatsnew_0180.enhancements.sas:  ``pd.read_sas()`` changes
^^^  ``read_sas`` has gained the ability to read SAS7BDAT files, including compressed files.  The files can be read in entirety, or incrementally.  For full details see `here <io.sas>`. (`4052`)  .. _whatsnew_0180.enhancements.other:  Other enhancements
^^^  - Handle truncated floats in SAS xport files (`11713`)
- Added option to hide index in ``Series.to_string`` (`11729`)
- ``read_excel`` now supports s3 urls of the format ``s3://bucketname/filename`` (`11447`)
- add support for ``AWS_S3_HOST`` env variable when reading from s3 (`12198`)
- A simple version of ``Panel.round()`` is now implemented (`11763`)
- For Python 3.x, ``round(DataFrame)``, ``round(Series)``, ``round(Panel)`` will work (`11763`)
- ``sys.getsizeof(obj)`` returns the memory usage of a pandas object, including the  values it contains (`11597`)
- ``Series`` gained an ``is_unique`` attribute (`11946`)

- ``DataFrame.quantile`` and ``Series.quantile`` now accept ``interpolation`` keyword (`10174`).
- Added ``DataFrame.style.format`` for more flexible formatting of cell values (`11692`)
- ``DataFrame.select_dtypes`` now allows the ``np.float16`` type code (`11990`)
- ``pivot_table()`` now accepts most iterables for the ``values`` parameter (`12017`)
- Added Google `BigQuery`` service account authentication support, which enables authentication on remote servers. (`11881`, `12572`). For further details see `here_
- ``HDFStore`` is now iterable: ``for k in store`` is equivalent to ``for k in store.keys()`` (`12221`).
- Add missing methods/fields to ``.dt`` for ``Period`` (`8848`)
- The entire code base has been ``PEP``-ified (`12096`)  .. _whatsnew_0180.api_breaking:  Backwards incompatible API changes
~~~  - the leading white spaces have been removed from the output of ``.to_string(index=False)`` method (`11833`)
- the ``out`` parameter has been removed from the ``Series.round()`` method. (`11763`)
- ``DataFrame.round()`` leaves non-numeric columns unchanged in its return, rather than raises. (`11885`)
- ``DataFrame.head(0)`` and ``DataFrame.tail(0)`` return empty frames, rather than ``self``.  (`11937`)
- ``Series.head(0)`` and ``Series.tail(0)`` return empty series, rather than ``self``.  (`11937`)
- ``to_msgpack`` and ``read_msgpack`` encoding now defaults to ``'utf-8'``. (`12170`)
- the order of keyword arguments to text file parsing functions (``.read_csv()``, ``.read_table()``, ``.read_fwf()``) changed to group related arguments. (`11555`)
- ``NaTType.isoformat`` now returns the string ``'NaT`` to allow the result to  be passed to the constructor of ``Timestamp``. (`12300`)  NaT and Timedelta operations
^^^  ``NaT`` and ``Timedelta`` have expanded arithmetic operations, which are extended to ``Series``
arithmetic where applicable.  Operations defined for ``datetime64[ns]`` or ``timedelta64[ns]``
are now also defined for ``NaT`` (`11564`).  ``NaT`` now supports arithmetic operations with integers and floats.  .. ipython:: python  pd.NaT * 1  pd.NaT * 1.5  pd.NaT / 2  pd.NaT * np.nan  ``NaT`` defines more arithmetic operations with ``datetime64[ns]`` and ``timedelta64[ns]``.  .. ipython:: python  pd.NaT / pd.NaT  pd.Timedelta('1s') / pd.NaT  ``NaT`` may represent either a ``datetime64[ns]`` null or a ``timedelta64[ns]`` null.
Given the ambiguity, it is treated as a ``timedelta64[ns]``, which allows more operations
to succeed.  .. ipython:: python  pd.NaT + pd.NaT  # same as  pd.Timedelta('1s') + pd.Timedelta('1s')  as opposed to  .. code-block:: ipython  In [3]: pd.Timestamp('19900315') + pd.Timestamp('19900315')  TypeError: unsupported operand type(s) for +: 'Timestamp' and 'Timestamp'  However, when wrapped in a ``Series`` whose ``dtype`` is ``datetime64[ns]`` or ``timedelta64[ns]``,
the ``dtype`` information is respected.  .. code-block:: ipython  In [1]: pd.Series([pd.NaT], dtype='<M8[ns]') + pd.Series([pd.NaT], dtype='<M8[ns]')  TypeError: can only operate on a datetimes for subtraction,  but the operator [__add__] was passed  .. ipython:: python  pd.Series([pd.NaT], dtype='<m8[ns]') + pd.Series([pd.NaT], dtype='<m8[ns]')  ``Timedelta`` division by ``floats`` now works.  .. ipython:: python  pd.Timedelta('1s') / 2.0  Subtraction by ``Timedelta`` in a ``Series`` by a ``Timestamp`` works (`11925`)  .. ipython:: python  ser = pd.Series(pd.timedelta_range('1 day', periods=3))  ser  pd.Timestamp('2012-01-01') - ser  ``NaT.isoformat()`` now returns ``'NaT'``. This change allows
``pd.Timestamp`` to rehydrate any timestamp like object from its isoformat
(`12300`).  Changes to msgpack

^^^  Forward incompatible changes in ``msgpack`` writing format were made over 0.17.0 and 0.18.0; older versions of pandas cannot read files packed by newer versions (`12129`, `10527`)  Bugs in ``to_msgpack`` and ``read_msgpack`` introduced in 0.17.0 and fixed in 0.18.0, caused files packed in Python 2 unreadable by Python 3 (`12142`). The following table describes the backward and forward compat of msgpacks.  .. warning::  +---+---+  | Packed with  | Can be unpacked with  |  +===+===+  | pre-0.17 / Python 2  | any  |  +---+---+  | pre-0.17 / Python 3  | any  |  +---+---+  | 0.17 / Python 2  | - ==0.17 / Python 2  |  |  | - >=0.18 / any Python  |  +---+---+  | 0.17 / Python 3  | >=0.18 / any Python  |  +---+---+  | 0.18  | >= 0.18  |  +---+---+  0.18.0 is backward-compatible for reading files packed by older versions, except for files packed with 0.17 in Python 2, in which case only they can only be unpacked in Python 2.  Signature change for .rank
^^^  ``Series.rank`` and ``DataFrame.rank`` now have the same signature (`11759`)  Previous signature  .. code-block:: ipython  In [3]: pd.Series([0,1]).rank(method='average', na_option='keep',  ascending=True, pct=False)  Out[3]:  0  1  1  2  dtype: float64  In [4]: pd.DataFrame([0,1]).rank(axis=0, numeric_only=None,  method='average', na_option='keep',  ascending=True, pct=False)  Out[4]:  0  0  1  1  2  New signature  .. ipython:: python  pd.Series([0,1]).rank(axis=0, method='average', numeric_only=False,  na_option='keep', ascending=True, pct=False)  pd.DataFrame([0,1]).rank(axis=0, method='average', numeric_only=False,  na_option='keep', ascending=True, pct=False)  Bug in QuarterBegin with n=0
^^^  In previous versions, the behavior of the QuarterBegin offset was inconsistent
depending on the date when the ``n`` parameter was 0. (`11406`)  The general semantics of anchored offsets for ``n=0`` is to not move the date
when it is an anchor point (e.g., a quarter start date), and otherwise roll
forward to the next anchor point.  .. ipython:: python  d = pd.Timestamp('2014-02-01')  d  d + pd.offsets.QuarterBegin(n=0, startingMonth=2)  d + pd.offsets.QuarterBegin(n=0, startingMonth=1)  For the ``QuarterBegin`` offset in previous versions, the date would be rolled
*backwards* if date was in the same month as the quarter start date.  .. code-block:: ipython  In [3]: d = pd.Timestamp('2014-02-15')  In [4]: d + pd.offsets.QuarterBegin(n=0, startingMonth=2)  Out[4]: Timestamp('2014-02-01 000')  This behavior has been corrected in version 0.18.0, which is consistent with
other anchored offsets like ``MonthBegin`` and ``YearBegin``.  .. ipython:: python  d = pd.Timestamp('2014-02-15')  d + pd.offsets.QuarterBegin(n=0, startingMonth=2)  .. _whatsnew_0180.breaking.resample:  Resample API
^^^  Like the change in the window functions API `above <whatsnew_0180.enhancements.moments>`, ``.resample(...)`` is changing to have a more groupby-like API. (`11732`, `12702`, `12202`, `12332`, `12334`, `12348`, `12448`).  .. ipython:: python  np.random.seed(1234)  df = pd.DataFrame(np.random.rand(10,4),  columns=list('ABCD'),  index=pd.date_range('2010-01-01 0900',  periods=10, freq='s'))  df  **Previous API**:  You would write a resampling operation that immediately evaluates. If a ``how`` parameter was not provided, it

would default to ``how='mean'``.  .. code-block:: ipython  In [6]: df.resample('2s')  Out[6]:  A  B  C  D  2010-01-01 0900  0.485748  0.447351  0.357096  0.793615  2010-01-01 0902  0.820801  0.794317  0.364034  0.531096  2010-01-01 0904  0.433985  0.314582  0.424104  0.625733  2010-01-01 0906  0.624988  0.609738  0.633165  0.612452  2010-01-01 0908  0.510470  0.534317  0.573201  0.806949  You could also specify a ``how`` directly  .. code-block:: ipython  In [7]: df.resample('2s', how='sum')  Out[7]:  A  B  C  D  2010-01-01 0900  0.971495  0.894701  0.714192  1.587231  2010-01-01 0902  1.641602  1.588635  0.728068  1.062191  2010-01-01 0904  0.867969  0.629165  0.848208  1.251465  2010-01-01 0906  1.249976  1.219477  1.266330  1.224904  2010-01-01 0908  1.020940  1.068634  1.146402  1.613897  **New API**:  Now, you can write ``.resample(..)`` as a 2-stage operation like ``.groupby(...)``, which
yields a ``Resampler``.  .. ipython:: python  r = df.resample('2s')  r  Downsampling
"""  You can then use this object to perform operations.
These are downsampling operations (going from a higher frequency to a lower one).  .. ipython:: python  r.mean()  .. ipython:: python  r.sum()  Furthermore, resample now supports ``getitem`` operations to perform the resample on specific columns.  .. ipython:: python  r[['A','C']].mean()  and ``.aggregate`` type operations.  .. ipython:: python  r.agg({'A' : 'mean', 'B' : 'sum'})  These accessors can of course, be combined  .. ipython:: python  r[['A','B']].agg(['mean','sum'])  Upsampling
"""  .. currentmodule:: pandas.tseries.resample  Upsampling operations take you from a lower frequency to a higher frequency. These are now
performed with the ``Resampler`` objects with `~Resampler.backfill`,
`~Resampler.ffill`, `~Resampler.fillna` and `~Resampler.asfreq` methods.  .. ipython:: python  s = pd.Series(np.arange(5, dtype='int64'),  index=pd.date_range('2010-01-01', periods=5, freq='Q'))  s  Previously  .. code-block:: ipython  In [6]: s.resample('M', fill_method='ffill')  Out[6]:  2010-03-31  0  2010-04-30  0  2010-05-31  0  2010-06-30  1  2010-07-31  1  2010-08-31  1  2010-09-30  2  2010-10-31  2  2010-11-30  2  2010-12-31  3  2011-01-31  3  2011-02-28  3  2011-03-31  4  Freq: M, dtype: int64  New API  .. ipython:: python  s.resample('M').ffill()  .. note::  In the new API, you can either downsample OR upsample. The prior implementation would allow you to pass an aggregator function (like ``mean``) even though you were upsampling, providing a bit of confusion.  Previous API will work but with deprecations

"""  .. warning::  This new API for resample includes some internal changes for the prior-to-0.18.0 API, to work with a deprecation warning in most cases, as the resample operation returns a deferred object. We can intercept operations and just do what the (pre 0.18.0) API did (with a warning). Here is a typical use case:  .. code-block:: ipython  In [4]: r = df.resample('2s')  In [6]: r*10  pandas/tseries/resample.py FutureWarning: .resample() is now a deferred operation  use .resample(...).mean() instead of .resample(...)  Out[6]:  A  B  C  D  2010-01-01 0900  4.857476  4.473507  3.570960  7.936154  2010-01-01 0902  8.208011  7.943173  3.640340  5.310957  2010-01-01 0904  4.339846  3.145823  4.241039  6.257326  2010-01-01 0906  6.249881  6.097384  6.331650  6.124518  2010-01-01 0908  5.104699  5.343172  5.732009  8.069486  However, getting and assignment operations directly on a ``Resampler`` will raise a ``ValueError``:  .. code-block:: ipython  In [7]: r.iloc[0] = 5  ValueError: .resample() is now a deferred operation  use .resample(...).mean() instead of .resample(...)  There is a situation where the new API can not perform all the operations when using original code.  This code is intending to resample every 2s, take the ``mean`` AND then take the ``min`` of those results.  .. code-block:: ipython  In [4]: df.resample('2s').min()  Out[4]:  A  0.433985  B  0.314582  C  0.357096  D  0.531096  dtype: float64  The new API will:  .. ipython:: python  df.resample('2s').min()  The good news is the return dimensions will differ between the new API and the old API, so this should loudly raise  an exception.  To replicate the original operation  .. ipython:: python  df.resample('2s').mean().min()  Changes to eval
^^^  In prior versions, new columns assignments in an ``eval`` expression resulted
in an inplace change to the ``DataFrame``. (`9297`, `8664`, `10486`)  .. ipython:: python  df = pd.DataFrame({'a': np.linspace(0, 10, 5), 'b': range(5)})  df  .. ipython:: python  df.eval('c = a + b', inplace=True)  .. code-block:: ipython  In [12]: df.eval('c = a + b')  FutureWarning: eval expressions containing an assignment currentlydefault to operating inplace.  This will change in a future version of pandas, use inplace=True to avoid this warning.  In [13]: df  Out[13]:  a  b  c  0  0.0  0  0.0  1  2.5  1  3.5  2  5.0  2  7.0  3  7.5  3  10.5  4  10.0  4  14.0  In version 0.18.0, a new ``inplace`` keyword was added to choose whether the
assignment should be done inplace or return a copy.  .. ipython:: python  df  df.eval('d = c - b', inplace=False)  df  df.eval('d = c - b', inplace=True)  df  .. warning::  For backwards compatibility, ``inplace`` defaults to ``True`` if not specified.  This will change in a future version of pandas. If your code depends on an  inplace assignment you should update to explicitly set ``inplace=True``  The ``inplace`` keyword parameter was also added the ``query`` method.  .. ipython:: python  df.query('a > 5')  df.query('a > 5', inplace=True)  df  .. warning::  Note that the default value for ``inplace`` in a ``query``  is ``False``, which is consistent with prior versions.  ``eval`` has also been updated to allow multi-line expressions for multiple
assignments.  These expressions will be evaluated one at a time in order.  Only
assignments are valid for multi-line expressions.  .. ipython:: python  df  df.eval("""  e = d + a  f = e - 22  g = f / 2.0""", inplace=True)  df  .. _whatsnew_0180.api:  Other API changes
^^^

^^^
- ``DataFrame.between_time`` and ``Series.between_time`` now only parse a fixed set of time strings. Parsing of date strings is no longer supported and raises a ``ValueError``. (`11818`)  .. ipython:: python  s = pd.Series(range(10), pd.date_range('2015-01-01', freq='H', periods=10))  s.between_time("7:00am", "9:00am")  This will now raise.  .. code-block:: ipython  In [2]: s.between_time('20150101 0700','20150101 0900')  ValueError: Cannot convert arg ['20150101 0700'] to a time.  - ``.memory_usage()`` now includes values in the index, as does memory_usage in ``.info()`` (`11597`)
- ``DataFrame.to_latex()`` now supports non-ascii encodings (eg ``utf-8``) in Python 2 with the parameter ``encoding`` (`7061`)
- ``pandas.merge()`` and ``DataFrame.merge()`` will show a specific error message when trying to merge with an object that is not of type ``DataFrame`` or a subclass (`12081`)
- ``DataFrame.unstack`` and ``Series.unstack`` now take ``fill_value`` keyword to allow direct replacement of missing values when an unstack results in missing values in the resulting ``DataFrame``. As an added benefit, specifying ``fill_value`` will preserve the data type of the original stacked data.  (`9746`)
- As part of the new API for `window functions <whatsnew_0180.enhancements.moments>` and `resampling <whatsnew_0180.breaking.resample>`, aggregation functions have been clarified, raising more informative error messages on invalid aggregations. (`9052`). A full set of examples are presented in `groupby <groupby.aggregate>`.
- Statistical functions for ``NDFrame`` objects (like ``sum(), mean(), min()``) will now raise if non-numpy-compatible arguments are passed in for ``**kwargs`` (`12301`)
- ``.to_latex`` and ``.to_html`` gain a ``decimal`` parameter like ``.to_csv``; the default is ``'.'`` (`12031`)
- More helpful error message when constructing a ``DataFrame`` with empty data but with indices (`8020`)
- ``.describe()`` will now properly handle bool dtype as a categorical (`6625`)
- More helpful error message with an invalid ``.transform`` with user defined input (`10165`)
- Exponentially weighted functions now allow specifying alpha directly (`10789`) and raise ``ValueError`` if parameters violate ``0 < alpha <= 1`` (`12492`)  .. _whatsnew_0180.deprecations:  Deprecations
^^^  .. _whatsnew_0180.window_deprecations:  - The functions ``pd.rolling_*``, ``pd.expanding_*``, and ``pd.ewm*`` are deprecated and replaced by the corresponding method call. Note that  the new suggested syntax includes all of the arguments (even if default) (`11603`)  .. code-block:: ipython  In [1]: s = pd.Series(range(3))  In [2]: pd.rolling_mean(s,window=2,min_periods=1)  FutureWarning: pd.rolling_mean is deprecated for Series and  will be removed in a future version, replace with  Series.rolling(min_periods=1,window=2,center=False).mean()  Out[2]:  0  0.0  1  0.5  2  1.5  dtype: float64  In [3]: pd.rolling_cov(s, s, window=2)  FutureWarning: pd.rolling_cov is deprecated for Series and  will be removed in a future version, replace with  Series.rolling(window=2).cov(other=<Series>)  Out[3]:  0  NaN  1  0.5  2  0.5  dtype: float64  - The ``freq`` and ``how`` arguments to the ``.rolling``, ``.expanding``, and ``.ewm`` (new) functions are deprecated, and will be removed in a future version. You can simply resample the input prior to creating a window function. (`11603`).  For example, instead of ``s.rolling(window=5,freq='D').max()`` to get the max value on a rolling 5 Day window, one could use ``s.resample('D').mean().rolling(window=5).max()``, which first resamples the data to daily data, then provides a rolling 5 day window.  - ``pd.tseries.frequencies.get_offset_name`` function is deprecated. Use offset's ``.freqstr`` property as alternative (`11192`)

- ``pandas.stats.fama_macbeth`` routines are deprecated and will be removed in a future version (`6077`)
- ``pandas.stats.ols``, ``pandas.stats.plm`` and ``pandas.stats.var`` routines are deprecated and will be removed in a future version (`6077`)
- show a ``FutureWarning`` rather than a ``DeprecationWarning`` on using long-time deprecated syntax in ``HDFStore.select``, where the ``where`` clause is not a string-like (`12027`)  - The ``pandas.options.display.mpl_style`` configuration has been deprecated  and will be removed in a future version of pandas. This functionality  is better handled by matplotlib's `style sheets`_ (`11783`).  .. _style sheets: http://matplotlib.org/users/style_sheets.html  .. _whatsnew_0180.float_indexers:  Removal of deprecated float indexers
^^^  In `4892` indexing with floating point numbers on a non-``Float64Index`` was deprecated (in version 0.14.0).
In 0.18.0, this deprecation warning is removed and these will now raise a ``TypeError``. (`12165`, `12333`)  .. ipython:: python  s = pd.Series([1, 2, 3], index=[4, 5, 6])  s  s2 = pd.Series([1, 2, 3], index=list('abc'))  s2  Previous behavior:  .. code-block:: ipython  # this is label indexing  In [2]: s[5.0]  FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point  Out[2]: 2  # this is positional indexing  In [3]: s.iloc[1.0]  FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point  Out[3]: 2  # this is label indexing  In [4]: s.loc[5.0]  FutureWarning: scalar indexers for index type Int64Index should be integers and not floating point  Out[4]: 2  # .ix would coerce 1.0 to the positional 1, and index  In [5]: s2.ix[1.0] = 10  FutureWarning: scalar indexers for index type Index should be integers and not floating point  In [6]: s2  Out[6]:  a  1  b  10  c  3  dtype: int64  New behavior:  For iloc, getting & setting via a float scalar will always raise.  .. code-block:: ipython  In [3]: s.iloc[2.0]  TypeError: cannot do label indexing on <class 'pandas.indexes.numeric.Int64Index'> with these indexers [2.0] of <type 'float'>  Other indexers will coerce to a like integer for both getting and setting. The ``FutureWarning`` has been dropped for ``.loc``, ``.ix`` and ``[]``.  .. ipython:: python  s[5.0]  s.loc[5.0]  and setting  .. ipython:: python  s_copy = s.copy()  s_copy[5.0] = 10  s_copy  s_copy = s.copy()  s_copy.loc[5.0] = 10  s_copy  Positional setting with ``.ix`` and a float indexer will ADD this value to the index, rather than previously setting the value by position.  .. code-block:: ipython  In [3]: s2.ix[1.0] = 10  In [4]: s2  Out[4]:  a  1  b  2  c  3  1.0  10  dtype: int64  Slicing will also coerce integer-like floats to integers for a non-``Float64Index``.  .. ipython:: python  s.loc[5.0:6]  Note that for floats that are NOT coercible to ints, the label based bounds will be excluded  .. ipython:: python  s.loc[5.1:6]  Float indexing on a ``Float64Index`` is unchanged.  .. ipython:: python  s = pd.Series([1, 2, 3], index=np.arange(3.))  s[1.0]  s[1.0:2.5]  .. _whatsnew_0180.prior_deprecations:  Removal of prior version deprecations/changes
^^^  - Removal of ``rolling_corr_pairwise`` in favor of ``.rolling().corr(pairwise=True)`` (`4950`)
- Removal of ``expanding_corr_pairwise`` in favor of ``.expanding().corr(pairwise=True)`` (`4950`)
- Removal of ``DataMatrix`` module. This was not imported into the pandas namespace in any event (`12111`)
- Removal of ``cols`` keyword in favor of ``subset`` in ``DataFrame.duplicated()`` and ``DataFrame.drop_duplicates()`` (`6680`)
- Removal of the ``read_frame`` and ``frame_query`` (both aliases for ``pd.read_sql``)  and ``write_frame`` (alias of ``to_sql``) functions in the ``pd.io.sql`` namespace,  deprecated since 0.14.0 (`6292`).
- Removal of the ``order`` keyword from ``.factorize()`` (`6930`)  .. _whatsnew_0180.performance:  Performance improvements
~~~  - Improved performance of ``andrews_curves`` (`11534`)

- Improved huge ``DatetimeIndex``, ``PeriodIndex`` and ``TimedeltaIndex``'s ops performance including ``NaT`` (`10277`)
- Improved performance of ``pandas.concat`` (`11958`)
- Improved performance of ``StataReader`` (`11591`)
- Improved performance in construction of ``Categoricals`` with ``Series`` of datetimes containing ``NaT`` (`12077`)  - Improved performance of ISO 8601 date parsing for dates without separators (`11899`), leading zeros (`11871`) and with white space preceding the time zone (`9714`)  .. _whatsnew_0180.bug_fixes:  Bug fixes
~~~  - Bug in ``GroupBy.size`` when data-frame is empty. (`11699`)
- Bug in ``Period.end_time`` when a multiple of time period is requested (`11738`)
- Regression in ``.clip`` with tz-aware datetimes (`11838`)
- Bug in ``date_range`` when the boundaries fell on the frequency (`11804`, `12409`)
- Bug in consistency of passing nested dicts to ``.groupby(...).agg(...)`` (`9052`)
- Accept unicode in ``Timedelta`` constructor (`11995`)
- Bug in value label reading for ``StataReader`` when reading incrementally (`12014`)
- Bug in vectorized ``DateOffset`` when ``n`` parameter is ``0`` (`11370`)
- Compat for numpy 1.11 w.r.t. ``NaT`` comparison changes (`12049`)
- Bug in ``read_csv`` when reading from a ``StringIO`` in threads (`11790`)
- Bug in not treating ``NaT`` as a missing value in datetimelikes when factorizing & with ``Categoricals`` (`12077`)
- Bug in getitem when the values of a ``Series`` were tz-aware (`12089`)
- Bug in ``Series.str.get_dummies`` when one of the variables was 'name' (`12180`)
- Bug in ``pd.concat`` while concatenating tz-aware NaT series. (`11693`, `11755`, `12217`)
- Bug in ``pd.read_stata`` with version <= 108 files (`12232`)
- Bug in ``Series.resample`` using a frequency of ``Nano`` when the index is a ``DatetimeIndex`` and contains non-zero nanosecond parts (`12037`)
- Bug in resampling with ``.nunique`` and a sparse index (`12352`)
- Removed some compiler warnings (`12471`)
- Work around compat issues with ``boto`` in python 3.5 (`11915`)
- Bug in ``NaT`` subtraction from ``Timestamp`` or ``DatetimeIndex`` with timezones (`11718`)
- Bug in subtraction of ``Series`` of a single tz-aware ``Timestamp`` (`12290`)
- Use compat iterators in PY2 to support ``.next()`` (`12299`)
- Bug in ``Timedelta.round`` with negative values (`11690`)
- Bug in ``.loc`` against ``CategoricalIndex`` may result in normal ``Index`` (`11586`)
- Bug in ``DataFrame.info`` when duplicated column names exist (`11761`)
- Bug in ``.copy`` of datetime tz-aware objects (`11794`)
- Bug in ``Series.apply`` and ``Series.map`` where ``timedelta64`` was not boxed (`11349`)
- Bug in ``DataFrame.set_index()`` with tz-aware ``Series`` (`12358`)  - Bug in subclasses of ``DataFrame`` where ``AttributeError`` did not propagate (`11808`)
- Bug groupby on tz-aware data where selection not returning ``Timestamp`` (`11616`)
- Bug in ``pd.read_clipboard`` and ``pd.to_clipboard`` functions not supporting Unicode; upgrade included ``pyperclip`` to v1.5.15 (`9263`)
- Bug in ``DataFrame.query`` containing an assignment (`8664`)  - Bug in ``from_msgpack`` where ``__contains__()`` fails for columns of the unpacked ``DataFrame``, if the ``DataFrame`` has object columns. (`11880`)
- Bug in ``.resample`` on categorical data with ``TimedeltaIndex`` (`12169`)  - Bug in timezone info lost when broadcasting scalar datetime to ``DataFrame`` (`11682`)
- Bug in ``Index`` creation from ``Timestamp`` with mixed tz coerces to UTC (`11488`)
- Bug in ``to_numeric`` where it does not raise if input is more than one dimension (`11776`)
- Bug in parsing timezone offset strings with non-zero minutes (`11708`)
- Bug in ``df.plot`` using incorrect colors for bar plots under matplotlib 1.5+ (`11614`)
- Bug in the ``groupby`` ``plot`` method when using keyword arguments (`11805`).

- Bug in ``DataFrame.duplicated`` and ``drop_duplicates`` causing spurious matches when setting ``keep=False`` (`11864`)
- Bug in ``.loc`` result with duplicated key may have ``Index`` with incorrect dtype (`11497`)
- Bug in ``pd.rolling_median`` where memory allocation failed even with sufficient memory (`11696`)
- Bug in ``DataFrame.style`` with spurious zeros (`12134`)
- Bug in ``DataFrame.style`` with integer columns not starting at 0 (`12125`)
- Bug in ``.style.bar`` may not rendered properly using specific browser (`11678`)
- Bug in rich comparison of ``Timedelta`` with a ``numpy.array`` of ``Timedelta`` that caused an infinite recursion (`11835`)
- Bug in ``DataFrame.round`` dropping column index name (`11986`)
- Bug in ``df.replace`` while replacing value in mixed dtype ``Dataframe`` (`11698`)
- Bug in ``Index`` prevents copying name of passed ``Index``, when a new name is not provided (`11193`)
- Bug in ``read_excel`` failing to read any non-empty sheets when empty sheets exist and ``sheetname=None`` (`11711`)
- Bug in ``read_excel`` failing to raise ``NotImplemented`` error when keywords ``parse_dates`` and ``date_parser`` are provided (`11544`)
- Bug in ``read_sql`` with ``pymysql`` connections failing to return chunked data (`11522`)
- Bug in ``.to_csv`` ignoring formatting parameters ``decimal``, ``na_rep``, ``float_format`` for float indexes (`11553`)
- Bug in ``Int64Index`` and ``Float64Index`` preventing the use of the modulo operator (`9244`)
- Bug in ``MultiIndex.drop`` for not lexsorted MultiIndexes (`12078`)  - Bug in ``DataFrame`` when masking an empty ``DataFrame`` (`11859`)  - Bug in ``.plot`` potentially modifying the ``colors`` input when the number of columns didn't match the number of series provided (`12039`).
- Bug in ``Series.plot`` failing when index has a ``CustomBusinessDay`` frequency (`7222`).
- Bug in ``.to_sql`` for ``datetime.time`` values with sqlite fallback (`8341`)
- Bug in ``read_excel`` failing to read data with one column when ``squeeze=True`` (`12157`)
- Bug in ``read_excel`` failing to read one empty column (`12292`, `9002`)
- Bug in ``.groupby`` where a ``KeyError`` was not raised for a wrong column if there was only one row in the dataframe (`11741`)
- Bug in ``.read_csv`` with dtype specified on empty data producing an error (`12048`)
- Bug in ``.read_csv`` where strings like ``'2E'`` are treated as valid floats (`12237`)
- Bug in building *pandas* with debugging symbols (`12123`)  - Removed ``millisecond`` property of ``DatetimeIndex``. This would always raise a ``ValueError`` (`12019`).
- Bug in ``Series`` constructor with read-only data (`11502`)
- Removed ``pandas._testing.choice()``.  Should use ``np.random.choice()``, instead. (`12386`)
- Bug in ``.loc`` setitem indexer preventing the use of a TZ-aware DatetimeIndex (`12050`)
- Bug in ``.style`` indexes and MultiIndexes not appearing (`11655`)
- Bug in ``to_msgpack`` and ``from_msgpack`` which did not correctly serialize or deserialize ``NaT`` (`12307`).
- Bug in ``.skew`` and ``.kurt`` due to roundoff error for highly similar values (`11974`)
- Bug in ``Timestamp`` constructor where microsecond resolution was lost if HHMMSS were not separated with ':' (`10041`)
- Bug in ``buffer_rd_bytes`` src->buffer could be freed more than once if reading failed, causing a segfault (`12098`)  - Bug in ``crosstab`` where arguments with non-overlapping indexes would return a ``KeyError`` (`10291`)  - Bug in ``DataFrame.apply`` in which reduction was not being prevented for cases in which ``dtype`` was not a numpy dtype (`12244`)
- Bug when initializing categorical series with a scalar value. (`12336`)
- Bug when specifying a UTC ``DatetimeIndex`` by setting ``utc=True`` in ``.to_datetime`` (`11934`)
- Bug when increasing the buffer size of CSV reader in ``read_csv`` (`12494`)

- Bug when setting columns of a ``DataFrame`` with duplicate column names (`12344`)  .. _whatsnew_0.18.0.contributors:  Contributors
~~~  .. contributors:: v0.17.1..v0.18.0

{{ header }}  This is a minor bug-fix release from 0.18.0 and includes a large number of
bug fixes along with several new features, enhancements, and performance improvements.
We recommend that all users upgrade to this version.  Highlights include:  - ``.groupby(...)`` has been enhanced to provide convenient syntax when working with ``.rolling(..)``, ``.expanding(..)`` and ``.resample(..)`` per group, see `here <whatsnew_0181.deferred_ops>`
- ``pd.to_datetime()`` has gained the ability to assemble dates from a ``DataFrame``, see `here <whatsnew_0181.enhancements.assembling>`
- Method chaining improvements, see `here <whatsnew_0181.enhancements.method_chain>`.
- Custom business hour offset, see `here <whatsnew_0181.enhancements.custombusinesshour>`.
- Many bug fixes in the handling of ``sparse``, see `here <whatsnew_0181.sparse>`
- Expanded the Tutorials section_. (`13045`).  .. contents:: What's new in v0.18.1  none  .. _whatsnew_0181.new_features:  New features
~~~  .. _whatsnew_0181.enhancements.custombusinesshour:  Custom business hour
^^^  The ``CustomBusinessHour`` is a mixture of ``BusinessHour`` and ``CustomBusinessDay`` which
allows you to specify arbitrary holidays. For details,
see `Custom Business Hour <timeseries.custombusinesshour>` (`11514`)  .. ipython:: python  from pandas.tseries.offsets import CustomBusinessHour  from pandas.tseries.holiday import USFederalHolidayCalendar  bhour_us = CustomBusinessHour(calendar=USFederalHolidayCalendar())  Friday before MLK Day  .. ipython:: python  import datetime  dt = datetime.datetime(2014, 1, 17, 15)  dt + bhour_us  Tuesday after MLK Day (Monday is skipped because it's a holiday)  .. ipython:: python  dt + bhour_us * 2  .. _whatsnew_0181.deferred_ops:  Method ``.groupby(..)`` syntax with window and resample operations
^^^  ``.groupby(...)`` has been enhanced to provide convenient syntax when working with ``.rolling(..)``, ``.expanding(..)`` and ``.resample(..)`` per group, see (`12486`, `12738`).  You can now use ``.rolling(..)`` and ``.expanding(..)`` as methods on groupbys. These return another deferred object (similar to what ``.rolling()`` and ``.expanding()`` do on ungrouped pandas objects). You can then operate on these ``RollingGroupby`` objects in a similar manner.  Previously you would have to do this to get a rolling window mean per-group:  .. ipython:: python  df = pd.DataFrame({"A": [1] * 20 + [2] * 12 + [3] * 8, "B": np.arange(40)})  df  .. ipython:: python  df.groupby("A").apply(lambda x: x.rolling(4).B.mean())  Now you can do:  .. ipython:: python  df.groupby("A").rolling(4).B.mean()  For ``.resample(..)`` type of operations, previously you would have to:  .. ipython:: python  df = pd.DataFrame(  {  "date": pd.date_range(start="2016-01-01", periods=4, freq="W"),  "group": [1, 1, 2, 2],  "val": [5, 6, 7, 8],  }  ).set_index("date")  df  .. ipython:: python  df.groupby("group").apply(lambda x: x.resample("1D").ffill())  Now you can do:  .. ipython:: python  df.groupby("group").resample("1D").ffill()  .. _whatsnew_0181.enhancements.method_chain:  Method chaining improvements
^^^  The following methods / indexers now accept a ``callable``. It is intended to make
these more useful in method chains, see the `documentation <indexing.callable>`.
(`11485`, `12533`)  - ``.where()`` and ``.mask()``
- ``.loc[]``, ``iloc[]`` and ``.ix[]``
- ``[]`` indexing  Methods ``.where()`` and ``.mask()``
"""  These can accept a callable for the condition and ``other``
arguments.  .. ipython:: python  df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})  df.where(lambda x: x > 4, lambda x: x + 10)  Methods ``.loc[]``, ``.iloc[]``, ``.ix[]``
"""  These can accept a callable, and a tuple of callable as a slicer. The callable

can return a valid boolean indexer or anything which is valid for these indexer's input.  .. ipython:: python  # callable returns bool indexer  df.loc[lambda x: x.A >= 2, lambda x: x.sum() > 10]  # callable returns list of labels  df.loc[lambda x: [1, 2], lambda x: ["A", "B"]]  Indexing with ``[]``
"""  Finally, you can use a callable in ``[]`` indexing of Series, DataFrame and Panel.
The callable must return a valid input for ``[]`` indexing depending on its
class and index type.  .. ipython:: python  df[lambda x: "A"]  Using these methods / indexers, you can chain data selection operations
without using temporary variable.  .. ipython:: python  bb = pd.read_csv("data/baseball.csv", index_col="id")  (bb.groupby(["year", "team"]).sum(numeric_only=True).loc[lambda df: df.r > 100])  .. _whatsnew_0181.partial_string_indexing:  Partial string indexing on ``DatetimeIndex`` when part of a ``MultiIndex``
^^^  Partial string indexing now matches on ``DateTimeIndex`` when part of a ``MultiIndex`` (`10331`)  .. ipython:: python  dft2 = pd.DataFrame(  np.random.randn(20, 1),  columns=["A"],  index=pd.MultiIndex.from_product(  [pd.date_range("20130101", periods=10, freq="12H"), ["a", "b"]]  ),  )  dft2  dft2.loc["2013-01-05"]  On other levels  .. ipython:: python  idx = pd.IndexSlice  dft2 = dft2.swaplevel(0, 1).sort_index()  dft2  dft2.loc[idx[:, "2013-01-05"], :]  .. _whatsnew_0181.enhancements.assembling:  Assembling datetimes
^^^  ``pd.to_datetime()`` has gained the ability to assemble datetimes from a passed in ``DataFrame`` or a dict. (`8158`).  .. ipython:: python  df = pd.DataFrame(  {"year": [2015, 2016], "month": [2, 3], "day": [4, 5], "hour": [2, 3]}  )  df  Assembling using the passed frame.  .. ipython:: python  pd.to_datetime(df)  You can pass only the columns that you need to assemble.  .. ipython:: python  pd.to_datetime(df[["year", "month", "day"]])  .. _whatsnew_0181.other:  Other enhancements
^^^  - ``pd.read_csv()`` now supports ``delim_whitespace=True`` for the Python engine (`12958`)
- ``pd.read_csv()`` now supports opening ZIP files that contains a single CSV, via extension inference or explicit ``compression='zip'`` (`12175`)
- ``pd.read_csv()`` now supports opening files using xz compression, via extension inference or explicit ``compression='xz'`` is specified; ``xz`` compressions is also supported by ``DataFrame.to_csv`` in the same way (`11852`)
- ``pd.read_msgpack()`` now always gives writeable ndarrays even when compression is used (`12359`).
- ``pd.read_msgpack()`` now supports serializing and de-serializing categoricals with msgpack (`12573`)
- ``.to_json()`` now supports ``NDFrames`` that contain categorical and sparse data (`10778`)
- ``interpolate()`` now supports ``method='akima'`` (`7588`).
- ``pd.read_excel()`` now accepts path objects (e.g. ``pathlib.Path``, ``py.path.local``) for the file path, in line with other ``read_*`` functions (`12655`)
- Added ``.weekday_name`` property as a component to ``DatetimeIndex`` and the ``.dt`` accessor. (`11128`)  - ``Index.take`` now handles ``allow_fill`` and ``fill_value`` consistently (`12631`)  .. ipython:: python  idx = pd.Index([1.0, 2.0, 3.0, 4.0], dtype="float")  # default, allow_fill=True, fill_value=None  idx.take([2, -1])  idx.take([2, -1], fill_value=True)  - ``Index`` now supports ``.str.get_dummies()`` which returns ``MultiIndex``, see `Creating Indicator Variables <text.indicator>` (`10008`, `10103`)  .. ipython:: python  idx = pd.Index(["a|b", "a|c", "b|c"])  idx.str.get_dummies("|")  - ``pd.crosstab()`` has gained a ``normalize`` argument for normalizing frequency tables (`12569`). Examples in the updated docs `here <reshaping.crosstabulations>`.
- ``.resample(..).interpolate()`` is now supported (`12925`)
- ``.isin()`` now accepts passed ``sets`` (`12988`)  .. _whatsnew_0181.sparse:  Sparse changes

~~~  These changes conform sparse handling to return the correct types and work to make a smoother experience with indexing.  ``SparseArray.take`` now returns a scalar for scalar input, ``SparseArray`` for others. Furthermore, it handles a negative indexer with the same rule as ``Index`` (`10560`, `12796`)  .. code-block:: python  s = pd.SparseArray([np.nan, np.nan, 1, 2, 3, np.nan, 4, 5, np.nan, 6])  s.take(0)  s.take([1, 2, 3])  - Bug in ``SparseSeries[]`` indexing with ``Ellipsis`` raises ``KeyError`` (`9467`)
- Bug in ``SparseArray[]`` indexing with tuples are not handled properly (`12966`)
- Bug in ``SparseSeries.loc[]`` with list-like input raises ``TypeError`` (`10560`)
- Bug in ``SparseSeries.iloc[]`` with scalar input may raise ``IndexError`` (`10560`)
- Bug in ``SparseSeries.loc[]``, ``.iloc[]`` with ``slice`` returns ``SparseArray``, rather than ``SparseSeries`` (`10560`)
- Bug in ``SparseDataFrame.loc[]``, ``.iloc[]`` may results in dense ``Series``, rather than ``SparseSeries`` (`12787`)
- Bug in ``SparseArray`` addition ignores ``fill_value`` of right hand side (`12910`)
- Bug in ``SparseArray`` mod raises ``AttributeError`` (`12910`)
- Bug in ``SparseArray`` pow calculates ``1 ** np.nan`` as ``np.nan`` which must be 1 (`12910`)
- Bug in ``SparseArray`` comparison output may incorrect result or raise ``ValueError`` (`12971`)
- Bug in ``SparseSeries.__repr__`` raises ``TypeError`` when it is longer than ``max_rows`` (`10560`)
- Bug in ``SparseSeries.shape`` ignores ``fill_value`` (`10452`)
- Bug in ``SparseSeries`` and ``SparseArray`` may have different ``dtype`` from its dense values (`12908`)
- Bug in ``SparseSeries.reindex`` incorrectly handle ``fill_value`` (`12797`)
- Bug in ``SparseArray.to_frame()`` results in ``DataFrame``, rather than ``SparseDataFrame`` (`9850`)
- Bug in ``SparseSeries.value_counts()`` does not count ``fill_value`` (`6749`)
- Bug in ``SparseArray.to_dense()`` does not preserve ``dtype`` (`10648`)
- Bug in ``SparseArray.to_dense()`` incorrectly handle ``fill_value`` (`12797`)
- Bug in ``pd.concat()`` of ``SparseSeries`` results in dense (`10536`)
- Bug in ``pd.concat()`` of ``SparseDataFrame`` incorrectly handle ``fill_value`` (`9765`)
- Bug in ``pd.concat()`` of ``SparseDataFrame`` may raise ``AttributeError`` (`12174`)
- Bug in ``SparseArray.shift()`` may raise ``NameError`` or ``TypeError`` (`12908`)  .. _whatsnew_0181.api:  API changes
~~~  .. _whatsnew_0181.api.groubynth:  Method ``.groupby(..).nth()`` changes
^^^  The index in ``.groupby(..).nth()`` output is now more consistent when the ``as_index`` argument is passed (`11039`):  .. ipython:: python  df = pd.DataFrame({"A": ["a", "b", "a"], "B": [1, 2, 3]})  df  Previous behavior:  .. code-block:: ipython  In [3]: df.groupby('A', as_index=True)['B'].nth(0)  Out[3]:  0  1  1  2  Name: B, dtype: int64  In [4]: df.groupby('A', as_index=False)['B'].nth(0)  Out[4]:  0  1  1  2  Name: B, dtype: int64  New behavior:  .. ipython:: python  df.groupby("A", as_index=True)["B"].nth(0)  df.groupby("A", as_index=False)["B"].nth(0)  Furthermore, previously, a ``.groupby`` would always sort, regardless if ``sort=False`` was passed with ``.nth()``.  .. ipython:: python  np.random.seed(1234)  df = pd.DataFrame(np.random.randn(100, 2), columns=["a", "b"])  df["c"] = np.random.randint(0, 4, 100)  Previous behavior:  .. code-block:: ipython  In [4]: df.groupby('c', sort=True).nth(1)  Out[4]:  a  b  c  0 -0.334077  0.002118  1  0.036142 -2.074978  2 -0.720589  0.887163  3  0.859588 -0.636524  In [5]: df.groupby('c', sort=False).nth(1)  Out[5]:  a  b  c  0 -0.334077  0.002118  1  0.036142 -2.074978  2 -0.720589  0.887163  3  0.859588 -0.636524  New behavior:  .. ipython:: python  df.groupby("c", sort=True).nth(1)  df.groupby("c", sort=False).nth(1)  .. _whatsnew_0181.numpy_compatibility:  NumPy function compatibility

^^^  Compatibility between pandas array-like methods (e.g. ``sum`` and ``take``) and their ``numpy``
counterparts has been greatly increased by augmenting the signatures of the ``pandas`` methods so
as to accept arguments that can be passed in from ``numpy``, even if they are not necessarily
used in the ``pandas`` implementation (`12644`, `12638`, `12687`)  - ``.searchsorted()`` for ``Index`` and ``TimedeltaIndex`` now accept a ``sorter`` argument to maintain compatibility with numpy's ``searchsorted`` function (`12238`)
- Bug in numpy compatibility of ``np.round()`` on a ``Series`` (`12600`)  An example of this signature augmentation is illustrated below:  .. code-block:: python  sp = pd.SparseDataFrame([1, 2, 3])  sp  Previous behaviour:  .. code-block:: ipython  In [2]: np.cumsum(sp, axis=0)  ...  TypeError: cumsum() takes at most 2 arguments (4 given)  New behaviour:  .. code-block:: python  np.cumsum(sp, axis=0)  .. _whatsnew_0181.apply_resample:  Using ``.apply`` on GroupBy resampling
^^^  Using ``apply`` on resampling groupby operations (using a ``pd.TimeGrouper``) now has the same output types as similar ``apply`` calls on other groupby operations. (`11742`).  .. ipython:: python  df = pd.DataFrame(  {"date": pd.to_datetime(["10/10/2000", "11/10/2000"]), "value": [10, 13]}  )  df  Previous behavior:  .. code-block:: ipython  In [1]: df.groupby(pd.TimeGrouper(key='date',  ...:  freq='M')).apply(lambda x: x.value.sum())  Out[1]:  ...  TypeError: cannot concatenate a non-NDFrame object  # Output is a Series  In [2]: df.groupby(pd.TimeGrouper(key='date',  ...:  freq='M')).apply(lambda x: x[['value']].sum())  Out[2]:  date  2000-10-31  value  10  2000-11-30  value  13  dtype: int64  New behavior:  .. code-block:: ipython  # Output is a Series  In [55]: df.groupby(pd.TimeGrouper(key='date',  ...:  freq='M')).apply(lambda x: x.value.sum())  Out[55]:  date  2000-10-31  10  2000-11-30  13  Freq: M, dtype: int64  # Output is a DataFrame  In [56]: df.groupby(pd.TimeGrouper(key='date',  ...:  freq='M')).apply(lambda x: x[['value']].sum())  Out[56]:  value  date  2000-10-31  10  2000-11-30  13  .. _whatsnew_0181.read_csv_exceptions:  Changes in ``read_csv`` exceptions
^^^  In order to standardize the ``read_csv`` API for both the ``c`` and ``python`` engines, both will now raise an
``EmptyDataError``, a subclass of ``ValueError``, in response to empty columns or header (`12493`, `12506`)  Previous behaviour:  .. code-block:: ipython  In [1]: import io  In [2]: df = pd.read_csv(io.StringIO(''), engine='c')  ...  ValueError: No columns to parse from file  In [3]: df = pd.read_csv(io.StringIO(''), engine='python')  ...  StopIteration  New behaviour:  .. code-block:: ipython  In [1]: df = pd.read_csv(io.StringIO(''), engine='c')  ...  pandas.io.common.EmptyDataError: No columns to parse from file  In [2]: df = pd.read_csv(io.StringIO(''), engine='python')  ...  pandas.io.common.EmptyDataError: No columns to parse from file  In addition to this error change, several others have been made as well:  - ``CParserError`` now sub-classes ``ValueError`` instead of just a ``Exception`` (`12551`)
- A ``CParserError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when the ``c`` engine cannot parse a column (`12506`)
- A ``ValueError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when the ``c`` engine encounters a ``NaN`` value in an integer column (`12506`)
- A ``ValueError`` is now raised instead of a generic ``Exception`` in ``read_csv`` when ``true_values`` is specified, and the ``c`` engine encounters an element in a column containing unencodable bytes (`12506`)
- ``pandas.parser.OverflowError`` exception has been removed and has been replaced with Python's built-in ``OverflowError`` exception (`12506`)
- ``pd.read_csv()`` no longer allows a combination of strings and integers for the ``usecols`` parameter (`12678`)  .. _whatsnew_0181.api.to_datetime:  Method ``to_datetime`` error changes

^^^  Bugs in ``pd.to_datetime()`` when passing a ``unit`` with convertible entries and ``errors='coerce'`` or non-convertible with ``errors='ignore'``. Furthermore, an ``OutOfBoundsDateime`` exception will be raised when an out-of-range value is encountered for that unit when ``errors='raise'``. (`11758`, `13052`, `13059`)  Previous behaviour:  .. code-block:: ipython  In [27]: pd.to_datetime(1420043460, unit='s', errors='coerce')  Out[27]: NaT  In [28]: pd.to_datetime(111, unit='D', errors='ignore')  OverflowError: Python int too large to convert to C long  In [29]: pd.to_datetime(111, unit='D', errors='raise')  OverflowError: Python int too large to convert to C long  New behaviour:  .. code-block:: ipython  In [2]: pd.to_datetime(1420043460, unit='s', errors='coerce')  Out[2]: Timestamp('2014-12-31 1600')  In [3]: pd.to_datetime(111, unit='D', errors='ignore')  Out[3]: 111  In [4]: pd.to_datetime(111, unit='D', errors='raise')  OutOfBoundsDatetime: cannot convert input with unit 'D'  .. _whatsnew_0181.api.other:  Other API changes
^^^  - ``.swaplevel()`` for ``Series``, ``DataFrame``, ``Panel``, and ``MultiIndex`` now features defaults for its first two parameters ``i`` and ``j`` that swap the two innermost levels of the index. (`12934`)
- ``.searchsorted()`` for ``Index`` and ``TimedeltaIndex`` now accept a ``sorter`` argument to maintain compatibility with numpy's ``searchsorted`` function (`12238`)
- ``Period`` and ``PeriodIndex`` now raises ``IncompatibleFrequency`` error which inherits ``ValueError`` rather than raw ``ValueError`` (`12615`)
- ``Series.apply`` for category dtype now applies the passed function to each of the ``.categories`` (and not the ``.codes``), and returns a ``category`` dtype if possible (`12473`)
- ``read_csv`` will now raise a ``TypeError`` if ``parse_dates`` is neither a boolean, list, or dictionary (matches the doc-string) (`5636`)
- The default for ``.query()/.eval()`` is now ``engine=None``, which will use ``numexpr`` if it's installed; otherwise it will fallback to the ``python`` engine. This mimics the pre-0.18.1 behavior if ``numexpr`` is installed (and which, previously, if numexpr was not installed, ``.query()/.eval()`` would raise). (`12749`)
- ``pd.show_versions()`` now includes ``pandas_datareader`` version (`12740`)
- Provide a proper ``__name__`` and ``__qualname__`` attributes for generic functions (`12021`)
- ``pd.concat(ignore_index=True)`` now uses ``RangeIndex`` as default (`12695`)
- ``pd.merge()`` and ``DataFrame.join()`` will show a ``UserWarning`` when merging/joining a single- with a multi-leveled dataframe (`9455`, `12219`)
- Compat with ``scipy`` > 0.17 for deprecated ``piecewise_polynomial`` interpolation method; support for the replacement ``from_derivatives`` method (`12887`)  .. _whatsnew_0181.deprecations:  Deprecations
^^^  - The method name ``Index.sym_diff()`` is deprecated and can be replaced by ``Index.symmetric_difference()`` (`12591`)
- The method name ``Categorical.sort()`` is deprecated in favor of ``Categorical.sort_values()`` (`12882`)  .. _whatsnew_0181.performance:  Performance improvements
~~~  - Improved speed of SAS reader (`12656`, `12961`)
- Performance improvements in ``.groupby(..).cumcount()`` (`11039`)
- Improved memory usage in ``pd.read_csv()`` when using ``skiprows=an_integer`` (`13005`)
- Improved performance of ``DataFrame.to_sql`` when checking case sensitivity for tables. Now only checks if table has been created correctly when table name is not lower case. (`12876`)
- Improved performance of ``Period`` construction and time series plotting (`12903`, `11831`).
- Improved performance of ``.str.encode()`` and ``.str.decode()`` methods (`13008`)
- Improved performance of ``to_numeric`` if input is numeric dtype (`12777`)

- Improved performance of sparse arithmetic with ``IntIndex`` (`13036`)  .. _whatsnew_0181.bug_fixes:  Bug fixes
~~~
- ``usecols`` parameter in ``pd.read_csv`` is now respected even when the lines of a CSV file are not even (`12203`)
- Bug in ``groupby.transform(..)`` when ``axis=1`` is specified with a non-monotonic ordered index (`12713`)
- Bug in ``Period`` and ``PeriodIndex`` creation raises ``KeyError`` if ``freq="Minute"`` is specified. Note that "Minute" freq is deprecated in v0.17.0, and recommended to use ``freq="T"`` instead (`11854`)
- Bug in ``.resample(...).count()`` with a ``PeriodIndex`` always raising a ``TypeError`` (`12774`)
- Bug in ``.resample(...)`` with a ``PeriodIndex`` casting to a ``DatetimeIndex`` when empty (`12868`)
- Bug in ``.resample(...)`` with a ``PeriodIndex`` when resampling to an existing frequency (`12770`)
- Bug in printing data which contains ``Period`` with different ``freq`` raises ``ValueError`` (`12615`)
- Bug in ``Series`` construction with ``Categorical`` and ``dtype='category'`` is specified (`12574`)
- Bugs in concatenation with a coercible dtype was too aggressive, resulting in different dtypes in output formatting when an object was longer than ``display.max_rows`` (`12411`, `12045`, `11594`, `10571`, `12211`)
- Bug in ``float_format`` option with option not being validated as a callable. (`12706`)
- Bug in ``GroupBy.filter`` when ``dropna=False`` and no groups fulfilled the criteria (`12768`)
- Bug in ``__name__`` of ``.cum*`` functions (`12021`)
- Bug in ``.astype()`` of a ``Float64Inde/Int64Index`` to an ``Int64Index`` (`12881`)
- Bug in round tripping an integer based index in ``.to_json()/.read_json()`` when ``orient='index'`` (the default) (`12866`)
- Bug in plotting ``Categorical`` dtypes cause error when attempting stacked bar plot (`13019`)
- Compat with >= ``numpy`` 1.11 for ``NaT`` comparisons (`12969`)
- Bug in ``.drop()`` with a non-unique ``MultiIndex``. (`12701`)
- Bug in ``.concat`` of datetime tz-aware and naive DataFrames (`12467`)
- Bug in correctly raising a ``ValueError`` in ``.resample(..).fillna(..)`` when passing a non-string (`12952`)
- Bug fixes in various encoding and header processing issues in ``pd.read_sas()`` (`12659`, `12654`, `12647`, `12809`)
- Bug in ``pd.crosstab()`` where would silently ignore ``aggfunc`` if ``values=None`` (`12569`).
- Potential segfault in ``DataFrame.to_json`` when serialising ``datetime.time`` (`11473`).
- Potential segfault in ``DataFrame.to_json`` when attempting to serialise 0d array (`11299`).
- Segfault in ``to_json`` when attempting to serialise a ``DataFrame`` or ``Series`` with non-ndarray values; now supports serialization of ``category``, ``sparse``, and ``datetime64[ns, tz]`` dtypes (`10778`).
- Bug in ``DataFrame.to_json`` with unsupported dtype not passed to default handler (`12554`).
- Bug in ``.align`` not returning the sub-class (`12983`)
- Bug in aligning a ``Series`` with a ``DataFrame`` (`13037`)
- Bug in ``ABCPanel`` in which ``Panel4D`` was not being considered as a valid instance of this generic type (`12810`)  - Bug in consistency of ``.name`` on ``.groupby(..).apply(..)`` cases (`12363`)  - Bug in ``Timestamp.__repr__`` that caused ``pprint`` to fail in nested structures (`12622`)
- Bug in ``Timedelta.min`` and ``Timedelta.max``, the properties now report the true minimum/maximum ``timedeltas`` as recognized by pandas. See the `documentation <timedeltas.limitations>`. (`12727`)
- Bug in ``.quantile()`` with interpolation may coerce to ``float`` unexpectedly (`12772`)
- Bug in ``.quantile()`` with empty ``Series`` may return scalar rather than empty ``Series`` (`12772`)  - Bug in ``.loc`` with out-of-bounds in a large indexer would raise ``IndexError`` rather than ``KeyError`` (`12527`)

- Bug in resampling when using a ``TimedeltaIndex`` and ``.asfreq()``, would previously not include the final fencepost (`12926`)  - Bug in equality testing with a ``Categorical`` in a ``DataFrame`` (`12564`)
- Bug in ``GroupBy.first()``, ``.last()`` returns incorrect row when ``TimeGrouper`` is used (`7453`)  - Bug in ``pd.read_csv()`` with the ``c`` engine when specifying ``skiprows`` with newlines in quoted items (`10911`, `12775`)
- Bug in ``DataFrame`` timezone lost when assigning tz-aware datetime ``Series`` with alignment (`12981`)  - Bug in ``.value_counts()`` when ``normalize=True`` and ``dropna=True`` where nulls still contributed to the normalized count (`12558`)
- Bug in ``Series.value_counts()`` loses name if its dtype is ``category`` (`12835`)
- Bug in ``Series.value_counts()`` loses timezone info (`12835`)
- Bug in ``Series.value_counts(normalize=True)`` with ``Categorical`` raises ``UnboundLocalError`` (`12835`)
- Bug in ``Panel.fillna()`` ignoring ``inplace=True`` (`12633`)
- Bug in ``pd.read_csv()`` when specifying ``names``, ``usecols``, and ``parse_dates`` simultaneously with the ``c`` engine (`9755`)
- Bug in ``pd.read_csv()`` when specifying ``delim_whitespace=True`` and ``lineterminator`` simultaneously with the ``c`` engine (`12912`)
- Bug in ``Series.rename``, ``DataFrame.rename`` and ``DataFrame.rename_axis`` not treating ``Series`` as mappings to relabel (`12623`).
- Clean in ``.rolling.min`` and ``.rolling.max`` to enhance dtype handling (`12373`)
- Bug in ``groupby`` where complex types are coerced to float (`12902`)
- Bug in ``Series.map`` raises ``TypeError`` if its dtype is ``category`` or tz-aware ``datetime`` (`12473`)  - Bugs on 32bit platforms for some test comparisons (`12972`)
- Bug in index coercion when falling back from ``RangeIndex`` construction (`12893`)
- Better error message in window functions when invalid argument (e.g. a float window) is passed (`12669`)  - Bug in slicing subclassed ``DataFrame`` defined to return subclassed ``Series`` may return normal ``Series`` (`11559`)  - Bug in ``.str`` accessor methods may raise ``ValueError`` if input has ``name`` and the result is ``DataFrame`` or ``MultiIndex`` (`12617`)
- Bug in ``DataFrame.last_valid_index()`` and ``DataFrame.first_valid_index()`` on empty frames (`12800`)  - Bug in ``CategoricalIndex.get_loc`` returns different result from regular ``Index`` (`12531`)
- Bug in ``PeriodIndex.resample`` where name not propagated (`12769`)  - Bug in ``date_range`` ``closed`` keyword and timezones (`12684`).  - Bug in ``pd.concat`` raises ``AttributeError`` when input data contains tz-aware datetime and timedelta (`12620`)
- Bug in ``pd.concat`` did not handle empty ``Series`` properly (`11082`)  - Bug in ``.plot.bar`` alignment when ``width`` is specified with ``int`` (`12979`)  - Bug in ``fill_value`` is ignored if the argument to a binary operator is a constant (`12723`)  - Bug in ``pd.read_html()`` when using bs4 flavor and parsing table with a header and only one column (`9178`)  - Bug in ``.pivot_table`` when ``margins=True`` and ``dropna=True`` where nulls still contributed to margin count (`12577`)
- Bug in ``.pivot_table`` when ``dropna=False`` where table index/column names disappear (`12133`)
- Bug in ``pd.crosstab()`` when ``margins=True`` and ``dropna=False`` which raised (`12642`)  - Bug in ``Series.name`` when ``name`` attribute can be a hashable type (`12610`)  - Bug in ``.describe()`` resets categorical columns information (`11558`)
- Bug where ``loffset`` argument was not applied when calling ``resample().count()`` on a timeseries (`12725`)
- ``pd.read_excel()`` now accepts column names associated with keyword argument ``names`` (`12870`)
- Bug in ``pd.to_numeric()`` with ``Index`` returns ``np.ndarray``, rather than ``Index`` (`12777`)
- Bug in ``pd.to_numeric()`` with datetime-like may raise ``TypeError`` (`12777`)
- Bug in ``pd.to_numeric()`` with scalar raises ``ValueError`` (`12777`)  .. _whatsnew_0.18.1.contributors:  Contributors

~~~  .. contributors:: v0.18.0..v0.18.1

{{ header }}  This is a major release from 0.18.1 and includes number of API changes, several new features,
users upgrade to this version.  Highlights include:  - `merge_asof` for asof-style time-series joining, see `here <whatsnew_0190.enhancements.asof_merge>`
- ``.rolling()`` is now time-series aware, see `here <whatsnew_0190.enhancements.rolling_ts>`
- `read_csv` now supports parsing ``Categorical`` data, see `here <whatsnew_0190.enhancements.read_csv_categorical>`
- A function `union_categorical` has been added for combining categoricals, see `here <whatsnew_0190.enhancements.union_categoricals>`
- ``PeriodIndex`` now has its own ``period`` dtype, and changed to be more consistent with other ``Index`` classes. See `here <whatsnew_0190.api.period>`
- Sparse data structures gained enhanced support of ``int`` and ``bool`` dtypes, see `here <whatsnew_0190.sparse>`
- Comparison operations with ``Series`` no longer ignores the index, see `here <whatsnew_0190.api.series_ops>` for an overview of the API changes.
- Introduction of a pandas development API for utility functions, see `here <whatsnew_0190.dev_api>`.
- Deprecation of `Panel4D`` and ``PanelND``. We recommend to represent these types of n-dimensional data with the `xarray package_.
- Removal of the previously deprecated modules ``pandas.io.data``, ``pandas.io.wb``, ``pandas.tools.rplot``.  .. warning::  pandas >= 0.19.0 will no longer silence numpy ufunc warnings upon import, see `here <whatsnew_0190.errstate>`.  .. contents:: What's new in v0.19.0  none  .. _whatsnew_0190.new_features:  New features
~~~  .. _whatsnew_0190.enhancements.asof_merge:  Function ``merge_asof`` for asof-style time-series joining
^^^  A long-time requested feature has been added through the `merge_asof` function, to
support asof style joining of time-series (`1870`, `13695`, `13709`, `13902`). Full documentation is
`here <merging.merge_asof>`.  The `merge_asof` performs an asof merge, which is similar to a left-join
except that we match on nearest key rather than equal keys.  .. ipython:: python  left = pd.DataFrame({"a": [1, 5, 10], "left_val": ["a", "b", "c"]})  right = pd.DataFrame({"a": [1, 2, 3, 6, 7], "right_val": [1, 2, 3, 6, 7]})  left  right  We typically want to match exactly when possible, and use the most
recent value otherwise.  .. ipython:: python  pd.merge_asof(left, right, on="a")  We can also match rows ONLY with prior data, and not an exact match.  .. ipython:: python  pd.merge_asof(left, right, on="a", allow_exact_matches=False)  In a typical time-series example, we have ``trades`` and ``quotes`` and we want to ``asof-join`` them.
This also illustrates using the ``by`` parameter to group data before merging.  .. ipython:: python  trades = pd.DataFrame(  {  "time": pd.to_datetime(  [  "20160525 1300.023",  "20160525 1300.038",  "20160525 1300.048",  "20160525 1300.048",  "20160525 1300.048",  ]  ),  "ticker": ["MSFT", "MSFT", "GOOG", "GOOG", "AAPL"],  "price": [51.95, 51.95, 720.77, 720.92, 98.00],  "quantity": [75, 155, 100, 100, 100],  },  columns=["time", "ticker", "price", "quantity"],  )  quotes = pd.DataFrame(  {  "time": pd.to_datetime(  [  "20160525 1300.023",  "20160525 1300.023",  "20160525 1300.030",  "20160525 1300.041",  "20160525 1300.048",  "20160525 1300.049",  "20160525 1300.072",  "20160525 1300.075",  ]  ),  "ticker": ["GOOG", "MSFT", "MSFT", "MSFT", "GOOG", "AAPL", "GOOG", "MSFT"],  "bid": [720.50, 51.95, 51.97, 51.99, 720.50, 97.99, 720.50, 52.01],  "ask": [720.93, 51.96, 51.98, 52.00, 720.93, 98.01, 720.88, 52.03],  },  columns=["time", "ticker", "bid", "ask"],  )  .. ipython:: python  trades  quotes  An asof merge joins on the ``on``, typically a datetimelike field, which is ordered, and
in this case we are using a grouper in the ``by`` field. This is like a left-outer join, except

that forward filling happens automatically taking the most recent non-NaN value.  .. ipython:: python  pd.merge_asof(trades, quotes, on="time", by="ticker")  This returns a merged DataFrame with the entries in the same order as the original left
passed DataFrame (``trades`` in this case), with the fields of the ``quotes`` merged.  .. _whatsnew_0190.enhancements.rolling_ts:  Method ``.rolling()`` is now time-series aware
^^^  ``.rolling()`` objects are now time-series aware and can accept a time-series offset (or convertible) for the ``window`` argument (`13327`, `12995`).
See the full documentation `here <window.generic>`.  .. ipython:: python  dft = pd.DataFrame(  {"B": [0, 1, 2, np.nan, 4]},  index=pd.date_range("20130101 0900", periods=5, freq="s"),  )  dft  This is a regular frequency index. Using an integer window parameter works to roll along the window frequency.  .. ipython:: python  dft.rolling(2).sum()  dft.rolling(2, min_periods=1).sum()  Specifying an offset allows a more intuitive specification of the rolling frequency.  .. ipython:: python  dft.rolling("2s").sum()  Using a non-regular, but still monotonic index, rolling with an integer window does not impart any special calculation.  .. ipython:: python  dft = pd.DataFrame(  {"B": [0, 1, 2, np.nan, 4]},  index=pd.Index(  [  pd.Timestamp("20130101 0900"),  pd.Timestamp("20130101 0902"),  pd.Timestamp("20130101 0903"),  pd.Timestamp("20130101 0905"),  pd.Timestamp("20130101 0906"),  ],  name="foo",  ),  )  dft  dft.rolling(2).sum()  Using the time-specification generates variable windows for this sparse data.  .. ipython:: python  dft.rolling("2s").sum()  Furthermore, we now allow an optional ``on`` parameter to specify a column (rather than the
default of the index) in a DataFrame.  .. ipython:: python  dft = dft.reset_index()  dft  dft.rolling("2s", on="foo").sum()  .. _whatsnew_0190.enhancements.read_csv_dupe_col_names_support:  Method ``read_csv`` has improved support for duplicate column names
^^^  .. ipython:: python  from io import StringIO  `Duplicate column names <io.dupe_names>` are now supported in `read_csv` whether
they are in the file or passed in as the ``names`` parameter (`7160`, `9424`)  .. ipython:: python  data = "0,1,2\n3,4,5"  names = ["a", "b", "a"]  **Previous behavior**:  .. code-block:: ipython  In [2]: pd.read_csv(StringIO(data), names=names)  Out[2]:  a  b  a  0  2  1  2  1  5  4  5  The first ``a`` column contained the same data as the second ``a`` column, when it should have
contained the values ``[0, 3]``.  **New behavior**:  .. ipython:: python  pd.read_csv(StringIO(data), names=names)  .. _whatsnew_0190.enhancements.read_csv_categorical:  Method ``read_csv`` supports parsing ``Categorical`` directly
^^^  The `read_csv` function now supports parsing a ``Categorical`` column when
specified as a dtype (`10153`).  Depending on the structure of the data,
this can result in a faster parse time and lower memory usage compared to

converting to ``Categorical`` after parsing.  See the io `docs here <io.categorical>`.  .. ipython:: python  data = """  col1,col2,col3  a,b,1  a,b,2  c,d,3  """  pd.read_csv(StringIO(data))  pd.read_csv(StringIO(data)).dtypes  pd.read_csv(StringIO(data), dtype="category").dtypes  Individual columns can be parsed as a ``Categorical`` using a dict specification  .. ipython:: python  pd.read_csv(StringIO(data), dtype={"col1": "category"}).dtypes  .. note::  The resulting categories will always be parsed as strings (object dtype).  If the categories are numeric they can be converted using the  `to_numeric` function, or as appropriate, another converter  such as `to_datetime`.  .. ipython:: python  df = pd.read_csv(StringIO(data), dtype="category")  df.dtypes  df["col3"]  new_categories = pd.to_numeric(df["col3"].cat.categories)  df["col3"] = df["col3"].cat.rename_categories(new_categories)  df["col3"]  .. _whatsnew_0190.enhancements.union_categoricals:  Categorical concatenation
^^^  - A function `union_categoricals` has been added for combining categoricals, see `Unioning Categoricals<categorical.union>` (`13361`, `13763`, `13846`, `14173`)  .. ipython:: python  from pandas.api.types import union_categoricals  a = pd.Categorical(["b", "c"])  b = pd.Categorical(["a", "b"])  union_categoricals([a, b])  - ``concat`` and ``append`` now can concat ``category`` dtypes with different ``categories`` as ``object`` dtype (`13524`)  .. ipython:: python  s1 = pd.Series(["a", "b"], dtype="category")  s2 = pd.Series(["b", "c"], dtype="category")  **Previous behavior**:  .. code-block:: ipython  In [1]: pd.concat([s1, s2])  ValueError: incompatible categories in categorical concat  **New behavior**:  .. ipython:: python  pd.concat([s1, s2])  .. _whatsnew_0190.enhancements.semi_month_offsets:  Semi-month offsets
^^^  pandas has gained new frequency offsets, ``SemiMonthEnd`` ('SM') and ``SemiMonthBegin`` ('SMS').
These provide date offsets anchored (by default) to the 15th and end of month, and 15th and 1st of month respectively.
(`1543`)  .. ipython:: python  from pandas.tseries.offsets import SemiMonthEnd, SemiMonthBegin  **SemiMonthEnd**:  .. ipython:: python  pd.Timestamp("2016-01-01") + SemiMonthEnd()  pd.date_range("2015-01-01", freq="SM", periods=4)  **SemiMonthBegin**:  .. ipython:: python  pd.Timestamp("2016-01-01") + SemiMonthBegin()  pd.date_range("2015-01-01", freq="SMS", periods=4)  Using the anchoring suffix, you can also specify the day of month to use instead of the 15th.  .. ipython:: python  pd.date_range("2015-01-01", freq="SMS-16", periods=4)  pd.date_range("2015-01-01", freq="SM-14", periods=4)  .. _whatsnew_0190.enhancements.index:  New Index methods
^^^  The following methods and options are added to ``Index``, to be more consistent with the ``Series`` and ``DataFrame`` API.  ``Index`` now supports the ``.where()`` function for same shape indexing (`13170`)  .. ipython:: python  idx = pd.Index(["a", "b", "c"])  idx.where([True, False, True])  ``Index`` now supports ``.dropna()`` to exclude missing values (`6194`)  .. ipython:: python  idx = pd.Index([1, 2, np.nan, 4])  idx.dropna()  For ``MultiIndex``, values are dropped if any level is missing by default. Specifying

``how='all'`` only drops values where all levels are missing.  .. ipython:: python  midx = pd.MultiIndex.from_arrays([[1, 2, np.nan, 4], [1, 2, np.nan, np.nan]])  midx  midx.dropna()  midx.dropna(how="all")  ``Index`` now supports ``.str.extractall()`` which returns a ``DataFrame``, see the `docs here <text.extractall>` (`10008`, `13156`)  .. ipython:: python  idx = pd.Index(["a1a2", "b1", "c1"])  idx.str.extractall(r"[ab](?P<digit>\d)")  ``Index.astype()`` now accepts an optional boolean argument ``copy``, which allows optional copying if the requirements on dtype are satisfied (`13209`)  .. _whatsnew_0190.gbq:  Google BigQuery enhancements
^^^  - The read_gbq` method has gained the ``dialect`` argument to allow users to specify whether to use BigQuery's legacy SQL or BigQuery's standard SQL. See the `docs_ for more details (`13615`).
- The `~DataFrame.to_gbq` method now allows the DataFrame column order to differ from the destination table schema (`11359`).  .. _whatsnew_0190.errstate:  Fine-grained NumPy errstate
^^^  Previous versions of pandas would permanently silence numpy's ufunc error handling when ``pandas`` was imported. pandas did this in order to silence the warnings that would arise from using numpy ufuncs on missing data, which are usually represented as ``NaN`` s. Unfortunately, this silenced legitimate warnings arising in non-pandas code in the application. Starting with 0.19.0, pandas will use the ``numpy.errstate`` context manager to silence these warnings in a more fine-grained manner, only around where these operations are actually used in the pandas code base. (`13109`, `13145`)  After upgrading pandas, you may see *new* `RuntimeWarnings`` being issued from your code. These are likely legitimate, and the underlying cause likely existed in the code when using previous versions of pandas that simply silenced the warning. Use `numpy.errstate_ around the source of the ``RuntimeWarning`` to control how these conditions are handled.  .. _whatsnew_0190.get_dummies_dtypes:  Method ``get_dummies`` now returns integer dtypes
^^^  The ``pd.get_dummies`` function now returns dummy-encoded columns as small integers, rather than floats (`8725`). This should provide an improved memory footprint.  **Previous behavior**:  .. code-block:: ipython  In [1]: pd.get_dummies(['a', 'b', 'a', 'c']).dtypes  Out[1]:  a  float64  b  float64  c  float64  dtype: object  **New behavior**:  .. ipython:: python  pd.get_dummies(["a", "b", "a", "c"]).dtypes  .. _whatsnew_0190.enhancements.to_numeric_downcast:  Downcast values to smallest possible dtype in ``to_numeric``
^^^  ``pd.to_numeric()`` now accepts a ``downcast`` parameter, which will downcast the data if possible to smallest specified numerical dtype (`13352`)  .. ipython:: python  s = ["1", 2, 3]  pd.to_numeric(s, downcast="unsigned")  pd.to_numeric(s, downcast="integer")  .. _whatsnew_0190.dev_api:  pandas development API
^^^  As part of making pandas API more uniform and accessible in the future, we have created a standard
sub-package of pandas, ``pandas.api`` to hold public API's. We are starting by exposing type
introspection functions in ``pandas.api.types``. More sub-packages and officially sanctioned API's
will be published in future versions of pandas (`13147`, `13634`)  The following are now part of this API:  .. ipython:: python  import pprint  from pandas.api import types  funcs = [f for f in dir(types) if not f.startswith("_")]  pprint.pprint(funcs)  .. note::  Calling these functions from the internal module ``pandas.core.common`` will now show a ``DeprecationWarning`` (`13990`)  .. _whatsnew_0190.enhancements.other:  Other enhancements

^^^  - ``Timestamp`` can now accept positional and keyword parameters similar to `datetime.datetime` (`10758`, `11630`)  .. ipython:: python  pd.Timestamp(2012, 1, 1)  pd.Timestamp(year=2012, month=1, day=1, hour=8, minute=30)  - The ``.resample()`` function now accepts a ``on=`` or ``level=`` parameter for resampling on a datetimelike column or ``MultiIndex`` level (`13500`)  .. ipython:: python  df = pd.DataFrame(  {"date": pd.date_range("2015-01-01", freq="W", periods=5), "a": np.arange(5)},  index=pd.MultiIndex.from_arrays(  [[1, 2, 3, 4, 5], pd.date_range("2015-01-01", freq="W", periods=5)],  names=["v", "d"],  ),  )  df  df.resample("M", on="date")[["a"]].sum()  df.resample("M", level="d")[["a"]].sum()  - The `.get_credentials()`` method of ``GbqConnector`` can now first try to fetch `the application default credentials_. See the docs for more details (`13577`).
- The ``.tz_localize()`` method of ``DatetimeIndex`` and ``Timestamp`` has gained the ``errors`` keyword, so you can potentially coerce nonexistent timestamps to ``NaT``. The default behavior remains to raising a ``NonExistentTimeError`` (`13057`)
- ``.to_hdf/read_hdf()`` now accept path objects (e.g. ``pathlib.Path``, ``py.path.local``) for the file path (`11773`)
- The ``pd.read_csv()`` with ``engine='python'`` has gained support for the  ``decimal`` (`12933`), ``na_filter`` (`13321`) and the ``memory_map`` option (`13381`).
- Consistent with the Python API, ``pd.read_csv()`` will now interpret ``+inf`` as positive infinity (`13274`)
- The ``pd.read_html()`` has gained support for the ``na_values``, ``converters``, ``keep_default_na``  options (`13461`)
- ``Categorical.astype()`` now accepts an optional boolean argument ``copy``, effective when dtype is categorical (`13209`)
- ``DataFrame`` has gained the ``.asof()`` method to return the last non-NaN values according to the selected subset (`13358`)
- The ``DataFrame`` constructor will now respect key ordering if a list of ``OrderedDict`` objects are passed in (`13304`)
- ``pd.read_html()`` has gained support for the ``decimal`` option (`12907`)
- ``Series`` has gained the properties ``.is_monotonic``, ``.is_monotonic_increasing``, ``.is_monotonic_decreasing``, similar to ``Index`` (`13336`)
- ``DataFrame.to_sql()`` now allows a single value as the SQL type for all columns (`11886`).
- ``Series.append`` now supports the ``ignore_index`` option (`13677`)
- ``.to_stata()`` and ``StataWriter`` can now write variable labels to Stata dta files using a dictionary to make column names to labels (`13535`, `13536`)
- ``.to_stata()`` and ``StataWriter`` will automatically convert ``datetime64[ns]`` columns to Stata format ``%tc``, rather than raising a ``ValueError`` (`12259`)
- ``read_stata()`` and ``StataReader`` raise with a more explicit error message when reading Stata files with repeated value labels when ``convert_categoricals=True`` (`13923`)
- ``DataFrame.style`` will now render sparsified MultiIndexes (`11655`)
- ``DataFrame.style`` will now show column level names (e.g. ``DataFrame.columns.names``) (`13775`)
- ``DataFrame`` has gained support to re-order the columns based on the values  in a row using ``df.sort_values(by='...', axis=1)`` (`10806`)  .. ipython:: python  df = pd.DataFrame({"A": [2, 7], "B": [3, 5], "C": [4, 8]}, index=["row1", "row2"])  df  df.sort_values(by="row2", axis=1)  - Added documentation to `I/O<io.dtypes>` regarding the perils of reading in columns with mixed dtypes and how to handle it (`13746`)
- `~DataFrame.to_html` now has a ``border`` argument to control the value in the opening ``<table>`` tag. The default is the value of the ``html.border`` option, which defaults to 1. This also affects the notebook HTML repr, but since Jupyter's CSS includes a border-width attribute, the visual effect is the same. (`11563`).
- Raise ``ImportError`` in the sql functions when ``sqlalchemy`` is not installed and a connection string is used (`11920`).

- Compatibility with matplotlib 2.0. Older versions of pandas should also work with matplotlib 2.0 (`1333`)
- ``Timestamp``, ``Period``, ``DatetimeIndex``, ``PeriodIndex`` and ``.dt`` accessor have gained a ``.is_leap_year`` property to check whether the date belongs to a leap year. (`13727`)
- ``astype()`` will now accept a dict of column name to data types mapping as the ``dtype`` argument. (`12086`)
- The ``pd.read_json`` and ``DataFrame.to_json`` has gained support for reading and writing json lines with ``lines`` option see `Line delimited json <io.jsonl>` (`9180`)
- `read_excel` now supports the true_values and false_values keyword arguments (`13347`)
- ``groupby()`` will now accept a scalar and a single-element list for specifying ``level`` on a non-``MultiIndex`` grouper. (`13907`)
- Non-convertible dates in an excel date column will be returned without conversion and the column will be ``object`` dtype, rather than raising an exception (`10001`).
- ``pd.Timedelta(None)`` is now accepted and will return ``NaT``, mirroring ``pd.Timestamp`` (`13687`)
- ``pd.read_stata()`` can now handle some format 111 files, which are produced by SAS when generating Stata dta files (`11526`)
- ``Series`` and ``Index`` now support ``divmod`` which will return a tuple of  series or indices. This behaves like a standard binary operator with regards  to broadcasting rules (`14208`).  .. _whatsnew_0190.api:  API changes
~~~  ``Series.tolist()`` will now return Python types
^^^  ``Series.tolist()`` will now return Python types in the output, mimicking NumPy ``.tolist()`` behavior (`10904`)  .. ipython:: python  s = pd.Series([1, 2, 3])  **Previous behavior**:  .. code-block:: ipython  In [7]: type(s.tolist()[0])  Out[7]:  <class 'numpy.int64'>  **New behavior**:  .. ipython:: python  type(s.tolist()[0])  .. _whatsnew_0190.api.series_ops:  ``Series`` operators for different indexes
^^^  Following ``Series`` operators have been changed to make all operators consistent,
including ``DataFrame`` (`1134`, `4581`, `13538`)  - ``Series`` comparison operators now raise ``ValueError`` when ``index`` are different.
- ``Series`` logical operators align both ``index`` of left and right hand side.  .. warning::  Until 0.18.1, comparing ``Series`` with the same length, would succeed even if  the ``.index`` are different (the result ignores ``.index``). As of 0.19.0, this will raises ``ValueError`` to be more strict. This section also describes how to keep previous behavior or align different indexes, using the flexible comparison methods like ``.eq``.  As a result, ``Series`` and ``DataFrame`` operators behave as below:  Arithmetic operators
"""  Arithmetic operators align both ``index`` (no changes).  .. ipython:: python  s1 = pd.Series([1, 2, 3], index=list("ABC"))  s2 = pd.Series([2, 2, 2], index=list("ABD"))  s1 + s2  df1 = pd.DataFrame([1, 2, 3], index=list("ABC"))  df2 = pd.DataFrame([2, 2, 2], index=list("ABD"))  df1 + df2  Comparison operators

"""  Comparison operators raise ``ValueError`` when ``.index`` are different.  **Previous behavior** (``Series``):  ``Series`` compared values ignoring the ``.index`` as long as both had the same length:  .. code-block:: ipython  In [1]: s1 == s2  Out[1]:  A  False  B  True  C  False  dtype: bool  **New behavior** (``Series``):  .. code-block:: ipython  In [2]: s1 == s2  Out[2]:  ValueError: Can only compare identically-labeled Series objects  .. note::  To achieve the same result as previous versions (compare values based on locations ignoring ``.index``), compare both ``.values``.  .. ipython:: python  s1.values == s2.values  If you want to compare ``Series`` aligning its ``.index``, see flexible comparison methods section below:  .. ipython:: python  s1.eq(s2)  **Current behavior** (``DataFrame``, no change):  .. code-block:: ipython  In [3]: df1 == df2  Out[3]:  ValueError: Can only compare identically-labeled DataFrame objects  Logical operators
"""  Logical operators align both ``.index`` of left and right hand side.  **Previous behavior** (``Series``), only left hand side ``index`` was kept:  .. code-block:: ipython  In [4]: s1 = pd.Series([True, False, True], index=list('ABC'))  In [5]: s2 = pd.Series([True, True, True], index=list('ABD'))  In [6]: s1 & s2  Out[6]:  A  True  B  False  C  False  dtype: bool  **New behavior** (``Series``):  .. ipython:: python  s1 = pd.Series([True, False, True], index=list("ABC"))  s2 = pd.Series([True, True, True], index=list("ABD"))  s1 & s2  .. note::  ``Series`` logical operators fill a ``NaN`` result with ``False``.  .. note::  To achieve the same result as previous versions (compare values based on only left hand side index), you can use ``reindex_like``:  .. ipython:: python  s1 & s2.reindex_like(s1)  **Current behavior** (``DataFrame``, no change):  .. ipython:: python  df1 = pd.DataFrame([True, False, True], index=list("ABC"))  df2 = pd.DataFrame([True, True, True], index=list("ABD"))  df1 & df2  Flexible comparison methods
"""  ``Series`` flexible comparison methods like ``eq``, ``ne``, ``le``, ``lt``, ``ge`` and ``gt`` now align both ``index``. Use these operators if you want to compare two ``Series``
which has the different ``index``.  .. ipython:: python  s1 = pd.Series([1, 2, 3], index=["a", "b", "c"])  s2 = pd.Series([2, 2, 2], index=["b", "c", "d"])  s1.eq(s2)  s1.ge(s2)  Previously, this worked the same as comparison operators (see above).  .. _whatsnew_0190.api.promote:  ``Series`` type promotion on assignment
^^^  A ``Series`` will now correctly promote its dtype for assignment with incompat values to the current dtype (`13234`)  .. ipython:: python  s = pd.Series()  **Previous behavior**:  .. code-block:: ipython  In [2]: s["a"] = pd.Timestamp("2016-01-01")  In [3]: s["b"] = 3.0  TypeError: invalid type promotion  **New behavior**:  .. ipython:: python  s["a"] = pd.Timestamp("2016-01-01")  s["b"] = 3.0  s  s.dtype  .. _whatsnew_0190.api.to_datetime_coerce:  Function ``.to_datetime()`` changes
^^^  Previously if ``.to_datetime()`` encountered mixed integers/floats and strings, but no datetimes with ``errors='coerce'`` it would convert all to ``NaT``.  **Previous behavior**:  .. code-block:: ipython  In [2]: pd.to_datetime([1, 'foo'], errors='coerce')  Out[2]: DatetimeIndex(['NaT', 'NaT'], dtype='datetime64[ns]', freq=None)  **Current behavior**:  This will now convert integers/floats with the default unit of ``ns``.  .. ipython:: python  pd.to_datetime([1, "foo"], errors="coerce")  Bug fixes related to ``.to_datetime()``:  - Bug in ``pd.to_datetime()`` when passing integers or floats, and no ``unit`` and ``errors='coerce'`` (`13180`).
- Bug in ``pd.to_datetime()`` when passing invalid data types (e.g. bool); will now respect the ``errors`` keyword (`13176`)
- Bug in ``pd.to_datetime()`` which overflowed on ``int8``, and ``int16`` dtypes (`13451`)

- Bug in ``pd.to_datetime()`` raise ``AttributeError`` with ``NaN`` and the other string is not valid when ``errors='ignore'`` (`12424`)
- Bug in ``pd.to_datetime()`` did not cast floats correctly when ``unit`` was specified, resulting in truncated datetime (`13834`)  .. _whatsnew_0190.api.merging:  Merging changes
^^^  Merging will now preserve the dtype of the join keys (`8596`)  .. ipython:: python  df1 = pd.DataFrame({"key": [1], "v1": [10]})  df1  df2 = pd.DataFrame({"key": [1, 2], "v1": [20, 30]})  df2  **Previous behavior**:  .. code-block:: ipython  In [5]: pd.merge(df1, df2, how='outer')  Out[5]:  key  v1  0  1.0  10.0  1  1.0  20.0  2  2.0  30.0  In [6]: pd.merge(df1, df2, how='outer').dtypes  Out[6]:  key  float64  v1  float64  dtype: object  **New behavior**:  We are able to preserve the join keys  .. ipython:: python  pd.merge(df1, df2, how="outer")  pd.merge(df1, df2, how="outer").dtypes  Of course if you have missing values that are introduced, then the
resulting dtype will be upcast, which is unchanged from previous.  .. ipython:: python  pd.merge(df1, df2, how="outer", on="key")  pd.merge(df1, df2, how="outer", on="key").dtypes  .. _whatsnew_0190.api.describe:  Method ``.describe()`` changes
^^^  Percentile identifiers in the index of a ``.describe()`` output will now be rounded to the least precision that keeps them distinct (`13104`)  .. ipython:: python  s = pd.Series([0, 1, 2, 3, 4])  df = pd.DataFrame([0, 1, 2, 3, 4])  **Previous behavior**:  The percentiles were rounded to at most one decimal place, which could raise ``ValueError`` for a data frame if the percentiles were duplicated.  .. code-block:: ipython  In [3]: s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.999])  Out[3]:  count  5.000  mean  2.000  std  1.581139  min  0.000  0.0%  0.000400  0.1%  0.002000  0.1%  0.004000  50%  2.000  99.9%  3.996000  100.0%  3.998000  100.0%  3.999600  max  4.000  dtype: float64  In [4]: df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.999])  Out[4]:  ...  ValueError: cannot reindex from a duplicate axis  **New behavior**:  .. ipython:: python  s.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.999])  df.describe(percentiles=[0.0001, 0.0005, 0.001, 0.999, 0.9995, 0.999])  Furthermore:  - Passing duplicated ``percentiles`` will now raise a ``ValueError``.
- Bug in ``.describe()`` on a DataFrame with a mixed-dtype column index, which would previously raise a ``TypeError`` (`13288`)  .. _whatsnew_0190.api.period:  ``Period`` changes
^^^  The ``PeriodIndex`` now has ``period`` dtype
"""  ``PeriodIndex`` now has its own ``period`` dtype. The ``period`` dtype is a
pandas extension dtype like ``category`` or the `timezone aware dtype <timeseries.timezone_series>` (``datetime64[ns, tz]``) (`13941`).
As a consequence of this change, ``PeriodIndex`` no longer has an integer dtype:  **Previous behavior**:  .. code-block:: ipython  In [1]: pi = pd.PeriodIndex(['2016-08-01'], freq='D')  In [2]: pi  Out[2]: PeriodIndex(['2016-08-01'], dtype='int64', freq='D')  In [3]: pd.api.types.is_integer_dtype(pi)  Out[3]: True  In [4]: pi.dtype  Out[4]: dtype('int64')  **New behavior**:  .. ipython:: python  pi = pd.PeriodIndex(["2016-08-01"], freq="D")  pi  pd.api.types.is_integer_dtype(pi)  pd.api.types.is_period_dtype(pi)  pi.dtype  type(pi.dtype)  .. _whatsnew_0190.api.periodnat:  ``Period('NaT')`` now returns ``pd.NaT``

"""  Previously, ``Period`` has its own ``Period('NaT')`` representation different from ``pd.NaT``. Now ``Period('NaT')`` has been changed to return ``pd.NaT``. (`12759`, `13582`)  **Previous behavior**:  .. code-block:: ipython  In [5]: pd.Period('NaT', freq='D')  Out[5]: Period('NaT', 'D')  **New behavior**:  These result in ``pd.NaT`` without providing ``freq`` option.  .. ipython:: python  pd.Period("NaT")  pd.Period(None)  To be compatible with ``Period`` addition and subtraction, ``pd.NaT`` now supports addition and subtraction with ``int``. Previously it raised ``ValueError``.  **Previous behavior**:  .. code-block:: ipython  In [5]: pd.NaT + 1  ...  ValueError: Cannot add integral value to Timestamp without freq.  **New behavior**:  .. ipython:: python  pd.NaT + 1  pd.NaT - 1  ``PeriodIndex.values`` now returns array of ``Period`` object
"""  ``.values`` is changed to return an array of ``Period`` objects, rather than an array
of integers (`13988`).  **Previous behavior**:  .. code-block:: ipython  In [6]: pi = pd.PeriodIndex(['2011-01', '2011-02'], freq='M')  In [7]: pi.values  Out[7]: array([492, 493])  **New behavior**:  .. ipython:: python  pi = pd.PeriodIndex(["2011-01", "2011-02"], freq="M")  pi.values  .. _whatsnew_0190.api.setops:  Index ``+`` / ``-`` no longer used for set operations
^^^  Addition and subtraction of the base Index type and of DatetimeIndex
(not the numeric index types)
previously performed set operations (set union and difference). This
behavior was already deprecated since 0.15.0 (in favor using the specific
``.union()`` and ``.difference()`` methods), and is now disabled. When
possible, ``+`` and ``-`` are now used for element-wise operations, for
example for concatenating strings or subtracting datetimes
(`8227`, `14127`).  Previous behavior:  .. code-block:: ipython  In [1]: pd.Index(['a', 'b']) + pd.Index(['a', 'c'])  FutureWarning: using '+' to provide set union with Indexes is deprecated, use '|' or .union()  Out[1]: Index(['a', 'b', 'c'], dtype='object')  **New behavior**: the same operation will now perform element-wise addition:  .. ipython:: python  pd.Index(["a", "b"]) + pd.Index(["a", "c"])  Note that numeric Index objects already performed element-wise operations.
For example, the behavior of adding two integer Indexes is unchanged.
The base ``Index`` is now made consistent with this behavior.  .. ipython:: python  pd.Index([1, 2, 3]) + pd.Index([2, 3, 4])  Further, because of this change, it is now possible to subtract two
DatetimeIndex objects resulting in a TimedeltaIndex:  **Previous behavior**:  .. code-block:: ipython  In [1]: (pd.DatetimeIndex(['2016-01-01', '2016-01-02'])  ...:  - pd.DatetimeIndex(['2016-01-02', '2016-01-03']))  FutureWarning: using '-' to provide set differences with datetimelike Indexes is deprecated, use .difference()  Out[1]: DatetimeIndex(['2016-01-01'], dtype='datetime64[ns]', freq=None)  **New behavior**:  .. ipython:: python  (  pd.DatetimeIndex(["2016-01-01", "2016-01-02"])  - pd.DatetimeIndex(["2016-01-02", "2016-01-03"])  )  .. _whatsnew_0190.api.difference:  ``Index.difference`` and ``.symmetric_difference`` changes
^^^  ``Index.difference`` and ``Index.symmetric_difference`` will now, more consistently, treat ``NaN`` values as any other values. (`13514`)  .. ipython:: python  idx1 = pd.Index([1, 2, 3, np.nan])  idx2 = pd.Index([0, 1, np.nan])  **Previous behavior**:  .. code-block:: ipython  In [3]: idx1.difference(idx2)  Out[3]: Float64Index([nan, 2.0, 3.0], dtype='float64')  In [4]: idx1.symmetric_difference(idx2)  Out[4]: Float64Index([0.0, nan, 2.0, 3.0], dtype='float64')  **New behavior**:  .. ipython:: python  idx1.difference(idx2)  idx1.symmetric_difference(idx2)  .. _whatsnew_0190.api.unique_index:  ``Index.unique`` consistently returns ``Index``
^^^  ``Index.unique()`` now returns unique values as an
``Index`` of the appropriate ``dtype``. (`13395`).
Previously, most ``Index`` classes returned ``np.ndarray``, and ``DatetimeIndex``,

``TimedeltaIndex`` and ``PeriodIndex`` returned ``Index`` to keep metadata like timezone.  **Previous behavior**:  .. code-block:: ipython  In [1]: pd.Index([1, 2, 3]).unique()  Out[1]: array([1, 2, 3])  In [2]: pd.DatetimeIndex(['2011-01-01', '2011-01-02',  ...:  '2011-01-03'], tz='Asia/Tokyo').unique()  Out[2]:  DatetimeIndex(['2011-01-01 000+09:00', '2011-01-02 000+09:00',  '2011-01-03 000+09:00'],  dtype='datetime64[ns, Asia/Tokyo]', freq=None)  **New behavior**:  .. ipython:: python  pd.Index([1, 2, 3]).unique()  pd.DatetimeIndex(  ["2011-01-01", "2011-01-02", "2011-01-03"], tz="Asia/Tokyo"  ).unique()  .. _whatsnew_0190.api.multiindex:  ``MultiIndex`` constructors, ``groupby`` and ``set_index`` preserve categorical dtypes
^^^  ``MultiIndex.from_arrays`` and ``MultiIndex.from_product`` will now preserve categorical dtype
in ``MultiIndex`` levels (`13743`, `13854`).  .. ipython:: python  cat = pd.Categorical(["a", "b"], categories=list("bac"))  lvl1 = ["foo", "bar"]  midx = pd.MultiIndex.from_arrays([cat, lvl1])  midx  **Previous behavior**:  .. code-block:: ipython  In [4]: midx.levels[0]  Out[4]: Index(['b', 'a', 'c'], dtype='object')  In [5]: midx.get_level_values[0]  Out[5]: Index(['a', 'b'], dtype='object')  **New behavior**: the single level is now a ``CategoricalIndex``:  .. ipython:: python  midx.levels[0]  midx.get_level_values(0)  An analogous change has been made to ``MultiIndex.from_product``.
As a consequence, ``groupby`` and ``set_index`` also preserve categorical dtypes in indexes  .. ipython:: python  df = pd.DataFrame({"A": [0, 1], "B": [10, 11], "C": cat})  df_grouped = df.groupby(by=["A", "C"]).first()  df_set_idx = df.set_index(["A", "C"])  **Previous behavior**:  .. code-block:: ipython  In [11]: df_grouped.index.levels[1]  Out[11]: Index(['b', 'a', 'c'], dtype='object', name='C')  In [12]: df_grouped.reset_index().dtypes  Out[12]:  A  int64  C  object  B  float64  dtype: object  In [13]: df_set_idx.index.levels[1]  Out[13]: Index(['b', 'a', 'c'], dtype='object', name='C')  In [14]: df_set_idx.reset_index().dtypes  Out[14]:  A  int64  C  object  B  int64  dtype: object  **New behavior**:  .. ipython:: python  df_grouped.index.levels[1]  df_grouped.reset_index().dtypes  df_set_idx.index.levels[1]  df_set_idx.reset_index().dtypes  .. _whatsnew_0190.api.autogenerated_chunksize_index:  Function ``read_csv`` will progressively enumerate chunks
^^^  When `read_csv` is called with ``chunksize=n`` and without specifying an index,
each chunk used to have an independently generated index from ``0`` to ``n-1``.
They are now given instead a progressive index, starting from ``0`` for the first chunk,
from ``n`` for the second, and so on, so that, when concatenated, they are identical to
the result of calling `read_csv` without the ``chunksize=`` argument
(`12185`).  .. ipython:: python  data = "A,B\n0,1\n2,3\n4,5\n6,7"  **Previous behavior**:  .. code-block:: ipython  In [2]: pd.concat(pd.read_csv(StringIO(data), chunksize=2))  Out[2]:  A  B  0  0  1  1  2  3  0  4  5  1  6  7  **New behavior**:  .. ipython:: python  pd.concat(pd.read_csv(StringIO(data), chunksize=2))  .. _whatsnew_0190.sparse:  Sparse changes
^^^  These changes allow pandas to handle sparse data with more dtypes, and for work to make a smoother experience with data handling.  Types ``int64`` and ``bool`` support enhancements

"""  Sparse data structures now gained enhanced support of ``int64`` and ``bool`` ``dtype`` (`667`, `13849`).  Previously, sparse data were ``float64`` dtype by default, even if all inputs were of ``int`` or ``bool`` dtype. You had to specify ``dtype`` explicitly to create sparse data with ``int64`` dtype. Also, ``fill_value`` had to be specified explicitly because the default was ``np.nan`` which doesn't appear in ``int64`` or ``bool`` data.  .. code-block:: ipython  In [1]: pd.SparseArray([1, 2, 0, 0])  Out[1]:  [1.0, 2.0, 0.0, 0.0]  Fill: nan  IntIndex  Indices: array([0, 1, 2, 3], dtype=int32)  # specifying int64 dtype, but all values are stored in sp_values because  # fill_value default is np.nan  In [2]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64)  Out[2]:  [1, 2, 0, 0]  Fill: nan  IntIndex  Indices: array([0, 1, 2, 3], dtype=int32)  In [3]: pd.SparseArray([1, 2, 0, 0], dtype=np.int64, fill_value=0)  Out[3]:  [1, 2, 0, 0]  Fill: 0  IntIndex  Indices: array([0, 1], dtype=int32)  As of v0.19.0, sparse data keeps the input dtype, and uses more appropriate ``fill_value`` defaults (``0`` for ``int64`` dtype, ``False`` for ``bool`` dtype).  .. ipython:: python  pd.arrays.SparseArray([1, 2, 0, 0], dtype=np.int64)  pd.arrays.SparseArray([True, False, False, False])  See the `docs <sparse.dtype>` for more details.  Operators now preserve dtypes
"""  - Sparse data structure now can preserve ``dtype`` after arithmetic ops (`13848`)  .. code-block:: python  s = pd.SparseSeries([0, 2, 0, 1], fill_value=0, dtype=np.int64)  s.dtype  s + 1  - Sparse data structure now support ``astype`` to convert internal ``dtype`` (`13900`)  .. code-block:: python  s = pd.SparseSeries([1.0, 0.0, 2.0, 0.0], fill_value=0)  s  s.astype(np.int64)  ``astype`` fails if data contains values which cannot be converted to specified ``dtype``.
Note that the limitation is applied to ``fill_value`` which default is ``np.nan``.  .. code-block:: ipython  In [7]: pd.SparseSeries([1., np.nan, 2., np.nan], fill_value=np.nan).astype(np.int64)  Out[7]:  ValueError: unable to coerce current fill_value nan to int64 dtype  Other sparse fixes
"""  - Subclassed ``SparseDataFrame`` and ``SparseSeries`` now preserve class types when slicing or transposing. (`13787`)
- ``SparseArray`` with ``bool`` dtype now supports logical (bool) operators (`14000`)
- Bug in ``SparseSeries`` with ``MultiIndex`` ``[]`` indexing may raise ``IndexError`` (`13144`)
- Bug in ``SparseSeries`` with ``MultiIndex`` ``[]`` indexing result may have normal ``Index`` (`13144`)
- Bug in ``SparseDataFrame`` in which ``axis=None`` did not default to ``axis=0`` (`13048`)
- Bug in ``SparseSeries`` and ``SparseDataFrame`` creation with ``object`` dtype may raise ``TypeError`` (`11633`)
- Bug in ``SparseDataFrame`` doesn't respect passed ``SparseArray`` or ``SparseSeries`` 's dtype and ``fill_value``  (`13866`)
- Bug in ``SparseArray`` and ``SparseSeries`` don't apply ufunc to ``fill_value`` (`13853`)
- Bug in ``SparseSeries.abs`` incorrectly keeps negative ``fill_value`` (`13853`)
- Bug in single row slicing on multi-type ``SparseDataFrame`` s, types were previously forced to float (`13917`)
- Bug in ``SparseSeries`` slicing changes integer dtype to float (`8292`)
- Bug in ``SparseDataFarme`` comparison ops may raise ``TypeError`` (`13001`)
- Bug in ``SparseDataFarme.isnull`` raises ``ValueError`` (`8276`)
- Bug in ``SparseSeries`` representation with ``bool`` dtype may raise ``IndexError`` (`13110`)
- Bug in ``SparseSeries`` and ``SparseDataFrame`` of ``bool`` or ``int64`` dtype may display its values like ``float64`` dtype (`13110`)
- Bug in sparse indexing using ``SparseArray`` with ``bool`` dtype may return incorrect result  (`13985`)
- Bug in ``SparseArray`` created from ``SparseSeries`` may lose ``dtype`` (`13999`)

- Bug in ``SparseSeries`` comparison with dense returns normal ``Series`` rather than ``SparseSeries`` (`13999`)  .. _whatsnew_0190.indexer_dtype:  Indexer dtype changes
^^^  .. note::  This change only affects 64 bit python running on Windows, and only affects relatively advanced  indexing operations  Methods such as ``Index.get_indexer`` that return an indexer array, coerce that array to a "platform int", so that it can be
directly used in 3rd party library operations like ``numpy.take``.  Previously, a platform int was defined as ``np.int_``
which corresponds to a C integer, but the correct type, and what is being used now, is ``np.intp``, which corresponds
to the C integer size that can hold a pointer (`3033`, `13972`).  These types are the same on many platform, but for 64 bit python on Windows,
``np.int_`` is 32 bits, and ``np.intp`` is 64 bits.  Changing this behavior improves performance for many
operations on that platform.  **Previous behavior**:  .. code-block:: ipython  In [1]: i = pd.Index(['a', 'b', 'c'])  In [2]: i.get_indexer(['b', 'b', 'c']).dtype  Out[2]: dtype('int32')  **New behavior**:  .. code-block:: ipython  In [1]: i = pd.Index(['a', 'b', 'c'])  In [2]: i.get_indexer(['b', 'b', 'c']).dtype  Out[2]: dtype('int64')  .. _whatsnew_0190.api.other:  Other API changes
^^^  - ``Timestamp.to_pydatetime`` will issue a ``UserWarning`` when ``warn=True``, and the instance has a non-zero number of nanoseconds, previously this would print a message to stdout (`14101`).
- ``Series.unique()`` with datetime and timezone now returns return array of ``Timestamp`` with timezone (`13565`).
- ``Panel.to_sparse()`` will raise a ``NotImplementedError`` exception when called (`13778`).
- ``Index.reshape()`` will raise a ``NotImplementedError`` exception when called (`12882`).
- ``.filter()`` enforces mutual exclusion of the keyword arguments (`12399`).
- ``eval``'s upcasting rules for ``float32`` types have been updated to be more consistent with NumPy's rules.  New behavior will not upcast to ``float64`` if you multiply a pandas ``float32`` object by a scalar float64 (`12388`).
- An ``UnsupportedFunctionCall`` error is now raised if NumPy ufuncs like ``np.mean`` are called on groupby or resample objects (`12811`).
- ``__setitem__`` will no longer apply a callable rhs as a function instead of storing it. Call ``where`` directly to get the previous behavior (`13299`).
- Calls to ``.sample()`` will respect the random seed set via ``numpy.random.seed(n)`` (`13161`)
- ``Styler.apply`` is now more strict about the outputs your function must return. For ``axis=0`` or ``axis=1``, the output shape must be identical. For ``axis=None``, the output must be a DataFrame with identical columns and index labels (`13222`).
- ``Float64Index.astype(int)`` will now raise ``ValueError`` if ``Float64Index`` contains ``NaN`` values (`13149`)
- ``TimedeltaIndex.astype(int)`` and ``DatetimeIndex.astype(int)`` will now return ``Int64Index`` instead of ``np.array`` (`13209`)
- Passing ``Period`` with multiple frequencies to normal ``Index`` now returns ``Index`` with ``object`` dtype (`13664`)
- ``PeriodIndex.fillna`` with ``Period`` has different freq now coerces to ``object`` dtype (`13664`)
- Faceted boxplots from ``DataFrame.boxplot(by=col)`` now return a ``Series`` when ``return_type`` is not None. Previously these returned an ``OrderedDict``. Note that when ``return_type=None``, the default, these still return a 2-D NumPy array (`12216`, `7096`).
- ``pd.read_hdf`` will now raise a ``ValueError`` instead of ``KeyError``, if a mode other than ``r``, ``r+`` and ``a`` is supplied. (`13623`)

- ``pd.read_csv()``, ``pd.read_table()``, and ``pd.read_hdf()`` raise the builtin ``FileNotFoundError`` exception for Python 3.x when called on a nonexistent file; this is back-ported as ``IOError`` in Python 2.x (`14086`)
- More informative exceptions are passed through the csv parser. The exception type would now be the original exception type instead of ``CParserError`` (`13652`).
- ``pd.read_csv()`` in the C engine will now issue a ``ParserWarning`` or raise a ``ValueError`` when ``sep`` encoded is more than one character long (`14065`)
- ``DataFrame.values`` will now return ``float64`` with a ``DataFrame`` of mixed ``int64`` and ``uint64`` dtypes, conforming to ``np.find_common_type`` (`10364`, `13917`)
- ``.groupby.groups`` will now return a dictionary of ``Index`` objects, rather than a dictionary of ``np.ndarray`` or ``lists`` (`14293`)  .. _whatsnew_0190.deprecations:  Deprecations
~~~
- ``Series.reshape`` and ``Categorical.reshape`` have been deprecated and will be removed in a subsequent release (`12882`, `12882`)
- ``PeriodIndex.to_datetime`` has been deprecated in favor of ``PeriodIndex.to_timestamp`` (`8254`)
- ``Timestamp.to_datetime`` has been deprecated in favor of ``Timestamp.to_pydatetime`` (`8254`)
- ``Index.to_datetime`` and ``DatetimeIndex.to_datetime`` have been deprecated in favor of ``pd.to_datetime`` (`8254`)
- ``pandas.core.datetools`` module has been deprecated and will be removed in a subsequent release (`14094`)
- ``SparseList`` has been deprecated and will be removed in a future version (`13784`)
- ``DataFrame.to_html()`` and ``DataFrame.to_latex()`` have dropped the ``colSpace`` parameter in favor of ``col_space`` (`13857`)
- ``DataFrame.to_sql()`` has deprecated the ``flavor`` parameter, as it is superfluous when SQLAlchemy is not installed (`13611`)
- Deprecated ``read_csv`` keywords:  - ``compact_ints`` and ``use_unsigned`` have been deprecated and will be removed in a future version (`13320`)  - ``buffer_lines`` has been deprecated and will be removed in a future version (`13360`)  - ``as_recarray`` has been deprecated and will be removed in a future version (`13373`)  - ``skip_footer`` has been deprecated in favor of ``skipfooter`` and will be removed in a future version (`13349`)  - top-level ``pd.ordered_merge()`` has been renamed to ``pd.merge_ordered()`` and the original name will be removed in a future version (`13358`)
- ``Timestamp.offset`` property (and named arg in the constructor), has been deprecated in favor of ``freq`` (`12160`)
- ``pd.tseries.util.pivot_annual`` is deprecated. Use ``pivot_table`` as alternative, an example is `here <cookbook.pivot>` (`736`)
- ``pd.tseries.util.isleapyear`` has been deprecated and will be removed in a subsequent release. Datetime-likes now have a ``.is_leap_year`` property (`13727`)
- `Panel4D`` and ``PanelND`` constructors are deprecated and will be removed in a future version. The recommended way to represent these types of n-dimensional data are with the `xarray package_. pandas provides a `~Panel4D.to_xarray` method to automate this conversion (`13564`).
- ``pandas.tseries.frequencies.get_standard_freq`` is deprecated. Use  ``pandas.tseries.frequencies.to_offset(freq).rule_code`` instead (`13874`)
- ``pandas.tseries.frequencies.to_offset``'s ``freqstr`` keyword is deprecated in favor of ``freq`` (`13874`)
- ``Categorical.from_array`` has been deprecated and will be removed in a future version (`13854`)  .. _whatsnew_0190.prior_deprecations:  Removal of prior version deprecations/changes
~~~  - The ``SparsePanel`` class has been removed (`13778`)
- The ``pd.sandbox`` module has been removed in favor of the external library ``pandas-qt`` (`13670`)
- The ``pandas.io.data`` and ``pandas.io.wb`` modules are removed in favor of  the pandas-datareader package_ (`13724`).

- The ``pandas.tools.rplot`` module has been removed in favor of  the seaborn package_ (`13855`)
- ``DataFrame.to_csv()`` has dropped the ``engine`` parameter, as was deprecated in 0.17.1 (`11274`, `13419`)
- ``DataFrame.to_dict()`` has dropped the ``outtype`` parameter in favor of ``orient`` (`13627`, `8486`)
- ``pd.Categorical`` has dropped setting of the ``ordered`` attribute directly in favor of the ``set_ordered`` method (`13671`)
- ``pd.Categorical`` has dropped the ``levels`` attribute in favor of ``categories`` (`8376`)
- ``DataFrame.to_sql()`` has dropped the ``mysql`` option for the ``flavor`` parameter (`13611`)
- ``Panel.shift()`` has dropped the ``lags`` parameter in favor of ``periods`` (`14041`)
- ``pd.Index`` has dropped the ``diff`` method in favor of ``difference`` (`13669`)
- ``pd.DataFrame`` has dropped the ``to_wide`` method in favor of ``to_panel`` (`14039`)
- ``Series.to_csv`` has dropped the ``nanRep`` parameter in favor of ``na_rep`` (`13804`)
- ``Series.xs``, ``DataFrame.xs``, ``Panel.xs``, ``Panel.major_xs``, and ``Panel.minor_xs`` have dropped the ``copy`` parameter (`13781`)
- ``str.split`` has dropped the ``return_type`` parameter in favor of ``expand`` (`13701`)
- Removal of the legacy time rules (offset aliases), deprecated since 0.17.0 (this has been alias since 0.8.0) (`13590`, `13868`). Now legacy time rules raises ``ValueError``. For the list of currently supported offsets, see `here <timeseries.offset_aliases>`.
- The default value for the ``return_type`` parameter for ``DataFrame.plot.box`` and ``DataFrame.boxplot`` changed from ``None`` to ``"axes"``. These methods will now return a matplotlib axes by default instead of a dictionary of artists. See `here <visualization.box.return>` (`6581`).
- The ``tquery`` and ``uquery`` functions in the ``pandas.io.sql`` module are removed (`5950`).  .. _whatsnew_0190.performance:  Performance improvements
~~~  - Improved performance of sparse ``IntIndex.intersect`` (`13082`)
- Improved performance of sparse arithmetic with ``BlockIndex`` when the number of blocks are large, though recommended to use ``IntIndex`` in such cases (`13082`)
- Improved performance of ``DataFrame.quantile()`` as it now operates per-block (`11623`)
- Improved performance of float64 hash table operations, fixing some very slow indexing and groupby operations in python 3 (`13166`, `13334`)
- Improved performance of ``DataFrameGroupBy.transform`` (`12737`)
- Improved performance of ``Index`` and ``Series`` ``.duplicated`` (`10235`)
- Improved performance of ``Index.difference`` (`12044`)
- Improved performance of ``RangeIndex.is_monotonic_increasing`` and ``is_monotonic_decreasing`` (`13749`)
- Improved performance of datetime string parsing in ``DatetimeIndex`` (`13692`)
- Improved performance of hashing ``Period`` (`12817`)
- Improved performance of ``factorize`` of datetime with timezone (`13750`)
- Improved performance of by lazily creating indexing hashtables on larger Indexes (`14266`)
- Improved performance of ``groupby.groups`` (`14293`)
- Unnecessary materializing of a MultiIndex when introspecting for memory usage (`14308`)  .. _whatsnew_0190.bug_fixes:  Bug fixes
~~~  - Bug in ``groupby().shift()``, which could cause a segfault or corruption in rare circumstances when grouping by columns with missing values (`13813`)
- Bug in ``groupby().cumsum()`` calculating ``cumprod`` when ``axis=1``. (`13994`)
- Bug in ``pd.to_timedelta()`` in which the ``errors`` parameter was not being respected (`13613`)
- Bug in ``io.json.json_normalize()``, where non-ascii keys raised an exception (`13213`)
- Bug when passing a not-default-indexed ``Series`` as ``xerr`` or ``yerr`` in ``.plot()`` (`11858`)
- Bug in area plot draws legend incorrectly if subplot is enabled or legend is moved after plot (matplotlib 1.5.0 is required to draw area plot legend properly) (`9161`, `13544`)

- Bug in ``DataFrame`` assignment with an object-dtyped ``Index`` where the resultant column is mutable to the original object. (`13522`)
- Bug in matplotlib ``AutoDataFormatter``; this restores the second scaled formatting and re-adds micro-second scaled formatting (`13131`)
- Bug in selection from a ``HDFStore`` with a fixed format and ``start`` and/or ``stop`` specified will now return the selected range (`8287`)
- Bug in ``Categorical.from_codes()`` where an unhelpful error was raised when an invalid ``ordered`` parameter was passed in (`14058`)
- Bug in ``Series`` construction from a tuple of integers on windows not returning default dtype (int64) (`13646`)
- Bug in ``TimedeltaIndex`` addition with a Datetime-like object where addition overflow was not being caught (`14068`)
- Bug in ``.groupby(..).resample(..)`` when the same object is called multiple times (`13174`)
- Bug in ``.to_records()`` when index name is a unicode string (`13172`)
- Bug in calling ``.memory_usage()`` on object which doesn't implement (`12924`)
- Regression in ``Series.quantile`` with nans (also shows up in ``.median()`` and ``.describe()`` ); furthermore now names the ``Series`` with the quantile (`13098`, `13146`)
- Bug in ``SeriesGroupBy.transform`` with datetime values and missing groups (`13191`)
- Bug where empty ``Series`` were incorrectly coerced in datetime-like numeric operations (`13844`)
- Bug in ``Categorical`` constructor when passed a ``Categorical`` containing datetimes with timezones (`14190`)
- Bug in ``Series.str.extractall()`` with ``str`` index raises ``ValueError``  (`13156`)
- Bug in ``Series.str.extractall()`` with single group and quantifier  (`13382`)
- Bug in ``DatetimeIndex`` and ``Period`` subtraction raises ``ValueError`` or ``AttributeError`` rather than ``TypeError`` (`13078`)
- Bug in ``Index`` and ``Series`` created with ``NaN`` and ``NaT`` mixed data may not have ``datetime64`` dtype  (`13324`)
- Bug in ``Index`` and ``Series`` may ignore ``np.datetime64('nat')`` and ``np.timdelta64('nat')`` to infer dtype (`13324`)
- Bug in ``PeriodIndex`` and ``Period`` subtraction raises ``AttributeError`` (`13071`)
- Bug in ``PeriodIndex`` construction returning a ``float64`` index in some circumstances (`13067`)
- Bug in ``.resample(..)`` with a ``PeriodIndex`` not changing its ``freq`` appropriately when empty (`13067`)
- Bug in ``.resample(..)`` with a ``PeriodIndex`` not retaining its type or name with an empty ``DataFrame`` appropriately when empty (`13212`)
- Bug in ``groupby(..).apply(..)`` when the passed function returns scalar values per group (`13468`).
- Bug in ``groupby(..).resample(..)`` where passing some keywords would raise an exception (`13235`)
- Bug in ``.tz_convert`` on a tz-aware ``DateTimeIndex`` that relied on index being sorted for correct results (`13306`)
- Bug in ``.tz_localize`` with ``dateutil.tz.tzlocal`` may return incorrect result (`13583`)
- Bug in ``DatetimeTZDtype`` dtype with ``dateutil.tz.tzlocal`` cannot be regarded as valid dtype (`13583`)
- Bug in ``pd.read_hdf()`` where attempting to load an HDF file with a single dataset, that had one or more categorical columns, failed unless the key argument was set to the name of the dataset. (`13231`)
- Bug in ``.rolling()`` that allowed a negative integer window in construction of the ``Rolling()`` object, but would later fail on aggregation (`13383`)
- Bug in ``Series`` indexing with tuple-valued data and a numeric index (`13509`)
- Bug in printing ``pd.DataFrame`` where unusual elements with the ``object`` dtype were causing segfaults (`13717`)
- Bug in ranking ``Series`` which could result in segfaults (`13445`)
- Bug in various index types, which did not propagate the name of passed index (`12309`)
- Bug in ``DatetimeIndex``, which did not honour the ``copy=True`` (`13205`)

- Bug in ``DatetimeIndex.is_normalized`` returns incorrectly for normalized date_range in case of local timezones (`13459`)
- Bug in ``pd.concat`` and ``.append`` may coerces ``datetime64`` and ``timedelta`` to ``object`` dtype containing python built-in ``datetime`` or ``timedelta`` rather than ``Timestamp`` or ``Timedelta`` (`13626`)
- Bug in ``PeriodIndex.append`` may raises ``AttributeError`` when the result is ``object`` dtype (`13221`)
- Bug in ``CategoricalIndex.append`` may accept normal ``list`` (`13626`)
- Bug in ``pd.concat`` and ``.append`` with the same timezone get reset to UTC (`7795`)
- Bug in ``Series`` and ``DataFrame`` ``.append`` raises ``AmbiguousTimeError`` if data contains datetime near DST boundary (`13626`)
- Bug in ``DataFrame.to_csv()`` in which float values were being quoted even though quotations were specified for non-numeric values only (`12922`, `13259`)
- Bug in ``DataFrame.describe()`` raising ``ValueError`` with only boolean columns (`13898`)
- Bug in ``MultiIndex`` slicing where extra elements were returned when level is non-unique (`12896`)
- Bug in ``.str.replace`` does not raise ``TypeError`` for invalid replacement (`13438`)
- Bug in ``MultiIndex.from_arrays`` which didn't check for input array lengths matching (`13599`)
- Bug in ``cartesian_product`` and ``MultiIndex.from_product`` which may raise with empty input arrays (`12258`)
- Bug in ``pd.read_csv()`` which may cause a segfault or corruption when iterating in large chunks over a stream/file under rare circumstances (`13703`)
- Bug in ``pd.read_csv()`` which caused errors to be raised when a dictionary containing scalars is passed in for ``na_values`` (`12224`)
- Bug in ``pd.read_csv()`` which caused BOM files to be incorrectly parsed by not ignoring the BOM (`4793`)
- Bug in ``pd.read_csv()`` with ``engine='python'`` which raised errors when a numpy array was passed in for ``usecols`` (`12546`)
- Bug in ``pd.read_csv()`` where the index columns were being incorrectly parsed when parsed as dates with a ``thousands`` parameter (`14066`)
- Bug in ``pd.read_csv()`` with ``engine='python'`` in which ``NaN`` values weren't being detected after data was converted to numeric values (`13314`)
- Bug in ``pd.read_csv()`` in which the ``nrows`` argument was not properly validated for both engines (`10476`)
- Bug in ``pd.read_csv()`` with ``engine='python'`` in which infinities of mixed-case forms were not being interpreted properly (`13274`)
- Bug in ``pd.read_csv()`` with ``engine='python'`` in which trailing ``NaN`` values were not being parsed (`13320`)
- Bug in ``pd.read_csv()`` with ``engine='python'`` when reading from a ``tempfile.TemporaryFile`` on Windows with Python 3 (`13398`)
- Bug in ``pd.read_csv()`` that prevents ``usecols`` kwarg from accepting single-byte unicode strings (`13219`)
- Bug in ``pd.read_csv()`` that prevents ``usecols`` from being an empty set (`13402`)
- Bug in ``pd.read_csv()`` in the C engine where the NULL character was not being parsed as NULL (`14012`)
- Bug in ``pd.read_csv()`` with ``engine='c'`` in which NULL ``quotechar`` was not accepted even though ``quoting`` was specified as ``None`` (`13411`)
- Bug in ``pd.read_csv()`` with ``engine='c'`` in which fields were not properly cast to float when quoting was specified as non-numeric (`13411`)
- Bug in ``pd.read_csv()`` in Python 2.x with non-UTF8 encoded, multi-character separated data (`3404`)
- Bug in ``pd.read_csv()``, where aliases for utf-xx (e.g. UTF-xx, UTF_xx, utf_xx) raised UnicodeDecodeError (`13549`)
- Bug in ``pd.read_csv``, ``pd.read_table``, ``pd.read_fwf``, ``pd.read_stata`` and ``pd.read_sas`` where files were opened by parsers but not closed if both ``chunksize`` and ``iterator`` were ``None``. (`13940`)

- Bug in ``StataReader``, ``StataWriter``, ``XportReader`` and ``SAS7BDATReader`` where a file was not properly closed when an error was raised. (`13940`)
- Bug in ``pd.pivot_table()`` where ``margins_name`` is ignored when ``aggfunc`` is a list (`13354`)
- Bug in ``pd.Series.str.zfill``, ``center``, ``ljust``, ``rjust``, and ``pad`` when passing non-integers, did not raise ``TypeError`` (`13598`)
- Bug in checking for any null objects in a ``TimedeltaIndex``, which always returned ``True`` (`13603`)
- Bug in ``Series`` arithmetic raises ``TypeError`` if it contains datetime-like as ``object`` dtype (`13043`)
- Bug ``Series.isnull()`` and ``Series.notnull()`` ignore ``Period('NaT')``  (`13737`)
- Bug ``Series.fillna()`` and ``Series.dropna()`` don't affect to ``Period('NaT')``  (`13737`
- Bug in ``.fillna(value=np.nan)`` incorrectly raises ``KeyError`` on a ``category`` dtyped ``Series`` (`14021`)
- Bug in extension dtype creation where the created types were not is/identical (`13285`)
- Bug in ``.resample(..)`` where incorrect warnings were triggered by IPython introspection (`13618`)
- Bug in ``NaT`` - ``Period`` raises ``AttributeError`` (`13071`)
- Bug in ``Series`` comparison may output incorrect result if rhs contains ``NaT`` (`9005`)
- Bug in ``Series`` and ``Index`` comparison may output incorrect result if it contains ``NaT`` with ``object`` dtype (`13592`)
- Bug in ``Period`` addition raises ``TypeError`` if ``Period`` is on right hand side (`13069`)
- Bug in ``Period`` and ``Series`` or ``Index`` comparison raises ``TypeError`` (`13200`)
- Bug in ``pd.set_eng_float_format()`` that would prevent NaN and Inf from formatting (`11981`)
- Bug in ``.unstack`` with ``Categorical`` dtype resets ``.ordered`` to ``True`` (`13249`)
- Clean some compile time warnings in datetime parsing (`13607`)
- Bug in ``factorize`` raises ``AmbiguousTimeError`` if data contains datetime near DST boundary (`13750`)
- Bug in ``.set_index`` raises ``AmbiguousTimeError`` if new index contains DST boundary and multi levels (`12920`)
- Bug in ``.shift`` raises ``AmbiguousTimeError`` if data contains datetime near DST boundary (`13926`)
- Bug in ``pd.read_hdf()`` returns incorrect result when a ``DataFrame`` with a ``categorical`` column and a query which doesn't match any values (`13792`)
- Bug in ``.iloc`` when indexing with a non lexsorted MultiIndex (`13797`)
- Bug in ``.loc`` when indexing with date strings in a reverse sorted ``DatetimeIndex`` (`14316`)
- Bug in ``Series`` comparison operators when dealing with zero dim NumPy arrays (`13006`)
- Bug in ``.combine_first`` may return incorrect ``dtype`` (`7630`, `10567`)
- Bug in ``groupby`` where ``apply`` returns different result depending on whether first result is ``None`` or not (`12824`)
- Bug in ``groupby(..).nth()`` where the group key is included inconsistently if called after ``.head()/.tail()`` (`12839`)
- Bug in ``.to_html``, ``.to_latex`` and ``.to_string`` silently ignore custom datetime formatter passed through the ``formatters`` key word (`10690`)
- Bug in ``DataFrame.iterrows()``, not yielding a ``Series`` subclasse if defined (`13977`)
- Bug in ``pd.to_numeric`` when ``errors='coerce'`` and input contains non-hashable objects (`13324`)
- Bug in invalid ``Timedelta`` arithmetic and comparison may raise ``ValueError`` rather than ``TypeError`` (`13624`)
- Bug in invalid datetime parsing in ``to_datetime`` and ``DatetimeIndex`` may raise ``TypeError`` rather than ``ValueError`` (`11169`, `11287`)
- Bug in ``Index`` created with tz-aware ``Timestamp`` and mismatched ``tz`` option incorrectly coerces timezone (`13692`)
- Bug in ``DatetimeIndex`` with nanosecond frequency does not include timestamp specified with ``end`` (`13672`)
- Bug in ``Series`` when setting a slice with a ``np.timedelta64`` (`14155`)
- Bug in ``Index`` raises ``OutOfBoundsDatetime`` if ``datetime`` exceeds ``datetime64[ns]`` bounds, rather than coercing to ``object`` dtype (`13663`)

- Bug in ``Index`` may ignore specified ``datetime64`` or ``timedelta64`` passed as ``dtype``  (`13981`)
- Bug in ``RangeIndex`` can be created without no arguments rather than raises ``TypeError`` (`13793`)
- Bug in ``.value_counts()`` raises ``OutOfBoundsDatetime`` if data exceeds ``datetime64[ns]`` bounds (`13663`)
- Bug in ``DatetimeIndex`` may raise ``OutOfBoundsDatetime`` if input ``np.datetime64`` has other unit than ``ns`` (`9114`)
- Bug in ``Series`` creation with ``np.datetime64`` which has other unit than ``ns`` as ``object`` dtype results in incorrect values (`13876`)
- Bug in ``resample`` with timedelta data where data was casted to float (`13119`).
- Bug in ``pd.isnull()`` ``pd.notnull()`` raise ``TypeError`` if input datetime-like has other unit than ``ns`` (`13389`)
- Bug in ``pd.merge()`` may raise ``TypeError`` if input datetime-like has other unit than ``ns`` (`13389`)
- Bug in ``HDFStore``/``read_hdf()`` discarded ``DatetimeIndex.name`` if ``tz`` was set (`13884`)
- Bug in ``Categorical.remove_unused_categories()`` changes ``.codes`` dtype to platform int (`13261`)
- Bug in ``groupby`` with ``as_index=False`` returns all NaN's when grouping on multiple columns including a categorical one (`13204`)
- Bug in ``df.groupby(...)[...]`` where getitem with ``Int64Index`` raised an error (`13731`)
- Bug in the CSS classes assigned to ``DataFrame.style`` for index names. Previously they were assigned ``"col_heading level<n> col<c>"`` where ``n`` was the number of levels + 1. Now they are assigned ``"index_name level<n>"``, where ``n`` is the correct level for that MultiIndex.
- Bug where ``pd.read_gbq()`` could throw ``ImportError: No module named discovery`` as a result of a naming conflict with another python package called apiclient  (`13454`)
- Bug in ``Index.union`` returns an incorrect result with a named empty index (`13432`)
- Bugs in ``Index.difference`` and ``DataFrame.join`` raise in Python3 when using mixed-integer indexes (`13432`, `12814`)
- Bug in subtract tz-aware ``datetime.datetime`` from tz-aware ``datetime64`` series (`14088`)
- Bug in ``.to_excel()`` when DataFrame contains a MultiIndex which contains a label with a NaN value (`13511`)
- Bug in invalid frequency offset string like "D1", "-2-3H" may not raise ``ValueError`` (`13930`)
- Bug in ``concat`` and ``groupby`` for hierarchical frames with ``RangeIndex`` levels (`13542`).
- Bug in ``Series.str.contains()`` for Series containing only ``NaN`` values of ``object`` dtype (`14171`)
- Bug in ``agg()`` function on groupby dataframe changes dtype of ``datetime64[ns]`` column to ``float64`` (`12821`)
- Bug in using NumPy ufunc with ``PeriodIndex`` to add or subtract integer raise ``IncompatibleFrequency``. Note that using standard operator like ``+`` or ``-`` is recommended, because standard operators use more efficient path (`13980`)
- Bug in operations on ``NaT`` returning ``float`` instead of ``datetime64[ns]`` (`12941`)
- Bug in ``Series`` flexible arithmetic methods (like ``.add()``) raises ``ValueError`` when ``axis=None`` (`13894`)
- Bug in ``DataFrame.to_csv()`` with ``MultiIndex`` columns in which a stray empty line was added (`6618`)
- Bug in ``DatetimeIndex``, ``TimedeltaIndex`` and ``PeriodIndex.equals()`` may return ``True`` when input isn't ``Index`` but contains the same values (`13107`)
- Bug in assignment against datetime with timezone may not work if it contains datetime near DST boundary (`14146`)
- Bug in ``pd.eval()`` and ``HDFStore`` query truncating long float literals with python 2 (`14241`)
- Bug in ``Index`` raises ``KeyError`` displaying incorrect column when column is not in the df and columns contains duplicate values (`13822`)
- Bug in ``Period`` and ``PeriodIndex`` creating wrong dates when frequency has combined offset aliases (`13874`)

- Bug in ``.to_string()`` when called with an integer ``line_width`` and ``index=False`` raises an UnboundLocalError exception because ``idx`` referenced before assignment.
- Bug in ``eval()`` where the ``resolvers`` argument would not accept a list (`14095`)
- Bugs in ``stack``, ``get_dummies``, ``make_axis_dummies`` which don't preserve categorical dtypes in (multi)indexes (`13854`)
- ``PeriodIndex`` can now accept ``list`` and ``array`` which contains ``pd.NaT`` (`13430`)
- Bug in ``df.groupby`` where ``.median()`` returns arbitrary values if grouped dataframe contains empty bins (`13629`)
- Bug in ``Index.copy()`` where ``name`` parameter was ignored (`14302`)  .. _whatsnew_0.19.0.contributors:  Contributors
~~~  .. contributors:: v0.18.1..v0.19.0

Version 0.19.1 (November 3, 2016)

{{ header }}  .. ipython:: python  from pandas import *  # noqa F401, F403  This is a minor bug-fix release from 0.19.0 and includes some small regression fixes,
bug fixes and performance improvements.
We recommend that all users upgrade to this version.  .. contents:: What's new in v0.19.1  none  .. _whatsnew_0191.performance:  Performance improvements
~~~  - Fixed performance regression in factorization of ``Period`` data (`14338`)
- Fixed performance regression in ``Series.asof(where)`` when ``where`` is a scalar (`14461`)
- Improved performance in ``DataFrame.asof(where)`` when ``where`` is a scalar (`14461`)
- Improved performance in ``.to_json()`` when ``lines=True`` (`14408`)
- Improved performance in certain types of ``loc`` indexing with a MultiIndex (`14551`).  .. _whatsnew_0191.bug_fixes:  Bug fixes
~~~  - Source installs from PyPI will now again work without ``cython`` installed, as in previous versions (`14204`)
- Compat with Cython 0.25 for building (`14496`)
- Fixed regression where user-provided file handles were closed in ``read_csv`` (c engine) (`14418`).
- Fixed regression in ``DataFrame.quantile`` when missing values where present in some columns (`14357`).
- Fixed regression in ``Index.difference`` where the ``freq`` of a ``DatetimeIndex`` was incorrectly set (`14323`)
- Added back ``pandas.core.common.array_equivalent`` with a deprecation warning (`14555`).
- Bug in ``pd.read_csv`` for the C engine in which quotation marks were improperly parsed in skipped rows (`14459`)
- Bug in ``pd.read_csv`` for Python 2.x in which Unicode quote characters were no longer being respected (`14477`)
- Fixed regression in ``Index.append`` when categorical indices were appended (`14545`).
- Fixed regression in ``pd.DataFrame`` where constructor fails when given dict with ``None`` value (`14381`)
- Fixed regression in ``DatetimeIndex._maybe_cast_slice_bound`` when index is empty (`14354`).
- Bug in localizing an ambiguous timezone when a boolean is passed (`14402`)
- Bug in ``TimedeltaIndex`` addition with a Datetime-like object where addition overflow in the negative direction was not being caught (`14068`, `14453`)
- Bug in string indexing against data with ``object`` ``Index`` may raise ``AttributeError`` (`14424`)
- Correctly raise ``ValueError`` on empty input to ``pd.eval()`` and ``df.query()`` (`13139`)
- Bug in ``RangeIndex.intersection`` when result is a empty set (`14364`).
- Bug in groupby-transform broadcasting that could cause incorrect dtype coercion (`14457`)
- Bug in ``Series.__setitem__`` which allowed mutating read-only arrays (`14359`).
- Bug in ``DataFrame.insert`` where multiple calls with duplicate columns can fail (`14291`)
- ``pd.merge()`` will raise ``ValueError`` with non-boolean parameters in passed boolean type arguments (`14434`)
- Bug in ``Timestamp`` where dates very near the minimum (1677-09) could underflow on creation (`14415`)
- Bug in ``pd.concat`` where names of the ``keys`` were not propagated to the resulting ``MultiIndex`` (`14252`)
- Bug in ``pd.concat`` where ``axis`` cannot take string parameters ``'rows'`` or ``'columns'`` (`14369`)
- Bug in ``pd.concat`` with dataframes heterogeneous in length and tuple ``keys`` (`14438`)
- Bug in ``MultiIndex.set_levels`` where illegal level values were still set after raising an error (`13754`)
- Bug in ``DataFrame.to_json`` where ``lines=True`` and a value contained a ``}`` character (`14391`)
- Bug in ``df.groupby`` causing an ``AttributeError`` when grouping a single index frame by a column and the index level (`14327`)
- Bug in ``df.groupby`` where ``TypeError`` raised when ``pd.Grouper(key=...)`` is passed in a list (`14334`)
- Bug in ``pd.pivot_table`` may raise ``TypeError`` or ``ValueError`` when ``index`` or ``columns``  is not scalar and ``values`` is not specified (`14380`)  .. _whatsnew_0.19.1.contributors:  Contributors
~~~  .. contributors:: v0.19.0..v0.19.1

{{ header }}  .. ipython:: python  from pandas import *  # noqa F401, F403  This is a minor bug-fix release in the 0.19.x series and includes some small regression fixes,
We recommend that all users upgrade to this version.  Highlights include:  - Compatibility with Python 3.6
- Added a Pandas Cheat Sheet_. (`13202`).  .. contents:: What's new in v0.19.2  none  .. _whatsnew_0192.enhancements:  Enhancements
~~~  The ``pd.merge_asof()``, added in 0.19.0, gained some improvements:  - ``pd.merge_asof()`` gained ``left_index``/``right_index`` and ``left_by``/``right_by`` arguments (`14253`)
- ``pd.merge_asof()`` can take multiple columns in ``by`` parameter and has specialized dtypes for better performance (`13936`)  .. _whatsnew_0192.performance:  Performance improvements
~~~  - Performance regression with ``PeriodIndex`` (`14822`)
- Performance regression in indexing with getitem (`14930`)
- Improved performance of ``.replace()`` (`12745`)
- Improved performance ``Series`` creation with a datetime index and dictionary data (`14894`)  .. _whatsnew_0192.bug_fixes:  Bug fixes
~~~
- Compat with python 3.6 for pickling of some offsets (`14685`)
- Compat with python 3.6 for some indexing exception types (`14684`, `14689`)
- Compat with python 3.6 for deprecation warnings in the test suite (`14681`)
- Compat with python 3.6 for Timestamp pickles (`14689`)
- Compat with ``dateutil==2.6.0``; segfault reported in the testing suite (`14621`)
- Allow ``nanoseconds`` in ``Timestamp.replace`` as a kwarg (`14621`)
- Bug in ``pd.read_csv`` in which aliasing was being done for ``na_values`` when passed in as a dictionary (`14203`)
- Bug in ``pd.read_csv`` in which column indices for a dict-like ``na_values`` were not being respected (`14203`)
- Bug in ``pd.read_csv`` where reading files fails, if the number of headers is equal to the number of lines in the file (`14515`)
- Bug in ``pd.read_csv`` for the Python engine in which an unhelpful error message was being raised when multi-char delimiters were not being respected with quotes (`14582`)
- Fix bugs (`14734`, `13654`) in ``pd.read_sas`` and ``pandas.io.sas.sas7bdat.SAS7BDATReader`` that caused problems when reading a SAS file incrementally.
- Bug in ``pd.read_csv`` for the Python engine in which an unhelpful error message was being raised when ``skipfooter`` was not being respected by Python's CSV library (`13879`)
- Bug in ``.fillna()`` in which timezone aware datetime64 values were incorrectly rounded (`14872`)
- Bug in ``.groupby(..., sort=True)`` of a non-lexsorted MultiIndex when grouping with multiple levels (`14776`)
- Bug in ``pd.cut`` with negative values and a single bin (`14652`)
- Bug in ``pd.to_numeric`` where a 0 was not unsigned on a ``downcast='unsigned'`` argument (`14401`)
- Bug in plotting regular and irregular timeseries using shared axes  (``sharex=True`` or ``ax.twinx()``) (`13341`, `14322`).
- Bug in not propagating exceptions in parsing invalid datetimes, noted in python 3.6 (`14561`)
- Bug in resampling a ``DatetimeIndex`` in local TZ, covering a DST change, which would raise ``AmbiguousTimeError`` (`14682`)
- Bug in indexing that transformed ``RecursionError`` into ``KeyError`` or ``IndexingError`` (`14554`)
- Bug in ``HDFStore`` when writing a ``MultiIndex`` when using ``data_columns=True`` (`14435`)
- Bug in ``HDFStore.append()`` when writing a ``Series`` and passing a ``min_itemsize`` argument containing a value for the ``index`` (`11412`)
- Bug when writing to a ``HDFStore`` in ``table`` format with a ``min_itemsize`` value for the ``index`` and without asking to append (`10381`)
- Bug in ``Series.groupby.nunique()`` raising an ``IndexError`` for an empty ``Series`` (`12553`)
- Bug in ``DataFrame.nlargest`` and ``DataFrame.nsmallest`` when the index had duplicate values (`13412`)
- Bug in clipboard functions on linux with python2 with unicode and separators (`13747`)
- Bug in clipboard functions on Windows 10 and python 3 (`14362`, `12807`)

- Bug in ``.to_clipboard()`` and Excel compat (`12529`)
- Bug in ``DataFrame.combine_first()`` for integer columns (`14687`).
- Bug in ``pd.read_csv()`` in which the ``dtype`` parameter was not being respected for empty data (`14712`)
- Bug in ``pd.read_csv()`` in which the ``nrows`` parameter was not being respected for large input when using the C engine for parsing (`7626`)
- Bug in ``pd.merge_asof()`` could not handle timezone-aware DatetimeIndex when a tolerance was specified (`14844`)
- Explicit check in ``to_stata`` and ``StataWriter`` for out-of-range values when writing doubles (`14618`)
- Bug in ``.plot(kind='kde')`` which did not drop missing values to generate the KDE Plot, instead generating an empty plot. (`14821`)
- Bug in ``unstack()`` if called with a list of column(s) as an argument, regardless of the dtypes of all columns, they get coerced to ``object`` (`11847`)  .. _whatsnew_0.19.2.contributors:  Contributors
~~~  .. contributors:: v0.19.1..v0.19.2

{{ header }}  This is a major release from 0.19.2 and includes a number of API changes, deprecations, new features,
users upgrade to this version.  Highlights include:  - New ``.agg()`` API for Series/DataFrame similar to the groupby-rolling-resample API's, see `here <whatsnew_0200.enhancements.agg>`
- Integration with the ``feather-format``, including a new top-level ``pd.read_feather()`` and ``DataFrame.to_feather()`` method, see `here <io.feather>`.
- The ``.ix`` indexer has been deprecated, see `here <whatsnew_0200.api_breaking.deprecate_ix>`
- ``Panel`` has been deprecated, see `here <whatsnew_0200.api_breaking.deprecate_panel>`
- Addition of an ``IntervalIndex`` and ``Interval`` scalar type, see `here <whatsnew_0200.enhancements.intervalindex>`
- Improved user API when grouping by index levels in ``.groupby()``, see `here <whatsnew_0200.enhancements.groupby_access>`
- Improved support for ``UInt64`` dtypes, see `here <whatsnew_0200.enhancements.uint64_support>`
- A new orient for JSON serialization, ``orient='table'``, that uses the Table Schema spec and that gives the possibility for a more interactive repr in the Jupyter Notebook, see `here <whatsnew_0200.enhancements.table_schema>`
- Experimental support for exporting styled DataFrames (``DataFrame.style``) to Excel, see `here <whatsnew_0200.enhancements.style_excel>`
- Window binary corr/cov operations now return a MultiIndexed ``DataFrame`` rather than a ``Panel``, as ``Panel`` is now deprecated, see `here <whatsnew_0200.api_breaking.rolling_pairwise>`
- Support for S3 handling now uses ``s3fs``, see `here <whatsnew_0200.api_breaking.s3>`
- Google BigQuery support now uses the ``pandas-gbq`` library, see `here <whatsnew_0200.api_breaking.gbq>`  .. warning::  pandas has changed the internal structure and layout of the code base.  This can affect imports that are not from the top-level ``pandas.*`` namespace, please see the changes `here <whatsnew_0200.privacy>`.  Check the `API Changes <whatsnew_0200.api_breaking>` and `deprecations <whatsnew_0200.deprecations>` before updating.  .. note::  This is a combined release for 0.20.0 and 0.20.1.  Version 0.20.1 contains one additional change for backwards-compatibility with downstream projects using pandas' ``utils`` routines. (`16250`)  .. contents:: What's new in v0.20.0  none  .. _whatsnew_0200.enhancements:  New features
~~~  .. _whatsnew_0200.enhancements.agg:  Method ``agg`` API for DataFrame/Series
^^^  Series & DataFrame have been enhanced to support the aggregation API. This is a familiar API
from groupby, window operations, and resampling. This allows aggregation operations in a concise way
by using `~DataFrame.agg` and `~DataFrame.transform`. The full documentation
is `here <basics.aggregate>` (`1623`).  Here is a sample  .. ipython:: python  df = pd.DataFrame(np.random.randn(10, 3), columns=['A', 'B', 'C'],  index=pd.date_range('1/1/2000', periods=10))  df.iloc[3:7] = np.nan  df  One can operate using string function names, callables, lists, or dictionaries of these.  Using a single function is equivalent to ``.apply``.  .. ipython:: python  df.agg('sum')  Multiple aggregations with a list of functions.  .. ipython:: python  df.agg(['sum', 'min'])  Using a dict provides the ability to apply specific aggregations per column.
You will get a matrix-like output of all of the aggregators. The output has one column
per unique function. Those functions applied to a particular column will be ``NaN``:  .. ipython:: python  df.agg({'A': ['sum', 'min'], 'B': ['min', 'max']})  The API also supports a ``.transform()`` function for broadcasting results.  .. ipython:: python  df.transform(['abs', lambda x: x - x.min()])  When presented with mixed dtypes that cannot be aggregated, ``.agg()`` will only take the valid

aggregations. This is similar to how groupby ``.agg()`` works. (`15015`)  .. ipython:: python  df = pd.DataFrame({'A': [1, 2, 3],  'B': [1., 2., 3.],  'C': ['foo', 'bar', 'baz'],  'D': pd.date_range('20130101', periods=3)})  df.dtypes  .. code-block:: python  In [10]: df.agg(['min', 'sum'])  Out[10]:  A  B  C  D  min  1  1.0  bar 2013-01-01  sum  6  6.0  foobarbaz  NaT  .. _whatsnew_0200.enhancements.dataio_dtype:  Keyword argument ``dtype`` for data IO
^^^  The ``'python'`` engine for `read_csv`, as well as the `read_fwf` function for parsing
fixed-width text files and `read_excel` for parsing Excel files, now accept the ``dtype`` keyword argument for specifying the types of specific columns (`14295`). See the `io docs <io.dtypes>` for more information.  .. ipython:: python  from io import StringIO  .. ipython:: python  data = "a  b\n1  2\n3  4"  pd.read_fwf(StringIO(data)).dtypes  pd.read_fwf(StringIO(data), dtype={'a': 'float64', 'b': 'object'}).dtypes  .. _whatsnew_0120.enhancements.datetime_origin:  Method ``.to_datetime()`` has gained an ``origin`` parameter
^^^  `to_datetime` has gained a new parameter, ``origin``, to define a reference date
from where to compute the resulting timestamps when parsing numerical values with a specific ``unit`` specified. (`11276`, `11745`)  For example, with 1960-01-01 as the starting date:  .. ipython:: python  pd.to_datetime([1, 2, 3], unit='D', origin=pd.Timestamp('1960-01-01'))  The default is set at ``origin='unix'``, which defaults to ``1970-01-01 000``, which is
commonly called 'unix epoch' or POSIX time. This was the previous default, so this is a backward compatible change.  .. ipython:: python  pd.to_datetime([1, 2, 3], unit='D')  .. _whatsnew_0200.enhancements.groupby_access:  GroupBy enhancements
^^^  Strings passed to ``DataFrame.groupby()`` as the ``by`` parameter may now reference either column names or index level names. Previously, only column names could be referenced. This allows to easily group by a column and index level at the same time. (`5677`)  .. ipython:: python  arrays = [['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],  ['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two']]  index = pd.MultiIndex.from_arrays(arrays, names=['first', 'second'])  df = pd.DataFrame({'A': [1, 1, 1, 1, 2, 2, 3, 3],  'B': np.arange(8)},  index=index)  df  df.groupby(['second', 'A']).sum()  .. _whatsnew_0200.enhancements.compressed_urls:  Better support for compressed URLs in ``read_csv``
^^^  The compression code was refactored (`12688`). As a result, reading
dataframes from URLs in `read_csv` or `read_table` now supports
additional compression methods: ``xz``, ``bz2``, and ``zip`` (`14570`).
Previously, only ``gzip`` compression was supported. By default, compression of
URLs and paths are now inferred using their file extensions. Additionally,
support for bz2 compression in the python 2 C-engine improved (`14874`).  .. ipython:: python  url = ('https://github.com/{repo}/raw/{branch}/{path}'  .format(repo='pandas-dev/pandas',  branch='main',  path='pandas/tests/io/parser/data/salaries.csv.bz2'))  # default, infer compression  df = pd.read_csv(url, sep='\t', compression='infer')  # explicitly specify compression  df = pd.read_csv(url, sep='\t', compression='bz2')  df.head(2)  .. _whatsnew_0200.enhancements.pickle_compression:  Pickle file IO now supports compression
^^^  `read_pickle`, `DataFrame.to_pickle` and `Series.to_pickle`
can now read from and write to compressed pickle files. Compression methods
can be an explicit parameter or be inferred from the file extension.

See `the docs here. <io.pickle.compression>`  .. ipython:: python  df = pd.DataFrame({'A': np.random.randn(1000),  'B': 'foo',  'C': pd.date_range('20130101', periods=1000, freq='s')})  Using an explicit compression type  .. ipython:: python  df.to_pickle("data.pkl.compress", compression="gzip")  rt = pd.read_pickle("data.pkl.compress", compression="gzip")  rt.head()  The default is to infer the compression type from the extension (``compression='infer'``):  .. ipython:: python  df.to_pickle("data.pkl.gz")  rt = pd.read_pickle("data.pkl.gz")  rt.head()  df["A"].to_pickle("s1.pkl.bz2")  rt = pd.read_pickle("s1.pkl.bz2")  rt.head()  .. ipython:: python  import os  os.remove("data.pkl.compress")  os.remove("data.pkl.gz")  os.remove("s1.pkl.bz2")  .. _whatsnew_0200.enhancements.uint64_support:  UInt64 support improved
^^^  pandas has significantly improved support for operations involving unsigned,
or purely non-negative, integers. Previously, handling these integers would
result in improper rounding or data-type casting, leading to incorrect results.
Notably, a new numerical index, ``UInt64Index``, has been created (`14937`)  .. code-block:: ipython  In [1]: idx = pd.UInt64Index([1, 2, 3])  In [2]: df = pd.DataFrame({'A': ['a', 'b', 'c']}, index=idx)  In [3]: df.index  Out[3]: UInt64Index([1, 2, 3], dtype='uint64')  - Bug in converting object elements of array-like objects to unsigned 64-bit integers (`4471`, `14982`)
- Bug in ``Series.unique()`` in which unsigned 64-bit integers were causing overflow (`14721`)
- Bug in ``DataFrame`` construction in which unsigned 64-bit integer elements were being converted to objects (`14881`)
- Bug in ``pd.read_csv()`` in which unsigned 64-bit integer elements were being improperly converted to the wrong data types (`14983`)
- Bug in ``pd.unique()`` in which unsigned 64-bit integers were causing overflow (`14915`)
- Bug in ``pd.value_counts()`` in which unsigned 64-bit integers were being erroneously truncated in the output (`14934`)  .. _whatsnew_0200.enhancements.groupy_categorical:  GroupBy on categoricals
^^^  In previous versions, ``.groupby(..., sort=False)`` would fail with a ``ValueError`` when grouping on a categorical series with some categories not appearing in the data. (`13179`)  .. ipython:: python  chromosomes = np.r_[np.arange(1, 23).astype(str), ['X', 'Y']]  df = pd.DataFrame({  'A': np.random.randint(100),  'B': np.random.randint(100),  'C': np.random.randint(100),  'chromosomes': pd.Categorical(np.random.choice(chromosomes, 100),  categories=chromosomes,  ordered=True)})  df  **Previous behavior**:  .. code-block:: ipython

In [3]: df[df.chromosomes != '1'].groupby('chromosomes', sort=False).sum()
  ValueError: items in new_categories are not the same as in old categories  **New behavior**:  .. ipython:: python  df[df.chromosomes != '1'].groupby('chromosomes', sort=False).sum()  .. _whatsnew_0200.enhancements.table_schema:  Table schema output
^^^  The new orient ``'table'`` for `DataFrame.to_json`
will generate a `Table Schema`_ compatible string representation of
the data.  .. ipython:: python  df = pd.DataFrame(  {'A': [1, 2, 3],  'B': ['a', 'b', 'c'],  'C': pd.date_range('2016-01-01', freq='d', periods=3)},  index=pd.Index(range(3), name='idx'))  df  df.to_json(orient='table')  See `IO: Table Schema for more information <io.table_schema>`.  Additionally, the repr for ``DataFrame`` and ``Series`` can now publish
this JSON Table schema representation of the Series or DataFrame if you are
using IPython (or another frontend like `nteract`_ using the Jupyter messaging
protocol).
This gives frontends like the Jupyter notebook and `nteract`_
more flexibility in how they display pandas objects, since they have
more information about the data.
You must enable this by setting the ``display.html.table_schema`` option to ``True``.  .. _Table Schema: http://specs.frictionlessdata.io/json-table-schema/
.. _nteract: https://nteract.io/  .. _whatsnew_0200.enhancements.scipy_sparse:  SciPy sparse matrix from/to SparseDataFrame
^^^  pandas now supports creating sparse dataframes directly from ``scipy.sparse.spmatrix`` instances.
See the `documentation <sparse.scipysparse>` for more information. (`4343`)  All sparse formats are supported, but matrices that are not in `COOrdinate <scipy.sparse>` format will be converted, copying data as needed.  .. code-block:: python  from scipy.sparse import csr_matrix  arr = np.random.random(size=(1000, 5))  arr[arr < .9] = 0  sp_arr = csr_matrix(arr)  sp_arr  sdf = pd.SparseDataFrame(sp_arr)  sdf  To convert a ``SparseDataFrame`` back to sparse SciPy matrix in COO format, you can use:  .. code-block:: python  sdf.to_coo()  .. _whatsnew_0200.enhancements.style_excel:  Excel output for styled DataFrames
^^^  Experimental support has been added to export ``DataFrame.style`` formats to Excel using the ``openpyxl`` engine. (`15530`)  For example, after running the following, ``styled.xlsx`` renders as below:  .. ipython:: python  np.random.seed(24)  df = pd.DataFrame({'A': np.linspace(1, 10, 10)})  df = pd.concat([df, pd.DataFrame(np.random.RandomState(24).randn(10, 4),  columns=list('BCDE'))],  axis=1)  df.iloc[0, 2] = np.nan  df  styled = (df.style  .applymap(lambda val: 'color:red;' if val < 0 else 'color:black;')  .highlight_max())  styled.to_excel('styled.xlsx', engine='openpyxl')  .. ipython:: python  import os  os.remove('styled.xlsx')  See the `Style documentation </user_guide/style.ipynb#Export-to-Excel>` for more detail.  .. _whatsnew_0200.enhancements.intervalindex:  IntervalIndex
^^^  pandas has gained an ``IntervalIndex`` with its own dtype, ``interval`` as well as the ``Interval`` scalar type. These allow first-class support for interval
notation, specifically as a return type for the categories in `cut` and `qcut`. The ``IntervalIndex`` allows some unique indexing, see the
`docs <advanced.intervalindex>`. (`7640`, `8625`)  .. warning::  These indexing behaviors of the IntervalIndex are provisional and may change in a future version of pandas. Feedback on usage is welcome.  Previous behavior:  The returned categories were strings, representing Intervals  .. code-block:: ipython  In [1]: c = pd.cut(range(4), bins=2)  In [2]: c  Out[2]:  [(-0.003, 1.5], (-0.003, 1.5], (1.5, 3], (1.5, 3]]  Categories (2, object): [(-0.003, 1.5] < (1.5, 3]]  In [3]: c.categories  Out[3]: Index(['(-0.003, 1.5]', '(1.5, 3]'], dtype='object')  New behavior:  .. ipython:: python  c = pd.cut(range(4), bins=2)  c  c.categories  Furthermore, this allows one to bin *other* data with these same bins, with ``NaN`` representing a missing

value similar to other dtypes.  .. ipython:: python  pd.cut([0, 3, 5, 1], bins=c.categories)  An ``IntervalIndex`` can also be used in ``Series`` and ``DataFrame`` as the index.  .. ipython:: python  df = pd.DataFrame({'A': range(4),  'B': pd.cut([0, 3, 1, 1], bins=c.categories)  }).set_index('B')  df  Selecting via a specific interval:  .. ipython:: python  df.loc[pd.Interval(1.5, 3.0)]  Selecting via a scalar value that is contained *in* the intervals.  .. ipython:: python  df.loc[0]  .. _whatsnew_0200.enhancements.other:  Other enhancements
^^^  - ``DataFrame.rolling()`` now accepts the parameter ``closed='right'|'left'|'both'|'neither'`` to choose the rolling window-endpoint closedness. See the `documentation <window.endpoints>` (`13965`)
- ``Series.str.replace()`` now accepts a callable, as replacement, which is passed to ``re.sub`` (`15055`)
- ``Series.str.replace()`` now accepts a compiled regular expression as a pattern (`15446`)
- ``Series.sort_index`` accepts parameters ``kind`` and ``na_position`` (`13589`, `1444`)
- ``DataFrame`` and ``DataFrame.groupby()``  have gained a ``nunique()`` method to count the distinct values over an axis (`14336`, `15197`).
- ``DataFrame`` has gained a ``melt()`` method, equivalent to ``pd.melt()``, for unpivoting from a wide to long format (`12640`).
- ``pd.read_excel()`` now preserves sheet order when using ``sheetname=None`` (`9930`)
- Multiple offset aliases with decimal points are now supported (e.g. ``0.5min`` is parsed as ``30s``) (`8419`)
- ``.isnull()`` and ``.notnull()`` have been added to ``Index`` object to make them more consistent with the ``Series`` API (`15300`)
- New ``UnsortedIndexError`` (subclass of ``KeyError``) raised when indexing/slicing into an  unsorted MultiIndex (`11897`). This allows differentiation between errors due to lack  of sorting or an incorrect key. See `here <advanced.unsorted>`
- ``MultiIndex`` has gained a ``.to_frame()`` method to convert to a ``DataFrame`` (`12397`)
- ``pd.cut`` and ``pd.qcut`` now support datetime64 and timedelta64 dtypes (`14714`, `14798`)
- ``pd.qcut`` has gained the ``duplicates='raise'|'drop'`` option to control whether to raise on duplicated edges (`7751`)
- ``Series`` provides a ``to_excel`` method to output Excel files (`8825`)
- The ``usecols`` argument in ``pd.read_csv()`` now accepts a callable function as a value  (`14154`)
- The ``skiprows`` argument in ``pd.read_csv()`` now accepts a callable function as a value  (`10882`)
- The ``nrows`` and ``chunksize`` arguments in ``pd.read_csv()`` are supported if both are passed (`6774`, `15755`)
- ``DataFrame.plot`` now prints a title above each subplot if ``suplots=True`` and ``title`` is a list of strings (`14753`)
- `DataFrame.plot`` can pass the matplotlib 2.0 default color cycle as a single string as color parameter, see `here_. (`15516`)
- ``Series.interpolate()`` now supports timedelta as an index type with ``method='time'`` (`6424`)
- Addition of a ``level`` keyword to ``DataFrame/Series.rename`` to rename  labels in the specified level of a MultiIndex (`4160`).
- ``DataFrame.reset_index()`` will now interpret a tuple ``index.name`` as a key spanning across levels of ``columns``, if this is a ``MultiIndex`` (`16164`)
- ``Timedelta.isoformat`` method added for formatting Timedeltas as an `ISO 8601 duration`_. See the `Timedelta docs <timedeltas.isoformat>` (`15136`)
- ``.select_dtypes()`` now allows the string ``datetimetz`` to generically select datetimes with tz (`14910`)
- The ``.to_latex()`` method will now accept ``multicolumn`` and ``multirow`` arguments to use the accompanying LaTeX enhancements
- ``pd.merge_asof()`` gained the option ``direction='backward'|'forward'|'nearest'`` (`14887`)
- ``Series/DataFrame.asfreq()`` have gained a ``fill_value`` parameter, to fill missing values (`3715`).

- ``Series/DataFrame.resample.asfreq`` have gained a ``fill_value`` parameter, to fill missing values during resampling (`3715`).
- `pandas.util.hash_pandas_object` has gained the ability to hash a ``MultiIndex`` (`15224`)
- ``Series/DataFrame.squeeze()`` have gained the ``axis`` parameter. (`15339`)
- ``DataFrame.to_excel()`` has a new ``freeze_panes`` parameter to turn on Freeze Panes when exporting to Excel (`15160`)
- ``pd.read_html()`` will parse multiple header rows, creating a MultiIndex header. (`13434`).
- HTML table output skips ``colspan`` or ``rowspan`` attribute if equal to 1. (`15403`)
- `pandas.io.formats.style.Styler` template now has blocks for easier extension, see the `example notebook </user_guide/style.ipynb#Subclassing>` (`15649`)
- `Styler.render() <pandas.io.formats.style.Styler.render>` now accepts ``**kwargs`` to allow user-defined variables in the template (`15649`)
- Compatibility with Jupyter notebook 5.0; MultiIndex column labels are left-aligned and MultiIndex row-labels are top-aligned (`15379`)
- ``TimedeltaIndex`` now has a custom date-tick formatter specifically designed for nanosecond level precision (`8711`)
- ``pd.api.types.union_categoricals`` gained the ``ignore_ordered`` argument to allow ignoring the ordered attribute of unioned categoricals (`13410`). See the `categorical union docs <categorical.union>` for more information.
- ``DataFrame.to_latex()`` and ``DataFrame.to_string()`` now allow optional header aliases. (`15536`)
- Re-enable the ``parse_dates`` keyword of ``pd.read_excel()`` to parse string columns as dates (`14326`)
- Added ``.empty`` property to subclasses of ``Index``. (`15270`)
- Enabled floor division for ``Timedelta`` and ``TimedeltaIndex`` (`15828`)
- ``pandas.io.json.json_normalize()`` gained the option ``errors='ignore'|'raise'``; the default is ``errors='raise'`` which is backward compatible. (`14583`)
- ``pandas.io.json.json_normalize()`` with an empty ``list`` will return an empty ``DataFrame`` (`15534`)
- ``pandas.io.json.json_normalize()`` has gained a ``sep`` option that accepts ``str`` to separate joined fields; the default is ".", which is backward compatible. (`14883`)
- `MultiIndex.remove_unused_levels` has been added to facilitate `removing unused levels <advanced.shown_levels>`. (`15694`)
- ``pd.read_csv()`` will now raise a ``ParserError`` error whenever any parsing error occurs (`15913`, `15925`)
- ``pd.read_csv()`` now supports the ``error_bad_lines`` and ``warn_bad_lines`` arguments for the Python parser (`15925`)
- The ``display.show_dimensions`` option can now also be used to specify  whether the length of a ``Series`` should be shown in its repr (`7117`).
- ``parallel_coordinates()`` has gained a ``sort_labels`` keyword argument that sorts class labels and the colors assigned to them (`15908`)
- Options added to allow one to turn on/off using ``bottleneck`` and ``numexpr``, see `here <basics.accelerate>` (`16157`)
- ``DataFrame.style.bar()`` now accepts two more options to further customize the bar chart. Bar alignment is set with ``align='left'|'mid'|'zero'``, the default is "left", which is backward compatible; You can now pass a list of ``color=[color_negative, color_positive]``. (`14757`)  .. _ISO 8601 duration: https://en.wikipedia.org/wiki/ISO_8601#Durations  .. _whatsnew_0200.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew.api_breaking.io_compat:  Possible incompatibility for HDF5 formats created with pandas < 0.13.0
^^^  ``pd.TimeSeries`` was deprecated officially in 0.17.0, though has already been an alias since 0.13.0. It has
been dropped in favor of ``pd.Series``. (`15098`).  This *may* cause HDF5 files that were created in prior versions to become unreadable if ``pd.TimeSeries``

was used. This is most likely to be for pandas < 0.13.0. If you find yourself in this situation.
You can use a recent prior version of pandas to read in your HDF5 files,
then write them out again after applying the procedure below.  .. code-block:: ipython  In [2]: s = pd.TimeSeries([1, 2, 3], index=pd.date_range('20130101', periods=3))  In [3]: s  Out[3]:  2013-01-01  1  2013-01-02  2  2013-01-03  3  Freq: D, dtype: int64  In [4]: type(s)  Out[4]: pandas.core.series.TimeSeries  In [5]: s = pd.Series(s)  In [6]: s  Out[6]:  2013-01-01  1  2013-01-02  2  2013-01-03  3  Freq: D, dtype: int64  In [7]: type(s)  Out[7]: pandas.core.series.Series  .. _whatsnew_0200.api_breaking.index_map:  Map on Index types now return other Index types
^^^  ``map`` on an ``Index`` now returns an ``Index``, not a numpy array (`12766`)  .. ipython:: python  idx = pd.Index([1, 2])  idx  mi = pd.MultiIndex.from_tuples([(1, 2), (2, 4)])  mi  Previous behavior:  .. code-block:: ipython  In [5]: idx.map(lambda x: x * 2)  Out[5]: array([2, 4])  In [6]: idx.map(lambda x: (x, x * 2))  Out[6]: array([(1, 2), (2, 4)], dtype=object)  In [7]: mi.map(lambda x: x)  Out[7]: array([(1, 2), (2, 4)], dtype=object)  In [8]: mi.map(lambda x: x[0])  Out[8]: array([1, 2])  New behavior:  .. ipython:: python  idx.map(lambda x: x * 2)  idx.map(lambda x: (x, x * 2))  mi.map(lambda x: x)  mi.map(lambda x: x[0])  ``map`` on a ``Series`` with ``datetime64`` values may return ``int64`` dtypes rather than ``int32``  .. ipython:: python  s = pd.Series(pd.date_range('2011-01-02T00:00', '2011-01-02T02:00', freq='H')  .tz_localize('Asia/Tokyo'))  s  Previous behavior:  .. code-block:: ipython  In [9]: s.map(lambda x: x.hour)  Out[9]:  0  0  1  1  2  2  dtype: int32  New behavior:  .. ipython:: python  s.map(lambda x: x.hour)  .. _whatsnew_0200.api_breaking.index_dt_field:  Accessing datetime fields of Index now return Index
^^^  The datetime-related attributes (see `here <timeseries.components>`
for an overview) of ``DatetimeIndex``, ``PeriodIndex`` and ``TimedeltaIndex`` previously
returned numpy arrays. They will now return a new ``Index`` object, except
in the case of a boolean field, where the result will still be a boolean ndarray. (`15022`)  Previous behaviour:  .. code-block:: ipython  In [1]: idx = pd.date_range("2015-01-01", periods=5, freq='10H')  In [2]: idx.hour  Out[2]: array([ 0, 10, 20,  6, 16], dtype=int32)  New behavior:  .. ipython:: python  idx = pd.date_range("2015-01-01", periods=5, freq='10H')  idx.hour  This has the advantage that specific ``Index`` methods are still available on the
result. On the other hand, this might have backward incompatibilities: e.g.
compared to numpy arrays, ``Index`` objects are not mutable. To get the original
ndarray, you can always convert explicitly using ``np.asarray(idx.hour)``.  .. _whatsnew_0200.api_breaking.unique:  pd.unique will now be consistent with extension types
^^^  In prior versions, using `Series.unique` and `pandas.unique` on ``Categorical`` and tz-aware

data-types would yield different return types. These are now made consistent. (`15903`)  - Datetime tz-aware  Previous behaviour:  .. code-block:: ipython  # Series  In [5]: pd.Series([pd.Timestamp('20160101', tz='US/Eastern'),  ...:  pd.Timestamp('20160101', tz='US/Eastern')]).unique()  Out[5]: array([Timestamp('2016-01-01 000-0500', tz='US/Eastern')], dtype=object)  In [6]: pd.unique(pd.Series([pd.Timestamp('20160101', tz='US/Eastern'),  ...:  pd.Timestamp('20160101', tz='US/Eastern')]))  Out[6]: array(['2016-01-01T0500.000'], dtype='datetime64[ns]')  # Index  In [7]: pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),  ...:  pd.Timestamp('20160101', tz='US/Eastern')]).unique()  Out[7]: DatetimeIndex(['2016-01-01 000-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)  In [8]: pd.unique([pd.Timestamp('20160101', tz='US/Eastern'),  ...:  pd.Timestamp('20160101', tz='US/Eastern')])  Out[8]: array(['2016-01-01T0500.000'], dtype='datetime64[ns]')  New behavior:  .. ipython:: python  # Series, returns an array of Timestamp tz-aware  pd.Series([pd.Timestamp(r'20160101', tz=r'US/Eastern'),  pd.Timestamp(r'20160101', tz=r'US/Eastern')]).unique()  pd.unique(pd.Series([pd.Timestamp('20160101', tz='US/Eastern'),  pd.Timestamp('20160101', tz='US/Eastern')]))  # Index, returns a DatetimeIndex  pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),  pd.Timestamp('20160101', tz='US/Eastern')]).unique()  pd.unique(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),  pd.Timestamp('20160101', tz='US/Eastern')]))  - Categoricals  Previous behaviour:  .. code-block:: ipython  In [1]: pd.Series(list('baabc'), dtype='category').unique()  Out[1]:  [b, a, c]  Categories (3, object): [b, a, c]  In [2]: pd.unique(pd.Series(list('baabc'), dtype='category'))  Out[2]: array(['b', 'a', 'c'], dtype=object)  New behavior:  .. ipython:: python  # returns a Categorical  pd.Series(list('baabc'), dtype='category').unique()  pd.unique(pd.Series(list('baabc'), dtype='category'))  .. _whatsnew_0200.api_breaking.s3:  S3 file handling
^^^  pandas now uses s3fs for handling S3 connections. This shouldn't break
any code. However, since ``s3fs`` is not a required dependency, you will need to install it separately, like ``boto``
in prior versions of pandas. (`11915`).  .. _whatsnew_0200.api_breaking.partial_string_indexing:  Partial string indexing changes
^^^  `DatetimeIndex Partial String Indexing <timeseries.partialindexing>` now works as an exact match, provided that string resolution coincides with index resolution, including a case when both are seconds (`14826`). See `Slice vs. Exact Match <timeseries.slice_vs_exact_match>` for details.  .. ipython:: python  df = pd.DataFrame({'a': [1, 2, 3]}, pd.DatetimeIndex(['2011-12-31 2359',  '2012-01-01 000',  '2012-01-01 0001']))
Previous behavior:  .. code-block:: ipython  In [4]: df['2011-12-31 2359']  Out[4]:  a  2011-12-31 2359  1  In [5]: df['a']['2011-12-31 2359']  Out[5]:  2011-12-31 2359  1  Name: a, dtype: int64  New behavior:  .. code-block:: ipython  In [4]: df['2011-12-31 2359']  KeyError: '2011-12-31 2359'  In [5]: df['a']['2011-12-31 2359']  Out[5]: 1  .. _whatsnew_0200.api_breaking.concat_dtypes:  Concat of different float dtypes will not automatically upcast
^^^  Previously, ``concat`` of multiple objects with different ``float`` dtypes would automatically upcast results to a dtype of ``float64``.
Now the smallest acceptable dtype will be used (`13247`)  .. ipython:: python  df1 = pd.DataFrame(np.array([1.0], dtype=np.float32, ndmin=2))  df1.dtypes  df2 = pd.DataFrame(np.array([np.nan], dtype=np.float32, ndmin=2))  df2.dtypes  Previous behavior:  .. code-block:: ipython  In [7]: pd.concat([df1, df2]).dtypes  Out[7]:  0  float64  dtype: object  New behavior:  .. ipython:: python  pd.concat([df1, df2]).dtypes  .. _whatsnew_0200.api_breaking.gbq:  pandas Google BigQuery support has moved

^^^  pandas has split off Google BigQuery support into a separate package ``pandas-gbq``. You can ``conda install pandas-gbq -c conda-forge`` or
``pip install pandas-gbq`` to get it. The functionality of `read_gbq` and `DataFrame.to_gbq` remain the same with the
currently released version of `pandas-gbq=0.1.4``. Documentation is now hosted `here_  (`15347`)  .. _whatsnew_0200.api_breaking.memory_usage:  Memory usage for Index is more accurate
^^^  In previous versions, showing ``.memory_usage()`` on a pandas structure that has an index, would only include actual index values and not include structures that facilitated fast indexing. This will generally be different for ``Index`` and ``MultiIndex`` and less-so for other index types. (`15237`)  Previous behavior:  .. code-block:: ipython  In [8]: index = pd.Index(['foo', 'bar', 'baz'])  In [9]: index.memory_usage(deep=True)  Out[9]: 180  In [10]: index.get_loc('foo')  Out[10]: 0  In [11]: index.memory_usage(deep=True)  Out[11]: 180  New behavior:  .. code-block:: ipython  In [8]: index = pd.Index(['foo', 'bar', 'baz'])  In [9]: index.memory_usage(deep=True)  Out[9]: 180  In [10]: index.get_loc('foo')  Out[10]: 0  In [11]: index.memory_usage(deep=True)  Out[11]: 260  .. _whatsnew_0200.api_breaking.sort_index:  DataFrame.sort_index changes
^^^  In certain cases, calling ``.sort_index()`` on a MultiIndexed DataFrame would return the *same* DataFrame without seeming to sort.
This would happen with a ``lexsorted``, but non-monotonic levels. (`15622`, `15687`, `14015`, `13431`, `15797`)  This is *unchanged* from prior versions, but shown for illustration purposes:  .. ipython:: python  df = pd.DataFrame(np.arange(6), columns=['value'],  index=pd.MultiIndex.from_product([list('BA'), range(3)]))  df  .. code-block:: python  In [87]: df.index.is_lexsorted()  Out[87]: False  In [88]: df.index.is_monotonic  Out[88]: False  Sorting works as expected  .. ipython:: python  df.sort_index()  .. code-block:: python  In [90]: df.sort_index().index.is_lexsorted()  Out[90]: True  In [91]: df.sort_index().index.is_monotonic  Out[91]: True  However, this example, which has a non-monotonic 2nd level,
doesn't behave as desired.  .. ipython:: python  df = pd.DataFrame({'value': [1, 2, 3, 4]},  index=pd.MultiIndex([['a', 'b'], ['bb', 'aa']],  [[0, 0, 1, 1], [0, 1, 0, 1]]))  df  Previous behavior:  .. code-block:: python  In [11]: df.sort_index()  Out[11]:  value  a bb  1  aa  2  b bb  3  aa  4  In [14]: df.sort_index().index.is_lexsorted()  Out[14]: True  In [15]: df.sort_index().index.is_monotonic  Out[15]: False  New behavior:  .. code-block:: python  In [94]: df.sort_index()  Out[94]:  value  a aa  2  bb  1  b aa  4  bb  3  [4 rows x 1 columns]  In [95]: df.sort_index().index.is_lexsorted()  Out[95]: True  In [96]: df.sort_index().index.is_monotonic  Out[96]: True  .. _whatsnew_0200.api_breaking.groupby_describe:  GroupBy describe formatting
^^^  The output formatting of ``groupby.describe()`` now labels the ``describe()`` metrics in the columns instead of the index.
This format is consistent with ``groupby.agg()`` when applying multiple functions at once. (`4792`)  Previous behavior:  .. code-block:: ipython  In [1]: df = pd.DataFrame({'A': [1, 1, 2, 2], 'B': [1, 2, 3, 4]})  In [2]: df.groupby('A').describe()  Out[2]:  B  A  1 count  2.000  mean  1.5000  std  0.707107  min  1.000  25%  1.25000  50%  1.5000  75%  1.75000  max  2.000  2 count  2.000  mean  3.5000  std  0.707107  min  3.000  25%  3.25000  50%  3.5000  75%  3.75000  max  4.000  In [3]: df.groupby('A').agg([np.mean, np.std, np.min, np.max])  Out[3]:  B  mean  std amin amax  A  1  1.5  0.707107  1  2  2  3.5  0.707107  3  4  New behavior:  .. ipython:: python  df = pd.DataFrame({'A': [1, 1, 2, 2], 'B': [1, 2, 3, 4]})  df.groupby('A').describe()  df.groupby('A').agg([np.mean, np.std, np.min, np.max])  .. _whatsnew_0200.api_breaking.rolling_pairwise:  Window binary corr/cov operations return a MultiIndex DataFrame

^^^  A binary window operation, like ``.corr()`` or ``.cov()``, when operating on a ``.rolling(..)``, ``.expanding(..)``, or ``.ewm(..)`` object,
will now return a 2-level ``MultiIndexed DataFrame`` rather than a ``Panel``, as ``Panel`` is now deprecated,
see `here <whatsnew_0200.api_breaking.deprecate_panel>`. These are equivalent in function,
but a MultiIndexed ``DataFrame`` enjoys more support in pandas.
See the section on `Windowed Binary Operations <window.cov_corr>` for more information. (`15677`)  .. ipython:: python  np.random.seed(1234)  df = pd.DataFrame(np.random.rand(100, 2),  columns=pd.Index(['A', 'B'], name='bar'),  index=pd.date_range('20160101',  periods=100, freq='D', name='foo'))  df.tail()  Previous behavior:  .. code-block:: ipython  In [2]: df.rolling(12).corr()  Out[2]:  <class 'pandas.core.panel.Panel'>  Dimensions: 100 (items) x 2 (major_axis) x 2 (minor_axis)  Items axis: 2016-01-01 000 to 2016-04-09 000  Major_axis axis: A to B  Minor_axis axis: A to B  New behavior:  .. ipython:: python  res = df.rolling(12).corr()  res.tail()  Retrieving a correlation matrix for a cross-section  .. ipython:: python  df.rolling(12).corr().loc['2016-04-07']  .. _whatsnew_0200.api_breaking.hdfstore_where:  HDFStore where string comparison
^^^  In previous versions most types could be compared to string column in a ``HDFStore``
usually resulting in an invalid comparison, returning an empty result frame. These comparisons will now raise a
``TypeError`` (`15492`)  .. ipython:: python  df = pd.DataFrame({'unparsed_date': ['2014-01-01', '2014-01-01']})  df.to_hdf('store.h5', 'key', format='table', data_columns=True)  df.dtypes  Previous behavior:  .. code-block:: ipython  In [4]: pd.read_hdf('store.h5', 'key', where='unparsed_date > ts')  File "<string>", line 1  (unparsed_date > 1970-01-01 0001.388552400)  ^  SyntaxError: invalid token  New behavior:  .. code-block:: ipython  In [18]: ts = pd.Timestamp('2014-01-01')  In [19]: pd.read_hdf('store.h5', 'key', where='unparsed_date > ts')  TypeError: Cannot compare 2014-01-01 000 of  type <class 'pandas.tslib.Timestamp'> to string column  .. ipython:: python  import os  os.remove('store.h5')  .. _whatsnew_0200.api_breaking.index_order:  Index.intersection and inner join now preserve the order of the left Index
^^^  `Index.intersection` now preserves the order of the calling ``Index`` (left)
instead of the other ``Index`` (right) (`15582`). This affects inner
joins, `DataFrame.join` and `merge`, and the ``.align`` method.  - ``Index.intersection``  .. ipython:: python  left = pd.Index([2, 1, 0])  left  right = pd.Index([1, 2, 3])  right  Previous behavior:  .. code-block:: ipython  In [4]: left.intersection(right)  Out[4]: Int64Index([1, 2], dtype='int64')  New behavior:  .. ipython:: python  left.intersection(right)  - ``DataFrame.join`` and ``pd.merge``  .. ipython:: python  left = pd.DataFrame({'a': [20, 10, 0]}, index=[2, 1, 0])  left  right = pd.DataFrame({'b': [100, 200, 300]}, index=[1, 2, 3])  right  Previous behavior:  .. code-block:: ipython  In [4]: left.join(right, how='inner')  Out[4]:  a  b  1  10  100  2  20  200  New behavior:  .. ipython:: python  left.join(right, how='inner')  .. _whatsnew_0200.api_breaking.pivot_table:  Pivot table always returns a DataFrame
^^^  The documentation for `pivot_table` states that a ``DataFrame`` is *always* returned. Here a bug
is fixed that allowed this to return a ``Series`` under certain circumstance. (`4386`)  .. ipython:: python  df = pd.DataFrame({'col1': [3, 4, 5],  'col2': ['C', 'D', 'E'],  'col3': [1, 3, 9]})  df  Previous behavior:  .. code-block:: ipython  In [2]: df.pivot_table('col1', index=['col3', 'col2'], aggfunc=np.sum)  Out[2]:  col3  col2  1  C  3  3  D  4  9  E  5  Name: col1, dtype: int64  New behavior:  .. ipython:: python  df.pivot_table('col1', index=['col3', 'col2'], aggfunc=np.sum)  .. _whatsnew_0200.api:  Other API changes

^^^  - ``numexpr`` version is now required to be >= 2.4.6 and it will not be used at all if this requisite is not fulfilled (`15213`).
- ``CParserError`` has been renamed to ``ParserError`` in ``pd.read_csv()`` and will be removed in the future (`12665`)
- ``SparseArray.cumsum()`` and ``SparseSeries.cumsum()`` will now always return ``SparseArray`` and ``SparseSeries`` respectively (`12855`)
- ``DataFrame.applymap()`` with an empty ``DataFrame`` will return a copy of the empty ``DataFrame`` instead of a ``Series`` (`8222`)
- ``Series.map()`` now respects default values of dictionary subclasses with a ``__missing__`` method, such as ``collections.Counter`` (`15999`)
- ``.loc`` has compat with ``.ix`` for accepting iterators, and NamedTuples (`15120`)
- ``interpolate()`` and ``fillna()`` will raise a ``ValueError`` if the ``limit`` keyword argument is not greater than 0. (`9217`)
- ``pd.read_csv()`` will now issue a ``ParserWarning`` whenever there are conflicting values provided by the ``dialect`` parameter and the user (`14898`)
- ``pd.read_csv()`` will now raise a ``ValueError`` for the C engine if the quote character is larger than one byte (`11592`)
- ``inplace`` arguments now require a boolean value, else a ``ValueError`` is thrown (`14189`)
- ``pandas.api.types.is_datetime64_ns_dtype`` will now report ``True`` on a tz-aware dtype, similar to ``pandas.api.types.is_datetime64_any_dtype``
- ``DataFrame.asof()`` will return a null filled ``Series`` instead the scalar ``NaN`` if a match is not found (`15118`)
- Specific support for ``copy.copy()`` and ``copy.deepcopy()`` functions on NDFrame objects (`15444`)
- ``Series.sort_values()`` accepts a one element list of bool for consistency with the behavior of ``DataFrame.sort_values()`` (`15604`)
- ``.merge()`` and ``.join()`` on ``category`` dtype columns will now preserve the category dtype when possible (`10409`)
- ``SparseDataFrame.default_fill_value`` will be 0, previously was ``nan`` in the return from ``pd.get_dummies(..., sparse=True)`` (`15594`)
- The default behaviour of ``Series.str.match`` has changed from extracting  groups to matching the pattern. The extracting behaviour was deprecated  since pandas version 0.13.0 and can be done with the ``Series.str.extract``  method (`5224`). As a consequence, the ``as_indexer`` keyword is  ignored (no longer needed to specify the new behaviour) and is deprecated.
- ``NaT`` will now correctly report ``False`` for datetimelike boolean operations such as ``is_month_start`` (`15781`)
- ``NaT`` will now correctly return ``np.nan`` for ``Timedelta`` and ``Period`` accessors such as ``days`` and ``quarter`` (`15782`)
- ``NaT`` will now returns ``NaT`` for ``tz_localize`` and ``tz_convert``  methods (`15830`)
- ``DataFrame`` and ``Panel`` constructors with invalid input will now raise ``ValueError`` rather than ``PandasError``, if called with scalar inputs and not axes (`15541`)
- ``DataFrame`` and ``Panel`` constructors with invalid input will now raise ``ValueError`` rather than ``pandas.core.common.PandasError``, if called with scalar inputs and not axes; The exception ``PandasError`` is removed as well. (`15541`)
- The exception ``pandas.core.common.AmbiguousIndexError`` is removed as it is not referenced (`15541`)  .. _whatsnew_0200.privacy:  Reorganization of the library: privacy changes
~~~  .. _whatsnew_0200.privacy.extensions:  Modules privacy has changed
^^^  Some formerly public python/c/c++/cython extension modules have been moved and/or renamed. These are all removed from the public API.
Furthermore, the ``pandas.core``, ``pandas.compat``, and ``pandas.util`` top-level modules are now considered to be PRIVATE.

If indicated, a deprecation warning will be issued if you reference these modules. (`12588`)  .. csv-table::  "Previous Location", "New Location", "Deprecated"  30, 30, 4  "pandas.lib", "pandas._libs.lib", "X"  "pandas.tslib", "pandas._libs.tslib", "X"  "pandas.computation", "pandas.core.computation", "X"  "pandas.msgpack", "pandas.io.msgpack", ""  "pandas.index", "pandas._libs.index", ""  "pandas.algos", "pandas._libs.algos", ""  "pandas.hashtable", "pandas._libs.hashtable", ""  "pandas.indexes", "pandas.core.indexes", ""  "pandas.json", "pandas._libs.json / pandas.io.json", "X"  "pandas.parser", "pandas._libs.parsers", "X"  "pandas.formats", "pandas.io.formats", ""  "pandas.sparse", "pandas.core.sparse", ""  "pandas.tools", "pandas.core.reshape", "X"  "pandas.types", "pandas.core.dtypes", "X"  "pandas.io.sas.saslib", "pandas.io.sas._sas", ""  "pandas._join", "pandas._libs.join", ""  "pandas._hash", "pandas._libs.hashing", ""  "pandas._period", "pandas._libs.period", ""  "pandas._sparse", "pandas._libs.sparse", ""  "pandas._testing", "pandas._libs.testing", ""  "pandas._window", "pandas._libs.window", ""  Some new subpackages are created with public functionality that is not directly
exposed in the top-level namespace: ``pandas.errors``, ``pandas.plotting`` and
``pandas.testing`` (more details below). Together with ``pandas.api.types`` and
certain functions in the ``pandas.io`` and ``pandas.tseries`` submodules,
these are now the public subpackages.  Further changes:  - The function `~pandas.api.types.union_categoricals` is now importable from ``pandas.api.types``, formerly from ``pandas.types.concat`` (`15998`)
- The type import ``pandas.tslib.NaTType`` is deprecated and can be replaced by using ``type(pandas.NaT)`` (`16146`)
- The public functions in ``pandas.tools.hashing`` deprecated from that locations, but are now importable from ``pandas.util`` (`16223`)
- The modules in ``pandas.util``: ``decorators``, ``print_versions``, ``doctools``, ``validators``, ``depr_module`` are now private. Only the functions exposed in ``pandas.util`` itself are public (`16223`)  .. _whatsnew_0200.privacy.errors:  ``pandas.errors``
^^^  We are adding a standard public module for all pandas exceptions & warnings ``pandas.errors``. (`14800`). Previously
these exceptions & warnings could be imported from ``pandas.core.common`` or ``pandas.io.common``. These exceptions and warnings
will be removed from the ``*.common`` locations in a future release. (`15541`)  The following are now part of this API:  .. code-block:: python  ['DtypeWarning',  'EmptyDataError',  'OutOfBoundsDatetime',  'ParserError',  'ParserWarning',  'PerformanceWarning',  'UnsortedIndexError',  'UnsupportedFunctionCall']  .. _whatsnew_0200.privacy.testing:  ``pandas.testing``
^^^  We are adding a standard module that exposes the public testing functions in ``pandas.testing`` (`9895`). Those functions can be used when writing tests for functionality using pandas objects.  The following testing functions are now part of this API:  - `testing.assert_frame_equal`
- `testing.assert_series_equal`
- `testing.assert_index_equal`  .. _whatsnew_0200.privacy.plotting:  ``pandas.plotting``
^^^  A new public ``pandas.plotting`` module has been added that holds plotting functionality that was previously in either ``pandas.tools.plotting`` or in the top-level namespace. See the `deprecations sections <whatsnew_0200.privacy.deprecate_plotting>` for more details.  .. _whatsnew_0200.privacy.development:  Other development changes
^^^  - Building pandas for development now requires ``cython >= 0.23`` (`14831`)
- Require at least 0.23 version of cython to avoid problems with character encodings (`14699`)
- Switched the test framework to use pytest_ (`13097`)
- Reorganization of tests directory layout (`14854`, `15707`).  .. _whatsnew_0200.deprecations:  Deprecations

~~~  .. _whatsnew_0200.api_breaking.deprecate_ix:  Deprecate ``.ix``
^^^  The ``.ix`` indexer is deprecated, in favor of the more strict ``.iloc`` and ``.loc`` indexers. ``.ix`` offers a lot of magic on the inference of what the user wants to do. More specifically, ``.ix`` can decide to index *positionally* OR via *labels*, depending on the data type of the index. This has caused quite a bit of user confusion over the years. The full indexing documentation is `here <indexing>`. (`14218`)  The recommended methods of indexing are:  - ``.loc`` if you want to *label* index
- ``.iloc`` if you want to *positionally* index.  Using `.ix`` will now show a ``DeprecationWarning`` with a link to some examples of how to convert code `here_.  .. ipython:: python  df = pd.DataFrame({'A': [1, 2, 3],  'B': [4, 5, 6]},  index=list('abc'))  df  Previous behavior, where you wish to get the 0th and the 2nd elements from the index in the 'A' column.  .. code-block:: ipython  In [3]: df.ix[[0, 2], 'A']  Out[3]:  a  1  c  3  Name: A, dtype: int64  Using ``.loc``. Here we will select the appropriate indexes from the index, then use *label* indexing.  .. ipython:: python  df.loc[df.index[[0, 2]], 'A']  Using ``.iloc``. Here we will get the location of the 'A' column, then use *positional* indexing to select things.  .. ipython:: python  df.iloc[[0, 2], df.columns.get_loc('A')]  .. _whatsnew_0200.api_breaking.deprecate_panel:  Deprecate Panel
^^^  ``Panel`` is deprecated and will be removed in a future version. The recommended way to represent 3-D data are
with a `MultiIndex`` on a ``DataFrame`` via the `~Panel.to_frame` or with the `xarray package_. pandas
provides a `~Panel.to_xarray` method to automate this conversion (`13563`).  .. code-block:: ipython  In [133]: import pandas._testing as tm  In [134]: p = tm.makePanel()  In [135]: p  Out[135]:  <class 'pandas.core.panel.Panel'>  Dimensions: 3 (items) x 3 (major_axis) x 4 (minor_axis)  Items axis: ItemA to ItemC  Major_axis axis: 2000-01-03 000 to 2000-01-05 000  Minor_axis axis: A to D  Convert to a MultiIndex DataFrame  .. code-block:: ipython  In [136]: p.to_frame()  Out[136]:  ItemA  ItemB  ItemC  major  minor  2000-01-03 A  0.628776 -1.409432  0.209395  B  0.988138 -1.347533 -0.896581  C  -0.938153  1.272395 -0.161137  D  -0.223019 -0.591863 -1.051539  2000-01-04 A  0.186494  1.422986 -0.592886  B  -0.072608  0.363565  1.104352  C  -1.239072 -1.449567  0.889157  D  2.123692 -0.414505 -0.319561  2000-01-05 A  0.952478 -2.147855 -1.473116  B  -0.550603 -0.014752 -0.431550  C  0.139683 -1.195524  0.288377  D  0.122273 -1.425795 -0.619993  [12 rows x 3 columns]  Convert to an xarray DataArray  .. code-block:: ipython  In [137]: p.to_xarray()  Out[137]:  <xarray.DataArray (items: 3, major_axis: 3, minor_axis: 4)>  array([[[ 0.628776,  0.988138, -0.938153, -0.223019],  [ 0.186494, -0.072608, -1.239072,  2.123692],  [ 0.952478, -0.550603,  0.139683,  0.122273]],  [[-1.409432, -1.347533,  1.272395, -0.591863],  [ 1.422986,  0.363565, -1.449567, -0.414505],  [-2.147855, -0.014752, -1.195524, -1.425795]],  [[ 0.209395, -0.896581, -0.161137, -1.051539],  [-0.592886,  1.104352,  0.889157, -0.319561],  [-1.473116, -0.43155 ,  0.288377, -0.619993]]])  Coordinates:  * items  (items) object 'ItemA' 'ItemB' 'ItemC'  * major_axis  (major_axis) datetime64[ns] 2000-01-03 2000-01-04 2000-01-05  * minor_axis  (minor_axis) object 'A' 'B' 'C' 'D'  .. _whatsnew_0200.api_breaking.deprecate_group_agg_dict:  Deprecate groupby.agg() with a dictionary when renaming
^^^  The ``.groupby(..).agg(..)``, ``.rolling(..).agg(..)``, and ``.resample(..).agg(..)``  syntax can accept a variable of inputs, including scalars,
list, and a dict of column names to scalars or lists. This provides a useful syntax for constructing multiple

(potentially different) aggregations.  However, ``.agg(..)`` can *also* accept a dict that allows 'renaming' of the result columns. This is a complicated and confusing syntax, as well as not consistent
between ``Series`` and ``DataFrame``. We are deprecating this 'renaming' functionality.  - We are deprecating passing a dict to a grouped/rolled/resampled ``Series``. This allowed  one to ``rename`` the resulting aggregation, but this had a completely different  meaning than passing a dictionary to a grouped ``DataFrame``, which accepts column-to-aggregations.
- We are deprecating passing a dict-of-dicts to a grouped/rolled/resampled ``DataFrame`` in a similar manner.  This is an illustrative example:  .. ipython:: python  df = pd.DataFrame({'A': [1, 1, 1, 2, 2],  'B': range(5),  'C': range(5)})  df  Here is a typical useful syntax for computing different aggregations for different columns. This
is a natural, and useful syntax. We aggregate from the dict-to-list by taking the specified
columns and applying the list of functions. This returns a ``MultiIndex`` for the columns (this is *not* deprecated).  .. ipython:: python  df.groupby('A').agg({'B': 'sum', 'C': 'min'})  Here's an example of the first deprecation, passing a dict to a grouped ``Series``. This
is a combination aggregation & renaming:  .. code-block:: ipython  In [6]: df.groupby('A').B.agg({'foo': 'count'})  FutureWarning: using a dict on a Series for aggregation  is deprecated and will be removed in a future version  Out[6]:  foo  A  1  3  2  2  You can accomplish the same operation, more idiomatically by:  .. ipython:: python  df.groupby('A').B.agg(['count']).rename(columns={'count': 'foo'})  Here's an example of the second deprecation, passing a dict-of-dict to a grouped ``DataFrame``:  .. code-block:: python  In [23]: (df.groupby('A')  ...:  .agg({'B': {'foo': 'sum'}, 'C': {'bar': 'min'}})  ...:  )  FutureWarning: using a dict with renaming is deprecated and  will be removed in a future version  Out[23]:  B  C  foo bar  A  1  3  0  2  7  3  You can accomplish nearly the same by:  .. ipython:: python  (df.groupby('A')  .agg({'B': 'sum', 'C': 'min'})  .rename(columns={'B': 'foo', 'C': 'bar'})  )  .. _whatsnew_0200.privacy.deprecate_plotting:  Deprecate .plotting
^^^  The ``pandas.tools.plotting`` module has been deprecated,  in favor of the top level ``pandas.plotting`` module. All the public plotting functions are now available
from ``pandas.plotting`` (`12548`).  Furthermore, the top-level ``pandas.scatter_matrix`` and ``pandas.plot_params`` are deprecated.
Users can import these from ``pandas.plotting`` as well.  Previous script:  .. code-block:: python  pd.tools.plotting.scatter_matrix(df)  pd.scatter_matrix(df)  Should be changed to:  .. code-block:: python  pd.plotting.scatter_matrix(df)  .. _whatsnew_0200.deprecations.other:  Other deprecations
^^^  - ``SparseArray.to_dense()`` has deprecated the ``fill`` parameter, as that parameter was not being respected (`14647`)
- ``SparseSeries.to_dense()`` has deprecated the ``sparse_only`` parameter (`14647`)
- ``Series.repeat()`` has deprecated the ``reps`` parameter in favor of ``repeats`` (`12662`)
- The ``Series`` constructor and ``.astype`` method have deprecated accepting timestamp dtypes without a frequency (e.g. ``np.datetime64``) for the ``dtype`` parameter (`15524`)
- ``Index.repeat()`` and ``MultiIndex.repeat()`` have deprecated the ``n`` parameter in favor of ``repeats`` (`12662`)
- ``Categorical.searchsorted()`` and ``Series.searchsorted()`` have deprecated the ``v`` parameter in favor of ``value`` (`12662`)
- ``TimedeltaIndex.searchsorted()``, ``DatetimeIndex.searchsorted()``, and ``PeriodIndex.searchsorted()`` have deprecated the ``key`` parameter in favor of ``value`` (`12662`)
- ``DataFrame.astype()`` has deprecated the ``raise_on_error`` parameter in favor of ``errors`` (`14878`)

- ``Series.sortlevel`` and ``DataFrame.sortlevel`` have been deprecated in favor of ``Series.sort_index`` and ``DataFrame.sort_index`` (`15099`)
- importing ``concat`` from ``pandas.tools.merge`` has been deprecated in favor of imports from the ``pandas`` namespace. This should only affect explicit imports (`15358`)
- ``Series/DataFrame/Panel.consolidate()`` been deprecated as a public method. (`15483`)
- The ``as_indexer`` keyword of ``Series.str.match()`` has been deprecated (ignored keyword) (`15257`).
- The following top-level pandas functions have been deprecated and will be removed in a future version (`13790`, `15940`)  * ``pd.pnow()``, replaced by ``Period.now()``  * ``pd.Term``, is removed, as it is not applicable to user code. Instead use in-line string expressions in the where clause when searching in HDFStore  * ``pd.Expr``, is removed, as it is not applicable to user code.  * ``pd.match()``, is removed.  * ``pd.groupby()``, replaced by using the ``.groupby()`` method directly on a ``Series/DataFrame``  * ``pd.get_store()``, replaced by a direct call to ``pd.HDFStore(...)``
- ``is_any_int_dtype``, ``is_floating_dtype``, and ``is_sequence`` are deprecated from ``pandas.api.types`` (`16042`)  .. _whatsnew_0200.prior_deprecations:  Removal of prior version deprecations/changes
~~~  - The ``pandas.rpy`` module is removed. Similar functionality can be accessed  through the rpy2_ project.  See the R interfacing docs_ for more details.
- The ``pandas.io.ga`` module with a ``google-analytics`` interface is removed (`11308`).  Similar functionality can be found in the Google2Pandas_ package.
- ``pd.to_datetime`` and ``pd.to_timedelta`` have dropped the ``coerce`` parameter in favor of ``errors`` (`13602`)
- `pandas.stats.fama_macbeth``, ``pandas.stats.ols``, ``pandas.stats.plm`` and ``pandas.stats.var``, as well as the top-level ``pandas.fama_macbeth`` and ``pandas.ols`` routines are removed. Similar functionality can be found in the `statsmodels_ package. (`11898`)
- The ``TimeSeries`` and ``SparseTimeSeries`` classes, aliases of ``Series``  and ``SparseSeries``, are removed (`10890`, `15098`).
- ``Series.is_time_series`` is dropped in favor of ``Series.index.is_all_dates`` (`15098`)
- The deprecated ``irow``, ``icol``, ``iget`` and ``iget_value`` methods are removed  in favor of ``iloc`` and ``iat`` as explained `here <whatsnew_0170.deprecations>` (`10711`).
- The deprecated ``DataFrame.iterkv()`` has been removed in favor of ``DataFrame.iteritems()`` (`10711`)
- The ``Categorical`` constructor has dropped the ``name`` parameter (`10632`)
- ``Categorical`` has dropped support for ``NaN`` categories (`10748`)
- The ``take_last`` parameter has been dropped from ``duplicated()``, ``drop_duplicates()``, ``nlargest()``, and ``nsmallest()`` methods (`10236`, `10792`, `10920`)
- ``Series``, ``Index``, and ``DataFrame`` have dropped the ``sort`` and ``order`` methods (`10726`)
- Where clauses in ``pytables`` are only accepted as strings and expressions types and not other data-types (`12027`)
- ``DataFrame`` has dropped the ``combineAdd`` and ``combineMult`` methods in favor of ``add`` and ``mul`` respectively (`10735`)  .. _whatsnew_0200.performance:  Performance improvements
~~~  - Improved performance of ``pd.wide_to_long()`` (`14779`)
- Improved performance of ``pd.factorize()`` by releasing the GIL with ``object`` dtype when inferred as strings (`14859`, `16057`)
- Improved performance of timeseries plotting with an irregular DatetimeIndex  (or with ``compat_x=True``) (`15073`).
- Improved performance of ``groupby().cummin()`` and ``groupby().cummax()`` (`15048`, `15109`, `15561`, `15635`)
- Improved performance and reduced memory when indexing with a ``MultiIndex`` (`15245`)
- When reading buffer object in ``read_sas()`` method without specified format, filepath string is inferred rather than buffer object. (`14947`)

- Improved performance of ``.rank()`` for categorical data (`15498`)
- Improved performance when using ``.unstack()`` (`15503`)
- Improved performance of merge/join on ``category`` columns (`10409`)
- Improved performance of ``drop_duplicates()`` on ``bool`` columns (`12963`)
- Improve performance of ``pd.core.groupby.GroupBy.apply`` when the applied  function used the ``.name`` attribute of the group DataFrame (`15062`).
- Improved performance of ``iloc`` indexing with a list or array (`15504`).
- Improved performance of ``Series.sort_index()`` with a monotonic index (`15694`)
- Improved performance in ``pd.read_csv()`` on some platforms with buffered reads (`16039`)  .. _whatsnew_0200.bug_fixes:  Bug fixes
~~~  Conversion
^^^  - Bug in ``Timestamp.replace`` now raises ``TypeError`` when incorrect argument names are given; previously this raised ``ValueError`` (`15240`)
- Bug in ``Timestamp.replace`` with compat for passing long integers (`15030`)
- Bug in ``Timestamp`` returning UTC based time/date attributes when a timezone was provided (`13303`, `6538`)
- Bug in ``Timestamp`` incorrectly localizing timezones during construction (`11481`, `15777`)
- Bug in ``TimedeltaIndex`` addition where overflow was being allowed without error (`14816`)
- Bug in ``TimedeltaIndex`` raising a ``ValueError`` when boolean indexing with ``loc`` (`14946`)
- Bug in catching an overflow in ``Timestamp`` + ``Timedelta/Offset`` operations (`15126`)
- Bug in ``DatetimeIndex.round()`` and ``Timestamp.round()`` floating point accuracy when rounding by milliseconds or less (`14440`, `15578`)
- Bug in ``astype()`` where ``inf`` values were incorrectly converted to integers. Now raises error now with ``astype()`` for Series and DataFrames (`14265`)
- Bug in ``DataFrame(..).apply(to_numeric)`` when values are of type decimal.Decimal. (`14827`)
- Bug in ``describe()`` when passing a numpy array which does not contain the median to the ``percentiles`` keyword argument (`14908`)
- Cleaned up ``PeriodIndex`` constructor, including raising on floats more consistently (`13277`)
- Bug in using ``__deepcopy__`` on empty NDFrame objects (`15370`)
- Bug in ``.replace()`` may result in incorrect dtypes. (`12747`, `15765`)
- Bug in ``Series.replace`` and ``DataFrame.replace`` which failed on empty replacement dicts (`15289`)
- Bug in ``Series.replace`` which replaced a numeric by string (`15743`)
- Bug in ``Index`` construction with ``NaN`` elements and integer dtype specified (`15187`)
- Bug in ``Series`` construction with a datetimetz (`14928`)
- Bug in ``Series.dt.round()`` inconsistent behaviour on ``NaT`` 's with different arguments (`14940`)
- Bug in ``Series`` constructor when both ``copy=True`` and ``dtype`` arguments are provided (`15125`)
- Incorrect dtyped ``Series`` was returned by comparison methods (e.g., ``lt``, ``gt``, ...) against a constant for an empty ``DataFrame`` (`15077`)
- Bug in ``Series.ffill()`` with mixed dtypes containing tz-aware datetimes. (`14956`)
- Bug in ``DataFrame.fillna()`` where the argument ``downcast`` was ignored when fillna value was of type ``dict`` (`15277`)
- Bug in ``.asfreq()``, where frequency was not set for empty ``Series`` (`14320`)
- Bug in ``DataFrame`` construction with nulls and datetimes in a list-like (`15869`)
- Bug in ``DataFrame.fillna()`` with tz-aware datetimes (`15855`)
- Bug in ``is_string_dtype``, ``is_timedelta64_ns_dtype``, and ``is_string_like_dtype`` in which an error was raised when ``None`` was passed in (`15941`)
- Bug in the return type of ``pd.unique`` on a ``Categorical``, which was returning an ndarray and not a ``Categorical`` (`15903`)
- Bug in ``Index.to_series()`` where the index was not copied (and so mutating later would change the original), (`15949`)
- Bug in indexing with partial string indexing with a len-1 DataFrame (`16071`)

- Bug in ``Series`` construction where passing invalid dtype didn't raise an error. (`15520`)  Indexing
^^^  - Bug in ``Index`` power operations with reversed operands (`14973`)
- Bug in ``DataFrame.sort_values()`` when sorting by multiple columns where one column is of type ``int64`` and contains ``NaT`` (`14922`)
- Bug in ``DataFrame.reindex()`` in which ``method`` was ignored when passing ``columns`` (`14992`)
- Bug in ``DataFrame.loc`` with indexing a ``MultiIndex`` with a ``Series`` indexer (`14730`, `15424`)
- Bug in ``DataFrame.loc`` with indexing a ``MultiIndex`` with a numpy array (`15434`)
- Bug in ``Series.asof`` which raised if the series contained all ``np.nan`` (`15713`)
- Bug in ``.at`` when selecting from a tz-aware column (`15822`)
- Bug in ``Series.where()`` and ``DataFrame.where()`` where array-like conditionals were being rejected (`15414`)
- Bug in ``Series.where()`` where TZ-aware data was converted to float representation (`15701`)
- Bug in ``.loc`` that would not return the correct dtype for scalar access for a DataFrame (`11617`)
- Bug in output formatting of a ``MultiIndex`` when names are integers (`12223`, `15262`)
- Bug in ``Categorical.searchsorted()`` where alphabetical instead of the provided categorical order was used (`14522`)
- Bug in ``Series.iloc`` where a ``Categorical`` object for list-like indexes input was returned, where a ``Series`` was expected. (`14580`)
- Bug in ``DataFrame.isin`` comparing datetimelike to empty frame (`15473`)
- Bug in ``.reset_index()`` when an all ``NaN`` level of a ``MultiIndex`` would fail (`6322`)
- Bug in ``.reset_index()`` when raising error for index name already present in ``MultiIndex`` columns (`16120`)
- Bug in creating a ``MultiIndex`` with tuples and not passing a list of names; this will now raise ``ValueError`` (`15110`)
- Bug in the HTML display with a ``MultiIndex`` and truncation (`14882`)
- Bug in the display of ``.info()`` where a qualifier (+) would always be displayed with a ``MultiIndex`` that contains only non-strings (`15245`)
- Bug in ``pd.concat()`` where the names of ``MultiIndex`` of resulting ``DataFrame`` are not handled correctly when ``None`` is presented in the names of ``MultiIndex`` of input ``DataFrame`` (`15787`)
- Bug in ``DataFrame.sort_index()`` and ``Series.sort_index()`` where ``na_position`` doesn't work with a ``MultiIndex`` (`14784`, `16604`)
- Bug in ``pd.concat()`` when combining objects with a ``CategoricalIndex`` (`16111`)
- Bug in indexing with a scalar and a ``CategoricalIndex`` (`16123`)  IO
^^  - Bug in ``pd.to_numeric()`` in which float and unsigned integer elements were being improperly casted (`14941`, `15005`)
- Bug in ``pd.read_fwf()`` where the skiprows parameter was not being respected during column width inference (`11256`)
- Bug in ``pd.read_csv()`` in which the ``dialect`` parameter was not being verified before processing (`14898`)
- Bug in ``pd.read_csv()`` in which missing data was being improperly handled with ``usecols`` (`6710`)
- Bug in ``pd.read_csv()`` in which a file containing a row with many columns followed by rows with fewer columns would cause a crash (`14125`)
- Bug in ``pd.read_csv()`` for the C engine where ``usecols`` were being indexed incorrectly with ``parse_dates`` (`14792`)
- Bug in ``pd.read_csv()`` with ``parse_dates`` when multi-line headers are specified (`15376`)
- Bug in ``pd.read_csv()`` with ``float_precision='round_trip'`` which caused a segfault when a text entry is parsed (`15140`)
- Bug in ``pd.read_csv()`` when an index was specified and no values were specified as null values (`15835`)
- Bug in ``pd.read_csv()`` in which certain invalid file objects caused the Python interpreter to crash (`15337`)
- Bug in ``pd.read_csv()`` in which invalid values for ``nrows`` and ``chunksize`` were allowed (`15767`)

- Bug in ``pd.read_csv()`` for the Python engine in which unhelpful error messages were being raised when parsing errors occurred (`15910`)
- Bug in ``pd.read_csv()`` in which the ``skipfooter`` parameter was not being properly validated (`15925`)
- Bug in ``pd.to_csv()`` in which there was numeric overflow when a timestamp index was being written (`15982`)
- Bug in ``pd.util.hashing.hash_pandas_object()`` in which hashing of categoricals depended on the ordering of categories, instead of just their values. (`15143`)
- Bug in ``.to_json()`` where ``lines=True`` and contents (keys or values) contain escaped characters (`15096`)
- Bug in ``.to_json()`` causing single byte ascii characters to be expanded to four byte unicode (`15344`)
- Bug in ``.to_json()`` for the C engine where rollover was not correctly handled for case where frac is odd and diff is exactly 0.5 (`15716`, `15864`)
- Bug in ``pd.read_json()`` for Python 2 where ``lines=True`` and contents contain non-ascii unicode characters (`15132`)
- Bug in ``pd.read_msgpack()`` in which ``Series`` categoricals were being improperly processed (`14901`)
- Bug in ``pd.read_msgpack()`` which did not allow loading of a dataframe with an index of type ``CategoricalIndex`` (`15487`)
- Bug in ``pd.read_msgpack()`` when deserializing a ``CategoricalIndex`` (`15487`)
- Bug in ``DataFrame.to_records()`` with converting a ``DatetimeIndex`` with a timezone (`13937`)
- Bug in ``DataFrame.to_records()`` which failed with unicode characters in column names (`11879`)
- Bug in ``.to_sql()`` when writing a DataFrame with numeric index names (`15404`).
- Bug in ``DataFrame.to_html()`` with ``index=False`` and ``max_rows`` raising in ``IndexError`` (`14998`)
- Bug in ``pd.read_hdf()`` passing a ``Timestamp`` to the ``where`` parameter with a non date column (`15492`)
- Bug in ``DataFrame.to_stata()`` and ``StataWriter`` which produces incorrectly formatted files to be produced for some locales (`13856`)
- Bug in ``StataReader`` and ``StataWriter`` which allows invalid encodings (`15723`)
- Bug in the ``Series`` repr not showing the length when the output was truncated (`15962`).  Plotting
^^^  - Bug in ``DataFrame.hist`` where ``plt.tight_layout`` caused an ``AttributeError``  (use ``matplotlib >= 2.0.1``) (`9351`)
- Bug in ``DataFrame.boxplot`` where ``fontsize`` was not applied to the tick labels on both axes (`15108`)
- Bug in the date and time converters pandas registers with matplotlib not handling multiple dimensions (`16026`)
- Bug in ``pd.scatter_matrix()`` could accept either ``color`` or ``c``, but not both (`14855`)  GroupBy/resample/rolling
^^^  - Bug in ``.groupby(..).resample()`` when passed the ``on=`` kwarg. (`15021`)
- Properly set ``__name__`` and ``__qualname__`` for ``Groupby.*`` functions (`14620`)
- Bug in ``GroupBy.get_group()`` failing with a categorical grouper (`15155`)
- Bug in ``.groupby(...).rolling(...)`` when ``on`` is specified and using a ``DatetimeIndex`` (`15130`, `13966`)
- Bug in groupby operations with ``timedelta64`` when passing ``numeric_only=False`` (`5724`)
- Bug in ``groupby.apply()`` coercing ``object`` dtypes to numeric types, when not all values were numeric (`14423`, `15421`, `15670`)
- Bug in ``resample``, where a non-string ``loffset`` argument would not be applied when resampling a timeseries (`13218`)
- Bug in ``DataFrame.groupby().describe()`` when grouping on ``Index`` containing tuples (`14848`)
- Bug in ``groupby().nunique()`` with a datetimelike-grouper where bins counts were incorrect (`13453`)
- Bug in ``groupby.transform()`` that would coerce the resultant dtypes back to the original (`10972`, `11444`)
- Bug in ``groupby.agg()`` incorrectly localizing timezone on ``datetime`` (`15426`, `10668`, `13046`)
- Bug in ``.rolling/expanding()`` functions where ``count()`` was not counting ``np.Inf``, nor handling ``object`` dtypes (`12541`)

- Bug in ``.rolling()`` where ``pd.Timedelta`` or ``datetime.timedelta`` was not accepted as a ``window`` argument (`15440`)
- Bug in ``Rolling.quantile`` function that caused a segmentation fault when called with a quantile value outside of the range [0, 1] (`15463`)
- Bug in ``DataFrame.resample().median()`` if duplicate column names are present (`14233`)  Sparse
^^^  - Bug in ``SparseSeries.reindex`` on single level with list of length 1 (`15447`)
- Bug in repr-formatting a ``SparseDataFrame`` after a value was set on (a copy of) one of its series (`15488`)
- Bug in ``SparseDataFrame`` construction with lists not coercing to dtype (`15682`)
- Bug in sparse array indexing in which indices were not being validated (`15863`)  Reshaping
^^^  - Bug in ``pd.merge_asof()`` where ``left_index`` or ``right_index`` caused a failure when multiple ``by`` was specified (`15676`)
- Bug in ``pd.merge_asof()`` where ``left_index``/``right_index`` together caused a failure when ``tolerance`` was specified (`15135`)
- Bug in ``DataFrame.pivot_table()`` where ``dropna=True`` would not drop all-NaN columns when the columns was a ``category`` dtype (`15193`)
- Bug in ``pd.melt()`` where passing a tuple value for ``value_vars`` caused a ``TypeError`` (`15348`)
- Bug in ``pd.pivot_table()`` where no error was raised when values argument was not in the columns (`14938`)
- Bug in ``pd.concat()`` in which concatenating with an empty dataframe with ``join='inner'`` was being improperly handled (`15328`)
- Bug with ``sort=True`` in ``DataFrame.join`` and ``pd.merge`` when joining on indexes (`15582`)
- Bug in ``DataFrame.nsmallest`` and ``DataFrame.nlargest`` where identical values resulted in duplicated rows (`15297`)
- Bug in `pandas.pivot_table` incorrectly raising ``UnicodeError`` when passing unicode input for ``margins`` keyword (`13292`)  Numeric
^^^  - Bug in ``.rank()`` which incorrectly ranks ordered categories (`15420`)
- Bug in ``.corr()`` and ``.cov()`` where the column and index were the same object (`14617`)
- Bug in ``.mode()`` where ``mode`` was not returned if was only a single value (`15714`)
- Bug in ``pd.cut()`` with a single bin on an all 0s array (`15428`)
- Bug in ``pd.qcut()`` with a single quantile and an array with identical values (`15431`)
- Bug in ``pandas.tools.utils.cartesian_product()`` with large input can cause overflow on windows (`15265`)
- Bug in ``.eval()`` which caused multi-line evals to fail with local variables not on the first line (`15342`)  Other
^^^  - Compat with SciPy 0.19.0 for testing on ``.interpolate()`` (`15662`)
- Compat for 32-bit platforms for ``.qcut/cut``; bins will now be ``int64`` dtype (`14866`)
- Bug in interactions with ``Qt`` when a ``QtApplication`` already exists (`14372`)
- Avoid use of ``np.finfo()`` during ``import pandas`` removed to mitigate deadlock on Python GIL misuse (`14641`)  .. _whatsnew_0.20.0.contributors:  Contributors
~~~  .. contributors:: v0.19.2..v0.20.0

{{ header }}  .. ipython:: python  from pandas import *  # noqa F401, F403  This is a minor bug-fix release in the 0.20.x series and includes some small regression fixes,
We recommend that all users upgrade to this version.  .. contents:: What's new in v0.20.2  none  .. _whatsnew_0202.enhancements:  Enhancements
~~~  - Unblocked access to additional compression types supported in pytables: 'blosc:blosclz, 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy', 'blosc:zlib', 'blosc:zstd' (`14478`)
- ``Series`` provides a ``to_latex`` method (`16180`)  - A new groupby method `~pandas.core.groupby.GroupBy.ngroup`,  parallel to the existing `~pandas.core.groupby.GroupBy.cumcount`,  has been added to return the group order (`11642`); see  `here <groupby.ngroup>`.  .. _whatsnew_0202.performance:  Performance improvements
~~~  - Performance regression fix when indexing with a list-like (`16285`)
- Performance regression fix for MultiIndexes (`16319`, `16346`)
- Improved performance of ``.clip()`` with scalar arguments (`15400`)
- Improved performance of groupby with categorical groupers (`16413`)
- Improved performance of ``MultiIndex.remove_unused_levels()`` (`16556`)  .. _whatsnew_0202.bug_fixes:  Bug fixes
~~~  - Silenced a warning on some Windows environments about "tput: terminal attributes: No such device or address" when  detecting the terminal size. This fix only applies to python 3 (`16496`)
- Bug in using ``pathlib.Path`` or ``py.path.local`` objects with io functions (`16291`)
- Bug in ``Index.symmetric_difference()`` on two equal MultiIndex's, results in a ``TypeError`` (`13490`)
- Bug in ``DataFrame.update()`` with ``overwrite=False`` and ``NaN values`` (`15593`)
- Passing an invalid engine to `read_csv` now raises an informative  ``ValueError`` rather than ``UnboundLocalError``. (`16511`)
- Bug in `unique` on an array of tuples (`16519`)
- Bug in `cut` when ``labels`` are set, resulting in incorrect label ordering (`16459`)
- Fixed a compatibility issue with IPython 6.0's tab completion showing deprecation warnings on ``Categoricals`` (`16409`)  Conversion
^^^  - Bug in `to_numeric` in which empty data inputs were causing a segfault of the interpreter (`16302`)
- Silence numpy warnings when broadcasting ``DataFrame`` to ``Series`` with comparison ops (`16378`, `16306`)  Indexing
^^^  - Bug in ``DataFrame.reset_index(level=)`` with single level index (`16263`)
- Bug in partial string indexing with a monotonic, but not strictly-monotonic, index incorrectly reversing the slice bounds (`16515`)
- Bug in ``MultiIndex.remove_unused_levels()`` that would not return a ``MultiIndex`` equal to the original. (`16556`)  IO
^^  - Bug in `read_csv` when ``comment`` is passed in a space delimited text file (`16472`)
- Bug in `read_csv` not raising an exception with nonexistent columns in ``usecols`` when it had the correct length (`14671`)
- Bug that would force importing of the clipboard routines unnecessarily, potentially causing an import error on startup (`16288`)
- Bug that raised ``IndexError`` when HTML-rendering an empty ``DataFrame`` (`15953`)
- Bug in `read_csv` in which tarfile object inputs were raising an error in Python 2.x for the C engine (`16530`)
- Bug where ``DataFrame.to_html()`` ignored the ``index_names`` parameter (`16493`)
- Bug where ``pd.read_hdf()`` returns numpy strings for index names (`13492`)  - Bug in ``HDFStore.select_as_multiple()`` where start/stop arguments were not respected (`16209`)  Plotting
^^^  - Bug in ``DataFrame.plot`` with a single column and a list-like ``color`` (`3486`)
- Bug in ``plot`` where ``NaT`` in ``DatetimeIndex`` results in ``Timestamp.min`` (`12405`)
- Bug in ``DataFrame.boxplot`` where ``figsize`` keyword was not respected for non-grouped boxplots (`11959`)  GroupBy/resample/rolling
^^^  - Bug in creating a time-based rolling window on an empty ``DataFrame`` (`15819`)
- Bug in ``rolling.cov()`` with offset window (`16058`)

- Bug in ``.resample()`` and ``.groupby()`` when aggregating on integers (`16361`)  Sparse
^^^  - Bug in construction of ``SparseDataFrame`` from ``scipy.sparse.dok_matrix`` (`16179`)  Reshaping
^^^  - Bug in ``DataFrame.stack`` with unsorted levels in ``MultiIndex`` columns (`16323`)
- Bug in ``pd.wide_to_long()`` where no error was raised when ``i`` was not a unique identifier (`16382`)
- Bug in ``Series.isin(..)`` with a list of tuples (`16394`)
- Bug in construction of a ``DataFrame`` with mixed dtypes including an all-NaT column. (`16395`)
- Bug in ``DataFrame.agg()`` and ``Series.agg()`` with aggregating on non-callable attributes (`16405`)  Numeric
^^^
- Bug in ``.interpolate()``, where ``limit_direction`` was not respected when ``limit=None`` (default) was passed (`16282`)  Categorical
^^^  - Fixed comparison operations considering the order of the categories when both categoricals are unordered (`16014`)  Other
^^^  - Bug in ``DataFrame.drop()`` with an empty-list with non-unique indices (`16270`)  .. _whatsnew_0.20.2.contributors:  Contributors
~~~  .. contributors:: v0.20.0..v0.20.2

Version 0.20.3 (July 7, 2017)

{{ header }}  .. ipython:: python  from pandas import *  # noqa F401, F403  This is a minor bug-fix release in the 0.20.x series and includes some small regression fixes
and bug fixes. We recommend that all users upgrade to this version.  .. contents:: What's new in v0.20.3  none  .. _whatsnew_0203.bug_fixes:  Bug fixes
~~~  - Fixed a bug in failing to compute rolling computations of a column-MultiIndexed ``DataFrame`` (`16789`, `16825`)
- Fixed a pytest marker failing downstream packages' tests suites (`16680`)  Conversion
^^^  - Bug in pickle compat prior to the v0.20.x series, when ``UTC`` is a timezone in a Series/DataFrame/Index (`16608`)
- Bug in ``Series`` construction when passing a ``Series`` with ``dtype='category'`` (`16524`).
- Bug in `DataFrame.astype` when passing a ``Series`` as the ``dtype`` kwarg. (`16717`).  Indexing
^^^  - Bug in ``Float64Index`` causing an empty array instead of ``None`` to be returned from ``.get(np.nan)`` on a Series whose index did not contain any ``NaN`` s (`8569`)
- Bug in ``MultiIndex.isin`` causing an error when passing an empty iterable (`16777`)
- Fixed a bug in a slicing DataFrame/Series that have a  ``TimedeltaIndex`` (`16637`)  IO
^^  - Bug in `read_csv` in which files weren't opened as binary files by the C engine on Windows, causing EOF characters mid-field, which would fail (`16039`, `16559`, `16675`)
- Bug in `read_hdf` in which reading a ``Series`` saved to an HDF file in 'fixed' format fails when an explicit ``mode='r'`` argument is supplied (`16583`)
- Bug in `DataFrame.to_latex` where ``bold_rows`` was wrongly specified to be ``True`` by default, whereas in reality row labels remained non-bold whatever parameter provided. (`16707`)
- Fixed an issue with `DataFrame.style` where generated element ids were not unique (`16780`)
- Fixed loading a ``DataFrame`` with a ``PeriodIndex``, from a ``format='fixed'`` HDFStore, in Python 3, that was written in Python 2 (`16781`)  Plotting
^^^  - Fixed regression that prevented RGB and RGBA tuples from being used as color arguments (`16233`)
- Fixed an issue with `DataFrame.plot.scatter` that incorrectly raised a ``KeyError`` when categorical data is used for plotting (`16199`)  Reshaping
^^^  - ``PeriodIndex`` / ``TimedeltaIndex.join`` was missing the ``sort=`` kwarg (`16541`)
- Bug in joining on a ``MultiIndex`` with a ``category`` dtype for a level (`16627`).
- Bug in `merge` when merging/joining with multiple categorical columns (`16767`)  Categorical
^^^  - Bug in ``DataFrame.sort_values`` not respecting the ``kind`` parameter with categorical data (`16793`)  .. _whatsnew_0.20.3.contributors:  Contributors
~~~  .. contributors:: v0.20.2..v0.20.3

{{ header }}  .. ipython:: python  from pandas import * # noqa F401, F403  This is a major release from 0.20.3 and includes a number of API changes, deprecations, new features,
users upgrade to this version.  Highlights include:  - Integration with Apache Parquet_, including a new top-level `read_parquet` function and `DataFrame.to_parquet` method, see `here <whatsnew_0210.enhancements.parquet>`.
- New user-facing `pandas.api.types.CategoricalDtype` for specifying  categoricals independent of the data, see `here <whatsnew_0210.enhancements.categorical_dtype>`.
- The behavior of `sum`` and ``prod`` on all-NaN Series/DataFrames is now consistent and no longer depends on whether `bottleneck_ is installed, and ``sum`` and ``prod`` on empty Series now return NaN instead of 0, see `here <whatsnew_0210.api_breaking.bottleneck>`.
- Compatibility fixes for pypy, see `here <whatsnew_0210.pypy>`.
- Additions to the ``drop``, ``reindex`` and ``rename`` API to make them more consistent, see `here <whatsnew_0210.enhancements.drop_api>`.
- Addition of the new methods ``DataFrame.infer_objects`` (see `here <whatsnew_0210.enhancements.infer_objects>`) and ``GroupBy.pipe`` (see `here <whatsnew_0210.enhancements.GroupBy_pipe>`).
- Indexing with a list of labels, where one or more of the labels is missing, is deprecated and will raise a KeyError in a future version, see `here <whatsnew_0210.api_breaking.loc>`.  Check the `API Changes <whatsnew_0210.api_breaking>` and `deprecations <whatsnew_0210.deprecations>` before updating.  .. contents:: What's new in v0.21.0  none  2  .. _whatsnew_0210.enhancements:  New features
~~~  .. _whatsnew_0210.enhancements.parquet:  Integration with Apache Parquet file format
^^^  Integration with Apache Parquet_, including a new top-level `read_parquet` and `DataFrame.to_parquet` method, see `here <io.parquet>` (`15838`, `17438`).  Apache Parquet_ provides a cross-language, binary file format for reading and writing data frames efficiently.
Parquet is designed to faithfully serialize and de-serialize ``DataFrame`` s, supporting all of the pandas
dtypes, including extension dtypes such as datetime with timezones.  This functionality depends on either the pyarrow_ or fastparquet_ library.
For more details, see `the IO docs on Parquet <io.parquet>`.  .. _whatsnew_0210.enhancements.infer_objects:  Method ``infer_objects`` type conversion
^^^  The `DataFrame.infer_objects` and `Series.infer_objects`
methods have been added to perform dtype inference on object columns, replacing
some of the functionality of the deprecated ``convert_objects``
method. See the documentation `here <basics.object_conversion>`
for more details. (`11221`)  This method only performs soft conversions on object columns, converting Python objects
to native types, but not any coercive conversions. For example:  .. ipython:: python  df = pd.DataFrame({'A': [1, 2, 3],  'B': np.array([1, 2, 3], dtype='object'),  'C': ['1', '2', '3']})  df.dtypes  df.infer_objects().dtypes  Note that column ``'C'`` was not converted - only scalar numeric types
will be converted to a new type.  Other types of conversion should be accomplished
using the `to_numeric` function (or `to_datetime`, `to_timedelta`).  .. ipython:: python  df = df.infer_objects()  df['C'] = pd.to_numeric(df['C'], errors='coerce')  df.dtypes  .. _whatsnew_0210.enhancements.attribute_access:  Improved warnings when attempting to create columns
^^^  New users are often puzzled by the relationship between column operations and
attribute access on ``DataFrame`` instances (`7175`). One specific
instance of this confusion is attempting to create a new column by setting an

attribute on the ``DataFrame``:  .. code-block:: ipython  In [1]: df = pd.DataFrame({'one': [1., 2., 3.]})  In [2]: df.two = [4, 5, 6]  This does not raise any obvious exceptions, but also does not create a new column:  .. code-block:: ipython  In [3]: df  Out[3]:  one  0  1.0  1  2.0  2  3.0  Setting a list-like data structure into a new attribute now raises a ``UserWarning`` about the potential for unexpected behavior. See `Attribute Access <indexing.attribute_access>`.  .. _whatsnew_0210.enhancements.drop_api:  Method ``drop`` now also accepts index/columns keywords
^^^  The `~DataFrame.drop` method has gained ``index``/``columns`` keywords as an
alternative to specifying the ``axis``. This is similar to the behavior of ``reindex``
(`12392`).  For example:  .. ipython:: python  df = pd.DataFrame(np.arange(8).reshape(2, 4),  columns=['A', 'B', 'C', 'D'])  df  df.drop(['B', 'C'], axis=1)  # the following is now equivalent  df.drop(columns=['B', 'C'])  .. _whatsnew_0210.enhancements.rename_reindex_axis:  Methods ``rename``, ``reindex`` now also accept axis keyword
^^^  The `DataFrame.rename` and `DataFrame.reindex` methods have gained
the ``axis`` keyword to specify the axis to target with the operation
(`12392`).  Here's ``rename``:  .. ipython:: python  df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})  df.rename(str.lower, axis='columns')  df.rename(id, axis='index')  And ``reindex``:  .. ipython:: python  df.reindex(['A', 'B', 'C'], axis='columns')  df.reindex([0, 1, 3], axis='index')  The "index, columns" style continues to work as before.  .. ipython:: python  df.rename(index=id, columns=str.lower)  df.reindex(index=[0, 1, 3], columns=['A', 'B', 'C'])  We *highly* encourage using named arguments to avoid confusion when using either
style.  .. _whatsnew_0210.enhancements.categorical_dtype:  ``CategoricalDtype`` for specifying categoricals
^^^  `pandas.api.types.CategoricalDtype` has been added to the public API and
expanded to include the ``categories`` and ``ordered`` attributes. A
``CategoricalDtype`` can be used to specify the set of categories and
orderedness of an array, independent of the data. This can be useful for example,
when converting string data to a ``Categorical`` (`14711`,
`15078`, `16015`, `17643`):  .. ipython:: python  from pandas.api.types import CategoricalDtype  s = pd.Series(['a', 'b', 'c', 'a'])  # strings  dtype = CategoricalDtype(categories=['a', 'b', 'c', 'd'], ordered=True)  s.astype(dtype)  One place that deserves special mention is in `read_csv`. Previously, with
``dtype={'col': 'category'}``, the returned values and categories would always
be strings.  .. ipython:: python  from io import StringIO  .. ipython:: python  data = 'A,B\na,1\nb,2\nc,3'  pd.read_csv(StringIO(data), dtype={'B': 'category'}).B.cat.categories  Notice the "object" dtype.  With a ``CategoricalDtype`` of all numerics, datetimes, or
timedeltas, we can automatically convert to the correct type  .. ipython:: python  dtype = {'B': CategoricalDtype([1, 2, 3])}  pd.read_csv(StringIO(data), dtype=dtype).B.cat.categories  The values have been correctly interpreted as integers.  The ``.dtype`` property of a ``Categorical``, ``CategoricalIndex`` or a
``Series`` with categorical type will now return an instance of
``CategoricalDtype``. While the repr has changed, ``str(CategoricalDtype())`` is
still the string ``'category'``. We'll take this moment to remind users that the
*preferred* way to detect categorical data is to use
`pandas.api.types.is_categorical_dtype`, and not ``str(dtype) == 'category'``.  See the `CategoricalDtype docs <categorical.categoricaldtype>` for more.  .. _whatsnew_0210.enhancements.GroupBy_pipe:  ``GroupBy`` objects now have a ``pipe`` method
^^^  ``GroupBy`` objects now have a ``pipe`` method, similar to the one on
``DataFrame`` and ``Series``, that allow for functions that take a

``GroupBy`` to be composed in a clean, readable syntax. (`17871`)  For a concrete example on combining ``.groupby`` and ``.pipe`` , imagine having a
DataFrame with columns for stores, products, revenue and sold quantity. We'd like to
do a groupwise calculation of *prices* (i.e. revenue/quantity) per store and per product.
We could do this in a multi-step operation, but expressing it in terms of piping can make the
code more readable.  First we set the data:  .. ipython:: python  import numpy as np  n = 1000  df = pd.DataFrame({'Store': np.random.choice(['Store_1', 'Store_2'], n),  'Product': np.random.choice(['Product_1',  'Product_2',  'Product_3'  ], n),  'Revenue': (np.random.random(n) * 50 + 10).round(2),  'Quantity': np.random.randint(1, 10, size=n)})  df.head(2)  Now, to find prices per store/product, we can simply do:  .. ipython:: python  (df.groupby(['Store', 'Product'])  .pipe(lambda grp: grp.Revenue.sum() / grp.Quantity.sum())  .unstack().round(2))  See the `documentation <groupby.pipe>` for more.  .. _whatsnew_0210.enhancements.rename_categories:  ``Categorical.rename_categories`` accepts a dict-like
^^^  `~Series.cat.rename_categories` now accepts a dict-like argument for
``new_categories``. The previous categories are looked up in the dictionary's
keys and replaced if found. The behavior of missing and extra keys is the same
as in `DataFrame.rename`.  .. ipython:: python  c = pd.Categorical(['a', 'a', 'b'])  c.rename_categories({"a": "eh", "b": "bee"})  .. warning::  To assist with upgrading pandas, ``rename_categories`` treats ``Series`` as  list-like. Typically, Series are considered to be dict-like (e.g. in  ``.rename``, ``.map``). In a future version of pandas ``rename_categories``  will change to treat them as dict-like. Follow the warning message's  recommendations for writing future-proof code.  .. code-block:: ipython  In [33]: c.rename_categories(pd.Series([0, 1], index=['a', 'c']))  FutureWarning: Treating Series 'new_categories' as a list-like and using the values.  In a future version, 'rename_categories' will treat Series like a dictionary.  For dict-like, use 'new_categories.to_dict()'  For list-like, use 'new_categories.values'.  Out[33]:  [0, 0, 1]  Categories (2, int64): [0, 1]  .. _whatsnew_0210.enhancements.other:  Other enhancements
^^^  New functions or methods
"""  - `~pandas.core.resample.Resampler.nearest` is added to support nearest-neighbor upsampling (`17496`).
- `~pandas.Index` has added support for a ``to_frame`` method (`15230`).  New keywords
"""  - Added a ``skipna`` parameter to `~pandas.api.types.infer_dtype` to  support type inference in the presence of missing values (`17059`).
- `Series.to_dict` and `DataFrame.to_dict` now support an ``into`` keyword which allows you to specify the ``collections.Mapping`` subclass that you would like returned.  The default is ``dict``, which is backwards compatible. (`16122`)
- `Series.set_axis` and `DataFrame.set_axis` now support the ``inplace`` parameter. (`14636`)
- Series.to_pickle` and `DataFrame.to_pickle` have gained a ``protocol`` parameter (`16252`). By default, this parameter is set to `HIGHEST_PROTOCOL_
- `read_feather` has gained the ``nthreads`` parameter for multi-threaded operations (`16359`)
- `DataFrame.clip()` and `Series.clip()` have gained an ``inplace`` argument. (`15388`)
- `crosstab` has gained a ``margins_name`` parameter to define the name of the row / column that will contain the totals when ``margins=True``. (`15972`)
- `read_json` now accepts a ``chunksize`` parameter that can be used when ``lines=True``. If ``chunksize`` is passed, read_json now returns an iterator which reads in ``chunksize`` lines with each iteration. (`17048`)
- `read_json` and `~DataFrame.to_json` now accept a ``compression`` argument which allows them to transparently handle compressed files. (`17798`)  Various enhancements

"""  - Improved the import time of pandas by about 2.25x.  (`16764`)
- Support for `PEP 519 -- Adding a file system path protocol  <https://www.python.org/dev/peps/pep-0519/>`_ on most readers (e.g.  `read_csv`) and writers (e.g. `DataFrame.to_csv`) (`13823`).
- Added a ``__fspath__`` method to ``pd.HDFStore``, ``pd.ExcelFile``,  and ``pd.ExcelWriter`` to work properly with the file system path protocol (`13823`).
- The ``validate`` argument for `merge` now checks whether a merge is one-to-one, one-to-many, many-to-one, or many-to-many. If a merge is found to not be an example of specified merge type, an exception of type ``MergeError`` will be raised. For more, see `here <merging.validation>` (`16270`)
- Added support for PEP 518 (``pyproject.toml``) to the build system (`16745`)
- `RangeIndex.append` now returns a ``RangeIndex`` object when possible (`16212`)
- `Series.rename_axis` and `DataFrame.rename_axis` with ``inplace=True`` now return ``None`` while renaming the axis inplace. (`15704`)
- `api.types.infer_dtype` now infers decimals. (`15690`)
- `DataFrame.select_dtypes` now accepts scalar values for include/exclude as well as list-like. (`16855`)
- `date_range` now accepts 'YS' in addition to 'AS' as an alias for start of year. (`9313`)
- `date_range` now accepts 'Y' in addition to 'A' as an alias for end of year. (`9313`)
- `DataFrame.add_prefix` and `DataFrame.add_suffix` now accept strings containing the '%' character. (`17151`)
- Read/write methods that infer compression (`read_csv`, `read_table`, `read_pickle`, and `~DataFrame.to_pickle`) can now infer from path-like objects, such as ``pathlib.Path``. (`17206`)
- `read_sas` now recognizes much more of the most frequently used date (datetime) formats in SAS7BDAT files. (`15871`)
- `DataFrame.items` and `Series.items` are now present in both Python 2 and 3 and is lazy in all cases. (`13918`, `17213`)
- `pandas.io.formats.style.Styler.where` has been implemented as a convenience for `pandas.io.formats.style.Styler.applymap`. (`17474`)
- `MultiIndex.is_monotonic_decreasing` has been implemented.  Previously returned ``False`` in all cases. (`16554`)
- `read_excel` raises ``ImportError`` with a better message if ``xlrd`` is not installed. (`17613`)
- `DataFrame.assign` will preserve the original order of ``**kwargs`` for Python 3.6+ users instead of sorting the column names. (`14207`)
- `Series.reindex`, `DataFrame.reindex`, `Index.get_indexer` now support list-like argument for ``tolerance``. (`17367`)  .. _whatsnew_0210.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_0210.api_breaking.deps:  Dependencies have increased minimum versions
^^^  We have updated our minimum supported versions of dependencies (`15206`, `15543`, `15214`).
If installed, we now require:  +---+---+---+  | Package  | Minimum Version | Required |  +===+===+===+  | Numpy  | 1.9.0  |  X  |  +---+---+---+  | Matplotlib  | 1.4.3  |  |  +---+---+---+  | Scipy  | 0.14.0  |  |  +---+---+---+  | Bottleneck  | 1.0.0  |  |  +---+---+---+  Additionally, support has been dropped for Python 3.4 (`15251`).  .. _whatsnew_0210.api_breaking.bottleneck:  Sum/prod of all-NaN or empty Series/DataFrames is now consistently NaN
^^^  .. note::  The changes described here have been partially reverted. See  the `v0.22.0 Whatsnew <whatsnew_0220>` for more.  The behavior of ``sum`` and ``prod`` on all-NaN Series/DataFrames no longer depends on

whether bottleneck_ is installed, and return value of ``sum`` and ``prod`` on an empty Series has changed (`9422`, `15507`).  Calling ``sum`` or ``prod`` on an empty or all-``NaN`` ``Series``, or columns of a ``DataFrame``, will result in ``NaN``. See the `docs <missing_data.numeric_sum>`.  .. ipython:: python  s = pd.Series([np.nan])  Previously WITHOUT ``bottleneck`` installed:  .. code-block:: ipython  In [2]: s.sum()  Out[2]: np.nan  Previously WITH ``bottleneck``:  .. code-block:: ipython  In [2]: s.sum()  Out[2]: 0.0  New behavior, without regard to the bottleneck installation:  .. ipython:: python  s.sum()  Note that this also changes the sum of an empty ``Series``. Previously this always returned 0 regardless of a ``bottleneck`` installation:  .. code-block:: ipython  In [1]: pd.Series([]).sum()  Out[1]: 0  but for consistency with the all-NaN case, this was changed to return NaN as well:  .. ipython:: python  pd.Series([]).sum()  .. _whatsnew_0210.api_breaking.loc:  Indexing with a list with missing labels is deprecated
^^^  Previously, selecting with a list of labels, where one or more labels were missing would always succeed, returning ``NaN`` for missing labels.
This will now show a ``FutureWarning``. In the future this will raise a ``KeyError`` (`15747`).
This warning will trigger on a ``DataFrame`` or a ``Series`` for using ``.loc[]``  or ``[[]]`` when passing a list-of-labels with at least 1 missing label.
See the `deprecation docs <indexing.deprecate_loc_reindex_listlike>`.  .. ipython:: python  s = pd.Series([1, 2, 3])  s  Previous behavior  .. code-block:: ipython  In [4]: s.loc[[1, 2, 3]]  Out[4]:  1  2.0  2  3.0  3  NaN  dtype: float64  Current behavior  .. code-block:: ipython  In [4]: s.loc[[1, 2, 3]]  Passing list-likes to .loc or [] with any missing label will raise  KeyError in the future, you can use .reindex() as an alternative.  See the documentation here:  https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike  Out[4]:  1  2.0  2  3.0  3  NaN  dtype: float64  The idiomatic way to achieve selecting potentially not-found elements is via ``.reindex()``  .. ipython:: python  s.reindex([1, 2, 3])  Selection with all keys found is unchanged.  .. ipython:: python  s.loc[[1, 2]]  .. _whatsnew_0210.api.na_changes:  NA naming changes
^^^  In order to promote more consistency among the pandas API, we have added additional top-level
functions `isna` and `notna` that are aliases for `isnull` and `notnull`.
The naming scheme is now more consistent with methods like ``.dropna()`` and ``.fillna()``. Furthermore
in all cases where ``.isnull()`` and ``.notnull()`` methods are defined, these have additional methods
named ``.isna()`` and ``.notna()``, these are included for classes ``Categorical``,
``Index``, ``Series``, and ``DataFrame``. (`15001`).  The configuration option ``pd.options.mode.use_inf_as_null`` is deprecated, and ``pd.options.mode.use_inf_as_na`` is added as a replacement.  .. _whatsnew_0210.api_breaking.iteration_scalars:  Iteration of Series/Index will now return Python scalars

^^^  Previously, when using certain iteration methods for a ``Series`` with dtype ``int`` or ``float``, you would receive a ``numpy`` scalar, e.g. a ``np.int64``, rather than a Python ``int``. Issue (`10904`) corrected this for ``Series.tolist()`` and ``list(Series)``. This change makes all iteration methods consistent, in particular, for ``__iter__()`` and ``.map()``; note that this only affects int/float dtypes. (`13236`, `13258`, `14216`).  .. ipython:: python  s = pd.Series([1, 2, 3])  s  Previously:  .. code-block:: ipython  In [2]: type(list(s)[0])  Out[2]: numpy.int64  New behavior:  .. ipython:: python  type(list(s)[0])  Furthermore this will now correctly box the results of iteration for `DataFrame.to_dict` as well.  .. ipython:: python  d = {'a': [1], 'b': ['b']}  df = pd.DataFrame(d)  Previously:  .. code-block:: ipython  In [8]: type(df.to_dict()['a'][0])  Out[8]: numpy.int64  New behavior:  .. ipython:: python  type(df.to_dict()['a'][0])  .. _whatsnew_0210.api_breaking.loc_with_index:  Indexing with a Boolean Index
^^^  Previously when passing a boolean ``Index`` to ``.loc``, if the index of the ``Series/DataFrame`` had ``boolean`` labels,
you would get a label based selection, potentially duplicating result labels, rather than a boolean indexing selection
(where ``True`` selects elements), this was inconsistent how a boolean numpy array indexed. The new behavior is to
act like a boolean numpy array indexer. (`17738`)  Previous behavior:  .. ipython:: python  s = pd.Series([1, 2, 3], index=[False, True, False])  s  .. code-block:: ipython  In [59]: s.loc[pd.Index([True, False, True])]  Out[59]:  True  2  False  1  False  3  True  2  dtype: int64  Current behavior  .. ipython:: python  s.loc[pd.Index([True, False, True])]  Furthermore, previously if you had an index that was non-numeric (e.g. strings), then a boolean Index would raise a ``KeyError``.
This will now be treated as a boolean indexer.  Previously behavior:  .. ipython:: python  s = pd.Series([1, 2, 3], index=['a', 'b', 'c'])  s  .. code-block:: ipython  In [39]: s.loc[pd.Index([True, False, True])]  KeyError: "None of [Index([True, False, True], dtype='object')] are in the [index]"  Current behavior  .. ipython:: python  s.loc[pd.Index([True, False, True])]  .. _whatsnew_0210.api_breaking.period_index_resampling:  ``PeriodIndex`` resampling

^^^  In previous versions of pandas, resampling a ``Series``/``DataFrame`` indexed by a ``PeriodIndex`` returned a ``DatetimeIndex`` in some cases (`12884`). Resampling to a multiplied frequency now returns a ``PeriodIndex`` (`15944`). As a minor enhancement, resampling a ``PeriodIndex`` can now handle ``NaT`` values (`13224`)  Previous behavior:  .. code-block:: ipython  In [1]: pi = pd.period_range('2017-01', periods=12, freq='M')  In [2]: s = pd.Series(np.arange(12), index=pi)  In [3]: resampled = s.resample('2Q').mean()  In [4]: resampled  Out[4]:  2017-03-31  1.0  2017-09-30  5.5  2018-03-31  10.0  Freq: 2Q-DEC, dtype: float64  In [5]: resampled.index  Out[5]: DatetimeIndex(['2017-03-31', '2017-09-30', '2018-03-31'], dtype='datetime64[ns]', freq='2Q-DEC')  New behavior:  .. ipython:: python  pi = pd.period_range('2017-01', periods=12, freq='M')  s = pd.Series(np.arange(12), index=pi)  resampled = s.resample('2Q').mean()  resampled  resampled.index  Upsampling and calling ``.ohlc()`` previously returned a ``Series``, basically identical to calling ``.asfreq()``. OHLC upsampling now returns a DataFrame with columns ``open``, ``high``, ``low`` and ``close`` (`13083`). This is consistent with downsampling and ``DatetimeIndex`` behavior.  Previous behavior:  .. code-block:: ipython  In [1]: pi = pd.period_range(start='2000-01-01', freq='D', periods=10)  In [2]: s = pd.Series(np.arange(10), index=pi)  In [3]: s.resample('H').ohlc()  Out[3]:  2000-01-01 00:00  0.0  ...  2000-01-10 23:00  NaN  Freq: H, Length: 240, dtype: float64  In [4]: s.resample('M').ohlc()  Out[4]:  open  high  low  close  2000-01  0  9  0  9  New behavior:  .. ipython:: python  pi = pd.period_range(start='2000-01-01', freq='D', periods=10)  s = pd.Series(np.arange(10), index=pi)  s.resample('H').ohlc()  s.resample('M').ohlc()  .. _whatsnew_0210.api_breaking.pandas_eval:  Improved error handling during item assignment in pd.eval
^^^  `eval` will now raise a ``ValueError`` when item assignment malfunctions, or
inplace operations are specified, but there is no item assignment in the expression (`16732`)  .. ipython:: python  arr = np.array([1, 2, 3])  Previously, if you attempted the following expression, you would get a not very helpful error message:  .. code-block:: ipython  In [3]: pd.eval("a = 1 + 2", target=arr, inplace=True)  ...  IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`)  and integer or boolean arrays are valid indices  This is a very long way of saying numpy arrays don't support string-item indexing. With this
change, the error message is now this:  .. code-block:: python  In [3]: pd.eval("a = 1 + 2", target=arr, inplace=True)  ...  ValueError: Cannot assign expression output to target  It also used to be possible to evaluate expressions inplace, even if there was no item assignment:  .. code-block:: ipython  In [4]: pd.eval("1 + 2", target=arr, inplace=True)  Out[4]: 3  However, this input does not make much sense because the output is not being assigned to
the target. Now, a ``ValueError`` will be raised when such an input is passed in:  .. code-block:: ipython  In [4]: pd.eval("1 + 2", target=arr, inplace=True)  ...  ValueError: Cannot operate inplace if there is no assignment  .. _whatsnew_0210.api_breaking.dtype_conversions:  Dtype conversions
^^^  Previously assignments, ``.where()`` and ``.fillna()`` with a ``bool`` assignment, would coerce to same the type (e.g. int / float), or raise for datetimelikes. These will now preserve the bools with ``object`` dtypes. (`16821`).  .. ipython:: python  s = pd.Series([1, 2, 3])  .. code-block:: python  In [5]: s[1] = True  In [6]: s  Out[6]:  0  1  1  1  2  3  dtype: int64  New behavior  .. ipython:: python  s[1] = True  s  Previously, as assignment to a datetimelike with a non-datetimelike would coerce the

non-datetime-like item being assigned (`14145`).  .. ipython:: python  s = pd.Series([pd.Timestamp('2011-01-01'), pd.Timestamp('2012-01-01')])  .. code-block:: python  In [1]: s[1] = 1  In [2]: s  Out[2]:  0  2011-01-01 000.000  1  1970-01-01 000.0001  dtype: datetime64[ns]  These now coerce to ``object`` dtype.  .. ipython:: python  s[1] = 1  s  - Inconsistent behavior in ``.where()`` with datetimelikes which would raise rather than coerce to ``object`` (`16402`)
- Bug in assignment against ``int64`` data with ``np.ndarray`` with ``float64`` dtype may keep ``int64`` dtype (`14001`)  .. _whatsnew_210.api.multiindex_single:  MultiIndex constructor with a single level
^^^  The ``MultiIndex`` constructors no longer squeezes a MultiIndex with all
length-one levels down to a regular ``Index``. This affects all the
``MultiIndex`` constructors. (`17178`)  Previous behavior:  .. code-block:: ipython  In [2]: pd.MultiIndex.from_tuples([('a',), ('b',)])  Out[2]: Index(['a', 'b'], dtype='object')  Length 1 levels are no longer special-cased. They behave exactly as if you had
length 2+ levels, so a `MultiIndex` is always returned from all of the
``MultiIndex`` constructors:  .. ipython:: python  pd.MultiIndex.from_tuples([('a',), ('b',)])  .. _whatsnew_0210.api.utc_localization_with_series:  UTC localization with Series
^^^  Previously, `to_datetime` did not localize datetime ``Series`` data when ``utc=True`` was passed. Now, `to_datetime` will correctly localize ``Series`` with a ``datetime64[ns, UTC]`` dtype to be consistent with how list-like and ``Index`` data are handled. (`6415`).  Previous behavior  .. ipython:: python  s = pd.Series(['20130101 000'] * 3)  .. code-block:: ipython  In [12]: pd.to_datetime(s, utc=True)  Out[12]:  0  2013-01-01  1  2013-01-01  2  2013-01-01  dtype: datetime64[ns]  New behavior  .. ipython:: python  pd.to_datetime(s, utc=True)  Additionally, DataFrames with datetime columns that were parsed by `read_sql_table` and `read_sql_query` will also be localized to UTC only if the original SQL columns were timezone aware datetime columns.  .. _whatsnew_0210.api.consistency_of_range_functions:  Consistency of range functions
^^^  In previous versions, there were some inconsistencies between the various range functions: `date_range`, `bdate_range`, `period_range`, `timedelta_range`, and `interval_range`. (`17471`).  One of the inconsistent behaviors occurred when the ``start``, ``end`` and ``period`` parameters were all specified, potentially leading to ambiguous ranges.  When all three parameters were passed, ``interval_range`` ignored the ``period`` parameter, ``period_range`` ignored the ``end`` parameter, and the other range functions raised.  To promote consistency among the range functions, and avoid potentially ambiguous ranges, ``interval_range`` and ``period_range`` will now raise when all three parameters are passed.  Previous behavior:  .. code-block:: ipython  In [2]: pd.interval_range(start=0, end=4, periods=6)  Out[2]:  IntervalIndex([(0, 1], (1, 2], (2, 3]]  closed='right',  dtype='interval[int64]')  In [3]: pd.period_range(start='2017Q1', end='2017Q4', periods=6, freq='Q')  Out[3]: PeriodIndex(['2017Q1', '2017Q2', '2017Q3', '2017Q4', '2018Q1', '2018Q2'], dtype='period[Q-DEC]', freq='Q-DEC')  New behavior:  .. code-block:: ipython

In [2]: pd.interval_range(start=0, end=4, periods=6)
  ValueError: Of the three parameters: start, end, and periods, exactly two must be specified

In [3]: pd.period_range(start='2017Q1', end='2017Q4', periods=6, freq='Q')
  ValueError: Of the three parameters: start, end, and periods, exactly two must be specified  Additionally, the endpoint parameter ``end`` was not included in the intervals produced by ``interval_range``.  However, all other range functions include ``end`` in their output.  To promote consistency among the range functions, ``interval_range`` will now include ``end`` as the right endpoint of the final interval, except if ``freq`` is specified in a way which skips ``end``.  Previous behavior:  .. code-block:: ipython  In [4]: pd.interval_range(start=0, end=4)  Out[4]:  IntervalIndex([(0, 1], (1, 2], (2, 3]]  closed='right',  dtype='interval[int64]')  New behavior:  .. ipython:: python  pd.interval_range(start=0, end=4)  .. _whatsnew_0210.api.mpl_converters:  No automatic Matplotlib converters
^^^  pandas no longer registers our ``date``, ``time``, ``datetime``,
``datetime64``, and ``Period`` converters with matplotlib when pandas is
imported. Matplotlib plot methods (``plt.plot``, ``ax.plot``, ...), will not
nicely format the x-axis for ``DatetimeIndex`` or ``PeriodIndex`` values. You
must explicitly register these methods:  pandas built-in ``Series.plot`` and ``DataFrame.plot`` *will* register these
converters on first-use (`17710`).  .. note::  This change has been temporarily reverted in pandas 0.21.1,  for more details see `here <whatsnew_0211.converters>`.  .. _whatsnew_0210.api:  Other API changes
^^^  - The Categorical constructor no longer accepts a scalar for the ``categories`` keyword. (`16022`)
- Accessing a non-existent attribute on a closed `~pandas.HDFStore` will now  raise an ``AttributeError`` rather than a ``ClosedFileError`` (`16301`)
- `read_csv` now issues a ``UserWarning`` if the ``names`` parameter contains duplicates (`17095`)
- `read_csv` now treats ``'null'`` and ``'n/a'`` strings as missing values by default (`16471`, `16078`)
- `pandas.HDFStore`'s string representation is now faster and less detailed. For the previous behavior, use ``pandas.HDFStore.info()``. (`16503`).
- Compression defaults in HDF stores now follow pytables standards. Default is no compression and if ``complib`` is missing and ``complevel`` > 0 ``zlib`` is used (`15943`)
- ``Index.get_indexer_non_unique()`` now returns a ndarray indexer rather than an ``Index``; this is consistent with ``Index.get_indexer()`` (`16819`)
- Removed the ``@slow`` decorator from ``pandas._testing``, which caused issues for some downstream packages' test suites. Use ``@pytest.mark.slow`` instead, which achieves the same thing (`16850`)
- Moved definition of ``MergeError`` to the ``pandas.errors`` module.
- The signature of `Series.set_axis` and `DataFrame.set_axis` has been changed from ``set_axis(axis, labels)`` to ``set_axis(labels, axis=0)``, for consistency with the rest of the API. The old signature is deprecated and will show a ``FutureWarning`` (`14636`)
- `Series.argmin` and `Series.argmax` will now raise a ``TypeError`` when used with ``object`` dtypes, instead of a ``ValueError`` (`13595`)
- `Period` is now immutable, and will now raise an ``AttributeError`` when a user tries to assign a new value to the ``ordinal`` or ``freq`` attributes (`17116`).
- `to_datetime` when passed a tz-aware ``origin=`` kwarg will now raise a more informative ``ValueError`` rather than a ``TypeError`` (`16842`)
- `to_datetime` now raises a ``ValueError`` when format includes ``%W`` or ``%U`` without also including day of the week and calendar year (`16774`)
- Renamed non-functional ``index`` to ``index_col`` in `read_stata` to improve API consistency (`16342`)
- Bug in `DataFrame.drop` caused boolean labels ``False`` and ``True`` to be treated as labels 0 and 1 respectively when dropping indices from a numeric index. This will now raise a ValueError (`16877`)

- Restricted DateOffset keyword arguments.  Previously, ``DateOffset`` subclasses allowed arbitrary keyword arguments which could lead to unexpected behavior.  Now, only valid arguments will be accepted. (`17176`).  .. _whatsnew_0210.deprecations:  Deprecations
~~~  - `DataFrame.from_csv` and `Series.from_csv` have been deprecated in favor of `read_csv()` (`4191`)
- `read_excel()` has deprecated ``sheetname`` in favor of ``sheet_name`` for consistency with ``.to_excel()`` (`10559`).
- `read_excel()` has deprecated ``parse_cols`` in favor of ``usecols`` for consistency with `read_csv` (`4988`)
- `read_csv()` has deprecated the ``tupleize_cols`` argument. Column tuples will always be converted to a ``MultiIndex`` (`17060`)
- `DataFrame.to_csv` has deprecated the ``tupleize_cols`` argument. MultiIndex columns will be always written as rows in the CSV file (`17060`)
- The ``convert`` parameter has been deprecated in the ``.take()`` method, as it was not being respected (`16948`)
- ``pd.options.html.border`` has been deprecated in favor of ``pd.options.display.html.border`` (`15793`).
- `SeriesGroupBy.nth` has deprecated ``True`` in favor of ``'all'`` for its kwarg ``dropna`` (`11038`).
- `DataFrame.as_blocks` is deprecated, as this is exposing the internal implementation (`17302`)
- ``pd.TimeGrouper`` is deprecated in favor of `pandas.Grouper` (`16747`)
- ``cdate_range`` has been deprecated in favor of `bdate_range`, which has gained ``weekmask`` and ``holidays`` parameters for building custom frequency date ranges. See the `documentation <timeseries.custom-freq-ranges>` for more details (`17596`)
- passing ``categories`` or ``ordered`` kwargs to `Series.astype` is deprecated, in favor of passing a `CategoricalDtype <whatsnew_0210.enhancements.categorical_dtype>` (`17636`)
- ``.get_value`` and ``.set_value`` on ``Series``, ``DataFrame``, ``Panel``, ``SparseSeries``, and ``SparseDataFrame`` are deprecated in favor of using ``.iat[]`` or ``.at[]`` accessors (`15269`)
- Passing a non-existent column in ``.to_excel(..., columns=)`` is deprecated and will raise a ``KeyError`` in the future (`17295`)
- ``raise_on_error`` parameter to `Series.where`, `Series.mask`, `DataFrame.where`, `DataFrame.mask` is deprecated, in favor of ``errors=`` (`14968`)
- Using `DataFrame.rename_axis` and `Series.rename_axis` to alter index or column *labels* is now deprecated in favor of using ``.rename``. ``rename_axis`` may still be used to alter the name of the index or columns (`17833`).
- `~DataFrame.reindex_axis` has been deprecated in favor of `~DataFrame.reindex`. See `here <whatsnew_0210.enhancements.rename_reindex_axis>` for more (`17833`).  .. _whatsnew_0210.deprecations.select:  Series.select and DataFrame.select
^^^  The `Series.select` and `DataFrame.select` methods are deprecated in favor of using ``df.loc[labels.map(crit)]`` (`12401`)  .. ipython:: python  df = pd.DataFrame({'A': [1, 2, 3]}, index=['foo', 'bar', 'baz'])  .. code-block:: ipython  In [3]: df.select(lambda x: x in ['bar', 'baz'])  FutureWarning: select is deprecated and will be removed in a future release. You can use .loc[crit] as a replacement  Out[3]:  A  bar  2  baz  3  .. ipython:: python  df.loc[df.index.map(lambda x: x in ['bar', 'baz'])]  .. _whatsnew_0210.deprecations.argmin_min:  Series.argmax and Series.argmin
^^^  The behavior of `Series.argmax` and `Series.argmin` have been deprecated in favor of `Series.idxmax` and `Series.idxmin`, respectively (`16830`).  For compatibility with NumPy arrays, ``pd.Series`` implements ``argmax`` and
``argmin``. Since pandas 0.13.0, ``argmax`` has been an alias for
`pandas.Series.idxmax`, and ``argmin`` has been an alias for
`pandas.Series.idxmin`. They return the *label* of the maximum or minimum,
rather than the *position*.  We've deprecated the current behavior of ``Series.argmax`` and
``Series.argmin``. Using either of these will emit a ``FutureWarning``. Use
`Series.idxmax` if you want the label of the maximum. Use

``Series.values.argmax()`` if you want the position of the maximum. Likewise for
the minimum. In a future release ``Series.argmax`` and ``Series.argmin`` will
return the position of the maximum or minimum.  .. _whatsnew_0210.prior_deprecations:  Removal of prior version deprecations/changes
~~~  - `read_excel()` has dropped the ``has_index_names`` parameter (`10967`)
- The ``pd.options.display.height`` configuration has been dropped (`3663`)
- The ``pd.options.display.line_width`` configuration has been dropped (`2881`)
- The ``pd.options.display.mpl_style`` configuration has been dropped (`12190`)
- ``Index`` has dropped the ``.sym_diff()`` method in favor of ``.symmetric_difference()`` (`12591`)
- ``Categorical`` has dropped the ``.order()`` and ``.sort()`` methods in favor of ``.sort_values()`` (`12882`)
- `eval` and `DataFrame.eval` have changed the default of ``inplace`` from ``None`` to ``False`` (`11149`)
- The function ``get_offset_name`` has been dropped in favor of the ``.freqstr`` attribute for an offset (`11834`)
- pandas no longer tests for compatibility with hdf5-files created with pandas < 0.11 (`17404`).  .. _whatsnew_0210.performance:  Performance improvements
~~~  - Improved performance of instantiating `SparseDataFrame` (`16773`)
- `Series.dt` no longer performs frequency inference, yielding a large speedup when accessing the attribute (`17210`)
- Improved performance of `~Series.cat.set_categories` by not materializing the values (`17508`)
- `Timestamp.microsecond` no longer re-computes on attribute access (`17331`)
- Improved performance of the `CategoricalIndex` for data that is already categorical dtype (`17513`)
- Improved performance of `RangeIndex.min` and `RangeIndex.max` by using ``RangeIndex`` properties to perform the computations (`17607`)  .. _whatsnew_0210.docs:  Documentation changes
~~~  - Several ``NaT`` method docstrings (e.g. `NaT.ctime`) were incorrect (`17327`)
- The documentation has had references to versions < v0.17 removed and cleaned up (`17442`, `17442`, `17404` & `17504`)  .. _whatsnew_0210.bug_fixes:  Bug fixes
^^^  - Bug in assignment against datetime-like data with ``int`` may incorrectly convert to datetime-like (`14145`)
- Bug in assignment against ``int64`` data with ``np.ndarray`` with ``float64`` dtype may keep ``int64`` dtype (`14001`)
- Fixed the return type of ``IntervalIndex.is_non_overlapping_monotonic`` to be a Python ``bool`` for consistency with similar attributes/methods.  Previously returned a ``numpy.bool_``. (`17237`)
- Bug in ``IntervalIndex.is_non_overlapping_monotonic`` when intervals are closed on both sides and overlap at a point (`16560`)
- Bug in `Series.fillna` returns frame when ``inplace=True`` and ``value`` is dict (`16156`)
- Bug in `Timestamp.weekday_name` returning a UTC-based weekday name when localized to a timezone (`17354`)
- Bug in ``Timestamp.replace`` when replacing ``tzinfo`` around DST changes (`15683`)
- Bug in ``Timedelta`` construction and arithmetic that would not propagate the ``Overflow`` exception (`17367`)
- Bug in `~DataFrame.astype` converting to object dtype when passed extension type classes (``DatetimeTZDtype``, ``CategoricalDtype``) rather than instances. Now a ``TypeError`` is raised when a class is passed (`17780`).
- Bug in `to_numeric` in which elements were not always being coerced to numeric when ``errors='coerce'`` (`17007`, `17125`)
- Bug in ``DataFrame`` and ``Series`` constructors where ``range`` objects are converted to ``int32`` dtype on Windows instead of ``int64`` (`16804`)  Indexing
^^^  - When called with a null slice (e.g. ``df.iloc[:]``), the ``.iloc`` and ``.loc`` indexers return a shallow copy of the original object. Previously they returned the original object. (`13873`).

- When called on an unsorted ``MultiIndex``, the ``loc`` indexer now will raise ``UnsortedIndexError`` only if proper slicing is used on non-sorted levels (`16734`).
- Fixes regression in 0.20.3 when indexing with a string on a ``TimedeltaIndex`` (`16896`).
- Fixed `TimedeltaIndex.get_loc` handling of ``np.timedelta64`` inputs (`16909`).
- Fix `MultiIndex.sort_index` ordering when ``ascending`` argument is a list, but not all levels are specified, or are in a different order (`16934`).
- Fixes bug where indexing with ``np.inf`` caused an ``OverflowError`` to be raised (`16957`)
- Bug in reindexing on an empty ``CategoricalIndex`` (`16770`)
- Fixes ``DataFrame.loc`` for setting with alignment and tz-aware ``DatetimeIndex`` (`16889`)
- Avoids ``IndexError`` when passing an Index or Series to ``.iloc`` with older numpy (`17193`)
- Allow unicode empty strings as placeholders in multilevel columns in Python 2 (`17099`)
- Bug in ``.iloc`` when used with inplace addition or assignment and an int indexer on a ``MultiIndex`` causing the wrong indexes to be read from and written to (`17148`)
- Bug in ``.isin()`` in which checking membership in empty ``Series`` objects raised an error (`16991`)
- Bug in ``CategoricalIndex`` reindexing in which specified indices containing duplicates were not being respected (`17323`)
- Bug in intersection of ``RangeIndex`` with negative step (`17296`)
- Bug in ``IntervalIndex`` where performing a scalar lookup fails for included right endpoints of non-overlapping monotonic decreasing indexes (`16417`, `17271`)
- Bug in `DataFrame.first_valid_index` and `DataFrame.last_valid_index` when no valid entry (`17400`)
- Bug in `Series.rename` when called with a callable, incorrectly alters the name of the ``Series``, rather than the name of the ``Index``. (`17407`)
- Bug in `String.str_get` raises ``IndexError`` instead of inserting NaNs when using a negative index. (`17704`)  IO
^^  - Bug in `read_hdf` when reading a timezone aware index from ``fixed`` format HDFStore (`17618`)
- Bug in `read_csv` in which columns were not being thoroughly de-duplicated (`17060`)
- Bug in `read_csv` in which specified column names were not being thoroughly de-duplicated (`17095`)
- Bug in `read_csv` in which non integer values for the header argument generated an unhelpful / unrelated error message (`16338`)
- Bug in `read_csv` in which memory management issues in exception handling, under certain conditions, would cause the interpreter to segfault (`14696`, `16798`).
- Bug in `read_csv` when called with ``low_memory=False`` in which a CSV with at least one column > 2GB in size would incorrectly raise a ``MemoryError`` (`16798`).
- Bug in `read_csv` when called with a single-element list ``header`` would return a ``DataFrame`` of all NaN values (`7757`)
- Bug in `DataFrame.to_csv` defaulting to 'ascii' encoding in Python 3, instead of 'utf-8' (`17097`)
- Bug in `read_stata` where value labels could not be read when using an iterator (`16923`)
- Bug in `read_stata` where the index was not set (`16342`)
- Bug in `read_html` where import check fails when run in multiple threads (`16928`)
- Bug in `read_csv` where automatic delimiter detection caused a ``TypeError`` to be thrown when a bad line was encountered rather than the correct error message (`13374`)
- Bug in `DataFrame.to_html` with ``notebook=True`` where DataFrames with named indices or non-MultiIndex indices had undesired horizontal or vertical alignment for column or row labels, respectively (`16792`)
- Bug in `DataFrame.to_html` in which there was no validation of the ``justify`` parameter (`17527`)
- Bug in `HDFStore.select` when reading a contiguous mixed-data table featuring VLArray (`17021`)

- Bug in `to_json` where several conditions (including objects with unprintable symbols, objects with deep recursion, overlong labels) caused segfaults instead of raising the appropriate exception (`14256`)  Plotting
^^^
- Bug in plotting methods using ``secondary_y`` and ``fontsize`` not setting secondary axis font size (`12565`)
- Bug when plotting ``timedelta`` and ``datetime`` dtypes on y-axis (`16953`)
- Line plots no longer assume monotonic x data when calculating xlims, they show the entire lines now even for unsorted x data. (`11310`, `11471`)
- With matplotlib 2.0.0 and above, calculation of x limits for line plots is left to matplotlib, so that its new default settings are applied. (`15495`)
- Bug in ``Series.plot.bar`` or ``DataFrame.plot.bar`` with ``y`` not respecting user-passed ``color`` (`16822`)
- Bug causing ``plotting.parallel_coordinates`` to reset the random seed when using random colors (`17525`)  GroupBy/resample/rolling
^^^  - Bug in ``DataFrame.resample(...).size()`` where an empty ``DataFrame`` did not return a ``Series`` (`14962`)
- Bug in `infer_freq` causing indices with 2-day gaps during the working week to be wrongly inferred as business daily (`16624`)
- Bug in ``.rolling(...).quantile()`` which incorrectly used different defaults than `Series.quantile()` and `DataFrame.quantile()` (`9413`, `16211`)
- Bug in ``groupby.transform()`` that would coerce boolean dtypes back to float (`16875`)
- Bug in ``Series.resample(...).apply()`` where an empty ``Series`` modified the source index and did not return the name of a ``Series`` (`14313`)
- Bug in ``.rolling(...).apply(...)`` with a ``DataFrame`` with a ``DatetimeIndex``, a ``window`` of a timedelta-convertible and ``min_periods >= 1`` (`15305`)
- Bug in ``DataFrame.groupby`` where index and column keys were not recognized correctly when the number of keys equaled the number of elements on the groupby axis (`16859`)
- Bug in ``groupby.nunique()`` with ``TimeGrouper`` which cannot handle ``NaT`` correctly (`17575`)
- Bug in ``DataFrame.groupby`` where a single level selection from a ``MultiIndex`` unexpectedly sorts (`17537`)
- Bug in ``DataFrame.groupby`` where spurious warning is raised when ``Grouper`` object is used to override ambiguous column name (`17383`)
- Bug in ``TimeGrouper`` differs when passes as a list and as a scalar (`17530`)  Sparse
^^^  - Bug in ``SparseSeries`` raises ``AttributeError`` when a dictionary is passed in as data (`16905`)
- Bug in `SparseDataFrame.fillna` not filling all NaNs when frame was instantiated from SciPy sparse matrix (`16112`)
- Bug in `SparseSeries.unstack` and `SparseDataFrame.stack` (`16614`, `15045`)
- Bug in `make_sparse` treating two numeric/boolean data, which have same bits, as same when array ``dtype`` is ``object`` (`17574`)
- `SparseArray.all` and `SparseArray.any` are now implemented to handle ``SparseArray``, these were used but not implemented (`17570`)  Reshaping
^^^
- Joining/Merging with a non unique ``PeriodIndex`` raised a ``TypeError`` (`16871`)
- Bug in `crosstab` where non-aligned series of integers were casted to float (`17005`)
- Bug in merging with categorical dtypes with datetimelikes incorrectly raised a ``TypeError`` (`16900`)
- Bug when using `isin` on a large object series and large comparison array (`16012`)
- Fixes regression from 0.20, `Series.aggregate` and `DataFrame.aggregate` allow dictionaries as return values again (`16741`)
- Fixes dtype of result with integer dtype input, from `pivot_table` when called with ``margins=True`` (`17013`)
- Bug in `crosstab` where passing two ``Series`` with the same name raised a ``KeyError`` (`13279`)
- `Series.argmin`, `Series.argmax`, and their counterparts on ``DataFrame`` and groupby objects work correctly with floating point data that contains infinite values (`13595`).

- Bug in `unique` where checking a tuple of strings raised a ``TypeError`` (`17108`)
- Bug in `concat` where order of result index was unpredictable if it contained non-comparable elements (`17344`)
- Fixes regression when sorting by multiple columns on a ``datetime64`` dtype ``Series`` with ``NaT`` values (`16836`)
- Bug in `pivot_table` where the result's columns did not preserve the categorical dtype of ``columns`` when ``dropna`` was ``False`` (`17842`)
- Bug in ``DataFrame.drop_duplicates`` where dropping with non-unique column names raised a ``ValueError`` (`17836`)
- Bug in `unstack` which, when called on a list of levels, would discard the ``fillna`` argument (`13971`)
- Bug in the alignment of ``range`` objects and other list-likes with ``DataFrame`` leading to operations being performed row-wise instead of column-wise (`17901`)  Numeric
^^^
- Bug in ``.clip()`` with ``axis=1`` and a list-like for ``threshold`` is passed; previously this raised ``ValueError`` (`15390`)
- `Series.clip()` and `DataFrame.clip()` now treat NA values for upper and lower arguments as ``None`` instead of raising ``ValueError`` (`17276`).  Categorical
^^^
- Bug in `Series.isin` when called with a categorical (`16639`)
- Bug in the categorical constructor with empty values and categories causing the ``.categories`` to be an empty ``Float64Index`` rather than an empty ``Index`` with object dtype (`17248`)
- Bug in categorical operations with `Series.cat <categorical.cat>` not preserving the original Series' name (`17509`)
- Bug in `DataFrame.merge` failing for categorical columns with boolean/int data types (`17187`)
- Bug in constructing a ``Categorical``/``CategoricalDtype`` when the specified ``categories`` are of categorical type (`17884`).  .. _whatsnew_0210.pypy:  PyPy
^^^  - Compatibility with PyPy in `read_csv` with ``usecols=[<unsorted ints>]`` and  `read_json` (`17351`)
- Split tests into cases for CPython and PyPy where needed, which highlights the fragility  of index matching with ``float('nan')``, ``np.nan`` and ``NAT`` (`17351`)
- Fix `DataFrame.memory_usage` to support PyPy. Objects on PyPy do not have a fixed size,  so an approximation is used instead (`17228`)  Other
^^^
- Bug where some inplace operators were not being wrapped and produced a copy when invoked (`12962`)
- Bug in `eval` where the ``inplace`` parameter was being incorrectly handled (`16732`)  .. _whatsnew_0.21.0.contributors:  Contributors
~~~  .. contributors:: v0.20.3..v0.21.0

{{ header }}  .. ipython:: python  from pandas import *  # noqa F401, F403  This is a minor bug-fix release in the 0.21.x series and includes some small regression fixes,
We recommend that all users upgrade to this version.  Highlights include:  - Temporarily restore matplotlib datetime plotting functionality. This should  resolve issues for users who implicitly relied on pandas to plot datetimes  with matplotlib. See `here <whatsnew_0211.converters>`.
- Improvements to the Parquet IO functions introduced in 0.21.0. See  `here <whatsnew_0211.enhancements.parquet>`.  .. contents:: What's new in v0.21.1  none  .. _whatsnew_0211.converters:  Restore Matplotlib datetime converter registration
~~~  pandas implements some matplotlib converters for nicely formatting the axis
labels on plots with ``datetime`` or ``Period`` values. Prior to pandas 0.21.0,
these were implicitly registered with matplotlib, as a side effect of ``import
pandas``.  In pandas 0.21.0, we required users to explicitly register the
converter. This caused problems for some users who relied on those converters
being present for regular ``matplotlib.pyplot`` plotting methods, so we're
temporarily reverting that change; pandas 0.21.1 again registers the converters on
import, just like before 0.21.0.  We've added a new option to control the converters:
``pd.options.plotting.matplotlib.register_converters``. By default, they are
registered. Toggling this to ``False`` removes pandas' formatters and restore
any converters we overwrote when registering them (`18301`).  We're working with the matplotlib developers to make this easier. We're trying
to balance user convenience (automatically registering the converters) with
import performance and best practices (importing pandas shouldn't have the side
effect of overwriting any custom converters you've already set). In the future
we hope to have most of the datetime formatting functionality in matplotlib,
with just the pandas-specific converters in pandas. We'll then gracefully
deprecate the automatic registration of converters in favor of users explicitly
registering them when they want them.  .. _whatsnew_0211.enhancements:  New features
~~~  .. _whatsnew_0211.enhancements.parquet:  Improvements to the Parquet IO functionality
^^^  - `DataFrame.to_parquet` will now write non-default indexes when the  underlying engine supports it. The indexes will be preserved when reading  back in with `read_parquet` (`18581`).
- `read_parquet` now allows to specify the columns to read from a parquet file (`18154`)
- `read_parquet` now allows to specify kwargs which are passed to the respective engine (`18216`)  .. _whatsnew_0211.enhancements.other:  Other enhancements
^^^  - `Timestamp.timestamp` is now available in Python 2.7. (`17329`)
- `Grouper` and `TimeGrouper` now have a friendly repr output (`18203`).  .. _whatsnew_0211.deprecations:  Deprecations
~~~  - ``pandas.tseries.register`` has been renamed to  `pandas.plotting.register_matplotlib_converters` (`18301`)  .. _whatsnew_0211.performance:  Performance improvements
~~~  - Improved performance of plotting large series/dataframes (`18236`).  .. _whatsnew_0211.bug_fixes:  Bug fixes
^^^  - Bug in `TimedeltaIndex` subtraction could incorrectly overflow when ``NaT`` is present (`17791`)
- Bug in `DatetimeIndex` subtracting datetimelike from DatetimeIndex could fail to overflow (`18020`)
- Bug in `IntervalIndex.copy` when copying and ``IntervalIndex`` with non-default ``closed`` (`18339`)
- Bug in `DataFrame.to_dict` where columns of datetime that are tz-aware were not converted to required arrays when used with ``orient='records'``, raising ``TypeError`` (`18372`)
- Bug in `DateTimeIndex` and `date_range` where mismatching tz-aware ``start`` and ``end`` timezones would not raise an err if ``end.tzinfo`` is None (`18431`)
- Bug in `Series.fillna` which raised when passed a long integer on Python 2 (`18159`).  Indexing

^^^  - Bug in a boolean comparison of a ``datetime.datetime`` and a ``datetime64[ns]`` dtype Series (`17965`)
- Bug where a ``MultiIndex`` with more than a million records was not raising ``AttributeError`` when trying to access a missing attribute (`18165`)
- Bug in `IntervalIndex` constructor when a list of intervals is passed with non-default ``closed`` (`18334`)
- Bug in ``Index.putmask`` when an invalid mask passed (`18368`)
- Bug in masked assignment of a ``timedelta64[ns]`` dtype ``Series``, incorrectly coerced to float (`18493`)  IO
^^  - Bug in `~pandas.io.stata.StataReader` not converting date/time columns with display formatting addressed (`17990`). Previously columns with display formatting were normally left as ordinal numbers and not converted to datetime objects.
- Bug in `read_csv` when reading a compressed UTF-16 encoded file (`18071`)
- Bug in `read_csv` for handling null values in index columns when specifying ``na_filter=False`` (`5239`)
- Bug in `read_csv` when reading numeric category fields with high cardinality (`18186`)
- Bug in `DataFrame.to_csv` when the table had ``MultiIndex`` columns, and a list of strings was passed in for ``header`` (`5539`)
- Bug in parsing integer datetime-like columns with specified format in ``read_sql`` (`17855`).
- Bug in `DataFrame.to_msgpack` when serializing data of the ``numpy.bool_`` datatype (`18390`)
- Bug in `read_json` not decoding when reading line delimited JSON from S3 (`17200`)
- Bug in `pandas.io.json.json_normalize` to avoid modification of ``meta`` (`18610`)
- Bug in `to_latex` where repeated MultiIndex values were not printed even though a higher level index differed from the previous row (`14484`)
- Bug when reading NaN-only categorical columns in `HDFStore` (`18413`)
- Bug in `DataFrame.to_latex` with ``longtable=True`` where a latex multicolumn always spanned over three columns (`17959`)  Plotting
^^^  - Bug in ``DataFrame.plot()`` and ``Series.plot()`` with `DatetimeIndex` where a figure generated by them is not pickleable in Python 3 (`18439`)  GroupBy/resample/rolling
^^^  - Bug in ``DataFrame.resample(...).apply(...)`` when there is a callable that returns different columns (`15169`)
- Bug in ``DataFrame.resample(...)`` when there is a time change (DST) and resampling frequency is 12h or higher (`15549`)
- Bug in ``pd.DataFrameGroupBy.count()`` when counting over a datetimelike column (`13393`)
- Bug in ``rolling.var`` where calculation is inaccurate with a zero-valued array (`18430`)  Reshaping
^^^  - Error message in ``pd.merge_asof()`` for key datatype mismatch now includes datatype of left and right key (`18068`)
- Bug in ``pd.concat`` when empty and non-empty DataFrames or Series are concatenated (`18178` `18187`)
- Bug in ``DataFrame.filter(...)`` when `unicode` is passed as a condition in Python 2 (`13101`)
- Bug when merging empty DataFrames when ``np.seterr(divide='raise')`` is set (`17776`)  Numeric
^^^  - Bug in ``pd.Series.rolling.skew()`` and ``rolling.kurt()`` with all equal values has floating issue (`18044`)  Categorical
^^^  - Bug in `DataFrame.astype` where casting to 'category' on an empty ``DataFrame`` causes a segmentation fault (`18004`)
- Error messages in the testing module have been improved when items have different ``CategoricalDtype`` (`18069`)
- ``CategoricalIndex`` can now correctly take a ``pd.api.types.CategoricalDtype`` as its dtype (`18116`)
- Bug in ``Categorical.unique()`` returning read-only ``codes``  array when all categories were ``NaN`` (`18051`)
- Bug in ``DataFrame.groupby(axis=1)`` with a ``CategoricalIndex`` (`18432`)  String
^^^  - `Series.str.split()` will now propagate ``NaN`` values across all expanded columns instead of ``None`` (`18450`)  .. _whatsnew_0.21.1.contributors:  Contributors
~~~  .. contributors:: v0.21.0..v0.21.1

{{ header }}  .. ipython:: python  from pandas import *  # noqa F401, F403  This is a major release from 0.21.1 and includes a single, API-breaking change.
We recommend that all users upgrade to this version after carefully reading the
release note (singular!).  .. _whatsnew_0220.api_breaking:  Backwards incompatible API changes
~~~  pandas 0.22.0 changes the handling of empty and all-*NA* sums and products. The
summary is that  * The sum of an empty or all-*NA* ``Series`` is now ``0``
* The product of an empty or all-*NA* ``Series`` is now ``1``
* We've added a ``min_count`` parameter to ``.sum()`` and ``.prod()`` controlling  the minimum number of valid values for the result to be valid. If fewer than  ``min_count`` non-*NA* values are present, the result is *NA*. The default is  ``0``. To return ``NaN``, the 0.21 behavior, use ``min_count=1``.  Some background: In pandas 0.21, we fixed a long-standing inconsistency
in the return value of all-*NA* series depending on whether or not bottleneck
was installed. See `whatsnew_0210.api_breaking.bottleneck`. At the same
time, we changed the sum and prod of an empty ``Series`` to also be ``NaN``.  Based on feedback, we've partially reverted those changes.  Arithmetic operations
^^^  The default sum for empty or all-*NA* ``Series`` is now ``0``.  *pandas 0.21.x*  .. code-block:: ipython  In [1]: pd.Series([]).sum()  Out[1]: nan  In [2]: pd.Series([np.nan]).sum()  Out[2]: nan  *pandas 0.22.0*  .. ipython:: python  pd.Series([]).sum()  pd.Series([np.nan]).sum()  The default behavior is the same as pandas 0.20.3 with bottleneck installed. It
also matches the behavior of NumPy's ``np.nansum`` on empty and all-*NA* arrays.  To have the sum of an empty series return ``NaN`` (the default behavior of
pandas 0.20.3 without bottleneck, or pandas 0.21.x), use the ``min_count``
keyword.  .. ipython:: python  pd.Series([]).sum(min_count=1)  Thanks to the ``skipna`` parameter, the ``.sum`` on an all-*NA*
series is conceptually the same as the ``.sum`` of an empty one with
``skipna=True`` (the default).  .. ipython:: python  pd.Series([np.nan]).sum(min_count=1)  # skipna=True by default  The ``min_count`` parameter refers to the minimum number of *non-null* values
required for a non-NA sum or product.  `Series.prod` has been updated to behave the same as `Series.sum`,
returning ``1`` instead.  .. ipython:: python  pd.Series([]).prod()  pd.Series([np.nan]).prod()  pd.Series([]).prod(min_count=1)  These changes affect `DataFrame.sum` and `DataFrame.prod` as well.
Finally, a few less obvious places in pandas are affected by this change.  Grouping by a Categorical
^^^  Grouping by a ``Categorical`` and summing now returns ``0`` instead of
``NaN`` for categories with no observations. The product now returns ``1``
instead of ``NaN``.  *pandas 0.21.x*  .. code-block:: ipython  In [8]: grouper = pd.Categorical(['a', 'a'], categories=['a', 'b'])  In [9]: pd.Series([1, 2]).groupby(grouper).sum()  Out[9]:  a  3.0  b  NaN  dtype: float64  *pandas 0.22*  .. ipython:: python  grouper = pd.Categorical(["a", "a"], categories=["a", "b"])  pd.Series([1, 2]).groupby(grouper).sum()  To restore the 0.21 behavior of returning ``NaN`` for unobserved groups,
use ``min_count>=1``.  .. ipython:: python  pd.Series([1, 2]).groupby(grouper).sum(min_count=1)  Resample
^^^  The sum and product of all-*NA* bins has changed from ``NaN`` to ``0`` for

sum and ``1`` for product.  *pandas 0.21.x*  .. code-block:: ipython  In [11]: s = pd.Series([1, 1, np.nan, np.nan],  ...:  index=pd.date_range('2017', periods=4))  ...: s  Out[11]:  2017-01-01  1.0  2017-01-02  1.0  2017-01-03  NaN  2017-01-04  NaN  Freq: D, dtype: float64  In [12]: s.resample('2d').sum()  Out[12]:  2017-01-01  2.0  2017-01-03  NaN  Freq: 2D, dtype: float64  *pandas 0.22.0*  .. ipython:: python  s = pd.Series([1, 1, np.nan, np.nan], index=pd.date_range("2017", periods=4))  s.resample("2d").sum()  To restore the 0.21 behavior of returning ``NaN``, use ``min_count>=1``.  .. ipython:: python  s.resample("2d").sum(min_count=1)  In particular, upsampling and taking the sum or product is affected, as
upsampling introduces missing values even if the original series was
entirely valid.  *pandas 0.21.x*  .. code-block:: ipython  In [14]: idx = pd.DatetimeIndex(['2017-01-01', '2017-01-02'])  In [15]: pd.Series([1, 2], index=idx).resample('12H').sum()  Out[15]:  2017-01-01 000  1.0  2017-01-01 1200  NaN  2017-01-02 000  2.0  Freq: 12H, dtype: float64  *pandas 0.22.0*  .. ipython:: python  idx = pd.DatetimeIndex(["2017-01-01", "2017-01-02"])  pd.Series([1, 2], index=idx).resample("12H").sum()  Once again, the ``min_count`` keyword is available to restore the 0.21 behavior.  .. ipython:: python  pd.Series([1, 2], index=idx).resample("12H").sum(min_count=1)  Rolling and expanding
^^^  Rolling and expanding already have a ``min_periods`` keyword that behaves
similar to ``min_count``. The only case that changes is when doing a rolling
or expanding sum with ``min_periods=0``. Previously this returned ``NaN``,
when fewer than ``min_periods`` non-*NA* values were in the window. Now it
returns ``0``.  *pandas 0.21.1*  .. code-block:: ipython  In [17]: s = pd.Series([np.nan, np.nan])  In [18]: s.rolling(2, min_periods=0).sum()  Out[18]:  0  NaN  1  NaN  dtype: float64  *pandas 0.22.0*  .. ipython:: python  s = pd.Series([np.nan, np.nan])  s.rolling(2, min_periods=0).sum()  The default behavior of ``min_periods=None``, implying that ``min_periods``
equals the window size, is unchanged.  Compatibility
~~~  If you maintain a library that should work across pandas versions, it
may be easiest to exclude pandas 0.21 from your requirements. Otherwise, all your
``sum()`` calls would need to check if the ``Series`` is empty before summing.  With setuptools, in your ``setup.py`` use::  install_requires=['pandas!=0.21.*', ...]  With conda, use  .. code-block:: yaml  requirements:  run:  - pandas !=0.21.0,!=0.21.1  Note that the inconsistency in the return value for all-*NA* series is still
there for pandas 0.20.3 and earlier. Avoiding pandas 0.21 will only help with
the empty case.  .. _whatsnew_0.22.0.contributors:  Contributors
~~~  .. contributors:: v0.21.1..v0.22.0

{{ header }}  .. ipython:: python  from pandas import * # noqa F401, F403  This is a major release from 0.22.0 and includes a number of API changes,
deprecations, new features, enhancements, and performance improvements along
with a large number of bug fixes. We recommend that all users upgrade to this
version.  Highlights include:  - `Round-trippable JSON format with 'table' orient <whatsnew_0230.enhancements.round-trippable_json>`.
- `Instantiation from dicts respects order for Python 3.6+ <whatsnew_0230.api_breaking.dict_insertion_order>`.
- `Dependent column arguments for assign <whatsnew_0230.enhancements.assign_dependent>`.
- `Merging / sorting on a combination of columns and index levels <whatsnew_0230.enhancements.merge_on_columns_and_levels>`.
- `Extending pandas with custom types <whatsnew_023.enhancements.extension>`.
- `Excluding unobserved categories from groupby <whatsnew_0230.enhancements.categorical_grouping>`.
- `Changes to make output shape of DataFrame.apply consistent <whatsnew_0230.api_breaking.apply>`.  Check the `API Changes <whatsnew_0230.api_breaking>` and `deprecations <whatsnew_0230.deprecations>` before updating.  .. warning::  Starting January 1, 2019, pandas feature releases will support Python 3 only.  See Dropping Python 2.7 for more.  .. contents:: What's new in v0.23.0  none  2  .. _whatsnew_0230.enhancements:  New features
~~~  .. _whatsnew_0230.enhancements.round-trippable_json:  JSON read/write round-trippable with ``orient='table'``
^^^  A ``DataFrame`` can now be written to and subsequently read back via JSON while preserving metadata through usage of the ``orient='table'`` argument (see `18912` and `9146`). Previously, none of the available ``orient`` values guaranteed the preservation of dtypes and index names, amongst other metadata.  .. ipython:: python  df = pd.DataFrame({'foo': [1, 2, 3, 4],  'bar': ['a', 'b', 'c', 'd'],  'baz': pd.date_range('2018-01-01', freq='d', periods=4),  'qux': pd.Categorical(['a', 'b', 'c', 'c'])},  index=pd.Index(range(4), name='idx'))  df  df.dtypes  df.to_json('test.json', orient='table')  new_df = pd.read_json('test.json', orient='table')  new_df  new_df.dtypes  Please note that the string ``index`` is not supported with the round trip format, as it is used by default in ``write_json`` to indicate a missing index name.  .. ipython:: python  df.index.name = 'index'  df.to_json('test.json', orient='table')  new_df = pd.read_json('test.json', orient='table')  new_df  new_df.dtypes  .. ipython:: python  import os  os.remove('test.json')  .. _whatsnew_0230.enhancements.assign_dependent:  Method ``.assign()`` accepts dependent arguments
^^^  The `DataFrame.assign` now accepts dependent keyword arguments for python version later than 3.6 (see also `PEP 468
<https://www.python.org/dev/peps/pep-0468/>`_). Later keyword arguments may now refer to earlier ones if the argument is a callable. See the
`documentation here <dsintro.chained_assignment>` (`14207`)  .. ipython:: python  df = pd.DataFrame({'A': [1, 2, 3]})  df  df.assign(B=df.A, C=lambda x: x['A'] + x['B'])  .. warning::  This may subtly change the behavior of your code when you're  using ``.assign()`` to update an existing column. Previously, callables  referring to other variables being updated would get the "old" values  Previous behavior:  .. code-block:: ipython  In [2]: df = pd.DataFrame({"A": [1, 2, 3]})  In [3]: df.assign(A=lambda df: df.A + 1, C=lambda df: df.A * -1)  Out[3]:  A  C  0  2 -1  1  3 -2  2  4 -3  New behavior:  .. ipython:: python  df.assign(A=df.A + 1, C=lambda df: df.A * -1)  .. _whatsnew_0230.enhancements.merge_on_columns_and_levels:  Merging on a combination of columns and index levels
^^^  Strings passed to `DataFrame.merge` as the ``on``, ``left_on``, and ``right_on``
parameters may now refer to either column names or index level names.
This enables merging ``DataFrame`` instances on a combination of index levels
and columns without resetting indexes. See the `Merge on columns and

levels <merging.merge_on_columns_and_levels>` documentation section.
(`14355`)  .. ipython:: python  left_index = pd.Index(['K0', 'K0', 'K1', 'K2'], name='key1')  left = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],  'B': ['B0', 'B1', 'B2', 'B3'],  'key2': ['K0', 'K1', 'K0', 'K1']},  index=left_index)  right_index = pd.Index(['K0', 'K1', 'K2', 'K2'], name='key1')  right = pd.DataFrame({'C': ['C0', 'C1', 'C2', 'C3'],  'D': ['D0', 'D1', 'D2', 'D3'],  'key2': ['K0', 'K0', 'K0', 'K1']},  index=right_index)  left.merge(right, on=['key1', 'key2'])  .. _whatsnew_0230.enhancements.sort_by_columns_and_levels:  Sorting by a combination of columns and index levels
^^^  Strings passed to `DataFrame.sort_values` as the ``by`` parameter may
now refer to either column names or index level names.  This enables sorting
``DataFrame`` instances by a combination of index levels and columns without
resetting indexes. See the `Sorting by Indexes and Values
<basics.sort_indexes_and_values>` documentation section.
(`14353`)  .. ipython:: python  # Build MultiIndex  idx = pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('a', 2),  ('b', 2), ('b', 1), ('b', 1)])  idx.names = ['first', 'second']  # Build DataFrame  df_multi = pd.DataFrame({'A': np.arange(6, 0, -1)},  index=idx)  df_multi  # Sort by 'second' (index) and 'A' (column)  df_multi.sort_values(by=['second', 'A'])  .. _whatsnew_023.enhancements.extension:  Extending pandas with custom types (experimental)
^^^  pandas now supports storing array-like objects that aren't necessarily 1-D NumPy
arrays as columns in a DataFrame or values in a Series. This allows third-party
libraries to implement extensions to NumPy's types, similar to how pandas
implemented categoricals, datetimes with timezones, periods, and intervals.  As a demonstration, we'll use cyberpandas_, which provides an ``IPArray`` type
for storing ip addresses.  .. code-block:: ipython  In [1]: from cyberpandas import IPArray  In [2]: values = IPArray([  ...:  0,  ...:  3232235777,  ...:  42540766452641154071740215577757643572  ...: ])  ...:  ...:  ``IPArray`` isn't a normal 1-D NumPy array, but because it's a pandas
`~pandas.api.extensions.ExtensionArray`, it can be stored properly inside pandas' containers.  .. code-block:: ipython  In [3]: ser = pd.Series(values)  In [4]: ser  Out[4]:  0  0.0.0.0  1  192.168.1.1  2  200185a3:370:7334  dtype: ip  Notice that the dtype is ``ip``. The missing value semantics of the underlying
array are respected:  .. code-block:: ipython  In [5]: ser.isna()  Out[5]:  0  True  1  False  2  False  dtype: bool  For more, see the `extension types <extending.extension-types>`
documentation. If you build an extension array, publicize it on our
`ecosystem page <ecosystem.extensions>`.  .. _cyberpandas: https://cyberpandas.readthedocs.io/en/latest/  .. _whatsnew_0230.enhancements.categorical_grouping:  New ``observed`` keyword for excluding unobserved categories in ``GroupBy``
^^^  Grouping by a categorical includes the unobserved categories in the output.
When grouping by multiple categorical columns, this means you get the cartesian product of all the
categories, including combinations where there are no observations, which can result in a large
number of groups. We have added a keyword ``observed`` to control this behavior, it defaults to

``observed=False`` for backward-compatibility. (`14942`, `8138`, `15217`, `17594`, `8669`, `20583`, `20902`)  .. ipython:: python  cat1 = pd.Categorical(["a", "a", "b", "b"],  categories=["a", "b", "z"], ordered=True)  cat2 = pd.Categorical(["c", "d", "c", "d"],  categories=["c", "d", "y"], ordered=True)  df = pd.DataFrame({"A": cat1, "B": cat2, "values": [1, 2, 3, 4]})  df['C'] = ['foo', 'bar'] * 2  df  To show all values, the previous behavior:  .. ipython:: python  df.groupby(['A', 'B', 'C'], observed=False).count()  To show only observed values:  .. ipython:: python  df.groupby(['A', 'B', 'C'], observed=True).count()  For pivoting operations, this behavior is *already* controlled by the ``dropna`` keyword:  .. ipython:: python  cat1 = pd.Categorical(["a", "a", "b", "b"],  categories=["a", "b", "z"], ordered=True)  cat2 = pd.Categorical(["c", "d", "c", "d"],  categories=["c", "d", "y"], ordered=True)  df = pd.DataFrame({"A": cat1, "B": cat2, "values": [1, 2, 3, 4]})  df  .. ipython:: python  pd.pivot_table(df, values='values', index=['A', 'B'],  dropna=True)  pd.pivot_table(df, values='values', index=['A', 'B'],  dropna=False)  .. _whatsnew_0230.enhancements.window_raw:  Rolling/Expanding.apply() accepts ``raw=False`` to pass a ``Series`` to the function
^^^  `Series.rolling().apply() <pandas.core.window.Rolling.apply>`, `DataFrame.rolling().apply() <pandas.core.window.Rolling.apply>`,
`Series.expanding().apply() <pandas.core.window.Expanding.apply>`, and `DataFrame.expanding().apply() <pandas.core.window.Expanding.apply>` have gained a ``raw=None`` parameter.
This is similar to `DataFame.apply`. This parameter, if ``True`` allows one to send a ``np.ndarray`` to the applied function. If ``False`` a ``Series`` will be passed. The
default is ``None``, which preserves backward compatibility, so this will default to ``True``, sending an ``np.ndarray``.
In a future version the default will be changed to ``False``, sending a ``Series``. (`5071`, `20584`)  .. ipython:: python  s = pd.Series(np.arange(5), np.arange(5) + 1)  s  Pass a ``Series``:  .. ipython:: python  s.rolling(2, min_periods=1).apply(lambda x: x.iloc[-1], raw=False)  Mimic the original behavior of passing a ndarray:  .. ipython:: python  s.rolling(2, min_periods=1).apply(lambda x: x[-1], raw=True)  .. _whatsnew_0210.enhancements.limit_area:  ``DataFrame.interpolate`` has gained the ``limit_area`` kwarg
^^^  `DataFrame.interpolate` has gained a ``limit_area`` parameter to allow further control of which ``NaN`` s are replaced.
Use ``limit_area='inside'`` to fill only NaNs surrounded by valid values or use ``limit_area='outside'`` to fill only ``NaN`` s
outside the existing valid values while preserving those inside.  (`16284`) See the `full documentation here <missing_data.interp_limits>`.  .. ipython:: python  ser = pd.Series([np.nan, np.nan, 5, np.nan, np.nan,  np.nan, 13, np.nan, np.nan])  ser  Fill one consecutive inside value in both directions  .. ipython:: python  ser.interpolate(limit_direction='both', limit_area='inside', limit=1)  Fill all consecutive outside values backward  .. ipython:: python  ser.interpolate(limit_direction='backward', limit_area='outside')  Fill all consecutive outside values in both directions  .. ipython:: python  ser.interpolate(limit_direction='both', limit_area='outside')  .. _whatsnew_0210.enhancements.get_dummies_dtype:  Function ``get_dummies`` now supports ``dtype`` argument

^^^  The `get_dummies` now accepts a ``dtype`` argument, which specifies a dtype for the new columns. The default remains uint8. (`18330`)  .. ipython:: python  df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'c': [5, 6]})  pd.get_dummies(df, columns=['c']).dtypes  pd.get_dummies(df, columns=['c'], dtype=bool).dtypes  .. _whatsnew_0230.enhancements.timedelta_mod:  Timedelta mod method
^^^  ``mod`` (%) and ``divmod`` operations are now defined on ``Timedelta`` objects
when operating with either timedelta-like or with numeric arguments.
See the `documentation here <timedeltas.mod_divmod>`. (`19365`)  .. ipython:: python  td = pd.Timedelta(hours=37)  td % pd.Timedelta(minutes=45)  .. _whatsnew_0230.enhancements.ran_inf:  Method ``.rank()`` handles ``inf`` values when ``NaN`` are present
^^^  In previous versions, ``.rank()`` would assign ``inf`` elements ``NaN`` as their ranks. Now ranks are calculated properly. (`6945`)  .. ipython:: python  s = pd.Series([-np.inf, 0, 1, np.nan, np.inf])  s  Previous behavior:  .. code-block:: ipython  In [11]: s.rank()  Out[11]:  0  1.0  1  2.0  2  3.0  3  NaN  4  NaN  dtype: float64  Current behavior:  .. ipython:: python  s.rank()  Furthermore, previously if you rank ``inf`` or ``-inf`` values together with ``NaN`` values, the calculation won't distinguish ``NaN`` from infinity when using 'top' or 'bottom' argument.  .. ipython:: python  s = pd.Series([np.nan, np.nan, -np.inf, -np.inf])  s  Previous behavior:  .. code-block:: ipython  In [15]: s.rank(na_option='top')  Out[15]:  0  2.5  1  2.5  2  2.5  3  2.5  dtype: float64  Current behavior:  .. ipython:: python  s.rank(na_option='top')  These bugs were squashed:  - Bug in `DataFrame.rank` and `Series.rank` when ``method='dense'`` and ``pct=True`` in which percentile ranks were not being used with the number of distinct observations (`15630`)
- Bug in `Series.rank` and `DataFrame.rank` when ``ascending='False'`` failed to return correct ranks for infinity if ``NaN`` were present (`19538`)
- Bug in `DataFrameGroupBy.rank` where ranks were incorrect when both infinity and ``NaN`` were present (`20561`)  .. _whatsnew_0230.enhancements.str_cat_align:  ``Series.str.cat`` has gained the ``join`` kwarg
^^^  Previously, `Series.str.cat` did not -- in contrast to most of ``pandas`` -- align `Series` on their index before concatenation (see `18657`).
The method has now gained a keyword ``join`` to control the manner of alignment, see examples below and `here <text.concatenate>`.  In v.0.23 ``join`` will default to None (meaning no alignment), but this default will change to ``'left'`` in a future version of pandas.  .. ipython:: python  s = pd.Series(['a', 'b', 'c', 'd'])  t = pd.Series(['b', 'd', 'e', 'c'], index=[1, 3, 4, 2])  s.str.cat(t)  s.str.cat(t, join='left', na_rep='-')  Furthermore, `Series.str.cat` now works for ``CategoricalIndex`` as well (previously raised a ``ValueError``; see `20842`).  .. _whatsnew_0230.enhancements.astype_category:  ``DataFrame.astype`` performs column-wise conversion to ``Categorical``
^^^  `DataFrame.astype` can now perform column-wise conversion to ``Categorical`` by supplying the string ``'category'`` or
a `~pandas.api.types.CategoricalDtype`. Previously, attempting this would raise a ``NotImplementedError``. See the

`categorical.objectcreation` section of the documentation for more details and examples. (`12860`, `18099`)  Supplying the string ``'category'`` performs column-wise conversion, with only labels appearing in a given column set as categories:  .. ipython:: python  df = pd.DataFrame({'A': list('abca'), 'B': list('bccd')})  df = df.astype('category')  df['A'].dtype  df['B'].dtype  Supplying a ``CategoricalDtype`` will make the categories in each column consistent with the supplied dtype:  .. ipython:: python  from pandas.api.types import CategoricalDtype  df = pd.DataFrame({'A': list('abca'), 'B': list('bccd')})  cdt = CategoricalDtype(categories=list('abcd'), ordered=True)  df = df.astype(cdt)  df['A'].dtype  df['B'].dtype  .. _whatsnew_0230.enhancements.other:  Other enhancements
^^^  - Unary ``+`` now permitted for ``Series`` and ``DataFrame`` as  numeric operator (`16073`)
- Better support for `~pandas.io.formats.style.Styler.to_excel` output with the ``xlsxwriter`` engine. (`16149`)
- `pandas.tseries.frequencies.to_offset` now accepts leading '+' signs e.g. '+1h'. (`18171`)
- `MultiIndex.unique` now supports the ``level=`` argument, to get unique values from a specific index level (`17896`)
- `pandas.io.formats.style.Styler` now has method ``hide_index()`` to determine whether the index will be rendered in output (`14194`)
- `pandas.io.formats.style.Styler` now has method ``hide_columns()`` to determine whether columns will be hidden in output (`14194`)
- Improved wording of ``ValueError`` raised in `to_datetime` when ``unit=`` is passed with a non-convertible value (`14350`)
- `Series.fillna` now accepts a Series or a dict as a ``value`` for a categorical dtype (`17033`)
- `pandas.read_clipboard` updated to use qtpy, falling back to PyQt5 and then PyQt4, adding compatibility with Python3 and multiple python-qt bindings (`17722`)
- Improved wording of ``ValueError`` raised in `read_csv` when the ``usecols`` argument cannot match all columns. (`17301`)
- `DataFrame.corrwith` now silently drops non-numeric columns when passed a Series. Before, an exception was raised (`18570`).
- `IntervalIndex` now supports time zone aware ``Interval`` objects (`18537`, `18538`)
- `Series` / `DataFrame` tab completion also returns identifiers in the first level of a `MultiIndex`. (`16326`)
- `read_excel()` has gained the ``nrows`` parameter (`16645`)
- `DataFrame.append` can now in more cases preserve the type of the calling dataframe's columns (e.g. if both are ``CategoricalIndex``) (`18359`)
- `DataFrame.to_json` and `Series.to_json` now accept an ``index`` argument which allows the user to exclude the index from the JSON output (`17394`)
- ``IntervalIndex.to_tuples()`` has gained the ``na_tuple`` parameter to control whether NA is returned as a tuple of NA, or NA itself (`18756`)
- ``Categorical.rename_categories``, ``CategoricalIndex.rename_categories`` and `Series.cat.rename_categories`  can now take a callable as their argument (`18862`)
- `Interval` and `IntervalIndex` have gained a ``length`` attribute (`18789`)
- ``Resampler`` objects now have a functioning `~pandas.core.resample.Resampler.pipe` method.  Previously, calls to ``pipe`` were diverted to  the ``mean`` method (`17905`).
- `~pandas.api.types.is_scalar` now returns ``True`` for ``DateOffset`` objects (`18943`).
- `DataFrame.pivot` now accepts a list for the ``values=`` kwarg (`17160`).
- Added `pandas.api.extensions.register_dataframe_accessor`,  `pandas.api.extensions.register_series_accessor`, and  `pandas.api.extensions.register_index_accessor`, accessor for libraries downstream of pandas  to register custom accessors like ``.cat`` on pandas objects. See  `Registering Custom Accessors <extending.register-accessors>` for more (`14781`).  - ``IntervalIndex.astype`` now supports conversions between subtypes when passed an ``IntervalDtype`` (`19197`)

- `IntervalIndex` and its associated constructor methods (``from_arrays``, ``from_breaks``, ``from_tuples``) have gained a ``dtype`` parameter (`19262`)
- Added `pandas.core.groupby.SeriesGroupBy.is_monotonic_increasing` and `pandas.core.groupby.SeriesGroupBy.is_monotonic_decreasing` (`17015`)
- For subclassed ``DataFrames``, `DataFrame.apply` will now preserve the ``Series`` subclass (if defined) when passing the data to the applied function (`19822`)
- `DataFrame.from_dict` now accepts a ``columns`` argument that can be used to specify the column names when ``orient='index'`` is used (`18529`)
- Added option `display.html.use_mathjax`` so `MathJax can be disabled when rendering tables in ``Jupyter`` notebooks (`19856`, `19824`)
- `DataFrame.replace` now supports the ``method`` parameter, which can be used to specify the replacement method when ``to_replace`` is a scalar, list or tuple and ``value`` is ``None`` (`19632`)
- `Timestamp.month_name`, `DatetimeIndex.month_name`, and `Series.dt.month_name` are now available (`12805`)
- `Timestamp.day_name` and `DatetimeIndex.day_name` are now available to return day names with a specified locale (`12806`)
- `DataFrame.to_sql` now performs a multi-value insert if the underlying connection supports itk rather than inserting row by row.  ``SQLAlchemy`` dialects supporting multi-value inserts include: ``mysql``, ``postgresql``, ``sqlite`` and any dialect with ``supports_multivalues_insert``. (`14315`, `8953`)
- `read_html` now accepts a ``displayed_only`` keyword argument to controls whether or not hidden elements are parsed (``True`` by default) (`20027`)
- `read_html` now reads all ``<tbody>`` elements in a ``<table>``, not just the first. (`20690`)
- `~pandas.core.window.Rolling.quantile` and `~pandas.core.window.Expanding.quantile` now accept the ``interpolation`` keyword, ``linear`` by default (`20497`)
- zip compression is supported via ``compression=zip`` in `DataFrame.to_pickle`, `Series.to_pickle`, `DataFrame.to_csv`, `Series.to_csv`, `DataFrame.to_json`, `Series.to_json`. (`17778`)
- `~pandas.tseries.offsets.WeekOfMonth` constructor now supports ``n=0`` (`20517`).
- `DataFrame` and `Series` now support matrix multiplication (``@``) operator (`10259`) for Python>=3.5
- Updated `DataFrame.to_gbq` and `pandas.read_gbq` signature and documentation to reflect changes from  the pandas-gbq library version 0.4.0. Adds intersphinx mapping to pandas-gbq  library. (`20564`)
- Added new writer for exporting Stata dta files in version 117, ``StataWriter117``.  This format supports exporting strings with lengths up to 2,000,000 characters (`16450`)
- `to_hdf` and `read_hdf` now accept an ``errors`` keyword argument to control encoding error handling (`20835`)
- `cut` has gained the ``duplicates='raise'|'drop'`` option to control whether to raise on duplicated edges (`20947`)
- `date_range`, `timedelta_range`, and `interval_range` now return a linearly spaced index if ``start``, ``stop``, and ``periods`` are specified, but ``freq`` is not. (`20808`, `20983`, `20976`)  .. _whatsnew_0230.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_0230.api_breaking.deps:  Dependencies have increased minimum versions
^^^  We have updated our minimum supported versions of dependencies (`15184`).
If installed, we now require:  +---+---+---+---+
| Package  | Minimum Version | Required |  Issue  |
| python-dateutil | 2.5.0  |  X  | `15184`|
| openpyxl  | 2.4.0  |  | `15184`|
| beautifulsoup4  | 4.2.1  |  | `20082`|
| setuptools  | 24.2.0  |  | `20698`|
+---+---+---+---+  .. _whatsnew_0230.api_breaking.dict_insertion_order:  Instantiation from dicts preserves dict insertion order for Python 3.6+
^^^  Until Python 3.6, dicts in Python had no formally defined ordering. For Python
version 3.6 and later, dicts are ordered by insertion order, see
PEP 468.
pandas will use the dict's insertion order, when creating a ``Series`` or

PEP 468.
``DataFrame`` from a dict and you're using Python version 3.6 or
higher. (`19884`)  Previous behavior (and current behavior if on Python < 3.6):  .. code-block:: ipython  In [16]: pd.Series({'Income': 2000,  ...:  'Expenses': -1500,  ...:  'Taxes': -200,  ...:  'Net result': 300})  Out[16]:  Expenses  -1500  Income  2000  Net result  300  Taxes  -200  dtype: int64  Note the Series above is ordered alphabetically by the index values.  New behavior (for Python >= 3.6):  .. ipython:: python  pd.Series({'Income': 2000,  'Expenses': -1500,  'Taxes': -200,  'Net result': 300})  Notice that the Series is now ordered by insertion order. This new behavior is
used for all relevant pandas types (``Series``, ``DataFrame``, ``SparseSeries``
and ``SparseDataFrame``).  If you wish to retain the old behavior while using Python >= 3.6, you can use
``.sort_index()``:  .. ipython:: python  pd.Series({'Income': 2000,  'Expenses': -1500,  'Taxes': -200,  'Net result': 300}).sort_index()  .. _whatsnew_0230.api_breaking.deprecate_panel:  Deprecate Panel
^^^  ``Panel`` was deprecated in the 0.20.x release, showing as a ``DeprecationWarning``. Using ``Panel`` will now show a ``FutureWarning``. The recommended way to represent 3-D data are
provides a `~Panel.to_xarray` method to automate this conversion (`13563`, `18324`).  .. code-block:: ipython  In [75]: import pandas._testing as tm  In [76]: p = tm.makePanel()  In [77]: p  Out[77]:  <class 'pandas.core.panel.Panel'>  Dimensions: 3 (items) x 3 (major_axis) x 4 (minor_axis)  Items axis: ItemA to ItemC  Major_axis axis: 2000-01-03 000 to 2000-01-05 000  Minor_axis axis: A to D  Convert to a MultiIndex DataFrame  .. code-block:: ipython  In [78]: p.to_frame()  Out[78]:  ItemA  ItemB  ItemC  major  minor  2000-01-03 A  0.469112  0.721555  0.404705  B  -1.135632  0.271860 -1.039268  C  0.119209  0.276232 -1.344312  D  -2.104569  0.113648 -0.109050  2000-01-04 A  -0.282863 -0.706771  0.577046  B  1.212112 -0.424972 -0.370647  C  -1.044236 -1.087401  0.844885  D  -0.494929 -1.478427  1.643563  2000-01-05 A  -1.509059 -1.039575 -1.715002  B  -0.173215  0.567020 -1.157892  C  -0.861849 -0.673690  1.075770  D  1.071804  0.524988 -1.469388  [12 rows x 3 columns]  Convert to an xarray DataArray  .. code-block:: ipython  In [79]: p.to_xarray()  Out[79]:  <xarray.DataArray (items: 3, major_axis: 3, minor_axis: 4)>  array([[[ 0.469112, -1.135632,  0.119209, -2.104569],  [-0.282863,  1.212112, -1.044236, -0.494929],  [-1.509059, -0.173215, -0.861849,  1.071804]],  [[ 0.721555,  0.27186 ,  0.276232,  0.113648],  [-0.706771, -0.424972, -1.087401, -1.478427],  [-1.039575,  0.56702 , -0.67369 ,  0.524988]],  [[ 0.404705, -1.039268, -1.344312, -0.10905 ],  [ 0.577046, -0.370647,  0.844885,  1.643563],  [-1.715002, -1.157892,  1.07577 , -1.469388]]])  Coordinates:  * items  (items) object 'ItemA' 'ItemB' 'ItemC'  * major_axis  (major_axis) datetime64[ns] 2000-01-03 2000-01-04 2000-01-05  * minor_axis  (minor_axis) object 'A' 'B' 'C' 'D'  .. _whatsnew_0230.api_breaking.core_common:  pandas.core.common removals
^^^  The following error & warning messages are removed from ``pandas.core.common`` (`13634`, `19769`):  - ``PerformanceWarning``
- ``UnsupportedFunctionCall``
- ``UnsortedIndexError``
- ``AbstractMethodError``  These are available from import from ``pandas.errors`` (since 0.19.0).  .. _whatsnew_0230.api_breaking.apply:  Changes to make output of ``DataFrame.apply`` consistent
^^^  `DataFrame.apply` was inconsistent when applying an arbitrary user-defined-function that returned a list-like with ``axis=1``. Several bugs and inconsistencies

are resolved. If the applied function returns a Series, then pandas will return a DataFrame; otherwise a Series will be returned, this includes the case
where a list-like (e.g. ``tuple`` or ``list`` is returned) (`16353`, `17437`, `17970`, `17348`, `17892`, `18573`,
`17602`, `18775`, `18901`, `18919`).  .. ipython:: python  df = pd.DataFrame(np.tile(np.arange(3), 6).reshape(6, -1) + 1,  columns=['A', 'B', 'C'])  df  Previous behavior: if the returned shape happened to match the length of original columns, this would return a ``DataFrame``.
If the return shape did not match, a ``Series`` with lists was returned.  .. code-block:: python  In [3]: df.apply(lambda x: [1, 2, 3], axis=1)  Out[3]:  A  B  C  0  1  2  3  1  1  2  3  2  1  2  3  3  1  2  3  4  1  2  3  5  1  2  3  In [4]: df.apply(lambda x: [1, 2], axis=1)  Out[4]:  0  [1, 2]  1  [1, 2]  2  [1, 2]  3  [1, 2]  4  [1, 2]  5  [1, 2]  dtype: object  New behavior: When the applied function returns a list-like, this will now *always* return a ``Series``.  .. ipython:: python  df.apply(lambda x: [1, 2, 3], axis=1)  df.apply(lambda x: [1, 2], axis=1)  To have expanded columns, you can use ``result_type='expand'``  .. ipython:: python  df.apply(lambda x: [1, 2, 3], axis=1, result_type='expand')  To broadcast the result across the original columns (the old behaviour for
list-likes of the correct length), you can use ``result_type='broadcast'``.
The shape must match the original columns.  .. ipython:: python  df.apply(lambda x: [1, 2, 3], axis=1, result_type='broadcast')  Returning a ``Series`` allows one to control the exact return structure and column names:  .. ipython:: python  df.apply(lambda x: pd.Series([1, 2, 3], index=['D', 'E', 'F']), axis=1)  .. _whatsnew_0230.api_breaking.concat:  Concatenation will no longer sort
^^^  In a future version of pandas `pandas.concat` will no longer sort the non-concatenation axis when it is not already aligned.
The current behavior is the same as the previous (sorting), but now a warning is issued when ``sort`` is not specified and the non-concatenation axis is not aligned (`4588`).  .. ipython:: python  df1 = pd.DataFrame({"a": [1, 2], "b": [1, 2]}, columns=['b', 'a'])  df2 = pd.DataFrame({"a": [4, 5]})  pd.concat([df1, df2])  To keep the previous behavior (sorting) and silence the warning, pass ``sort=True``  .. ipython:: python  pd.concat([df1, df2], sort=True)  To accept the future behavior (no sorting), pass ``sort=False``  .. ipython  pd.concat([df1, df2], sort=False)  Note that this change also applies to `DataFrame.append`, which has also received a ``sort`` keyword for controlling this behavior.  .. _whatsnew_0230.api_breaking.build_changes:  Build changes
^^^  - Building pandas for development now requires ``cython >= 0.24`` (`18613`)
- Building from source now explicitly requires ``setuptools`` in ``setup.py`` (`18113`)
- Updated conda recipe to be in compliance with conda-build 3.0+ (`18002`)  .. _whatsnew_0230.api_breaking.index_division_by_zero:  Index division by zero fills correctly

^^^  Division operations on ``Index`` and subclasses will now fill division of positive numbers by zero with ``np.inf``, division of negative numbers by zero with ``-np.inf`` and ``0 / 0`` with ``np.nan``.  This matches existing ``Series`` behavior. (`19322`, `19347`)  Previous behavior:  .. code-block:: ipython  In [6]: index = pd.Int64Index([-1, 0, 1])  In [7]: index / 0  Out[7]: Int64Index([0, 0, 0], dtype='int64')  # Previous behavior yielded different results depending on the type of zero in the divisor  In [8]: index / 0.0  Out[8]: Float64Index([-inf, nan, inf], dtype='float64')  In [9]: index = pd.UInt64Index([0, 1])  In [10]: index / np.array([0, 0], dtype=np.uint64)  Out[10]: UInt64Index([0, 0], dtype='uint64')  In [11]: pd.RangeIndex(1, 5) / 0  ZeroDivisionError: integer division or modulo by zero  Current behavior:  .. code-block:: ipython  In [12]: index = pd.Int64Index([-1, 0, 1])  # division by zero gives -infinity where negative,  # +infinity where positive, and NaN for 0 / 0  In [13]: index / 0  # The result of division by zero should not depend on  # whether the zero is int or float  In [14]: index / 0.0  In [15]: index = pd.UInt64Index([0, 1])  In [16]: index / np.array([0, 0], dtype=np.uint64)  In [17]: pd.RangeIndex(1, 5) / 0  .. _whatsnew_0230.api_breaking.extract:  Extraction of matching patterns from strings
^^^  By default, extracting matching patterns from strings with `str.extract` used to return a
``Series`` if a single group was being extracted (a ``DataFrame`` if more than one group was
extracted). As of pandas 0.23.0 `str.extract` always returns a ``DataFrame``, unless
``expand`` is set to ``False``. Finally, ``None`` was an accepted value for
the ``expand`` parameter (which was equivalent to ``False``), but now raises a ``ValueError``. (`11386`)  Previous behavior:  .. code-block:: ipython  In [1]: s = pd.Series(['number 10', '12 eggs'])  In [2]: extracted = s.str.extract(r'.*(\d\d).*')  In [3]: extracted  Out [3]:  0  10  1  12  dtype: object  In [4]: type(extracted)  Out [4]:  pandas.core.series.Series  New behavior:  .. ipython:: python  s = pd.Series(['number 10', '12 eggs'])  extracted = s.str.extract(r'.*(\d\d).*')  extracted  type(extracted)  To restore previous behavior, simply set ``expand`` to ``False``:  .. ipython:: python  s = pd.Series(['number 10', '12 eggs'])  extracted = s.str.extract(r'.*(\d\d).*', expand=False)  extracted  type(extracted)  .. _whatsnew_0230.api_breaking.cdt_ordered:  Default value for the ``ordered`` parameter of ``CategoricalDtype``

^^^  The default value of the ``ordered`` parameter for `~pandas.api.types.CategoricalDtype` has changed from ``False`` to ``None`` to allow updating of ``categories`` without impacting ``ordered``.  Behavior should remain consistent for downstream objects, such as `Categorical` (`18790`)  In previous versions, the default value for the ``ordered`` parameter was ``False``.  This could potentially lead to the ``ordered`` parameter unintentionally being changed from ``True`` to ``False`` when users attempt to update ``categories`` if ``ordered`` is not explicitly specified, as it would silently default to ``False``.  The new behavior for ``ordered=None`` is to retain the existing value of ``ordered``.  New behavior:  .. code-block:: ipython  In [2]: from pandas.api.types import CategoricalDtype  In [3]: cat = pd.Categorical(list('abcaba'), ordered=True, categories=list('cba'))  In [4]: cat  Out[4]:  [a, b, c, a, b, a]  Categories (3, object): [c < b < a]  In [5]: cdt = CategoricalDtype(categories=list('cbad'))  In [6]: cat.astype(cdt)  Out[6]:  [a, b, c, a, b, a]  Categories (4, object): [c < b < a < d]  Notice in the example above that the converted ``Categorical`` has retained ``ordered=True``.  Had the default value for ``ordered`` remained as ``False``, the converted ``Categorical`` would have become unordered, despite ``ordered=False`` never being explicitly specified.  To change the value of ``ordered``, explicitly pass it to the new dtype, e.g. ``CategoricalDtype(categories=list('cbad'), ordered=False)``.  Note that the unintentional conversion of ``ordered`` discussed above did not arise in previous versions due to separate bugs that prevented ``astype`` from doing any type of category to category conversion (`10696`, `18593`).  These bugs have been fixed in this release, and motivated changing the default value of ``ordered``.  .. _whatsnew_0230.api_breaking.pretty_printing:  Better pretty-printing of DataFrames in a terminal
^^^
Previously, the default value for the maximum number of columns was
``pd.options.display.max_columns=20``. This meant that relatively wide data
frames would not fit within the terminal width, and pandas would introduce line
breaks to display these 20 columns. This resulted in an output that was
relatively difficult to read:  If Python runs in a terminal, the maximum number of columns is now determined
automatically so that the printed data frame fits within the current terminal
width (``pd.options.display.max_columns=0``) (`17023`). If Python runs
as a Jupyter kernel (such as the Jupyter QtConsole or a Jupyter notebook, as
well as in many IDEs), this value cannot be inferred automatically and is thus
set to ``20`` as in previous versions. In a terminal, this results in a much
nicer output:  Note that if you don't like the new default, you can always set this option
yourself. To revert to the old setting, you can run this line:  .. code-block:: python  pd.options.display.max_columns = 20  .. _whatsnew_0230.api.datetimelike:  Datetimelike API changes
^^^  - The default ``Timedelta`` constructor now accepts an ``ISO 8601 Duration`` string as an argument (`19040`)
- Subtracting ``NaT`` from a `Series` with ``dtype='datetime64[ns]'`` returns a ``Series`` with ``dtype='timedelta64[ns]'`` instead of ``dtype='datetime64[ns]'`` (`18808`)
- Addition or subtraction of ``NaT`` from `TimedeltaIndex` will return ``TimedeltaIndex`` instead of ``DatetimeIndex`` (`19124`)
- `DatetimeIndex.shift` and `TimedeltaIndex.shift` will now raise ``NullFrequencyError`` (which subclasses ``ValueError``, which was raised in older versions) when the index object frequency is ``None`` (`19147`)
- Addition and subtraction of ``NaN`` from a `Series` with ``dtype='timedelta64[ns]'`` will raise a ``TypeError`` instead of treating the ``NaN`` as ``NaT`` (`19274`)
- ``NaT`` division with `datetime.timedelta` will now return ``NaN`` instead of raising (`17876`)

- Operations between a `Series` with dtype ``dtype='datetime64[ns]'`` and a `PeriodIndex` will correctly raises ``TypeError`` (`18850`)
- Subtraction of `Series` with timezone-aware ``dtype='datetime64[ns]'`` with mismatched timezones will raise ``TypeError`` instead of ``ValueError`` (`18817`)
- `Timestamp` will no longer silently ignore unused or invalid ``tz`` or ``tzinfo`` keyword arguments (`17690`)
- `Timestamp` will no longer silently ignore invalid ``freq`` arguments (`5168`)
- `CacheableOffset` and `WeekDay` are no longer available in the ``pandas.tseries.offsets`` module (`17830`)
- ``pandas.tseries.frequencies.get_freq_group()`` and ``pandas.tseries.frequencies.DAYS`` are removed from the public API (`18034`)
- `Series.truncate` and `DataFrame.truncate` will raise a ``ValueError`` if the index is not sorted instead of an unhelpful ``KeyError`` (`17935`)
- `Series.first` and `DataFrame.first` will now raise a ``TypeError``  rather than ``NotImplementedError`` when index is not a `DatetimeIndex` (`20725`).
- `Series.last` and `DataFrame.last` will now raise a ``TypeError``  rather than ``NotImplementedError`` when index is not a `DatetimeIndex` (`20725`).
- Restricted ``DateOffset`` keyword arguments. Previously, ``DateOffset`` subclasses allowed arbitrary keyword arguments which could lead to unexpected behavior. Now, only valid arguments will be accepted. (`17176`, `18226`).
- `pandas.merge` provides a more informative error message when trying to merge on timezone-aware and timezone-naive columns (`15800`)
- For `DatetimeIndex` and `TimedeltaIndex` with ``freq=None``, addition or subtraction of integer-dtyped array or ``Index`` will raise ``NullFrequencyError`` instead of ``TypeError`` (`19895`)
- `Timestamp` constructor now accepts a ``nanosecond`` keyword or positional argument (`18898`)
- `DatetimeIndex` will now raise an ``AttributeError`` when the ``tz`` attribute is set after instantiation (`3746`)
- `DatetimeIndex` with a ``pytz`` timezone will now return a consistent ``pytz`` timezone (`18595`)  .. _whatsnew_0230.api.other:  Other API changes
^^^  - `Series.astype` and `Index.astype` with an incompatible dtype will now raise a ``TypeError`` rather than a ``ValueError`` (`18231`)
- ``Series`` construction with an ``object`` dtyped tz-aware datetime and ``dtype=object`` specified, will now return an ``object`` dtyped ``Series``, previously this would infer the datetime dtype (`18231`)
- A `Series` of ``dtype=category`` constructed from an empty ``dict`` will now have categories of ``dtype=object`` rather than ``dtype=float64``, consistently with the case in which an empty list is passed (`18515`)
- All-NaN levels in a ``MultiIndex`` are now assigned ``float`` rather than ``object`` dtype, promoting consistency with ``Index`` (`17929`).
- Levels names of a ``MultiIndex`` (when not None) are now required to be unique: trying to create a ``MultiIndex`` with repeated names will raise a ``ValueError`` (`18872`)
- Both construction and renaming of ``Index``/``MultiIndex`` with non-hashable ``name``/``names`` will now raise ``TypeError`` (`20527`)
- `Index.map` can now accept ``Series`` and dictionary input objects (`12756`, `18482`, `18509`).
- `DataFrame.unstack` will now default to filling with ``np.nan`` for ``object`` columns. (`12815`)
- `IntervalIndex` constructor will raise if the ``closed`` parameter conflicts with how the input data is inferred to be closed (`18421`)
- Inserting missing values into indexes will work for all types of indexes and automatically insert the correct type of missing value (``NaN``, ``NaT``, etc.) regardless of the type passed in (`18295`)
- When created with duplicate labels, ``MultiIndex`` now raises a ``ValueError``. (`17464`)
- `Series.fillna` now raises a ``TypeError`` instead of a ``ValueError`` when passed a list, tuple or DataFrame as a ``value`` (`18293`)

- `pandas.DataFrame.merge` no longer casts a ``float`` column to ``object`` when merging on ``int`` and ``float`` columns (`16572`)
- `pandas.merge` now raises a ``ValueError`` when trying to merge on incompatible data types (`9780`)
- The default NA value for `UInt64Index` has changed from 0 to ``NaN``, which impacts methods that mask with NA, such as ``UInt64Index.where()`` (`18398`)
- Refactored ``setup.py`` to use ``find_packages`` instead of explicitly listing out all subpackages (`18535`)
- Rearranged the order of keyword arguments in `read_excel()` to align with `read_csv()` (`16672`)
- `wide_to_long` previously kept numeric-like suffixes as ``object`` dtype. Now they are cast to numeric if possible (`17627`)
- In `read_excel`, the ``comment`` argument is now exposed as a named parameter (`18735`)
- The options ``html.border`` and ``mode.use_inf_as_null`` were deprecated in prior versions, these will now show ``FutureWarning`` rather than a ``DeprecationWarning`` (`19003`)
- `IntervalIndex` and ``IntervalDtype`` no longer support categorical, object, and string subtypes (`19016`)
- ``IntervalDtype`` now returns ``True`` when compared against ``'interval'`` regardless of subtype, and ``IntervalDtype.name`` now returns ``'interval'`` regardless of subtype (`18980`)
- ``KeyError`` now raises instead of ``ValueError`` in `~DataFrame.drop`, `~Panel.drop`, `~Series.drop`, `~Index.drop` when dropping a non-existent element in an axis with duplicates (`19186`)
- `Series.to_csv` now accepts a ``compression`` argument that works in the same way as the ``compression`` argument in `DataFrame.to_csv` (`18958`)
- Set operations (union, difference...) on `IntervalIndex` with incompatible index types will now raise a ``TypeError`` rather than a ``ValueError`` (`19329`)
- `DateOffset` objects render more simply, e.g. ``<DateOffset: days=1>`` instead of ``<DateOffset: kwds={'days': 1}>`` (`19403`)
- ``Categorical.fillna`` now validates its ``value`` and ``method`` keyword arguments. It now raises when both or none are specified, matching the behavior of `Series.fillna` (`19682`)
- ``pd.to_datetime('today')`` now returns a datetime, consistent with ``pd.Timestamp('today')``; previously ``pd.to_datetime('today')`` returned a ``.normalized()`` datetime (`19935`)
- `Series.str.replace` now takes an optional ``regex`` keyword which, when set to ``False``, uses literal string replacement rather than regex replacement (`16808`)
- `DatetimeIndex.strftime` and `PeriodIndex.strftime` now return an ``Index`` instead of a numpy array to be consistent with similar accessors (`20127`)
- Constructing a Series from a list of length 1 no longer broadcasts this list when a longer index is specified (`19714`, `20391`).
- `DataFrame.to_dict` with ``orient='index'`` no longer casts int columns to float for a DataFrame with only int and float columns (`18580`)
- A user-defined-function that is passed to `Series.rolling().aggregate() <pandas.core.window.Rolling.aggregate>`, `DataFrame.rolling().aggregate() <pandas.core.window.Rolling.aggregate>`, or its expanding cousins, will now *always* be passed a ``Series``, rather than a ``np.array``; ``.apply()`` only has the ``raw`` keyword, see `here <whatsnew_0230.enhancements.window_raw>`. This is consistent with the signatures of ``.aggregate()`` across pandas (`20584`)
- Rolling and Expanding types raise ``NotImplementedError`` upon iteration (`11704`).  .. _whatsnew_0230.deprecations:  Deprecations
~~~  - ``Series.from_array`` and ``SparseSeries.from_array`` are deprecated. Use the normal constructor ``Series(..)`` and ``SparseSeries(..)`` instead (`18213`).
- ``DataFrame.as_matrix`` is deprecated. Use ``DataFrame.values`` instead (`18458`).

- ``Series.asobject``, ``DatetimeIndex.asobject``, ``PeriodIndex.asobject`` and ``TimeDeltaIndex.asobject`` have been deprecated. Use ``.astype(object)`` instead (`18572`)
- Grouping by a tuple of keys now emits a ``FutureWarning`` and is deprecated.  In the future, a tuple passed to ``'by'`` will always refer to a single key  that is the actual tuple, instead of treating the tuple as multiple keys. To  retain the previous behavior, use a list instead of a tuple (`18314`)
- ``Series.valid`` is deprecated. Use `Series.dropna` instead (`18800`).
- `read_excel` has deprecated the ``skip_footer`` parameter. Use ``skipfooter`` instead (`18836`)
- `ExcelFile.parse` has deprecated ``sheetname`` in favor of ``sheet_name`` for consistency with `read_excel` (`20920`).
- The ``is_copy`` attribute is deprecated and will be removed in a future version (`18801`).
- ``IntervalIndex.from_intervals`` is deprecated in favor of the `IntervalIndex` constructor (`19263`)
- ``DataFrame.from_items`` is deprecated. Use `DataFrame.from_dict` instead, or ``DataFrame.from_dict(OrderedDict())`` if you wish to preserve the key order (`17320`, `17312`)
- Indexing a `MultiIndex` or a `FloatIndex` with a list containing some missing keys will now show a `FutureWarning`, which is consistent with other types of indexes (`17758`).  - The ``broadcast`` parameter of ``.apply()`` is deprecated in favor of ``result_type='broadcast'`` (`18577`)
- The ``reduce`` parameter of ``.apply()`` is deprecated in favor of ``result_type='reduce'`` (`18577`)
- The ``order`` parameter of `factorize` is deprecated and will be removed in a future release (`19727`)
- `Timestamp.weekday_name`, `DatetimeIndex.weekday_name`, and `Series.dt.weekday_name` are deprecated in favor of `Timestamp.day_name`, `DatetimeIndex.day_name`, and `Series.dt.day_name` (`12806`)  - ``pandas.tseries.plotting.tsplot`` is deprecated. Use `Series.plot` instead (`18627`)
- ``Index.summary()`` is deprecated and will be removed in a future version (`18217`)
- ``NDFrame.get_ftype_counts()`` is deprecated and will be removed in a future version (`18243`)
- The ``convert_datetime64`` parameter in `DataFrame.to_records` has been deprecated and will be removed in a future version. The NumPy bug motivating this parameter has been resolved. The default value for this parameter has also changed from ``True`` to ``None`` (`18160`).
- `Series.rolling().apply() <pandas.core.window.Rolling.apply>`, `DataFrame.rolling().apply() <pandas.core.window.Rolling.apply>`,  `Series.expanding().apply() <pandas.core.window.Expanding.apply>`, and `DataFrame.expanding().apply() <pandas.core.window.Expanding.apply>` have deprecated passing an ``np.array`` by default. One will need to pass the new ``raw`` parameter to be explicit about what is passed (`20584`)
- The ``data``, ``base``, ``strides``, ``flags`` and ``itemsize`` properties  of the ``Series`` and ``Index`` classes have been deprecated and will be  removed in a future version (`20419`).
- ``DatetimeIndex.offset`` is deprecated. Use ``DatetimeIndex.freq`` instead (`20716`)
- Floor division between an integer ndarray and a `Timedelta` is deprecated. Divide by `Timedelta.value` instead (`19761`)
- Setting ``PeriodIndex.freq`` (which was not guaranteed to work correctly) is deprecated. Use `PeriodIndex.asfreq` instead (`20678`)
- ``Index.get_duplicates()`` is deprecated and will be removed in a future version (`20239`)
- The previous default behavior of negative indices in ``Categorical.take`` is deprecated. In a future version it will change from meaning missing values to meaning positional indices from the right. The future behavior is consistent with `Series.take` (`20664`).
- Passing multiple axes to the ``axis`` parameter in `DataFrame.dropna` has been deprecated and will be removed in a future version (`20987`)  .. _whatsnew_0230.prior_deprecations:  Removal of prior version deprecations/changes

~~~  - Warnings against the obsolete usage ``Categorical(codes, categories)``, which were emitted for instance when the first two arguments to ``Categorical()`` had different dtypes, and recommended the use of ``Categorical.from_codes``, have now been removed (`8074`)
- The ``levels`` and ``labels`` attributes of a ``MultiIndex`` can no longer be set directly (`4039`).
- ``pd.tseries.util.pivot_annual`` has been removed (deprecated since v0.19). Use ``pivot_table`` instead (`18370`)
- ``pd.tseries.util.isleapyear`` has been removed (deprecated since v0.19). Use ``.is_leap_year`` property in Datetime-likes instead (`18370`)
- ``pd.ordered_merge`` has been removed (deprecated since v0.19). Use ``pd.merge_ordered`` instead (`18459`)
- The ``SparseList`` class has been removed (`14007`)
- The ``pandas.io.wb`` and ``pandas.io.data`` stub modules have been removed (`13735`)
- ``Categorical.from_array`` has been removed (`13854`)
- The ``freq`` and ``how`` parameters have been removed from the ``rolling``/``expanding``/``ewm`` methods of DataFrame  and Series (deprecated since v0.18). Instead, resample before calling the methods. (`18601` & `18668`)
- ``DatetimeIndex.to_datetime``, ``Timestamp.to_datetime``, ``PeriodIndex.to_datetime``, and ``Index.to_datetime`` have been removed (`8254`, `14096`, `14113`)
- `read_csv` has dropped the ``skip_footer`` parameter (`13386`)
- `read_csv` has dropped the ``as_recarray`` parameter (`13373`)
- `read_csv` has dropped the ``buffer_lines`` parameter (`13360`)
- `read_csv` has dropped the ``compact_ints`` and ``use_unsigned`` parameters (`13323`)
- The ``Timestamp`` class has dropped the ``offset`` attribute in favor of ``freq`` (`13593`)
- The ``Series``, ``Categorical``, and ``Index`` classes have dropped the ``reshape`` method (`13012`)
- ``pandas.tseries.frequencies.get_standard_freq`` has been removed in favor of ``pandas.tseries.frequencies.to_offset(freq).rule_code`` (`13874`)
- The ``freqstr`` keyword has been removed from ``pandas.tseries.frequencies.to_offset`` in favor of ``freq`` (`13874`)
- The ``Panel4D`` and ``PanelND`` classes have been removed (`13776`)
- The ``Panel`` class has dropped the ``to_long`` and ``toLong`` methods (`19077`)
- The options ``display.line_with`` and ``display.height`` are removed in favor of ``display.width`` and ``display.max_rows`` respectively (`4391`, `19107`)
- The ``labels`` attribute of the ``Categorical`` class has been removed in favor of `Categorical.codes` (`7768`)
- The ``flavor`` parameter have been removed from `to_sql` method (`13611`)
- The modules ``pandas.tools.hashing`` and ``pandas.util.hashing`` have been removed (`16223`)
- The top-level functions ``pd.rolling_*``, ``pd.expanding_*`` and ``pd.ewm*`` have been removed (Deprecated since v0.18).  Instead, use the DataFrame/Series methods `~DataFrame.rolling`, `~DataFrame.expanding` and `~DataFrame.ewm` (`18723`)
- Imports from ``pandas.core.common`` for functions such as ``is_datetime64_dtype`` are now removed. These are located in ``pandas.api.types``. (`13634`, `19769`)
- The ``infer_dst`` keyword in `Series.tz_localize`, `DatetimeIndex.tz_localize`  and `DatetimeIndex` have been removed. ``infer_dst=True`` is equivalent to  ``ambiguous='infer'``, and ``infer_dst=False`` to ``ambiguous='raise'`` (`7963`).
- When ``.resample()`` was changed from an eager to a lazy operation, like ``.groupby()`` in v0.18.0, we put in place compatibility (with a ``FutureWarning``),  so operations would continue to work. This is now fully removed, so a ``Resampler`` will no longer forward compat operations (`20554`)
- Remove long deprecated ``axis=None`` parameter from ``.replace()`` (`20271`)  .. _whatsnew_0230.performance:  Performance improvements
~~~  - Indexers on ``Series`` or ``DataFrame`` no longer create a reference cycle (`17956`)
- Added a keyword argument, ``cache``, to `to_datetime` that improved the performance of converting duplicate datetime arguments (`11665`)

- `DateOffset` arithmetic performance is improved (`18218`)
- Converting a ``Series`` of ``Timedelta`` objects to days, seconds, etc... sped up through vectorization of underlying methods (`18092`)
- Improved performance of ``.map()`` with a ``Series/dict`` input (`15081`)
- The overridden ``Timedelta`` properties of days, seconds and microseconds have been removed, leveraging their built-in Python versions instead (`18242`)
- ``Series`` construction will reduce the number of copies made of the input data in certain cases (`17449`)
- Improved performance of `Series.dt.date` and `DatetimeIndex.date` (`18058`)
- Improved performance of `Series.dt.time` and `DatetimeIndex.time` (`18461`)
- Improved performance of `IntervalIndex.symmetric_difference()` (`18475`)
- Improved performance of ``DatetimeIndex`` and ``Series`` arithmetic operations with Business-Month and Business-Quarter frequencies (`18489`)
- `Series` / `DataFrame` tab completion limits to 100 values, for better performance. (`18587`)
- Improved performance of `DataFrame.median` with ``axis=1`` when bottleneck is not installed (`16468`)
- Improved performance of `MultiIndex.get_loc` for large indexes, at the cost of a reduction in performance for small ones (`18519`)
- Improved performance of `MultiIndex.remove_unused_levels` when there are no unused levels, at the cost of a reduction in performance when there are (`19289`)
- Improved performance of `Index.get_loc` for non-unique indexes (`19478`)
- Improved performance of pairwise ``.rolling()`` and ``.expanding()`` with ``.cov()`` and ``.corr()`` operations (`17917`)
- Improved performance of `pandas.core.groupby.GroupBy.rank` (`15779`)
- Improved performance of variable ``.rolling()`` on ``.min()`` and ``.max()`` (`19521`)
- Improved performance of `pandas.core.groupby.GroupBy.ffill` and `pandas.core.groupby.GroupBy.bfill` (`11296`)
- Improved performance of `pandas.core.groupby.GroupBy.any` and `pandas.core.groupby.GroupBy.all` (`15435`)
- Improved performance of `pandas.core.groupby.GroupBy.pct_change` (`19165`)
- Improved performance of `Series.isin` in the case of categorical dtypes (`20003`)
- Improved performance of ``getattr(Series, attr)`` when the Series has certain index types. This manifested in slow printing of large Series with a ``DatetimeIndex`` (`19764`)
- Fixed a performance regression for `GroupBy.nth` and `GroupBy.last` with some object columns (`19283`)
- Improved performance of `pandas.core.arrays.Categorical.from_codes` (`18501`)  .. _whatsnew_0230.docs:  Documentation changes
~~~  Thanks to all of the contributors who participated in the pandas Documentation
Sprint, which took place on March 10th. We had about 500 participants from over
30 locations across the world. You should notice that many of the
`API docstrings <api>` have greatly improved.  There were too many simultaneous contributions to include a release note for each
improvement, but this `GitHub search`_ should give you an idea of how many docstrings
were improved.  Special thanks to `Marc Garcia`_ for organizing the sprint. For more information,
read the `NumFOCUS blogpost`_ recapping the sprint.  .. _GitHub search: https://github.com/pandas-dev/pandas/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3ADocs+created%3A2018-03-10..2018-03-15+
.. _NumFOCUS blogpost: https://www.numfocus.org/blog/worldwide-pandas-sprint/
.. _Marc Garcia: https://github.com/datapythonista  - Changed spelling of "numpy" to "NumPy", and "python" to "Python". (`19017`)
- Consistency when introducing code samples, using either colon or period.  Rewrote some sentences for greater clarity, added more dynamic references  to functions, methods and classes.  (`18941`, `18948`, `18973`, `19017`)
- Added a reference to `DataFrame.assign` in the concatenate section of the merging documentation (`18665`)  .. _whatsnew_0230.bug_fixes:  Bug fixes
~~~  Categorical

^^^  .. warning::  A class of bugs were introduced in pandas 0.21 with ``CategoricalDtype`` that  affects the correctness of operations like ``merge``, ``concat``, and  indexing when comparing multiple unordered ``Categorical`` arrays that have  the same categories, but in a different order. We highly recommend upgrading  or manually aligning your categories before doing these operations.  - Bug in ``Categorical.equals`` returning the wrong result when comparing two  unordered ``Categorical`` arrays with the same categories, but in a different  order (`16603`)
- Bug in `pandas.api.types.union_categoricals` returning the wrong result  when for unordered categoricals with the categories in a different order.  This affected `pandas.concat` with Categorical data (`19096`).
- Bug in `pandas.merge` returning the wrong result when joining on an  unordered ``Categorical`` that had the same categories but in a different  order (`19551`)
- Bug in `CategoricalIndex.get_indexer` returning the wrong result when  ``target`` was an unordered ``Categorical`` that had the same categories as  ``self`` but in a different order (`19551`)
- Bug in `Index.astype` with a categorical dtype where the resultant index is not converted to a `CategoricalIndex` for all types of index (`18630`)
- Bug in `Series.astype` and ``Categorical.astype()`` where an existing categorical data does not get updated (`10696`, `18593`)
- Bug in `Series.str.split` with ``expand=True`` incorrectly raising an IndexError on empty strings (`20002`).
- Bug in `Index` constructor with ``dtype=CategoricalDtype(...)`` where ``categories`` and ``ordered`` are not maintained (`19032`)
- Bug in `Series` constructor with scalar and ``dtype=CategoricalDtype(...)`` where ``categories`` and ``ordered`` are not maintained (`19565`)
- Bug in ``Categorical.__iter__`` not converting to Python types (`19909`)
- Bug in `pandas.factorize` returning the unique codes for the ``uniques``. This now returns a ``Categorical`` with the same dtype as the input (`19721`)
- Bug in `pandas.factorize` including an item for missing values in the ``uniques`` return value (`19721`)
- Bug in `Series.take` with categorical data interpreting ``-1`` in ``indices`` as missing value markers, rather than the last element of the Series (`20664`)  Datetimelike
^^^  - Bug in `Series.__sub__` subtracting a non-nanosecond ``np.datetime64`` object from a ``Series`` gave incorrect results (`7996`)
- Bug in `DatetimeIndex`, `TimedeltaIndex` addition and subtraction of zero-dimensional integer arrays gave incorrect results (`19012`)
- Bug in `DatetimeIndex` and `TimedeltaIndex` where adding or subtracting an array-like of ``DateOffset`` objects either raised (``np.array``, ``pd.Index``) or broadcast incorrectly (``pd.Series``) (`18849`)
- Bug in `Series.__add__` adding Series with dtype ``timedelta64[ns]`` to a timezone-aware ``DatetimeIndex`` incorrectly dropped timezone information (`13905`)
- Adding a ``Period`` object to a ``datetime`` or ``Timestamp`` object will now correctly raise a ``TypeError`` (`17983`)
- Bug in `Timestamp` where comparison with an array of ``Timestamp`` objects would result in a ``RecursionError`` (`15183`)
- Bug in `Series` floor-division where operating on a scalar ``timedelta`` raises an exception (`18846`)
- Bug in `DatetimeIndex` where the repr was not showing high-precision time values at the end of a day (e.g., 2359.999) (`19030`)
- Bug in ``.astype()`` to non-ns timedelta units would hold the incorrect dtype (`19176`, `19223`, `12425`)
- Bug in subtracting `Series` from ``NaT`` incorrectly returning ``NaT`` (`19158`)
- Bug in `Series.truncate` which raises ``TypeError`` with a monotonic ``PeriodIndex`` (`17717`)
- Bug in `~DataFrame.pct_change` using ``periods`` and ``freq`` returned different length outputs (`7292`)

- Bug in comparison of `DatetimeIndex` against ``None`` or ``datetime.date`` objects raising ``TypeError`` for ``==`` and ``!=`` comparisons instead of all-``False`` and all-``True``, respectively (`19301`)
- Bug in `Timestamp` and `to_datetime` where a string representing a barely out-of-bounds timestamp would be incorrectly rounded down instead of raising ``OutOfBoundsDatetime`` (`19382`)
- Bug in `Timestamp.floor` `DatetimeIndex.floor` where time stamps far in the future and past were not rounded correctly (`19206`)
- Bug in `to_datetime` where passing an out-of-bounds datetime with ``errors='coerce'`` and ``utc=True`` would raise ``OutOfBoundsDatetime`` instead of parsing to ``NaT`` (`19612`)
- Bug in `DatetimeIndex` and `TimedeltaIndex` addition and subtraction where name of the returned object was not always set consistently. (`19744`)
- Bug in `DatetimeIndex` and `TimedeltaIndex` addition and subtraction where operations with numpy arrays raised ``TypeError`` (`19847`)
- Bug in `DatetimeIndex` and `TimedeltaIndex` where setting the ``freq`` attribute was not fully supported (`20678`)  Timedelta
^^^  - Bug in `Timedelta.__mul__` where multiplying by ``NaT`` returned ``NaT`` instead of raising a ``TypeError`` (`19819`)
- Bug in `Series` with ``dtype='timedelta64[ns]'`` where addition or subtraction of ``TimedeltaIndex`` had results cast to ``dtype='int64'`` (`17250`)
- Bug in `Series` with ``dtype='timedelta64[ns]'`` where addition or subtraction of ``TimedeltaIndex`` could return a ``Series`` with an incorrect name (`19043`)
- Bug in `Timedelta.__floordiv__` and `Timedelta.__rfloordiv__` dividing by many incompatible numpy objects was incorrectly allowed (`18846`)
- Bug where dividing a scalar timedelta-like object with `TimedeltaIndex` performed the reciprocal operation (`19125`)
- Bug in `TimedeltaIndex` where division by a ``Series`` would return a ``TimedeltaIndex`` instead of a ``Series`` (`19042`)
- Bug in `Timedelta.__add__`, `Timedelta.__sub__` where adding or subtracting a ``np.timedelta64`` object would return another ``np.timedelta64`` instead of a ``Timedelta`` (`19738`)
- Bug in `Timedelta.__floordiv__`, `Timedelta.__rfloordiv__` where operating with a ``Tick`` object would raise a ``TypeError`` instead of returning a numeric value (`19738`)
- Bug in `Period.asfreq` where periods near ``datetime(1, 1, 1)`` could be converted incorrectly (`19643`, `19834`)
- Bug in `Timedelta.total_seconds()` causing precision errors, for example ``Timedelta('30S').total_seconds()==30.0004`` (`19458`)
- Bug in `Timedelta.__rmod__` where operating with a ``numpy.timedelta64`` returned a ``timedelta64`` object instead of a ``Timedelta`` (`19820`)
- Multiplication of `TimedeltaIndex` by ``TimedeltaIndex`` will now raise ``TypeError`` instead of raising ``ValueError`` in cases of length mismatch (`19333`)
- Bug in indexing a `TimedeltaIndex` with a ``np.timedelta64`` object which was raising a ``TypeError`` (`20393`)  Timezones
^^^  - Bug in creating a ``Series`` from an array that contains both tz-naive and tz-aware values will result in a ``Series`` whose dtype is tz-aware instead of object (`16406`)
- Bug in comparison of timezone-aware `DatetimeIndex` against ``NaT`` incorrectly raising ``TypeError`` (`19276`)
- Bug in `DatetimeIndex.astype` when converting between timezone aware dtypes, and converting from timezone aware to naive (`18951`)
- Bug in comparing `DatetimeIndex`, which failed to raise ``TypeError`` when attempting to compare timezone-aware and timezone-naive datetimelike objects (`18162`)
- Bug in localization of a naive, datetime string in a ``Series`` constructor with a ``datetime64[ns, tz]`` dtype (`174151`)
- `Timestamp.replace` will now handle Daylight Savings transitions gracefully (`18319`)

- Bug in tz-aware `DatetimeIndex` where addition/subtraction with a `TimedeltaIndex` or array with ``dtype='timedelta64[ns]'`` was incorrect (`17558`)
- Bug in `DatetimeIndex.insert` where inserting ``NaT`` into a timezone-aware index incorrectly raised (`16357`)
- Bug in `DataFrame` constructor, where tz-aware Datetimeindex and a given column name will result in an empty ``DataFrame`` (`19157`)
- Bug in `Timestamp.tz_localize` where localizing a timestamp near the minimum or maximum valid values could overflow and return a timestamp with an incorrect nanosecond value (`12677`)
- Bug when iterating over `DatetimeIndex` that was localized with fixed timezone offset that rounded nanosecond precision to microseconds (`19603`)
- Bug in `DataFrame.diff` that raised an ``IndexError`` with tz-aware values (`18578`)
- Bug in `melt` that converted tz-aware dtypes to tz-naive (`15785`)
- Bug in `Dataframe.count` that raised an ``ValueError``, if `Dataframe.dropna` was called for a single column with timezone-aware values. (`13407`)  Offsets
^^^  - Bug in `WeekOfMonth` and `Week` where addition and subtraction did not roll correctly (`18510`, `18672`, `18864`)
- Bug in `WeekOfMonth` and `LastWeekOfMonth` where default keyword arguments for constructor raised ``ValueError`` (`19142`)
- Bug in `FY5253Quarter`, `LastWeekOfMonth` where rollback and rollforward behavior was inconsistent with addition and subtraction behavior (`18854`)
- Bug in `FY5253` where ``datetime`` addition and subtraction incremented incorrectly for dates on the year-end but not normalized to midnight (`18854`)
- Bug in `FY5253` where date offsets could incorrectly raise an ``AssertionError`` in arithmetic operations (`14774`)  Numeric
^^^
- Bug in `Series` constructor with an int or float list where specifying ``dtype=str``, ``dtype='str'`` or ``dtype='U'`` failed to convert the data elements to strings (`16605`)
- Bug in `Index` multiplication and division methods where operating with a ``Series`` would return an ``Index`` object instead of a ``Series`` object (`19042`)
- Bug in the `DataFrame` constructor in which data containing very large positive or very large negative numbers was causing ``OverflowError`` (`18584`)
- Bug in `Index` constructor with ``dtype='uint64'`` where int-like floats were not coerced to `UInt64Index` (`18400`)
- Bug in `DataFrame` flex arithmetic (e.g. ``df.add(other, fill_value=foo)``) with a ``fill_value`` other than ``None`` failed to raise ``NotImplementedError`` in corner cases where either the frame or ``other`` has length zero (`19522`)
- Multiplication and division of numeric-dtyped `Index` objects with timedelta-like scalars returns ``TimedeltaIndex`` instead of raising ``TypeError`` (`19333`)
- Bug where ``NaN`` was returned instead of 0 by `Series.pct_change` and `DataFrame.pct_change` when ``fill_method`` is not ``None`` (`19873`)  Strings
^^^
- Bug in `Series.str.get` with a dictionary in the values and the index not in the keys, raising ``KeyError`` (`20671`)  Indexing
^^^  - Bug in `Index` construction from list of mixed type tuples (`18505`)
- Bug in `Index.drop` when passing a list of both tuples and non-tuples (`18304`)
- Bug in `DataFrame.drop`, `Panel.drop`, `Series.drop`, `Index.drop` where no ``KeyError`` is raised when dropping a non-existent element from an axis that contains duplicates (`19186`)
- Bug in indexing a datetimelike ``Index`` that raised ``ValueError`` instead of ``IndexError`` (`18386`).
- `Index.to_series` now accepts ``index`` and ``name`` kwargs (`18699`)
- `DatetimeIndex.to_series` now accepts ``index`` and ``name`` kwargs (`18699`)
- Bug in indexing non-scalar value from ``Series`` having non-unique ``Index`` will return value flattened (`17610`)
- Bug in indexing with iterator containing only missing keys, which raised no error (`20748`)

- Fixed inconsistency in ``.ix`` between list and scalar keys when the index has integer dtype and does not include the desired keys (`20753`)
- Bug in ``__setitem__`` when indexing a `DataFrame` with a 2-d boolean ndarray (`18582`)
- Bug in ``str.extractall`` when there were no matches empty `Index` was returned instead of appropriate `MultiIndex` (`19034`)
- Bug in `IntervalIndex` where empty and purely NA data was constructed inconsistently depending on the construction method (`18421`)
- Bug in `IntervalIndex.symmetric_difference` where the symmetric difference with a non-``IntervalIndex`` did not raise (`18475`)
- Bug in `IntervalIndex` where set operations that returned an empty ``IntervalIndex`` had the wrong dtype (`19101`)
- Bug in `DataFrame.drop_duplicates` where no ``KeyError`` is raised when passing in columns that don't exist on the ``DataFrame`` (`19726`)
- Bug in ``Index`` subclasses constructors that ignore unexpected keyword arguments (`19348`)
- Bug in `Index.difference` when taking difference of an ``Index`` with itself (`20040`)
- Bug in `DataFrame.first_valid_index` and `DataFrame.last_valid_index` in presence of entire rows of NaNs in the middle of values (`20499`).
- Bug in `IntervalIndex` where some indexing operations were not supported for overlapping or non-monotonic ``uint64`` data (`20636`)
- Bug in ``Series.is_unique`` where extraneous output in stderr is shown if Series contains objects with ``__ne__`` defined (`20661`)
- Bug in ``.loc`` assignment with a single-element list-like incorrectly assigns as a list (`19474`)
- Bug in partial string indexing on a ``Series/DataFrame`` with a monotonic decreasing ``DatetimeIndex`` (`19362`)
- Bug in performing in-place operations on a ``DataFrame`` with a duplicate ``Index`` (`17105`)
- Bug in `IntervalIndex.get_loc` and `IntervalIndex.get_indexer` when used with an `IntervalIndex` containing a single interval (`17284`, `20921`)
- Bug in ``.loc`` with a ``uint64`` indexer (`20722`)  MultiIndex
^^^  - Bug in `MultiIndex.__contains__` where non-tuple keys would return ``True`` even if they had been dropped (`19027`)
- Bug in `MultiIndex.set_labels` which would cause casting (and potentially clipping) of the new labels if the ``level`` argument is not 0 or a list like [0, 1, ... ]  (`19057`)
- Bug in `MultiIndex.get_level_values` which would return an invalid index on level of ints with missing values (`17924`)
- Bug in `MultiIndex.unique` when called on empty `MultiIndex` (`20568`)
- Bug in `MultiIndex.unique` which would not preserve level names (`20570`)
- Bug in `MultiIndex.remove_unused_levels` which would fill nan values (`18417`)
- Bug in `MultiIndex.from_tuples` which would fail to take zipped tuples in python3 (`18434`)
- Bug in `MultiIndex.get_loc` which would fail to automatically cast values between float and int (`18818`, `15994`)
- Bug in `MultiIndex.get_loc` which would cast boolean to integer labels (`19086`)
- Bug in `MultiIndex.get_loc` which would fail to locate keys containing ``NaN`` (`18485`)
- Bug in `MultiIndex.get_loc` in large `MultiIndex`, would fail when levels had different dtypes (`18520`)
- Bug in indexing where nested indexers having only numpy arrays are handled incorrectly (`19686`)  IO
^^  - `read_html` now rewinds seekable IO objects after parse failure, before attempting to parse with a new parser. If a parser errors and the object is non-seekable, an informative error is raised suggesting the use of a different parser (`17975`)
- `DataFrame.to_html` now has an option to add an id to the leading ``<table>`` tag (`8496`)
- Bug in `read_msgpack` with a non existent file is passed in Python 2 (`15296`)
- Bug in `read_csv` where a ``MultiIndex`` with duplicate columns was not being mangled appropriately (`18062`)
- Bug in `read_csv` where missing values were not being handled properly when ``keep_default_na=False`` with dictionary ``na_values`` (`19227`)

- Bug in `read_csv` causing heap corruption on 32-bit, big-endian architectures (`20785`)
- Bug in `read_sas` where a file with 0 variables gave an ``AttributeError`` incorrectly. Now it gives an ``EmptyDataError`` (`18184`)
- Bug in `DataFrame.to_latex()` where pairs of braces meant to serve as invisible placeholders were escaped (`18667`)
- Bug in `DataFrame.to_latex()` where a ``NaN`` in a ``MultiIndex`` would cause an ``IndexError`` or incorrect output (`14249`)
- Bug in `DataFrame.to_latex()` where a non-string index-level name would result in an ``AttributeError`` (`19981`)
- Bug in `DataFrame.to_latex()` where the combination of an index name and the ``index_names=False`` option would result in incorrect output (`18326`)
- Bug in `DataFrame.to_latex()` where a ``MultiIndex`` with an empty string as its name would result in incorrect output (`18669`)
- Bug in `DataFrame.to_latex()` where missing space characters caused wrong escaping and produced non-valid latex in some cases (`20859`)
- Bug in `read_json` where large numeric values were causing an ``OverflowError`` (`18842`)
- Bug in `DataFrame.to_parquet` where an exception was raised if the write destination is S3 (`19134`)
- `Interval` now supported in `DataFrame.to_excel` for all Excel file types (`19242`)
- `Timedelta` now supported in `DataFrame.to_excel` for all Excel file types (`19242`, `9155`, `19900`)
- Bug in `pandas.io.stata.StataReader.value_labels` raising an ``AttributeError`` when called on very old files. Now returns an empty dict (`19417`)
- Bug in `read_pickle` when unpickling objects with `TimedeltaIndex` or `Float64Index` created with pandas prior to version 0.20 (`19939`)
- Bug in `pandas.io.json.json_normalize` where sub-records are not properly normalized if any sub-records values are NoneType (`20030`)
- Bug in ``usecols`` parameter in `read_csv` where error is not raised correctly when passing a string. (`20529`)
- Bug in `HDFStore.keys` when reading a file with a soft link causes exception (`20523`)
- Bug in `HDFStore.select_column` where a key which is not a valid store raised an ``AttributeError`` instead of a ``KeyError`` (`17912`)  Plotting
^^^  - Better error message when attempting to plot but matplotlib is not installed (`19810`).
- `DataFrame.plot` now raises a ``ValueError`` when the ``x`` or ``y`` argument is improperly formed (`18671`)
- Bug in `DataFrame.plot` when ``x`` and ``y`` arguments given as positions caused incorrect referenced columns for line, bar and area plots (`20056`)
- Bug in formatting tick labels with ``datetime.time()`` and fractional seconds (`18478`).
- `Series.plot.kde` has exposed the args ``ind`` and ``bw_method`` in the docstring (`18461`). The argument ``ind`` may now also be an integer (number of sample points).
- `DataFrame.plot` now supports multiple columns to the ``y`` argument (`19699`)  GroupBy/resample/rolling
^^^  - Bug when grouping by a single column and aggregating with a class like ``list`` or ``tuple`` (`18079`)
- Fixed regression in `DataFrame.groupby` which would not emit an error when called with a tuple key not in the index (`18798`)
- Bug in `DataFrame.resample` which silently ignored unsupported (or mistyped) options for ``label``, ``closed`` and ``convention`` (`19303`)
- Bug in `DataFrame.groupby` where tuples were interpreted as lists of keys rather than as keys (`17979`, `18249`)
- Bug in `DataFrame.groupby` where aggregation by ``first``/``last``/``min``/``max`` was causing timestamps to lose precision (`19526`)
- Bug in `DataFrame.transform` where particular aggregation functions were being incorrectly cast to match the dtype(s) of the grouped data (`19200`)
- Bug in `DataFrame.groupby` passing the ``on=`` kwarg, and subsequently using ``.apply()`` (`17813`)

- Bug in `DataFrame.resample().aggregate <pandas.core.resample.Resampler.aggregate>` not raising a ``KeyError`` when aggregating a non-existent column (`16766`, `19566`)
- Bug in `DataFrameGroupBy.cumsum` and `DataFrameGroupBy.cumprod` when ``skipna`` was passed (`19806`)
- Bug in `DataFrame.resample` that dropped timezone information (`13238`)
- Bug in `DataFrame.groupby` where transformations using ``np.all`` and ``np.any`` were raising a ``ValueError`` (`20653`)
- Bug in `DataFrame.resample` where ``ffill``, ``bfill``, ``pad``, ``backfill``, ``fillna``, ``interpolate``, and ``asfreq`` were ignoring ``loffset``. (`20744`)
- Bug in `DataFrame.groupby` when applying a function that has mixed data types and the user supplied function can fail on the grouping column (`20949`)
- Bug in `DataFrameGroupBy.rolling().apply() <pandas.core.window.Rolling.apply>` where operations performed against the associated `DataFrameGroupBy` object could impact the inclusion of the grouped item(s) in the result (`14013`)  Sparse
^^^  - Bug in which creating a `SparseDataFrame` from a dense ``Series`` or an unsupported type raised an uncontrolled exception (`19374`)
- Bug in `SparseDataFrame.to_csv` causing exception (`19384`)
- Bug in `SparseSeries.memory_usage` which caused segfault by accessing non sparse elements (`19368`)
- Bug in constructing a `SparseArray`: if ``data`` is a scalar and ``index`` is defined it will coerce to ``float64`` regardless of scalar's dtype. (`19163`)  Reshaping
^^^  - Bug in `DataFrame.merge` where referencing a ``CategoricalIndex`` by name, where the ``by`` kwarg would ``KeyError`` (`20777`)
- Bug in `DataFrame.stack` which fails trying to sort mixed type levels under Python 3 (`18310`)
- Bug in `DataFrame.unstack` which casts int to float if ``columns`` is a ``MultiIndex`` with unused levels (`17845`)
- Bug in `DataFrame.unstack` which raises an error if ``index`` is a ``MultiIndex`` with unused labels on the unstacked level (`18562`)
- Fixed construction of a `Series` from a ``dict`` containing ``NaN`` as key (`18480`)
- Fixed construction of a `DataFrame` from a ``dict`` containing ``NaN`` as key (`18455`)
- Disabled construction of a `Series` where len(index) > len(data) = 1, which previously would broadcast the data item, and now raises a ``ValueError`` (`18819`)
- Suppressed error in the construction of a `DataFrame` from a ``dict`` containing scalar values when the corresponding keys are not included in the passed index (`18600`)  - Fixed (changed from ``object`` to ``float64``) dtype of `DataFrame` initialized with axes, no data, and ``dtype=int`` (`19646`)
- Bug in `Series.rank` where ``Series`` containing ``NaT`` modifies the ``Series`` inplace (`18521`)
- Bug in `cut` which fails when using readonly arrays (`18773`)
- Bug in `DataFrame.pivot_table` which fails when the ``aggfunc`` arg is of type string.  The behavior is now consistent with other methods like ``agg`` and ``apply`` (`18713`)
- Bug in `DataFrame.merge` in which merging using ``Index`` objects as vectors raised an Exception (`19038`)
- Bug in `DataFrame.stack`, `DataFrame.unstack`, `Series.unstack` which were not returning subclasses (`15563`)
- Bug in timezone comparisons, manifesting as a conversion of the index to UTC in ``.concat()`` (`18523`)
- Bug in `concat` when concatenating sparse and dense series it returns only a ``SparseDataFrame``. Should be a ``DataFrame``. (`18914`, `18686`, and `16874`)
- Improved error message for `DataFrame.merge` when there is no common merge key (`19427`)
- Bug in `DataFrame.join` which does an ``outer`` instead of a ``left`` join when being called with multiple DataFrames and some have non-unique indices (`19624`)
- `Series.rename` now accepts ``axis`` as a kwarg (`18589`)
- Bug in `~DataFrame.rename` where an Index of same-length tuples was converted to a MultiIndex (`19497`)

- Comparisons between `Series` and `Index` would return a ``Series`` with an incorrect name, ignoring the ``Index``'s name attribute (`19582`)
- Bug in `qcut` where datetime and timedelta data with ``NaT`` present raised a ``ValueError`` (`19768`)
- Bug in DataFrame.iterrows`, which would infers strings not compliant to `ISO8601 to datetimes (`19671`)
- Bug in `Series` constructor with ``Categorical`` where a ``ValueError`` is not raised when an index of different length is given (`19342`)
- Bug in `DataFrame.astype` where column metadata is lost when converting to categorical or a dictionary of dtypes (`19920`)
- Bug in `cut` and `qcut` where timezone information was dropped (`19872`)
- Bug in `Series` constructor with a ``dtype=str``, previously raised in some cases (`19853`)
- Bug in `get_dummies`, and `select_dtypes`, where duplicate column names caused incorrect behavior (`20848`)
- Bug in `isna`, which cannot handle ambiguous typed lists (`20675`)
- Bug in `concat` which raises an error when concatenating TZ-aware dataframes and all-NaT dataframes (`12396`)
- Bug in `concat` which raises an error when concatenating empty TZ-aware series (`18447`)  Other
^^^  - Improved error message when attempting to use a Python keyword as an identifier in a ``numexpr`` backed query (`18221`)
- Bug in accessing a `pandas.get_option`, which raised ``KeyError`` rather than ``OptionError`` when looking up a non-existent option key in some cases (`19789`)
- Bug in `testing.assert_series_equal` and `testing.assert_frame_equal` for Series or DataFrames with differing unicode data (`20503`)  .. _whatsnew_0.23.0.contributors:  Contributors
~~~  .. contributors:: v0.22.0..v0.23.0

{{ header }}  This is a minor bug-fix release in the 0.23.x series and includes some small regression fixes
and bug fixes. We recommend that all users upgrade to this version.  .. warning::  Starting January 1, 2019, pandas feature releases will support Python 3 only.  See Dropping Python 2.7 for more.  .. contents:: What's new in v0.23.1  none  .. _whatsnew_0231.fixed_regressions:  Fixed regressions
~~~  **Comparing Series with datetime.date**  We've reverted a 0.23.0 change to comparing a `Series` holding datetimes and a ``datetime.date`` object (`21152`).
In pandas 0.22 and earlier, comparing a Series holding datetimes and ``datetime.date`` objects would coerce the ``datetime.date`` to a datetime before comparing.
This was inconsistent with Python, NumPy, and `DatetimeIndex`, which never consider a datetime and ``datetime.date`` equal.  In 0.23.0, we unified operations between DatetimeIndex and Series, and in the process changed comparisons between a Series of datetimes and ``datetime.date`` without warning.  We've temporarily restored the 0.22.0 behavior, so datetimes and dates may again compare equal, but restore the 0.23.0 behavior in a future release.  To summarize, here's the behavior in 0.22.0, 0.23.0, 0.23.1:  .. code-block:: python  # 0.22.0... Silently coerce the datetime.date  >>> import datetime  >>> pd.Series(pd.date_range('2017', periods=2)) == datetime.date(2017, 1, 1)  0  True  1  False  dtype: bool  # 0.23.0... Do not coerce the datetime.date  >>> pd.Series(pd.date_range('2017', periods=2)) == datetime.date(2017, 1, 1)  0  False  1  False  dtype: bool  # 0.23.1... Coerce the datetime.date with a warning  >>> pd.Series(pd.date_range('2017', periods=2)) == datetime.date(2017, 1, 1)  /bin/python FutureWarning: Comparing Series of datetimes with 'datetime.date'.  Currently, the  'datetime.date' is coerced to a datetime. In the future pandas will  not coerce, and the values not compare equal to the 'datetime.date'.  To retain the current behavior, convert the 'datetime.date' to a  datetime with 'pd.Timestamp'.  #!/bin/python3  0  True  1  False  dtype: bool  In addition, ordering comparisons will raise a ``TypeError`` in the future.  **Other fixes**  - Reverted the ability of `~DataFrame.to_sql` to perform multivalue  inserts as this caused regression in certain cases (`21103`).  In the future this will be made configurable.
- Fixed regression in the `DatetimeIndex.date` and `DatetimeIndex.time`  attributes in case of timezone-aware data: `DatetimeIndex.time` returned  a tz-aware time instead of tz-naive (`21267`) and `DatetimeIndex.date`  returned incorrect date when the input date has a non-UTC timezone (`21230`).
- Fixed regression in `pandas.io.json.json_normalize` when called with ``None`` values  in nested levels in JSON, and to not drop keys with value as ``None`` (`21158`, `21356`).
- Bug in `~DataFrame.to_csv` causes encoding error when compression and encoding are specified (`21241`, `21118`)
- Bug preventing pandas from being importable with -OO optimization (`21071`)
- Bug in `Categorical.fillna` incorrectly raising a ``TypeError`` when ``value`` the individual categories are iterable and ``value`` is an iterable (`21097`, `19788`)
- Fixed regression in constructors coercing NA values like ``None`` to strings when passing ``dtype=str`` (`21083`)
- Regression in `pivot_table` where an ordered ``Categorical`` with missing  values for the pivot's ``index`` would give a mis-aligned result (`21133`)
- Fixed regression in merging on boolean index/columns (`21119`).  .. _whatsnew_0231.performance:  Performance improvements
~~~  - Improved performance of `CategoricalIndex.is_monotonic_increasing`, `CategoricalIndex.is_monotonic_decreasing` and `CategoricalIndex.is_monotonic` (`21025`)
- Improved performance of `CategoricalIndex.is_unique` (`21107`)  .. _whatsnew_0231.bug_fixes:  Bug fixes

~~~  **Groupby/resample/rolling**  - Bug in `DataFrame.agg` where applying multiple aggregation functions to a `DataFrame` with duplicated column names would cause a stack overflow (`21063`)
- Bug in `pandas.core.groupby.GroupBy.ffill` and `pandas.core.groupby.GroupBy.bfill` where the fill within a grouping would not always be applied as intended due to the implementations' use of a non-stable sort (`21207`)
- Bug in `pandas.core.groupby.GroupBy.rank` where results did not scale to 100% when specifying ``method='dense'`` and ``pct=True``
- Bug in `pandas.DataFrame.rolling` and `pandas.Series.rolling` which incorrectly accepted a 0 window size rather than raising (`21286`)  **Data-type specific**  - Bug in `Series.str.replace()` where the method throws ``TypeError`` on Python 3.5.2 (`21078`)
- Bug in `Timedelta` where passing a float with a unit would prematurely round the float precision (`14156`)
- Bug in `pandas.testing.assert_index_equal` which raised ``AssertionError`` incorrectly, when comparing two `CategoricalIndex` objects with param ``check_categorical=False`` (`19776`)  **Sparse**  - Bug in `SparseArray.shape` which previously only returned the shape `SparseArray.sp_values` (`21126`)  **Indexing**  - Bug in `Series.reset_index` where appropriate error was not raised with an invalid level name (`20925`)
- Bug in `interval_range` when ``start``/``periods`` or ``end``/``periods`` are specified with float ``start`` or ``end`` (`21161`)
- Bug in `MultiIndex.set_names` where error raised for a ``MultiIndex`` with ``nlevels == 1`` (`21149`)
- Bug in `IntervalIndex` constructors where creating an ``IntervalIndex`` from categorical data was not fully supported (`21243`, `21253`)
- Bug in `MultiIndex.sort_index` which was not guaranteed to sort correctly with ``level=1``; this was also causing data misalignment in particular `DataFrame.stack` operations (`20994`, `20945`, `21052`)  **Plotting**  - New keywords (sharex, sharey) to turn on/off sharing of x/y-axis by subplots generated with pandas.DataFrame().groupby().boxplot() (`20968`)  **I/O**  - Bug in IO methods specifying ``compression='zip'`` which produced uncompressed zip archives (`17778`, `21144`)
- Bug in `DataFrame.to_stata` which prevented exporting DataFrames to buffers and most file-like objects (`21041`)
- Bug in `read_stata` and `StataReader` which did not correctly decode utf-8 strings on Python 3 from Stata 14 files (dta version 118) (`21244`)
- Bug in IO JSON `read_json` reading empty JSON schema with ``orient='table'`` back to `DataFrame` caused an error (`21287`)  **Reshaping**  - Bug in `concat` where error was raised in concatenating `Series` with numpy scalar and tuple names (`21015`)
- Bug in `concat` warning message providing the wrong guidance for future behavior (`21101`)  **Other**  - Tab completion on `Index` in IPython no longer outputs deprecation warnings (`21125`)
- Bug preventing pandas being used on Windows without C++ redistributable installed (`21106`)  .. _whatsnew_0.23.1.contributors:  Contributors
~~~  .. contributors:: v0.23.0..v0.23.1

and bug fixes. We recommend that all users upgrade to this version.  .. note::  pandas 0.23.2 is first pandas release that's compatible with  Python 3.7 (`20552`)  .. warning::  Starting January 1, 2019, pandas feature releases will support Python 3 only.  See Dropping Python 2.7 for more.  .. contents:: What's new in v0.23.2  none  .. _whatsnew_0232.enhancements:  Logical reductions over entire DataFrame
~~~  `DataFrame.all` and `DataFrame.any` now accept ``axis=None`` to reduce over all axes to a scalar (`19976`)  .. ipython:: python  df = pd.DataFrame({"A": [1, 2], "B": [True, False]})  df.all(axis=None)  This also provides compatibility with NumPy 1.15, which now dispatches to ``DataFrame.all``.
With NumPy 1.15 and pandas 0.23.1 or earlier, `numpy.all` will no longer reduce over every axis:  .. code-block:: python  >>> # NumPy 1.15, pandas 0.23.1  >>> np.any(pd.DataFrame({"A": [False], "B": [False]}))  A  False  B  False  dtype: bool  With pandas 0.23.2, that will correctly return False, as it did with NumPy < 1.15.  .. ipython:: python  np.any(pd.DataFrame({"A": [False], "B": [False]}))  .. _whatsnew_0232.fixed_regressions:  Fixed regressions
~~~  - Fixed regression in `to_csv` when handling file-like object incorrectly (`21471`)
- Re-allowed duplicate level names of a ``MultiIndex``. Accessing a level that has a duplicate name by name still raises an error (`19029`).
- Bug in both `DataFrame.first_valid_index` and `Series.first_valid_index` raised for a row index having duplicate values (`21441`)
- Fixed printing of DataFrames with hierarchical columns with long names (`21180`)
- Fixed regression in `~DataFrame.reindex` and `~DataFrame.groupby`  with a MultiIndex or multiple keys that contains categorical datetime-like values (`21390`).
- Fixed regression in unary negative operations with object dtype (`21380`)
- Bug in `Timestamp.ceil` and `Timestamp.floor` when timestamp is a multiple of the rounding frequency (`21262`)
- Fixed regression in `to_clipboard` that defaulted to copying dataframes with space delimited instead of tab delimited (`21104`)  Build changes
~~~  - The source and binary distributions no longer include test data files, resulting in smaller download sizes. Tests relying on these data files will be skipped when using ``pandas.test()``. (`19320`)  .. _whatsnew_0232.bug_fixes:  Bug fixes
~~~  **Conversion**  - Bug in constructing `Index` with an iterator or generator (`21470`)
- Bug in `Series.nlargest` for signed and unsigned integer dtypes when the minimum value is present (`21426`)  **Indexing**  - Bug in `Index.get_indexer_non_unique` with categorical key (`21448`)
- Bug in comparison operations for `MultiIndex` where error was raised on equality / inequality comparison involving a MultiIndex with ``nlevels == 1`` (`21149`)
- Bug in `DataFrame.drop` behaviour is not consistent for unique and non-unique indexes (`21494`)
- Bug in `DataFrame.duplicated` with a large number of columns causing a 'maximum recursion depth exceeded' (`21524`).  **I/O**  - Bug in `read_csv` that caused it to incorrectly raise an error when ``nrows=0``, ``low_memory=True``, and ``index_col`` was not ``None`` (`21141`)
- Bug in `json_normalize` when formatting the ``record_prefix`` with integer columns (`21536`)  **Categorical**  - Bug in rendering `Series` with ``Categorical`` dtype in rare conditions under Python 2.7 (`21002`)  **Timezones**  - Bug in `Timestamp` and `DatetimeIndex` where passing a `Timestamp` localized after a DST transition would return a datetime before the DST transition (`20854`)
- Bug in comparing `DataFrame` with tz-aware `DatetimeIndex` columns with a DST transition that raised a ``KeyError`` (`19970`)
- Bug in `DatetimeIndex.shift` where an ``AssertionError`` would raise when shifting across DST (`8616`)

- Bug in `Timestamp` constructor where passing an invalid timezone offset designator (``Z``) would not raise a ``ValueError`` (`8910`)
- Bug in `Timestamp.replace` where replacing at a DST boundary would retain an incorrect offset (`7825`)
- Bug in `DatetimeIndex.reindex` when reindexing a tz-naive and tz-aware `DatetimeIndex` (`8306`)
- Bug in `DatetimeIndex.resample` when downsampling across a DST boundary (`8531`)  **Timedelta**  - Bug in `Timedelta` where non-zero timedeltas shorter than 1 microsecond were considered False (`21484`)  .. _whatsnew_0.23.2.contributors:  Contributors
~~~  .. contributors:: v0.23.1..v0.23.2

What's new in 0.23.3 (July 7, 2018)

{{ header }}  This release fixes a build issue with the sdist for Python 3.7 (`21785`)
There are no other changes.  .. _whatsnew_0.23.3.contributors:  Contributors
~~~  .. contributors:: v0.23.2..v0.23.3

What's new in 0.23.4 (August 3, 2018)

and bug fixes. We recommend that all users upgrade to this version.  .. warning::  Starting January 1, 2019, pandas feature releases will support Python 3 only.  See Dropping Python 2.7 for more.  .. contents:: What's new in v0.23.4  none  .. _whatsnew_0234.fixed_regressions:  Fixed regressions
~~~  - Python 3.7 with Windows gave all missing values for rolling variance calculations (`21813`)  .. _whatsnew_0234.bug_fixes:  Bug fixes
~~~  **Groupby/resample/rolling**  - Bug where calling `DataFrameGroupBy.agg` with a list of functions including ``ohlc`` as the non-initial element would raise a ``ValueError`` (`21716`)
- Bug in ``roll_quantile`` caused a memory leak when calling ``.rolling(...).quantile(q)`` with ``q`` in (0,1) (`21965`)  **Missing**  - Bug in `Series.clip` and `DataFrame.clip` cannot accept list-like threshold containing ``NaN`` (`19992`)  .. _whatsnew_0.23.4.contributors:  Contributors
~~~  .. contributors:: v0.23.3..v0.23.4

.. warning::  The 0.24.x series of releases will be the last to support Python 2. Future feature  releases will support Python 3 only. See Dropping Python 2.7 for more  details.  {{ header }}  This is a major release from 0.23.4 and includes a number of API changes, new
features, enhancements, and performance improvements along with a large number
of bug fixes.  Highlights include:  * `Optional Integer NA Support <whatsnew_0240.enhancements.intna>`
* `New APIs for accessing the array backing a Series or Index <whatsnew_0240.values_api>`
* `A new top-level method for creating arrays <whatsnew_0240.enhancements.array>`
* `Store Interval and Period data in a Series or DataFrame <whatsnew_0240.enhancements.interval>`
* `Support for joining on two MultiIndexes <whatsnew_0240.enhancements.join_with_two_multiindexes>`  Check the `API Changes <whatsnew_0240.api_breaking>` and `deprecations <whatsnew_0240.deprecations>` before updating.  These are the changes in pandas 0.24.0. See `release` for a full changelog
including other versions of pandas.  Enhancements
~~~  .. _whatsnew_0240.enhancements.intna:  Optional integer NA support
^^^  pandas has gained the ability to hold integer dtypes with missing values. This long requested feature is enabled through the use of `extension types <extending.extension-types>`.  .. note::  IntegerArray is currently experimental. Its API or implementation may  change without warning.  We can construct a ``Series`` with the specified dtype. The dtype string ``Int64`` is a pandas ``ExtensionDtype``. Specifying a list or array using the traditional missing value
marker of ``np.nan`` will infer to integer dtype. The display of the ``Series`` will also use the ``NaN`` to indicate missing values in string outputs. (`20700`, `20747`, `22441`, `21789`, `22346`)  .. ipython:: python  s = pd.Series([1, 2, np.nan], dtype='Int64')  s  Operations on these dtypes will propagate ``NaN`` as other pandas operations.  .. ipython:: python  # arithmetic  s + 1  # comparison  s == 1  # indexing  s.iloc[1:3]  # operate with other dtypes  s + s.iloc[1:3].astype('Int8')  # coerce when needed  s + 0.01  These dtypes can operate as part of a ``DataFrame``.  .. ipython:: python  df = pd.DataFrame({'A': s, 'B': [1, 1, 3], 'C': list('aab')})  df  df.dtypes  These dtypes can be merged, reshaped, and casted.  .. ipython:: python  pd.concat([df[['A']], df[['B', 'C']]], axis=1).dtypes  df['A'].astype(float)  Reduction and groupby operations such as ``sum`` work.  .. ipython:: python  df.sum()  df.groupby('B').A.sum()  .. warning::  The Integer NA support currently uses the capitalized dtype version, e.g. ``Int8`` as compared to the traditional ``int8``. This may be changed at a future date.  See `integer_na` for more.  .. _whatsnew_0240.values_api:  Accessing the values in a Series or Index
^^^  `Series.array` and `Index.array` have been added for extracting the array backing a
``Series`` or ``Index``. (`19954`, `23623`)  .. ipython:: python  idx = pd.period_range('2000', periods=4)  idx.array  pd.Series(idx).array  Historically, this would have been done with ``series.values``, but with
``.values`` it was unclear whether the returned value would be the actual array,
some transformation of it, or one of pandas custom arrays (like
``Categorical``). For example, with `PeriodIndex`, ``.values`` generates
a new ndarray of period objects each time.  .. ipython:: python  idx.values  id(idx.values)  id(idx.values)  If you need an actual NumPy array, use `Series.to_numpy` or `Index.to_numpy`.  .. ipython:: python  idx.to_numpy()  pd.Series(idx).to_numpy()  For Series and Indexes backed by normal NumPy arrays, `Series.array` will return a
new `arrays.PandasArray`, which is a thin (no-copy) wrapper around a
`numpy.ndarray`. `~arrays.PandasArray` isn't especially useful on its own,
but it does provide the same interface as any extension array defined in pandas or by

a third-party library.  .. ipython:: python  ser = pd.Series([1, 2, 3])  ser.array  ser.to_numpy()  We haven't removed or deprecated `Series.values` or `DataFrame.values`, but we
highly recommend and using ``.array`` or ``.to_numpy()`` instead.  See `Dtypes <basics.dtypes>` and `Attributes and Underlying Data <basics.attrs>` for more.  .. _whatsnew_0240.enhancements.array:  ``pandas.array``: a new top-level method for creating arrays
^^^  A new top-level method `array` has been added for creating 1-dimensional arrays (`22860`).
This can be used to create any `extension array <extending.extension-types>`, including
extension arrays registered by `3rd party libraries <ecosystem.extensions>`.
See the `dtypes docs <basics.dtypes>` for more on extension arrays.  .. ipython:: python  pd.array([1, 2, np.nan], dtype='Int64')  pd.array(['a', 'b', 'c'], dtype='category')  Passing data for which there isn't dedicated extension type (e.g. float, integer, etc.)
will return a new `arrays.PandasArray`, which is just a thin (no-copy)
wrapper around a `numpy.ndarray` that satisfies the pandas extension array interface.  .. ipython:: python  pd.array([1, 2, 3])  On their own, a `~arrays.PandasArray` isn't a very useful object.
But if you need write low-level code that works generically for any
`~pandas.api.extensions.ExtensionArray`, `~arrays.PandasArray`
satisfies that need.  Notice that by default, if no ``dtype`` is specified, the dtype of the returned
array is inferred from the data. In particular, note that the first example of
``[1, 2, np.nan]`` would have returned a floating-point array, since ``NaN``
is a float.  .. ipython:: python  pd.array([1, 2, np.nan])  .. _whatsnew_0240.enhancements.interval:  Storing Interval and Period data in Series and DataFrame
^^^  `Interval` and `Period` data may now be stored in a `Series` or `DataFrame`, in addition to an
`IntervalIndex` and `PeriodIndex` like previously (`19453`, `22862`).  .. ipython:: python  ser = pd.Series(pd.interval_range(0, 5))  ser  ser.dtype  For periods:  .. ipython:: python  pser = pd.Series(pd.period_range("2000", freq="D", periods=5))  pser  pser.dtype  Previously, these would be cast to a NumPy array with object dtype. In general,
this should result in better performance when storing an array of intervals or periods
in a `Series` or column of a `DataFrame`.  Use `Series.array` to extract the underlying array of intervals or periods
from the ``Series``:  .. ipython:: python  ser.array  pser.array  These return an instance of `arrays.IntervalArray` or `arrays.PeriodArray`,
the new extension arrays that back interval and period data.  .. warning::  For backwards compatibility, `Series.values` continues to return  a NumPy array of objects for Interval and Period data. We recommend  using `Series.array` when you need the array of data stored in the  ``Series``, and `Series.to_numpy` when you know you need a NumPy array.  See `Dtypes <basics.dtypes>` and `Attributes and Underlying Data <basics.attrs>`  for more.  .. _whatsnew_0240.enhancements.join_with_two_multiindexes:  Joining with two multi-indexes
^^^  `DataFrame.merge` and `DataFrame.join` can now be used to join multi-indexed ``Dataframe`` instances on the overlapping index levels (`6360`)  See the `Merge, join, and concatenate

<merging.Join_with_two_multi_indexes>` documentation section.  .. ipython:: python  index_left = pd.MultiIndex.from_tuples([('K0', 'X0'), ('K0', 'X1'),  ('K1', 'X2')],  names=['key', 'X'])  left = pd.DataFrame({'A': ['A0', 'A1', 'A2'],  'B': ['B0', 'B1', 'B2']}, index=index_left)  index_right = pd.MultiIndex.from_tuples([('K0', 'Y0'), ('K1', 'Y1'),  ('K2', 'Y2'), ('K2', 'Y3')],  names=['key', 'Y'])  right = pd.DataFrame({'C': ['C0', 'C1', 'C2', 'C3'],  'D': ['D0', 'D1', 'D2', 'D3']}, index=index_right)  left.join(right)  For earlier versions this can be done using the following.  .. ipython:: python  pd.merge(left.reset_index(), right.reset_index(),  on=['key'], how='inner').set_index(['key', 'X', 'Y'])  .. _whatsnew_0240.enhancements.read_html:  Function ``read_html`` enhancements
^^^  `read_html` previously ignored ``colspan`` and ``rowspan`` attributes.
Now it understands them, treating them as sequences of cells with the same
value. (`17054`)  .. ipython:: python  result = pd.read_html("""  <table>  <thead>  <tr>  <th>A</th><th>B</th><th>C</th>  </tr>  </thead>  <tbody>  <tr>  <td colspan="2">1</td><td>2</td>  </tr>  </tbody>  </table>""")  *Previous behavior*:  .. code-block:: ipython  In [13]: result  Out [13]:  [  A  B  C  0  1  2 NaN]  *New behavior*:  .. ipython:: python  result  New ``Styler.pipe()`` method
^^^
The `~pandas.io.formats.style.Styler` class has gained a
`~pandas.io.formats.style.Styler.pipe` method.  This provides a
convenient way to apply users' predefined styling functions, and can help reduce
"boilerplate" when using DataFrame styling functionality repeatedly within a notebook. (`23229`)  .. ipython:: python  df = pd.DataFrame({'N': [1250, 1500, 1750], 'X': [0.25, 0.35, 0.50]})  def format_and_align(styler):  return (styler.format({'N': '{:,}', 'X': '{:.1%}'})  .set_properties(**{'text-align': 'right'}))  df.style.pipe(format_and_align).set_caption('Summary of results.')  Similar methods already exist for other classes in pandas, including `DataFrame.pipe`,
`GroupBy.pipe() <pandas.core.groupby.GroupBy.pipe>`, and `Resampler.pipe() <pandas.core.resample.Resampler.pipe>`.  .. _whatsnew_0240.enhancements.rename_axis:  Renaming names in a MultiIndex
^^^  `DataFrame.rename_axis` now supports ``index`` and ``columns`` arguments
and `Series.rename_axis` supports ``index`` argument (`19978`).  This change allows a dictionary to be passed so that some of the names
of a ``MultiIndex`` can be changed.  Example:  .. ipython:: python  mi = pd.MultiIndex.from_product([list('AB'), list('CD'), list('EF')],  names=['AB', 'CD', 'EF'])  df = pd.DataFrame(list(range(len(mi))), index=mi, columns=['N'])  df  df.rename_axis(index={'CD': 'New'})  See the `Advanced documentation on renaming<advanced.index_names>` for more details.  .. _whatsnew_0240.enhancements.other:  Other enhancements
^^^  - `merge` now directly allows merge between objects of type ``DataFrame`` and named ``Series``, without the need to convert the ``Series`` object into a ``DataFrame`` beforehand (`21220`)
- ``ExcelWriter`` now accepts ``mode`` as a keyword argument, enabling append to existing workbooks when using the ``openpyxl`` engine (`3441`)
- ``FrozenList`` has gained the ``.union()`` and ``.difference()`` methods. This functionality greatly simplifies groupby's that rely on explicitly excluding certain columns. See `Splitting an object into groups <groupby.split>` for more information (`15475`, `15506`).
- `DataFrame.to_parquet` now accepts ``index`` as an argument, allowing  the user to override the engine's default behavior to include or omit the  dataframe's indexes from the resulting Parquet file. (`20768`)
- `read_feather` now accepts ``columns`` as an argument, allowing the user to specify which columns should be read. (`24025`)

- `DataFrame.corr` and `Series.corr` now accept a callable for generic calculation methods of correlation, e.g. histogram intersection (`22684`)
- `DataFrame.to_string` now accepts ``decimal`` as an argument, allowing the user to specify which decimal separator should be used in the output. (`23614`)
- `DataFrame.to_html` now accepts ``render_links`` as an argument, allowing the user to generate HTML with links to any URLs that appear in the DataFrame.  See the `section on writing HTML <io.html>` in the IO docs for example usage. (`2679`)
- `pandas.read_csv` now supports pandas extension types as an argument to ``dtype``, allowing the user to use pandas extension types when reading CSVs. (`23228`)
- The `~DataFrame.shift` method now accepts ``fill_value`` as an argument, allowing the user to specify a value which will be used instead of NA/NaT in the empty periods. (`15486`)
- `to_datetime` now supports the ``%Z`` and ``%z`` directive when passed into ``format`` (`13486`)
- `Series.mode` and `DataFrame.mode` now support the ``dropna`` parameter which can be used to specify whether ``NaN``/``NaT`` values should be considered (`17534`)
- `DataFrame.to_csv` and `Series.to_csv` now support the ``compression`` keyword when a file handle is passed. (`21227`)
- `Index.droplevel` is now implemented also for flat indexes, for compatibility with `MultiIndex` (`21115`)
- `Series.droplevel` and `DataFrame.droplevel` are now implemented (`20342`)
- Added support for reading from/writing to Google Cloud Storage via the ``gcsfs`` library (`19454`, `23094`)
- `DataFrame.to_gbq` and `read_gbq` signature and documentation updated to  reflect changes from the `pandas-gbq library version 0.8.0  <https://pandas-gbq.readthedocs.io/en/latest/changelog.html#changelog-0-8-0>`__.  Adds a ``credentials`` argument, which enables the use of any kind of  `google-auth credentials  <https://google-auth.readthedocs.io/en/latest/>`__. (`21627`,  `22557`, `23662`)
- New method `HDFStore.walk` will recursively walk the group hierarchy of an HDF5 file (`10932`)
- `read_html` copies cell data across ``colspan`` and ``rowspan``, and it treats all-``th`` table rows as headers if ``header`` kwarg is not given and there is no ``thead`` (`17054`)
- `Series.nlargest`, `Series.nsmallest`, `DataFrame.nlargest`, and `DataFrame.nsmallest` now accept the value ``"all"`` for the ``keep`` argument. This keeps all ties for the nth largest/smallest value (`16818`)
- `IntervalIndex` has gained the `~IntervalIndex.set_closed` method to change the existing ``closed`` value (`21670`)
- `~DataFrame.to_csv`, `~Series.to_csv`, `~DataFrame.to_json`, and `~Series.to_json` now support ``compression='infer'`` to infer compression based on filename extension (`15008`).  The default compression for ``to_csv``, ``to_json``, and ``to_pickle`` methods has been updated to ``'infer'`` (`22004`).
- `DataFrame.to_sql` now supports writing ``TIMESTAMP WITH TIME ZONE`` types for supported databases. For databases that don't support timezones, datetime data will be stored as timezone unaware local timestamps. See the `io.sql_datetime_data` for implications (`9086`).
- `to_timedelta` now supports iso-formatted timedelta strings (`21877`)
- `Series` and `DataFrame` now support `Iterable` objects in the constructor (`2193`)
- `DatetimeIndex` has gained the `DatetimeIndex.timetz` attribute. This returns the local time with timezone information. (`21358`)
- `~Timestamp.round`, `~Timestamp.ceil`, and `~Timestamp.floor` for `DatetimeIndex` and `Timestamp`  now support an ``ambiguous`` argument for handling datetimes that are rounded to ambiguous times (`18946`)  and a ``nonexistent`` argument for handling datetimes that are rounded to nonexistent times. See `timeseries.timezone_nonexistent` (`22647`)
- The result of `~DataFrame.resample` is now iterable similar to ``groupby()`` (`15314`).

- `Series.resample` and `DataFrame.resample` have gained the `pandas.core.resample.Resampler.quantile` (`15023`).
- `DataFrame.resample` and `Series.resample` with a `PeriodIndex` will now respect the ``base`` argument in the same fashion as with a `DatetimeIndex`. (`23882`)
- `pandas.api.types.is_list_like` has gained a keyword ``allow_sets`` which is ``True`` by default; if ``False``,  all instances of ``set`` will not be considered "list-like" anymore (`23061`)
- `Index.to_frame` now supports overriding column name(s) (`22580`).
- `Categorical.from_codes` now can take a ``dtype`` parameter as an alternative to passing ``categories`` and ``ordered`` (`24398`).
- New attribute ``__git_version__`` will return git commit sha of current build (`21295`).
- Compatibility with Matplotlib 3.0 (`22790`).
- Added `Interval.overlaps`, `arrays.IntervalArray.overlaps`, and `IntervalIndex.overlaps` for determining overlaps between interval-like objects (`21998`)
- `read_fwf` now accepts keyword ``infer_nrows`` (`15138`).
- `~DataFrame.to_parquet` now supports writing a ``DataFrame`` as a directory of parquet files partitioned by a subset of the columns when ``engine = 'pyarrow'`` (`23283`)
- `Timestamp.tz_localize`, `DatetimeIndex.tz_localize`, and `Series.tz_localize` have gained the ``nonexistent`` argument for alternative handling of nonexistent times. See `timeseries.timezone_nonexistent` (`8917`, `24466`)
- `Index.difference`, `Index.intersection`, `Index.union`, and `Index.symmetric_difference` now have an optional ``sort`` parameter to control whether the results should be sorted if possible (`17839`, `24471`)
- `read_excel()` now accepts ``usecols`` as a list of column names or callable (`18273`)
- `MultiIndex.to_flat_index` has been added to flatten multiple levels into a single-level `Index` object.
- `DataFrame.to_stata` and `pandas.io.stata.StataWriter117` can write mixed string columns to Stata strl format (`23633`)
- `DataFrame.between_time` and `DataFrame.at_time` have gained the ``axis`` parameter (`8839`)
- `DataFrame.to_records` now accepts ``index_dtypes`` and ``column_dtypes`` parameters to allow different data types in stored column and index records (`18146`)
- `IntervalIndex` has gained the `~IntervalIndex.is_overlapping` attribute to indicate if the ``IntervalIndex`` contains any overlapping intervals (`23309`)
- `pandas.DataFrame.to_sql` has gained the ``method`` argument to control SQL insertion clause. See the `insertion method <io.sql.method>` section in the documentation. (`8953`)
- `DataFrame.corrwith` now supports Spearman's rank correlation, Kendall's tau as well as callable correlation methods. (`21925`)
- `DataFrame.to_json`, `DataFrame.to_csv`, `DataFrame.to_pickle`, and other export methods now support tilde(~) in path argument. (`23473`)  .. _whatsnew_0240.api_breaking:  Backwards incompatible API changes
~~~  pandas 0.24.0 includes a number of API breaking changes.  .. _whatsnew_0240.api_breaking.deps:  Increased minimum versions for dependencies
^^^  We have updated our minimum supported versions of dependencies (`21242`, `18742`, `23774`, `24767`).
If installed, we now require:  +---+---+---+
| Package  | Minimum Version | Required |
+===+===+===+
| numpy  | 1.12.0  |  X  |
| bottleneck  | 1.2.0  |  |
| fastparquet  | 0.2.1  |  |
| matplotlib  | 2.0.0  |  |
| numexpr  | 2.6.1  |  |
| pandas-gbq  | 0.8.0  |  |
| pyarrow  | 0.9.0  |  |
| pytables  | 3.4.2  |  |
| scipy  | 0.18.1  |  |
| xlrd  | 1.0.0  |  |
| pytest (dev)  | 3.6  |  |
+---+---+---+  Additionally we no longer depend on ``feather-format`` for feather based storage
and replaced it with references to ``pyarrow`` (`21639` and `23053`).  .. _whatsnew_0240.api_breaking.csv_line_terminator:  ``os.linesep`` is used for ``line_terminator`` of ``DataFrame.to_csv``

^^^  `DataFrame.to_csv` now uses `os.linesep` rather than ``'\n'``
for the default line terminator (`20353`).
This change only affects when running on Windows, where ``'\r\n'`` was used for line terminator
even when ``'\n'`` was passed in ``line_terminator``.  *Previous behavior* on Windows:  .. code-block:: ipython  In [1]: data = pd.DataFrame({"string_with_lf": ["a\nbc"],  ...:  "string_with_crlf": ["a\r\nbc"]})  In [2]: # When passing file PATH to to_csv,  ...: # line_terminator does not work, and csv is saved with '\r\n'.  ...: # Also, this converts all '\n's in the data to '\r\n'.  ...: data.to_csv("test.csv", index=False, line_terminator='\n')  In [3]: with open("test.csv", mode='rb') as f:  ...:  print(f.read())  Out[3]: b'string_with_lf,string_with_crlf\r\n"a\r\nbc","a\r\r\nbc"\r\n'  In [4]: # When passing file OBJECT with newline option to  ...: # to_csv, line_terminator works.  ...: with open("test2.csv", mode='w', newline='\n') as f:  ...:  data.to_csv(f, index=False, line_terminator='\n')  In [5]: with open("test2.csv", mode='rb') as f:  ...:  print(f.read())  Out[5]: b'string_with_lf,string_with_crlf\n"a\nbc","a\r\nbc"\n'  *New behavior* on Windows:  Passing ``line_terminator`` explicitly, set the ``line terminator`` to that character.  .. code-block:: ipython  In [1]: data = pd.DataFrame({"string_with_lf": ["a\nbc"],  ...:  "string_with_crlf": ["a\r\nbc"]})  In [2]: data.to_csv("test.csv", index=False, line_terminator='\n')  In [3]: with open("test.csv", mode='rb') as f:  ...:  print(f.read())  Out[3]: b'string_with_lf,string_with_crlf\n"a\nbc","a\r\nbc"\n'  On Windows, the value of ``os.linesep`` is ``'\r\n'``, so if ``line_terminator`` is not
set, ``'\r\n'`` is used for line terminator.  .. code-block:: ipython  In [1]: data = pd.DataFrame({"string_with_lf": ["a\nbc"],  ...:  "string_with_crlf": ["a\r\nbc"]})  In [2]: data.to_csv("test.csv", index=False)  In [3]: with open("test.csv", mode='rb') as f:  ...:  print(f.read())  Out[3]: b'string_with_lf,string_with_crlf\r\n"a\nbc","a\r\nbc"\r\n'  For file objects, specifying ``newline`` is not sufficient to set the line terminator.
You must pass in the ``line_terminator`` explicitly, even in this case.  .. code-block:: ipython  In [1]: data = pd.DataFrame({"string_with_lf": ["a\nbc"],  ...:  "string_with_crlf": ["a\r\nbc"]})  In [2]: with open("test2.csv", mode='w', newline='\n') as f:  ...:  data.to_csv(f, index=False)  In [3]: with open("test2.csv", mode='rb') as f:  ...:  print(f.read())  Out[3]: b'string_with_lf,string_with_crlf\r\n"a\nbc","a\r\nbc"\r\n'  .. _whatsnew_0240.bug_fixes.nan_with_str_dtype:  Proper handling of ``np.NaN`` in a string data-typed column with the Python engine
^^^  There was bug in `read_excel` and `read_csv` with the Python
engine, where missing values turned to ``'nan'`` with ``dtype=str`` and
``na_filter=True``. Now, these missing values are converted to the string
missing indicator, ``np.nan``. (`20377`)  .. ipython:: python  from io import StringIO  *Previous behavior*:  .. code-block:: ipython  In [5]: data = 'a,b,c\n1,,3\n4,5,6'  In [6]: df = pd.read_csv(StringIO(data), engine='python', dtype=str, na_filter=True)  In [7]: df.loc[0, 'b']  Out[7]:  'nan'  *New behavior*:  .. ipython:: python  data = 'a,b,c\n1,,3\n4,5,6'  df = pd.read_csv(StringIO(data), engine='python', dtype=str, na_filter=True)  df.loc[0, 'b']  Notice how we now instead output ``np.nan`` itself instead of a stringified form of it.  .. _whatsnew_0240.api.timezone_offset_parsing:  Parsing datetime strings with timezone offsets
^^^  Previously, parsing datetime strings with UTC offsets with `to_datetime`
or `DatetimeIndex` would automatically convert the datetime to UTC
without timezone localization. This is inconsistent from parsing the same

datetime string with `Timestamp` which would preserve the UTC
offset in the ``tz`` attribute. Now, `to_datetime` preserves the UTC
offset in the ``tz`` attribute when all the datetime strings have the same
UTC offset (`17697`, `11736`, `22457`)  *Previous behavior*:  .. code-block:: ipython  In [2]: pd.to_datetime("2015-11-18 1500+05:30")  Out[2]: Timestamp('2015-11-18 1000')  In [3]: pd.Timestamp("2015-11-18 1500+05:30")  Out[3]: Timestamp('2015-11-18 1500+0530', tz='pytz.FixedOffset(330)')  # Different UTC offsets would automatically convert the datetimes to UTC (without a UTC timezone)  In [4]: pd.to_datetime(["2015-11-18 1500+05:30", "2015-11-18 1600+06:30"])  Out[4]: DatetimeIndex(['2015-11-18 1000', '2015-11-18 1000'], dtype='datetime64[ns]', freq=None)  *New behavior*:  .. ipython:: python  pd.to_datetime("2015-11-18 1500+05:30")  pd.Timestamp("2015-11-18 1500+05:30")  Parsing datetime strings with the same UTC offset will preserve the UTC offset in the ``tz``  .. ipython:: python  pd.to_datetime(["2015-11-18 1500+05:30"] * 2)  Parsing datetime strings with different UTC offsets will now create an Index of
``datetime.datetime`` objects with different UTC offsets  .. ipython:: python  idx = pd.to_datetime(["2015-11-18 1500+05:30",  "2015-11-18 1600+06:30"])  idx  idx[0]  idx[1]  Passing ``utc=True`` will mimic the previous behavior but will correctly indicate
that the dates have been converted to UTC  .. ipython:: python  pd.to_datetime(["2015-11-18 1500+05:30",  "2015-11-18 1600+06:30"], utc=True)  .. _whatsnew_0240.api_breaking.read_csv_mixed_tz:  Parsing mixed-timezones with `read_csv`
^^^  `read_csv` no longer silently converts mixed-timezone columns to UTC (`24987`).  *Previous behavior*  .. code-block:: python  >>> import io  >>> content = """\  ... a  ... 2000-01-01T000+05:00  ... 2000-01-01T000+06:00"""  >>> df = pd.read_csv(io.StringIO(content), parse_dates=['a'])  >>> df.a  0  1999-12-31 1900  1  1999-12-31 1800  Name: a, dtype: datetime64[ns]  *New behavior*  .. ipython:: python  import io  content = """\  a  2000-01-01T000+05:00  2000-01-01T000+06:00"""  df = pd.read_csv(io.StringIO(content), parse_dates=['a'])  df.a  As can be seen, the ``dtype`` is object; each value in the column is a string.
To convert the strings to an array of datetimes, the ``date_parser`` argument  .. ipython:: python  df = pd.read_csv(io.StringIO(content), parse_dates=['a'],  date_parser=lambda col: pd.to_datetime(col, utc=True))  df.a  See `whatsnew_0240.api.timezone_offset_parsing` for more.  .. _whatsnew_0240.api_breaking.period_end_time:  Time values in ``dt.end_time`` and ``to_timestamp(how='end')``
^^^  The time values in `Period` and `PeriodIndex` objects are now set
to '2359.999' when calling `Series.dt.end_time`, `Period.end_time`,
`PeriodIndex.end_time`, `Period.to_timestamp()` with ``how='end'``,
or `PeriodIndex.to_timestamp()` with ``how='end'`` (`17157`)  *Previous behavior*:  .. code-block:: ipython  In [2]: p = pd.Period('2017-01-01', 'D')  In [3]: pi = pd.PeriodIndex([p])  In [4]: pd.Series(pi).dt.end_time[0]  Out[4]: Timestamp(2017-01-01 000)  In [5]: p.end_time  Out[5]: Timestamp(2017-01-01 2359.999)  *New behavior*:  Calling `Series.dt.end_time` will now result in a time of '2359.999' as
is the case with `Period.end_time`, for example  .. ipython:: python  p = pd.Period('2017-01-01', 'D')  pi = pd.PeriodIndex([p])  pd.Series(pi).dt.end_time[0]  p.end_time  .. _whatsnew_0240.api_breaking.datetime_unique:  Series.unique for timezone-aware data
^^^  The return type of `Series.unique` for datetime with timezone values has changed

from an `numpy.ndarray` of `Timestamp` objects to a `arrays.DatetimeArray` (`24024`).  .. ipython:: python  ser = pd.Series([pd.Timestamp('2000', tz='UTC'),  pd.Timestamp('2000', tz='UTC')])  *Previous behavior*:  .. code-block:: ipython  In [3]: ser.unique()  Out[3]: array([Timestamp('2000-01-01 000+000', tz='UTC')], dtype=object)  *New behavior*:  .. ipython:: python  ser.unique()  .. _whatsnew_0240.api_breaking.sparse_values:  Sparse data structure refactor
^^^  ``SparseArray``, the array backing ``SparseSeries`` and the columns in a ``SparseDataFrame``,
is now an extension array (`21978`, `19056`, `22835`).
To conform to this interface and for consistency with the rest of pandas, some API breaking
changes were made:  - ``SparseArray`` is no longer a subclass of `numpy.ndarray`. To convert a ``SparseArray`` to a NumPy array, use `numpy.asarray`.
- ``SparseArray.dtype`` and ``SparseSeries.dtype`` are now instances of `SparseDtype`, rather than ``np.dtype``. Access the underlying dtype with ``SparseDtype.subtype``.
- ``numpy.asarray(sparse_array)`` now returns a dense array with all the values, not just the non-fill-value values (`14167`)
- ``SparseArray.take`` now matches the API of `pandas.api.extensions.ExtensionArray.take` (`19506`):  * The default value of ``allow_fill`` has changed from ``False`` to ``True``.  * The ``out`` and ``mode`` parameters are now longer accepted (previously, this raised if they were specified).  * Passing a scalar for ``indices`` is no longer allowed.  - The result of `concat` with a mix of sparse and dense Series is a Series with sparse values, rather than a ``SparseSeries``.
- ``SparseDataFrame.combine`` and ``DataFrame.combine_first`` no longer supports combining a sparse column with a dense column while preserving the sparse subtype. The result will be an object-dtype SparseArray.
- Setting `SparseArray.fill_value` to a fill value with a different dtype is now allowed.
- ``DataFrame[column]`` is now a `Series` with sparse values, rather than a `SparseSeries`, when slicing a single column with sparse values (`23559`).
- The result of `Series.where` is now a ``Series`` with sparse values, like with other extension arrays (`24077`)  Some new warnings are issued for operations that require or are likely to materialize a large dense array:  - A `errors.PerformanceWarning` is issued when using fillna with a ``method``, as a dense array is constructed to create the filled array. Filling with a ``value`` is the efficient way to fill a sparse array.
- A `errors.PerformanceWarning` is now issued when concatenating sparse Series with differing fill values. The fill value from the first sparse array continues to be used.  In addition to these API breaking changes, many `Performance Improvements and Bug Fixes have been made <whatsnew_0240.bug_fixes.sparse>`.  Finally, a ``Series.sparse`` accessor was added to provide sparse-specific methods like `Series.sparse.from_coo`.  .. ipython:: python  s = pd.Series([0, 0, 1, 1, 1], dtype='Sparse[int]')  s.sparse.density  .. _whatsnew_0240.api_breaking.get_dummies:  `get_dummies` always returns a DataFrame
^^^  Previously, when ``sparse=True`` was passed to `get_dummies`, the return value could be either
a `DataFrame` or a `SparseDataFrame`, depending on whether all or a just a subset
of the columns were dummy-encoded. Now, a `DataFrame` is always returned (`24284`).  *Previous behavior*  The first `get_dummies` returns a `DataFrame` because the column ``A``
is not dummy encoded. When just ``["B", "C"]`` are passed to ``get_dummies``,

then all the columns are dummy-encoded, and a `SparseDataFrame` was returned.  .. code-block:: ipython  In [2]: df = pd.DataFrame({"A": [1, 2], "B": ['a', 'b'], "C": ['a', 'a']})  In [3]: type(pd.get_dummies(df, sparse=True))  Out[3]: pandas.core.frame.DataFrame  In [4]: type(pd.get_dummies(df[['B', 'C']], sparse=True))  Out[4]: pandas.core.sparse.frame.SparseDataFrame  .. ipython:: python  df = pd.DataFrame({"A": [1, 2], "B": ['a', 'b'], "C": ['a', 'a']})  *New behavior*  Now, the return type is consistently a `DataFrame`.  .. ipython:: python  type(pd.get_dummies(df, sparse=True))  type(pd.get_dummies(df[['B', 'C']], sparse=True))  .. note::  There's no difference in memory usage between a `SparseDataFrame`  and a `DataFrame` with sparse values. The memory usage will  be the same as in the previous version of pandas.  .. _whatsnew_0240.api_breaking.frame_to_dict_index_orient:  Raise ValueError in ``DataFrame.to_dict(orient='index')``
^^^  Bug in `DataFrame.to_dict` raises ``ValueError`` when used with
``orient='index'`` and a non-unique index instead of losing data (`22801`)  .. ipython:: python  df = pd.DataFrame({'a': [1, 2], 'b': [0.5, 0.75]}, index=['A', 'A'])  df  df.to_dict(orient='index')  .. _whatsnew_0240.api.datetimelike.normalize:  Tick DateOffset normalize restrictions
^^^  Creating a ``Tick`` object (`Day`, `Hour`, `Minute`,
`Second`, `Milli`, `Micro`, `Nano`) with
``normalize=True`` is no longer supported.  This prevents unexpected behavior
where addition could fail to be monotone or associative.  (`21427`)  *Previous behavior*:  .. code-block:: ipython  In [2]: ts = pd.Timestamp('2018-06-11 1814')  In [3]: ts  Out[3]: Timestamp('2018-06-11 1814')  In [4]: tic = pd.offsets.Hour(n=2, normalize=True)  ...:  In [5]: tic  Out[5]: <2 * Hours>  In [6]: ts + tic  Out[6]: Timestamp('2018-06-11 000')  In [7]: ts + tic + tic + tic == ts + (tic + tic + tic)  Out[7]: False  *New behavior*:  .. ipython:: python  ts = pd.Timestamp('2018-06-11 1814')  tic = pd.offsets.Hour(n=2)  ts + tic + tic + tic == ts + (tic + tic + tic)  .. _whatsnew_0240.api.datetimelike:  .. _whatsnew_0240.api.period_subtraction:  Period subtraction
^^^  Subtraction of a ``Period`` from another ``Period`` will give a ``DateOffset``.
instead of an integer (`21314`)  *Previous behavior*:  .. code-block:: ipython  In [2]: june = pd.Period('June 2018')  In [3]: april = pd.Period('April 2018')  In [4]: june - april  Out [4]: 2  *New behavior*:  .. ipython:: python  june = pd.Period('June 2018')  april = pd.Period('April 2018')  june - april  Similarly, subtraction of a ``Period`` from a ``PeriodIndex`` will now return
an ``Index`` of ``DateOffset`` objects instead of an ``Int64Index``  *Previous behavior*:  .. code-block:: ipython  In [2]: pi = pd.period_range('June 2018', freq='M', periods=3)  In [3]: pi - pi[0]  Out[3]: Int64Index([0, 1, 2], dtype='int64')  *New behavior*:  .. ipython:: python  pi = pd.period_range('June 2018', freq='M', periods=3)  pi - pi[0]  .. _whatsnew_0240.api.timedelta64_subtract_nan:  Addition/subtraction of ``NaN`` from `DataFrame`
^^^  Adding or subtracting ``NaN`` from a `DataFrame` column with
``timedelta64[ns]`` dtype will now raise a ``TypeError`` instead of returning
all-``NaT``.  This is for compatibility with ``TimedeltaIndex`` and
``Series`` behavior (`22163`)  .. ipython:: python  df = pd.DataFrame([pd.Timedelta(days=1)])  df  *Previous behavior*:  .. code-block:: ipython  In [4]: df = pd.DataFrame([pd.Timedelta(days=1)])  In [5]: df - np.nan  Out[5]:  0  0 NaT  *New behavior*:  .. code-block:: ipython  In [2]: df - np.nan  ...  TypeError: unsupported operand type(s) for -: 'TimedeltaIndex' and 'float'  .. _whatsnew_0240.api.dataframe_cmp_broadcasting:  DataFrame comparison operations broadcasting changes
^^^
Previously, the broadcasting behavior of `DataFrame` comparison
operations (``==``, ``!=``, ...) was inconsistent with the behavior of

^^^
arithmetic operations (``+``, ``-``, ...).  The behavior of the comparison
operations has been changed to match the arithmetic operations in these cases.
(`22880`)  The affected cases are:  - operating against a 2-dimensional ``np.ndarray`` with either 1 row or 1 column will now broadcast the same way a ``np.ndarray`` would (`23000`).
- a list or tuple with length matching the number of rows in the `DataFrame` will now raise ``ValueError`` instead of operating column-by-column (`22880`.
- a list or tuple with length matching the number of columns in the `DataFrame` will now operate row-by-row instead of raising ``ValueError`` (`22880`).  .. ipython:: python  arr = np.arange(6).reshape(3, 2)  df = pd.DataFrame(arr)  df  *Previous behavior*:  .. code-block:: ipython  In [5]: df == arr[[0], :]  ...: # comparison previously broadcast where arithmetic would raise  Out[5]:  0  1  0  True  True  1  False  False  2  False  False  In [6]: df + arr[[0], :]  ...  ValueError: Unable to coerce to DataFrame, shape must be (3, 2): given (1, 2)  In [7]: df == (1, 2)  ...: # length matches number of columns;  ...: # comparison previously raised where arithmetic would broadcast  ...  ValueError: Invalid broadcasting comparison [(1, 2)] with block values  In [8]: df + (1, 2)  Out[8]:  0  1  0  1  3  1  3  5  2  5  7  In [9]: df == (1, 2, 3)  ...:  # length matches number of rows  ...:  # comparison previously broadcast where arithmetic would raise  Out[9]:  0  1  0  False  True  1  True  False  2  False  False  In [10]: df + (1, 2, 3)  ...  ValueError: Unable to coerce to Series, length must be 2: given 3  *New behavior*:  .. ipython:: python  # Comparison operations and arithmetic operations both broadcast.  df == arr[[0], :]  df + arr[[0], :]  .. ipython:: python  # Comparison operations and arithmetic operations both broadcast.  df == (1, 2)  df + (1, 2)  .. code-block:: ipython  # Comparison operations and arithmetic operations both raise ValueError.  In [6]: df == (1, 2, 3)  ...  ValueError: Unable to coerce to Series, length must be 2: given 3  In [7]: df + (1, 2, 3)  ...  ValueError: Unable to coerce to Series, length must be 2: given 3  .. _whatsnew_0240.api.dataframe_arithmetic_broadcasting:  DataFrame arithmetic operations broadcasting changes
^^^  `DataFrame` arithmetic operations when operating with 2-dimensional
``np.ndarray`` objects now broadcast in the same way as ``np.ndarray``
broadcast.  (`23000`)  .. ipython:: python  arr = np.arange(6).reshape(3, 2)  df = pd.DataFrame(arr)  df  *Previous behavior*:  .. code-block:: ipython  In [5]: df + arr[[0], :]  # 1 row, 2 columns  ...  ValueError: Unable to coerce to DataFrame, shape must be (3, 2): given (1, 2)  In [6]: df + arr[:, [1]]  # 1 column, 3 rows  ...  ValueError: Unable to coerce to DataFrame, shape must be (3, 2): given (3, 1)  *New behavior*:  .. ipython:: python  df + arr[[0], :]  # 1 row, 2 columns  df + arr[:, [1]]  # 1 column, 3 rows  .. _whatsnew_0240.api.incompatibilities:  Series and Index data-dtype incompatibilities
^^^  ``Series`` and ``Index`` constructors now raise when the
data is incompatible with a passed ``dtype=`` (`15832`)  *Previous behavior*:  .. code-block:: ipython  In [4]: pd.Series([-1], dtype="uint64")  Out [4]:  0  18446744073709551615  dtype: uint64  *New behavior*:  .. code-block:: ipython  In [4]: pd.Series([-1], dtype="uint64")  Out [4]:  ...  OverflowError: Trying to coerce negative values to unsigned integers  .. _whatsnew_0240.api.concat_categorical:  Concatenation changes
^^^  Calling `pandas.concat` on a ``Categorical`` of ints with NA values now
causes them to be processed as objects when concatenating with anything

other than another ``Categorical`` of ints (`19214`)  .. ipython:: python  s = pd.Series([0, 1, np.nan])  c = pd.Series([0, 1, np.nan], dtype="category")  *Previous behavior*  .. code-block:: ipython  In [3]: pd.concat([s, c])  Out[3]:  0  0.0  1  1.0  2  NaN  0  0.0  1  1.0  2  NaN  dtype: float64  *New behavior*  .. ipython:: python  pd.concat([s, c])  Datetimelike API changes
^^^  - For `DatetimeIndex` and `TimedeltaIndex` with non-``None`` ``freq`` attribute, addition or subtraction of integer-dtyped array or ``Index`` will return an object of the same class (`19959`)
- `DateOffset` objects are now immutable. Attempting to alter one of these will now raise ``AttributeError`` (`21341`)
- `PeriodIndex` subtraction of another ``PeriodIndex`` will now return an object-dtype `Index` of `DateOffset` objects instead of raising a ``TypeError`` (`20049`)
- `cut` and `qcut` now returns a `DatetimeIndex` or `TimedeltaIndex` bins when the input is datetime or timedelta dtype respectively and ``retbins=True`` (`19891`)
- `DatetimeIndex.to_period` and `Timestamp.to_period` will issue a warning when timezone information will be lost (`21333`)
- `PeriodIndex.tz_convert` and `PeriodIndex.tz_localize` have been removed (`21781`)  .. _whatsnew_0240.api.other:  Other API changes
^^^  - A newly constructed empty `DataFrame` with integer as the ``dtype`` will now only be cast to ``float64`` if ``index`` is specified (`22858`)
- `Series.str.cat` will now raise if ``others`` is a ``set`` (`23009`)
- Passing scalar values to `DatetimeIndex` or `TimedeltaIndex` will now raise ``TypeError`` instead of ``ValueError`` (`23539`)
- ``max_rows`` and ``max_cols`` parameters removed from `HTMLFormatter` since truncation is handled by `DataFrameFormatter` (`23818`)
- `read_csv` will now raise a ``ValueError`` if a column with missing values is declared as having dtype ``bool`` (`20591`)
- The column order of the resultant `DataFrame` from `MultiIndex.to_frame` is now guaranteed to match the `MultiIndex.names` order. (`22420`)
- Incorrectly passing a `DatetimeIndex` to `MultiIndex.from_tuples`, rather than a sequence of tuples, now raises a ``TypeError`` rather than a ``ValueError`` (`24024`)
- `pd.offsets.generate_range` argument ``time_rule`` has been removed; use ``offset`` instead (`24157`)
- In 0.23.x, pandas would raise a ``ValueError`` on a merge of a numeric column (e.g. ``int`` dtyped column) and an ``object`` dtyped column (`9780`). We have re-enabled the ability to merge ``object`` and other dtypes; pandas will still raise on a merge between a numeric and an ``object`` dtyped column that is composed only of strings (`21681`)
- Accessing a level of a ``MultiIndex`` with a duplicate name (e.g. in  `~MultiIndex.get_level_values`) now raises a ``ValueError`` instead of a ``KeyError`` (`21678`).
- Invalid construction of ``IntervalDtype`` will now always raise a ``TypeError`` rather than a ``ValueError`` if the subdtype is invalid (`21185`)
- Trying to reindex a ``DataFrame`` with a non unique ``MultiIndex`` now raises a ``ValueError`` instead of an ``Exception`` (`21770`)
- `Index` subtraction will attempt to operate element-wise instead of raising ``TypeError`` (`19369`)
- `pandas.io.formats.style.Styler` supports a ``number-format`` property when using `~pandas.io.formats.style.Styler.to_excel` (`22015`)
- `DataFrame.corr` and `Series.corr` now raise a ``ValueError`` along with a helpful error message instead of a ``KeyError`` when supplied with an invalid method (`22298`)
- `shift` will now always return a copy, instead of the previous behaviour of returning self when shifting by 0 (`22397`)
- `DataFrame.set_index` now gives a better (and less frequent) KeyError, raises a ``ValueError`` for incorrect types,  and will not fail on duplicate column names with ``drop=True``. (`22484`)

- Slicing a single row of a DataFrame with multiple ExtensionArrays of the same type now preserves the dtype, rather than coercing to object (`22784`)
- `DateOffset` attribute ``_cacheable`` and method ``_should_cache`` have been removed (`23118`)
- `Series.searchsorted`, when supplied a scalar value to search for, now returns a scalar instead of an array (`23801`).
- `Categorical.searchsorted`, when supplied a scalar value to search for, now returns a scalar instead of an array (`23466`).
- `Categorical.searchsorted` now raises a ``KeyError`` rather that a ``ValueError``, if a searched for key is not found in its categories (`23466`).
- `Index.hasnans` and `Series.hasnans` now always return a python boolean. Previously, a python or a numpy boolean could be returned, depending on circumstances (`23294`).
- The order of the arguments of `DataFrame.to_html` and `DataFrame.to_string` is rearranged to be consistent with each other. (`23614`)
- `CategoricalIndex.reindex` now raises a ``ValueError`` if the target index is non-unique and not equal to the current index. It previously only raised if the target index was not of a categorical dtype (`23963`).
- `Series.to_list` and `Index.to_list` are now aliases of ``Series.tolist`` respectively ``Index.tolist`` (`8826`)
- The result of ``SparseSeries.unstack`` is now a `DataFrame` with sparse values, rather than a `SparseDataFrame` (`24372`).
- `DatetimeIndex` and `TimedeltaIndex` no longer ignore the dtype precision. Passing a non-nanosecond resolution dtype will raise a ``ValueError`` (`24753`)  .. _whatsnew_0240.api.extension:  Extension type changes
~~~  **Equality and hashability**  pandas now requires that extension dtypes be hashable (i.e. the respective
``ExtensionDtype`` objects; hashability is not a requirement for the values
of the corresponding ``ExtensionArray``). The base class implements
a default ``__eq__`` and ``__hash__``. If you have a parametrized dtype, you should
update the ``ExtensionDtype._metadata`` tuple to match the signature of your
``__init__`` method. See `pandas.api.extensions.ExtensionDtype` for more (`22476`).  **New and changed methods**  - `~pandas.api.types.ExtensionArray.dropna` has been added (`21185`)
- `~pandas.api.types.ExtensionArray.repeat` has been added (`24349`)
- The ``ExtensionArray`` constructor, ``_from_sequence`` now take the keyword arg ``copy=False`` (`21185`)
- `pandas.api.extensions.ExtensionArray.shift` added as part of the basic ``ExtensionArray`` interface (`22387`).
- `~pandas.api.types.ExtensionArray.searchsorted` has been added (`24350`)
- Support for reduction operations such as ``sum``, ``mean`` via opt-in base class method override (`22762`)
- `ExtensionArray.isna` is allowed to return an ``ExtensionArray`` (`22325`).  **Dtype changes**  - ``ExtensionDtype`` has gained the ability to instantiate from string dtypes, e.g. ``decimal`` would instantiate a registered ``DecimalDtype``; furthermore  the ``ExtensionDtype`` has gained the method ``construct_array_type`` (`21185`)
- Added ``ExtensionDtype._is_numeric`` for controlling whether an extension dtype is considered numeric (`22290`).
- Added `pandas.api.types.register_extension_dtype` to register an extension type with pandas (`22664`)
- Updated the ``.type`` attribute for ``PeriodDtype``, ``DatetimeTZDtype``, and ``IntervalDtype`` to be instances of the dtype (``Period``, ``Timestamp``, and ``Interval`` respectively) (`22938`)  .. _whatsnew_0240.enhancements.extension_array_operators:  **Operator support**  A ``Series`` based on an ``ExtensionArray`` now supports arithmetic and comparison
operators (`19577`). There are two approaches for providing operator support for an ``ExtensionArray``:  1. Define each of the operators on your ``ExtensionArray`` subclass.

2. Use an operator implementation from pandas that depends on operators that are already defined  on the underlying elements (scalars) of the ``ExtensionArray``.  See the `ExtensionArray Operator Support
<extending.extension.operator>` documentation section for details on both
ways of adding operator support.  **Other changes**  - A default repr for `pandas.api.extensions.ExtensionArray` is now provided (`23601`).
- `ExtensionArray._formatting_values` is deprecated. Use `ExtensionArray._formatter` instead. (`23601`)
- An ``ExtensionArray`` with a boolean dtype now works correctly as a boolean indexer. `pandas.api.types.is_bool_dtype` now properly considers them boolean (`22326`)  **Bug fixes**  - Bug in `Series.get` for ``Series`` using ``ExtensionArray`` and integer index (`21257`)
- `~Series.shift` now dispatches to `ExtensionArray.shift` (`22386`)
- `Series.combine()` works correctly with `~pandas.api.extensions.ExtensionArray` inside of `Series` (`20825`)
- `Series.combine()` with scalar argument now works for any function type (`21248`)
- `Series.astype` and `DataFrame.astype` now dispatch to `ExtensionArray.astype` (`21185`).
- Slicing a single row of a ``DataFrame`` with multiple ExtensionArrays of the same type now preserves the dtype, rather than coercing to object (`22784`)
- Bug when concatenating multiple ``Series`` with different extension dtypes not casting to object dtype (`22994`)
- Series backed by an ``ExtensionArray`` now work with `util.hash_pandas_object` (`23066`)
- `DataFrame.stack` no longer converts to object dtype for DataFrames where each column has the same extension dtype. The output Series will have the same dtype as the columns (`23077`).
- `Series.unstack` and `DataFrame.unstack` no longer convert extension arrays to object-dtype ndarrays. Each column in the output ``DataFrame`` will now have the same dtype as the input (`23077`).
- Bug when grouping `Dataframe.groupby()` and aggregating on ``ExtensionArray`` it was not returning the actual ``ExtensionArray`` dtype (`23227`).
- Bug in `pandas.merge` when merging on an extension array-backed column (`23020`).  .. _whatsnew_0240.deprecations:  Deprecations
~~~  - `MultiIndex.labels` has been deprecated and replaced by `MultiIndex.codes`.  The functionality is unchanged. The new name better reflects the natures of  these codes and makes the ``MultiIndex`` API more similar to the API for `CategoricalIndex` (`13443`).  As a consequence, other uses of the name ``labels`` in ``MultiIndex`` have also been deprecated and replaced with ``codes``:  - You should initialize a ``MultiIndex`` instance using a parameter named ``codes`` rather than ``labels``.  - ``MultiIndex.set_labels`` has been deprecated in favor of `MultiIndex.set_codes`.  - For method `MultiIndex.copy`, the ``labels`` parameter has been deprecated and replaced by a ``codes`` parameter.
- `DataFrame.to_stata`, `read_stata`, `StataReader` and `StataWriter` have deprecated the ``encoding`` argument. The encoding of a Stata dta file is determined by the file type and cannot be changed (`21244`)
- `MultiIndex.to_hierarchical` is deprecated and will be removed in a future version (`21613`)
- `Series.ptp` is deprecated. Use ``numpy.ptp`` instead (`21614`)
- `Series.compress` is deprecated. Use ``Series[condition]`` instead (`18262`)
- The signature of `Series.to_csv` has been uniformed to that of `DataFrame.to_csv`: the name of the first argument is now ``path_or_buf``, the order of subsequent arguments has changed, the ``header`` argument now defaults to ``True``. (`19715`)
- `Categorical.from_codes` has deprecated providing float values for the ``codes`` argument. (`21767`)
- `pandas.read_table` is deprecated. Instead, use `read_csv` passing ``sep='\t'`` if necessary. This deprecation has been removed in 0.25.0. (`21948`)

- `Series.str.cat` has deprecated using arbitrary list-likes *within* list-likes. A list-like container may still contain  many ``Series``, ``Index`` or 1-dimensional ``np.ndarray``, or alternatively, only scalar values. (`21950`)
- `FrozenNDArray.searchsorted` has deprecated the ``v`` parameter in favor of ``value`` (`14645`)
- `DatetimeIndex.shift` and `PeriodIndex.shift` now accept ``periods`` argument instead of ``n`` for consistency with `Index.shift` and `Series.shift`. Using ``n`` throws a deprecation warning (`22458`, `22912`)
- The ``fastpath`` keyword of the different Index constructors is deprecated (`23110`).
- `Timestamp.tz_localize`, `DatetimeIndex.tz_localize`, and `Series.tz_localize` have deprecated the ``errors`` argument in favor of the ``nonexistent`` argument (`8917`)
- The class ``FrozenNDArray`` has been deprecated. When unpickling, ``FrozenNDArray`` will be unpickled to ``np.ndarray`` once this class is removed (`9031`)
- The methods `DataFrame.update` and `Panel.update` have deprecated the ``raise_conflict=False|True`` keyword in favor of ``errors='ignore'|'raise'`` (`23585`)
- The methods `Series.str.partition` and `Series.str.rpartition` have deprecated the ``pat`` keyword in favor of ``sep`` (`22676`)
- Deprecated the ``nthreads`` keyword of `pandas.read_feather` in favor of ``use_threads`` to reflect the changes in ``pyarrow>=0.11.0``. (`23053`)
- `pandas.read_excel` has deprecated accepting ``usecols`` as an integer. Please pass in a list of ints from 0 to ``usecols`` inclusive instead (`23527`)
- Constructing a `TimedeltaIndex` from data with ``datetime64``-dtyped data is deprecated, will raise ``TypeError`` in a future version (`23539`)
- Constructing a `DatetimeIndex` from data with ``timedelta64``-dtyped data is deprecated, will raise ``TypeError`` in a future version (`23675`)
- The ``keep_tz=False`` option (the default) of the ``keep_tz`` keyword of  `DatetimeIndex.to_series` is deprecated (`17832`).
- Timezone converting a tz-aware ``datetime.datetime`` or `Timestamp` with `Timestamp` and the ``tz`` argument is now deprecated. Instead, use `Timestamp.tz_convert` (`23579`)
- `pandas.api.types.is_period` is deprecated in favor of ``pandas.api.types.is_period_dtype`` (`23917`)
- `pandas.api.types.is_datetimetz` is deprecated in favor of ``pandas.api.types.is_datetime64tz`` (`23917`)
- Creating a `TimedeltaIndex`, `DatetimeIndex`, or `PeriodIndex` by passing range arguments ``start``, ``end``, and ``periods`` is deprecated in favor of `timedelta_range`, `date_range`, or `period_range` (`23919`)
- Passing a string alias like ``'datetime64[ns, UTC]'`` as the ``unit`` parameter to `DatetimeTZDtype` is deprecated. Use `DatetimeTZDtype.construct_from_string` instead (`23990`).
- The ``skipna`` parameter of `~pandas.api.types.infer_dtype` will switch to ``True`` by default in a future version of pandas (`17066`, `24050`)
- In `Series.where` with Categorical data, providing an ``other`` that is not present in the categories is deprecated. Convert the categorical to a different dtype or add the ``other`` to the categories first (`24077`).
- `Series.clip_lower`, `Series.clip_upper`, `DataFrame.clip_lower` and `DataFrame.clip_upper` are deprecated and will be removed in a future version. Use ``Series.clip(lower=threshold)``, ``Series.clip(upper=threshold)`` and the equivalent ``DataFrame`` methods (`24203`)
- `Series.nonzero` is deprecated and will be removed in a future version (`18262`)
- Passing an integer to `Series.fillna` and `DataFrame.fillna` with ``timedelta64[ns]`` dtypes is deprecated, will raise ``TypeError`` in a future version.  Use ``obj.fillna(pd.Timedelta(...))`` instead (`24694`)
- ``Series.cat.categorical``, ``Series.cat.name`` and ``Series.cat.index`` have been deprecated. Use the attributes on ``Series.cat`` or ``Series`` directly. (`24751`).

- Passing a dtype without a precision like ``np.dtype('datetime64')`` or ``timedelta64`` to `Index`, `DatetimeIndex` and `TimedeltaIndex` is now deprecated. Use the nanosecond-precision dtype instead (`24753`).  .. _whatsnew_0240.deprecations.datetimelike_int_ops:  Integer addition/subtraction with datetimes and timedeltas is deprecated
^^^  In the past, users could—in some cases—add or subtract integers or integer-dtype
arrays from `Timestamp`, `DatetimeIndex` and `TimedeltaIndex`.  This usage is now deprecated.  Instead add or subtract integer multiples of
the object's ``freq`` attribute (`21939`, `23878`).  *Previous behavior*:  .. code-block:: ipython  In [5]: ts = pd.Timestamp('1994-05-06 1216', freq=pd.offsets.Hour())  In [6]: ts + 2  Out[6]: Timestamp('1994-05-06 1416', freq='H')  In [7]: tdi = pd.timedelta_range('1D', periods=2)  In [8]: tdi - np.array([2, 1])  Out[8]: TimedeltaIndex(['-1 days', '1 days'], dtype='timedelta64[ns]', freq=None)  In [9]: dti = pd.date_range('2001-01-01', periods=2, freq='7D')  In [10]: dti + pd.Index([1, 2])  Out[10]: DatetimeIndex(['2001-01-08', '2001-01-22'], dtype='datetime64[ns]', freq=None)  *New behavior*:  .. ipython:: python  ts = pd.Timestamp('1994-05-06 1216', freq=pd.offsets.Hour())  ts + 2 * ts.freq  tdi = pd.timedelta_range('1D', periods=2)  tdi - np.array([2 * tdi.freq, 1 * tdi.freq])  dti = pd.date_range('2001-01-01', periods=2, freq='7D')  dti + pd.Index([1 * dti.freq, 2 * dti.freq])  .. _whatsnew_0240.deprecations.integer_tz:  Passing integer data and a timezone to DatetimeIndex
^^^  The behavior of `DatetimeIndex` when passed integer data and
a timezone is changing in a future version of pandas. Previously, these
were interpreted as wall times in the desired timezone. In the future,
these will be interpreted as wall times in UTC, which are then converted
to the desired timezone (`24559`).  The default behavior remains the same, but issues a warning:  .. code-block:: ipython  In [3]: pd.DatetimeIndex([9466848000], tz="US/Central")  /bin/ipython FutureWarning:  Passing integer-dtype data and a timezone to DatetimeIndex. Integer values  will be interpreted differently in a future version of pandas. Previously,  these were viewed as datetime64[ns] values representing the wall time  *in the specified timezone*. In the future, these will be viewed as  datetime64[ns] values representing the wall time *in UTC*. This is similar  to a nanosecond-precision UNIX epoch. To accept the future behavior, use  pd.to_datetime(integer_data, utc=True).tz_convert(tz)  To keep the previous behavior, use  pd.to_datetime(integer_data).tz_localize(tz)  #!/bin/python3  Out[3]: DatetimeIndex(['2000-01-01 000-06:00'], dtype='datetime64[ns, US/Central]', freq=None)  As the warning message explains, opt in to the future behavior by specifying that
the integer values are UTC, and then converting to the final timezone:  .. ipython:: python  pd.to_datetime([9466848000], utc=True).tz_convert('US/Central')  The old behavior can be retained with by localizing directly to the final timezone:  .. ipython:: python  pd.to_datetime([9466848000]).tz_localize('US/Central')  .. _whatsnew_0240.deprecations.tz_aware_array:  Converting timezone-aware Series and Index to NumPy arrays
^^^  The conversion from a `Series` or `Index` with timezone-aware
datetime data will change to preserve timezones by default (`23569`).  NumPy doesn't have a dedicated dtype for timezone-aware datetimes.
In the past, converting a `Series` or `DatetimeIndex` with
timezone-aware datatimes would convert to a NumPy array by  1. converting the tz-aware data to UTC
2. dropping the timezone-info
3. returning a `numpy.ndarray` with ``datetime64[ns]`` dtype  Future versions of pandas will preserve the timezone information by returning an

object-dtype NumPy array where each value is a `Timestamp` with the correct
timezone attached  .. ipython:: python  ser = pd.Series(pd.date_range('2000', periods=2, tz="CET"))  ser  The default behavior remains the same, but issues a warning  .. code-block:: python  In [8]: np.asarray(ser)  /bin/ipython FutureWarning: Converting timezone-aware DatetimeArray to timezone-naive  ndarray with 'datetime64[ns]' dtype. In the future, this will return an ndarray  with 'object' dtype where each element is a 'pandas.Timestamp' with the correct 'tz'.  To accept the future behavior, pass 'dtype=object'.  To keep the old behavior, pass 'dtype="datetime64[ns]"'.  #!/bin/python3  Out[8]:  array(['1999-12-31T2300.000', '2000-01-01T2300.000'],  dtype='datetime64[ns]')  The previous or future behavior can be obtained, without any warnings, by specifying
the ``dtype``  *Previous behavior*  .. ipython:: python  np.asarray(ser, dtype='datetime64[ns]')  *Future behavior*  .. ipython:: python  # New behavior  np.asarray(ser, dtype=object)  Or by using `Series.to_numpy`  .. ipython:: python  ser.to_numpy()  ser.to_numpy(dtype="datetime64[ns]")  All the above applies to a `DatetimeIndex` with tz-aware values as well.  .. _whatsnew_0240.prior_deprecations:  Removal of prior version deprecations/changes
~~~  - The ``LongPanel`` and ``WidePanel`` classes have been removed (`10892`)
- `Series.repeat` has renamed the ``reps`` argument to ``repeats`` (`14645`)
- Several private functions were removed from the (non-public) module ``pandas.core.common`` (`22001`)
- Removal of the previously deprecated module ``pandas.core.datetools`` (`14105`, `14094`)
- Strings passed into `DataFrame.groupby` that refer to both column and index levels will raise a ``ValueError`` (`14432`)
- `Index.repeat` and `MultiIndex.repeat` have renamed the ``n`` argument to ``repeats`` (`14645`)
- The ``Series`` constructor and ``.astype`` method will now raise a ``ValueError`` if timestamp dtypes are passed in without a unit (e.g. ``np.datetime64``) for the ``dtype`` parameter (`15987`)
- Removal of the previously deprecated ``as_indexer`` keyword completely from ``str.match()`` (`22356`, `6581`)
- The modules ``pandas.types``, ``pandas.computation``, and ``pandas.util.decorators`` have been removed (`16157`, `16250`)
- Removed the ``pandas.formats.style`` shim for `pandas.io.formats.style.Styler` (`16059`)
- ``pandas.pnow``, ``pandas.match``, ``pandas.groupby``, ``pd.get_store``, ``pd.Expr``, and ``pd.Term`` have been removed (`15538`, `15940`)
- `Categorical.searchsorted` and `Series.searchsorted` have renamed the ``v`` argument to ``value`` (`14645`)
- ``pandas.parser``, ``pandas.lib``, and ``pandas.tslib`` have been removed (`15537`)
- `Index.searchsorted` have renamed the ``key`` argument to ``value`` (`14645`)
- ``DataFrame.consolidate`` and ``Series.consolidate`` have been removed (`15501`)
- Removal of the previously deprecated module ``pandas.json`` (`19944`)
- The module ``pandas.tools`` has been removed (`15358`, `16005`)
- `SparseArray.get_values` and `SparseArray.to_dense` have dropped the ``fill`` parameter (`14686`)
- ``DataFrame.sortlevel`` and ``Series.sortlevel`` have been removed (`15099`)
- `SparseSeries.to_dense` has dropped the ``sparse_only`` parameter (`14686`)
- `DataFrame.astype` and `Series.astype` have renamed the ``raise_on_error`` argument to ``errors`` (`14967`)
- ``is_sequence``, ``is_any_int_dtype``, and ``is_floating_dtype`` have been removed from ``pandas.api.types`` (`16163`, `16189`)  .. _whatsnew_0240.performance:  Performance improvements

~~~  - Slicing Series and DataFrames with an monotonically increasing `CategoricalIndex`  is now very fast and has speed comparable to slicing with an ``Int64Index``.  The speed increase is both when indexing by label (using .loc) and position(.iloc) (`20395`)  Slicing a monotonically increasing `CategoricalIndex` itself (i.e. ``ci[1000:2000]``)  shows similar speed improvements as above (`21659`)
- Improved performance of `CategoricalIndex.equals` when comparing to another `CategoricalIndex` (`24023`)
- Improved performance of `Series.describe` in case of numeric dtpyes (`21274`)
- Improved performance of `pandas.core.groupby.GroupBy.rank` when dealing with tied rankings (`21237`)
- Improved performance of `DataFrame.set_index` with columns consisting of `Period` objects (`21582`, `21606`)
- Improved performance of `Series.at` and `Index.get_value` for Extension Arrays values (e.g. `Categorical`) (`24204`)
- Improved performance of membership checks in `Categorical` and `CategoricalIndex`  (i.e. ``x in cat``-style checks are much faster). `CategoricalIndex.contains`  is likewise much faster (`21369`, `21508`)
- Improved performance of `HDFStore.groups` (and dependent functions like  `HDFStore.keys`.  (i.e. ``x in store`` checks are much faster)  (`21372`)
- Improved the performance of `pandas.get_dummies` with ``sparse=True`` (`21997`)
- Improved performance of `IndexEngine.get_indexer_non_unique` for sorted, non-unique indexes (`9466`)
- Improved performance of `PeriodIndex.unique` (`23083`)
- Improved performance of `concat` for ``Series`` objects (`23404`)
- Improved performance of `DatetimeIndex.normalize` and `Timestamp.normalize` for timezone naive or UTC datetimes (`23634`)
- Improved performance of `DatetimeIndex.tz_localize` and various ``DatetimeIndex`` attributes with dateutil UTC timezone (`23772`)
- Fixed a performance regression on Windows with Python 3.7 of `read_csv` (`23516`)
- Improved performance of `Categorical` constructor for ``Series`` objects (`23814`)
- Improved performance of `~DataFrame.where` for Categorical data (`24077`)
- Improved performance of iterating over a `Series`. Using `DataFrame.itertuples` now creates iterators  without internally allocating lists of all elements (`20783`)
- Improved performance of `Period` constructor, additionally benefitting ``PeriodArray`` and ``PeriodIndex`` creation (`24084`, `24118`)
- Improved performance of tz-aware `DatetimeArray` binary operations (`24491`)  .. _whatsnew_0240.bug_fixes:  Bug fixes
^^^  - Bug in `Categorical.from_codes` where ``NaN`` values in ``codes`` were silently converted to ``0`` (`21767`). In the future this will raise a ``ValueError``. Also changes the behavior of ``.from_codes([1.1, 2.0])``.
- Bug in `Categorical.sort_values` where ``NaN`` values were always positioned in front regardless of ``na_position`` value. (`22556`).
- Bug when indexing with a boolean-valued ``Categorical``. Now a boolean-valued ``Categorical`` is treated as a boolean mask (`22665`)
- Constructing a `CategoricalIndex` with empty values and boolean categories was raising a ``ValueError`` after a change to dtype coercion (`22702`).
- Bug in `Categorical.take` with a user-provided ``fill_value`` not encoding the ``fill_value``, which could result in a ``ValueError``, incorrect results, or a segmentation fault (`23296`).
- In `Series.unstack`, specifying a ``fill_value`` not present in the categories now raises a ``TypeError`` rather than ignoring the ``fill_value`` (`23284`)
- Bug when resampling `DataFrame.resample()` and aggregating on categorical data, the categorical dtype was getting lost. (`23227`)
- Bug in many methods of the ``.str``-accessor, which always failed on calling the ``CategoricalIndex.str`` constructor (`23555`, `23556`)

- Bug in `Series.where` losing the categorical dtype for categorical data (`24077`)
- Bug in `Categorical.apply` where ``NaN`` values could be handled unpredictably. They now remain unchanged (`24241`)
- Bug in `Categorical` comparison methods incorrectly raising ``ValueError`` when operating against a `DataFrame` (`24630`)
- Bug in `Categorical.set_categories` where setting fewer new categories with ``rename=True`` caused a segmentation fault (`24675`)  Datetimelike
^^^  - Fixed bug where two `DateOffset` objects with different ``normalize`` attributes could evaluate as equal (`21404`)
- Fixed bug where `Timestamp.resolution` incorrectly returned 1-microsecond ``timedelta`` instead of 1-nanosecond `Timedelta` (`21336`, `21365`)
- Bug in `to_datetime` that did not consistently return an `Index` when ``box=True`` was specified (`21864`)
- Bug in `DatetimeIndex` comparisons where string comparisons incorrectly raises ``TypeError`` (`22074`)
- Bug in `DatetimeIndex` comparisons when comparing against ``timedelta64[ns]`` dtyped arrays; in some cases ``TypeError`` was incorrectly raised, in others it incorrectly failed to raise (`22074`)
- Bug in `DatetimeIndex` comparisons when comparing against object-dtyped arrays (`22074`)
- Bug in `DataFrame` with ``datetime64[ns]`` dtype addition and subtraction with ``Timedelta``-like objects (`22005`, `22163`)
- Bug in `DataFrame` with ``datetime64[ns]`` dtype addition and subtraction with ``DateOffset`` objects returning an ``object`` dtype instead of ``datetime64[ns]`` dtype (`21610`, `22163`)
- Bug in `DataFrame` with ``datetime64[ns]`` dtype comparing against ``NaT`` incorrectly (`22242`, `22163`)
- Bug in `DataFrame` with ``datetime64[ns]`` dtype subtracting ``Timestamp``-like object incorrectly returned ``datetime64[ns]`` dtype instead of ``timedelta64[ns]`` dtype (`8554`, `22163`)
- Bug in `DataFrame` with ``datetime64[ns]`` dtype subtracting ``np.datetime64`` object with non-nanosecond unit failing to convert to nanoseconds (`18874`, `22163`)
- Bug in `DataFrame` comparisons against ``Timestamp``-like objects failing to raise ``TypeError`` for inequality checks with mismatched types (`8932`, `22163`)
- Bug in `DataFrame` with mixed dtypes including ``datetime64[ns]`` incorrectly raising ``TypeError`` on equality comparisons (`13128`, `22163`)
- Bug in `DataFrame.values` returning a `DatetimeIndex` for a single-column ``DataFrame`` with tz-aware datetime values. Now a 2-D `numpy.ndarray` of `Timestamp` objects is returned (`24024`)
- Bug in `DataFrame.eq` comparison against ``NaT`` incorrectly returning ``True`` or ``NaN`` (`15697`, `22163`)
- Bug in `DatetimeIndex` subtraction that incorrectly failed to raise ``OverflowError`` (`22492`, `22508`)
- Bug in `DatetimeIndex` incorrectly allowing indexing with ``Timedelta`` object (`20464`)
- Bug in `DatetimeIndex` where frequency was being set if original frequency was ``None`` (`22150`)
- Bug in rounding methods of `DatetimeIndex` (`~DatetimeIndex.round`, `~DatetimeIndex.ceil`, `~DatetimeIndex.floor`) and `Timestamp` (`~Timestamp.round`, `~Timestamp.ceil`, `~Timestamp.floor`) could give rise to loss of precision (`22591`)
- Bug in `to_datetime` with an `Index` argument that would drop the ``name`` from the result (`21697`)
- Bug in `PeriodIndex` where adding or subtracting a `timedelta` or `Tick` object produced incorrect results (`22988`)
- Bug in the `Series` repr with period-dtype data missing a space before the data (`23601`)
- Bug in `date_range` when decrementing a start date to a past end date by a negative frequency (`23270`)
- Bug in `Series.min` which would return ``NaN`` instead of ``NaT`` when called on a series of ``NaT`` (`23282`)
- Bug in `Series.combine_first` not properly aligning categoricals, so that missing values in ``self`` where not filled by valid values from ``other`` (`24147`)

- Bug in `DataFrame.combine` with datetimelike values raising a TypeError (`23079`)
- Bug in `date_range` with frequency of ``Day`` or higher where dates sufficiently far in the future could wrap around to the past instead of raising ``OutOfBoundsDatetime`` (`14187`)
- Bug in `period_range` ignoring the frequency of ``start`` and ``end`` when those are provided as `Period` objects (`20535`).
- Bug in `PeriodIndex` with attribute ``freq.n`` greater than 1 where adding a `DateOffset` object would return incorrect results (`23215`)
- Bug in `Series` that interpreted string indices as lists of characters when setting datetimelike values (`23451`)
- Bug in `DataFrame` when creating a new column from an ndarray of `Timestamp` objects with timezones creating an object-dtype column, rather than datetime with timezone (`23932`)
- Bug in `Timestamp` constructor which would drop the frequency of an input `Timestamp` (`22311`)
- Bug in `DatetimeIndex` where calling ``np.array(dtindex, dtype=object)`` would incorrectly return an array of ``long`` objects (`23524`)
- Bug in `Index` where passing a timezone-aware `DatetimeIndex` and ``dtype=object`` would incorrectly raise a ``ValueError`` (`23524`)
- Bug in `Index` where calling ``np.array(dtindex, dtype=object)`` on a timezone-naive `DatetimeIndex` would return an array of ``datetime`` objects instead of `Timestamp` objects, potentially losing nanosecond portions of the timestamps (`23524`)
- Bug in `Categorical.__setitem__` not allowing setting with another ``Categorical`` when both are unordered and have the same categories, but in a different order (`24142`)
- Bug in `date_range` where using dates with millisecond resolution or higher could return incorrect values or the wrong number of values in the index (`24110`)
- Bug in `DatetimeIndex` where constructing a `DatetimeIndex` from a `Categorical` or `CategoricalIndex` would incorrectly drop timezone information (`18664`)
- Bug in `DatetimeIndex` and `TimedeltaIndex` where indexing with ``Ellipsis`` would incorrectly lose the index's ``freq`` attribute (`21282`)
- Clarified error message produced when passing an incorrect ``freq`` argument to `DatetimeIndex` with ``NaT`` as the first entry in the passed data (`11587`)
- Bug in `to_datetime` where ``box`` and ``utc`` arguments were ignored when passing a `DataFrame` or ``dict`` of unit mappings (`23760`)
- Bug in `Series.dt` where the cache would not update properly after an in-place operation (`24408`)
- Bug in `PeriodIndex` where comparisons against an array-like object with length 1 failed to raise ``ValueError`` (`23078`)
- Bug in `DatetimeIndex.astype`, `PeriodIndex.astype` and `TimedeltaIndex.astype` ignoring the sign of the ``dtype`` for unsigned integer dtypes (`24405`).
- Fixed bug in `Series.max` with ``datetime64[ns]``-dtype failing to return ``NaT`` when nulls are present and ``skipna=False`` is passed (`24265`)
- Bug in `to_datetime` where arrays of ``datetime`` objects containing both timezone-aware and timezone-naive ``datetimes`` would fail to raise ``ValueError`` (`24569`)
- Bug in `to_datetime` with invalid datetime format doesn't coerce input to ``NaT`` even if ``errors='coerce'`` (`24763`)  Timedelta
^^^
- Bug in `DataFrame` with ``timedelta64[ns]`` dtype division by ``Timedelta``-like scalar incorrectly returning ``timedelta64[ns]`` dtype instead of ``float64`` dtype (`20088`, `22163`)
- Bug in adding a `Index` with object dtype to a `Series` with ``timedelta64[ns]`` dtype incorrectly raising (`22390`)
- Bug in multiplying a `Series` with numeric dtype against a ``timedelta`` object (`22390`)
- Bug in `Series` with numeric dtype when adding or subtracting an array or ``Series`` with ``timedelta64`` dtype (`22390`)
- Bug in `Index` with numeric dtype when multiplying or dividing an array with dtype ``timedelta64`` (`22390`)

- Bug in `TimedeltaIndex` incorrectly allowing indexing with ``Timestamp`` object (`20464`)
- Fixed bug where subtracting `Timedelta` from an object-dtyped array would raise ``TypeError`` (`21980`)
- Fixed bug in adding a `DataFrame` with all-`timedelta64[ns]` dtypes to a `DataFrame` with all-integer dtypes returning incorrect results instead of raising ``TypeError`` (`22696`)
- Bug in `TimedeltaIndex` where adding a timezone-aware datetime scalar incorrectly returned a timezone-naive `DatetimeIndex` (`23215`)
- Bug in `TimedeltaIndex` where adding ``np.timedelta64('NaT')`` incorrectly returned an all-``NaT`` `DatetimeIndex` instead of an all-``NaT`` `TimedeltaIndex` (`23215`)
- Bug in `Timedelta` and `to_timedelta()` have inconsistencies in supported unit string (`21762`)
- Bug in `TimedeltaIndex` division where dividing by another `TimedeltaIndex` raised ``TypeError`` instead of returning a `Float64Index` (`23829`, `22631`)
- Bug in `TimedeltaIndex` comparison operations where comparing against non-``Timedelta``-like objects would raise ``TypeError`` instead of returning all-``False`` for ``__eq__`` and all-``True`` for ``__ne__`` (`24056`)
- Bug in `Timedelta` comparisons when comparing with a ``Tick`` object incorrectly raising ``TypeError`` (`24710`)  Timezones
^^^  - Bug in `Index.shift` where an ``AssertionError`` would raise when shifting across DST (`8616`)
- Bug in `Series.replace` with ``datetime64[ns, tz]`` data when replacing ``NaT`` (`11792`)
- Bug in `Timestamp` when passing different string date formats with a timezone offset would produce different timezone offsets (`12064`)
- Bug when comparing a tz-naive `Timestamp` to a tz-aware `DatetimeIndex` which would coerce the `DatetimeIndex` to tz-naive (`12601`)
- Bug in `Series.truncate` with a tz-aware `DatetimeIndex` which would cause a core dump (`9243`)
- Bug in `Series` constructor which would coerce tz-aware and tz-naive `Timestamp` to tz-aware (`13051`)
- Bug in `Index` with ``datetime64[ns, tz]`` dtype that did not localize integer data correctly (`20964`)
- Bug in `DatetimeIndex` where constructing with an integer and tz would not localize correctly (`12619`)
- Fixed bug where `DataFrame.describe` and `Series.describe` on tz-aware datetimes did not show ``first`` and ``last`` result (`21328`)
- Bug in `DatetimeIndex` comparisons failing to raise ``TypeError`` when comparing timezone-aware ``DatetimeIndex`` against ``np.datetime64`` (`22074`)
- Bug in ``DataFrame`` assignment with a timezone-aware scalar (`19843`)
- Bug in `DataFrame.asof` that raised a ``TypeError`` when attempting to compare tz-naive and tz-aware timestamps (`21194`)
- Bug when constructing a `DatetimeIndex` with `Timestamp` constructed with the ``replace`` method across DST (`18785`)
- Bug when setting a new value with `DataFrame.loc` with a `DatetimeIndex` with a DST transition (`18308`, `20724`)
- Bug in `Index.unique` that did not re-localize tz-aware dates correctly (`21737`)
- Bug when indexing a `Series` with a DST transition (`21846`)
- Bug in `DataFrame.resample` and `Series.resample` where an ``AmbiguousTimeError`` or ``NonExistentTimeError`` would raise if a timezone aware timeseries ended on a DST transition (`19375`, `10117`)
- Bug in `DataFrame.drop` and `Series.drop` when specifying a tz-aware Timestamp key to drop from a `DatetimeIndex` with a DST transition (`21761`)
- Bug in `DatetimeIndex` constructor where ``NaT`` and ``dateutil.tz.tzlocal`` would raise an ``OutOfBoundsDatetime`` error (`23807`)

- Bug in `DatetimeIndex.tz_localize` and `Timestamp.tz_localize` with ``dateutil.tz.tzlocal`` near a DST transition that would return an incorrectly localized datetime (`23807`)
- Bug in `Timestamp` constructor where a ``dateutil.tz.tzutc`` timezone passed with a ``datetime.datetime`` argument would be converted to a ``pytz.UTC`` timezone (`23807`)
- Bug in `to_datetime` where ``utc=True`` was not respected when specifying a ``unit`` and ``errors='ignore'`` (`23758`)
- Bug in `to_datetime` where ``utc=True`` was not respected when passing a `Timestamp` (`24415`)
- Bug in `DataFrame.any` returns wrong value when ``axis=1`` and the data is of datetimelike type (`23070`)
- Bug in `DatetimeIndex.to_period` where a timezone aware index was converted to UTC first before creating `PeriodIndex` (`22905`)
- Bug in `DataFrame.tz_localize`, `DataFrame.tz_convert`, `Series.tz_localize`, and `Series.tz_convert` where ``copy=False`` would mutate the original argument inplace (`6326`)
- Bug in `DataFrame.max` and `DataFrame.min` with ``axis=1`` where a `Series` with ``NaN`` would be returned when all columns contained the same timezone (`10390`)  Offsets
^^^  - Bug in `FY5253` where date offsets could incorrectly raise an ``AssertionError`` in arithmetic operations (`14774`)
- Bug in `DateOffset` where keyword arguments ``week`` and ``milliseconds`` were accepted and ignored.  Passing these will now raise ``ValueError`` (`19398`)
- Bug in adding `DateOffset` with `DataFrame` or `PeriodIndex` incorrectly raising ``TypeError`` (`23215`)
- Bug in comparing `DateOffset` objects with non-DateOffset objects, particularly strings, raising ``ValueError`` instead of returning ``False`` for equality checks and ``True`` for not-equal checks (`23524`)  Numeric
^^^  - Bug in `Series` ``__rmatmul__`` doesn't support matrix vector multiplication (`21530`)
- Bug in `factorize` fails with read-only array (`12813`)
- Fixed bug in `unique` handled signed zeros inconsistently: for some inputs 0.0 and -0.0 were treated as equal and for some inputs as different. Now they are treated as equal for all inputs (`21866`)
- Bug in `DataFrame.agg`, `DataFrame.transform` and `DataFrame.apply` where,  when supplied with a list of functions and ``axis=1`` (e.g. ``df.apply(['sum', 'mean'], axis=1)``),  a ``TypeError`` was wrongly raised. For all three methods such calculation are now done correctly. (`16679`).
- Bug in `Series` comparison against datetime-like scalars and arrays (`22074`)
- Bug in `DataFrame` multiplication between boolean dtype and integer returning ``object`` dtype instead of integer dtype (`22047`, `22163`)
- Bug in `DataFrame.apply` where, when supplied with a string argument and additional positional or keyword arguments (e.g. ``df.apply('sum', min_count=1)``), a ``TypeError`` was wrongly raised (`22376`)
- Bug in `DataFrame.astype` to extension dtype may raise ``AttributeError`` (`22578`)
- Bug in `DataFrame` with ``timedelta64[ns]`` dtype arithmetic operations with ``ndarray`` with integer dtype incorrectly treating the narray as ``timedelta64[ns]`` dtype (`23114`)
- Bug in `Series.rpow` with object dtype ``NaN`` for ``1 ** NA`` instead of ``1`` (`22922`).
- `Series.agg` can now handle numpy NaN-aware methods like `numpy.nansum` (`19629`)
- Bug in `Series.rank` and `DataFrame.rank` when ``pct=True`` and more than 2\ `24` rows are present resulted in percentages greater than 1.0 (`18271`)
- Calls such as `DataFrame.round` with a non-unique `CategoricalIndex` now return expected data. Previously, data would be improperly duplicated (`21809`).
- Added ``log10``, ``floor`` and ``ceil`` to the list of supported functions in `DataFrame.eval` (`24139`, `24353`)
- Logical operations ``&, |, ^`` between `Series` and `Index` will no longer raise ``ValueError`` (`22092`)

- Checking PEP 3141 numbers in `~pandas.api.types.is_scalar` function returns ``True`` (`22903`)
- Reduction methods like `Series.sum` now accept the default value of ``keepdims=False`` when called from a NumPy ufunc, rather than raising a ``TypeError``. Full support for ``keepdims`` has not been implemented (`24356`).  Conversion
^^^  - Bug in `DataFrame.combine_first` in which column types were unexpectedly converted to float (`20699`)
- Bug in `DataFrame.clip` in which column types are not preserved and casted to float (`24162`)
- Bug in `DataFrame.clip` when order of columns of dataframes doesn't match, result observed is wrong in numeric values (`20911`)
- Bug in `DataFrame.astype` where converting to an extension dtype when duplicate column names are present causes a ``RecursionError`` (`24704`)  Strings
^^^  - Bug in `Index.str.partition` was not nan-safe (`23558`).
- Bug in `Index.str.split` was not nan-safe (`23677`).
- Bug `Series.str.contains` not respecting the ``na`` argument for a ``Categorical`` dtype ``Series`` (`22158`)
- Bug in `Index.str.cat` when the result contained only ``NaN`` (`24044`)  Interval
^^^  - Bug in the `IntervalIndex` constructor where the ``closed`` parameter did not always override the inferred ``closed`` (`19370`)
- Bug in the ``IntervalIndex`` repr where a trailing comma was missing after the list of intervals (`20611`)
- Bug in `Interval` where scalar arithmetic operations did not retain the ``closed`` value (`22313`)
- Bug in `IntervalIndex` where indexing with datetime-like values raised a ``KeyError`` (`20636`)
- Bug in ``IntervalTree`` where data containing ``NaN`` triggered a warning and resulted in incorrect indexing queries with `IntervalIndex` (`23352`)  Indexing
^^^  - Bug in `DataFrame.ne` fails if columns contain column name "dtype" (`22383`)
- The traceback from a ``KeyError`` when asking ``.loc`` for a single missing label is now shorter and more clear (`21557`)
- `PeriodIndex` now emits a ``KeyError`` when a malformed string is looked up, which is consistent with the behavior of `DatetimeIndex` (`22803`)
- When ``.ix`` is asked for a missing integer label in a `MultiIndex` with a first level of integer type, it now raises a ``KeyError``, consistently with the case of a flat `Int64Index`, rather than falling back to positional indexing (`21593`)
- Bug in `Index.reindex` when reindexing a tz-naive and tz-aware `DatetimeIndex` (`8306`)
- Bug in `Series.reindex` when reindexing an empty series with a ``datetime64[ns, tz]`` dtype (`20869`)
- Bug in `DataFrame` when setting values with ``.loc`` and a timezone aware `DatetimeIndex` (`11365`)
- ``DataFrame.__getitem__`` now accepts dictionaries and dictionary keys as list-likes of labels, consistently with ``Series.__getitem__`` (`21294`)
- Fixed ``DataFrame[np.nan]`` when columns are non-unique (`21428`)
- Bug when indexing `DatetimeIndex` with nanosecond resolution dates and timezones (`11679`)
- Bug where indexing with a Numpy array containing negative values would mutate the indexer (`21867`)
- Bug where mixed indexes wouldn't allow integers for ``.at`` (`19860`)
- ``Float64Index.get_loc`` now raises ``KeyError`` when boolean key passed. (`19087`)
- Bug in `DataFrame.loc` when indexing with an `IntervalIndex` (`19977`)
- `Index` no longer mangles ``None``, ``NaN`` and ``NaT``, i.e. they are treated as three different keys. However, for numeric Index all three are still coerced to a ``NaN`` (`22332`)
- Bug in ``scalar in Index`` if scalar is a float while the ``Index`` is of integer dtype (`22085`)
- Bug in `MultiIndex.set_levels` when levels value is not subscriptable (`23273`)
- Bug where setting a timedelta column by ``Index`` causes it to be casted to double, and therefore lose precision (`23511`)
- Bug in `Index.union` and `Index.intersection` where name of the ``Index`` of the result was not computed correctly for certain cases (`9943`, `9862`)

- Bug in `Index` slicing with boolean `Index` may raise ``TypeError`` (`22533`)
- Bug in ``PeriodArray.__setitem__`` when accepting slice and list-like value (`23978`)
- Bug in `DatetimeIndex`, `TimedeltaIndex` where indexing with ``Ellipsis`` would lose their ``freq`` attribute (`21282`)
- Bug in ``iat`` where using it to assign an incompatible value would create a new column (`23236`)  Missing
^^^  - Bug in `DataFrame.fillna` where a ``ValueError`` would raise when one column contained a ``datetime64[ns, tz]`` dtype (`15522`)
- Bug in `Series.hasnans` that could be incorrectly cached and return incorrect answers if null elements are introduced after an initial call (`19700`)
- `Series.isin` now treats all NaN-floats as equal also for ``np.object_``-dtype. This behavior is consistent with the behavior for float64 (`22119`)
- `unique` no longer mangles NaN-floats and the ``NaT``-object for ``np.object_``-dtype, i.e. ``NaT`` is no longer coerced to a NaN-value and is treated as a different entity. (`22295`)
- `DataFrame` and `Series` now properly handle numpy masked arrays with hardened masks. Previously, constructing a DataFrame or Series from a masked array with a hard mask would create a pandas object containing the underlying value, rather than the expected NaN. (`24574`)
- Bug in `DataFrame` constructor where ``dtype`` argument was not honored when handling numpy masked record arrays. (`24874`)  MultiIndex
^^^  - Bug in `io.formats.style.Styler.applymap` where ``subset=`` with `MultiIndex` slice would reduce to `Series` (`19861`)
- Removed compatibility for `MultiIndex` pickles prior to version 0.8.0; compatibility with `MultiIndex` pickles from version 0.13 forward is maintained (`21654`)
- `MultiIndex.get_loc_level` (and as a consequence, ``.loc`` on a ``Series`` or ``DataFrame`` with a `MultiIndex` index) will now raise a ``KeyError``, rather than returning an empty ``slice``, if asked a label which is present in the ``levels`` but is unused (`2221`)
- `MultiIndex` has gained the `MultiIndex.from_frame`, it allows constructing a `MultiIndex` object from a `DataFrame` (`22420`)
- Fix ``TypeError`` in Python 3 when creating `MultiIndex` in which some levels have mixed types, e.g. when some labels are tuples (`15457`)  IO
^^  - Bug in `read_csv` in which a column specified with ``CategoricalDtype`` of boolean categories was not being correctly coerced from string values to booleans (`20498`)
- Bug in `read_csv` in which unicode column names were not being properly recognized with Python 2.x (`13253`)
- Bug in `DataFrame.to_sql` when writing timezone aware data (``datetime64[ns, tz]`` dtype) would raise a ``TypeError`` (`9086`)
- Bug in `DataFrame.to_sql` where a naive `DatetimeIndex` would be written as ``TIMESTAMP WITH TIMEZONE`` type in supported databases, e.g. PostgreSQL (`23510`)
- Bug in `read_excel()` when ``parse_cols`` is specified with an empty dataset (`9208`)
- `read_html()` no longer ignores all-whitespace ``<tr>`` within ``<thead>`` when considering the ``skiprows`` and ``header`` arguments. Previously, users had to decrease their ``header`` and ``skiprows`` values on such tables to work around the issue. (`21641`)
- `read_excel()` will correctly show the deprecation warning for previously deprecated ``sheetname`` (`17994`)
- `read_csv()` and `read_table()` will throw ``UnicodeError`` and not coredump on badly encoded strings (`22748`)
- `read_csv()` will correctly parse timezone-aware datetimes (`22256`)
- Bug in `read_csv()` in which memory management was prematurely optimized for the C engine when the data was being read in chunks (`23509`)
- Bug in `read_csv()` in unnamed columns were being improperly identified when extracting a multi-index (`23687`)
- `read_sas()` will parse numbers in sas7bdat-files that have width less than 8 bytes correctly. (`21616`)

- `read_sas()` will correctly parse sas7bdat files with many columns (`22628`)
- `read_sas()` will correctly parse sas7bdat files with data page types having also bit 7 set (so page type is 128 + 256 = 384) (`16615`)
- Bug in `read_sas()` in which an incorrect error was raised on an invalid file format. (`24548`)
- Bug in `detect_client_encoding` where potential ``IOError`` goes unhandled when importing in a mod_wsgi process due to restricted access to stdout. (`21552`)
- Bug in `DataFrame.to_html()` with ``index=False`` misses truncation indicators (...) on truncated DataFrame (`15019`, `22783`)
- Bug in `DataFrame.to_html()` with ``index=False`` when both columns and row index are ``MultiIndex`` (`22579`)
- Bug in `DataFrame.to_html()` with ``index_names=False`` displaying index name (`22747`)
- Bug in `DataFrame.to_html()` with ``header=False`` not displaying row index names (`23788`)
- Bug in `DataFrame.to_html()` with ``sparsify=False`` that caused it to raise ``TypeError`` (`22887`)
- Bug in `DataFrame.to_string()` that broke column alignment when ``index=False`` and width of first column's values is greater than the width of first column's header (`16839`, `13032`)
- Bug in `DataFrame.to_string()` that caused representations of `DataFrame` to not take up the whole window (`22984`)
- Bug in `DataFrame.to_csv` where a single level MultiIndex incorrectly wrote a tuple. Now just the value of the index is written (`19589`).
- `HDFStore` will raise ``ValueError`` when the ``format`` kwarg is passed to the constructor (`13291`)
- Bug in `HDFStore.append` when appending a `DataFrame` with an empty string column and ``min_itemsize`` < 8 (`12242`)
- Bug in `read_csv()` in which memory leaks occurred in the C engine when parsing ``NaN`` values due to insufficient cleanup on completion or error (`21353`)
- Bug in `read_csv()` in which incorrect error messages were being raised when ``skipfooter`` was passed in along with ``nrows``, ``iterator``, or ``chunksize`` (`23711`)
- Bug in `read_csv()` in which `MultiIndex` index names were being improperly handled in the cases when they were not provided (`23484`)
- Bug in `read_csv()` in which unnecessary warnings were being raised when the dialect's values conflicted with the default arguments (`23761`)
- Bug in `read_html()` in which the error message was not displaying the valid flavors when an invalid one was provided (`23549`)
- Bug in `read_excel()` in which extraneous header names were extracted, even though none were specified (`11733`)
- Bug in `read_excel()` in which column names were not being properly converted to string sometimes in Python 2.x (`23874`)
- Bug in `read_excel()` in which ``index_col=None`` was not being respected and parsing index columns anyway (`18792`, `20480`)
- Bug in `read_excel()` in which ``usecols`` was not being validated for proper column names when passed in as a string (`20480`)
- Bug in `DataFrame.to_dict` when the resulting dict contains non-Python scalars in the case of numeric data (`23753`)
- `DataFrame.to_string()`, `DataFrame.to_html()`, `DataFrame.to_latex()` will correctly format output when a string is passed as the ``float_format`` argument (`21625`, `22270`)
- Bug in `read_csv` that caused it to raise ``OverflowError`` when trying to use 'inf' as ``na_value`` with integer index column (`17128`)
- Bug in `read_csv` that caused the C engine on Python 3.6+ on Windows to improperly read CSV filenames with accented or special characters (`15086`)
- Bug in `read_fwf` in which the compression type of a file was not being properly inferred (`22199`)
- Bug in `pandas.io.json.json_normalize` that caused it to raise ``TypeError`` when two consecutive elements of ``record_path`` are dicts (`22706`)

- Bug in `DataFrame.to_stata`, `pandas.io.stata.StataWriter` and `pandas.io.stata.StataWriter117` where a exception would leave a partially written and invalid dta file (`23573`)
- Bug in `DataFrame.to_stata` and `pandas.io.stata.StataWriter117` that produced invalid files when using strLs with non-ASCII characters (`23573`)
- Bug in `HDFStore` that caused it to raise ``ValueError`` when reading a Dataframe in Python 3 from fixed format written in Python 2 (`24510`)
- Bug in `DataFrame.to_string()` and more generally in the floating ``repr`` formatter. Zeros were not trimmed if ``inf`` was present in a columns while it was the case with NA values. Zeros are now trimmed as in the presence of NA (`24861`).
- Bug in the ``repr`` when truncating the number of columns and having a wide last column (`24849`).  Plotting
^^^  - Bug in `DataFrame.plot.scatter` and `DataFrame.plot.hexbin` caused x-axis label and ticklabels to disappear when colorbar was on in IPython inline backend (`10611`, `10678`, and `20455`)
- Bug in plotting a Series with datetimes using `matplotlib.axes.Axes.scatter` (`22039`)
- Bug in `DataFrame.plot.bar` caused bars to use multiple colors instead of a single one (`20585`)
- Bug in validating color parameter caused extra color to be appended to the given color array. This happened to multiple plotting functions using matplotlib. (`20726`)  GroupBy/resample/rolling
^^^  - Bug in `pandas.core.window.Rolling.min` and `pandas.core.window.Rolling.max` with ``closed='left'``, a datetime-like index and only one entry in the series leading to segfault (`24718`)
- Bug in `pandas.core.groupby.GroupBy.first` and `pandas.core.groupby.GroupBy.last` with ``as_index=False`` leading to the loss of timezone information (`15884`)
- Bug in `DateFrame.resample` when downsampling across a DST boundary (`8531`)
- Bug in date anchoring for `DateFrame.resample` with offset `Day` when n > 1 (`24127`)
- Bug where ``ValueError`` is wrongly raised when calling `~pandas.core.groupby.SeriesGroupBy.count` method of a  ``SeriesGroupBy`` when the grouping variable only contains NaNs and numpy version < 1.13 (`21956`).
- Multiple bugs in `pandas.core.window.Rolling.min` with ``closed='left'`` and a  datetime-like index leading to incorrect results and also segfault. (`21704`)
- Bug in `pandas.core.resample.Resampler.apply` when passing positional arguments to applied func (`14615`).
- Bug in `Series.resample` when passing ``numpy.timedelta64`` to ``loffset`` kwarg (`7687`).
- Bug in `pandas.core.resample.Resampler.asfreq` when frequency of ``TimedeltaIndex`` is a subperiod of a new frequency (`13022`).
- Bug in `pandas.core.groupby.SeriesGroupBy.mean` when values were integral but could not fit inside of int64, overflowing instead. (`22487`)
- `pandas.core.groupby.RollingGroupby.agg` and `pandas.core.groupby.ExpandingGroupby.agg` now support multiple aggregation functions as parameters (`15072`)
- Bug in `DataFrame.resample` and `Series.resample` when resampling by a weekly offset (``'W'``) across a DST transition (`9119`, `21459`)
- Bug in `DataFrame.expanding` in which the ``axis`` argument was not being respected during aggregations (`23372`)
- Bug in `pandas.core.groupby.GroupBy.transform` which caused missing values when the input function can accept a `DataFrame` but renames it (`23455`).
- Bug in `pandas.core.groupby.GroupBy.nth` where column order was not always preserved (`20760`)
- Bug in `pandas.core.groupby.GroupBy.rank` with ``method='dense'`` and ``pct=True`` when a group has only one member would raise a ``ZeroDivisionError`` (`23666`).
- Calling `pandas.core.groupby.GroupBy.rank` with empty groups and ``pct=True`` was raising a ``ZeroDivisionError`` (`22519`)
- Bug in `DataFrame.resample` when resampling ``NaT`` in ``TimeDeltaIndex`` (`13223`).

- Bug in `DataFrame.groupby` did not respect the ``observed`` argument when selecting a column and instead always used ``observed=False`` (`23970`)
- Bug in `pandas.core.groupby.SeriesGroupBy.pct_change` or `pandas.core.groupby.DataFrameGroupBy.pct_change` would previously work across groups when calculating the percent change, where it now correctly works per group (`21200`, `21235`).
- Bug preventing hash table creation with very large number (2^32) of rows (`22805`)
- Bug in groupby when grouping on categorical causes ``ValueError`` and incorrect grouping if ``observed=True`` and ``nan`` is present in categorical column (`24740`, `21151`).  Reshaping
^^^  - Bug in `pandas.concat` when joining resampled DataFrames with timezone aware index (`13783`)
- Bug in `pandas.concat` when joining only ``Series`` the ``names`` argument of ``concat`` is no longer ignored (`23490`)
- Bug in `Series.combine_first` with ``datetime64[ns, tz]`` dtype which would return tz-naive result (`21469`)
- Bug in `Series.where` and `DataFrame.where` with ``datetime64[ns, tz]`` dtype (`21546`)
- Bug in `DataFrame.where` with an empty DataFrame and empty ``cond`` having non-bool dtype (`21947`)
- Bug in `Series.mask` and `DataFrame.mask` with ``list`` conditionals (`21891`)
- Bug in `DataFrame.replace` raises RecursionError when converting OutOfBounds ``datetime64[ns, tz]`` (`20380`)
- `pandas.core.groupby.GroupBy.rank` now raises a ``ValueError`` when an invalid value is passed for argument ``na_option`` (`22124`)
- Bug in `get_dummies` with Unicode attributes in Python 2 (`22084`)
- Bug in `DataFrame.replace` raises ``RecursionError`` when replacing empty lists (`22083`)
- Bug in `Series.replace` and `DataFrame.replace` when dict is used as the ``to_replace`` value and one key in the dict is another key's value, the results were inconsistent between using integer key and using string key (`20656`)
- Bug in `DataFrame.drop_duplicates` for empty ``DataFrame`` which incorrectly raises an error (`20516`)
- Bug in `pandas.wide_to_long` when a string is passed to the stubnames argument and a column name is a substring of that stubname (`22468`)
- Bug in `merge` when merging ``datetime64[ns, tz]`` data that contained a DST transition (`18885`)
- Bug in `merge_asof` when merging on float values within defined tolerance (`22981`)
- Bug in `pandas.concat` when concatenating a multicolumn DataFrame with tz-aware data against a DataFrame with a different number of columns (`22796`)
- Bug in `merge_asof` where confusing error message raised when attempting to merge with missing values (`23189`)
- Bug in `DataFrame.nsmallest` and `DataFrame.nlargest` for dataframes that have a `MultiIndex` for columns (`23033`).
- Bug in `pandas.melt` when passing column names that are not present in ``DataFrame`` (`23575`)
- Bug in `DataFrame.append` with a `Series` with a dateutil timezone would raise a ``TypeError`` (`23682`)
- Bug in `Series` construction when passing no data and ``dtype=str`` (`22477`)
- Bug in `cut` with ``bins`` as an overlapping ``IntervalIndex`` where multiple bins were returned per item instead of raising a ``ValueError`` (`23980`)
- Bug in `pandas.concat` when joining ``Series`` datetimetz with ``Series`` category would lose timezone (`23816`)
- Bug in `DataFrame.join` when joining on partial MultiIndex would drop names (`20452`).
- `DataFrame.nlargest` and `DataFrame.nsmallest` now returns the correct n values when keep != 'all' also when tied on the first columns (`22752`)
- Constructing a DataFrame with an index argument that wasn't already an instance of `~pandas.core.Index` was broken (`2227`).
- Bug in `DataFrame` prevented list subclasses to be used to construction (`21226`)

- Bug in `DataFrame.unstack` and `DataFrame.pivot_table` returning a misleading error message when the resulting DataFrame has more elements than int32 can handle. Now, the error message is improved, pointing towards the actual problem (`20601`)
- Bug in `DataFrame.unstack` where a ``ValueError`` was raised when unstacking timezone aware values (`18338`)
- Bug in `DataFrame.stack` where timezone aware values were converted to timezone naive values (`19420`)
- Bug in `merge_asof` where a ``TypeError`` was raised when ``by_col`` were timezone aware values (`21184`)
- Bug showing an incorrect shape when throwing error during ``DataFrame`` construction. (`20742`)  .. _whatsnew_0240.bug_fixes.sparse:  Sparse
^^^  - Updating a boolean, datetime, or timedelta column to be Sparse now works (`22367`)
- Bug in `Series.to_sparse` with Series already holding sparse data not constructing properly (`22389`)
- Providing a ``sparse_index`` to the SparseArray constructor no longer defaults the na-value to ``np.nan`` for all dtypes. The correct na_value for ``data.dtype`` is now used.
- Bug in ``SparseArray.nbytes`` under-reporting its memory usage by not including the size of its sparse index.
- Improved performance of `Series.shift` for non-NA ``fill_value``, as values are no longer converted to a dense array.
- Bug in ``DataFrame.groupby`` not including ``fill_value`` in the groups for non-NA ``fill_value`` when grouping by a sparse column (`5078`)
- Bug in unary inversion operator (``~``) on a ``SparseSeries`` with boolean values. The performance of this has also been improved (`22835`)
- Bug in `SparseArary.unique` not returning the unique values (`19595`)
- Bug in `SparseArray.nonzero` and `SparseDataFrame.dropna` returning shifted/incorrect results (`21172`)
- Bug in `DataFrame.apply` where dtypes would lose sparseness (`23744`)
- Bug in `concat` when concatenating a list of `Series` with all-sparse values changing the ``fill_value`` and converting to a dense Series (`24371`)  Style
^^^  - `~pandas.io.formats.style.Styler.background_gradient` now takes a ``text_color_threshold`` parameter to automatically lighten the text color based on the luminance of the background color. This improves readability with dark background colors without the need to limit the background colormap range. (`21258`)
- `~pandas.io.formats.style.Styler.background_gradient` now also supports tablewise application (in addition to rowwise and columnwise) with ``axis=None`` (`15204`)
- `~pandas.io.formats.style.Styler.bar` now also supports tablewise application (in addition to rowwise and columnwise) with ``axis=None`` and setting clipping range with ``vmin`` and ``vmax`` (`21548` and `21526`). ``NaN`` values are also handled properly.  Build changes
^^^  - Building pandas for development now requires ``cython >= 0.28.2`` (`21688`)
- Testing pandas now requires `hypothesis>=3.58``.  You can find `the Hypothesis docs here, and a pandas-specific introduction `in the contributing guide <using-hypothesis>`. (`22280`)
- Building pandas on macOS now targets minimum macOS 10.9 if run on macOS 10.9 or above (`23424`)  Other
^^^  - Bug where C variables were declared with external linkage causing import errors if certain other C libraries were imported before pandas. (`24113`)  .. _whatsnew_0.24.0.contributors:  Contributors
~~~  .. contributors:: v0.23.4..v0.24.0

What's new in 0.24.1 (February 3, 2019)

.. warning::  The 0.24.x series of releases will be the last to support Python 2. Future feature  releases will support Python 3 only. See Dropping Python 2.7 for more.  {{ header }}  These are the changes in pandas 0.24.1. See `release` for a full changelog
including other versions of pandas. See `whatsnew_0240` for the 0.24.0 changelog.  .. _whatsnew_0241.api:  API changes
~~~  Changing the ``sort`` parameter for `Index` set operations
^^^  The default ``sort`` value for `Index.union` has changed from ``True`` to ``None`` (`24959`).
The default *behavior*, however, remains the same: the result is sorted, unless  1. ``self`` and ``other`` are identical
2. ``self`` or ``other`` is empty
3. ``self`` or ``other`` contain values that can not be compared (a ``RuntimeWarning`` is raised).  This change will allow ``sort=True`` to mean "always sort" in a future release.  The same change applies to `Index.difference` and `Index.symmetric_difference`, which
would not sort the result when the values could not be compared.  The ``sort`` option for `Index.intersection` has changed in three ways.  1. The default has changed from ``True`` to ``False``, to restore the  pandas 0.23.4 and earlier behavior of not sorting by default.
2. The behavior of ``sort=True`` can now be obtained with ``sort=None``.  This will sort the result only if the values in ``self`` and ``other``  are not identical.
3. The value ``sort=True`` is no longer allowed. A future version of pandas  will properly support ``sort=True`` meaning "always sort".  .. _whatsnew_0241.regressions:  Fixed regressions
~~~  - Fixed regression in `DataFrame.to_dict` with ``records`` orient raising an  ``AttributeError`` when the ``DataFrame`` contained more than 255 columns, or  wrongly converting column names that were not valid python identifiers (`24939`, `24940`).
- Fixed regression in `read_sql` when passing certain queries with MySQL/pymysql (`24988`).
- Fixed regression in `Index.intersection` incorrectly sorting the values by default (`24959`).
- Fixed regression in `merge` when merging an empty ``DataFrame`` with multiple timezone-aware columns on one of the timezone-aware columns (`25014`).
- Fixed regression in `Series.rename_axis` and `DataFrame.rename_axis` where passing ``None`` failed to remove the axis name (`25034`)
- Fixed regression in `to_timedelta` with ``box=False`` incorrectly returning a ``datetime64`` object instead of a ``timedelta64`` object (`24961`)
- Fixed regression where custom hashable types could not be used as column keys in `DataFrame.set_index` (`24969`)  .. _whatsnew_0241.bug_fixes:  Bug fixes
~~~  **Reshaping**  - Bug in `DataFrame.groupby` with `Grouper` when there is a time change (DST) and grouping frequency is ``'1d'`` (`24972`)  **Visualization**  - Fixed the warning for implicitly registered matplotlib converters not showing. See `whatsnew_0211.converters` for more (`24963`).  **Other**  - Fixed AttributeError when printing a DataFrame's HTML repr after accessing the IPython config object (`25036`)  .. _whatsnew_0.241.contributors:  Contributors
~~~  .. Including the contributors hardcoded for this release, as backporting with  MeeseeksDev loses the commit authors  A total of 7 people contributed patches to this release. People with a "+" by their names contributed a patch for the first time.  * Alex Buchkovsky
* Roman Yurchak
* h-vetinari
* jbrockmendel
* Jeremy Schendel
* Joris Van den Bossche
* Tom Augspurger

What's new in 0.24.2 (March 12, 2019)

.. warning::  The 0.24.x series of releases will be the last to support Python 2. Future feature  releases will support Python 3 only. See Dropping Python 2.7 for more.  {{ header }}  These are the changes in pandas 0.24.2. See `release` for a full changelog
including other versions of pandas.  .. _whatsnew_0242.regressions:  Fixed regressions
~~~  - Fixed regression in `DataFrame.all` and `DataFrame.any` where ``bool_only=True`` was ignored (`25101`)
- Fixed issue in ``DataFrame`` construction with passing a mixed list of mixed types could segfault. (`25075`)
- Fixed regression in `DataFrame.apply` causing ``RecursionError`` when ``dict``-like classes were passed as argument. (`25196`)
- Fixed regression in `DataFrame.replace` where ``regex=True`` was only replacing patterns matching the start of the string (`25259`)
- Fixed regression in `DataFrame.duplicated()`, where empty dataframe was not returning a boolean dtyped Series. (`25184`)
- Fixed regression in `Series.min` and `Series.max` where ``numeric_only=True`` was ignored when the ``Series`` contained ``Categorical`` data (`25299`)
- Fixed regression in subtraction between `Series` objects with ``datetime64[ns]`` dtype incorrectly raising ``OverflowError`` when the ``Series`` on the right contains null values (`25317`)
- Fixed regression in `TimedeltaIndex` where ``np.sum(index)`` incorrectly returned a zero-dimensional object instead of a scalar (`25282`)
- Fixed regression in ``IntervalDtype`` construction where passing an incorrect string with 'Interval' as a prefix could result in a ``RecursionError``. (`25338`)
- Fixed regression in creating a period-dtype array from a read-only NumPy array of period objects. (`25403`)
- Fixed regression in `Categorical`, where constructing it from a categorical ``Series`` and an explicit ``categories=`` that differed from that in the ``Series`` created an invalid object which could trigger segfaults. (`25318`)
- Fixed regression in `to_timedelta` losing precision when converting floating data to ``Timedelta`` data (`25077`).
- Fixed pip installing from source into an environment without NumPy (`25193`)
- Fixed regression in `DataFrame.replace` where large strings of numbers would be coerced into ``int64``, causing an ``OverflowError`` (`25616`)
- Fixed regression in `factorize` when passing a custom ``na_sentinel`` value with ``sort=True`` (`25409`).
- Fixed regression in `DataFrame.to_csv` writing duplicate line endings with gzip compress (`25311`)  .. _whatsnew_0242.bug_fixes:  Bug fixes
~~~  **I/O**  - Better handling of terminal printing when the terminal dimensions are not known (`25080`)
- Bug in reading a HDF5 table-format ``DataFrame`` created in Python 2, in Python 3 (`24925`)
- Bug in reading a JSON with ``orient='table'`` generated by `DataFrame.to_json` with ``index=False`` (`25170`)
- Bug where float indexes could have misaligned values when printing (`25061`)  **Categorical**

- Bug where calling `Series.replace` on categorical data could return a ``Series`` with incorrect dimensions (`24971`)

**Reshaping**  - Bug in `~pandas.core.groupby.GroupBy.transform` where applying a function to a timezone aware column would return a timezone naive result (`24198`)
- Bug in `DataFrame.join` when joining on a timezone aware `DatetimeIndex` (`23931`)  **Visualization**  - Bug in `Series.plot` where a secondary y axis could not be set to log scale (`25545`)  **Other**  - Bug in `Series.is_unique` where single occurrences of ``NaN`` were not considered unique (`25180`)
- Bug in `merge` when merging an empty ``DataFrame`` with an ``Int64`` column or a non-empty ``DataFrame`` with an ``Int64`` column that is all ``NaN`` (`25183`)
- Bug in ``IntervalTree`` where a ``RecursionError`` occurs upon construction due to an overflow when adding endpoints, which also causes `IntervalIndex` to crash during indexing operations (`25485`)
- Bug in `Series.size` raising for some extension-array-backed ``Series``, rather than returning the size (`25580`)
- Bug in resampling raising for nullable integer-dtype columns (`25580`)  .. _whatsnew_0242.contributors:  Contributors
~~~  .. Including the contributors hardcoded for this release, as backporting with  MeeseeksDev loses the commit authors  A total of 25 people contributed patches to this release. People with a "+" by their names contributed a patch for the first time.  * Albert Villanova del Moral
* Arno Veenstra +
* chris-b1
* Devin Petersohn +
* EternalLearner42 +
* Flavien Lambert +
* gfyoung
* Gioia Ballin
* Jeff Reback
* Johan von Forstner +
* Josh
* Justin Zheng
* Kendall Masse
* Matthew Roeschke
* Max Bolingbroke +
* rbenes +
* Sterling Paramore +
* Tao He +
* Thomas A Caswell
* Vibhu Agarwal +
* William Ayd
* Zach Angell

.. warning::  Starting with the 0.25.x series of releases, pandas only supports Python 3.5.3 and higher.  See Dropping Python 2.7 for more details.  .. warning::  The minimum supported Python version will be bumped to 3.6 in a future release.  .. warning::  ``Panel`` has been fully removed. For N-D labeled data structures, please  use xarray  .. warning::  `read_pickle` and `read_msgpack` are only guaranteed backwards compatible back to  pandas version 0.20.3 (`27082`)  {{ header }}  These are the changes in pandas 0.25.0. See `release` for a full changelog
~~~  .. _whatsnew_0250.enhancements.agg_relabel:  GroupBy aggregation with relabeling
^^^  pandas has added special groupby behavior, known as "named aggregation", for naming the
output columns when applying multiple aggregation functions to specific columns (`18366`, `26512`).  .. ipython:: python  animals = pd.DataFrame({'kind': ['cat', 'dog', 'cat', 'dog'],  'height': [9.1, 6.0, 9.5, 34.0],  'weight': [7.9, 7.5, 9.9, 198.0]})  animals  animals.groupby("kind").agg(  min_height=pd.NamedAgg(column='height', aggfunc='min'),  max_height=pd.NamedAgg(column='height', aggfunc='max'),  average_weight=pd.NamedAgg(column='weight', aggfunc=np.mean),  )  Pass the desired columns names as the ``**kwargs`` to ``.agg``. The values of ``**kwargs``
should be tuples where the first element is the column selection, and the second element is the
aggregation function to apply. pandas provides the ``pandas.NamedAgg`` namedtuple to make it clearer
what the arguments to the function are, but plain tuples are accepted as well.  .. ipython:: python  animals.groupby("kind").agg(  min_height=('height', 'min'),  max_height=('height', 'max'),  average_weight=('weight', np.mean),  )  Named aggregation is the recommended replacement for the deprecated "dict-of-dicts"
approach to naming the output of column-specific aggregations (`whatsnew_0200.api_breaking.deprecate_group_agg_dict`).  A similar approach is now available for Series groupby objects as well. Because there's no need for
column selection, the values can just be the functions to apply  .. ipython:: python  animals.groupby("kind").height.agg(  min_height="min",  max_height="max",  )  This type of aggregation is the recommended alternative to the deprecated behavior when passing
a dict to a Series groupby aggregation (`whatsnew_0200.api_breaking.deprecate_group_agg_dict`).  See `groupby.aggregate.named` for more.  .. _whatsnew_0250.enhancements.multiple_lambdas:  GroupBy aggregation with multiple lambdas
^^^  You can now provide multiple lambda functions to a list-like aggregation in
`pandas.core.groupby.GroupBy.agg` (`26430`).  .. ipython:: python  animals.groupby('kind').height.agg([  lambda x: x.iloc[0], lambda x: x.iloc[-1]  ])  animals.groupby('kind').agg([  lambda x: x.iloc[0] - x.iloc[1],  lambda x: x.iloc[0] + x.iloc[1]  ])  Previously, these raised a ``SpecificationError``.  .. _whatsnew_0250.enhancements.multi_index_repr:  Better repr for MultiIndex
^^^  Printing of `MultiIndex` instances now shows tuples of each row and ensures
that the tuple items are vertically aligned, so it's now easier to understand
the structure of the ``MultiIndex``. (`13480`):  The repr now looks like this:  .. ipython:: python  pd.MultiIndex.from_product([['a', 'abc'], range(500)])  Previously, outputting a `MultiIndex` printed all the ``levels`` and
``codes`` of the ``MultiIndex``, which was visually unappealing and made
the output more difficult to navigate. For example (limiting the range to 5):  .. code-block:: ipython  In [1]: pd.MultiIndex.from_product([['a', 'abc'], range(5)])  Out[1]: MultiIndex(levels=[['a', 'abc'], [0, 1, 2, 3]],  ...:  codes=[[0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3]])  In the new repr, all values will be shown, if the number of rows is smaller
than `options.display.max_seq_items` (default: 100 items). Horizontally,
the output will truncate, if it's wider than `options.display.width`

(default: 80 characters).  .. _whatsnew_0250.enhancements.shorter_truncated_repr:  Shorter truncated repr for Series and DataFrame
^^^  Currently, the default display options of pandas ensure that when a Series
or DataFrame has more than 60 rows, its repr gets truncated to this maximum
of 60 rows (the ``display.max_rows`` option). However, this still gives
a repr that takes up a large part of the vertical screen estate. Therefore,
a new option ``display.min_rows`` is introduced with a default of 10 which
determines the number of rows showed in the truncated repr:  - For small Series or DataFrames, up to ``max_rows`` number of rows is shown  (default: 60).
- For larger Series of DataFrame with a length above ``max_rows``, only  ``min_rows`` number of rows is shown (default: 10, i.e. the first and last  5 rows).  This dual option allows to still see the full content of relatively small
objects (e.g. ``df.head(20)`` shows all 20 rows), while giving a brief repr
for large objects.  To restore the previous behaviour of a single threshold, set
``pd.options.display.min_rows = None``.  .. _whatsnew_0250.enhancements.json_normalize_with_max_level:  JSON normalize with max_level param support
^^^  `json_normalize` normalizes the provided input dict to all
nested levels. The new max_level parameter provides more control over
which level to end normalization (`23843`):  The repr now looks like this:  .. code-block:: ipython  from pandas.io.json import json_normalize  data = [{  'CreatedBy': {'Name': 'User001'},  'Lookup': {'TextField': 'Some text',  'UserField': {'Id': 'ID001', 'Name': 'Name001'}},  'Image': {'a': 'b'}  }]  json_normalize(data, max_level=1)  .. _whatsnew_0250.enhancements.explode:  Series.explode to split list-like values to rows
^^^  `Series` and `DataFrame` have gained the `DataFrame.explode` methods to transform list-likes to individual rows. See `section on Exploding list-like column <reshaping.explode>` in docs for more information (`16538`, `10511`)  Here is a typical usecase. You have comma separated string in a column.  .. ipython:: python  df = pd.DataFrame([{'var1': 'a,b,c', 'var2': 1},  {'var1': 'd,e,f', 'var2': 2}])  df  Creating a long form ``DataFrame`` is now straightforward using chained operations  .. ipython:: python  df.assign(var1=df.var1.str.split(',')).explode('var1')  .. _whatsnew_0250.enhancements.other:  Other enhancements
^^^
- `DataFrame.plot` keywords ``logy``, ``logx`` and ``loglog`` can now accept the value ``'sym'`` for symlog scaling. (`24867`)
- Added support for ISO week year format ('%G-%V-%u') when parsing datetimes using `to_datetime` (`16607`)
- Indexing of ``DataFrame`` and ``Series`` now accepts zerodim ``np.ndarray`` (`24919`)
- `Timestamp.replace` now supports the ``fold`` argument to disambiguate DST transition times (`25017`)
- `DataFrame.at_time` and `Series.at_time` now support `datetime.time` objects with timezones (`24043`)
- `DataFrame.pivot_table` now accepts an ``observed`` parameter which is passed to underlying calls to `DataFrame.groupby` to speed up grouping categorical data. (`24923`)
- ``Series.str`` has gained `Series.str.casefold` method to removes all case distinctions present in a string (`25405`)
- `DataFrame.set_index` now works for instances of ``abc.Iterator``, provided their output is of the same length as the calling frame (`22484`, `24984`)
- `DatetimeIndex.union` now supports the ``sort`` argument. The behavior of the sort parameter matches that of `Index.union` (`24994`)
- `RangeIndex.union` now supports the ``sort`` argument. If ``sort=False`` an unsorted ``Int64Index`` is always returned. ``sort=None`` is the default and returns a monotonically increasing ``RangeIndex`` if possible or a sorted ``Int64Index`` if not (`24471`)
- `TimedeltaIndex.intersection` now also supports the ``sort`` keyword (`24471`)

- `DataFrame.rename` now supports the ``errors`` argument to raise errors when attempting to rename nonexistent keys (`13473`)
- Added `api.frame.sparse` for working with a ``DataFrame`` whose values are sparse (`25681`)
- `RangeIndex` has gained `~RangeIndex.start`, `~RangeIndex.stop`, and `~RangeIndex.step` attributes (`25710`)
- `datetime.timezone` objects are now supported as arguments to timezone methods and constructors (`25065`)
- `DataFrame.query` and `DataFrame.eval` now supports quoting column names with backticks to refer to names with spaces (`6508`)
- `merge_asof` now gives a more clear error message when merge keys are categoricals that are not equal (`26136`)
- `pandas.core.window.Rolling` supports exponential (or Poisson) window type (`21303`)
- Error message for missing required imports now includes the original import error's text (`23868`)
- `DatetimeIndex` and `TimedeltaIndex` now have a ``mean`` method (`24757`)
- `DataFrame.describe` now formats integer percentiles without decimal point (`26660`)
- Added support for reading SPSS .sav files using `read_spss` (`26537`)
- Added new option ``plotting.backend`` to be able to select a plotting backend different than the existing ``matplotlib`` one. Use ``pandas.set_option('plotting.backend', '<backend-module>')`` where ``<backend-module`` is a library implementing the pandas plotting API (`14130`)
- `pandas.offsets.BusinessHour` supports multiple opening hours intervals (`15481`)
- `read_excel` can now use ``openpyxl`` to read Excel files via the ``engine='openpyxl'`` argument. This will become the default in a future release (`11499`)
- `pandas.io.excel.read_excel` supports reading OpenDocument tables. Specify ``engine='odf'`` to enable. Consult the `IO User Guide <io.ods>` for more details (`9070`)
- `Interval`, `IntervalIndex`, and `~arrays.IntervalArray` have gained an `~Interval.is_empty` attribute denoting if the given interval(s) are empty (`27219`)  .. _whatsnew_0250.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_0250.api_breaking.utc_offset_indexing:  Indexing with date strings with UTC offsets
^^^  Indexing a `DataFrame` or `Series` with a `DatetimeIndex` with a
date string with a UTC offset would previously ignore the UTC offset. Now, the UTC offset
is respected in indexing. (`24076`, `16785`)  .. ipython:: python  df = pd.DataFrame([0], index=pd.DatetimeIndex(['2019-01-01'], tz='US/Pacific'))  df  *Previous behavior*:  .. code-block:: ipython  In [3]: df['2019-01-01 000+04:00':'2019-01-01 0100+04:00']  Out[3]:  0  2019-01-01 000-08:00  0  *New behavior*:  .. ipython:: python  df['2019-01-01 1200+04:00':'2019-01-01 1300+04:00']  .. _whatsnew_0250.api_breaking.multi_indexing:  ``MultiIndex`` constructed from levels and codes
^^^  Constructing a `MultiIndex` with ``NaN`` levels or codes value < -1 was allowed previously.
Now, construction with codes value < -1 is not allowed and ``NaN`` levels' corresponding codes
would be reassigned as -1. (`19387`)  *Previous behavior*:  .. code-block:: ipython  In [1]: pd.MultiIndex(levels=[[np.nan, None, pd.NaT, 128, 2]],  ...:  codes=[[0, -1, 1, 2, 3, 4]])  ...:  Out[1]: MultiIndex(levels=[[nan, None, NaT, 128, 2]],  codes=[[0, -1, 1, 2, 3, 4]])  In [2]: pd.MultiIndex(levels=[[1, 2]], codes=[[0, -2]])  Out[2]: MultiIndex(levels=[[1, 2]],  codes=[[0, -2]])  *New behavior*:  .. ipython:: python  pd.MultiIndex(levels=[[np.nan, None, pd.NaT, 128, 2]],  codes=[[0, -1, 1, 2, 3, 4]])  pd.MultiIndex(levels=[[1, 2]], codes=[[0, -2]])  .. _whatsnew_0250.api_breaking.groupby_apply_first_group_once:  ``GroupBy.apply`` on ``DataFrame`` evaluates first group only once
^^^  The implementation of `DataFrameGroupBy.apply() <pandas.core.groupby.DataFrameGroupBy.apply>`
previously evaluated the supplied function consistently twice on the first group
to infer if it is safe to use a fast code path. Particularly for functions with

side effects, this was an undesired behavior and may have led to surprises. (`2936`, `2656`, `7739`, `10519`, `12155`, `20084`, `21417`)  Now every group is evaluated only a single time.  .. ipython:: python  df = pd.DataFrame({"a": ["x", "y"], "b": [1, 2]})  df  def func(group):  print(group.name)  return group  *Previous behavior*:  .. code-block:: python  In [3]: df.groupby('a').apply(func)  x  x  y  Out[3]:  a  b  0  x  1  1  y  2  *New behavior*:  .. code-block:: python  In [3]: df.groupby('a').apply(func)  x  y  Out[3]:  a  b  0  x  1  1  y  2  Concatenating sparse values
^^^  When passed DataFrames whose values are sparse, `concat` will now return a
`Series` or `DataFrame` with sparse values, rather than a `SparseDataFrame` (`25702`).  .. ipython:: python  df = pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])})  *Previous behavior*:  .. code-block:: ipython  In [2]: type(pd.concat([df, df]))  pandas.core.sparse.frame.SparseDataFrame  *New behavior*:  .. ipython:: python  type(pd.concat([df, df]))  This now matches the existing behavior of `concat` on ``Series`` with sparse values.
`concat` will continue to return a ``SparseDataFrame`` when all the values
are instances of ``SparseDataFrame``.  This change also affects routines using `concat` internally, like `get_dummies`,
which now returns a `DataFrame` in all cases (previously a ``SparseDataFrame`` was
returned if all the columns were dummy encoded, and a `DataFrame` otherwise).  Providing any ``SparseSeries`` or ``SparseDataFrame`` to `concat` will
cause a ``SparseSeries`` or ``SparseDataFrame`` to be returned, as before.  The ``.str``-accessor performs stricter type checks
^^^  Due to the lack of more fine-grained dtypes, `Series.str` so far only checked whether the data was
of ``object`` dtype. `Series.str` will now infer the dtype data *within* the Series; in particular,
``'bytes'``-only data will raise an exception (except for `Series.str.decode`, `Series.str.get`,
`Series.str.len`, `Series.str.slice`), see `23163`, `23011`, `23551`.  *Previous behavior*:  .. code-block:: python  In [1]: s = pd.Series(np.array(['a', 'ba', 'cba'], 'S'), dtype=object)  In [2]: s  Out[2]:  0  b'a'  1  b'ba'  2  b'cba'  dtype: object  In [3]: s.str.startswith(b'a')  Out[3]:  0  True  1  False  2  False  dtype: bool  *New behavior*:  .. ipython:: python  s = pd.Series(np.array(['a', 'ba', 'cba'], 'S'), dtype=object)  s  s.str.startswith(b'a')  .. _whatsnew_0250.api_breaking.groupby_categorical:  Categorical dtypes are preserved during GroupBy
^^^  Previously, columns that were categorical, but not the groupby key(s) would be converted to ``object`` dtype during groupby operations. pandas now will preserve these dtypes. (`18502`)  .. ipython:: python  cat = pd.Categorical(["foo", "bar", "bar", "qux"], ordered=True)  df = pd.DataFrame({'payload': [-1, -2, -1, -2], 'col': cat})  df  df.dtypes  *Previous Behavior*:  .. code-block:: python  In [5]: df.groupby('payload').first().col.dtype  Out[5]: dtype('O')  *New Behavior*:  .. ipython:: python  df.groupby('payload').first().col.dtype  .. _whatsnew_0250.api_breaking.incompatible_index_unions:  Incompatible Index type unions
^^^  When performing `Index.union` operations between objects of incompatible dtypes,
the result will be a base `Index` of dtype ``object``. This behavior holds true for
unions between `Index` objects that previously would have been prohibited. The dtype
of empty `Index` objects will now be evaluated before performing union operations
rather than simply returning the other `Index` object. `Index.union` can now be

considered commutative, such that ``A.union(B) == B.union(A)`` (`23525`).  *Previous behavior*:  .. code-block:: python  In [1]: pd.period_range('19910905', periods=2).union(pd.Int64Index([1, 2, 3]))  ...  ValueError: can only call with other PeriodIndex-ed objects  In [2]: pd.Index([], dtype=object).union(pd.Index([1, 2, 3]))  Out[2]: Int64Index([1, 2, 3], dtype='int64')  *New behavior*:  .. code-block:: python  In [3]: pd.period_range('19910905', periods=2).union(pd.Int64Index([1, 2, 3]))  Out[3]: Index([1991-09-05, 1991-09-06, 1, 2, 3], dtype='object')  In [4]: pd.Index([], dtype=object).union(pd.Index([1, 2, 3]))  Out[4]: Index([1, 2, 3], dtype='object')  Note that integer- and floating-dtype indexes are considered "compatible". The integer
values are coerced to floating point, which may result in loss of precision. See
`indexing.set_ops` for more.  ``DataFrame`` GroupBy ffill/bfill no longer return group labels
^^^  The methods ``ffill``, ``bfill``, ``pad`` and ``backfill`` of
`DataFrameGroupBy <pandas.core.groupby.DataFrameGroupBy>`
previously included the group labels in the return value, which was
inconsistent with other groupby transforms. Now only the filled values
are returned. (`21521`)  .. ipython:: python  df = pd.DataFrame({"a": ["x", "y"], "b": [1, 2]})  df  *Previous behavior*:  .. code-block:: python  In [3]: df.groupby("a").ffill()  Out[3]:  a  b  0  x  1  1  y  2  *New behavior*:  .. ipython:: python  df.groupby("a").ffill()  ``DataFrame`` describe on an empty Categorical / object column will return top and freq
^^^  When calling `DataFrame.describe` with an empty categorical / object
column, the 'top' and 'freq' columns were previously omitted, which was inconsistent with
the output for non-empty columns. Now the 'top' and 'freq' columns will always be included,
with `numpy.nan` in the case of an empty `DataFrame` (`26397`)  .. ipython:: python  df = pd.DataFrame({"empty_col": pd.Categorical([])})  df  *Previous behavior*:  .. code-block:: python  In [3]: df.describe()  Out[3]:  empty_col  count  0  unique  0  *New behavior*:  .. ipython:: python  df.describe()  ``__str__`` methods now call ``__repr__`` rather than vice versa
^^^  pandas has until now mostly defined string representations in a pandas objects'
``__str__``/``__unicode__``/``__bytes__`` methods, and called ``__str__`` from the ``__repr__``
method, if a specific ``__repr__`` method is not found. This is not needed for Python3.
In pandas 0.25, the string representations of pandas objects are now generally
defined in ``__repr__``, and calls to ``__str__`` in general now pass the call on to
the ``__repr__``, if a specific ``__str__`` method doesn't exist, as is standard for Python.
This change is backward compatible for direct usage of pandas, but if you subclass
pandas objects *and* give your subclasses specific ``__str__``/``__repr__`` methods,
you may have to adjust your ``__str__``/``__repr__`` methods (`26495`).  .. _whatsnew_0250.api_breaking.interval_indexing:  Indexing an ``IntervalIndex`` with ``Interval`` objects
^^^  Indexing methods for `IntervalIndex` have been modified to require exact matches only for `Interval` queries.
``IntervalIndex`` methods previously matched on any overlapping ``Interval``.  Behavior with scalar points, e.g. querying
with an integer, is unchanged (`16316`).  .. ipython:: python  ii = pd.IntervalIndex.from_tuples([(0, 4), (1, 5), (5, 8)])  ii  The ``in`` operator (``__contains__``) now only returns ``True`` for exact matches to ``Intervals`` in the ``IntervalIndex``, whereas

this would previously return ``True`` for any ``Interval`` overlapping an ``Interval`` in the ``IntervalIndex``.  *Previous behavior*:  .. code-block:: python  In [4]: pd.Interval(1, 2, closed='neither') in ii  Out[4]: True  In [5]: pd.Interval(-10, 10, closed='both') in ii  Out[5]: True  *New behavior*:  .. ipython:: python  pd.Interval(1, 2, closed='neither') in ii  pd.Interval(-10, 10, closed='both') in ii  The `~IntervalIndex.get_loc` method now only returns locations for exact matches to ``Interval`` queries, as opposed to the previous behavior of
returning locations for overlapping matches.  A ``KeyError`` will be raised if an exact match is not found.  *Previous behavior*:  .. code-block:: python  In [6]: ii.get_loc(pd.Interval(1, 5))  Out[6]: array([0, 1])  In [7]: ii.get_loc(pd.Interval(2, 6))  Out[7]: array([0, 1, 2])  *New behavior*:  .. code-block:: python  In [6]: ii.get_loc(pd.Interval(1, 5))  Out[6]: 1

In [7]: ii.get_loc(pd.Interval(2, 6))
  KeyError: Interval(2, 6, closed='right')  Likewise, `~IntervalIndex.get_indexer` and `~IntervalIndex.get_indexer_non_unique` will also only return locations for exact matches
to ``Interval`` queries, with ``-1`` denoting that an exact match was not found.  These indexing changes extend to querying a `Series` or `DataFrame` with an ``IntervalIndex`` index.  .. ipython:: python  s = pd.Series(list('abc'), index=ii)  s  Selecting from a ``Series`` or ``DataFrame`` using ``[]`` (``__getitem__``) or ``loc`` now only returns exact matches for ``Interval`` queries.  *Previous behavior*:  .. code-block:: python  In [8]: s[pd.Interval(1, 5)]  Out[8]:  (0, 4]  a  (1, 5]  b  dtype: object  In [9]: s.loc[pd.Interval(1, 5)]  Out[9]:  (0, 4]  a  (1, 5]  b  dtype: object  *New behavior*:  .. ipython:: python  s[pd.Interval(1, 5)]  s.loc[pd.Interval(1, 5)]  Similarly, a ``KeyError`` will be raised for non-exact matches instead of returning overlapping matches.  *Previous behavior*:  .. code-block:: python  In [9]: s[pd.Interval(2, 3)]  Out[9]:  (0, 4]  a  (1, 5]  b  dtype: object  In [10]: s.loc[pd.Interval(2, 3)]  Out[10]:  (0, 4]  a  (1, 5]  b  dtype: object  *New behavior*:  .. code-block:: python

In [6]: s[pd.Interval(2, 3)]
  KeyError: Interval(2, 3, closed='right')

In [7]: s.loc[pd.Interval(2, 3)]
  KeyError: Interval(2, 3, closed='right')  The `~IntervalIndex.overlaps` method can be used to create a boolean indexer that replicates the
previous behavior of returning overlapping matches.  *New behavior*:  .. ipython:: python  idxr = s.index.overlaps(pd.Interval(2, 3))  idxr  s[idxr]  s.loc[idxr]  .. _whatsnew_0250.api_breaking.ufunc:  Binary ufuncs on Series now align
^^^  Applying a binary ufunc like `numpy.power` now aligns the inputs
when both are `Series` (`23293`).  .. ipython:: python  s1 = pd.Series([1, 2, 3], index=['a', 'b', 'c'])  s2 = pd.Series([3, 4, 5], index=['d', 'c', 'b'])  s1  s2  *Previous behavior*  .. code-block:: ipython  In [5]: np.power(s1, s2)  Out[5]:  a  1  b  16  c  243  dtype: int64  *New behavior*  .. ipython:: python  np.power(s1, s2)  This matches the behavior of other binary operations in pandas, like `Series.add`.
To retain the previous behavior, convert the other ``Series`` to an array before
applying the ufunc.  .. ipython:: python  np.power(s1, s2.array)  Categorical.argsort now places missing values at the end
^^^  `Categorical.argsort` now places missing values at the end of the array, making it
consistent with NumPy and the rest of pandas (`21801`).  .. ipython:: python  cat = pd.Categorical(['b', None, 'a'], categories=['a', 'b'], ordered=True)  *Previous behavior*  .. code-block:: ipython  In [2]: cat = pd.Categorical(['b', None, 'a'], categories=['a', 'b'], ordered=True)  In [3]: cat.argsort()  Out[3]: array([1, 2, 0])  In [4]: cat[cat.argsort()]  Out[4]:  [NaN, a, b]  categories (2, object): [a < b]  *New behavior*  .. ipython:: python  cat.argsort()  cat[cat.argsort()]  .. _whatsnew_0250.api_breaking.list_of_dict:  Column order is preserved when passing a list of dicts to DataFrame
^^^  Starting with Python 3.7 the key-order of `dict`` is `guaranteed. In practice, this has been true since
Python 3.6. The `DataFrame` constructor now treats a list of dicts in the same way as
it does a list of ``OrderedDict``, i.e. preserving the order of the dicts.
This change applies only when pandas is running on Python>=3.6 (`27309`).  .. ipython:: python  data = [  {'name': 'Joe', 'state': 'NY', 'age': 18},  {'name': 'Jane', 'state': 'KY', 'age': 19, 'hobby': 'Minecraft'},  {'name': 'Jean', 'state': 'OK', 'age': 20, 'finances': 'good'}  ]  *Previous Behavior*:  The columns were lexicographically sorted previously,  .. code-block:: python  In [1]: pd.DataFrame(data)  Out[1]:  age finances  hobby  name state  0  18  NaN  NaN  Joe  NY  1  19  NaN  Minecraft  Jane  KY  2  20  good  NaN  Jean  OK  *New Behavior*:  The column order now matches the insertion-order of the keys in the ``dict``,
considering all the records from top to bottom. As a consequence, the column
order of the resulting DataFrame has changed compared to previous pandas versions.  .. ipython:: python  pd.DataFrame(data)  .. _whatsnew_0250.api_breaking.deps:  Increased minimum versions for dependencies
^^^  Due to dropping support for Python 2.7, a number of optional dependencies have updated minimum versions (`25725`, `24942`, `25752`).
Independently, some minimum supported versions of dependencies were updated (`23519`, `25554`).
| numpy  | 1.13.3  |  X  |
| pytz  | 2015.4  |  X  |
| python-dateutil | 2.6.1  |  X  |
| bottleneck  | 1.2.1  |  |
| numexpr  | 2.6.2  |  |
| pytest (dev)  | 4.0.2  |  |
+---+---+---+  For optional libraries the general recommendation is to use the latest version.
The following table lists the lowest version per library that is currently being tested throughout the development of pandas.
Optional libraries below the lowest tested version may still work, but are not considered supported.  | Package  | Minimum Version |
+===+===+

+===+===+
| beautifulsoup4  | 4.6.0  |  | fastparquet  | 0.2.1  |  | gcsfs  | 0.2.2  |  | lxml  | 3.8.0  |  | matplotlib  | 2.2.2  |  | openpyxl  | 2.4.8  |  | pyarrow  | 0.9.0  |  | pymysql  | 0.7.1  |  | pytables  | 3.4.2  |  | scipy  | 0.19.0  |  | sqlalchemy  | 1.1.4  |  | xarray  | 0.8.2  |  | xlrd  | 1.1.0  |  | xlsxwriter  | 0.9.8  |  | xlwt  | 1.2.0  |  See `install.dependencies` and `install.optional_dependencies` for more.  .. _whatsnew_0250.api.other:  Other API changes
^^^  - `DatetimeTZDtype` will now standardize pytz timezones to a common timezone instance (`24713`)
- `Timestamp` and `Timedelta` scalars now implement the `to_numpy` method as aliases to `Timestamp.to_datetime64` and `Timedelta.to_timedelta64`, respectively. (`24653`)
- `Timestamp.strptime` will now rise a ``NotImplementedError`` (`25016`)
- Comparing `Timestamp` with unsupported objects now returns obj:`NotImplemented` instead of raising ``TypeError``. This implies that unsupported rich comparisons are delegated to the other object, and are now consistent with Python 3 behavior for ``datetime`` objects (`24011`)
- Bug in `DatetimeIndex.snap` which didn't preserving the ``name`` of the input `Index` (`25575`)
- The ``arg`` argument in `pandas.core.groupby.DataFrameGroupBy.agg` has been renamed to ``func`` (`26089`)
- The ``arg`` argument in `pandas.core.window._Window.aggregate` has been renamed to ``func`` (`26372`)
- Most pandas classes had a ``__bytes__`` method, which was used for getting a python2-style bytestring representation of the object. This method has been removed as a part of dropping Python2 (`26447`)
- The ``.str``-accessor has been disabled for 1-level `MultiIndex`, use `MultiIndex.to_flat_index` if necessary (`23679`)
- Removed support of gtk package for clipboards (`26563`)
- Using an unsupported version of Beautiful Soup 4 will now raise an ``ImportError`` instead of a ``ValueError`` (`27063`)
- `Series.to_excel` and `DataFrame.to_excel` will now raise a ``ValueError`` when saving timezone aware data. (`27008`, `7056`)
- `ExtensionArray.argsort` places NA values at the end of the sorted array. (`21801`)
- `DataFrame.to_hdf` and `Series.to_hdf` will now raise a ``NotImplementedError`` when saving a `MultiIndex` with extension data types for a ``fixed`` format. (`7775`)
- Passing duplicate ``names`` in `read_csv` will now raise a ``ValueError`` (`17346`)  .. _whatsnew_0250.deprecations:  Deprecations
~~~  Sparse subclasses
^^^  The ``SparseSeries`` and ``SparseDataFrame`` subclasses are deprecated. Their functionality is better-provided
by a ``Series`` or ``DataFrame`` with sparse values.  **Previous way**  .. code-block:: python  df = pd.SparseDataFrame({"A": [0, 0, 1, 2]})  df.dtypes  **New way**  .. ipython:: python  df = pd.DataFrame({"A": pd.arrays.SparseArray([0, 0, 1, 2])})  df.dtypes  The memory usage of the two approaches is identical. See `sparse.migration` for more (`19239`).  msgpack format
^^^  The msgpack format is deprecated as of 0.25 and will be removed in a future version. It is recommended to use pyarrow for on-the-wire transmission of pandas objects. (`27084`)  Other deprecations
^^^  - The deprecated ``.ix[]`` indexer now raises a more visible ``FutureWarning`` instead of ``DeprecationWarning`` (`26438`).
- Deprecated the ``units=M`` (months) and ``units=Y`` (year) parameters for ``units`` of `pandas.to_timedelta`, `pandas.Timedelta` and `pandas.TimedeltaIndex` (`16344`)
- `pandas.concat` has deprecated the ``join_axes``-keyword. Instead, use `DataFrame.reindex` or `DataFrame.reindex_like` on the result or on the inputs (`21951`)
- The `SparseArray.values` attribute is deprecated. You can use ``np.asarray(...)`` or  the `SparseArray.to_dense` method instead (`26421`).

- The functions `pandas.to_datetime` and `pandas.to_timedelta` have deprecated the ``box`` keyword. Instead, use `to_numpy` or `Timestamp.to_datetime64` or `Timedelta.to_timedelta64`. (`24416`)
- The `DataFrame.compound` and `Series.compound` methods are deprecated and will be removed in a future version (`26405`).
- The internal attributes ``_start``, ``_stop`` and ``_step`` attributes of `RangeIndex` have been deprecated.  Use the public attributes `~RangeIndex.start`, `~RangeIndex.stop` and `~RangeIndex.step` instead (`26581`).
- The `Series.ftype`, `Series.ftypes` and `DataFrame.ftypes` methods are deprecated and will be removed in a future version.  Instead, use `Series.dtype` and `DataFrame.dtypes` (`26705`).
- The `Series.get_values`, `DataFrame.get_values`, `Index.get_values`,  `SparseArray.get_values` and `Categorical.get_values` methods are deprecated.  One of ``np.asarray(..)`` or `~Series.to_numpy` can be used instead (`19617`).
- The 'outer' method on NumPy ufuncs, e.g. ``np.subtract.outer`` has been deprecated on `Series` objects. Convert the input to an array with `Series.array` first (`27186`)
- `Timedelta.resolution` is deprecated and replaced with `Timedelta.resolution_string`.  In a future version, `Timedelta.resolution` will be changed to behave like the standard library `datetime.timedelta.resolution` (`21344`)
- `read_table` has been undeprecated. (`25220`)
- `Index.dtype_str` is deprecated. (`18262`)
- `Series.imag` and `Series.real` are deprecated. (`18262`)
- `Series.put` is deprecated. (`18262`)
- `Index.item` and `Series.item` is deprecated. (`18262`)
- The default value ``ordered=None`` in `~pandas.api.types.CategoricalDtype` has been deprecated in favor of ``ordered=False``. When converting between categorical types ``ordered=True`` must be explicitly passed in order to be preserved. (`26336`)
- `Index.contains` is deprecated. Use ``key in index`` (``__contains__``) instead (`17753`).
- `DataFrame.get_dtype_counts` is deprecated. (`18262`)
- `Categorical.ravel` will return a `Categorical` instead of a ``np.ndarray`` (`27199`)  .. _whatsnew_0250.prior_deprecations:  Removal of prior version deprecations/changes
~~~
- Removed ``Panel`` (`25047`, `25191`, `25231`)
- Removed the previously deprecated ``sheetname`` keyword in `read_excel` (`16442`, `20938`)
- Removed the previously deprecated ``TimeGrouper`` (`16942`)
- Removed the previously deprecated ``parse_cols`` keyword in `read_excel` (`16488`)
- Removed the previously deprecated ``pd.options.html.border`` (`16970`)
- Removed the previously deprecated ``convert_objects`` (`11221`)
- Removed the previously deprecated ``select`` method of ``DataFrame`` and ``Series`` (`17633`)
- Removed the previously deprecated behavior of `Series` treated as list-like in `~Series.cat.rename_categories` (`17982`)
- Removed the previously deprecated ``DataFrame.reindex_axis`` and ``Series.reindex_axis`` (`17842`)
- Removed the previously deprecated behavior of altering column or index labels with `Series.rename_axis` or `DataFrame.rename_axis` (`17842`)
- Removed the previously deprecated ``tupleize_cols`` keyword argument in `read_html`, `read_csv`, and `DataFrame.to_csv` (`17877`, `17820`)
- Removed the previously deprecated ``DataFrame.from.csv`` and ``Series.from_csv`` (`17812`)
- Removed the previously deprecated ``raise_on_error`` keyword argument in `DataFrame.where` and `DataFrame.mask` (`17744`)
- Removed the previously deprecated ``ordered`` and ``categories`` keyword arguments in ``astype`` (`17742`)
- Removed the previously deprecated ``cdate_range`` (`17691`)
- Removed the previously deprecated ``True`` option for the ``dropna`` keyword argument in `SeriesGroupBy.nth` (`17493`)
- Removed the previously deprecated ``convert`` keyword argument in `Series.take` and `DataFrame.take` (`17352`)

- Removed the previously deprecated behavior of arithmetic operations with ``datetime.date`` objects (`21152`)  .. _whatsnew_0250.performance:  Performance improvements
~~~  - Significant speedup in `SparseArray` initialization that benefits most operations, fixing performance regression introduced in v0.20.0 (`24985`)
- `DataFrame.to_stata()` is now faster when outputting data with any string or non-native endian columns (`25045`)
- Improved performance of `Series.searchsorted`. The speedup is especially large when the dtype is  int8/int16/int32 and the searched key is within the integer bounds for the dtype (`22034`)
- Improved performance of `pandas.core.groupby.GroupBy.quantile` (`20405`)
- Improved performance of slicing and other selected operation on a `RangeIndex` (`26565`, `26617`, `26722`)
- `RangeIndex` now performs standard lookup without instantiating an actual hashtable, hence saving memory (`16685`)
- Improved performance of `read_csv` by faster tokenizing and faster parsing of small float numbers (`25784`)
- Improved performance of `read_csv` by faster parsing of N/A and boolean values (`25804`)
- Improved performance of `IntervalIndex.is_monotonic`, `IntervalIndex.is_monotonic_increasing` and `IntervalIndex.is_monotonic_decreasing` by removing conversion to `MultiIndex` (`24813`)
- Improved performance of `DataFrame.to_csv` when writing datetime dtypes (`25708`)
- Improved performance of `read_csv` by much faster parsing of ``MM/YYY`` and ``DD/MM/YYY`` datetime formats (`25922`)
- Improved performance of nanops for dtypes that cannot store NaNs. Speedup is particularly prominent for `Series.all` and `Series.any` (`25070`)
- Improved performance of `Series.map` for dictionary mappers on categorical series by mapping the categories instead of mapping all values (`23785`)
- Improved performance of `IntervalIndex.intersection` (`24813`)
- Improved performance of `read_csv` by faster concatenating date columns without extra conversion to string for integer/float zero and float ``NaN``; by faster checking the string for the possibility of being a date (`25754`)
- Improved performance of `IntervalIndex.is_unique` by removing conversion to ``MultiIndex`` (`24813`)
- Restored performance of `DatetimeIndex.__iter__` by re-enabling specialized code path (`26702`)
- Improved performance when building `MultiIndex` with at least one `CategoricalIndex` level (`22044`)
- Improved performance by removing the need for a garbage collect when checking for ``SettingWithCopyWarning`` (`27031`)
- For `to_datetime` changed default value of cache parameter to ``True`` (`26043`)
- Improved performance of `DatetimeIndex` and `PeriodIndex` slicing given non-unique, monotonic data (`27136`).
- Improved performance of `pd.read_json` for index-oriented data. (`26773`)
- Improved performance of `MultiIndex.shape` (`27384`).  .. _whatsnew_0250.bug_fixes:  Bug fixes
^^^  - Bug in `DataFrame.at` and `Series.at` that would raise exception if the index was a `CategoricalIndex` (`20629`)
- Fixed bug in comparison of ordered `Categorical` that contained missing values with a scalar which sometimes incorrectly resulted in ``True`` (`26504`)
- Bug in `DataFrame.dropna` when the `DataFrame` has a `CategoricalIndex` containing `Interval` objects incorrectly raised a ``TypeError`` (`25087`)  Datetimelike
^^^  - Bug in `to_datetime` which would raise an (incorrect) ``ValueError`` when called with a date far into the future and the ``format`` argument specified instead of raising ``OutOfBoundsDatetime`` (`23830`)
- Bug in `to_datetime` which would raise ``InvalidIndexError: Reindexing only valid with uniquely valued Index objects`` when called with ``cache=True``, with ``arg`` including at least two different elements from the set ``{None, numpy.nan, pandas.NaT}`` (`22305`)

- Bug in `DataFrame` and `Series` where timezone aware data with ``dtype='datetime64[ns]`` was not cast to naive (`25843`)
- Improved `Timestamp` type checking in various datetime functions to prevent exceptions when using a subclassed ``datetime`` (`25851`)
- Bug in `Series` and `DataFrame` repr where ``np.datetime64('NaT')`` and ``np.timedelta64('NaT')`` with ``dtype=object`` would be represented as ``NaN`` (`25445`)
- Bug in `to_datetime` which does not replace the invalid argument with ``NaT`` when error is set to coerce (`26122`)
- Bug in adding `DateOffset` with nonzero month to `DatetimeIndex` would raise ``ValueError`` (`26258`)
- Bug in `to_datetime` which raises unhandled ``OverflowError`` when called with mix of invalid dates and ``NaN`` values with ``format='%Y%m%d'`` and ``error='coerce'`` (`25512`)
- Bug in `isin` for datetimelike indexes; `DatetimeIndex`, `TimedeltaIndex` and `PeriodIndex` where the ``levels`` parameter was ignored. (`26675`)
- Bug in `to_datetime` which raises ``TypeError`` for ``format='%Y%m%d'`` when called for invalid integer dates with length >= 6 digits with ``errors='ignore'``
- Bug when comparing a `PeriodIndex` against a zero-dimensional numpy array (`26689`)
- Bug in constructing a ``Series`` or ``DataFrame`` from a numpy ``datetime64`` array with a non-ns unit and out-of-bound timestamps generating rubbish data, which will now correctly raise an ``OutOfBoundsDatetime`` error (`26206`).
- Bug in `date_range` with unnecessary ``OverflowError`` being raised for very large or very small dates (`26651`)
- Bug where adding `Timestamp` to a ``np.timedelta64`` object would raise instead of returning a `Timestamp` (`24775`)
- Bug where comparing a zero-dimensional numpy array containing a ``np.datetime64`` object to a `Timestamp` would incorrect raise ``TypeError`` (`26916`)

- Bug in `to_datetime` which would raise ``ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True`` when called with ``cache=True``, with ``arg`` including datetime strings with different offset (`26097`)

Timedelta
^^^  - Bug in `TimedeltaIndex.intersection` where for non-monotonic indices in some cases an empty ``Index`` was returned when in fact an intersection existed (`25913`)
- Bug with comparisons between `Timedelta` and ``NaT`` raising ``TypeError`` (`26039`)
- Bug when adding or subtracting a `BusinessHour` to a `Timestamp` with the resulting time landing in a following or prior day respectively (`26381`)
- Bug when comparing a `TimedeltaIndex` against a zero-dimensional numpy array (`26689`)  Timezones
^^^  - Bug in `DatetimeIndex.to_frame` where timezone aware data would be converted to timezone naive data (`25809`)
- Bug in `to_datetime` with ``utc=True`` and datetime strings that would apply previously parsed UTC offsets to subsequent arguments (`24992`)
- Bug in `Timestamp.tz_localize` and `Timestamp.tz_convert` does not propagate ``freq`` (`25241`)
- Bug in `Series.at` where setting `Timestamp` with timezone raises ``TypeError`` (`25506`)
- Bug in `DataFrame.update` when updating with timezone aware data would return timezone naive data (`25807`)
- Bug in `to_datetime` where an uninformative ``RuntimeError`` was raised when passing a naive `Timestamp` with datetime strings with mixed UTC offsets (`25978`)
- Bug in `to_datetime` with ``unit='ns'`` would drop timezone information from the parsed argument (`26168`)
- Bug in `DataFrame.join` where joining a timezone aware index with a timezone aware column would result in a column of ``NaN`` (`26335`)
- Bug in `date_range` where ambiguous or nonexistent start or end times were not handled by the ``ambiguous`` or ``nonexistent`` keywords respectively (`27088`)
- Bug in `DatetimeIndex.union` when combining a timezone aware and timezone unaware `DatetimeIndex` (`21671`)
- Bug when applying a numpy reduction function (e.g. `numpy.minimum`) to a timezone aware `Series` (`15552`)  Numeric
^^^  - Bug in `to_numeric` in which large negative numbers were being improperly handled (`24910`)
- Bug in `to_numeric` in which numbers were being coerced to float, even though ``errors`` was not ``coerce`` (`24910`)
- Bug in `to_numeric` in which invalid values for ``errors`` were being allowed (`26466`)
- Bug in `format` in which floating point complex numbers were not being formatted to proper display precision and trimming (`25514`)
- Bug in error messages in `DataFrame.corr` and `Series.corr`. Added the possibility of using a callable. (`25729`)
- Bug in `Series.divmod` and `Series.rdivmod` which would raise an (incorrect) ``ValueError`` rather than return a pair of `Series` objects as result (`25557`)
- Raises a helpful exception when a non-numeric index is sent to `interpolate` with methods which require numeric index. (`21662`)
- Bug in `~pandas.eval` when comparing floats with scalar operators, for example: ``x < -0.1`` (`25928`)
- Fixed bug where casting all-boolean array to integer extension array failed (`25211`)
- Bug in ``divmod`` with a `Series` object containing zeros incorrectly raising ``AttributeError`` (`26987`)

- Inconsistency in `Series` floor-division (`//`) and ``divmod`` filling positive//zero with ``NaN`` instead of ``Inf`` (`27321`)

^^^

- Bug in `DataFrame.astype()` when passing a dict of columns and types the ``errors`` parameter was ignored. (`25905`)

Strings
^^^  - Bug in the ``__name__`` attribute of several methods of `Series.str`, which were set incorrectly (`23551`)

- Improved error message when passing `Series` of wrong dtype to `Series.str.cat` (`22722`)
  Interval
^^^  - Construction of `Interval` is restricted to numeric, `Timestamp` and `Timedelta` endpoints (`23013`)
- Fixed bug in `Series`/`DataFrame` not displaying ``NaN`` in `IntervalIndex` with missing values (`25984`)
- Bug in `IntervalIndex.get_loc` where a ``KeyError`` would be incorrectly raised for a decreasing `IntervalIndex` (`25860`)
- Bug in `Index` constructor where passing mixed closed `Interval` objects would result in a ``ValueError`` instead of an ``object`` dtype ``Index`` (`27172`)  Indexing
^^^  - Improved exception message when calling `DataFrame.iloc` with a list of non-numeric objects (`25753`).
- Improved exception message when calling ``.iloc`` or ``.loc`` with a boolean indexer with different length (`26658`).
- Bug in ``KeyError`` exception message when indexing a `MultiIndex` with a non-existent key not displaying the original key (`27250`).
- Bug in ``.iloc`` and ``.loc`` with a boolean indexer not raising an ``IndexError`` when too few items are passed (`26658`).
- Bug in `DataFrame.loc` and `Series.loc` where ``KeyError`` was not raised for a ``MultiIndex`` when the key was less than or equal to the number of levels in the `MultiIndex` (`14885`).
- Bug in which `DataFrame.append` produced an erroneous warning indicating that a ``KeyError`` will be thrown in the future when the data to be appended contains new columns (`22252`).
- Bug in which `DataFrame.to_csv` caused a segfault for a reindexed data frame, when the indices were single-level `MultiIndex` (`26303`).
- Fixed bug where assigning a `arrays.PandasArray` to a `pandas.core.frame.DataFrame` would raise error (`26390`)
- Allow keyword arguments for callable local reference used in the `DataFrame.query` string (`26426`)
- Fixed a ``KeyError`` when indexing a `MultiIndex` level with a list containing exactly one label, which is missing (`27148`)
- Bug which produced ``AttributeError`` on partial matching `Timestamp` in a `MultiIndex`  (`26944`)
- Bug in `Categorical` and  `CategoricalIndex` with `Interval` values when using the ``in`` operator (``__contains``) with objects that are not comparable to the values in the ``Interval`` (`23705`)
- Bug in `DataFrame.loc` and `DataFrame.iloc` on a `DataFrame` with a single timezone-aware datetime64[ns] column incorrectly returning a scalar instead of a `Series` (`27110`)
- Bug in `CategoricalIndex` and `Categorical` incorrectly raising ``ValueError`` instead of ``TypeError`` when a list is passed using the ``in`` operator (``__contains__``) (`21729`)
- Bug in setting a new value in a `Series` with a `Timedelta` object incorrectly casting the value to an integer (`22717`)
- Bug in `Series` setting a new key (``__setitem__``) with a timezone-aware datetime incorrectly raising ``ValueError`` (`12862`)
- Bug in `DataFrame.iloc` when indexing with a read-only indexer (`17192`)
- Bug in `Series` setting an existing tuple key (``__setitem__``) with timezone-aware datetime values incorrectly raising ``TypeError`` (`20441`)  Missing
^^^  - Fixed misleading exception message in `Series.interpolate` if argument ``order`` is required, but omitted (`10633`, `24014`).
- Fixed class type displayed in exception message in `DataFrame.dropna` if invalid ``axis`` parameter passed (`2555`)

- A ``ValueError`` will now be thrown by `DataFrame.fillna` when ``limit`` is not a positive integer (`27042`)

^^^

- Bug in which incorrect exception raised by `Timedelta` when testing the membership of `MultiIndex` (`24570`)

IO
^^  - Bug in `DataFrame.to_html()` where values were truncated using display options instead of outputting the full content (`17004`)
- Fixed bug in missing text when using `to_clipboard` if copying utf-16 characters in Python 3 on Windows (`25040`)
- Bug in `read_json` for ``orient='table'`` when it tries to infer dtypes by default, which is not applicable as dtypes are already defined in the JSON schema (`21345`)
- Bug in `read_json` for ``orient='table'`` and float index, as it infers index dtype by default, which is not applicable because index dtype is already defined in the JSON schema (`25433`)
- Bug in `read_json` for ``orient='table'`` and string of float column names, as it makes a column name type conversion to `Timestamp`, which is not applicable because column names are already defined in the JSON schema (`25435`)
- Bug in `json_normalize` for ``errors='ignore'`` where missing values in the input data, were filled in resulting ``DataFrame`` with the string ``"nan"`` instead of ``numpy.nan`` (`25468`)
- `DataFrame.to_html` now raises ``TypeError`` when using an invalid type for the ``classes`` parameter instead of ``AssertionError`` (`25608`)
- Bug in `DataFrame.to_string` and `DataFrame.to_latex` that would lead to incorrect output when the ``header`` keyword is used (`16718`)
- Bug in `read_csv` not properly interpreting the UTF8 encoded filenames on Windows on Python 3.6+ (`15086`)
- Improved performance in `pandas.read_stata` and `pandas.io.stata.StataReader` when converting columns that have missing values (`25772`)
- Bug in `DataFrame.to_html` where header numbers would ignore display options when rounding (`17280`)
- Bug in `read_hdf` where reading a table from an HDF5 file written directly with PyTables fails with a ``ValueError`` when using a sub-selection via the ``start`` or ``stop`` arguments (`11188`)
- Bug in `read_hdf` not properly closing store after a ``KeyError`` is raised (`25766`)
- Improved the explanation for the failure when value labels are repeated in Stata dta files and suggested work-arounds (`25772`)
- Improved `pandas.read_stata` and `pandas.io.stata.StataReader` to read incorrectly formatted 118 format files saved by Stata (`25960`)
- Improved the ``col_space`` parameter in `DataFrame.to_html` to accept a string so CSS length values can be set correctly (`25941`)
- Fixed bug in loading objects from S3 that contain ``#`` characters in the URL (`25945`)
- Adds ``use_bqstorage_api`` parameter to `read_gbq` to speed up downloads of large data frames. This feature requires version 0.10.0 of the ``pandas-gbq`` library as well as the ``google-cloud-bigquery-storage`` and ``fastavro`` libraries. (`26104`)
- Fixed memory leak in `DataFrame.to_json` when dealing with numeric data (`24889`)
- Bug in `read_json` where date strings with ``Z`` were not converted to a UTC timezone (`26168`)
- Added ``cache_dates=True`` parameter to `read_csv`, which allows to cache unique dates when they are parsed (`25990`)
- `DataFrame.to_excel` now raises a ``ValueError`` when the caller's dimensions exceed the limitations of Excel (`26051`)
- Fixed bug in `pandas.read_csv` where a BOM would result in incorrect parsing using engine='python' (`26545`)
- `read_excel` now raises a ``ValueError`` when input is of type `pandas.io.excel.ExcelFile` and ``engine`` param is passed since `pandas.io.excel.ExcelFile` has an engine defined (`26566`)
- Bug while selecting from `HDFStore` with ``where=''`` specified (`26610`).
- Fixed bug in `DataFrame.to_excel()` where custom objects (i.e. ``PeriodIndex``) inside merged cells were not being converted into types safe for the Excel writer (`27006`)
- Bug in `read_hdf` where reading a timezone aware `DatetimeIndex` would raise a ``TypeError`` (`11926`)
- Bug in `to_msgpack` and `read_msgpack` which would raise a ``ValueError`` rather than a ``FileNotFoundError`` for an invalid path (`27160`)

- Fixed bug in `DataFrame.to_parquet` which would raise a ``ValueError`` when the dataframe had no columns (`27339`)
- Allow parsing of `PeriodDtype` columns when using `read_csv` (`26934`)  Plotting
^^^  - Fixed bug where `api.extensions.ExtensionArray` could not be used in matplotlib plotting (`25587`)
- Bug in an error message in `DataFrame.plot`. Improved the error message if non-numerics are passed to `DataFrame.plot` (`25481`)
- Bug in incorrect ticklabel positions when plotting an index that are non-numeric / non-datetime (`7612`, `15912`, `22334`)
- Fixed bug causing plots of `PeriodIndex` timeseries to fail if the frequency is a multiple of the frequency rule code (`14763`)

- Fixed bug when plotting a `DatetimeIndex` with ``datetime.timezone.utc`` timezone (`17173`)

GroupBy/resample/rolling
^^^  - Bug in `pandas.core.resample.Resampler.agg` with a timezone aware index where ``OverflowError`` would raise when passing a list of functions (`22660`)
- Bug in `pandas.core.groupby.DataFrameGroupBy.nunique` in which the names of column levels were lost (`23222`)
- Bug in `pandas.core.groupby.GroupBy.agg` when applying an aggregation function to timezone aware data (`23683`)
- Bug in `pandas.core.groupby.GroupBy.first` and `pandas.core.groupby.GroupBy.last` where timezone information would be dropped (`21603`)
- Bug in `pandas.core.groupby.GroupBy.size` when grouping only NA values (`23050`)
- Bug in `Series.groupby` where ``observed`` kwarg was previously ignored (`24880`)
- Bug in `Series.groupby` where using ``groupby`` with a `MultiIndex` Series with a list of labels equal to the length of the series caused incorrect grouping (`25704`)
- Ensured that ordering of outputs in ``groupby`` aggregation functions is consistent across all versions of Python (`25692`)
- Ensured that result group order is correct when grouping on an ordered ``Categorical`` and specifying ``observed=True`` (`25871`, `25167`)
- Bug in `pandas.core.window.Rolling.min` and `pandas.core.window.Rolling.max` that caused a memory leak (`25893`)
- Bug in `pandas.core.window.Rolling.count` and ``pandas.core.window.Expanding.count`` was previously ignoring the ``axis`` keyword (`13503`)
- Bug in `pandas.core.groupby.GroupBy.idxmax` and `pandas.core.groupby.GroupBy.idxmin` with datetime column would return incorrect dtype (`25444`, `15306`)
- Bug in `pandas.core.groupby.GroupBy.cumsum`, `pandas.core.groupby.GroupBy.cumprod`, `pandas.core.groupby.GroupBy.cummin` and `pandas.core.groupby.GroupBy.cummax` with categorical column having absent categories, would return incorrect result or segfault (`16771`)
- Bug in `pandas.core.groupby.GroupBy.nth` where NA values in the grouping would return incorrect results (`26011`)
- Bug in `pandas.core.groupby.SeriesGroupBy.transform` where transforming an empty group would raise a ``ValueError`` (`26208`)
- Bug in `pandas.core.frame.DataFrame.groupby` where passing a `pandas.core.groupby.grouper.Grouper` would return incorrect groups when using the ``.groups`` accessor (`26326`)
- Bug in `pandas.core.groupby.GroupBy.agg` where incorrect results are returned for uint64 columns. (`26310`)
- Bug in `pandas.core.window.Rolling.median` and `pandas.core.window.Rolling.quantile` where MemoryError is raised with empty window (`26005`)
- Bug in `pandas.core.window.Rolling.median` and `pandas.core.window.Rolling.quantile` where incorrect results are returned with ``closed='left'`` and ``closed='neither'`` (`26005`)
- Improved `pandas.core.window.Rolling`, `pandas.core.window.Window` and `pandas.core.window.ExponentialMovingWindow` functions to exclude nuisance columns from results instead of raising errors and raise a ``DataError`` only if all columns are nuisance (`12537`)
- Bug in `pandas.core.window.Rolling.max` and `pandas.core.window.Rolling.min` where incorrect results are returned with an empty variable window (`26005`)
- Raise a helpful exception when an unsupported weighted window function is used as an argument of `pandas.core.window.Window.aggregate` (`26597`)  Reshaping
^^^  - Bug in `pandas.merge` adds a string of ``None``, if ``None`` is assigned in suffixes instead of remain the column name as-is (`24782`).
- Bug in `merge` when merging by index name would sometimes result in an incorrectly numbered index (missing index values are now assigned NA) (`24212`, `25009`)
- `to_records` now accepts dtypes to its ``column_dtypes`` parameter (`24895`)
- Bug in `concat` where order of ``OrderedDict`` (and ``dict`` in Python 3.6+) is not respected, when passed in as  ``objs`` argument (`21510`)
- Bug in `pivot_table` where columns with ``NaN`` values are dropped even if ``dropna`` argument is ``False``, when the ``aggfunc`` argument contains a ``list`` (`22159`)

- Bug in `concat` where the resulting ``freq`` of two `DatetimeIndex` with the same ``freq`` would be dropped (`3232`).
- Bug in `merge` where merging with equivalent Categorical dtypes was raising an error (`22501`)
- bug in `DataFrame` instantiating with a dict of iterators or generators (e.g. ``pd.DataFrame({'A': reversed(range(3))})``) raised an error (`26349`).
- Bug in `DataFrame` instantiating with a ``range`` (e.g. ``pd.DataFrame(range(3))``) raised an error (`26342`).
- Bug in `DataFrame` constructor when passing non-empty tuples would cause a segmentation fault (`25691`)
- Bug in `Series.apply` failed when the series is a timezone aware `DatetimeIndex` (`25959`)
- Bug in `pandas.cut` where large bins could incorrectly raise an error due to an integer overflow (`26045`)
- Bug in `DataFrame.sort_index` where an error is thrown when a multi-indexed ``DataFrame`` is sorted on all levels with the initial level sorted last (`26053`)
- Bug in `Series.nlargest` treats ``True`` as smaller than ``False`` (`26154`)
- Bug in `DataFrame.pivot_table` with a `IntervalIndex` as pivot index would raise ``TypeError`` (`25814`)
- Bug in which `DataFrame.from_dict` ignored order of ``OrderedDict`` when ``orient='index'`` (`8425`).
- Bug in `DataFrame.transpose` where transposing a DataFrame with a timezone-aware datetime column would incorrectly raise ``ValueError`` (`26825`)
- Bug in `pivot_table` when pivoting a timezone aware column as the ``values`` would remove timezone information (`14948`)
- Bug in `merge_asof` when specifying multiple ``by`` columns where one is ``datetime64[ns, tz]`` dtype (`26649`)  Sparse
^^^  - Significant speedup in `SparseArray` initialization that benefits most operations, fixing performance regression introduced in v0.20.0 (`24985`)
- Bug in `SparseFrame` constructor where passing ``None`` as the data would cause ``default_fill_value`` to be ignored (`16807`)
- Bug in `SparseDataFrame` when adding a column in which the length of values does not match length of index, ``AssertionError`` is raised instead of raising ``ValueError`` (`25484`)
- Introduce a better error message in `Series.sparse.from_coo` so it returns a ``TypeError`` for inputs that are not coo matrices (`26554`)
- Bug in `numpy.modf` on a `SparseArray`. Now a tuple of `SparseArray` is returned (`26946`).  Build changes
^^^  - Fix install error with PyPy on macOS (`26536`)  ExtensionArray
^^^  - Bug in `factorize` when passing an ``ExtensionArray`` with a custom ``na_sentinel`` (`25696`).
- `Series.count` miscounts NA values in ExtensionArrays (`26835`)
- Added ``Series.__array_ufunc__`` to better handle NumPy ufuncs applied to Series backed by extension arrays (`23293`).
- Keyword argument ``deep`` has been removed from `ExtensionArray.copy` (`27083`)  Other
^^^  - Removed unused C functions from vendored UltraJSON implementation (`26198`)
- Allow `Index` and `RangeIndex` to be passed to numpy ``min`` and ``max`` functions (`26125`)
- Use actual class name in repr of empty objects of a ``Series`` subclass (`27001`).
- Bug in `DataFrame` where passing an object array of timezone-aware ``datetime`` objects would incorrectly raise ``ValueError`` (`13287`)  .. _whatsnew_0.250.contributors:  Contributors
~~~  .. contributors:: v0.24.2..v0.25.0

These are the changes in pandas 0.25.1. See `release` for a full changelog
including other versions of pandas.  IO and LZMA
~~~  Some users may unknowingly have an incomplete Python installation lacking the ``lzma`` module from the standard library. In this case, ``import pandas`` failed due to an ``ImportError`` (`27575`).
pandas will now warn, rather than raising an ``ImportError`` if the ``lzma`` module is not present. Any subsequent attempt to use ``lzma`` methods will raise a ``RuntimeError``.
A possible fix for the lack of the ``lzma`` module is to ensure you have the necessary libraries and then re-install Python.
For example, on MacOS installing Python with ``pyenv`` may lead to an incomplete Python installation due to unmet system dependencies at compilation time (like ``xz``). Compilation will succeed, but Python might fail at run time. The issue can be solved by installing the necessary dependencies and then re-installing Python.  .. _whatsnew_0251.bug_fixes:  Bug fixes
^^^  - Bug in `Categorical.fillna` that would replace all values, not just those that are ``NaN`` (`26215`)  Datetimelike
^^^  - Bug in `to_datetime` where passing a timezone-naive `DatetimeArray` or `DatetimeIndex` and ``utc=True`` would incorrectly return a timezone-naive result (`27733`)
- Bug in `Period.to_timestamp` where a `Period` outside the `Timestamp` implementation bounds (roughly 1677-09-21 to 2262-04-11) would return an incorrect `Timestamp` instead of raising ``OutOfBoundsDatetime`` (`19643`)
- Bug in iterating over `DatetimeIndex` when the underlying data is read-only (`28055`)  Timezones
^^^  - Bug in `Index` where a numpy object array with a timezone aware `Timestamp` and ``np.nan`` would not return a `DatetimeIndex` (`27011`)  Numeric
^^^  - Bug in `Series.interpolate` when using a timezone aware `DatetimeIndex` (`27548`)
- Bug when printing negative floating point complex numbers would raise an ``IndexError`` (`27484`)
- Bug where `DataFrame` arithmetic operators such as `DataFrame.mul` with a `Series` with axis=1 would raise an ``AttributeError`` on `DataFrame` larger than the minimum threshold to invoke numexpr (`27636`)
- Bug in `DataFrame` arithmetic where missing values in results were incorrectly masked with ``NaN`` instead of ``Inf`` (`27464`)  Conversion
^^^  - Improved the warnings for the deprecated methods `Series.real` and `Series.imag` (`27610`)  Interval
^^^  - Bug in `IntervalIndex` where ``dir(obj)`` would raise ``ValueError`` (`27571`)  Indexing
^^^  - Bug in partial-string indexing returning a NumPy array rather than a ``Series`` when indexing with a scalar like ``.loc['2015']`` (`27516`)
- Break reference cycle involving `Index` and other index classes to allow garbage collection of index objects without running the GC. (`27585`, `27840`)
- Fix regression in assigning values to a single column of a DataFrame with a ``MultiIndex`` columns (`27841`).
- Fix regression in ``.ix`` fallback with an ``IntervalIndex`` (`27865`).  Missing
^^^  - Bug in `pandas.isnull` or `pandas.isna` when the input is a type e.g. ``type(pandas.Series())`` (`27482`)  IO
^^  - Avoid calling ``S3File.s3`` when reading parquet, as this was removed in s3fs version 0.3.0 (`27756`)
- Better error message when a negative header is passed in `pandas.read_csv` (`27779`)
- Follow the ``min_rows`` display option (introduced in v0.25.0) correctly in the HTML repr in the notebook (`27991`).  Plotting
^^^  - Added a ``pandas_plotting_backends`` entrypoint group for registering plot backends. See `extending.plotting-backends` for more (`26747`).
- Fixed the re-instatement of Matplotlib datetime converters after calling  `pandas.plotting.deregister_matplotlib_converters` (`27481`).
- Fix compatibility issue with matplotlib when passing a pandas ``Index`` to a plot call (`27775`).  GroupBy/resample/rolling
^^^  - Fixed regression in `pands.core.groupby.DataFrameGroupBy.quantile` raising when multiple quantiles are given (`27526`)

- Bug in `pandas.core.groupby.DataFrameGroupBy.transform` where applying a timezone conversion lambda function would drop timezone information (`27496`)
- Bug in `pandas.core.groupby.GroupBy.nth` where ``observed=False`` was being ignored for Categorical groupers (`26385`)
- Bug in windowing over read-only arrays (`27766`)
- Fixed segfault in ``pandas.core.groupby.DataFrameGroupBy.quantile`` when an invalid quantile was passed (`27470`)  Reshaping
^^^  - A ``KeyError`` is now raised if ``.unstack()`` is called on a `Series` or `DataFrame` with a flat `Index` passing a name which is not the correct one (`18303`)
- Bug `merge_asof` could not merge `Timedelta` objects when passing ``tolerance`` kwarg (`27642`)
- Bug in `DataFrame.crosstab` when ``margins`` set to ``True`` and ``normalize`` is not ``False``, an error is raised. (`27500`)
- `DataFrame.join` now suppresses the ``FutureWarning`` when the sort parameter is specified (`21952`)
- Bug in `DataFrame.join` raising with readonly arrays (`27943`)  Sparse
^^^  - Bug in reductions for `Series` with Sparse dtypes (`27080`)  Other
^^^  - Bug in `Series.replace` and `DataFrame.replace` when replacing timezone-aware timestamps using a dict-like replacer (`27720`)
- Bug in `Series.rename` when using a custom type indexer. Now any value that isn't callable or dict-like is treated as a scalar. (`27814`)  .. _whatsnew_0.251.contributors:  Contributors
~~~  .. contributors:: v0.25.0..v0.25.1

What's new in 0.25.2 (October 15, 2019)

These are the changes in pandas 0.25.2. See `release` for a full changelog
including other versions of pandas.  .. note::  pandas 0.25.2 adds compatibility for Python 3.8 (`28147`).  .. _whatsnew_0252.bug_fixes:  Bug fixes
~~~  Indexing
^^^  - Fix regression in `DataFrame.reindex` not following the ``limit`` argument (`28631`).
- Fix regression in `RangeIndex.get_indexer` for decreasing `RangeIndex` where target values may be improperly identified as missing/present (`28678`)  IO
^^  - Fix regression in notebook display where ``<th>`` tags were missing for `DataFrame.index` values (`28204`).
- Regression in `~DataFrame.to_csv` where writing a `Series` or `DataFrame` indexed by an `IntervalIndex` would incorrectly raise a ``TypeError`` (`28210`)
- Fix `~DataFrame.to_csv` with ``ExtensionArray`` with list-like values (`28840`).  GroupBy/resample/rolling
^^^  - Bug incorrectly raising an ``IndexError`` when passing a list of quantiles to `pandas.core.groupby.DataFrameGroupBy.quantile` (`28113`).
- Bug in `pandas.core.groupby.GroupBy.shift`, `pandas.core.groupby.GroupBy.bfill` and `pandas.core.groupby.GroupBy.ffill` where timezone information would be dropped (`19995`, `27992`)  Other
^^^  - Compatibility with Python 3.8 in `DataFrame.query` (`27261`)
- Fix to ensure that tab-completion in an IPython console does not raise  warnings for deprecated attributes (`27900`).  .. _whatsnew_0.252.contributors:  Contributors
~~~  .. contributors:: v0.25.1..v0.25.2

What's new in 0.25.3 (October 31, 2019)

These are the changes in pandas 0.25.3. See `release` for a full changelog
including other versions of pandas.  .. _whatsnew_0253.bug_fixes:  Bug fixes
~~~  GroupBy/resample/rolling
^^^  - Bug in `DataFrameGroupBy.quantile` where NA values in the grouping could cause segfaults or incorrect results (`28882`)  Contributors
~~~  .. contributors:: v0.25.2..v0.25.3

Versions 0.4.1 through 0.4.3 (September 25 - October 9, 2011)

{{ header }}  New features
~~~  - Added Python 3 support using 2to3 (`200`)
- `Added <dsintro.name_attribute>` ``name`` attribute to ``Series``, now  prints as part of ``Series.__repr__``
- `Added <missing.isna>` instance methods ``isnull`` and ``notnull`` to  Series (`209`, `203`)
- `Added <basics.align>` ``Series.align`` method for aligning two series  with choice of join method (ENH56_)
- `Added <advanced.get_level_values>` method ``get_level_values`` to  ``MultiIndex`` (`188`)
- Set values in mixed-type ``DataFrame`` objects via ``.ix`` indexing attribute (`135`)
- Added new ``DataFrame`` `methods <basics.dtypes>`  ``get_dtype_counts`` and property ``dtypes`` (ENHdc_)
- Added `ignore_index <merging.ignore_index>` option to  ``DataFrame.append`` to stack DataFrames (ENH1b_)
- ``read_csv`` tries to `sniff <io.sniff>` delimiters using  ``csv.Sniffer`` (`146`)
- ``read_csv`` can `read <io.csv_multiindex>` multiple columns into a  ``MultiIndex``; DataFrame's ``to_csv`` method writes out a corresponding  ``MultiIndex`` (`151`)
- ``DataFrame.rename`` has a new ``copy`` parameter to `rename  <basics.rename>` a DataFrame in place (ENHed_)
- `Enable <reshaping.unstack_by_name>` unstacking by name (`142`)
- `Enable <advanced.sortlevel_byname>` ``sortlevel`` to work by level (`141`)  Performance enhancements
~~~  - Altered binary operations on differently-indexed SparseSeries objects  to use the integer-based (dense) alignment logic which is faster with a  larger number of blocks (`205`)
- Wrote faster Cython data alignment / merging routines resulting in  substantial speed increases
- Improved performance of ``isnull`` and ``notnull``, a regression from v0.3.0  (`187`)
- Refactored code related to ``DataFrame.join`` so that intermediate aligned  copies of the data in each ``DataFrame`` argument do not need to be created.  Substantial performance increases result (`176`)
- Substantially improved performance of generic ``Index.intersection`` and  ``Index.union``
- Implemented ``BlockManager.take`` resulting in significantly faster ``take``  performance on mixed-type ``DataFrame`` objects (`104`)
- Improved performance of ``Series.sort_index``
- Significant groupby performance enhancement: removed unnecessary integrity  checks in DataFrame internals that were slowing down slicing operations to  retrieve groups
- Optimized ``_ensure_index`` function resulting in performance savings in  type-checking Index objects
- Wrote fast time series merging / joining methods in Cython. Will be  integrated later into DataFrame.join and related functions  .. _ENH1b: https://github.com/pandas-dev/pandas/commit/1ba56251f0013ff7cd8834e9486cef2b10098371
.. _ENHdc: https://github.com/pandas-dev/pandas/commit/dca3c5c5a6a3769ee01465baca04cfdfa66a4f76
.. _ENHed: https://github.com/pandas-dev/pandas/commit/edd9f1945fc010a57fa0ae3b3444d1fffe592591
.. _ENH56: https://github.com/pandas-dev/pandas/commit/56e0c9ffafac79ce262b55a6a13e1b10a88fbe93  Contributors
~~~  .. contributors:: v0.4.1..v0.4.3

Version 0.5.0 (October 24, 2011)

~~~  - `Added <basics.df_join>` ``DataFrame.align`` method with standard join options
- `Added <io.parse_dates>` ``parse_dates`` option to ``read_csv`` and ``read_table`` methods to optionally try to parse dates in the index columns
- `Added <io.parse_dates>` ``nrows``, ``chunksize``, and ``iterator`` arguments to ``read_csv`` and ``read_table``. The last two return a new ``TextParser`` class capable of lazily iterating through chunks of a flat file (`242`)
- `Added <merging.multikey_join>` ability to join on multiple columns in ``DataFrame.join`` (`214`)
- Added private ``_get_duplicates`` function to ``Index`` for identifying duplicate values more easily (ENH5c_)
- `Added <indexing.df_cols>` column attribute access to DataFrame.
- `Added <indexing.df_cols>` Python tab completion hook for DataFrame columns. (`233`, `230`)
- `Implemented <basics.describe>` ``Series.describe`` for Series containing objects (`241`)
- `Added <merging.df_inner_join>` inner join option to ``DataFrame.join`` when joining on key(s) (`248`)
- `Implemented <indexing.df_cols>` selecting DataFrame columns by passing a list to ``__getitem__`` (`253`)
- `Implemented <indexing.set_ops>` & and | to intersect / union Index objects, respectively (`261`)
- `Added<reshaping.pivot>` ``pivot_table`` convenience function to pandas namespace (`234`)
- `Implemented <basics.rename_axis>` ``Panel.rename_axis`` function (`243`)
- DataFrame will show index level names in console output (`334`)
- `Implemented <advanced.take>` ``Panel.take``
- `Added<basics.console_output>` ``set_eng_float_format`` for alternate DataFrame floating point string formatting (ENH61_)
- `Added <indexing.set_index>` convenience ``set_index`` function for creating a DataFrame index from its existing columns
- `Implemented <groupby.multiindex>` ``groupby`` hierarchical index level name  (`223`)
- `Added <io.store_in_csv>` support for different delimiters in ``DataFrame.to_csv`` (`244`)  Performance enhancements
~~~  - VBENCH Major performance improvements in file parsing functions ``read_csv`` and ``read_table``
- VBENCH Added Cython function for converting tuples to ndarray very fast. Speeds up many MultiIndex-related operations
- VBENCH Refactored merging / joining code into a tidy class and disabled unnecessary computations in the float/object case, thus getting about 10% better performance (`211`)
- VBENCH Improved speed of ``DataFrame.xs`` on mixed-type DataFrame objects by about 5x, regression from 0.3.0 (`215`)
- VBENCH With new ``DataFrame.align`` method, speeding up binary operations between differently-indexed DataFrame objects by 10-25%.
- VBENCH Significantly sped up conversion of nested dict into DataFrame (`212`)
- VBENCH Significantly speed up DataFrame ``__repr__`` and ``count`` on large mixed-type DataFrame objects  .. _ENH61: https://github.com/pandas-dev/pandas/commit/6141961
.. _ENH5c: https://github.com/pandas-dev/pandas/commit/5ca6ff5d822ee4ddef1ec0d87b6d83d8b4bbd3eb  .. _whatsnew_0.5.0.contributors:  Contributors
~~~  .. contributors:: v0.4.0..v0.5.0

~~~
- `Added <reshaping.melt>` ``melt`` function to ``pandas.core.reshape``
- `Added <groupby.multiindex>` ``level`` parameter to group by level in Series and DataFrame descriptive statistics (`313`)
- `Added <basics.head_tail>` ``head`` and ``tail`` methods to Series, analogous to DataFrame (`296`)
- `Added <indexing.boolean>` ``Series.isin`` function which checks if each value is contained in a passed sequence (`289`)
- `Added <io.formatting>` ``float_format`` option to ``Series.to_string``
- `Added <io.parse_dates>` ``skip_footer`` (`291`) and ``converters`` (`343`) options to ``read_csv`` and ``read_table``
- `Added <indexing.duplicate>` ``drop_duplicates`` and ``duplicated`` functions for removing duplicate DataFrame rows and checking for duplicate rows, respectively (`319`)
- `Implemented <dsintro.boolean>` operators '&', '|', '^', '-' on DataFrame (`347`)
- `Added <basics.stats>` ``Series.mad``, mean absolute deviation
- `Added <timeseries.offsets>` ``QuarterEnd`` DateOffset (`321`)
- `Added <dsintro.numpy_interop>` ``dot`` to DataFrame (`65`)
- Added ``orient`` option to ``Panel.from_dict`` (`359`, `301`)
- `Added <basics.dataframe.from_dict>` ``orient`` option to ``DataFrame.from_dict``
- `Added <basics.dataframe.from_records>` passing list of tuples or list of lists to ``DataFrame.from_records`` (`357`)
- `Added <groupby.multiindex>` multiple levels to groupby (`103`)
- `Allow <basics.sorting>` multiple columns in ``by`` argument of ``DataFrame.sort_index`` (`92`, `362`)
- `Added <indexing.basics.get_value>` fast ``get_value`` and ``put_value`` methods to DataFrame (`360`)
- Added ``cov`` instance methods to Series and DataFrame (`194`, `362`)
- `Added <visualization.barplot>` ``kind='bar'`` option to ``DataFrame.plot`` (`348`)
- `Added <basics.idxmin>` ``idxmin`` and ``idxmax`` to Series and DataFrame (`286`)
- `Added <io.clipboard>` ``read_clipboard`` function to parse DataFrame from clipboard (`300`)
- `Added <basics.stats>` ``nunique`` function to Series for counting unique elements (`297`)
- `Made <basics.dataframe>` DataFrame constructor use Series name if no columns passed (`373`)
- `Support <io.parse_dates>` regular expressions in read_table/read_csv (`364`)
- `Added <io.html>` ``DataFrame.to_html`` for writing DataFrame to HTML (`387`)
- `Added <basics.dataframe>` support for MaskedArray data in DataFrame, masked values converted to NaN (`396`)
- `Added <visualization.box>` ``DataFrame.boxplot`` function (`368`)
- `Can <basics.apply>` pass extra args, kwds to DataFrame.apply (`376`)
- `Implement <merging.multikey_join>` ``DataFrame.join`` with vector ``on`` argument (`312`)
- `Added <visualization.basic>` ``legend`` boolean flag to ``DataFrame.plot`` (`324`)
- `Can <reshaping.stacking>` pass multiple levels to ``stack`` and ``unstack`` (`370`)
- `Can <reshaping.pivot>` pass multiple values columns to ``pivot_table`` (`381`)
- `Use <groupby.multiindex>` Series name in GroupBy for result index (`363`)
- `Added <basics.apply>` ``raw`` option to ``DataFrame.apply`` for performance if only need ndarray (`309`)
- Added proper, tested weighted least squares to standard and panel OLS (`303`)  Performance enhancements
~~~
- VBENCH Cythonized ``cache_readonly``, resulting in substantial micro-performance enhancements throughout the code base (`361`)
- VBENCH Special Cython matrix iterator for applying arbitrary reduction operations with 3-5x better performance than ``np.apply_along_axis`` (`309`)
- VBENCH Improved performance of ``MultiIndex.from_tuples``
- VBENCH Special Cython matrix iterator for applying arbitrary reduction operations
- VBENCH + DOCUMENT Add ``raw`` option to ``DataFrame.apply`` for getting better performance when
- VBENCH Faster cythonized count by level in Series and DataFrame (`341`)
- VBENCH? Significant GroupBy performance enhancement with multiple keys with many "empty" combinations

- VBENCH New Cython vectorized function ``map_infer`` speeds up ``Series.apply`` and ``Series.map`` significantly when passed elementwise Python function, motivated by (`355`)
- VBENCH Significantly improved performance of ``Series.order``, which also makes np.unique called on a Series faster (`327`)
- VBENCH Vastly improved performance of GroupBy on axes with a MultiIndex (`299`)  .. _whatsnew_0.6.0.contributors:  Contributors
~~~  .. contributors:: v0.5.0..v0.6.0

Version 0.6.1 (December 13, 2011)

New features
~~~
- Can append single rows (as Series) to a DataFrame
- Add Spearman and Kendall rank correlation  options to Series.corr and DataFrame.corr (`428`)
- `Added <indexing.basics.get_value>` ``get_value`` and ``set_value`` methods to  Series, DataFrame, and Panel for very low-overhead access (>2x faster in many  cases) to scalar elements (`437`, `438`). ``set_value`` is capable of  producing an enlarged object.
- Add PyQt table widget to sandbox (`435`)
- DataFrame.align can `accept Series arguments <basics.align.frame.series>`  and an `axis option <basics.df_join>` (`461`)
- Implement new `SparseArray <sparse.array>` and ``SparseList``  data structures. SparseSeries now derives from SparseArray (`463`)
- `Better console printing options <basics.console_output>` (`453`)
- Implement fast data ranking for Series and  DataFrame, fast versions of scipy.stats.rankdata (`428`)
- Implement ``DataFrame.from_items`` alternate  constructor (`444`)
- DataFrame.convert_objects method for `inferring better dtypes <basics.cast>`  for object columns (`302`)
- Add `rolling_corr_pairwise <window.corr_pairwise>` function for  computing Panel of correlation matrices (`189`)
- Add `margins <reshaping.pivot.margins>` option to `pivot_table  <reshaping.pivot>` for computing subgroup aggregates (`114`)
- Add ``Series.from_csv`` function (`482`)
- `Can pass <window.cov_corr>` DataFrame/DataFrame and  DataFrame/Series to rolling_corr/rolling_cov (GH #462)
- MultiIndex.get_level_values can `accept the level name <advanced.get_level_values>`  Performance improvements
~~~  - Improve memory usage of ``DataFrame.describe`` (do not copy data  unnecessarily) (PR #425)  - Optimize scalar value lookups in the general case by 25% or more in Series  and DataFrame  - Fix performance regression in cross-sectional count in DataFrame, affecting  DataFrame.dropna speed
- Column deletion in DataFrame copies no data (computes views on blocks) (GH  #158)  .. _whatsnew_0.6.1.contributors:  Contributors
~~~  .. contributors:: v0.6.0..v0.6.1

~~~  - New unified `merge function <merging.join>` for efficiently performing  full gamut of database / relational-algebra operations. Refactored existing  join methods to use the new infrastructure, resulting in substantial  performance gains (`220`, `249`, `267`)  - New `unified concatenation function <merging.concat>` for concatenating  Series, DataFrame or Panel objects along an axis. Can form union or  intersection of the other axes. Improves performance of ``Series.append`` and  ``DataFrame.append`` (`468`, `479`, `273`)  - Can pass multiple DataFrames to  ``DataFrame.append`` to concatenate (stack) and multiple Series to  ``Series.append`` too  - `Can<basics.dataframe.from_list_of_dicts>` pass list of dicts (e.g., a  list of JSON objects) to DataFrame constructor (`526`)  - You can now `set multiple columns <indexing.columns.multiple>` in a  DataFrame via ``__getitem__``, useful for transformation (`342`)  - Handle differently-indexed output values in ``DataFrame.apply`` (`498`)  .. code-block:: ipython  In [1]: df = pd.DataFrame(np.random.randn(10, 4))  In [2]: df.apply(lambda x: x.describe())  Out[2]:  0  1  2  3  count  10.000  10.000  10.000  10.000  mean  0.190912  -0.395125  -0.731920  -0.403130  std  0.730951  0.813266  1.112016  0.961912  min  -0.861849  -2.104569  -1.776904  -1.469388  25%  -0.411391  -0.698728  -1.501401  -1.076610  50%  0.380863  -0.228039  -1.191943  -1.004091  75%  0.658444  0.057974  -0.034326  0.461706  max  1.212112  0.577046  1.643563  1.071804  [8 rows x 4 columns]  - `Add<advanced.reorderlevels>` ``reorder_levels`` method to Series and  DataFrame (`534`)  - `Add<indexing.dictionarylike>` dict-like ``get`` function to DataFrame  and Panel (`521`)  - `Add<basics.iterrows>` ``DataFrame.iterrows`` method for efficiently  iterating through the rows of a DataFrame  - Add ``DataFrame.to_panel`` with code adapted from  ``LongPanel.to_long``  - `Add <basics.reindexing>` ``reindex_axis`` method added to DataFrame  - `Add <basics.stats>` ``level`` option to binary arithmetic functions on  ``DataFrame`` and ``Series``  - `Add <advanced.advanced_reindex>` ``level`` option to the ``reindex``  and ``align`` methods on Series and DataFrame for broadcasting values across  a level (`542`, `552`, others)  - Add attribute-based item access to  ``Panel`` and add IPython completion (`563`)  - `Add <visualization.basic>` ``logy`` option to ``Series.plot`` for  log-scaling on the Y axis  - `Add <io.formatting>` ``index`` and ``header`` options to  ``DataFrame.to_string``  - `Can <merging.multiple_join>` pass multiple DataFrames to  ``DataFrame.join`` to join on index (`115`)  - `Can <merging.multiple_join>` pass multiple Panels to ``Panel.join``  (`115`)  - `Added <io.formatting>` ``justify`` argument to ``DataFrame.to_string``  to allow different alignment of column headers  - `Add <groupby.attributes>` ``sort`` option to GroupBy to allow disabling  sorting of the group keys for potential speedups (`595`)  - `Can <basics.dataframe.from_series>` pass MaskedArray to Series  constructor (`563`)  - Add Panel item access via attributes  and IPython completion (`554`)  - Implement ``DataFrame.lookup``, fancy-indexing analogue for retrieving values  given a sequence of row and column labels (`338`)  - Can pass a `list of functions <groupby.aggregate.multifunc>` to  aggregate with groupby on a DataFrame, yielding an aggregated result with  hierarchical columns (`166`)  - Can call ``cummin`` and ``cummax`` on Series and DataFrame to get cumulative  minimum and maximum, respectively (`647`)  - ``value_range`` added as utility function to get min and max of a dataframe  (`288`)  - Added ``encoding`` argument to ``read_csv``, ``read_table``, ``to_csv`` and  ``from_csv`` for non-ascii text (`717`)  - `Added <basics.stats>` ``abs`` method to pandas objects  - `Added <reshaping.pivot>` ``crosstab`` function for easily computing frequency tables  - `Added <indexing.set_ops>` ``isin`` method to index objects  - `Added <advanced.xs>` ``level`` argument to ``xs`` method of DataFrame.  API changes to integer indexing

~~~  One of the potentially riskiest API changes in 0.7.0, but also one of the most
important, was a complete review of how **integer indexes** are handled with
regard to label-based indexing. Here is an example:  .. code-block:: ipython  In [3]: s = pd.Series(np.random.randn(10), index=range(0, 20, 2))  In [4]: s  Out[4]:  0  -1.294524  2  0.413738  4  0.276662  6  -0.472035  8  -0.013960  10  -0.362543  12  -0.006154  14  -0.923061  16  0.895717  18  0.805244  Length: 10, dtype: float64  In [5]: s[0]  Out[5]: -1.2945235902555294  In [6]: s[2]  Out[6]: 0.41373810535784006  In [7]: s[4]  Out[7]: 0.2766617129497566  This is all exactly identical to the behavior before. However, if you ask for a
key **not** contained in the Series, in versions 0.6.1 and prior, Series would
*fall back* on a location-based lookup. This now raises a ``KeyError``:  .. code-block:: ipython  In [2]: s[1]  KeyError: 1  This change also has the same impact on DataFrame:  .. code-block:: ipython  In [3]: df = pd.DataFrame(np.random.randn(8, 4), index=range(0, 16, 2))  In [4]: df  0  1  2  3  0  0.88427  0.3363 -0.1787  0.03162  2  0.14451 -0.1415  0.2504  0.58374  4  -1.44779 -0.9186 -1.4996  0.27163  6  -0.26598 -2.4184 -0.2658  0.11503  8  -0.58776  0.3144 -0.8566  0.61941  10  0.10940 -0.7175 -1.0108  0.47990  12 -1.16919 -0.3087 -0.6049 -0.43544  14 -0.07337  0.3410  0.0424 -0.16037  In [5]: df.ix[3]  KeyError: 3  In order to support purely integer-based indexing, the following methods have
been added:  .. csv-table::  "Method","Description"  40,60  ``Series.iget_value(i)``, Retrieve value stored at location ``i``  ``Series.iget(i)``, Alias for ``iget_value``  ``DataFrame.irow(i)``, Retrieve the ``i``-th row  ``DataFrame.icol(j)``, Retrieve the ``j``-th column  "``DataFrame.iget_value(i, j)``", Retrieve the value at row ``i`` and column ``j``  API tweaks regarding label-based slicing
~~~  Label-based slicing using ``ix`` now requires that the index be sorted
(monotonic) **unless** both the start and endpoint are contained in the index:  .. code-block:: python  In [1]: s = pd.Series(np.random.randn(6), index=list('gmkaec'))  In [2]: s  Out[2]:  g  -1.182230  m  -0.276183  k  -0.243550  a  1.628992  e  0.073308  c  -0.539890  dtype: float64  Then this is OK:  .. code-block:: python  In [3]: s.ix['k':'e']  Out[3]:  k  -0.243550  a  1.628992  e  0.073308  dtype: float64  But this is not:  .. code-block:: ipython  In [12]: s.ix['b':'h']  KeyError 'b'  If the index had been sorted, the "range selection" would have been possible:  .. code-block:: python  In [4]: s2 = s.sort_index()  In [5]: s2  Out[5]:  a  1.628992  c  -0.539890  e  0.073308  g  -1.182230  k  -0.243550  m  -0.276183  dtype: float64  In [6]: s2.ix['b':'h']  Out[6]:  c  -0.539890  e  0.073308  g  -1.182230  dtype: float64  Changes to Series ``[]`` operator
~~~  As as notational convenience, you can pass a sequence of labels or a label
slice to a Series when getting and setting values via ``[]`` (i.e. the
``__getitem__`` and ``__setitem__`` methods). The behavior will be the same as
passing similar input to ``ix`` **except in the case of integer indexing**:  .. code-block:: ipython  In [8]: s = pd.Series(np.random.randn(6), index=list('acegkm'))  In [9]: s  Out[9]:  a  -1.206412  c  2.565646  e  1.431256  g  1.340309  k  -1.170299  m  -0.226169  Length: 6, dtype: float64  In [10]: s[['m', 'a', 'c', 'e']]  Out[10]:  m  -0.226169  a  -1.206412  c  2.565646  e  1.431256  Length: 4, dtype: float64  In [11]: s['b':'l']  Out[11]:  c  2.565646  e  1.431256  g  1.340309  k  -1.170299  Length: 4, dtype: float64  In [12]: s['c':'k']  Out[12]:  c  2.565646  e  1.431256  g  1.340309  k  -1.170299  Length: 4, dtype: float64  In the case of integer indexes, the behavior will be exactly as before

(shadowing ``ndarray``):  .. code-block:: ipython  In [13]: s = pd.Series(np.random.randn(6), index=range(0, 12, 2))  In [14]: s[[4, 0, 2]]  Out[14]:  4  0.132003  0  0.410835  2  0.813850  Length: 3, dtype: float64  In [15]: s[1:5]  Out[15]:  2  0.813850  4  0.132003  6  -0.827317  8  -0.076467  Length: 4, dtype: float64  If you wish to do indexing with sequences and slicing on an integer index with
label semantics, use ``ix``.  Other API changes
~~~  - The deprecated ``LongPanel`` class has been completely removed  - If ``Series.sort`` is called on a column of a DataFrame, an exception will  now be raised. Before it was possible to accidentally mutate a DataFrame's  column by doing ``df[col].sort()`` instead of the side-effect free method  ``df[col].order()`` (`316`)  - Miscellaneous renames and deprecations which will (harmlessly) raise  ``FutureWarning``  - ``drop`` added as an optional parameter to ``DataFrame.reset_index`` (`699`)  Performance improvements
~~~  - `Cythonized GroupBy aggregations <groupby.aggregate.cython>` no longer  presort the data, thus achieving a significant speedup (`93`).  GroupBy  aggregations with Python functions significantly sped up by clever  manipulation of the ndarray data type in Cython (`496`).
- Better error message in DataFrame constructor when passed column labels  don't match data (`497`)
- Substantially improve performance of multi-GroupBy aggregation when a  Python function is passed, reuse ndarray object in Cython (`496`)
- Can store objects indexed by tuples and floats in HDFStore (`492`)
- Don't print length by default in Series.to_string, add ``length`` option (`489`)
- Improve Cython code for multi-groupby to aggregate without having to sort  the data (`93`)
- Improve MultiIndex reindexing speed by storing tuples in the MultiIndex,  test for backwards unpickling compatibility
- Improve column reindexing performance by using specialized Cython take  function
- Further performance tweaking of Series.__getitem__ for standard use cases
- Avoid Index dict creation in some cases (i.e. when getting slices, etc.),  regression from prior versions
- Friendlier error message in setup.py if NumPy not installed
- Use common set of NA-handling operations (sum, mean, etc.) in Panel class  also (`536`)
- Default name assignment when calling ``reset_index`` on DataFrame with a  regular (non-hierarchical) index (`476`)
- Use Cythonized groupers when possible in Series/DataFrame stat ops with  ``level`` parameter passed (`545`)
- Ported skiplist data structure to C to speed up ``rolling_median`` by about  5-10x in most typical use cases (`374`)  .. _whatsnew_0.7.0.contributors:  Contributors
~~~  .. contributors:: v0.6.1..v0.7.0

Version 0.7.1 (February 29, 2012)

{{ header }}  This release includes a few new features and addresses over a dozen bugs in
0.7.0.  New features
~~~  - Add ``to_clipboard`` function to pandas namespace for writing objects to  the system clipboard (`774`)  - Add ``itertuples`` method to DataFrame for iterating through the rows of a  dataframe as tuples (`818`)  - Add ability to pass fill_value and method to DataFrame and Series align  method (`806`, `807`)  - Add fill_value option to reindex, align methods (`784`)  - Enable concat to produce DataFrame from Series (`787`)  - Add ``between`` method to Series (`802`)  - Add HTML representation hook to DataFrame for the IPython HTML notebook  (`773`)  - Support for reading Excel 2007 XML documents using openpyxl  Performance improvements
~~~  - Improve performance and memory usage of fillna on DataFrame  - Can concatenate a list of Series along axis=1 to obtain a DataFrame (`787`)  .. _whatsnew_0.7.1.contributors:  Contributors
~~~  .. contributors:: v0.7.0..v0.7.1

Version 0.7.2 (March 16, 2012)

{{ header }}  This release targets bugs in 0.7.1, and adds a few minor features.  New features
~~~  - Add additional tie-breaking methods in DataFrame.rank (`874`)  - Add ascending parameter to rank in Series, DataFrame (`875`)  - Add coerce_float option to DataFrame.from_records (`893`)  - Add sort_columns parameter to allow unsorted plots (`918`)  - Enable column access via attributes on GroupBy (`882`)  - Can pass dict of values to DataFrame.fillna (`661`)  - Can select multiple hierarchical groups by passing list of values in .ix  (`134`)  - Add ``axis`` option to DataFrame.fillna (`174`)  - Add level keyword to ``drop`` for dropping values from a level (`159`)  Performance improvements
~~~  - Use khash for Series.value_counts, add raw function to algorithms.py (`861`)  - Intercept __builtin__.sum in groupby (`885`)  .. _whatsnew_0.7.2.contributors:  Contributors
~~~  .. contributors:: v0.7.1..v0.7.2

Version 0.7.3 (April 12, 2012)

{{ header }}  This is a minor release from 0.7.2 and fixes many minor bugs and adds a number
of nice new features. There are also a couple of API changes to note; these
should not affect very many users, and we are inclined to call them "bug fixes"
even though they do constitute a change in behavior. See the `full release
notes <release>` or issue
tracker on GitHub for a complete list.  New features
~~~  - New `fixed width file reader <io.fwf>`, ``read_fwf``
- New `scatter_matrix <visualization.scatter_matrix>` function for making  a scatter plot matrix  .. code-block:: python  from pandas.tools.plotting import scatter_matrix  scatter_matrix(df, alpha=0.2)  # noqa F821  - Add ``stacked`` argument to Series and DataFrame's ``plot`` method for  `stacked bar plots <visualization.barplot>`.  .. code-block:: python  df.plot(kind="bar", stacked=True)  # noqa F821  .. code-block:: python  df.plot(kind="barh", stacked=True)  # noqa F821  - Add log x and y `scaling options <visualization.basic>` to  ``DataFrame.plot`` and ``Series.plot``
- Add ``kurt`` methods to Series and DataFrame for computing kurtosis  NA boolean comparison API change
~~~  Reverted some changes to how NA values (represented typically as ``NaN`` or
``None``) are handled in non-numeric Series:  .. code-block:: ipython  In [1]: series = pd.Series(["Steve", np.nan, "Joe"])  In [2]: series == "Steve"  Out[2]:  0  True  1  False  2  False  Length: 3, dtype: bool  In [3]: series != "Steve"  Out[3]:  0  False  1  True  2  True  Length: 3, dtype: bool  In comparisons, NA / NaN will always come through as ``False`` except with
``!=`` which is ``True``. *Be very careful* with boolean arithmetic, especially
negation, in the presence of NA data. You may wish to add an explicit NA
filter into boolean array operations if you are worried about this:  .. code-block:: ipython  In [4]: mask = series == "Steve"  In [5]: series[mask & series.notnull()]  Out[5]:  0  Steve  Length: 1, dtype: object  While propagating NA in comparisons may seem like the right behavior to some
users (and you could argue on purely technical grounds that this is the right
thing to do), the evaluation was made that propagating NA everywhere, including
in numerical arrays, would cause a large amount of problems for users. Thus, a
"practicality beats purity" approach was taken. This issue may be revisited at
some point in the future.  Other API changes
~~~  When calling ``apply`` on a grouped Series, the return value will also be a
Series, to be more consistent with the ``groupby`` behavior with DataFrame:  .. code-block:: ipython  In [6]: df = pd.DataFrame(  ...:  {  ...:  "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],  ...:  "B": ["one", "one", "two", "three", "two", "two", "one", "three"],  ...:  "C": np.random.randn(8),  ...:  "D": np.random.randn(8),  ...:  }  ...: )  ...:  In [7]: df  Out[7]:  A  B  C  D  0  foo  one  0.469112 -0.861849  1  bar  one -0.282863 -2.104569  2  foo  two -1.509059 -0.494929  3  bar  three -1.135632  1.071804  4  foo  two  1.212112  0.721555  5  bar  two -0.173215 -0.706771  6  foo  one  0.119209 -1.039575  7  foo  three -1.044236  0.271860  [8 rows x 4 columns]  In [8]: grouped = df.groupby("A")["C"]  In [9]: grouped.describe()  Out[9]:  count  mean  std  min  25%  50%  75%  max  A  bar  3.0 -0.530570  0.526860 -1.135632 -0.709248 -0.282863 -0.228039 -0.173215  foo  5.0 -0.150572  1.113308 -1.509059 -1.044236  0.119209  0.469112  1.212112  [2 rows x 8 columns]  In [10]: grouped.apply(lambda x: x.sort_values()[-2:])  # top 2 values  Out[10]:  A  bar  1  -0.282863  5  -0.173215  foo  0  0.469112  4  1.212112  Name: C, Length: 4, dtype: float64  .. _whatsnew_0.7.3.contributors:  Contributors
~~~  .. contributors:: v0.7.2..v0.7.3

{{ header }}  This is a major release from 0.7.3 and includes extensive work on the time
series handling and processing infrastructure as well as a great deal of new
functionality throughout the library. It includes over 700 commits from more
than 20 distinct authors. Most pandas 0.7.3 and earlier users should not
experience any issues upgrading, but due to the migration to the NumPy
datetime64 dtype, there may be a number of bugs and incompatibilities
lurking. Lingering incompatibilities will be fixed ASAP in a 0.8.1 release if
necessary. See the `full release notes
on GitHub for a complete list.  Support for non-unique indexes
~~~  All objects can now work with non-unique indexes. Data alignment / join
operations work according to SQL join semantics (including, if application,
index duplication in many-to-many joins)  NumPy datetime64 dtype and 1.6 dependency
~~~  Time series data are now represented using NumPy's datetime64 dtype; thus,
pandas 0.8.0 now requires at least NumPy 1.6. It has been tested and verified
to work with the development version (1.7+) of NumPy as well which includes
some significant user-facing API changes. NumPy 1.6 also has a number of bugs
having to do with nanosecond resolution data, so I recommend that you steer
clear of NumPy 1.6's datetime64 API functions (though limited as they are) and
only interact with this data using the interface that pandas provides.  See the end of the 0.8.0 section for a "porting" guide listing potential issues
for users migrating legacy code bases from pandas 0.7 or earlier to 0.8.0.  Bug fixes to the 0.7.x series for legacy NumPy < 1.6 users will be provided as
they arise. There will be no more further development in 0.7.x beyond bug
fixes.  Time Series changes and improvements
~~~  .. note::  With this release, legacy scikits.timeseries users should be able to port  their code to use pandas.  .. note::  See `documentation <timeseries>` for overview of pandas timeseries API.  - New datetime64 representation **speeds up join operations and data  alignment**, **reduces memory usage**, and improve serialization /  deserialization performance significantly over datetime.datetime
- High performance and flexible **resample** method for converting from  high-to-low and low-to-high frequency. Supports interpolation, user-defined  aggregation functions, and control over how the intervals and result labeling  are defined. A suite of high performance Cython/C-based resampling functions  (including Open-High-Low-Close) have also been implemented.
- Revamp of `frequency aliases <timeseries.offset_aliases>` and support for  **frequency shortcuts** like '15min', or '1h30min'
- New `DatetimeIndex class <timeseries.datetimeindex>` supports both fixed  frequency and irregular time  series. Replaces now deprecated DateRange class
- New ``PeriodIndex`` and ``Period`` classes for representing  `time spans <timeseries.periods>` and performing **calendar logic**,  including the ``12 fiscal quarterly frequencies <timeseries.quarterly>``.  This is a partial port of, and a substantial enhancement to,  elements of the scikits.timeseries code base. Support for conversion between  PeriodIndex and DatetimeIndex
- New Timestamp data type subclasses ``datetime.datetime``, providing the same  interface while enabling working with nanosecond-resolution data. Also  provides `easy time zone conversions <timeseries.timezone>`.

- Enhanced support for `time zones <timeseries.timezone>`. Add  ``tz_convert`` and ``tz_localize`` methods to TimeSeries and DataFrame. All  timestamps are stored as UTC; Timestamps from DatetimeIndex objects with time  zone set will be localized to local time. Time zone conversions are therefore  essentially free. User needs to know very little about pytz library now; only  time zone names as strings are required. Time zone-aware timestamps are  equal if and only if their UTC timestamps match. Operations between time  zone-aware time series with different time zones will result in a UTC-indexed  time series.
- Time series **string indexing conveniences** / shortcuts: slice years, year  and month, and index values with strings
- Enhanced time series **plotting**; adaptation of scikits.timeseries  matplotlib-based plotting code
- New ``date_range``, ``bdate_range``, and ``period_range`` `factory  functions <timeseries.daterange>`
- Robust **frequency inference** function ``infer_freq`` and ``inferred_freq``  property of DatetimeIndex, with option to infer frequency on construction of  DatetimeIndex
- to_datetime function efficiently **parses array of strings** to  DatetimeIndex. DatetimeIndex will parse array or list of strings to  datetime64
- **Optimized** support for datetime64-dtype data in Series and DataFrame  columns
- New NaT (Not-a-Time) type to represent **NA** in timestamp arrays
- Optimize Series.asof for looking up **"as of" values** for arrays of  timestamps
- Milli, Micro, Nano date offset objects
- Can index time series with datetime.time objects to select all data at  particular **time of day** (``TimeSeries.at_time``) or **between two times**  (``TimeSeries.between_time``)
- Add `tshift <timeseries.advanced_datetime>` method for leading/lagging  using the frequency (if any) of the index, as opposed to a naive lead/lag  using shift  Other new features
~~~  - New `cut <reshaping.tile.cut>` and ``qcut`` functions (like R's cut  function) for computing a categorical variable from a continuous variable by  binning values either into value-based (``cut``) or quantile-based (``qcut``)  bins
- Rename ``Factor`` to ``Categorical`` and add a number of usability features
- Add `limit <missing_data.fillna.limit>` argument to fillna/reindex
- More flexible multiple function application in GroupBy, and can pass list  (name, function) tuples to get result in particular order with given names
- Add flexible `replace <missing_data.replace>` method for efficiently  substituting values
- Enhanced `read_csv/read_table <io.parse_dates>` for reading time series  data and converting multiple columns to dates
- Add `comments <io.comments>` option to parser functions: read_csv, etc.
- Add `dayfirst <io.dayfirst>` option to parser functions for parsing  international DD/MM/YYY dates
- Allow the user to specify the CSV reader `dialect <io.dialect>` to  control quoting etc.
- Handling `thousands <io.thousands>` separators in read_csv to improve  integer parsing.
- Enable unstacking of multiple levels in one shot. Alleviate ``pivot_table``  bugs (empty columns being introduced)
- Move to klib-based hash tables for indexing; better performance and less  memory usage than Python's dict
- Add first, last, min, max, and prod optimized GroupBy functions
- New `ordered_merge <merging.merge_ordered>` function
- Add flexible `comparison <basics.binop>` instance methods eq, ne, lt,  gt, etc. to DataFrame, Series
- Improve `scatter_matrix <visualization.scatter_matrix>` plotting  function and add histogram or kernel density estimates to diagonal
- Add `'kde' <visualization.kde>` plot option for density plots
- Support for converting DataFrame to R data.frame through rpy2
- Improved support for complex numbers in Series and DataFrame
- Add ``pct_change`` method to all data structures
- Add max_colwidth configuration option for DataFrame console output
- `Interpolate <missing_data.interpolate>` Series values using index values
- Can select multiple columns from GroupBy

- Add `update <merging.combine_first.update>` methods to Series/DataFrame  for updating values in place
- Add ``any`` and ``all`` method to DataFrame  New plotting methods
~~~  .. code-block:: python  import pandas as pd  fx = pd.read_pickle("data/fx_prices")  import matplotlib.pyplot as plt  ``Series.plot`` now supports a ``secondary_y`` option:  .. code-block:: python  plt.figure()  fx["FR"].plot(style="g")  fx["IT"].plot(style="k--", secondary_y=True)  Vytautas Jancauskas, the 2012 GSOC participant, has added many new plot
types. For example, ``'kde'`` is a new option:  .. code-block:: python  s = pd.Series(  np.concatenate((np.random.randn(1000), np.random.randn(1000) * 0.5 + 3))  )  plt.figure()  s.hist(density=True, alpha=0.2)  s.plot(kind="kde")  See `the plotting page <visualization.other>` for much more.  Other API changes
~~~  - Deprecation of ``offset``, ``time_rule``, and ``timeRule`` arguments names in  time series functions. Warnings will be printed until pandas 0.9 or 1.0.  Potential porting issues for pandas <= 0.7.3 users
~~~  The major change that may affect you in pandas 0.8.0 is that time series
indexes use NumPy's ``datetime64`` data type instead of ``dtype=object`` arrays
of Python's built-in ``datetime.datetime`` objects. ``DateRange`` has been
replaced by ``DatetimeIndex`` but otherwise behaved identically. But, if you
have code that converts ``DateRange`` or ``Index`` objects that used to contain
``datetime.datetime`` values to plain NumPy arrays, you may have bugs lurking
with code using scalar values because you are handing control over to NumPy:  .. ipython:: python  import datetime  rng = pd.date_range("1/1/2000", periods=10)  rng[5]  isinstance(rng[5], datetime.datetime)  rng_asarray = np.asarray(rng)  scalar_val = rng_asarray[5]  type(scalar_val)  pandas's ``Timestamp`` object is a subclass of ``datetime.datetime`` that has
nanosecond support (the ``nanosecond`` field store the nanosecond value between
0 and 999). It should substitute directly into any code that used
``datetime.datetime`` values before. Thus, I recommend not casting
``DatetimeIndex`` to regular NumPy arrays.  If you have code that requires an array of ``datetime.datetime`` objects, you
have a couple of options. First, the ``astype(object)`` method of ``DatetimeIndex``
produces an array of ``Timestamp`` objects:  .. ipython:: python  stamp_array = rng.astype(object)  stamp_array  stamp_array[5]  To get an array of proper ``datetime.datetime`` objects, use the
``to_pydatetime`` method:  .. ipython:: python  dt_array = rng.to_pydatetime()  dt_array  dt_array[5]  matplotlib knows how to handle ``datetime.datetime`` but not Timestamp
objects. While I recommend that you plot time series using ``TimeSeries.plot``,
you can either use ``to_pydatetime`` or register a converter for the Timestamp
type. See `matplotlib documentation
<http://matplotlib.org/api/units_api.html>`__ for more on this.  .. warning::  There are bugs in the user-facing API with the nanosecond datetime64 unit  in NumPy 1.6. In particular, the string version of the array shows garbage  values, and conversion to ``dtype=object`` is similarly broken.  .. ipython:: python  rng = pd.date_range("1/1/2000", periods=10)  rng  np.asarray(rng)  converted = np.asarray(rng, dtype=object)  converted[5]  **Trust me: don't panic**. If you are using NumPy 1.6 and restrict your  interaction with ``datetime64`` values to pandas's API you will be just  fine. There is nothing wrong with the data-type (a 64-bit integer  internally); all of the important data processing happens in pandas and is  heavily tested. I strongly recommend that you **do not work directly with  datetime64 arrays in NumPy 1.6** and only use the pandas API.  **Support for non-unique indexes**: In the latter case, you may have code

inside a ``try:... catch:`` block that failed due to the index not being
unique. In many cases it will no longer fail (some method like ``append`` still
check for uniqueness unless disabled). However, all is not lost: you can
inspect ``index.is_unique`` and raise an exception explicitly if it is
``False`` or go to a different code branch.  .. _whatsnew_0.8.0.contributors:  Contributors
~~~  .. contributors:: v0.7.3..v0.8.0

Version 0.8.1 (July 22, 2012)

{{ header }}  This release includes a few new features, performance enhancements, and over 30
bug fixes from 0.8.0.  New features include notably NA friendly string
processing functionality and a series of new plot types and options.  New features
~~~  - Add `vectorized string processing methods <text.string_methods>`  accessible via Series.str (`620`)  - Add option to disable adjustment in EWMA (`1584`)  - `Radviz plot <visualization.radviz>` (`1566`)  - `Parallel coordinates plot <visualization.parallel_coordinates>`  - `Bootstrap plot <visualization.bootstrap>`  - Per column styles and secondary y-axis plotting (`1559`)  - New datetime converters millisecond plotting  (`1599`)  - Add option to disable "sparse" display of hierarchical indexes (`1538`)  - Series/DataFrame's ``set_index`` method can `append levels  <indexing.set_index>` to an existing Index/MultiIndex (`1569`, `1577`)  Performance improvements
~~~  - Improved implementation of rolling min and max (thanks to `Bottleneck  <https://bottleneck.readthedocs.io>`__ !)  - Add accelerated ``'median'`` GroupBy option (`1358`)  - Significantly improve the performance of parsing ISO8601-format date  strings with ``DatetimeIndex`` or ``to_datetime`` (`1571`)  - Improve the performance of GroupBy on single-key aggregations and use with  Categorical types  - Significant datetime parsing performance improvements  .. _whatsnew_0.8.1.contributors:  Contributors
~~~  .. contributors:: v0.8.0..v0.8.1

Version 0.9.0 (October 7, 2012)

This is a major release from 0.8.1 and includes several new features and
enhancements along with a large number of bug fixes. New features include
vectorized unicode encoding/decoding for ``Series.str``, ``to_latex`` method to
DataFrame, more flexible parsing of boolean values, and enabling the download of
options data from Yahoo! Finance.  New features
~~~  - Add ``encode`` and ``decode`` for unicode handling to `vectorized  string processing methods <text.string_methods>` in Series.str  (`1706`)  - Add ``DataFrame.to_latex`` method (`1735`)  - Add convenient expanding window equivalents of all rolling_* ops (`1785`)  - Add Options class to pandas.io.data for fetching options data from Yahoo!  Finance (`1748`, `1739`)  - More flexible parsing of boolean values (Yes, No, TRUE, FALSE, etc)  (`1691`, `1295`)  - Add ``level`` parameter to ``Series.reset_index``  - ``TimeSeries.between_time`` can now select times across midnight (`1871`)  - Series constructor can now handle generator as input (`1679`)  - ``DataFrame.dropna`` can now take multiple axes (tuple/list) as input  (`924`)  - Enable ``skip_footer`` parameter in ``ExcelFile.parse`` (`1843`)  API changes
~~~  - The default column names when ``header=None`` and no columns names passed to  functions like ``read_csv`` has changed to be more Pythonic and amenable to  attribute access:  .. ipython:: python  import io  data = """  0,0,1  1,1,0  0,1,0  """  df = pd.read_csv(io.StringIO(data), header=None)  df  - Creating a Series from another Series, passing an index, will cause reindexing  to happen inside rather than treating the Series like an ndarray. Technically  improper usages like ``Series(df[col1], index=df[col2])`` that worked before  "by accident" (this was never intended) will lead to all NA Series in some  cases. To be perfectly clear:  .. ipython:: python  s1 = pd.Series([1, 2, 3])  s1  s2 = pd.Series(s1, index=["foo", "bar", "baz"])  s2  - Deprecated ``day_of_year`` API removed from PeriodIndex, use ``dayofyear``  (`1723`)  - Don't modify NumPy suppress printoption to True at import time  - The internal HDF5 data arrangement for DataFrames has been transposed.  Legacy  files will still be readable by HDFStore (`1834`, `1824`)  - Legacy cruft removed: pandas.stats.misc.quantileTS  - Use ISO8601 format for Period repr: monthly, daily, and on down (`1776`)  - Empty DataFrame columns are now created as object dtype. This will prevent a  class of TypeErrors that was occurring in code where the dtype of a column  would depend on the presence of data or not (e.g. a SQL query having results)  (`1783`)  - Setting parts of DataFrame/Panel using ix now aligns input Series/DataFrame  (`1630`)  - ``first`` and ``last`` methods in ``GroupBy`` no longer drop non-numeric  columns (`1809`)  - Resolved inconsistencies in specifying custom NA values in text parser.  ``na_values`` of type dict no longer override default NAs unless  ``keep_default_na`` is set to false explicitly (`1657`)  - ``DataFrame.dot`` will not do data alignment, and also work with Series  (`1915`)  See the `full release notes
on GitHub for a complete list.  .. _whatsnew_0.9.0.contributors:  Contributors
~~~  .. contributors:: v0.8.1..v0.9.0

{{ header }}  This is a bug fix release from 0.9.0 and includes several new features and
enhancements along with a large number of bug fixes. The new features include
by-column sort order for DataFrame and Series, improved NA handling for the rank
method, masking functions for DataFrame, and intraday time-series filtering for
DataFrame.  New features
~~~  - ``Series.sort``, ``DataFrame.sort``, and ``DataFrame.sort_index`` can now be  specified in a per-column manner to support multiple sort orders (`928`)  .. code-block:: ipython  In [2]: df = pd.DataFrame(np.random.randint(0, 2, (6, 3)),  ...:  columns=['A', 'B', 'C'])  In [3]: df.sort(['A', 'B'], ascending=[1, 0])  Out[3]:  A  B  C  3  0  1  1  4  0  1  1  2  0  0  1  0  1  0  0  1  1  0  0  5  1  0  0  - ``DataFrame.rank`` now supports additional argument values for the  ``na_option`` parameter so missing values can be assigned either the largest  or the smallest rank (`1508`, `2159`)  .. ipython:: python  df = pd.DataFrame(np.random.randn(6, 3), columns=['A', 'B', 'C'])  df.loc[2:4] = np.nan  df.rank()  df.rank(na_option='top')  df.rank(na_option='bottom')  - DataFrame has new ``where`` and ``mask`` methods to select values according to a  given boolean mask (`2109`, `2151`)  DataFrame currently supports slicing via a boolean vector the same length as the DataFrame (inside the ``[]``).  The returned DataFrame has the same number of columns as the original, but is sliced on its index.  .. ipython:: python  df = pd.DataFrame(np.random.randn(5, 3), columns=['A', 'B', 'C'])  df  df[df['A'] > 0]  If a DataFrame is sliced with a DataFrame based boolean condition (with the same size as the original DataFrame),  then a DataFrame the same size (index and columns) as the original is returned, with  elements that do not meet the boolean condition as ``NaN``. This is accomplished via  the new method ``DataFrame.where``. In addition, ``where`` takes an optional ``other`` argument for replacement.  .. ipython:: python  df[df > 0]  df.where(df > 0)  df.where(df > 0, -df)  Furthermore, ``where`` now aligns the input boolean condition (ndarray or DataFrame), such that partial selection  with setting is possible. This is analogous to partial setting via ``.ix`` (but on the contents rather than the axis labels)  .. ipython:: python  df2 = df.copy()  df2[df2[1:4] > 0] = 3  df2  ``DataFrame.mask`` is the inverse boolean operation of ``where``.  .. ipython:: python  df.mask(df <= 0)  - Enable referencing of Excel columns by their column names (`1936`)  .. code-block:: ipython  In [1]: xl = pd.ExcelFile('data/test.xls')  In [2]: xl.parse('Sheet1', index_col=0, parse_dates=True,  parse_cols='A:D')  - Added option to disable pandas-style tick locators and formatters  using ``series.plot(x_compat=True)`` or ``pandas.plot_params['x_compat'] =  True`` (`2205`)  - Existing TimeSeries methods ``at_time`` and ``between_time`` were added to  DataFrame (`2149`)  - DataFrame.dot can now accept ndarrays (`2042`)  - DataFrame.drop now supports non-unique indexes (`2101`)  - Panel.shift now supports negative periods (`2164`)  - DataFrame now support unary ~ operator (`2110`)  API changes

~~~  - Upsampling data with a PeriodIndex will result in a higher frequency  TimeSeries that spans the original time window  .. code-block:: ipython  In [1]: prng = pd.period_range('2012Q1', periods=2, freq='Q')  In [2]: s = pd.Series(np.random.randn(len(prng)), prng)  In [4]: s.resample('M')  Out[4]:  2012-01  -1.471992  2012-02  NaN  2012-03  NaN  2012-04  -0.493593  2012-05  NaN  2012-06  NaN  Freq: M, dtype: float64  - Period.end_time now returns the last nanosecond in the time interval  (`2124`, `2125`, `1764`)  .. ipython:: python  p = pd.Period('2012')  p.end_time  - File parsers no longer coerce to float or bool for columns that have custom  converters specified (`2184`)  .. ipython:: python  import io  data = ('A,B,C\n'  '0001,001,5\n'  '0002,002,6')  pd.read_csv(io.StringIO(data), converters={'A': lambda x: x.strip()})  See the `full release notes
on GitHub for a complete list.  .. _whatsnew_0.9.1.contributors:  Contributors
~~~  .. contributors:: v0.9.0..v0.9.1

These are the changes in pandas 1.0.0. See `release` for a full changelog
including other versions of pandas.  .. note::  The pandas 1.0 release removed a lot of functionality that was deprecated  in previous releases (see `below <whatsnew_100.prior_deprecations>`  for an overview). It is recommended to first upgrade to pandas 0.25 and to  ensure your code is working without warnings, before upgrading to pandas  1.0.  New deprecation policy
~~~  Starting with pandas 1.0.0, pandas will adopt a variant of `SemVer`_ to
version releases. Briefly,  * Deprecations will be introduced in minor releases (e.g. 1.1.0, 1.2.0, 2.1.0, ...)
* Deprecations will be enforced in major releases (e.g. 1.0.0, 2.0.0, 3.0.0, ...)
* API-breaking changes will be made only in major releases (except for experimental features)  See `policies.version` for more.  .. _2019 Pandas User Survey: https://pandas.pydata.org/community/blog/2019-user-survey.html
.. _SemVer: https://semver.org  {{ header }}  .. ---  Enhancements
~~~  .. _whatsnew_100.numba_rolling_apply:  Using Numba in ``rolling.apply`` and ``expanding.apply``
^^^  We've added an ``engine`` keyword to `~core.window.rolling.Rolling.apply` and `~core.window.expanding.Expanding.apply`
that allows the user to execute the routine using Numba_ instead of Cython.
Using the Numba engine can yield significant performance gains if the apply function can operate on numpy arrays and
the data set is larger (1 million rows or greater). For more details, see
`rolling apply documentation <window.numba_engine>` (`28987`, `30936`)  .. _whatsnew_100.custom_window:  Defining custom windows for rolling operations
^^^  We've added a `pandas.api.indexers.BaseIndexer` class that allows users to define how
window bounds are created during ``rolling`` operations. Users can define their own ``get_window_bounds``
method on a `pandas.api.indexers.BaseIndexer` subclass that will generate the start and end
indices used for each window during the rolling aggregation. For more details and example usage, see
the `custom window rolling documentation <window.custom_rolling_window>`  .. _whatsnew_100.to_markdown:  Converting to markdown
^^^  We've added `~DataFrame.to_markdown` for creating a markdown table (`11052`)  .. ipython:: python  df = pd.DataFrame({"A": [1, 2, 3], "B": [1, 2, 3]}, index=['a', 'a', 'b'])  print(df.to_markdown())  Experimental new features
~~~  .. _whatsnew_100.NA:  Experimental ``NA`` scalar to denote missing values
^^^  A new ``pd.NA`` value (singleton) is introduced to represent scalar missing
values. Up to now, pandas used several values to represent missing data: ``np.nan`` is used for this for float data, ``np.nan`` or
``None`` for object-dtype data and ``pd.NaT`` for datetime-like data. The
goal of ``pd.NA`` is to provide a "missing" indicator that can be used
consistently across data types. ``pd.NA`` is currently used by the nullable integer and boolean
data types and the new string data type (`28095`).  .. warning::  Experimental: the behaviour of ``pd.NA`` can still change without warning.  For example, creating a Series using the nullable integer dtype:  .. ipython:: python  s = pd.Series([1, 2, None], dtype="Int64")  s  s[2]  Compared to ``np.nan``, ``pd.NA`` behaves differently in certain operations.
In addition to arithmetic operations, ``pd.NA`` also propagates as "missing"
or "unknown" in comparison operations:  .. ipython:: python  np.nan > 1  pd.NA > 1  For logical operations, ``pd.NA`` follows the rules of the
*Kleene logic*). For example:  .. ipython:: python  pd.NA | True  For more, see `NA section <missing_data.NA>` in the user guide on missing
data.  .. _whatsnew_100.string:  Dedicated string data type
^^^  We've added `StringDtype`, an extension type dedicated to string data.

Previously, strings were typically stored in object-dtype NumPy arrays. (`29975`)  .. warning::  ``StringDtype`` is currently considered experimental. The implementation  and parts of the API may change without warning.  The ``'string'`` extension type solves several issues with object-dtype NumPy arrays:  1. You can accidentally store a *mixture* of strings and non-strings in an  ``object`` dtype array. A ``StringArray`` can only store strings.
2. ``object`` dtype breaks dtype-specific operations like `DataFrame.select_dtypes`.  There isn't a clear way to select *just* text while excluding non-text,  but still object-dtype columns.
3. When reading code, the contents of an ``object`` dtype array is less clear  than ``string``.  .. ipython:: python  pd.Series(['abc', None, 'def'], dtype=pd.StringDtype())  You can use the alias ``"string"`` as well.  .. ipython:: python  s = pd.Series(['abc', None, 'def'], dtype="string")  s  The usual string accessor methods work. Where appropriate, the return type
of the Series or columns of a DataFrame will also have string dtype.  .. ipython:: python  s.str.upper()  s.str.split('b', expand=True).dtypes  String accessor methods returning integers will return a value with `Int64Dtype`  .. ipython:: python  s.str.count("a")  We recommend explicitly using the ``string`` data type when working with strings.
See `text.types` for more.  .. _whatsnew_100.boolean:  Boolean data type with missing values support
^^^  We've added `BooleanDtype` / `~arrays.BooleanArray`, an extension
type dedicated to boolean data that can hold missing values. The default
``bool`` data type based on a bool-dtype NumPy array, the column can only hold
``True`` or ``False``, and not missing values. This new `~arrays.BooleanArray`
can store missing values as well by keeping track of this in a separate mask.
(`29555`, `30095`, `31131`)  .. ipython:: python  pd.Series([True, False, None], dtype=pd.BooleanDtype())  You can use the alias ``"boolean"`` as well.  .. ipython:: python  s = pd.Series([True, False, None], dtype="boolean")  s  .. _whatsnew_100.convert_dtypes:  Method ``convert_dtypes`` to ease use of supported extension dtypes
^^^  In order to encourage use of the extension dtypes ``StringDtype``,
``BooleanDtype``, ``Int64Dtype``, ``Int32Dtype``, etc., that support ``pd.NA``, the
methods `DataFrame.convert_dtypes` and `Series.convert_dtypes`
have been introduced. (`29752`) (`30929`)  Example:  .. ipython:: python  df = pd.DataFrame({'x': ['abc', None, 'def'],  'y': [1, 2, np.nan],  'z': [True, False, True]})  df  df.dtypes  .. ipython:: python  converted = df.convert_dtypes()  converted  converted.dtypes  This is especially useful after reading in data using readers such as `read_csv`
and `read_excel`.
See `here <missing_data.NA.conversion>` for a description.  .. _whatsnew_100.enhancements.other:  Other enhancements
~~~  - `DataFrame.to_string` added the ``max_colwidth`` parameter to control when wide columns are truncated (`9784`)
- Added the ``na_value`` argument to `Series.to_numpy`, `Index.to_numpy` and `DataFrame.to_numpy` to control the value used for missing data (`30322`)
- `MultiIndex.from_product` infers level names from inputs if not explicitly provided (`27292`)
- `DataFrame.to_latex` now accepts ``caption`` and ``label`` arguments (`25436`)
- DataFrames with `nullable integer <integer_na>`, the `new string dtype <text.types>`  and period data type can now be converted to ``pyarrow`` (>=0.15.0), which means that it is  supported in writing to the Parquet file format when using the ``pyarrow`` engine (`28368`).  Full roundtrip to parquet (writing and reading back in with `~DataFrame.to_parquet` / `read_parquet`)  is supported starting with pyarrow >= 0.16 (`20612`).
- `to_parquet` now appropriately handles the ``schema`` argument for user defined schemas in the pyarrow engine. (`30270`)

- `DataFrame.to_json` now accepts an ``indent`` integer argument to enable pretty printing of JSON output (`12004`)
- `read_stata` can read Stata 119 dta files. (`28250`)
- Implemented `pandas.core.window.Window.var` and `pandas.core.window.Window.std` functions (`26597`)
- Added ``encoding`` argument to `DataFrame.to_string` for non-ascii text (`28766`)
- Added ``encoding`` argument to `DataFrame.to_html` for non-ascii text (`28663`)
- `Styler.background_gradient` now accepts ``vmin`` and ``vmax`` arguments (`12145`)
- `Styler.format` added the ``na_rep`` parameter to help format the missing values (`21527`, `28358`)
- `read_excel` now can read binary Excel (``.xlsb``) files by passing ``engine='pyxlsb'``. For more details and example usage, see the `Binary Excel files documentation <io.xlsb>`. Closes `8540`.
- The ``partition_cols`` argument in `DataFrame.to_parquet` now accepts a string (`27117`)
- `pandas.read_json` now parses ``NaN``, ``Infinity`` and ``-Infinity`` (`12213`)
- DataFrame constructor preserve ``ExtensionArray`` dtype with ``ExtensionArray`` (`11363`)
- `DataFrame.sort_values` and `Series.sort_values` have gained ``ignore_index`` keyword to be able to reset index after sorting (`30114`)
- `DataFrame.sort_index` and `Series.sort_index` have gained ``ignore_index`` keyword to reset index (`30114`)
- `DataFrame.drop_duplicates` has gained ``ignore_index`` keyword to reset index (`30114`)
- Added new writer for exporting Stata dta files in versions 118 and 119, ``StataWriterUTF8``.  These files formats support exporting strings containing Unicode characters. Format 119 supports data sets with more than 32,767 variables (`23573`, `30959`)
- `Series.map` now accepts ``collections.abc.Mapping`` subclasses as a mapper (`29733`)
- Added an experimental `~DataFrame.attrs` for storing global metadata about a dataset (`29062`)
- `Timestamp.fromisocalendar` is now compatible with python 3.8 and above (`28115`)
- `DataFrame.to_pickle` and `read_pickle` now accept URL (`30163`)  .. ---  .. _whatsnew_100.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_100.api_breaking.MultiIndex._names:  Avoid using names from ``MultiIndex.levels``
^^^  As part of a larger refactor to `MultiIndex` the level names are now
stored separately from the levels (`27242`). We recommend using
`MultiIndex.names` to access the names, and `Index.set_names`
to update the names.  For backwards compatibility, you can still *access* the names via the levels.  .. ipython:: python  mi = pd.MultiIndex.from_product([[1, 2], ['a', 'b']], names=['x', 'y'])  mi.levels[0].name  However, it is no longer possible to *update* the names of the ``MultiIndex``
via the level.  .. ipython:: python  mi.levels[0].name = "new name"  mi.names  To update, use ``MultiIndex.set_names``, which returns a new ``MultiIndex``.  .. ipython:: python  mi2 = mi.set_names("new name", level=0)  mi2.names  New repr for `~pandas.arrays.IntervalArray`
^^^  `pandas.arrays.IntervalArray` adopts a new ``__repr__`` in accordance with other array classes (`25022`)  *pandas 0.25.x*  .. code-block:: ipython  In [1]: pd.arrays.IntervalArray.from_tuples([(0, 1), (2, 3)])  Out[2]:  IntervalArray([(0, 1], (2, 3]],  closed='right',  dtype='interval[int64]')  *pandas 1.0.0*  .. ipython:: python  pd.arrays.IntervalArray.from_tuples([(0, 1), (2, 3)])  ``DataFrame.rename`` now only accepts one positional argument
^^^  `DataFrame.rename` would previously accept positional arguments that would lead
to ambiguous or undefined behavior. From pandas 1.0, only the very first argument, which
maps labels to their new names along the default axis, is allowed to be passed by position

(`29136`).  .. ipython:: python  df = pd.DataFrame([[1]])  *pandas 0.25.x*  .. code-block:: ipython  In [1]: df = pd.DataFrame([[1]])  In [2]: df.rename({0: 1}, {0: 2})  Out[2]:  FutureWarning: ...Use named arguments to resolve ambiguity...  2  1  1  *pandas 1.0.0*  .. code-block:: ipython  In [3]: df.rename({0: 1}, {0: 2})  Traceback (most recent call last):  ...  TypeError: rename() takes from 1 to 2 positional arguments but 3 were given  Note that errors will now be raised when conflicting or potentially ambiguous arguments are provided.  *pandas 0.25.x*  .. code-block:: ipython  In [4]: df.rename({0: 1}, index={0: 2})  Out[4]:  0  1  1  In [5]: df.rename(mapper={0: 1}, index={0: 2})  Out[5]:  0  2  1  *pandas 1.0.0*  .. code-block:: ipython  In [6]: df.rename({0: 1}, index={0: 2})  Traceback (most recent call last):  ...  TypeError: Cannot specify both 'mapper' and any of 'index' or 'columns'  In [7]: df.rename(mapper={0: 1}, index={0: 2})  Traceback (most recent call last):  ...  TypeError: Cannot specify both 'mapper' and any of 'index' or 'columns'  You can still change the axis along which the first positional argument is applied by
supplying the ``axis`` keyword argument.  .. ipython:: python  df.rename({0: 1})  df.rename({0: 1}, axis=1)  If you would like to update both the index and column labels, be sure to use the respective
keywords.  .. ipython:: python  df.rename(index={0: 1}, columns={0: 2})  Extended verbose info output for `~pandas.DataFrame`
^^^  `DataFrame.info` now shows line numbers for the columns summary (`17304`)  *pandas 0.25.x*  .. code-block:: ipython  In [1]: df = pd.DataFrame({"int_col": [1, 2, 3],  ...  "text_col": ["a", "b", "c"],  ...  "float_col": [0.0, 0.1, 0.2]})  In [2]: df.info(verbose=True)  <class 'pandas.core.frame.DataFrame'>  RangeIndex: 3 entries, 0 to 2  Data columns (total 3 columns):  int_col  3 non-null int64  text_col  3 non-null object  float_col  3 non-null float64  dtypes: float64(1), int64(1), object(1)  memory usage: 152.0+ bytes  *pandas 1.0.0*  .. ipython:: python  df = pd.DataFrame({"int_col": [1, 2, 3],  "text_col": ["a", "b", "c"],  "float_col": [0.0, 0.1, 0.2]})  df.info(verbose=True)  `pandas.array` inference changes
^^^  `pandas.array` now infers pandas' new extension types in several cases (`29791`):  1. String data (including missing values) now returns a `arrays.StringArray`.
2. Integer data (including missing values) now returns a `arrays.IntegerArray`.
3. Boolean data (including missing values) now returns the new `arrays.BooleanArray`  *pandas 0.25.x*  .. code-block:: ipython  In [1]: pd.array(["a", None])  Out[1]:  <PandasArray>  ['a', None]  Length: 2, dtype: object  In [2]: pd.array([1, None])  Out[2]:  <PandasArray>  [1, None]  Length: 2, dtype: object  *pandas 1.0.0*  .. ipython:: python  pd.array(["a", None])  pd.array([1, None])  As a reminder, you can specify the ``dtype`` to disable all inference.  `arrays.IntegerArray` now uses `pandas.NA`
^^^  `arrays.IntegerArray` now uses `pandas.NA` rather than
`numpy.nan` as its missing value marker (`29964`).  *pandas 0.25.x*  .. code-block:: ipython  In [1]: a = pd.array([1, 2, None], dtype="Int64")  In [2]: a  Out[2]:  <IntegerArray>  [1, 2, NaN]  Length: 3, dtype: Int64  In [3]: a[2]  Out[3]:  nan  *pandas 1.0.0*  .. ipython:: python  a = pd.array([1, 2, None], dtype="Int64")  a  a[2]  This has a few API-breaking consequences.  **Converting to a NumPy ndarray**  When converting to a NumPy array missing values will be ``pd.NA``, which cannot
be converted to a float. So calling ``np.asarray(integer_array, dtype="float")``

will now raise.  *pandas 0.25.x*  .. code-block:: ipython  In [1]: np.asarray(a, dtype="float")  Out[1]:  array([ 1.,  2., nan])  *pandas 1.0.0*  .. ipython:: python  np.asarray(a, dtype="float")  Use `arrays.IntegerArray.to_numpy` with an explicit ``na_value`` instead.  .. ipython:: python  a.to_numpy(dtype="float", na_value=np.nan)  **Reductions can return** ``pd.NA``  When performing a reduction such as a sum with ``skipna=False``, the result
will now be ``pd.NA`` instead of ``np.nan`` in presence of missing values
(`30958`).  *pandas 0.25.x*  .. code-block:: ipython  In [1]: pd.Series(a).sum(skipna=False)  Out[1]:  nan  *pandas 1.0.0*  .. ipython:: python  pd.Series(a).sum(skipna=False)  **value_counts returns a nullable integer dtype**  `Series.value_counts` with a nullable integer dtype now returns a nullable
integer dtype for the values.  *pandas 0.25.x*  .. code-block:: ipython  In [1]: pd.Series([2, 1, 1, None], dtype="Int64").value_counts().dtype  Out[1]:  dtype('int64')  *pandas 1.0.0*  .. ipython:: python  pd.Series([2, 1, 1, None], dtype="Int64").value_counts().dtype  See `missing_data.NA` for more on the differences between `pandas.NA`
and `numpy.nan`.  `arrays.IntegerArray` comparisons return `arrays.BooleanArray`
^^^  Comparison operations on a `arrays.IntegerArray` now returns a
`arrays.BooleanArray` rather than a NumPy array (`29964`).  *pandas 0.25.x*  .. code-block:: ipython  In [1]: a = pd.array([1, 2, None], dtype="Int64")  In [2]: a  Out[2]:  <IntegerArray>  [1, 2, NaN]  Length: 3, dtype: Int64  In [3]: a > 1  Out[3]:  array([False,  True, False])  *pandas 1.0.0*  .. ipython:: python  a = pd.array([1, 2, None], dtype="Int64")  a > 1  Note that missing values now propagate, rather than always comparing unequal
like `numpy.nan`. See `missing_data.NA` for more.  By default `Categorical.min` now returns the minimum instead of np.nan
^^^  When `Categorical` contains ``np.nan``,
`Categorical.min` no longer return ``np.nan`` by default (skipna=True) (`25303`)  *pandas 0.25.x*  .. code-block:: ipython  In [1]: pd.Categorical([1, 2, np.nan], ordered=True).min()  Out[1]: nan  *pandas 1.0.0*  .. ipython:: python  pd.Categorical([1, 2, np.nan], ordered=True).min()  Default dtype of empty `pandas.Series`
^^^  Initialising an empty `pandas.Series` without specifying a dtype will raise a ``DeprecationWarning`` now
(`17261`). The default dtype will change from ``float64`` to ``object`` in future releases so that it is
consistent with the behaviour of `DataFrame` and `Index`.  *pandas 1.0.0*  .. code-block:: ipython  In [1]: pd.Series()  Out[2]:  DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.  Series([], dtype: float64)  Result dtype inference changes for resample operations
^^^  The rules for the result dtype in `DataFrame.resample` aggregations have changed for extension types (`31359`).
Previously, pandas would attempt to convert the result back to the original dtype, falling back to the usual
inference rules if that was not possible. Now, pandas will only return a result of the original dtype if the
scalar values in the result are instances of the extension dtype's scalar type.  .. ipython:: python  df = pd.DataFrame({"A": ['a', 'b']}, dtype='category',  index=pd.date_range('2000', periods=2))  df  *pandas 0.25.x*  .. code-block:: ipython  In [1]> df.resample("2D").agg(lambda x: 'a').A.dtype  Out[1]:  CategoricalDtype(categories=['a', 'b'], ordered=False)  *pandas 1.0.0*  .. ipython:: python  df.resample("2D").agg(lambda x: 'a').A.dtype  This fixes an inconsistency between ``resample`` and ``groupby``.
This also fixes a potential bug, where the **values** of the result might change

depending on how the results are cast back to the original dtype.  *pandas 0.25.x*  .. code-block:: ipython  In [1] df.resample("2D").agg(lambda x: 'c')  Out[1]:  A  0  NaN  *pandas 1.0.0*  .. ipython:: python  df.resample("2D").agg(lambda x: 'c')  .. _whatsnew_100.api_breaking.python:  Increased minimum version for Python
^^^  pandas 1.0.0 supports Python 3.6.1 and higher (`29212`).  .. _whatsnew_100.api_breaking.deps:  Increased minimum versions for dependencies
^^^  Some minimum supported versions of dependencies were updated (`29766`, `29723`).
| Package  | Minimum Version | Required | Changed |
| numpy  | 1.13.3  |  X  |  |
| pytz  | 2015.4  |  X  |  |
| python-dateutil | 2.6.1  |  X  |  |
| bottleneck  | 1.2.1  |  |  |
| numexpr  | 2.6.2  |  |  |
| pytest (dev)  | 4.0.2  |  |  |
+---+---+---+---+  For optional libraries the general recommendation is to use the latest version.
Optional libraries below the lowest tested version may still work, but are not considered supported.  +---+---+---+
| Package  | Minimum Version | Changed |
| beautifulsoup4  | 4.6.0  |  |
| fastparquet  | 0.3.2  |  X  |
| gcsfs  | 0.2.2  |  |
| lxml  | 3.8.0  |  |
| matplotlib  | 2.2.2  |  |
| numba  | 0.46.0  |  X  |
| openpyxl  | 2.5.7  |  X  |
| pyarrow  | 0.13.0  |  X  |
| pymysql  | 0.7.1  |  |
| s3fs  | 0.3.0  |  X  |
| scipy  | 0.19.0  |  |
| sqlalchemy  | 1.1.4  |  |
| xarray  | 0.8.2  |  |
| xlrd  | 1.1.0  |  |
| xlsxwriter  | 0.9.8  |  |
| xlwt  | 1.2.0  |  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  Build changes
^^^  pandas has added a pyproject.toml file and will no longer include
cythonized files in the source distribution uploaded to PyPI (`28341`, `20775`). If you're installing
a built distribution (wheel) or via conda, this shouldn't have any effect on you. If you're building pandas from
source, you should no longer need to install Cython into your build environment before calling ``pip install pandas``.  .. _whatsnew_100.api.other:  Other API changes
^^^  - `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` now raises on invalid operation names (`27489`)
- `pandas.api.types.infer_dtype` will now return "integer-na" for integer and ``np.nan`` mix (`27283`)
- `MultiIndex.from_arrays` will no longer infer names from arrays if ``names=None`` is explicitly provided (`27292`)
- In order to improve tab-completion, pandas does not include most deprecated attributes when introspecting a pandas object using ``dir`` (e.g. ``dir(df)``).  To see which attributes are excluded, see an object's ``_deprecations`` attribute, for example ``pd.DataFrame._deprecations`` (`28805`).
- The returned dtype of `unique` now matches the input dtype. (`27874`)
- Changed the default configuration value for ``options.matplotlib.register_converters`` from ``True`` to ``"auto"`` (`18720`).  Now, pandas custom formatters will only be applied to plots created by pandas, through `~DataFrame.plot`.  Previously, pandas' formatters would be applied to all plots created *after* a `~DataFrame.plot`.  See `units registration <whatsnew_100.matplotlib_units>` for more.
- `Series.dropna` has dropped its ``**kwargs`` argument in favor of a single ``how`` parameter.  Supplying anything else than ``how`` to ``**kwargs`` raised a ``TypeError`` previously (`29388`)
- When testing pandas, the new minimum required version of pytest is 5.0.1 (`29664`)

- `Series.str.__iter__` was deprecated and will be removed in future releases (`28277`).
- Added ``<NA>`` to the list of default NA values for `read_csv` (`30821`)  .. _whatsnew_100.api.documentation:  Documentation improvements
^^^  - Added new section on `scale` (`28315`).
- Added sub-section on `io.query_multi` for HDF5 datasets (`28791`).  .. ---  .. _whatsnew_100.deprecations:  Deprecations
~~~  - `Series.item` and `Index.item` have been _undeprecated_ (`29250`)
- ``Index.set_value`` has been deprecated. For a given index ``idx``, array ``arr``,  value in ``idx`` of ``idx_val`` and a new value of ``val``, ``idx.set_value(arr, idx_val, val)``  is equivalent to ``arr[idx.get_loc(idx_val)] = val``, which should be used instead (`28621`).
- `is_extension_type` is deprecated, `is_extension_array_dtype` should be used instead (`29457`)
- `eval` keyword argument "truediv" is deprecated and will be removed in a future version (`29812`)
- `DateOffset.isAnchored` and `DatetOffset.onOffset` are deprecated and will be removed in a future version, use `DateOffset.is_anchored` and `DateOffset.is_on_offset` instead (`30340`)
- ``pandas.tseries.frequencies.get_offset`` is deprecated and will be removed in a future version, use ``pandas.tseries.frequencies.to_offset`` instead (`4205`)
- `Categorical.take_nd` and `CategoricalIndex.take_nd` are deprecated, use `Categorical.take` and `CategoricalIndex.take` instead (`27745`)
- The parameter ``numeric_only`` of `Categorical.min` and `Categorical.max` is deprecated and replaced with ``skipna`` (`25303`)
- The parameter ``label`` in `lreshape` has been deprecated and will be removed in a future version (`29742`)
- ``pandas.core.index`` has been deprecated and will be removed in a future version, the public classes are available in the top-level namespace (`19711`)
- `pandas.json_normalize` is now exposed in the top-level namespace.  Usage of ``json_normalize`` as ``pandas.io.json.json_normalize`` is now deprecated and  it is recommended to use ``json_normalize`` as `pandas.json_normalize` instead (`27586`).
- The ``numpy`` argument of `pandas.read_json` is deprecated (`28512`).
- `DataFrame.to_stata`, `DataFrame.to_feather`, and `DataFrame.to_parquet` argument "fname" is deprecated, use "path" instead (`23574`)
- The deprecated internal attributes ``_start``, ``_stop`` and ``_step`` of `RangeIndex` now raise a ``FutureWarning`` instead of a ``DeprecationWarning`` (`26581`)
- The ``pandas.util.testing`` module has been deprecated. Use the public API in ``pandas.testing`` documented at `api.general.testing` (`16232`).
- ``pandas.SparseArray`` has been deprecated.  Use ``pandas.arrays.SparseArray`` (`arrays.SparseArray`) instead. (`30642`)
- The parameter ``is_copy`` of `Series.take` and `DataFrame.take` has been deprecated and will be removed in a future version. (`27357`)
- Support for multi-dimensional indexing (e.g. ``index[:, None]``) on a `Index` is deprecated and will be removed in a future version, convert to a numpy array before indexing instead (`30588`)
- The ``pandas.np`` submodule is now deprecated. Import numpy directly instead (`30296`)
- The ``pandas.datetime`` class is now deprecated. Import from ``datetime`` instead (`30610`)
- `~DataFrame.diff` will raise a ``TypeError`` rather than implicitly losing the dtype of extension types in the future. Convert to the correct dtype before calling ``diff`` instead (`31025`)  **Selecting Columns from a Grouped DataFrame**  When selecting columns from a `DataFrameGroupBy` object, passing individual keys (or a tuple of keys) inside single brackets is deprecated,

a list of items should be used instead. (`23566`) For example:  .. code-block:: ipython  df = pd.DataFrame({  "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],  "B": np.random.randn(8),  "C": np.random.randn(8),  })  g = df.groupby('A')  # single key, returns SeriesGroupBy  g['B']  # tuple of single key, returns SeriesGroupBy  g[('B',)]  # tuple of multiple keys, returns DataFrameGroupBy, raises FutureWarning  g[('B', 'C')]  # multiple keys passed directly, returns DataFrameGroupBy, raises FutureWarning  # (implicitly converts the passed strings into a single tuple)  g['B', 'C']  # proper way, returns DataFrameGroupBy  g[['B', 'C']]  .. ---  .. _whatsnew_100.prior_deprecations:  Removal of prior version deprecations/changes
~~~  **Removed SparseSeries and SparseDataFrame**  ``SparseSeries``, ``SparseDataFrame`` and the ``DataFrame.to_sparse`` method
have been removed (`28425`). We recommend using a ``Series`` or
``DataFrame`` with sparse values instead. See `sparse.migration` for help
with migrating existing code.  .. _whatsnew_100.matplotlib_units:  **Matplotlib unit registration**  Previously, pandas would register converters with matplotlib as a side effect of importing pandas (`18720`).
This changed the output of plots made via matplotlib plots after pandas was imported, even if you were using
matplotlib directly rather than `~DataFrame.plot`.  To use pandas formatters with a matplotlib plot, specify  .. code-block:: ipython  In [1]: import pandas as pd  In [2]: pd.options.plotting.matplotlib.register_converters = True  Note that plots created by `DataFrame.plot` and `Series.plot` *do* register the converters
automatically. The only behavior change is when plotting a date-like object via ``matplotlib.pyplot.plot``
or ``matplotlib.Axes.plot``. See `plotting.formatters` for more.  **Other removals**  - Removed the previously deprecated keyword "index" from `read_stata`, `StataReader`, and `StataReader.read`, use "index_col" instead (`17328`)
- Removed ``StataReader.data`` method, use `StataReader.read` instead (`9493`)
- Removed ``pandas.plotting._matplotlib.tsplot``, use `Series.plot` instead (`19980`)
- ``pandas.tseries.converter.register`` has been moved to `pandas.plotting.register_matplotlib_converters` (`18307`)
- `Series.plot` no longer accepts positional arguments, pass keyword arguments instead (`30003`)
- `DataFrame.hist` and `Series.hist` no longer allows ``figsize="default"``, specify figure size by passinig a tuple instead (`30003`)
- Floordiv of integer-dtyped array by `Timedelta` now raises ``TypeError`` (`21036`)
- `TimedeltaIndex` and `DatetimeIndex` no longer accept non-nanosecond dtype strings like "timedelta64" or "datetime64", use "timedelta64[ns]" and "datetime64[ns]" instead (`24806`)
- Changed the default "skipna" argument in `pandas.api.types.infer_dtype` from ``False`` to ``True`` (`24050`)
- Removed ``Series.ix`` and ``DataFrame.ix`` (`26438`)
- Removed ``Index.summary`` (`18217`)
- Removed the previously deprecated keyword "fastpath" from the `Index` constructor (`23110`)
- Removed ``Series.get_value``, ``Series.set_value``, ``DataFrame.get_value``, ``DataFrame.set_value`` (`17739`)
- Removed ``Series.compound`` and ``DataFrame.compound`` (`26405`)
- Changed the default "inplace" argument in `DataFrame.set_index` and `Series.set_axis` from ``None`` to ``False`` (`27600`)
- Removed ``Series.cat.categorical``, ``Series.cat.index``, ``Series.cat.name`` (`24751`)
- Removed the previously deprecated keyword "box" from `to_datetime` and `to_timedelta`; in addition these now always returns `DatetimeIndex`, `TimedeltaIndex`, `Index`, `Series`, or `DataFrame` (`24486`)
- `to_timedelta`, `Timedelta`, and `TimedeltaIndex` no longer allow "M", "y", or "Y" for the "unit" argument (`23264`)
- Removed the previously deprecated keyword "time_rule" from (non-public) ``offsets.generate_range``, which has been moved to `core.arrays._ranges.generate_range` (`24157`)

- `DataFrame.loc` or `Series.loc` with listlike indexers and missing labels will no longer reindex (`17295`)
- `DataFrame.to_excel` and `Series.to_excel` with non-existent columns will no longer reindex (`17295`)
- Removed the previously deprecated keyword "join_axes" from `concat`; use ``reindex_like`` on the result instead (`22318`)
- Removed the previously deprecated keyword "by" from `DataFrame.sort_index`, use `DataFrame.sort_values` instead (`10726`)
- Removed support for nested renaming in `DataFrame.aggregate`, `Series.aggregate`, `core.groupby.DataFrameGroupBy.aggregate`, `core.groupby.SeriesGroupBy.aggregate`, `core.window.rolling.Rolling.aggregate` (`18529`)
- Passing ``datetime64`` data to `TimedeltaIndex` or ``timedelta64`` data to ``DatetimeIndex`` now raises ``TypeError`` (`23539`, `23937`)
- Passing ``int64`` values to `DatetimeIndex` and a timezone now interprets the values as nanosecond timestamps in UTC, not wall times in the given timezone (`24559`)
- A tuple passed to `DataFrame.groupby` is now exclusively treated as a single key (`18314`)
- Removed ``Index.contains``, use ``key in index`` instead (`30103`)
- Addition and subtraction of ``int`` or integer-arrays is no longer allowed in `Timestamp`, `DatetimeIndex`, `TimedeltaIndex`, use ``obj + n * obj.freq`` instead of ``obj + n`` (`22535`)
- Removed ``Series.ptp`` (`21614`)
- Removed ``Series.from_array`` (`18258`)
- Removed ``DataFrame.from_items`` (`18458`)
- Removed ``DataFrame.as_matrix``, ``Series.as_matrix`` (`18458`)
- Removed ``Series.asobject`` (`18477`)
- Removed ``DataFrame.as_blocks``, ``Series.as_blocks``, ``DataFrame.blocks``, ``Series.blocks`` (`17656`)
- `pandas.Series.str.cat` now defaults to aligning ``others``, using ``join='left'`` (`27611`)
- `pandas.Series.str.cat` does not accept list-likes *within* list-likes anymore (`27611`)
- `Series.where` with ``Categorical`` dtype (or `DataFrame.where` with ``Categorical`` column) no longer allows setting new categories (`24114`)
- Removed the previously deprecated keywords "start", "end", and "periods" from the `DatetimeIndex`, `TimedeltaIndex`, and `PeriodIndex` constructors; use `date_range`, `timedelta_range`, and `period_range` instead (`23919`)
- Removed the previously deprecated keyword "verify_integrity" from the `DatetimeIndex` and `TimedeltaIndex` constructors (`23919`)
- Removed the previously deprecated keyword "fastpath" from ``pandas.core.internals.blocks.make_block`` (`19265`)
- Removed the previously deprecated keyword "dtype" from `Block.make_block_same_class` (`19434`)
- Removed ``ExtensionArray._formatting_values``. Use `ExtensionArray._formatter` instead. (`23601`)
- Removed ``MultiIndex.to_hierarchical`` (`21613`)
- Removed ``MultiIndex.labels``, use `MultiIndex.codes` instead (`23752`)
- Removed the previously deprecated keyword "labels" from the `MultiIndex` constructor, use "codes" instead (`23752`)
- Removed ``MultiIndex.set_labels``, use `MultiIndex.set_codes` instead (`23752`)
- Removed the previously deprecated keyword "labels" from `MultiIndex.set_codes`, `MultiIndex.copy`, `MultiIndex.drop`, use "codes" instead (`23752`)
- Removed support for legacy HDF5 formats (`29787`)
- Passing a dtype alias (e.g. 'datetime64[ns, UTC]') to `DatetimeTZDtype` is no longer allowed, use `DatetimeTZDtype.construct_from_string` instead (`23990`)
- Removed the previously deprecated keyword "skip_footer" from `read_excel`; use "skipfooter" instead (`18836`)
- `read_excel` no longer allows an integer value for the parameter ``usecols``, instead pass a list of integers from 0 to ``usecols`` inclusive (`23635`)
- Removed the previously deprecated keyword "convert_datetime64" from `DataFrame.to_records` (`18902`)
- Removed ``IntervalIndex.from_intervals`` in favor of the `IntervalIndex` constructor (`19263`)

- Changed the default "keep_tz" argument in `DatetimeIndex.to_series` from ``None`` to ``True`` (`23739`)
- Removed ``api.types.is_period`` and ``api.types.is_datetimetz`` (`23917`)
- Ability to read pickles containing `Categorical` instances created with pre-0.16 version of pandas has been removed (`27538`)
- Removed ``pandas.tseries.plotting.tsplot`` (`18627`)
- Removed the previously deprecated keywords "reduce" and "broadcast" from `DataFrame.apply` (`18577`)
- Removed the previously deprecated ``assert_raises_regex`` function in ``pandas._testing`` (`29174`)
- Removed the previously deprecated ``FrozenNDArray`` class in ``pandas.core.indexes.frozen`` (`29335`)
- Removed the previously deprecated keyword "nthreads" from `read_feather`, use "use_threads" instead (`23053`)
- Removed ``Index.is_lexsorted_for_tuple`` (`29305`)
- Removed support for nested renaming in `DataFrame.aggregate`, `Series.aggregate`, `core.groupby.DataFrameGroupBy.aggregate`, `core.groupby.SeriesGroupBy.aggregate`, `core.window.rolling.Rolling.aggregate` (`29608`)
- Removed ``Series.valid``; use `Series.dropna` instead (`18800`)
- Removed ``DataFrame.is_copy``, ``Series.is_copy`` (`18812`)
- Removed ``DataFrame.get_ftype_counts``, ``Series.get_ftype_counts`` (`18243`)
- Removed ``DataFrame.ftypes``, ``Series.ftypes``, ``Series.ftype`` (`26744`)
- Removed ``Index.get_duplicates``, use ``idx[idx.duplicated()].unique()`` instead (`20239`)
- Removed ``Series.clip_upper``, ``Series.clip_lower``, ``DataFrame.clip_upper``, ``DataFrame.clip_lower`` (`24203`)
- Removed the ability to alter `DatetimeIndex.freq`, `TimedeltaIndex.freq`, or `PeriodIndex.freq` (`20772`)
- Removed ``DatetimeIndex.offset`` (`20730`)
- Removed ``DatetimeIndex.asobject``, ``TimedeltaIndex.asobject``, ``PeriodIndex.asobject``, use ``astype(object)`` instead (`29801`)
- Removed the previously deprecated keyword "order" from `factorize` (`19751`)
- Removed the previously deprecated keyword "encoding" from `read_stata` and `DataFrame.to_stata` (`21400`)
- Changed the default "sort" argument in `concat` from ``None`` to ``False`` (`20613`)
- Removed the previously deprecated keyword "raise_conflict" from `DataFrame.update`, use "errors" instead (`23585`)
- Removed the previously deprecated keyword "n" from `DatetimeIndex.shift`, `TimedeltaIndex.shift`, `PeriodIndex.shift`, use "periods" instead (`22458`)
- Removed the previously deprecated keywords "how", "fill_method", and "limit" from `DataFrame.resample` (`30139`)
- Passing an integer to `Series.fillna` or `DataFrame.fillna` with ``timedelta64[ns]`` dtype now raises ``TypeError`` (`24694`)
- Passing multiple axes to `DataFrame.dropna` is no longer supported (`20995`)
- Removed ``Series.nonzero``, use ``to_numpy().nonzero()`` instead (`24048`)
- Passing floating dtype ``codes`` to `Categorical.from_codes` is no longer supported, pass ``codes.astype(np.int64)`` instead (`21775`)
- Removed the previously deprecated keyword "pat" from `Series.str.partition` and `Series.str.rpartition`, use "sep" instead (`23767`)
- Removed ``Series.put`` (`27106`)
- Removed ``Series.real``, ``Series.imag`` (`27106`)
- Removed ``Series.to_dense``, ``DataFrame.to_dense`` (`26684`)
- Removed ``Index.dtype_str``, use ``str(index.dtype)`` instead (`27106`)
- `Categorical.ravel` returns a `Categorical` instead of a ``ndarray`` (`27199`)
- The 'outer' method on Numpy ufuncs, e.g. ``np.subtract.outer`` operating on `Series` objects is no longer supported, and will raise ``NotImplementedError`` (`27198`)
- Removed ``Series.get_dtype_counts`` and ``DataFrame.get_dtype_counts`` (`27145`)
- Changed the default "fill_value" argument in `Categorical.take` from ``True`` to ``False`` (`20841`)

- Changed the default value for the ``raw`` argument in `Series.rolling().apply() <pandas.core.window.Rolling.apply>`, `DataFrame.rolling().apply() <pandas.core.window.Rolling.apply>`, `Series.expanding().apply() <pandas.core.window.Expanding.apply>`, and `DataFrame.expanding().apply() <pandas.core.window.Expanding.apply>` from ``None`` to ``False`` (`20584`)
- Removed deprecated behavior of `Series.argmin` and `Series.argmax`, use `Series.idxmin` and `Series.idxmax` for the old behavior (`16955`)
- Passing a tz-aware ``datetime.datetime`` or `Timestamp` into the `Timestamp` constructor with the ``tz`` argument now raises a ``ValueError`` (`23621`)
- Removed ``Series.base``, ``Index.base``, ``Categorical.base``, ``Series.flags``, ``Index.flags``, ``PeriodArray.flags``, ``Series.strides``, ``Index.strides``, ``Series.itemsize``, ``Index.itemsize``, ``Series.data``, ``Index.data`` (`20721`)
- Changed `Timedelta.resolution` to match the behavior of the standard library ``datetime.timedelta.resolution``, for the old behavior, use `Timedelta.resolution_string` (`26839`)
- Removed ``Timestamp.weekday_name``, ``DatetimeIndex.weekday_name``, and ``Series.dt.weekday_name`` (`18164`)
- Removed the previously deprecated keyword "errors" in `Timestamp.tz_localize`, `DatetimeIndex.tz_localize`, and `Series.tz_localize` (`22644`)
- Changed the default "ordered" argument in `CategoricalDtype` from ``None`` to ``False`` (`26336`)
- `Series.set_axis` and `DataFrame.set_axis` now require "labels" as the first argument and "axis" as an optional named parameter (`30089`)
- Removed ``to_msgpack``, ``read_msgpack``, ``DataFrame.to_msgpack``, ``Series.to_msgpack`` (`27103`)
- Removed ``Series.compress`` (`21930`)
- Removed the previously deprecated keyword "fill_value" from `Categorical.fillna`, use "value" instead (`19269`)
- Removed the previously deprecated keyword "data" from `andrews_curves`, use "frame" instead (`6956`)
- Removed the previously deprecated keyword "data" from `parallel_coordinates`, use "frame" instead (`6956`)
- Removed the previously deprecated keyword "colors" from `parallel_coordinates`, use "color" instead (`6956`)
- Removed the previously deprecated keywords "verbose" and "private_key" from `read_gbq` (`30200`)

- Calling ``np.array`` and ``np.asarray`` on tz-aware `Series` and `DatetimeIndex` will now return an object array of tz-aware `Timestamp` (`24596`)

.. ---  .. _whatsnew_100.performance:  Performance improvements
~~~  - Performance improvement in `DataFrame` arithmetic and comparison operations with scalars (`24990`, `29853`)
- Performance improvement in indexing with a non-unique `IntervalIndex` (`27489`)
- Performance improvement in `MultiIndex.is_monotonic` (`27495`)
- Performance improvement in `cut` when ``bins`` is an `IntervalIndex` (`27668`)
- Performance improvement when initializing a `DataFrame` using a ``range`` (`30171`)
- Performance improvement in `DataFrame.corr` when ``method`` is ``"spearman"`` (`28139`)
- Performance improvement in `DataFrame.replace` when provided a list of values to replace (`28099`)
- Performance improvement in `DataFrame.select_dtypes` by using vectorization instead of iterating over a loop (`28317`)
- Performance improvement in `Categorical.searchsorted` and  `CategoricalIndex.searchsorted` (`28795`)
- Performance improvement when comparing a `Categorical` with a scalar and the scalar is not found in the categories (`29750`)
- Performance improvement when checking if values in a `Categorical` are equal, equal or larger or larger than a given scalar.  The improvement is not present if checking if the `Categorical` is less than or less than or equal than the scalar (`29820`)
- Performance improvement in `Index.equals` and  `MultiIndex.equals` (`29134`)
- Performance improvement in `~pandas.api.types.infer_dtype` when ``skipna`` is ``True`` (`28814`)  .. ---  .. _whatsnew_100.bug_fixes:  Bug fixes
^^^  - Added test to assert the `fillna` raises the correct ``ValueError`` message when the value isn't a value from categories (`13628`)
- Bug in `Categorical.astype` where ``NaN`` values were handled incorrectly when casting to int (`28406`)
- `DataFrame.reindex` with a `CategoricalIndex` would fail when the targets contained duplicates, and wouldn't fail if the source contained duplicates (`28107`)
- Bug in `Categorical.astype` not allowing for casting to extension dtypes (`28668`)
- Bug where `merge` was unable to join on categorical and extension dtype columns (`28668`)
- `Categorical.searchsorted` and `CategoricalIndex.searchsorted` now work on unordered categoricals also (`21667`)
- Added test to assert roundtripping to parquet with `DataFrame.to_parquet` or `read_parquet` will preserve Categorical dtypes for string types (`27955`)
- Changed the error message in `Categorical.remove_categories` to always show the invalid removals as a set (`28669`)
- Using date accessors on a categorical dtyped `Series` of datetimes was not returning an object of the  same type as if one used the `.str.` / `.dt.` on a `Series` of that type. E.g. when accessing `Series.dt.tz_localize` on a  `Categorical` with duplicate entries, the accessor was skipping duplicates (`27952`)
- Bug in `DataFrame.replace` and `Series.replace` that would give incorrect results on categorical data (`26988`)
- Bug where calling `Categorical.min` or `Categorical.max` on an empty Categorical would raise a numpy exception (`30227`)
- The following methods now also correctly output values for unobserved categories when called through ``groupby(..., observed=False)`` (`17605`)  * `core.groupby.SeriesGroupBy.count`  * `core.groupby.SeriesGroupBy.size`  * `core.groupby.SeriesGroupBy.nunique`  * `core.groupby.SeriesGroupBy.nth`  Datetimelike
^^^
- Bug in `Series.__setitem__` incorrectly casting ``np.timedelta64("NaT")`` to ``np.datetime64("NaT")`` when inserting into a `Series` with datetime64 dtype (`27311`)
- Bug in `Series.dt` property lookups when the underlying data is read-only (`27529`)
- Bug in ``HDFStore.__getitem__`` incorrectly reading tz attribute created in Python 2 (`26443`)
- Bug in `to_datetime` where passing arrays of malformed ``str`` with errors="coerce" could incorrectly lead to raising ``ValueError`` (`28299`)
- Bug in `core.groupby.SeriesGroupBy.nunique` where ``NaT`` values were interfering with the count of unique values (`27951`)

- Bug in `Timestamp` subtraction when subtracting a `Timestamp` from a ``np.datetime64`` object incorrectly raising ``TypeError`` (`28286`)
- Addition and subtraction of integer or integer-dtype arrays with `Timestamp` will now raise ``NullFrequencyError`` instead of ``ValueError`` (`28268`)
- Bug in `Series` and `DataFrame` with integer dtype failing to raise ``TypeError`` when adding or subtracting a ``np.datetime64`` object (`28080`)
- Bug in `Series.astype`, `Index.astype`, and `DataFrame.astype` failing to handle ``NaT`` when casting to an integer dtype (`28492`)
- Bug in `Week` with ``weekday`` incorrectly raising ``AttributeError`` instead of ``TypeError`` when adding or subtracting an invalid type (`28530`)
- Bug in `DataFrame` arithmetic operations when operating with a `Series` with dtype ``'timedelta64[ns]'`` (`28049`)
- Bug in `core.groupby.generic.SeriesGroupBy.apply` raising ``ValueError`` when a column in the original DataFrame is a datetime and the column labels are not standard integers (`28247`)
- Bug in `pandas._config.localization.get_locales` where the ``locales -a`` encodes the locales list as windows-1252 (`23638`, `24760`, `27368`)
- Bug in `Series.var` failing to raise ``TypeError`` when called with ``timedelta64[ns]`` dtype (`28289`)
- Bug in `DatetimeIndex.strftime` and `Series.dt.strftime` where ``NaT`` was converted to the string ``'NaT'`` instead of ``np.nan`` (`29578`)
- Bug in masking datetime-like arrays with a boolean mask of an incorrect length not raising an ``IndexError`` (`30308`)
- Bug in `Timestamp.resolution` being a property instead of a class attribute (`29910`)
- Bug in `pandas.to_datetime` when called with ``None`` raising ``TypeError`` instead of returning ``NaT`` (`30011`)
- Bug in `pandas.to_datetime` failing for ``deques`` when using ``cache=True`` (the default) (`29403`)
- Bug in `Series.item` with ``datetime64`` or ``timedelta64`` dtype, `DatetimeIndex.item`, and `TimedeltaIndex.item` returning an integer instead of a `Timestamp` or `Timedelta` (`30175`)
- Bug in `DatetimeIndex` addition when adding a non-optimized `DateOffset` incorrectly dropping timezone information (`30336`)
- Bug in `DataFrame.drop` where attempting to drop non-existent values from a DatetimeIndex would yield a confusing error message (`30399`)
- Bug in `DataFrame.append` would remove the timezone-awareness of new data (`30238`)
- Bug in `Series.cummin` and `Series.cummax` with timezone-aware dtype incorrectly dropping its timezone (`15553`)
- Bug in `DatetimeArray`, `TimedeltaArray`, and `PeriodArray` where inplace addition and subtraction did not actually operate inplace (`24115`)
- Bug in `pandas.to_datetime` when called with ``Series`` storing ``IntegerArray`` raising ``TypeError`` instead of returning ``Series`` (`30050`)
- Bug in `date_range` with custom business hours as ``freq`` and given number of ``periods`` (`30593`)
- Bug in `PeriodIndex` comparisons with incorrectly casting integers to `Period` objects, inconsistent with the `Period` comparison behavior (`30722`)
- Bug in `DatetimeIndex.insert` raising a ``ValueError`` instead of a ``TypeError`` when trying to insert a timezone-aware `Timestamp` into a timezone-naive `DatetimeIndex`, or vice-versa (`30806`)  Timedelta
^^^

- Bug in subtracting a `TimedeltaIndex` or `TimedeltaArray` from a ``np.datetime64`` object (`29558`)

Timezones
^^^

Numeric
^^^
- Bug in `DataFrame.quantile` with zero-column `DataFrame` incorrectly raising (`23925`)
- `DataFrame` flex inequality comparisons methods (`DataFrame.lt`, `DataFrame.le`, `DataFrame.gt`, `DataFrame.ge`) with object-dtype and ``complex`` entries failing to raise ``TypeError`` like their `Series` counterparts (`28079`)
- Bug in `DataFrame` logical operations (``&``, ``|``, ``^``) not matching `Series` behavior by filling NA values (`28741`)
- Bug in `DataFrame.interpolate` where specifying axis by name references variable before it is assigned (`29142`)
- Bug in `Series.var` not computing the right value with a nullable integer dtype series not passing through ddof argument (`29128`)
- Improved error message when using ``frac`` > 1 and ``replace`` = False (`27451`)
- Bug in numeric indexes resulted in it being possible to instantiate an `Int64Index`, `UInt64Index`, or `Float64Index` with an invalid dtype (e.g. datetime-like) (`29539`)
- Bug in `UInt64Index` precision loss while constructing from a list with values in the ``np.uint64`` range (`29526`)
- Bug in `NumericIndex` construction that caused indexing to fail when integers in the ``np.uint64`` range were used (`28023`)
- Bug in `NumericIndex` construction that caused `UInt64Index` to be casted to `Float64Index` when integers in the ``np.uint64`` range were used to index a `DataFrame` (`28279`)
- Bug in `Series.interpolate` when using method=`index` with an unsorted index, would previously return incorrect results. (`21037`)
- Bug in `DataFrame.round` where a `DataFrame` with a `CategoricalIndex` of `IntervalIndex` columns would incorrectly raise a ``TypeError`` (`30063`)
- Bug in `Series.pct_change` and `DataFrame.pct_change` when there are duplicated indices (`30463`)
- Bug in `DataFrame` cumulative operations (e.g. cumsum, cummax) incorrect casting to object-dtype (`19296`)
- Bug in `~DataFrame.diff` losing the dtype for extension types (`30889`)
- Bug in `DataFrame.diff` raising an ``IndexError`` when one of the columns was a nullable integer dtype (`30967`)  Conversion
^^^

- Calling `Series.str.isalnum` (and other "ismethods") on an empty ``Series`` would return an ``object`` dtype instead of ``bool`` (`29624`)
  Interval
^^^  - Bug in `IntervalIndex.get_indexer` where a `Categorical` or `CategoricalIndex` ``target`` would incorrectly raise a ``TypeError`` (`30063`)
- Bug in ``pandas.core.dtypes.cast.infer_dtype_from_scalar`` where passing ``pandas_dtype=True`` did not infer `IntervalDtype` (`30337`)
- Bug in `Series` constructor where constructing a ``Series`` from a ``list`` of `Interval` objects resulted in ``object`` dtype instead of `IntervalDtype` (`23563`)
- Bug in `IntervalDtype` where the ``kind`` attribute was incorrectly set as ``None`` instead of ``"O"`` (`30568`)
- Bug in `IntervalIndex`, `~arrays.IntervalArray`, and `Series` with interval data where equality comparisons were incorrect (`24112`)  Indexing
^^^  - Bug in assignment using a reverse slicer (`26939`)
- Bug in `DataFrame.explode` would duplicate frame in the presence of duplicates in the index (`28010`)
- Bug in reindexing a `PeriodIndex` with another type of index that contained a ``Period`` (`28323`) (`28337`)
- Fix assignment of column via ``.loc`` with numpy non-ns datetime type (`27395`)
- Bug in `Float64Index.astype` where ``np.inf`` was not handled properly when casting to an integer dtype (`28475`)
- `Index.union` could fail when the left contained duplicates (`28257`)
- Bug when indexing with ``.loc`` where the index was a `CategoricalIndex` with non-string categories didn't work (`17569`, `30225`)
- `Index.get_indexer_non_unique` could fail with ``TypeError`` in some cases, such as when searching for ints in a string index (`28257`)
- Bug in `Float64Index.get_loc` incorrectly raising ``TypeError`` instead of ``KeyError`` (`29189`)
- Bug in `DataFrame.loc` with incorrect dtype when setting Categorical value in 1-row DataFrame (`25495`)
- `MultiIndex.get_loc` can't find missing values when input includes missing values (`19132`)
- Bug in `Series.__setitem__` incorrectly assigning values with boolean indexer when the length of new data matches the number of ``True`` values and new data is not a ``Series`` or an ``np.array`` (`30567`)
- Bug in indexing with a `PeriodIndex` incorrectly accepting integers representing years, use e.g. ``ser.loc["2007"]`` instead of ``ser.loc[2007]`` (`30763`)  Missing
^^^

^^^  - Constructor for `MultiIndex` verifies that the given ``sortorder`` is compatible with the actual ``lexsort_depth``  if ``verify_integrity`` parameter is ``True`` (the default) (`28735`)

- Series and MultiIndex ``.drop`` with ``MultiIndex`` raise exception if labels not in given in level (`8594`)

IO
^^  - `read_csv` now accepts binary mode file buffers when using the Python csv engine (`23779`)
- Bug in `DataFrame.to_json` where using a Tuple as a column or index value and using ``orient="columns"`` or ``orient="index"`` would produce invalid JSON (`20500`)
- Improve infinity parsing. `read_csv` now interprets ``Infinity``, ``+Infinity``, ``-Infinity`` as floating point values (`10065`)
- Bug in `DataFrame.to_csv` where values were truncated when the length of ``na_rep`` was shorter than the text input data. (`25099`)
- Bug in `DataFrame.to_string` where values were truncated using display options instead of outputting the full content (`9784`)
- Bug in `DataFrame.to_json` where a datetime column label would not be written out in ISO format with ``orient="table"`` (`28130`)
- Bug in `DataFrame.to_parquet` where writing to GCS would fail with ``engine='fastparquet'`` if the file did not already exist (`28326`)
- Bug in `read_hdf` closing stores that it didn't open when Exceptions are raised (`28699`)
- Bug in `DataFrame.read_json` where using ``orient="index"`` would not maintain the order (`28557`)
- Bug in `DataFrame.to_html` where the length of the ``formatters`` argument was not verified (`28469`)
- Bug in `DataFrame.read_excel` with ``engine='ods'`` when ``sheet_name`` argument references a non-existent sheet (`27676`)
- Bug in `pandas.io.formats.style.Styler` formatting for floating values not displaying decimals correctly (`13257`)
- Bug in `DataFrame.to_html` when using ``formatters=<list>`` and ``max_cols`` together. (`25955`)
- Bug in `Styler.background_gradient` not able to work with dtype ``Int64`` (`28869`)
- Bug in `DataFrame.to_clipboard` which did not work reliably in ipython (`22707`)
- Bug in `read_json` where default encoding was not set to ``utf-8`` (`29565`)
- Bug in `PythonParser` where str and bytes were being mixed when dealing with the decimal field (`29650`)
- `read_gbq` now accepts ``progress_bar_type`` to display progress bar while the data downloads. (`29857`)
- Bug in `pandas.io.json.json_normalize` where a missing value in the location specified by ``record_path`` would raise a ``TypeError`` (`30148`)
- `read_excel` now accepts binary data (`15914`)
- Bug in `read_csv` in which encoding handling was limited to just the string ``utf-16`` for the C engine (`24130`)  Plotting
^^^  - Bug in `Series.plot` not able to plot boolean values (`23719`)
- Bug in `DataFrame.plot` not able to plot when no rows (`27758`)
- Bug in `DataFrame.plot` producing incorrect legend markers when plotting multiple series on the same axis (`18222`)
- Bug in `DataFrame.plot` when ``kind='box'`` and data contains datetime or timedelta data. These types are now automatically dropped (`22799`)
- Bug in `DataFrame.plot.line` and `DataFrame.plot.area` produce wrong xlim in x-axis (`27686`, `25160`, `24784`)
- Bug where `DataFrame.boxplot` would not accept a ``color`` parameter like `DataFrame.plot.box` (`26214`)
- Bug in the ``xticks`` argument being ignored for `DataFrame.plot.bar` (`14119`)
- `set_option` now validates that the plot backend provided to ``'plotting.backend'`` implements the backend when the option is set, rather than when a plot is created (`28163`)
- `DataFrame.plot` now allow a ``backend`` keyword argument to allow changing between backends in one session (`28619`).
- Bug in color validation incorrectly raising for non-color styles (`29122`).
- Allow `DataFrame.plot.scatter` to plot ``objects`` and ``datetime`` type data (`18755`, `30391`)
- Bug in `DataFrame.hist`, ``xrot=0`` does not work with ``by`` and subplots (`30288`).  GroupBy/resample/rolling
^^^  - Bug in `core.groupby.DataFrameGroupBy.apply` only showing output from a single group when function returns an `Index` (`28652`)
- Bug in `DataFrame.groupby` with multiple groups where an ``IndexError`` would be raised if any group contained all NA values (`20519`)

- Bug in `pandas.core.resample.Resampler.size` and `pandas.core.resample.Resampler.count` returning wrong dtype when used with an empty `Series` or `DataFrame` (`28427`)
- Bug in `DataFrame.rolling` not allowing for rolling over datetimes when ``axis=1`` (`28192`)
- Bug in `DataFrame.rolling` not allowing rolling over multi-index levels (`15584`).
- Bug in `DataFrame.rolling` not allowing rolling on monotonic decreasing time indexes (`19248`).
- Bug in `DataFrame.groupby` not offering selection by column name when ``axis=1`` (`27614`)
- Bug in `core.groupby.DataFrameGroupby.agg` not able to use lambda function with named aggregation (`27519`)
- Bug in `DataFrame.groupby` losing column name information when grouping by a categorical column (`28787`)
- Remove error raised due to duplicated input functions in named aggregation in `DataFrame.groupby` and `Series.groupby`. Previously error will be raised if the same function is applied on the same column and now it is allowed if new assigned names are different. (`28426`)
- `core.groupby.SeriesGroupBy.value_counts` will be able to handle the case even when the `Grouper` makes empty groups (`28479`)
- Bug in `core.window.rolling.Rolling.quantile` ignoring ``interpolation`` keyword argument when used within a groupby (`28779`)
- Bug in `DataFrame.groupby` where ``any``, ``all``, ``nunique`` and transform functions would incorrectly handle duplicate column labels (`21668`)
- Bug in `core.groupby.DataFrameGroupBy.agg` with timezone-aware datetime64 column incorrectly casting results to the original dtype (`29641`)
- Bug in `DataFrame.groupby` when using axis=1 and having a single level columns index (`30208`)
- Bug in `DataFrame.groupby` when using nunique on axis=1 (`30253`)
- Bug in `.DataFrameGroupBy.quantile` and `.SeriesGroupBy.quantile` with multiple list-like q value and integer column names (`30289`)
- Bug in `.DataFrameGroupBy.pct_change` and `.SeriesGroupBy.pct_change` causes ``TypeError`` when ``fill_method`` is ``None`` (`30463`)
- Bug in `Rolling.count` and `Expanding.count` argument where ``min_periods`` was ignored (`26996`)  Reshaping
^^^  - Bug in `DataFrame.apply` that caused incorrect output with empty `DataFrame` (`28202`, `21959`)
- Bug in `DataFrame.stack` not handling non-unique indexes correctly when creating MultiIndex (`28301`)
- Bug in `pivot_table` not returning correct type ``float`` when ``margins=True`` and ``aggfunc='mean'`` (`24893`)
- Bug `merge_asof` could not use `datetime.timedelta` for ``tolerance`` kwarg (`28098`)
- Bug in `merge`, did not append suffixes correctly with MultiIndex (`28518`)
- `qcut` and `cut` now handle boolean input (`20303`)
- Fix to ensure all int dtypes can be used in `merge_asof` when using a tolerance value. Previously every non-int64 type would raise an erroneous ``MergeError`` (`28870`).
- Better error message in `get_dummies` when ``columns`` isn't a list-like value (`28383`)
- Bug in `Index.join` that caused infinite recursion error for mismatched ``MultiIndex`` name orders. (`25760`, `28956`)
- Bug `Series.pct_change` where supplying an anchored frequency would throw a ``ValueError`` (`28664`)
- Bug where `DataFrame.equals` returned True incorrectly in some cases when two DataFrames had the same columns in different orders (`28839`)
- Bug in `DataFrame.replace` that caused non-numeric replacer's dtype not respected (`26632`)
- Bug in `melt` where supplying mixed strings and numeric values for ``id_vars`` or ``value_vars`` would incorrectly raise a ``ValueError`` (`29718`)
- Dtypes are now preserved when transposing a ``DataFrame`` where each column is the same extension dtype (`30091`)
- Bug in `merge_asof` merging on a tz-aware ``left_index`` and ``right_on`` a tz-aware column (`29864`)
- Improved error message and docstring in `cut` and `qcut` when ``labels=True`` (`13318`)

- Bug in missing ``fill_na`` parameter to `DataFrame.unstack` with list of levels (`30740`)  Sparse
^^^
- Bug in `SparseDataFrame` arithmetic operations incorrectly casting inputs to float (`28107`)
- Bug in ``DataFrame.sparse`` returning a ``Series`` when there was a column named ``sparse`` rather than the accessor (`30758`)
- Fixed `operator.xor` with a boolean-dtype ``SparseArray``. Now returns a sparse result, rather than object dtype (`31025`)  ExtensionArray
^^^  - Bug in `arrays.PandasArray` when setting a scalar string (`28118`, `28150`).
- Bug where nullable integers could not be compared to strings (`28930`)
- Bug where `DataFrame` constructor raised ``ValueError`` with list-like data and ``dtype`` specified (`30280`)  Other
^^^
- Trying to set the ``display.precision``, ``display.max_rows`` or ``display.max_columns`` using `set_option` to anything but a ``None`` or a positive int will raise a ``ValueError`` (`23348`)
- Using `DataFrame.replace` with overlapping keys in a nested dictionary will no longer raise, now matching the behavior of a flat dictionary (`27660`)
- `DataFrame.to_csv` and `Series.to_csv` now support dicts as ``compression`` argument with key ``'method'`` being the compression method and others as additional compression options when the compression method is ``'zip'``. (`26023`)
- Bug in `Series.diff` where a boolean series would incorrectly raise a ``TypeError`` (`17294`)
- `Series.append` will no longer raise a ``TypeError`` when passed a tuple of ``Series`` (`28410`)
- Fix corrupted error message when calling ``pandas.libs._json.encode()`` on a 0d array (`18878`)
- Backtick quoting in `DataFrame.query` and `DataFrame.eval` can now also be used to use invalid identifiers like names that start with a digit, are python keywords, or are using single character operators. (`27017`)
- Bug in ``pd.core.util.hashing.hash_pandas_object`` where arrays containing tuples were incorrectly treated as non-hashable (`28969`)
- Bug in `DataFrame.append` that raised ``IndexError`` when appending with empty list (`28769`)
- Fix `AbstractHolidayCalendar` to return correct results for  years after 2030 (now goes up to 2200) (`27790`)
- Fixed `~arrays.IntegerArray` returning ``inf`` rather than ``NaN`` for operations dividing by ``0`` (`27398`)
- Fixed ``pow`` operations for `~arrays.IntegerArray` when the other value is ``0`` or ``1`` (`29997`)
- Bug in `Series.count` raises if use_inf_as_na is enabled (`29478`)
- Bug in `Index` where a non-hashable name could be set without raising ``TypeError`` (`29069`)
- Bug in `DataFrame` constructor when passing a 2D ``ndarray`` and an extension dtype (`12513`)
- Bug in `DataFrame.to_csv` when supplied a series with a ``dtype="string"`` and a ``na_rep``, the ``na_rep`` was being truncated to 2 characters. (`29975`)
- Bug where `DataFrame.itertuples` would incorrectly determine whether or not namedtuples could be used for dataframes of 255 columns (`28282`)
- Handle nested NumPy ``object`` arrays in `testing.assert_series_equal` for ExtensionArray implementations (`30841`)
- Bug in `Index` constructor incorrectly allowing 2-dimensional input arrays (`13601`, `27125`)  .. ---  .. _whatsnew_100.contributors:  Contributors
~~~  .. contributors:: v0.25.3..v1.0.0

What's new in 1.0.1 (February 5, 2020)

These are the changes in pandas 1.0.1. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_101.regressions:  Fixed regressions
~~~  - Fixed regression in `DataFrame` setting values with a slice (e.g. ``df[-4:] = 1``) indexing by label instead of position (`31469`)
- Fixed regression when indexing a ``Series`` or ``DataFrame`` indexed by ``DatetimeIndex`` with a slice containing a `datetime.date` (`31501`)
- Fixed regression in ``DataFrame.__setitem__`` raising an ``AttributeError`` with a `MultiIndex` and a non-monotonic indexer (`31449`)
- Fixed regression in `Series` multiplication when multiplying a numeric `Series` with >1000 elements with a timedelta-like scalar (`31457`)
- Fixed regression in ``.groupby().agg()`` raising an ``AssertionError`` for some reductions like ``min`` on object-dtype columns (`31522`)
- Fixed regression in ``.groupby()`` aggregations with categorical dtype using Cythonized reduction functions (e.g. ``first``) (`31450`)
- Fixed regression in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` if called with a function which returned a non-pandas non-scalar object (e.g. a list or numpy array) (`31441`)
- Fixed regression in `DataFrame.groupby` whereby taking the minimum or maximum of a column with period dtype would raise a ``TypeError``. (`31471`)
- Fixed regression in `DataFrame.groupby` with an empty DataFrame grouping by a level of a MultiIndex (`31670`).
- Fixed regression in `DataFrame.apply` with object dtype and non-reducing function (`31505`)
- Fixed regression in `to_datetime` when parsing non-nanosecond resolution datetimes (`31491`)
- Fixed regression in `~DataFrame.to_csv` where specifying an ``na_rep`` might truncate the values written (`31447`)
- Fixed regression in `Categorical` construction with ``numpy.str_`` categories (`31499`)
- Fixed regression in `DataFrame.loc` and `DataFrame.iloc` when selecting a row containing a single ``datetime64`` or ``timedelta64`` column (`31649`)
- Fixed regression where setting `pd.options.display.max_colwidth` was not accepting negative integer. In addition, this behavior has been deprecated in favor of using ``None`` (`31532`)
- Fixed regression in objTOJSON.c fix return-type warning (`31463`)
- Fixed regression in `qcut` when passed a nullable integer. (`31389`)
- Fixed regression in assigning to a `Series` using a nullable integer dtype (`31446`)
- Fixed performance regression when indexing a ``DataFrame`` or ``Series`` with a `MultiIndex` for the index using a list of labels (`31648`)
- Fixed regression in `read_csv` used in file like object ``RawIOBase`` is not recognize ``encoding`` option (`31575`)  .. ---  .. _whatsnew_101.deprecations:  Deprecations
~~~  - Support for negative integer for `pd.options.display.max_colwidth` is deprecated in favor of using ``None`` (`31532`)  .. ---  .. _whatsnew_101.bug_fixes:  Bug fixes
~~~  **Datetimelike**  - Fixed bug in `to_datetime` raising when ``cache=True`` and out-of-bound values are present (`31491`)  **Numeric**  - Bug in dtypes being lost in ``DataFrame.__invert__`` (``~`` operator) with mixed dtypes (`31183`)  and for extension-array backed ``Series`` and ``DataFrame`` (`23087`)  **Plotting**  - Plotting tz-aware timeseries no longer gives UserWarning (`31205`)  **Interval**  - Bug in `Series.shift` with ``interval`` dtype raising a ``TypeError`` when shifting an interval array of integers or datetimes (`34195`)  .. ---  .. _whatsnew_101.contributors:  Contributors
~~~  .. contributors:: v1.0.0..v1.0.1|HEAD

These are the changes in pandas 1.0.2. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_102.regressions:  Fixed regressions
~~~  **Groupby**  - Fixed regression in `.DataFrameGroupBy.agg` and `.SeriesGroupBy.agg` which were failing on frames with `MultiIndex` columns and a custom function (`31777`)
- Fixed regression in ``groupby(..).rolling(..).apply()`` (``RollingGroupby``) where the ``raw`` parameter was ignored (`31754`)
- Fixed regression in `rolling(..).corr() <pandas.core.window.rolling.Rolling.corr>` when using a time offset (`31789`)
- Fixed regression in `groupby(..).nunique() <pandas.core.groupby.DataFrameGroupBy.nunique>` which was modifying the original values if ``NaN`` values were present (`31950`)
- Fixed regression in ``DataFrame.groupby`` raising a ``ValueError`` from an internal operation (`31802`)
- Fixed regression in `.DataFrameGroupBy.agg` and `.SeriesGroupBy.agg` calling a user-provided function an extra time on an empty input (`31760`)  **I/O**  - Fixed regression in `read_csv` in which the ``encoding`` option was not recognized with certain file-like objects (`31819`)
- Fixed regression in `DataFrame.to_excel` when the ``columns`` keyword argument is passed (`31677`)
- Fixed regression in `ExcelFile` where the stream passed into the function was closed by the destructor. (`31467`)
- Fixed regression where `read_pickle` raised a ``UnicodeDecodeError`` when reading a py27 pickle with `MultiIndex` column (`31988`).  **Reindexing/alignment**  - Fixed regression in `Series.align` when ``other`` is a `DataFrame` and ``method`` is not ``None`` (`31785`)
- Fixed regression in `DataFrame.reindex` and `Series.reindex` when reindexing with (tz-aware) index and ``method=nearest`` (`26683`)
- Fixed regression in `DataFrame.reindex_like` on a `DataFrame` subclass raised an  ``AssertionError`` (`31925`)
- Fixed regression in `DataFrame` arithmetic operations with mis-matched columns (`31623`)  **Other**  - Fixed regression in joining on `DatetimeIndex` or `TimedeltaIndex` to preserve ``freq`` in simple cases (`32166`)
- Fixed regression in `Series.shift` with ``datetime64`` dtype when passing an integer ``fill_value`` (`32591`)
- Fixed regression in the repr of an object-dtype `Index` with bools and missing values (`32146`)  .. ---  Indexing with nullable boolean arrays
~~~  Previously indexing with a nullable Boolean array containing ``NA`` would raise a ``ValueError``, however this is now permitted with ``NA`` being treated as ``False``. (`31503`)  .. ipython:: python  s = pd.Series([1, 2, 3, 4])  mask = pd.array([True, True, False, None], dtype="boolean")  s  mask  *pandas 1.0.0-1.0.1*  .. code-block:: python  >>> s[mask]  Traceback (most recent call last):  ...  ValueError: cannot mask with array containing NA / NaN values  *pandas 1.0.2*  .. ipython:: python  s[mask]  .. _whatsnew_102.bug_fixes:  Bug fixes
~~~  **Datetimelike**  - Bug in `Series.astype` not copying for tz-naive and tz-aware ``datetime64`` dtype (`32490`)
- Bug where `to_datetime` would raise when passed ``pd.NA`` (`32213`)
- Improved error message when subtracting two `Timestamp` that result in an out-of-bounds `Timedelta` (`31774`)  **Categorical**  - Fixed bug where `Categorical.from_codes` improperly raised a ``ValueError`` when passed nullable integer codes. (`31779`)
- Fixed bug where `Categorical` constructor would raise a ``TypeError`` when given a numpy array containing ``pd.NA``. (`31927`)
- Bug in `Categorical` that would ignore or crash when calling `Series.replace` with a list-like ``to_replace`` (`31720`)  **I/O**  - Using ``pd.NA`` with `DataFrame.to_json` now correctly outputs a null value instead of an empty object (`31615`)
- Bug in `pandas.json_normalize` when value in meta path is not iterable (`31507`)
- Fixed pickling of ``pandas.NA``. Previously a new object was returned, which broke computations relying on ``NA`` being a singleton (`31847`)

- Fixed bug in parquet roundtrip with nullable unsigned integer dtypes (`31896`).  **Experimental dtypes**  - Fixed bug in `DataFrame.convert_dtypes` for columns that were already using the ``"string"`` dtype (`31731`).
- Fixed bug in `DataFrame.convert_dtypes` for series with mix of integers and strings (`32117`)
- Fixed bug in `DataFrame.convert_dtypes` where ``BooleanDtype`` columns were converted to ``Int64`` (`32287`)
- Fixed bug in setting values using a slice indexer with string dtype (`31772`)
- Fixed bug where `.DataFrameGroupBy.first`, `.SeriesGroupBy.first`, `.DataFrameGroupBy.last`, and `.SeriesGroupBy.last` would raise a ``TypeError`` when groups contained ``pd.NA`` in a column of object dtype (`32123`)
- Fixed bug where `DataFrameGroupBy.mean`, `DataFrameGroupBy.median`, `DataFrameGroupBy.var`, and `DataFrameGroupBy.std` would raise a ``TypeError`` on ``Int64`` dtype columns (`32219`)  **Strings**  - Using ``pd.NA`` with `Series.str.repeat` now correctly outputs a null value instead of raising error for vector inputs (`31632`)  **Rolling**  - Fixed rolling operations with variable window (defined by time duration) on decreasing time index (`32385`).  .. ---  .. _whatsnew_102.contributors:  Contributors
~~~  .. contributors:: v1.0.1..v1.0.2

What's new in 1.0.3 (March 17, 2020)

These are the changes in pandas 1.0.3. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_103.regressions:  Fixed regressions
~~~
- Fixed regression in ``resample.agg`` when the underlying data is non-writeable (`31710`)
- Fixed regression in `DataFrame` exponentiation with reindexing (`32685`)  .. _whatsnew_103.bug_fixes:  Bug fixes
~~~  Contributors
~~~  .. contributors:: v1.0.2..v1.0.3

What's new in 1.0.4 (May 28, 2020)

These are the changes in pandas 1.0.4. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_104.regressions:  Fixed regressions
~~~
- Fix regression where `Series.isna` and `DataFrame.isna` would raise for categorical dtype when ``pandas.options.mode.use_inf_as_na`` was set to ``True`` (`33594`)
- Fix regression in `.DataFrameGroupBy.first`, `.SeriesGroupBy.first`, `.DataFrameGroupBy.last`, and `.SeriesGroupBy.last` where None is not preserved in object dtype (`32800`)
- Fix regression in DataFrame reductions using ``numeric_only=True`` and ExtensionArrays (`33256`).
- Fix performance regression in ``memory_usage(deep=True)`` for object dtype (`33012`)
- Fix regression where `Categorical.replace` would replace with ``NaN`` whenever the new value and replacement value were equal (`33288`)
- Fix regression where an ordered `Categorical` containing only ``NaN`` values would raise rather than returning ``NaN`` when taking the minimum or maximum  (`33450`)
- Fix regression in `DataFrameGroupBy.agg` with dictionary input losing ``ExtensionArray`` dtypes (`32194`)
- Fix to preserve the ability to index with the "nearest" method with xarray's CFTimeIndex, an Index` subclass (`pydata/xarray#3751, `32905`).
- Fix regression in `DataFrame.describe` raising ``TypeError: unhashable type: 'dict'`` (`32409`)
- Fix regression in `DataFrame.replace` casts columns to ``object`` dtype if items in ``to_replace`` not in values (`32988`)
- Fix regression in `Series.groupby` would raise ``ValueError`` when grouping by `PeriodIndex` level (`34010`)
- Fix regression in `DataFrameGroupBy.rolling.apply` and `SeriesGroupBy.rolling.apply` ignoring args and kwargs parameters (`33433`)
- Fix regression in error message with ``np.min`` or ``np.max`` on unordered `Categorical` (`33115`)
- Fix regression in `DataFrame.loc` and `Series.loc` throwing an error when a ``datetime64[ns, tz]`` value is provided (`32395`)  .. _whatsnew_104.bug_fixes:  Bug fixes
~~~
- Bug in `SeriesGroupBy.first`, `SeriesGroupBy.last`, `SeriesGroupBy.min`, and `SeriesGroupBy.max` returning floats when applied to nullable Booleans (`33071`)
- Bug in `Rolling.min` and `Rolling.max`: Growing memory usage after multiple calls when using a fixed window (`30726`)
- Bug in `~DataFrame.to_parquet` was not raising ``PermissionError`` when writing to a private s3 bucket with invalid creds. (`27679`)
- Bug in `~DataFrame.to_csv` was silently failing when writing to an invalid s3 bucket. (`32486`)
- Bug in `read_parquet` was raising a ``FileNotFoundError`` when passed an s3 directory path. (`26388`)
- Bug in `~DataFrame.to_parquet` was throwing an ``AttributeError`` when writing a partitioned parquet file to s3 (`27596`)
- Bug in `.DataFrameGroupBy.quantile` and `.SeriesGroupBy.quantile` causes the quantiles to be shifted when the ``by`` axis contains ``NaN`` (`33200`, `33569`)  Contributors
~~~  .. contributors:: v1.0.3..v1.0.4

What's new in 1.0.5 (June 17, 2020)

These are the changes in pandas 1.0.5. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_105.regressions:  Fixed regressions
~~~  - Fix regression in `read_parquet` when reading from file-like objects  (`34467`).
- Fix regression in reading from public S3 buckets (`34626`).  Note this disables the ability to read Parquet files from directories on S3
again (`26388`, `34632`), which was added in the 1.0.4 release,
but is now targeted for pandas 1.1.0.  - Fixed regression in `~DataFrame.replace` raising an ``AssertionError`` when replacing values in an extension dtype with values of a different dtype (`34530`)  .. _whatsnew_105.bug_fixes:  Bug fixes
~~~  - Fixed building from source with Python 3.8 fetching the wrong version of NumPy (`34666`)  Contributors
~~~  .. contributors:: v1.0.4..v1.0.5|HEAD

These are the changes in pandas 1.1.0. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  Enhancements
~~~  .. _whatsnew_110.specify_missing_labels:  KeyErrors raised by loc specify missing labels
^^^
Previously, if labels were missing for a ``.loc`` call, a KeyError was raised stating that this was no longer supported.  Now the error message also includes a list of the missing labels (max 10 items, display width 80 characters). See `34272`.  .. _whatsnew_110.astype_string:  All dtypes can now be converted to ``StringDtype``
^^^  Previously, declaring or converting to `StringDtype` was in general only possible if the data was already only ``str`` or nan-like (`31204`).
`StringDtype` now works in all situations where ``astype(str)`` or ``dtype=str`` work:  For example, the below now works:  .. ipython:: python  ser = pd.Series([1, "abc", np.nan], dtype="string")  ser  ser[0]  pd.Series([1, 2, np.nan], dtype="Int64").astype("string")  .. _whatsnew_110.period_index_partial_string_slicing:  Non-monotonic PeriodIndex partial string slicing
^^^  `PeriodIndex` now supports partial string slicing for non-monotonic indexes, mirroring `DatetimeIndex` behavior (`31096`)  For example:  .. ipython:: python  dti = pd.date_range("2014-01-01", periods=30, freq="30D")  pi = dti.to_period("D")  ser_monotonic = pd.Series(np.arange(30), index=pi)  shuffler = list(range(0, 30, 2)) + list(range(1, 31, 2))  ser = ser_monotonic[shuffler]  ser  .. ipython:: python  ser["2014"]  ser.loc["May 2015"]  .. _whatsnew_110.dataframe_or_series_comparing:  Comparing two ``DataFrame`` or two ``Series`` and summarizing the differences
^^^  We've added `DataFrame.compare` and `Series.compare` for comparing two ``DataFrame`` or two ``Series`` (`30429`)  .. ipython:: python  df = pd.DataFrame(  {  "col1": ["a", "a", "b", "b", "a"],  "col2": [1.0, 2.0, 3.0, np.nan, 5.0],  "col3": [1.0, 2.0, 3.0, 4.0, 5.0]  },  columns=["col1", "col2", "col3"],  )  df  .. ipython:: python  df2 = df.copy()  df2.loc[0, 'col1'] = 'c'  df2.loc[2, 'col3'] = 4.0  df2  .. ipython:: python  df.compare(df2)  See `User Guide <merging.compare>` for more details.  .. _whatsnew_110.groupby_key:  Allow NA in groupby key
^^^  With `groupby <groupby.dropna>` , we've added a ``dropna`` keyword to `DataFrame.groupby` and `Series.groupby` in order to
allow ``NA`` values in group keys. Users can define ``dropna`` to ``False`` if they want to include
``NA`` values in groupby keys. The default is set to ``True`` for ``dropna`` to keep backwards
compatibility (`3729`)  .. ipython:: python  df_list = [[1, 2, 3], [1, None, 4], [2, 1, 3], [1, 2, 2]]  df_dropna = pd.DataFrame(df_list, columns=["a", "b", "c"])  df_dropna  .. ipython:: python  # Default ``dropna`` is set to True, which will exclude NaNs in keys  df_dropna.groupby(by=["b"], dropna=True).sum()  # In order to allow NaN in keys, set ``dropna`` to False  df_dropna.groupby(by=["b"], dropna=False).sum()  The default setting of ``dropna`` argument is ``True`` which means ``NA`` are not included in group keys.  .. _whatsnew_110.key_sorting:  Sorting with keys
^^^  We've added a ``key`` argument to the `DataFrame` and `Series` sorting methods, including
`DataFrame.sort_values`, `DataFrame.sort_index`, `Series.sort_values`,
and `Series.sort_index`. The ``key`` can be any callable function which is applied
column-by-column to each column used for sorting, before sorting is performed (`27237`).
See `sort_values with keys <basics.sort_value_key>` and `sort_index with keys
<basics.sort_index_key>` for more information.  .. ipython:: python  s = pd.Series(['C', 'a', 'B'])  s  .. ipython:: python  s.sort_values()  Note how this is sorted with capital letters first. If we apply the `Series.str.lower`
method, we get  .. ipython:: python  s.sort_values(key=lambda x: x.str.lower())  When applied to a ``DataFrame``, they key is applied per-column to all columns or a subset if

``by`` is specified, e.g.  .. ipython:: python  df = pd.DataFrame({'a': ['C', 'C', 'a', 'a', 'B', 'B'],  'b': [1, 2, 3, 4, 5, 6]})  df  .. ipython:: python  df.sort_values(by=['a'], key=lambda col: col.str.lower())  For more details, see examples and documentation in `DataFrame.sort_values`,
`Series.sort_values`, and `~DataFrame.sort_index`.  .. _whatsnew_110.timestamp_fold_support:  Fold argument support in Timestamp constructor
^^^  Timestamp:` now supports the keyword-only fold argument according to `PEP 495 similar to parent ``datetime.datetime`` class. It supports both accepting fold as an initialization argument and inferring fold from other constructor arguments (`25057`, `31338`). Support is limited to ``dateutil`` timezones as ``pytz`` doesn't support fold.  For example:  .. ipython:: python  ts = pd.Timestamp("2019-10-27 0100+00:00")  ts.fold  .. ipython:: python  ts = pd.Timestamp(year=2019, month=10, day=27, hour=1, minute=30,  tz="dateutil/Europe/London", fold=1)  ts  For more on working with fold, see `Fold subsection <timeseries.fold>` in the user guide.  .. _whatsnew_110.to_datetime_multiple_tzname_tzoffset_support:  Parsing timezone-aware format with different timezones in to_datetime
^^^  `to_datetime` now supports parsing formats containing timezone names (``%Z``) and UTC offsets (``%z``) from different timezones then converting them to UTC by setting ``utc=True``. This would return a `DatetimeIndex` with timezone at UTC as opposed to an `Index` with ``object`` dtype if ``utc=True`` is not set (`32792`).  For example:  .. ipython:: python  tz_strs = ["2010-01-01 1200 +0100", "2010-01-01 1200 -0100",  "2010-01-01 1200 +0300", "2010-01-01 1200 +0400"]  pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z', utc=True)  pd.to_datetime(tz_strs, format='%Y-%m-%d %H:%M:%S %z')  .. _whatsnew_110.grouper_resample_origin:  Grouper and resample now supports the arguments origin and offset
^^^  `Grouper` and `DataFrame.resample` now supports the arguments ``origin`` and ``offset``. It let the user control the timestamp on which to adjust the grouping. (`31809`)  The bins of the grouping are adjusted based on the beginning of the day of the time series starting point. This works well with frequencies that are multiples of a day (like ``30D``) or that divides a day (like ``90s`` or ``1min``). But it can create inconsistencies with some frequencies that do not meet this criteria. To change this behavior you can now specify a fixed timestamp with the argument ``origin``.  Two arguments are now deprecated (more information in the documentation of `DataFrame.resample`):  - ``base`` should be replaced by ``offset``.
- ``loffset`` should be replaced by directly adding an offset to the index `DataFrame` after being resampled.  Small example of the use of ``origin``:  .. ipython:: python  start, end = '2000-10-01 2300', '2000-10-02 000'  middle = '2000-10-02 000'  rng = pd.date_range(start, end, freq='7min')  ts = pd.Series(np.arange(len(rng)) * 3, index=rng)  ts  Resample with the default behavior ``'start_day'`` (origin is ``2000-10-01 000``):  .. ipython:: python  ts.resample('17min').sum()  ts.resample('17min', origin='start_day').sum()  Resample using a fixed origin:  .. ipython:: python  ts.resample('17min', origin='epoch').sum()  ts.resample('17min', origin='2000-01-01').sum()  If needed you can adjust the bins with the argument ``offset`` (a `Timedelta`) that would be added to the default ``origin``.  For a full example, see: `timeseries.adjust-the-start-of-the-bins`.  fsspec now used for filesystem handling
^^^  For reading and writing to filesystems other than local and reading from HTTP(S),
the optional dependency ``fsspec`` will be used to dispatch operations (`33452`).
This will give unchanged

functionality for S3 and GCS storage, which were already supported, but also add
support for several other storage implementations such as `Azure Datalake and Blob`_,
SSH, FTP, dropbox and github. For docs and capabilities, see the `fsspec docs`_.  The existing capability to interface with S3 and GCS will be unaffected by this
change, as ``fsspec`` will still bring in the same packages as before.  .. _Azure Datalake and Blob: https://github.com/fsspec/adlfs  .. _fsspec docs: https://filesystem-spec.readthedocs.io/en/latest/  .. _whatsnew_110.enhancements.other:  Other enhancements
^^^  - Compatibility with matplotlib 3.3.0 (`34850`)
- `IntegerArray.astype` now supports ``datetime64`` dtype (`32538`)
- `IntegerArray` now implements the ``sum`` operation (`33172`)
- Added `pandas.errors.InvalidIndexError` (`34570`).
- Added `DataFrame.value_counts` (`5377`)
- Added a `pandas.api.indexers.FixedForwardWindowIndexer` class to support forward-looking windows during ``rolling`` operations.
- Added a `pandas.api.indexers.VariableOffsetWindowIndexer` class to support ``rolling`` operations with non-fixed offsets (`34994`)
- `~DataFrame.describe` now includes a ``datetime_is_numeric`` keyword to control how datetime columns are summarized (`30164`, `34798`)
- `~pandas.io.formats.style.Styler` may now render CSS more efficiently where multiple cells have the same styling (`30876`)
- `~pandas.io.formats.style.Styler.highlight_null` now accepts ``subset`` argument (`31345`)
- When writing directly to a sqlite connection `DataFrame.to_sql` now supports the ``multi`` method (`29921`)
- `pandas.errors.OptionError` is now exposed in ``pandas.errors`` (`27553`)
- Added `api.extensions.ExtensionArray.argmax` and `api.extensions.ExtensionArray.argmin` (`24382`)
- `timedelta_range` will now infer a frequency when passed ``start``, ``stop``, and ``periods`` (`32377`)
- Positional slicing on a `IntervalIndex` now supports slices with ``step > 1`` (`31658`)
- `Series.str` now has a ``fullmatch`` method that matches a regular expression against the entire string in each row of the `Series`, similar to ``re.fullmatch`` (`32806`).
- `DataFrame.sample` will now also allow array-like and BitGenerator objects to be passed to ``random_state`` as seeds (`32503`)
- `Index.union` will now raise ``RuntimeWarning`` for `MultiIndex` objects if the object inside are unsortable. Pass ``sort=False`` to suppress this warning (`33015`)
- Added `Series.dt.isocalendar` and `DatetimeIndex.isocalendar` that returns a `DataFrame` with year, week, and day calculated according to the ISO 8601 calendar (`33206`, `34392`).
- The `DataFrame.to_feather` method now supports additional keyword  arguments (e.g. to set the compression) that are added in pyarrow 0.17  (`33422`).
- The `cut` will now accept parameter ``ordered`` with default ``ordered=True``. If ``ordered=False`` and no labels are provided, an error will be raised (`33141`)
- `DataFrame.to_csv`, `DataFrame.to_pickle`,  and `DataFrame.to_json` now support passing a dict of  compression arguments when using the ``gzip`` and ``bz2`` protocols.  This can be used to set a custom compression level, e.g.,  ``df.to_csv(path, compression={'method': 'gzip', 'compresslevel': 1}``  (`33196`)
- `melt` has gained an ``ignore_index`` (default ``True``) argument that, if set to ``False``, prevents the method from dropping the index (`17440`).
- `Series.update` now accepts objects that can be coerced to a `Series`,  such as ``dict`` and ``list``, mirroring the behavior of `DataFrame.update` (`33215`)
- `~pandas.core.groupby.DataFrameGroupBy.transform` and `~pandas.core.groupby.DataFrameGroupBy.aggregate` have gained ``engine`` and ``engine_kwargs`` arguments that support executing functions with ``Numba`` (`32854`, `33388`)

- `~pandas.core.resample.Resampler.interpolate` now supports SciPy interpolation method `scipy.interpolate.CubicSpline` as method ``cubicspline`` (`33670`)
- `~pandas.core.groupby.DataFrameGroupBy` and `~pandas.core.groupby.SeriesGroupBy` now implement the ``sample`` method for doing random sampling within groups (`31775`)
- `DataFrame.to_numpy` now supports the ``na_value`` keyword to control the NA sentinel in the output array (`33820`)
- Added `api.extension.ExtensionArray.equals` to the extension array interface, similar to `Series.equals` (`27081`)
- The minimum supported dta version has increased to 105 in `read_stata` and `~pandas.io.stata.StataReader`  (`26667`).
- `~DataFrame.to_stata` supports compression using the ``compression``  keyword argument. Compression can either be inferred or explicitly set using a string or a  dictionary containing both the method and any additional arguments that are passed to the  compression library. Compression was also added to the low-level Stata-file writers  `~pandas.io.stata.StataWriter`, `~pandas.io.stata.StataWriter117`,  and `~pandas.io.stata.StataWriterUTF8` (`26599`).
- `HDFStore.put` now accepts a ``track_times`` parameter. This parameter is passed to the ``create_table`` method of ``PyTables`` (`32682`).
- `Series.plot` and `DataFrame.plot` now accepts ``xlabel`` and ``ylabel`` parameters to present labels on x and y axis (`9093`).
- Made `pandas.core.window.rolling.Rolling` and `pandas.core.window.expanding.Expanding` iterable（`11704`)
- Made ``option_context`` a `contextlib.ContextDecorator`, which allows it to be used as a decorator over an entire function (`34253`).
- `DataFrame.to_csv` and `Series.to_csv` now accept an ``errors`` argument (`22610`)
- `~pandas.core.groupby.DataFrameGroupBy.groupby.transform` now allows ``func`` to be ``pad``, ``backfill`` and ``cumcount`` (`31269`).
- `read_json` now accepts an ``nrows`` parameter. (`33916`).
- `DataFrame.hist`, `Series.hist`, `core.groupby.DataFrameGroupBy.hist`, and `core.groupby.SeriesGroupBy.hist` have gained the ``legend`` argument. Set to True to show a legend in the histogram. (`6279`)
- `concat` and `~DataFrame.append` now preserve extension dtypes, for example  combining a nullable integer column with a numpy integer column will no longer  result in object dtype but preserve the integer dtype (`33607`, `34339`, `34095`).
- `read_gbq` now allows to disable progress bar (`33360`).
- `read_gbq` now supports the ``max_results`` kwarg from ``pandas-gbq`` (`34639`).
- `DataFrame.cov` and `Series.cov` now support a new parameter ``ddof`` to support delta degrees of freedom as in the corresponding numpy methods (`34611`).
- `DataFrame.to_html` and `DataFrame.to_string`'s ``col_space`` parameter now accepts a list or dict to change only some specific columns' width (`28917`).
- `DataFrame.to_excel` can now also write OpenOffice spreadsheet (.ods) files (`27222`)
- `~Series.explode` now accepts ``ignore_index`` to reset the index, similar to `pd.concat` or `DataFrame.sort_values` (`34932`).
- `DataFrame.to_markdown` and `Series.to_markdown` now accept ``index`` argument as an alias for tabulate's ``showindex`` (`32667`)
- `read_csv` now accepts string values like "0", "0.0", "1", "1.0" as convertible to the nullable Boolean dtype (`34859`)
- `pandas.core.window.ExponentialMovingWindow` now supports a ``times`` argument that allows ``mean`` to be calculated with observations spaced by the timestamps in ``times`` (`34839`)
- `DataFrame.agg` and `Series.agg` now accept named aggregation for renaming the output columns/indexes. (`26513`)
- ``compute.use_numba`` now exists as a configuration option that utilizes the numba engine when available (`33966`, `35374`)

- `Series.plot` now supports asymmetric error bars. Previously, if `Series.plot` received a "2xN" array with error values for ``yerr`` and/or ``xerr``, the left/lower values (first row) were mirrored, while the right/upper values (second row) were ignored. Now, the first row represents the left/lower error values and the second row the right/upper error values. (`9536`)  .. ---  .. _whatsnew_110.notable_bug_fixes:  Notable bug fixes
~~~  These are bug fixes that might have notable behavior changes.  ``MultiIndex.get_indexer`` interprets ``method`` argument correctly
^^^  This restores the behavior of `MultiIndex.get_indexer` with ``method='backfill'`` or ``method='pad'`` to the behavior before pandas 0.23.0. In particular, MultiIndexes are treated as a list of tuples and padding or backfilling is done with respect to the ordering of these lists of tuples (`29896`).  As an example of this, given:  .. ipython:: python  df = pd.DataFrame({  'a': [0, 0, 0, 0],  'b': [0, 2, 3, 4],  'c': ['A', 'B', 'C', 'D'],  }).set_index(['a', 'b'])  mi_2 = pd.MultiIndex.from_product([[0], [-1, 0, 1, 3, 4, 5]])  The differences in reindexing ``df`` with ``mi_2`` and using ``method='backfill'`` can be seen here:  *pandas >= 0.23, < 1.1.0*:  .. code-block:: ipython  In [1]: df.reindex(mi_2, method='backfill')  Out[1]:  c  0 -1  A  0  A  1  D  3  A  4  A  5  C  *pandas <0.23, >= 1.1.0*  .. ipython:: python  df.reindex(mi_2, method='backfill')  And the differences in reindexing ``df`` with ``mi_2`` and using ``method='pad'`` can be seen here:  *pandas >= 0.23, < 1.1.0*  .. code-block:: ipython  In [1]: df.reindex(mi_2, method='pad')  Out[1]:  c  0 -1  NaN  0  NaN  1  D  3  NaN  4  A  5  C  *pandas < 0.23, >= 1.1.0*  .. ipython:: python  df.reindex(mi_2, method='pad')  .. _whatsnew_110.notable_bug_fixes.indexing_raises_key_errors:  Failed label-based lookups always raise KeyError
^^^  Label lookups ``series[key]``, ``series.loc[key]`` and ``frame.loc[key]``
used to raise either ``KeyError`` or ``TypeError`` depending on the type of
key and type of `Index`.  These now consistently raise ``KeyError`` (`31867`)  .. ipython:: python  ser1 = pd.Series(range(3), index=[0, 1, 2])  ser2 = pd.Series(range(3), index=pd.date_range("2020-02-01", periods=3))  *Previous behavior*:  .. code-block:: ipython  In [3]: ser1[1.5]  ...  TypeError: cannot do label indexing on Int64Index with these indexers [1.5] of type float  In [4] ser1["foo"]  ...  KeyError: 'foo'  In [5]: ser1.loc[1.5]  ...  TypeError: cannot do label indexing on Int64Index with these indexers [1.5] of type float  In [6]: ser1.loc["foo"]  ...  KeyError: 'foo'  In [7]: ser2.loc[1]  ...  TypeError: cannot do label indexing on DatetimeIndex with these indexers [1] of type int  In [8]: ser2.loc[pd.Timestamp(0)]  ...  KeyError: Timestamp('1970-01-01 000')  *New behavior*:  .. code-block:: ipython  In [3]: ser1[1.5]  ...  KeyError: 1.5  In [4] ser1["foo"]  ...  KeyError: 'foo'  In [5]: ser1.loc[1.5]  ...  KeyError: 1.5  In [6]: ser1.loc["foo"]  ...  KeyError: 'foo'  In [7]: ser2.loc[1]  ...  KeyError: 1  In [8]: ser2.loc[pd.Timestamp(0)]  ...  KeyError: Timestamp('1970-01-01 000')  Similarly, `DataFrame.at` and `Series.at` will raise a ``TypeError`` instead of a ``ValueError`` if an incompatible key is passed, and ``KeyError`` if a missing key is passed, matching the behavior of ``.loc[]`` (`31722`)  .. _whatsnew_110.notable_bug_fixes.indexing_int_multiindex_raises_key_errors:  Failed Integer Lookups on MultiIndex Raise KeyError
^^^
Indexing with integers with a `MultiIndex` that has an integer-dtype
first level incorrectly failed to raise ``KeyError`` when one or more of

^^^
those integer keys is not present in the first level of the index (`33539`)  .. ipython:: python  idx = pd.Index(range(4))  dti = pd.date_range("2000-01-03", periods=3)  mi = pd.MultiIndex.from_product([idx, dti])  ser = pd.Series(range(len(mi)), index=mi)  *Previous behavior*:  .. code-block:: ipython  In [5]: ser[[5]]  Out[5]: Series([], dtype: int64)  *New behavior*:  .. code-block:: ipython  In [5]: ser[[5]]  ...  KeyError: '[5] not in index'  `DataFrame.merge` preserves right frame's row order
^^^
`DataFrame.merge` now preserves the right frame's row order when executing a right merge (`27453`)  .. ipython:: python  left_df = pd.DataFrame({'animal': ['dog', 'pig'],  'max_speed': [40, 11]})  right_df = pd.DataFrame({'animal': ['quetzal', 'pig'],  'max_speed': [80, 11]})  left_df  right_df  *Previous behavior*:  .. code-block:: python  >>> left_df.merge(right_df, on=['animal', 'max_speed'], how="right")  animal  max_speed  0  pig  11  1  quetzal  80  *New behavior*:  .. ipython:: python  left_df.merge(right_df, on=['animal', 'max_speed'], how="right")  .. ---  .. _whatsnew_110.notable_bug_fixes.assignment_to_multiple_columns:  Assignment to multiple columns of a DataFrame when some columns do not exist
^^^  Assignment to multiple columns of a `DataFrame` when some of the columns do not exist would previously assign the values to the last column. Now, new columns will be constructed with the right values. (`13658`)  .. ipython:: python  df = pd.DataFrame({'a': [0, 1, 2], 'b': [3, 4, 5]})  df  *Previous behavior*:  .. code-block:: ipython  In [3]: df[['a', 'c']] = 1  In [4]: df  Out[4]:  a  b  0  1  1  1  1  1  2  1  1  *New behavior*:  .. ipython:: python  df[['a', 'c']] = 1  df  .. _whatsnew_110.notable_bug_fixes.groupby_consistency:  Consistency across groupby reductions
^^^  Using `DataFrame.groupby` with ``as_index=True`` and the aggregation ``nunique`` would include the grouping column(s) in the columns of the result. Now the grouping column(s) only appear in the index, consistent with other reductions. (`32579`)  .. ipython:: python  df = pd.DataFrame({"a": ["x", "x", "y", "y"], "b": [1, 1, 2, 3]})  df  *Previous behavior*:  .. code-block:: ipython  In [3]: df.groupby("a", as_index=True).nunique()  Out[4]:  a  b  a  x  1  1  y  1  2  *New behavior*:  .. ipython:: python  df.groupby("a", as_index=True).nunique()  Using `DataFrame.groupby` with ``as_index=False`` and the function ``idxmax``, ``idxmin``, ``mad``, ``nunique``, ``sem``, ``skew``, or ``std`` would modify the grouping column. Now the grouping column remains unchanged, consistent with other reductions. (`21090`, `10355`)  *Previous behavior*:  .. code-block:: ipython  In [3]: df.groupby("a", as_index=False).nunique()  Out[4]:  a  b  0  1  1  1  1  2  *New behavior*:  .. ipython:: python  df.groupby("a", as_index=False).nunique()  The method `~pandas.core.groupby.DataFrameGroupBy.size` would previously ignore ``as_index=False``. Now the grouping columns are returned as columns, making the result a `DataFrame` instead of a `Series`. (`32599`)  *Previous behavior*:  .. code-block:: ipython  In [3]: df.groupby("a", as_index=False).size()  Out[4]:  a  x  2  y  2  dtype: int64  *New behavior*:  .. ipython:: python  df.groupby("a", as_index=False).size()  .. _whatsnew_110.api_breaking.groupby_results_lost_as_index_false:  `~pandas.core.groupby.DataFrameGroupby.agg` lost results with ``as_index=False`` when relabeling columns
^^^  Previously `~pandas.core.groupby.DataFrameGroupby.agg` lost the result columns, when the ``as_index`` option was
set to ``False`` and the result columns were relabeled. In this case the result values were replaced with

the previous index (`32240`).  .. ipython:: python  df = pd.DataFrame({"key": ["x", "y", "z", "x", "y", "z"],  "val": [1.0, 0.8, 2.0, 3.0, 3.6, 0.75]})  df  *Previous behavior*:  .. code-block:: ipython  In [2]: grouped = df.groupby("key", as_index=False)  In [3]: result = grouped.agg(min_val=pd.NamedAgg(column="val", aggfunc="min"))  In [4]: result  Out[4]:  min_val  0  x  1  y  2  z  *New behavior*:  .. ipython:: python  grouped = df.groupby("key", as_index=False)  result = grouped.agg(min_val=pd.NamedAgg(column="val", aggfunc="min"))  result  .. _whatsnew_110.notable_bug_fixes.apply_applymap_first_once:  apply and applymap on ``DataFrame`` evaluates first row/column only once
^^^  .. ipython:: python  df = pd.DataFrame({'a': [1, 2], 'b': [3, 6]})  def func(row):  print(row)  return row  *Previous behavior*:  .. code-block:: ipython  In [4]: df.apply(func, axis=1)  a  1  b  3  Name: 0, dtype: int64  a  1  b  3  Name: 0, dtype: int64  a  2  b  6  Name: 1, dtype: int64  Out[4]:  a  b  0  1  3  1  2  6  *New behavior*:  .. ipython:: python  df.apply(func, axis=1)  .. _whatsnew_110.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_110.api_breaking.testing.check_freq:  Added ``check_freq`` argument to ``testing.assert_frame_equal`` and ``testing.assert_series_equal``
^^^  The ``check_freq`` argument was added to `testing.assert_frame_equal` and `testing.assert_series_equal` in pandas 1.1.0 and defaults to ``True``. `testing.assert_frame_equal` and `testing.assert_series_equal` now raise ``AssertionError`` if the indexes do not have the same frequency. Before pandas 1.1.0, the index frequency was not checked.  Increased minimum versions for dependencies
^^^  Some minimum supported versions of dependencies were updated (`33718`, `29766`, `29723`, pytables >= 3.4.3).
| numpy  | 1.15.4  |  X  |  X  |
| python-dateutil | 2.7.3  |  X  |  X  |
| fastparquet  | 0.3.2  |  |
| fsspec  | 0.7.4  |  |
| gcsfs  | 0.6.0  |  X  |
| numba  | 0.46.0  |  |
| openpyxl  | 2.5.7  |  |
| pyarrow  | 0.13.0  |  |
| pytables  | 3.4.3  |  X  |
| s3fs  | 0.4.0  |  X  |
| scipy  | 1.2.0  |  X  |
| pandas-gbq  | 1.2.0  |  X  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  Development changes
^^^  - The minimum version of Cython is now the most recent bug-fix version (0.29.16) (`3334`).  .. _whatsnew_110.deprecations:  Deprecations

~~~  - Lookups on a `Series` with a single-item list containing a slice (e.g. ``ser[[slice(0, 4)]]``) are deprecated and will raise in a future version.  Either convert the list to a tuple, or pass the slice directly instead (`31333`)  - `DataFrame.mean` and `DataFrame.median` with ``numeric_only=None`` will include ``datetime64`` and ``datetime64tz`` columns in a future version (`29941`)
- Setting values with ``.loc`` using a positional slice is deprecated and will raise in a future version.  Use ``.loc`` with labels or ``.iloc`` with positions instead (`31840`)
- `DataFrame.to_dict` has deprecated accepting short names for ``orient`` and will raise in a future version (`32515`)
- `Categorical.to_dense` is deprecated and will be removed in a future version, use ``np.asarray(cat)`` instead (`32639`)
- The ``fastpath`` keyword in the ``SingleBlockManager`` constructor is deprecated and will be removed in a future version (`33092`)
- Providing ``suffixes`` as a ``set`` in `pandas.merge` is deprecated. Provide a tuple instead (`33740`, `34741`).
- Indexing a `Series` with a multi-dimensional indexer like ``[:, None]`` to return an ``ndarray`` now raises a ``FutureWarning``. Convert to a NumPy array before indexing instead (`27837`)
- `Index.is_mixed` is deprecated and will be removed in a future version, check ``index.inferred_type`` directly instead (`32922`)  - Passing any arguments but the first one to `read_html` as  positional arguments is deprecated. All other  arguments should be given as keyword arguments (`27573`).  - Passing any arguments but ``path_or_buf`` (the first one) to  `read_json` as positional arguments is deprecated. All  other arguments should be given as keyword arguments (`27573`).  - Passing any arguments but the first two to `read_excel` as  positional arguments is deprecated. All other  arguments should be given as keyword arguments (`27573`).  - `pandas.api.types.is_categorical` is deprecated and will be removed in a future version; use `pandas.api.types.is_categorical_dtype` instead (`33385`)
- `Index.get_value` is deprecated and will be removed in a future version (`19728`)
- `Series.dt.week` and `Series.dt.weekofyear` are deprecated and will be removed in a future version, use `Series.dt.isocalendar().week` instead (`33595`)
- `DatetimeIndex.week` and ``DatetimeIndex.weekofyear`` are deprecated and will be removed in a future version, use ``DatetimeIndex.isocalendar().week`` instead (`33595`)
- `DatetimeArray.week` and ``DatetimeArray.weekofyear`` are deprecated and will be removed in a future version, use ``DatetimeArray.isocalendar().week`` instead (`33595`)
- `DateOffset.__call__` is deprecated and will be removed in a future version, use ``offset + other`` instead (`34171`)
- `~pandas.tseries.offsets.BusinessDay.apply_index` is deprecated and will be removed in a future version. Use ``offset + other`` instead (`34580`)
- `DataFrame.tshift` and `Series.tshift` are deprecated and will be removed in a future version, use `DataFrame.shift` and `Series.shift` instead (`11631`)
- Indexing an `Index` object with a float key is deprecated, and will  raise an ``IndexError`` in the future. You can manually convert to an integer key  instead (`34191`).
- The ``squeeze`` keyword in `~DataFrame.groupby` is deprecated and will be removed in a future version (`32380`)
- The ``tz`` keyword in `Period.to_timestamp` is deprecated and will be removed in a future version; use ``per.to_timestamp(...).tz_localize(tz)`` instead (`34522`)
- `DatetimeIndex.to_perioddelta` is deprecated and will be removed in a future version.  Use ``index - index.to_period(freq).to_timestamp()`` instead (`34853`)
- `DataFrame.melt` accepting a ``value_name`` that already exists is deprecated, and will be removed in a future version (`34731`)

- The ``center`` keyword in the `DataFrame.expanding` function is deprecated and will be removed in a future version (`20647`)  .. ---  .. _whatsnew_110.performance:  Performance improvements
~~~  - Performance improvement in `Timedelta` constructor (`30543`)
- Performance improvement in `Timestamp` constructor (`30543`)
- Performance improvement in flex arithmetic ops between `DataFrame` and `Series` with ``axis=0`` (`31296`)
- Performance improvement in arithmetic ops between `DataFrame` and `Series` with ``axis=1`` (`33600`)
- The internal index method `~Index._shallow_copy` now copies cached attributes over to the new index,  avoiding creating these again on the new index. This can speed up many operations that depend on creating copies of  existing indexes (`28584`, `32640`, `32669`)
- Significant performance improvement when creating a `DataFrame` with  sparse values from ``scipy.sparse`` matrices using the  `DataFrame.sparse.from_spmatrix` constructor (`32821`,  `32825`,  `32826`, `32856`, `32858`).
- Performance improvement for groupby methods `~pandas.core.groupby.groupby.Groupby.first`  and `~pandas.core.groupby.groupby.Groupby.last` (`34178`)
- Performance improvement in `factorize` for nullable (integer and Boolean) dtypes (`33064`).
- Performance improvement when constructing `Categorical` objects (`33921`)
- Fixed performance regression in `pandas.qcut` and `pandas.cut` (`33921`)
- Performance improvement in reductions (``sum``, ``prod``, ``min``, ``max``) for nullable (integer and Boolean) dtypes (`30982`, `33261`, `33442`).
- Performance improvement in arithmetic operations between two `DataFrame` objects (`32779`)
- Performance improvement in `pandas.core.groupby.RollingGroupby` (`34052`)
- Performance improvement in arithmetic operations (``sub``, ``add``, ``mul``, ``div``) for `MultiIndex` (`34297`)
- Performance improvement in ``DataFrame[bool_indexer]`` when ``bool_indexer`` is a ``list`` (`33924`)
- Significant performance improvement of `io.formats.style.Styler.render` with styles added with various ways such as `io.formats.style.Styler.apply`, `io.formats.style.Styler.applymap` or `io.formats.style.Styler.bar` (`19917`)  .. ---  .. _whatsnew_110.bug_fixes:  Bug fixes
^^^  - Passing an invalid ``fill_value`` to `Categorical.take` raises a ``ValueError`` instead of ``TypeError`` (`33660`)
- Combining a `Categorical` with integer categories and which contains missing values with a float dtype column in operations such as `concat` or `~DataFrame.append` will now result in a float column instead of an object dtype column (`33607`)
- Bug where `merge` was unable to join on non-unique categorical indices (`28189`)
- Bug when passing categorical data to `Index` constructor along with ``dtype=object`` incorrectly returning a `CategoricalIndex` instead of object-dtype `Index` (`32167`)
- Bug where `Categorical` comparison operator ``__ne__`` would incorrectly evaluate to ``False`` when either element was missing (`32276`)
- `Categorical.fillna` now accepts `Categorical` ``other`` argument (`32420`)
- Repr of `Categorical` was not distinguishing between ``int`` and ``str`` (`33676`)  Datetimelike
^^^  - Passing an integer dtype other than ``int64`` to ``np.array(period_index, dtype=...)`` will now raise ``TypeError`` instead of incorrectly using ``int64`` (`32255`)
- `Series.to_timestamp` now raises a ``TypeError`` if the axis is not a `PeriodIndex`. Previously an ``AttributeError`` was raised (`33327`)
- `Series.to_period` now raises a ``TypeError`` if the axis is not a `DatetimeIndex`. Previously an ``AttributeError`` was raised (`33327`)
- `Period` no longer accepts tuples for the ``freq`` argument (`34658`)
- Bug in `Timestamp` where constructing a `Timestamp` from ambiguous epoch time and calling constructor again changed the `Timestamp.value` property (`24329`)

- `DatetimeArray.searchsorted`, `TimedeltaArray.searchsorted`, `PeriodArray.searchsorted` not recognizing non-pandas scalars and incorrectly raising ``ValueError`` instead of ``TypeError`` (`30950`)
- Bug in `Timestamp` where constructing `Timestamp` with dateutil timezone less than 128 nanoseconds before daylight saving time switch from winter to summer would result in nonexistent time (`31043`)
- Bug in `Period.to_timestamp`, `Period.start_time` with microsecond frequency returning a timestamp one nanosecond earlier than the correct time (`31475`)
- `Timestamp` raised a confusing error message when year, month or day is missing (`31200`)
- Bug in `DatetimeIndex` constructor incorrectly accepting ``bool``-dtype inputs (`32668`)
- Bug in `DatetimeIndex.searchsorted` not accepting a ``list`` or `Series` as its argument (`32762`)
- Bug where `PeriodIndex` raised when passed a `Series` of strings (`26109`)
- Bug in `Timestamp` arithmetic when adding or subtracting an ``np.ndarray`` with ``timedelta64`` dtype (`33296`)
- Bug in `DatetimeIndex.to_period` not inferring the frequency when called with no arguments (`33358`)
- Bug in `DatetimeIndex.tz_localize` incorrectly retaining ``freq`` in some cases where the original ``freq`` is no longer valid (`30511`)
- Bug in `DatetimeIndex.intersection` losing ``freq`` and timezone in some cases (`33604`)
- Bug in `DatetimeIndex.get_indexer` where incorrect output would be returned for mixed datetime-like targets (`33741`)
- Bug in `DatetimeIndex` addition and subtraction with some types of `DateOffset` objects incorrectly retaining an invalid ``freq`` attribute (`33779`)
- Bug in `DatetimeIndex` where setting the ``freq`` attribute on an index could silently change the ``freq`` attribute on another index viewing the same data (`33552`)
- `DataFrame.min` and `DataFrame.max` were not returning consistent results with `Series.min` and `Series.max` when called on objects initialized with empty `pd.to_datetime`
- Bug in `DatetimeIndex.intersection` and `TimedeltaIndex.intersection` with results not having the correct ``name`` attribute (`33904`)
- Bug in `DatetimeArray.__setitem__`, `TimedeltaArray.__setitem__`, `PeriodArray.__setitem__` incorrectly allowing values with ``int64`` dtype to be silently cast (`33717`)
- Bug in subtracting `TimedeltaIndex` from `Period` incorrectly raising ``TypeError`` in some cases where it should succeed and ``IncompatibleFrequency`` in some cases where it should raise ``TypeError`` (`33883`)
- Bug in constructing a `Series` or `Index` from a read-only NumPy array with non-ns  resolution which converted to object dtype instead of coercing to ``datetime64[ns]``  dtype when within the timestamp bounds (`34843`).
- The ``freq`` keyword in `Period`, `date_range`, `period_range`, `pd.tseries.frequencies.to_offset` no longer allows tuples, pass as string instead (`34703`)
- Bug in `DataFrame.append` when appending a `Series` containing a scalar tz-aware `Timestamp` to an empty `DataFrame` resulted in an object column instead of ``datetime64[ns, tz]`` dtype (`35038`)
- ``OutOfBoundsDatetime`` issues an improved error message when timestamp is out of implementation bounds. (`32967`)
- Bug in `AbstractHolidayCalendar.holidays` when no rules were defined (`31415`)
- Bug in `Tick` comparisons raising ``TypeError`` when comparing against timedelta-like objects (`34088`)
- Bug in `Tick` multiplication raising ``TypeError`` when multiplying by a float (`34486`)  Timedelta
^^^  - Bug in constructing a `Timedelta` with a high precision integer that would round the `Timedelta` components (`31354`)
- Bug in dividing ``np.nan`` or ``None`` by `Timedelta` incorrectly returning ``NaT`` (`31869`)
- `Timedelta` now understands ``µs`` as an identifier for microsecond (`32899`)

- `Timedelta` string representation now includes nanoseconds, when nanoseconds are non-zero (`9309`)
- Bug in comparing a `Timedelta` object against an ``np.ndarray`` with ``timedelta64`` dtype incorrectly viewing all entries as unequal (`33441`)
- Bug in `timedelta_range` that produced an extra point on a edge case (`30353`, `33498`)
- Bug in `DataFrame.resample` that produced an extra point on a edge case (`30353`, `13022`, `33498`)
- Bug in `DataFrame.resample` that ignored the ``loffset`` argument when dealing with timedelta (`7687`, `33498`)
- Bug in `Timedelta` and `pandas.to_timedelta` that ignored the ``unit`` argument for string input (`12136`)  Timezones
^^^  - Bug in `to_datetime` with ``infer_datetime_format=True`` where timezone names (e.g. ``UTC``) would not be parsed correctly (`33133`)  Numeric
^^^
- Bug in `DataFrame.floordiv` with ``axis=0`` not treating division-by-zero like `Series.floordiv` (`31271`)
- Bug in `to_numeric` with string argument ``"uint64"`` and ``errors="coerce"`` silently fails (`32394`)
- Bug in `to_numeric` with ``downcast="unsigned"`` fails for empty data (`32493`)
- Bug in `DataFrame.mean` with ``numeric_only=False`` and either ``datetime64`` dtype or ``PeriodDtype`` column incorrectly raising ``TypeError`` (`32426`)
- Bug in `DataFrame.count` with ``level="foo"`` and index level ``"foo"`` containing NaNs causes segmentation fault (`21824`)
- Bug in `DataFrame.diff` with ``axis=1`` returning incorrect results with mixed dtypes (`32995`)
- Bug in `DataFrame.corr` and `DataFrame.cov` raising when handling nullable integer columns with ``pandas.NA`` (`33803`)
- Bug in arithmetic operations between `DataFrame` objects with non-overlapping columns with duplicate labels causing an infinite loop (`35194`)
- Bug in `DataFrame` and `Series` addition and subtraction between object-dtype objects and ``datetime64`` dtype objects (`33824`)
- Bug in `Index.difference` giving incorrect results when comparing a `Float64Index` and object `Index` (`35217`)
- Bug in `DataFrame` reductions (e.g. ``df.min()``, ``df.max()``) with ``ExtensionArray`` dtypes (`34520`, `32651`)
- `Series.interpolate` and `DataFrame.interpolate` now raise a ValueError if ``limit_direction`` is ``'forward'`` or ``'both'`` and ``method`` is ``'backfill'`` or ``'bfill'`` or ``limit_direction`` is ``'backward'`` or ``'both'`` and ``method`` is ``'pad'`` or ``'ffill'`` (`34746`)  Conversion
^^^
- Bug in `Series` construction from NumPy array with big-endian ``datetime64`` dtype (`29684`)
- Bug in `Timedelta` construction with large nanoseconds keyword value (`32402`)
- Bug in `DataFrame` construction where sets would be duplicated rather than raising (`32582`)
- The `DataFrame` constructor no longer accepts a list of `DataFrame` objects. Because of changes to NumPy, `DataFrame` objects are now consistently treated as 2D objects, so a list of `DataFrame` objects is considered 3D, and no longer acceptable for the `DataFrame` constructor (`32289`).
- Bug in `DataFrame` when initiating a frame with lists and assign ``columns`` with nested list for ``MultiIndex`` (`32173`)
- Improved error message for invalid construction of list when creating a new index (`35190`)  Strings
^^^  - Bug in the `~Series.astype` method when converting "string" dtype data to nullable integer dtype (`32450`).
- Fixed issue where taking ``min`` or ``max`` of a ``StringArray`` or ``Series`` with ``StringDtype`` type would raise. (`31746`)
- Bug in `Series.str.cat` returning ``NaN`` output when other had `Index` type (`33425`)
- `pandas.api.dtypes.is_string_dtype` no longer incorrectly identifies categorical series as string.  Interval
^^^
- Bug in `IntervalArray` incorrectly allowing the underlying data to be changed when setting values (`32782`)  Indexing

^^^
^^^  - `DataFrame.xs` now raises a  ``TypeError`` if a ``level`` keyword is supplied and the axis is not a `MultiIndex`. Previously an ``AttributeError`` was raised (`33610`)
- Bug in slicing on a `DatetimeIndex` with a partial-timestamp dropping high-resolution indices near the end of a year, quarter, or month (`31064`)
- Bug in `PeriodIndex.get_loc` treating higher-resolution strings differently from `PeriodIndex.get_value` (`31172`)
- Bug in `Series.at` and `DataFrame.at` not matching ``.loc`` behavior when looking up an integer in a `Float64Index` (`31329`)
- Bug in `PeriodIndex.is_monotonic` incorrectly returning ``True`` when containing leading ``NaT`` entries (`31437`)
- Bug in `DatetimeIndex.get_loc` raising ``KeyError`` with converted-integer key instead of the user-passed key (`31425`)
- Bug in `Series.xs` incorrectly returning ``Timestamp`` instead of ``datetime64`` in some object-dtype cases (`31630`)
- Bug in `DataFrame.iat` incorrectly returning ``Timestamp`` instead of ``datetime`` in some object-dtype cases (`32809`)
- Bug in `DataFrame.at` when either columns or index is non-unique (`33041`)
- Bug in `Series.loc` and `DataFrame.loc` when indexing with an integer key on a object-dtype `Index` that is not all-integers (`31905`)
- Bug in `DataFrame.iloc.__setitem__` on a `DataFrame` with duplicate columns incorrectly setting values for all matching columns (`15686`, `22036`)
- Bug in `DataFrame.loc` and `Series.loc` with a `DatetimeIndex`, `TimedeltaIndex`, or `PeriodIndex` incorrectly allowing lookups of non-matching datetime-like dtypes (`32650`)
- Bug in `Series.__getitem__` indexing with non-standard scalars, e.g. ``np.dtype`` (`32684`)
- Bug in `Index` constructor where an unhelpful error message was raised for NumPy scalars (`33017`)
- Bug in `DataFrame.lookup` incorrectly raising an ``AttributeError`` when ``frame.index`` or ``frame.columns`` is not unique; this will now raise a ``ValueError`` with a helpful error message (`33041`)
- Bug in `Interval` where a `Timedelta` could not be added or subtracted from a `Timestamp` interval (`32023`)
- Bug in `DataFrame.copy` not invalidating _item_cache after copy caused post-copy value updates to not be reflected (`31784`)
- Fixed regression in `DataFrame.loc` and `Series.loc` throwing an error when a ``datetime64[ns, tz]`` value is provided (`32395`)
- Bug in `Series.__getitem__` with an integer key and a `MultiIndex` with leading integer level failing to raise ``KeyError`` if the key is not present in the first level (`33355`)
- Bug in `DataFrame.iloc` when slicing a single column `DataFrame` with ``ExtensionDtype`` (e.g. ``df.iloc[:, :1]``) returning an invalid result (`32957`)
- Bug in `DatetimeIndex.insert` and `TimedeltaIndex.insert` causing index ``freq`` to be lost when setting an element into an empty `Series` (`33573`)
- Bug in `Series.__setitem__` with an `IntervalIndex` and a list-like key of integers (`33473`)
- Bug in `Series.__getitem__` allowing missing labels with ``np.ndarray``, `Index`, `Series` indexers but not ``list``, these now all raise ``KeyError`` (`33646`)
- Bug in `DataFrame.truncate` and `Series.truncate` where index was assumed to be monotone increasing (`33756`)
- Indexing with a list of strings representing datetimes failed on `DatetimeIndex` or `PeriodIndex` (`11278`)
- Bug in `Series.at` when used with a `MultiIndex` would raise an exception on valid inputs (`26989`)
- Bug in `DataFrame.loc` with dictionary of values changes columns with dtype of ``int`` to ``float`` (`34573`)
- Bug in `Series.loc` when used with a `MultiIndex` would raise an ``IndexingError`` when accessing a ``None`` value (`34318`)
- Bug in `DataFrame.reset_index` and `Series.reset_index` would not preserve data types on an empty `DataFrame` or `Series` with a `MultiIndex` (`19602`)

- Bug in `Series` and `DataFrame` indexing with a ``time`` key on a `DatetimeIndex` with ``NaT`` entries (`35114`)  Missing
^^^
- Calling `fillna` on an empty `Series` now correctly returns a shallow copied object. The behaviour is now consistent with `Index`, `DataFrame` and a non-empty `Series` (`32543`).
- Bug in `Series.replace` when argument ``to_replace`` is of type dict/list and is used on a `Series` containing ``<NA>`` was raising a ``TypeError``. The method now handles this by ignoring ``<NA>`` values when doing the comparison for the replacement (`32621`)
- Bug in `~Series.any` and `~Series.all` incorrectly returning ``<NA>`` for all ``False`` or all ``True`` values using the nulllable Boolean dtype and with ``skipna=False`` (`33253`)
- Clarified documentation on interpolate with ``method=akima``. The ``der`` parameter must be scalar or ``None`` (`33426`)
- `DataFrame.interpolate` uses the correct axis convention now. Previously interpolating along columns lead to interpolation along indices and vice versa. Furthermore interpolating with methods ``pad``, ``ffill``, ``bfill`` and ``backfill`` are identical to using these methods with `DataFrame.fillna` (`12918`, `29146`)
- Bug in `DataFrame.interpolate` when called on a `DataFrame` with column names of string type was throwing a ValueError. The method is now independent of the type of the column names (`33956`)
- Passing `NA` into a format string using format specs will now work. For example ``"{:.1f}".format(pd.NA)`` would previously raise a ``ValueError``, but will now return the string ``"<NA>"`` (`34740`)
- Bug in `Series.map` not raising on invalid ``na_action`` (`32815`)  MultiIndex
^^^  - `DataFrame.swaplevels` now raises a ``TypeError`` if the axis is not a `MultiIndex`. Previously an ``AttributeError`` was raised (`31126`)
- Bug in `Dataframe.loc` when used with a `MultiIndex`. The returned values were not in the same order as the given inputs (`22797`)  .. ipython:: python  df = pd.DataFrame(np.arange(4),  index=[["a", "a", "b", "b"], [1, 2, 1, 2]])  # Rows are now ordered as the requested keys  df.loc[(['b', 'a'], [2, 1]), :]  - Bug in `MultiIndex.intersection` was not guaranteed to preserve order when ``sort=False``. (`31325`)
- Bug in `DataFrame.truncate` was dropping `MultiIndex` names. (`34564`)  .. ipython:: python  left = pd.MultiIndex.from_arrays([["b", "a"], [2, 1]])  right = pd.MultiIndex.from_arrays([["a", "b", "c"], [1, 2, 3]])  # Common elements are now guaranteed to be ordered by the left side  left.intersection(right, sort=False)  - Bug when joining two `MultiIndex` without specifying level with different columns. Return-indexers parameter was ignored. (`34074`)  IO
^^
- Passing a ``set`` as ``names`` argument to `pandas.read_csv`, `pandas.read_table`, or `pandas.read_fwf` will raise ``ValueError: Names should be an ordered collection.`` (`34946`)
- Bug in print-out when ``display.precision`` is zero. (`20359`)
- Bug in `read_json` where integer overflow was occurring when json contains big number strings. (`30320`)
- `read_csv` will now raise a ``ValueError`` when the arguments ``header`` and ``prefix`` both are not ``None``. (`27394`)
- Bug in `DataFrame.to_json` was raising ``NotFoundError`` when ``path_or_buf`` was an S3 URI (`28375`)
- Bug in `DataFrame.to_parquet` overwriting pyarrow's default for  ``coerce_timestamps``; following pyarrow's default allows writing nanosecond  timestamps with ``version="2.0"`` (`31652`).
- Bug in `read_csv` was raising ``TypeError`` when ``sep=None`` was used in combination with ``comment`` keyword (`31396`)
- Bug in `HDFStore` that caused it to set to ``int64`` the dtype of a ``datetime64`` column when reading a `DataFrame` in Python 3 from fixed format written in Python 2 (`31750`)

- `read_sas()` now handles dates and datetimes larger than `Timestamp.max` returning them as `datetime.datetime` objects (`20927`)
- Bug in `DataFrame.to_json` where ``Timedelta`` objects would not be serialized correctly with ``date_format="iso"`` (`28256`)
- `read_csv` will raise a ``ValueError`` when the column names passed in ``parse_dates`` are missing in the `Dataframe` (`31251`)
- Bug in `read_excel` where a UTF-8 string with a high surrogate would cause a segmentation violation (`23809`)
- Bug in `read_csv` was causing a file descriptor leak on an empty file (`31488`)
- Bug in `read_csv` was causing a segfault when there were blank lines between the header and data rows (`28071`)
- Bug in `read_csv` was raising a misleading exception on a permissions issue (`23784`)
- Bug in `read_csv` was raising an ``IndexError`` when ``header=None`` and two extra data columns
- Bug in `read_sas` was raising an ``AttributeError`` when reading files from Google Cloud Storage (`33069`)
- Bug in `DataFrame.to_sql` where an ``AttributeError`` was raised when saving an out of bounds date (`26761`)
- Bug in `read_excel` did not correctly handle multiple embedded spaces in OpenDocument text cells. (`32207`)
- Bug in `read_json` was raising ``TypeError`` when reading a ``list`` of Booleans into a `Series`. (`31464`)
- Bug in `pandas.io.json.json_normalize` where location specified by ``record_path`` doesn't point to an array. (`26284`)
- `pandas.read_hdf` has a more explicit error message when loading an  unsupported HDF file (`9539`)
- Bug in `~DataFrame.read_feather` was raising an ``ArrowIOError`` when reading an s3 or http file path (`29055`)
- Bug in `~DataFrame.to_excel` could not handle the column name ``render`` and was raising an ``KeyError`` (`34331`)
- Bug in `~SQLDatabase.execute` was raising a ``ProgrammingError`` for some DB-API drivers when the SQL statement contained the ``%`` character and no parameters were present (`34211`)
- Bug in `~pandas.io.stata.StataReader` which resulted in categorical variables with different dtypes when reading data using an iterator. (`31544`)
- `HDFStore.keys` has now an optional ``include`` parameter that allows the retrieval of all native HDF5 table names (`29916`)
- ``TypeError`` exceptions raised by `read_csv` and `read_table` were showing as ``parser_f`` when an unexpected keyword argument was passed (`25648`)
- Bug in `read_excel` for ODS files removes 0.0 values (`27222`)
- Bug in `ujson.encode` was raising an ``OverflowError`` with numbers larger than ``sys.maxsize`` (`34395`)
- Bug in `HDFStore.append_to_multiple` was raising a ``ValueError`` when the ``min_itemsize`` parameter is set (`11238`)
- Bug in `~HDFStore.create_table` now raises an error when ``column`` argument was not specified in ``data_columns`` on input (`28156`)
- `read_json` now could read line-delimited json file from a file url while ``lines`` and ``chunksize`` are set.
- Bug in `DataFrame.to_sql` when reading DataFrames with ``-np.inf`` entries with MySQL now has a more explicit ``ValueError`` (`34431`)
- Bug where capitalised files extensions were not decompressed by read_* functions (`35164`)
- Bug in `read_excel` that was raising a ``TypeError`` when ``header=None`` and ``index_col`` is given as a ``list`` (`31783`)
- Bug in `read_excel` where datetime values are used in the header in a `MultiIndex` (`34748`)
- `read_excel` no longer takes ``**kwds`` arguments. This means that passing in the keyword argument ``chunksize`` now raises a ``TypeError`` (previously raised a ``NotImplementedError``), while passing in the keyword argument ``encoding`` now raises a ``TypeError`` (`34464`)
- Bug in `DataFrame.to_records` was incorrectly losing timezone information in timezone-aware ``datetime64`` columns (`32535`)  Plotting

^^^  - `DataFrame.plot` for line/bar now accepts color by dictionary (`8193`).
- Bug in `DataFrame.plot.hist` where weights are not working for multiple columns (`33173`)
- Bug in `DataFrame.boxplot` and `DataFrame.plot.boxplot` lost color attributes of ``medianprops``, ``whiskerprops``, ``capprops`` and ``boxprops`` (`30346`)
- Bug in `DataFrame.hist` where the order of ``column`` argument was ignored (`29235`)
- Bug in `DataFrame.plot.scatter` that when adding multiple plots with different ``cmap``, colorbars always use the first ``cmap`` (`33389`)
- Bug in `DataFrame.plot.scatter` was adding a colorbar to the plot even if the argument ``c`` was assigned to a column containing color names (`34316`)
- Bug in `pandas.plotting.bootstrap_plot` was causing cluttered axes and overlapping labels (`34905`)
- Bug in `DataFrame.plot.scatter` caused an error when plotting variable marker sizes (`32904`)  GroupBy/resample/rolling
^^^  - Using a `pandas.api.indexers.BaseIndexer` with ``count``, ``min``, ``max``, ``median``, ``skew``,  ``cov``, ``corr`` will now return correct results for any monotonic `pandas.api.indexers.BaseIndexer` descendant (`32865`)
- `DataFrameGroupby.mean` and `SeriesGroupby.mean` (and similarly for `~DataFrameGroupby.median`, `~DataFrameGroupby.std` and `~DataFrameGroupby.var`) now raise a ``TypeError`` if a non-accepted keyword argument is passed into it. Previously an ``UnsupportedFunctionCall`` was raised (``AssertionError`` if ``min_count`` passed into `~DataFrameGroupby.median`) (`31485`)
- Bug in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` raising ``ValueError`` when the ``by`` axis is not sorted, has duplicates, and the applied ``func`` does not mutate passed in objects (`30667`)
- Bug in `DataFrameGroupBy.transform` produces an incorrect result with transformation functions (`30918`)
- Bug in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` were returning the wrong result when grouping by multiple keys of which some were categorical and others not (`32494`)
- Bug in `.DataFrameGroupBy.count` and `.SeriesGroupBy.count` causing segmentation fault when grouped-by columns contain NaNs (`32841`)
- Bug in `DataFrame.groupby` and `Series.groupby` produces inconsistent type when aggregating Boolean `Series` (`32894`)
- Bug in `DataFrameGroupBy.sum` and `SeriesGroupBy.sum` where a large negative number would be returned when the number of non-null values was below ``min_count`` for nullable integer dtypes (`32861`)
- Bug in `SeriesGroupBy.quantile` was raising on nullable integers (`33136`)
- Bug in `DataFrame.resample` where an ``AmbiguousTimeError`` would be raised when the resulting timezone aware `DatetimeIndex` had a DST transition at midnight (`25758`)
- Bug in `DataFrame.groupby` where a ``ValueError`` would be raised when grouping by a categorical column with read-only categories and ``sort=False`` (`33410`)
- Bug in `.DataFrameGroupBy.agg`, `.SeriesGroupBy.agg`, `.DataFrameGroupBy.transform`, `.SeriesGroupBy.transform`, `.DataFrameGroupBy.resample`, and `.SeriesGroupBy.resample` where subclasses are not preserved (`28330`)
- Bug in `SeriesGroupBy.agg` where any column name was accepted in the named aggregation of `SeriesGroupBy` previously. The behaviour now allows only ``str`` and callables else would raise ``TypeError``. (`34422`)
- Bug in `DataFrame.groupby` lost the name of the `Index` when one of the ``agg`` keys referenced an empty list (`32580`)
- Bug in `Rolling.apply` where ``center=True`` was ignored when ``engine='numba'`` was specified (`34784`)
- Bug in `DataFrame.ewm.cov` was throwing ``AssertionError`` for `MultiIndex` inputs (`34440`)
- Bug in `core.groupby.DataFrameGroupBy.quantile` raised ``TypeError`` for non-numeric types rather than dropping the columns (`27892`)

- Bug in `core.groupby.DataFrameGroupBy.transform` when ``func='nunique'`` and columns are of type ``datetime64``, the result would also be of type ``datetime64`` instead of ``int64`` (`35109`)
- Bug in `DataFrame.groupby` raising an ``AttributeError`` when selecting a column and aggregating with ``as_index=False`` (`35246`).
- Bug in `DataFrameGroupBy.first` and `DataFrameGroupBy.last` that would raise an unnecessary ``ValueError`` when grouping on multiple ``Categoricals`` (`34951`)  Reshaping
^^^  - Bug effecting all numeric and Boolean reduction methods not returning subclassed data type. (`25596`)
- Bug in `DataFrame.pivot_table` when only `MultiIndexed` columns is set (`17038`)
- Bug in `DataFrame.unstack` and `Series.unstack` can take tuple names in `MultiIndexed` data (`19966`)
- Bug in `DataFrame.pivot_table` when ``margin`` is ``True`` and only ``column`` is defined (`31016`)
- Fixed incorrect error message in `DataFrame.pivot` when ``columns`` is set to ``None``. (`30924`)
- Bug in `crosstab` when inputs are two `Series` and have tuple names, the output will keep a dummy `MultiIndex` as columns. (`18321`)
- `DataFrame.pivot` can now take lists for ``index`` and ``columns`` arguments (`21425`)
- Bug in `concat` where the resulting indices are not copied when ``copy=True`` (`29879`)
- Bug in `SeriesGroupBy.aggregate` was resulting in aggregations being overwritten when they shared the same name (`30880`)
- Bug where `Index.astype` would lose the `name` attribute when converting from ``Float64Index`` to ``Int64Index``, or when casting to an ``ExtensionArray`` dtype (`32013`)
- `Series.append` will now raise a ``TypeError`` when passed a `DataFrame` or a sequence containing `DataFrame` (`31413`)
- `DataFrame.replace` and `Series.replace` will raise a ``TypeError`` if ``to_replace`` is not an expected type. Previously the ``replace`` would fail silently (`18634`)
- Bug on inplace operation of a `Series` that was adding a column to the `DataFrame` from where it was originally dropped from (using ``inplace=True``) (`30484`)
- Bug in `DataFrame.apply` where callback was called with `Series` parameter even though ``raw=True`` requested. (`32423`)
- Bug in `DataFrame.pivot_table` losing timezone information when creating a `MultiIndex` level from a column with timezone-aware dtype (`32558`)
- Bug in `concat` where when passing a non-dict mapping as ``objs`` would raise a ``TypeError`` (`32863`)
- `DataFrame.agg` now provides more descriptive ``SpecificationError`` message when attempting to aggregate a non-existent column (`32755`)
- Bug in `DataFrame.unstack` when `MultiIndex` columns and `MultiIndex` rows were used (`32624`, `24729` and `28306`)
- Appending a dictionary to a `DataFrame` without passing ``ignore_index=True`` will raise ``TypeError: Can only append a dict if ignore_index=True`` instead of ``TypeError: Can only append a `Series` if ignore_index=True or if the `Series` has a name`` (`30871`)
- Bug in `DataFrame.corrwith()`, `DataFrame.memory_usage()`, `DataFrame.dot()`,  `DataFrame.idxmin()`, `DataFrame.idxmax()`, `DataFrame.duplicated()`, `DataFrame.isin()`,  `DataFrame.count()`, `Series.explode()`, `Series.asof()` and `DataFrame.asof()` not  returning subclassed types. (`31331`)
- Bug in `concat` was not allowing for concatenation of `DataFrame` and `Series` with duplicate keys (`33654`)
- Bug in `cut` raised an error when the argument ``labels`` contains duplicates (`33141`)
- Ensure only named functions can be used in `eval()` (`32460`)
- Bug in `Dataframe.aggregate` and `Series.aggregate` was causing a recursive loop in some cases (`34224`)
- Fixed bug in `melt` where melting `MultiIndex` columns with ``col_level > 0`` would raise a ``KeyError`` on ``id_vars`` (`34129`)
- Bug in `Series.where` with an empty `Series` and empty ``cond`` having non-bool dtype (`34592`)

- Fixed regression where `DataFrame.apply` would raise ``ValueError`` for elements with ``S`` dtype (`34529`)  Sparse
^^^
- Creating a `SparseArray` from timezone-aware dtype will issue a warning before dropping timezone information, instead of doing so silently (`32501`)
- Bug in `arrays.SparseArray.from_spmatrix` wrongly read scipy sparse matrix (`31991`)
- Bug in `Series.sum` with ``SparseArray`` raised a ``TypeError`` (`25777`)
- Bug where `DataFrame` containing an all-sparse `SparseArray` filled with ``NaN`` when indexed by a list-like (`27781`, `29563`)
- The repr of `SparseDtype` now includes the repr of its ``fill_value`` attribute. Previously it used ``fill_value``'s  string representation (`34352`)
- Bug where empty `DataFrame` could not be cast to `SparseDtype` (`33113`)
- Bug in `arrays.SparseArray` was returning the incorrect type when indexing a sparse dataframe with an iterable (`34526`, `34540`)  ExtensionArray
^^^  - Fixed bug where `Series.value_counts` would raise on empty input of ``Int64`` dtype (`33317`)
- Fixed bug in `concat` when concatenating `DataFrame` objects with non-overlapping columns resulting in object-dtype columns rather than preserving the extension dtype (`27692`, `33027`)
- Fixed bug where `StringArray.isna` would return ``False`` for NA values when ``pandas.options.mode.use_inf_as_na`` was set to ``True`` (`33655`)
- Fixed bug in `Series` construction with EA dtype and index but no data or scalar data fails (`26469`)
- Fixed bug that caused `Series.__repr__()` to crash for extension types whose elements are multidimensional arrays (`33770`).
- Fixed bug where `Series.update` would raise a ``ValueError`` for ``ExtensionArray`` dtypes with missing values (`33980`)
- Fixed bug where `StringArray.memory_usage` was not implemented (`33963`)
- Fixed bug where `DataFrameGroupBy` would ignore the ``min_count`` argument for aggregations on nullable Boolean dtypes (`34051`)
- Fixed bug where the constructor of `DataFrame` with ``dtype='string'`` would fail (`27953`, `33623`)
- Bug where `DataFrame` column set to scalar extension type was considered an object type rather than the extension type (`34832`)
- Fixed bug in `IntegerArray.astype` to correctly copy the mask as well (`34931`).  Other
^^^  - Set operations on an object-dtype `Index` now always return object-dtype results (`31401`)
- Fixed `pandas.testing.assert_series_equal` to correctly raise if the ``left`` argument is a different subclass with ``check_series_type=True`` (`32670`).
- Getting a missing attribute in a `DataFrame.query` or `DataFrame.eval` string raises the correct ``AttributeError`` (`32408`)
- Fixed bug in `pandas.testing.assert_series_equal` where dtypes were checked for ``Interval`` and ``ExtensionArray`` operands when ``check_dtype`` was ``False`` (`32747`)
- Bug in `DataFrame.__dir__` caused a segfault when using unicode surrogates in a column name (`25509`)
- Bug in `DataFrame.equals` and `Series.equals` in allowing subclasses to be equal (`34402`).  .. ---  .. _whatsnew_110.contributors:  Contributors
~~~  .. contributors:: v1.0.5..v1.1.0|HEAD

What's new in 1.1.1 (August 20, 2020)

These are the changes in pandas 1.1.1. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_111.regressions:  Fixed regressions
~~~  - Fixed regression in `CategoricalIndex.format` where, when stringified scalars had different lengths, the shorter string would be right-filled with spaces, so it had the same length as the longest string (`35439`)
- Fixed regression in `Series.truncate` when trying to truncate a single-element series (`35544`)
- Fixed regression where `DataFrame.to_numpy` would raise a ``RuntimeError`` for mixed dtypes when converting to ``str`` (`35455`)
- Fixed regression where `read_csv` would raise a ``ValueError`` when ``pandas.options.mode.use_inf_as_na`` was set to ``True`` (`35493`)
- Fixed regression where `pandas.testing.assert_series_equal` would raise an error when non-numeric dtypes were passed with ``check_exact=True`` (`35446`)
- Fixed regression in ``.groupby(..).rolling(..)`` where column selection was ignored (`35486`)
- Fixed regression where `DataFrame.interpolate` would raise a ``TypeError`` when the `DataFrame` was empty (`35598`)
- Fixed regression in `DataFrame.shift` with ``axis=1`` and heterogeneous dtypes (`35488`)
- Fixed regression in `DataFrame.diff` with read-only data (`35559`)
- Fixed regression in ``.groupby(..).rolling(..)`` where a segfault would occur with ``center=True`` and an odd number of values (`35552`)
- Fixed regression in `DataFrame.apply` where functions that altered the input in-place only operated on a single row (`35462`)
- Fixed regression in `DataFrame.reset_index` would raise a ``ValueError`` on empty `DataFrame` with a `MultiIndex` with a ``datetime64`` dtype level (`35606`, `35657`)
- Fixed regression where `pandas.merge_asof` would raise a ``UnboundLocalError`` when ``left_index``, ``right_index`` and ``tolerance`` were set (`35558`)
- Fixed regression in ``.groupby(..).rolling(..)`` where a custom ``BaseIndexer`` would be ignored (`35557`)
- Fixed regression in `DataFrame.replace` and `Series.replace` where compiled regular expressions would be ignored during replacement (`35680`)
- Fixed regression in `~pandas.core.groupby.DataFrameGroupBy.aggregate` where a list of functions would produce the wrong results if at least one of the functions did not aggregate (`35490`)
- Fixed memory usage issue when instantiating large `pandas.arrays.StringArray` (`35499`)  .. ---  .. _whatsnew_111.bug_fixes:  Bug fixes
~~~  - Bug in `~pandas.io.formats.style.Styler` whereby ``cell_ids`` argument had no effect due to other recent changes (`35588`) (`35663`)
- Bug in `pandas.testing.assert_series_equal` and `pandas.testing.assert_frame_equal` where extension dtypes were not ignored when ``check_dtypes`` was set to ``False`` (`35715`)
- Bug in `to_timedelta` fails when ``arg`` is a `Series` with ``Int64`` dtype containing null values (`35574`)
- Bug in ``.groupby(..).rolling(..)`` where passing ``closed`` with column selection would raise a ``ValueError`` (`35549`)
- Bug in `DataFrame` constructor failing to raise ``ValueError`` in some cases when ``data`` and ``index`` have mismatched lengths (`33437`)  .. ---  .. _whatsnew_111.contributors:  Contributors
~~~  .. contributors:: v1.1.0..v1.1.1

What's new in 1.1.2 (September 8, 2020)

These are the changes in pandas 1.1.2. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_112.regressions:  Fixed regressions
~~~
- Regression in `DatetimeIndex.intersection` incorrectly raising ``AssertionError`` when intersecting against a list (`35876`)
- Fix regression in updating a column inplace (e.g. using ``df['col'].fillna(.., inplace=True)``) (`35731`)
- Fix regression in `DataFrame.append` mixing tz-aware and tz-naive datetime columns (`35460`)
- Performance regression for `RangeIndex.format` (`35712`)
- Regression where `MultiIndex.get_loc` would return a slice spanning the full index when passed an empty list (`35878`)
- Fix regression in invalid cache after an indexing operation; this can manifest when setting which does not update the data (`35521`)
- Regression in `DataFrame.replace` where a ``TypeError`` would be raised when attempting to replace elements of type `Interval` (`35931`)
- Fix regression in pickle roundtrip of the ``closed`` attribute of `IntervalIndex` (`35658`)
- Fixed regression in `DataFrameGroupBy.agg` where a ``ValueError: buffer source array is read-only`` would be raised when the underlying array is read-only (`36014`)
- Fixed regression in `Series.groupby.rolling` number of levels of `MultiIndex` in input was compressed to one (`36018`)
- Fixed regression in `DataFrameGroupBy` on an empty `DataFrame` (`36197`)  .. ---  .. _whatsnew_112.bug_fixes:  Bug fixes
~~~
- Bug in `DataFrame.eval` with ``object`` dtype column binary operations (`35794`)
- Bug in `Series` constructor raising a ``TypeError`` when constructing sparse datetime64 dtypes (`35762`)
- Bug in `DataFrame.apply` with ``result_type="reduce"`` returning with incorrect index (`35683`)
- Bug in `Series.astype` and `DataFrame.astype` not respecting the ``errors`` argument when set to ``"ignore"`` for extension dtypes (`35471`)
- Bug in `DateTimeIndex.format` and `PeriodIndex.format` with ``name=True`` setting the first item to ``"None"`` where it should be ``""`` (`35712`)
- Bug in `Float64Index.__contains__` incorrectly raising ``TypeError`` instead of returning ``False`` (`35788`)
- Bug in `Series` constructor incorrectly raising a ``TypeError`` when passed an ordered set (`36044`)
- Bug in `Series.dt.isocalendar` and `DatetimeIndex.isocalendar` that returned incorrect year for certain dates (`36032`)
- Bug in `DataFrame` indexing returning an incorrect `Series` in some cases when the series has been altered and a cache not invalidated (`33675`)
- Bug in `DataFrame.corr` causing subsequent indexing lookups to be incorrect (`35882`)
- Bug in `import_optional_dependency` returning incorrect package names in cases where package name is different from import name (`35948`)
- Bug when setting empty `DataFrame` column to a `Series` in preserving name of index in frame (`31368`)  .. ---  .. _whatsnew_112.other:  Other
~~~
- `factorize` now supports ``na_sentinel=None`` to include NaN in the uniques of the values and remove ``dropna`` keyword which was unintentionally exposed to public facing API in 1.1 version from `factorize` (`35667`)
- `DataFrame.plot` and `Series.plot` raise ``UserWarning`` about usage of ``FixedFormatter`` and ``FixedLocator`` (`35684` and `35945`)  .. ---  .. _whatsnew_112.contributors:  Contributors
~~~  .. contributors:: v1.1.1..v1.1.2

What's new in 1.1.3 (October 5, 2020)

These are the changes in pandas 1.1.3. See `release` for a full changelog
~~~  Added support for new Python version
^^^  pandas 1.1.3 now supports Python 3.9 (`36296`).  Development Changes
^^^  - The minimum version of Cython is now the most recent bug-fix version (0.29.21) (`36296`).  .. ---  .. _whatsnew_113.regressions:  Fixed regressions
~~~
- Fixed regression in `DataFrame.agg`, `DataFrame.apply`, `Series.agg`, and `Series.apply` where internal suffix is exposed to the users when no relabelling is applied (`36189`)
- Fixed regression in `IntegerArray` unary plus and minus operations raising a ``TypeError`` (`36063`)
- Fixed regression when adding a `timedelta_range` to a `Timestamp` raised a ``ValueError`` (`35897`)
- Fixed regression in `Series.__getitem__` incorrectly raising when the input was a tuple (`35534`)
- Fixed regression in `Series.__getitem__` incorrectly raising when the input was a frozenset (`35747`)
- Fixed regression in modulo of `Index`, `Series` and `DataFrame` using ``numexpr`` using C not Python semantics (`36047`, `36526`)
- Fixed regression in `read_excel` with ``engine="odf"`` caused ``UnboundLocalError`` in some cases where cells had nested child nodes (`36122`, `35802`)
- Fixed regression in `DataFrame.replace` inconsistent replace when using a float in the replace method (`35376`)
- Fixed regression in `Series.loc` on a `Series` with a `MultiIndex` containing `Timestamp` raising ``InvalidIndexError`` (`35858`)
- Fixed regression in `DataFrame` and `Series` comparisons between numeric arrays and strings (`35700`, `36377`)
- Fixed regression in `DataFrame.apply` with ``raw=True`` and user-function returning string (`35940`)
- Fixed regression when setting empty `DataFrame` column to a `Series` in preserving name of index in frame (`36527`)
- Fixed regression in `Period` incorrect value for ordinal over the maximum timestamp (`36430`)
- Fixed regression in `read_table` raised ``ValueError`` when ``delim_whitespace`` was set to ``True`` (`35958`)
- Fixed regression in `Series.dt.normalize` when normalizing pre-epoch dates the result was shifted one day (`36294`)  .. ---  .. _whatsnew_113.bug_fixes:  Bug fixes
~~~
- Bug in `read_spss` where passing a ``pathlib.Path`` as ``path`` would raise a ``TypeError`` (`33666`)
- Bug in `Series.str.startswith` and `Series.str.endswith` with ``category`` dtype not propagating ``na`` parameter (`36241`)
- Bug in `Series` constructor where integer overflow would occur for sufficiently large scalar inputs when an index was provided (`36291`)
- Bug in `DataFrame.sort_values` raising an ``AttributeError`` when sorting on a key that casts column to categorical dtype (`36383`)
- Bug in `DataFrame.stack` raising a ``ValueError`` when stacking `MultiIndex` columns based on position when the levels had duplicate names (`36353`)
- Bug in `Series.astype` showing too much precision when casting from ``np.float32`` to string dtype (`36451`)
- Bug in `Series.isin` and `DataFrame.isin` when using ``NaN`` and a row length above 1,000,000 (`22205`)
- Bug in `cut` raising a ``ValueError`` when passed a `Series` of labels with ``ordered=False`` (`36603`)  .. ---  .. _whatsnew_113.other:  Other
~~~
- Reverted enhancement added in pandas-1.1.0 where `timedelta_range` infers a frequency when passed ``start``, ``stop``, and ``periods`` (`32377`)  .. ---  .. _whatsnew_113.contributors:  Contributors
~~~  .. contributors:: v1.1.2..v1.1.3

What's new in 1.1.4 (October 30, 2020)

These are the changes in pandas 1.1.4. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_114.regressions:  Fixed regressions
~~~
- Fixed regression in `read_csv` raising a ``ValueError`` when ``names`` was of type ``dict_keys`` (`36928`)
- Fixed regression in `read_csv` with more than 1M rows and specifying a ``index_col`` argument (`37094`)
- Fixed regression where attempting to mutate a `DateOffset` object would no longer raise an ``AttributeError`` (`36940`)
- Fixed regression where `DataFrame.agg` would fail with `TypeError` when passed positional arguments to be passed on to the aggregation function (`36948`).
- Fixed regression in `RollingGroupby` with ``sort=False`` not being respected (`36889`)
- Fixed regression in `Series.astype` converting ``None`` to ``"nan"`` when casting to string (`36904`)
- Fixed regression in `Series.rank` method failing for read-only data (`37290`)
- Fixed regression in `RollingGroupby` causing a segmentation fault with Index of dtype object (`36727`)
- Fixed regression in `DataFrame.resample(...).apply(...)` raised ``AttributeError`` when input was a `DataFrame` and only a `Series` was evaluated (`36951`)
- Fixed regression in ``DataFrame.groupby(..).std()`` with nullable integer dtype (`37415`)
- Fixed regression in `PeriodDtype` comparing both equal and unequal to its string representation (`37265`)
- Fixed regression where slicing `DatetimeIndex` raised `AssertionError` on irregular time series with ``pd.NaT`` or on unsorted indices (`36953` and `35509`)
- Fixed regression in certain offsets (`pd.offsets.Day() <pandas.tseries.offsets.Day>` and below) no longer being hashable (`37267`)
- Fixed regression in `StataReader` which required ``chunksize`` to be manually set when using an iterator to read a dataset (`37280`)
- Fixed regression in setitem with `DataFrame.iloc` which raised error when trying to set a value while filtering with a boolean list (`36741`)
- Fixed regression in setitem with a Series getting aligned before setting the values (`37427`)
- Fixed regression in `MultiIndex.is_monotonic_increasing` returning wrong results with ``NaN`` in at least one of the levels (`37220`)
- Fixed regression in inplace arithmetic operation (`+=`) on a Series not updating the parent DataFrame/Series (`36373`)  .. ---  .. _whatsnew_114.bug_fixes:  Bug fixes
~~~
- Bug causing ``groupby(...).sum()`` and similar to not preserve metadata (`29442`)
- Bug in `Series.isin` and `DataFrame.isin` raising a ``ValueError`` when the target was read-only (`37174`)
- Bug in `.DataFrameGroupBy.fillna` and `.SeriesGroupBy.fillna` that introduced a performance regression after 1.0.5 (`36757`)
- Bug in `DataFrame.info` was raising a ``KeyError`` when the DataFrame has integer column names (`37245`)
- Bug in `DataFrameGroupby.apply` would drop a `CategoricalIndex` when grouped on (`35792`)  .. ---  .. _whatsnew_114.contributors:  Contributors
~~~  .. contributors:: v1.1.3..v1.1.4

What's new in 1.1.5 (December 07, 2020)

These are the changes in pandas 1.1.5. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_115.regressions:  Fixed regressions
~~~
- Fixed regression in addition of a timedelta-like scalar to a `DatetimeIndex` raising incorrectly (`37295`)
- Fixed regression in `Series.groupby` raising when the `Index` of the `Series` had a tuple as its name (`37755`)
- Fixed regression in `DataFrame.loc` and `Series.loc` for ``__setitem__`` when one-dimensional tuple was given to select from `MultiIndex` (`37711`)
- Fixed regression in inplace operations on `Series` with ``ExtensionDtype`` with NumPy dtyped operand (`37910`)
- Fixed regression in metadata propagation for ``groupby`` iterator (`37343`)
- Fixed regression in `MultiIndex` constructed from a `DatetimeIndex` not retaining frequency (`35563`)
- Fixed regression in `Index` constructor raising a ``AttributeError`` when passed a `SparseArray` with datetime64 values (`35843`)
- Fixed regression in `DataFrame.unstack` with columns with integer dtype (`37115`)
- Fixed regression in indexing on a `Series` with ``CategoricalDtype`` after unpickling (`37631`)
- Fixed regression in `DataFrame.groupby` aggregation with out-of-bounds datetime objects in an object-dtype column (`36003`)
- Fixed regression in ``df.groupby(..).rolling(..)`` with the resulting `MultiIndex` when grouping by a label that is in the index (`37641`)
- Fixed regression in `DataFrame.fillna` not filling ``NaN`` after other operations such as `DataFrame.pivot` (`36495`).
- Fixed performance regression in ``df.groupby(..).rolling(..)`` (`38038`)
- Fixed regression in `MultiIndex.intersection` returning duplicates when at least one of the indexes had duplicates (`36915`)
- Fixed regression in `.DataFrameGroupBy.first`, `.SeriesGroupBy.first`, `.DataFrameGroupBy.last`, and `.SeriesGroupBy.last` where ``None`` was considered a non-NA value (`38286`)  .. ---  .. _whatsnew_115.bug_fixes:  Bug fixes
~~~
- Bug in pytables methods in python 3.9 (`38041`)  .. ---  .. _whatsnew_115.other:  Other
~~~
- Only set ``-Werror`` as a compiler flag in the CI jobs (`33315`, `33314`)  .. ---  .. _whatsnew_115.contributors:  Contributors
~~~  .. contributors:: v1.1.4..v1.1.5|HEAD

These are the changes in pandas 1.2.0. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. warning::  The xlwt package for writing old-style ``.xls``  excel files is no longer maintained.  The xlrd package is now only for reading  old-style ``.xls`` files.  Previously, the default argument ``engine=None`` to `~pandas.read_excel`  would result in using the ``xlrd`` engine in many cases, including new  Excel 2007+ (``.xlsx``) files.  If openpyxl  is installed,  many of these cases will now default to using the ``openpyxl`` engine.  See the `read_excel` documentation for more details.  Thus, it is strongly encouraged to install ``openpyxl`` to read Excel 2007+  (``.xlsx``) files.  **Please do not report issues when using ``xlrd`` to read ``.xlsx`` files.**  This is no longer supported, switch to using ``openpyxl`` instead.  Attempting to use the ``xlwt`` engine will raise a ``FutureWarning``  unless the option `io.excel.xls.writer` is set to ``"xlwt"``.  While this option is now deprecated and will also raise a ``FutureWarning``,  it can be globally set and the warning suppressed. Users are recommended to  write ``.xlsx`` files using the ``openpyxl`` engine instead.  .. ---  Enhancements
~~~  .. _whatsnew_120.duplicate_labels:  Optionally disallow duplicate labels
^^^  `Series` and `DataFrame` can now be created with ``allows_duplicate_labels=False`` flag to
control whether the index or columns can contain duplicate labels (`28394`). This can be used to
prevent accidental introduction of duplicate labels, which can affect downstream operations.  By default, duplicates continue to be allowed.  .. code-block:: ipython  In [1]: pd.Series([1, 2], index=['a', 'a'])  Out[1]:  a  1  a  2  Length: 2, dtype: int64  In [2]: pd.Series([1, 2], index=['a', 'a']).set_flags(allows_duplicate_labels=False)  ...  DuplicateLabelError: Index has duplicates.  positions  label  a  [0, 1]  pandas will propagate the ``allows_duplicate_labels`` property through many operations.  .. code-block:: ipython  In [3]: a = (  ...:  pd.Series([1, 2], index=['a', 'b'])  ...:  .set_flags(allows_duplicate_labels=False)  ...: )  In [4]: a  Out[4]:  a  1  b  2  Length: 2, dtype: int64  # An operation introducing duplicates  In [5]: a.reindex(['a', 'b', 'a'])  ...  DuplicateLabelError: Index has duplicates.  positions  label  a  [0, 2]  [1 rows x 1 columns]  .. warning::  This is an experimental feature. Currently, many methods fail to  propagate the ``allows_duplicate_labels`` value. In future versions  it is expected that every method taking or returning one or more  DataFrame or Series objects will propagate ``allows_duplicate_labels``.  See `duplicates` for more.  The ``allows_duplicate_labels`` flag is stored in the new `DataFrame.flags`
attribute. This stores global attributes that apply to the *pandas object*. This
differs from `DataFrame.attrs`, which stores information that applies to
the dataset.  Passing arguments to fsspec backends
^^^  Many read/write functions have acquired the ``storage_options`` optional argument,
to pass a dictionary of parameters to the storage backend. This allows, for
example, for passing credentials to S3 and GCS storage. The details of what
parameters can be passed to which backends can be found in the documentation
of the individual storage backends (detailed from the fsspec docs for
`builtin implementations`_ and linked to `external ones`_). See
Section `io.remote`.  `35655` added fsspec support (including ``storage_options``)
for reading excel files.  .. _builtin implementations: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
.. _external ones: https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations  .. _whatsnew_120.binary_handle_to_csv:  Support for binary file handles in ``to_csv``
^^^  `to_csv` supports file handles in binary mode (`19827` and `35058`)
with ``encoding`` (`13068` and `23854`) and ``compression`` (`22555`).

If pandas does not automatically detect whether the file handle is opened in binary or text mode,
it is necessary to provide ``mode="wb"``.  For example:  .. ipython:: python  import io  data = pd.DataFrame([0, 1, 2])  buffer = io.BytesIO()  data.to_csv(buffer, encoding="utf-8", compression="gzip")  Support for short caption and table position in ``to_latex``
^^^  `DataFrame.to_latex` now allows one to specify
a floating table position (`35281`)
and a short caption (`36267`).  The keyword ``position`` has been added to set the position.  .. ipython:: python  data = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})  table = data.to_latex(position='ht')  print(table)  Usage of the keyword ``caption`` has been extended.
Besides taking a single string as an argument,
one can optionally provide a tuple ``(full_caption, short_caption)``
to add a short caption macro.  .. ipython:: python  data = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})  table = data.to_latex(caption=('the full long caption', 'short caption'))  print(table)  .. _whatsnew_120.read_csv_table_precision_default:  Change in default floating precision for ``read_csv`` and ``read_table``
^^^  For the C parsing engine, the methods `read_csv` and `read_table` previously defaulted to a parser that
could read floating point numbers slightly incorrectly with respect to the last bit in precision.
The option ``floating_precision="high"`` has always been available to avoid this issue.
Beginning with this version, the default is now to use the more accurate parser by making
``floating_precision=None`` correspond to the high precision parser, and the new option
``floating_precision="legacy"`` to use the legacy parser. The change to using the higher precision
parser by default should have no impact on performance. (`17154`)  .. _whatsnew_120.floating:  Experimental nullable data types for float data
^^^  We've added `Float32Dtype` / `Float64Dtype` and `~arrays.FloatingArray`.
These are extension data types dedicated to floating point data that can hold the
``pd.NA`` missing value indicator (`32265`, `34307`).  While the default float data type already supports missing values using ``np.nan``,
these new data types use ``pd.NA`` (and its corresponding behavior) as the missing
value indicator, in line with the already existing nullable `integer <integer_na>`
and `boolean <boolean>` data types.  One example where the behavior of ``np.nan`` and ``pd.NA`` is different is
comparison operations:  .. ipython:: python  # the default NumPy float64 dtype  s1 = pd.Series([1.5, None])  s1  s1 > 1  .. ipython:: python  # the new nullable float64 dtype  s2 = pd.Series([1.5, None], dtype="Float64")  s2  s2 > 1  See the `missing_data.NA` doc section for more details on the behavior
when using the ``pd.NA`` missing value indicator.  As shown above, the dtype can be specified using the "Float64" or "Float32"
string (capitalized to distinguish it from the default "float64" data type).
Alternatively, you can also use the dtype object:  .. ipython:: python  pd.Series([1.5, None], dtype=pd.Float32Dtype())  Operations with the existing integer or boolean nullable data types that
give float results will now also use the nullable floating data types (`38178`).  .. warning::  Experimental: the new floating data types are currently experimental, and their  behavior or API may still change without warning. Especially the behavior  regarding NaN (distinct from NA missing values) is subject to change.  .. _whatsnew_120.index_name_preservation:  Index/column name preservation when aggregating
^^^  When aggregating using `concat` or the `DataFrame` constructor, pandas
will now attempt to preserve index and column names whenever possible (`35847`).
In the case where all inputs share a common name, this name will be assigned to the
result. When the input names do not all agree, the result will be unnamed. Here is an

example where the index name is preserved:  .. ipython:: python  idx = pd.Index(range(5), name='abc')  ser = pd.Series(range(5, 10), index=idx)  pd.concat({'x': ser[1:], 'y': ser[:-1]}, axis=1)  The same is true for `MultiIndex`, but the logic is applied separately on a
level-by-level basis.  .. _whatsnew_120.groupby_ewm:  GroupBy supports EWM operations directly
^^^  `.DataFrameGroupBy` now supports exponentially weighted window operations directly (`16037`).  .. ipython:: python  df = pd.DataFrame({'A': ['a', 'b', 'a', 'b'], 'B': range(4)})  df  df.groupby('A').ewm(com=1.0).mean()  Additionally `mean`` supports execution via `Numba_ with
the  ``engine`` and ``engine_kwargs`` arguments. Numba must be installed as an optional dependency
to use this feature.  .. _whatsnew_120.enhancements.other:  Other enhancements
^^^
- Added ``day_of_week`` (compatibility alias ``dayofweek``) property to `Timestamp`, `.DatetimeIndex`, `Period`, `PeriodIndex` (`9605`)
- Added ``day_of_year`` (compatibility alias ``dayofyear``) property to `Timestamp`, `.DatetimeIndex`, `Period`, `PeriodIndex` (`9605`)
- Added `~DataFrame.set_flags` for setting table-wide flags on a Series or DataFrame (`28394`)
- `DataFrame.applymap` now supports ``na_action`` (`23803`)
- `Index` with object dtype supports division and multiplication (`34160`)
- `io.sql.get_schema` now supports a ``schema`` keyword argument that will add a schema into the create table statement (`28486`)
- `DataFrame.explode` and `Series.explode` now support exploding of sets (`35614`)
- `DataFrame.hist` now supports time series (datetime) data (`32590`)
- `.Styler.set_table_styles` now allows the direct styling of rows and columns and can be chained (`35607`)
- `.Styler` now allows direct CSS class name addition to individual data cells (`36159`)
- `.Rolling.mean` and `.Rolling.sum` use Kahan summation to calculate the mean to avoid numerical problems (`10319`, `11645`, `13254`, `32761`, `36031`)
- `.DatetimeIndex.searchsorted`, `.TimedeltaIndex.searchsorted`, `PeriodIndex.searchsorted`, and `Series.searchsorted` with datetime-like dtypes will now try to cast string arguments (list-like and scalar) to the matching datetime-like type (`36346`)
- Added methods `IntegerArray.prod`, `IntegerArray.min`, and `IntegerArray.max` (`33790`)
- Calling a NumPy ufunc on a ``DataFrame`` with extension types now preserves the extension types when possible (`23743`)
- Calling a binary-input NumPy ufunc on multiple ``DataFrame`` objects now aligns, matching the behavior of binary operations and ufuncs on ``Series`` (`23743`).  This change has been reverted in pandas 1.2.1, and the behaviour to not align DataFrames  is deprecated instead, see the `the 1.2.1 release notes <whatsnew_121.ufunc_deprecation>`.
- Where possible `RangeIndex.difference` and `RangeIndex.symmetric_difference` will return `RangeIndex` instead of `Int64Index` (`36564`)
- `DataFrame.to_parquet` now supports `MultiIndex` for columns in parquet format (`34777`)
- `read_parquet` gained a ``use_nullable_dtypes=True`` option to use nullable dtypes that use ``pd.NA`` as missing value indicator where possible for the resulting DataFrame (default is ``False``, and only applicable for ``engine="pyarrow"``) (`31242`)
- Added `.Rolling.sem` and `Expanding.sem` to compute the standard error of the mean (`26476`)
- `.Rolling.var` and `.Rolling.std` use Kahan summation and Welford's Method to avoid numerical issues (`37051`)
- `DataFrame.corr` and `DataFrame.cov` use Welford's Method to avoid numerical issues (`37448`)
- `DataFrame.plot` now recognizes ``xlabel`` and ``ylabel`` arguments for plots of type ``scatter`` and ``hexbin`` (`37001`)
- `DataFrame` now supports the ``divmod`` operation (`37165`)
- `DataFrame.to_parquet` now returns a ``bytes`` object when no ``path`` argument is passed (`37105`)

- `.Rolling` now supports the ``closed`` argument for fixed windows (`34315`)
- `.DatetimeIndex` and `Series` with ``datetime64`` or ``datetime64tz`` dtypes now support ``std`` (`37436`)
- `Window` now supports all Scipy window types in ``win_type`` with flexible keyword argument support (`34556`)
- `testing.assert_index_equal` now has a ``check_order`` parameter that allows indexes to be checked in an order-insensitive manner (`37478`)
- `read_csv` supports memory-mapping for compressed files (`37621`)
- Add support for ``min_count`` keyword for `DataFrame.groupby` and `DataFrame.resample` for functions ``min``, ``max``, ``first`` and ``last`` (`37821`, `37768`)
- Improve error reporting for `DataFrame.merge` when invalid merge column definitions were given (`16228`)
- Improve numerical stability for `.Rolling.skew`, `.Rolling.kurt`, `Expanding.skew` and `Expanding.kurt` through implementation of Kahan summation (`6929`)
- Improved error reporting for subsetting columns of a `.DataFrameGroupBy` with ``axis=1`` (`37725`)
- Implement method ``cross`` for `DataFrame.merge` and `DataFrame.join` (`5401`)
- When `read_csv`, `read_sas` and `read_json` are called with ``chunksize``/``iterator`` they can be used in a ``with`` statement as they return context-managers (`38225`)
- Augmented the list of named colors available for styling Excel exports, enabling all of CSS4 colors (`38247`)  .. ---  .. _whatsnew_120.notable_bug_fixes:  Notable bug fixes
~~~  These are bug fixes that might have notable behavior changes.  Consistency of DataFrame Reductions
^^^
`DataFrame.any` and `DataFrame.all` with ``bool_only=True`` now
determines whether to exclude object-dtype columns on a column-by-column basis,
instead of checking if *all* object-dtype columns can be considered boolean.  This prevents pathological behavior where applying the reduction on a subset
of columns could result in a larger Series result. See (`37799`).  .. ipython:: python  df = pd.DataFrame({"A": ["foo", "bar"], "B": [True, False]}, dtype=object)  df["C"] = pd.Series([True, True])  *Previous behavior*:  .. code-block:: ipython  In [5]: df.all(bool_only=True)  Out[5]:  C  True  dtype: bool  In [6]: df[["B", "C"]].all(bool_only=True)  Out[6]:  B  False  C  True  dtype: bool  *New behavior*:  .. ipython:: python  In [5]: df.all(bool_only=True)  In [6]: df[["B", "C"]].all(bool_only=True)  Other DataFrame reductions with ``numeric_only=None`` will also avoid
this pathological behavior (`37827`):  .. ipython:: python  df = pd.DataFrame({"A": [0, 1, 2], "B": ["a", "b", "c"]}, dtype=object)  *Previous behavior*:  .. code-block:: ipython  In [3]: df.mean()  Out[3]: Series([], dtype: float64)  In [4]: df[["A"]].mean()  Out[4]:  A  1.0  dtype: float64  *New behavior*:  .. code-block:: ipython  In [3]: df.mean()  Out[3]:  A  1.0  dtype: float64  In [4]: df[["A"]].mean()  Out[4]:  A  1.0  dtype: float64  Moreover, DataFrame reductions with ``numeric_only=None`` will now be
consistent with their Series counterparts.  In particular, for
reductions where the Series method raises ``TypeError``, the
DataFrame reduction will now consider that column non-numeric
instead of casting to a NumPy array which may have different semantics (`36076`,
`28949`, `21020`).  .. ipython:: python  ser = pd.Series([0, 1], dtype="category", name="A")  df = ser.to_frame()  *Previous behavior*:  .. code-block:: ipython  In [5]: df.any()  Out[5]:  A  True  dtype: bool  *New behavior*:  .. code-block:: ipython  In [5]: df.any()  Out[5]: Series([], dtype: bool)  .. _whatsnew_120.api_breaking.python:  Increased minimum version for Python
^^^  pandas 1.2.0 supports Python 3.7.1 and higher (`35214`).  .. _whatsnew_120.api_breaking.deps:  Increased minimum versions for dependencies
^^^  Some minimum supported versions of dependencies were updated (`35214`).

| numpy  | 1.16.5  |  X  |  X  |
| pytz  | 2017.3  |  X  |  X  |
| python-dateutil | 2.7.3  |  X  |  |
| numexpr  | 2.6.8  |  |  X  |
| pytest (dev)  | 5.0.1  |  |  X  |
| mypy (dev)  | 0.782  |  |  X  |
| gcsfs  | 0.6.0  |  |
| lxml  | 4.3.0  |  X  |
| matplotlib  | 2.2.3  |  X  |
| openpyxl  | 2.6.0  |  X  |
| pyarrow  | 0.15.0  |  X  |
| pymysql  | 0.7.11  |  X  |
| pytables  | 3.5.1  |  X  |
| s3fs  | 0.4.0  |  |
| scipy  | 1.2.0  |  |
| sqlalchemy  | 1.2.8  |  X  |
| xarray  | 0.12.3  |  X  |
| xlrd  | 1.2.0  |  X  |
| xlsxwriter  | 1.0.2  |  X  |
| xlwt  | 1.3.0  |  X  |
| pandas-gbq  | 0.12.0  |  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  .. _whatsnew_120.api.other:  Other API changes
^^^  - Sorting in descending order is now stable for `Series.sort_values` and `Index.sort_values` for Datetime-like `Index` subclasses. This will affect sort order when sorting a DataFrame on multiple columns, sorting with a key function that produces duplicates, or requesting the sorting index when using `Index.sort_values`. When using `Series.value_counts`, the count of missing values is no longer necessarily last in the list of duplicate counts. Instead, its position corresponds to the position in the original Series. When using `Index.sort_values` for Datetime-like `Index` subclasses, NaTs ignored the ``na_position`` argument and were sorted to the beginning. Now they respect ``na_position``, the default being ``last``, same as other `Index` subclasses (`35992`)
- Passing an invalid ``fill_value`` to `Categorical.take`, `.DatetimeArray.take`, `TimedeltaArray.take`, or `PeriodArray.take` now raises a ``TypeError`` instead of a ``ValueError`` (`37733`)
- Passing an invalid ``fill_value`` to `Series.shift` with a ``CategoricalDtype`` now raises a ``TypeError`` instead of a ``ValueError`` (`37733`)
- Passing an invalid value to `IntervalIndex.insert` or `CategoricalIndex.insert` now raises a ``TypeError`` instead of a ``ValueError`` (`37733`)
- Attempting to reindex a Series with a `CategoricalIndex` with an invalid ``fill_value`` now raises a ``TypeError`` instead of a ``ValueError`` (`37733`)
- `CategoricalIndex.append` with an index that contains non-category values will now cast instead of raising ``TypeError`` (`38098`)  .. ---  .. _whatsnew_120.deprecations:  Deprecations
~~~
- Deprecated parameter ``inplace`` in `MultiIndex.set_codes` and `MultiIndex.set_levels` (`35626`)
- Deprecated parameter ``dtype`` of method `~Index.copy` for all `Index` subclasses. Use the `~Index.astype` method instead for changing dtype (`35853`)
- Deprecated parameters ``levels`` and ``codes`` in `MultiIndex.copy`. Use the `~MultiIndex.set_levels` and `~MultiIndex.set_codes` methods instead (`36685`)

- Date parser functions `~pandas.io.date_converters.parse_date_time`, `~pandas.io.date_converters.parse_date_fields`, `~pandas.io.date_converters.parse_all_fields` and `~pandas.io.date_converters.generic_parser` from ``pandas.io.date_converters`` are deprecated and will be removed in a future version; use `to_datetime` instead (`35741`)
- `DataFrame.lookup` is deprecated and will be removed in a future version, use `DataFrame.melt` and `DataFrame.loc` instead (`35224`)
- The method `Index.to_native_types` is deprecated. Use ``.astype(str)`` instead (`28867`)
- Deprecated indexing `DataFrame` rows with a single datetime-like string as ``df[string]`` (given the ambiguity whether it is indexing the rows or selecting a column), use ``df.loc[string]`` instead (`36179`)
- Deprecated `Index.is_all_dates` (`27744`)
- The default value of ``regex`` for `Series.str.replace` will change from ``True`` to ``False`` in a future release. In addition, single character regular expressions will *not* be treated as literal strings when ``regex=True`` is set (`24804`)
- Deprecated automatic alignment on comparison operations between `DataFrame` and `Series`, do ``frame, ser = frame.align(ser, axis=1, copy=False)`` before e.g. ``frame == ser`` (`28759`)
- `Rolling.count` with ``min_periods=None`` will default to the size of the window in a future version (`31302`)
- Using "outer" ufuncs on DataFrames to return 4d ndarray is now deprecated. Convert to an ndarray first (`23743`)
- Deprecated slice-indexing on tz-aware `DatetimeIndex` with naive ``datetime`` objects, to match scalar indexing behavior (`36148`)
- `Index.ravel` returning a ``np.ndarray`` is deprecated, in the future this will return a view on the same index (`19956`)
- Deprecate use of strings denoting units with 'M', 'Y' or 'y' in `~pandas.to_timedelta` (`3666`)
- `Index` methods ``&``, ``|``, and ``^`` behaving as the set operations `Index.intersection`, `Index.union`, and `Index.symmetric_difference`, respectively, are deprecated and in the future will behave as pointwise boolean operations matching `Series` behavior.  Use the named set methods instead (`36758`)
- `Categorical.is_dtype_equal` and `CategoricalIndex.is_dtype_equal` are deprecated, will be removed in a future version (`37545`)
- `Series.slice_shift` and `DataFrame.slice_shift` are deprecated, use `Series.shift` or `DataFrame.shift` instead (`37601`)
- Partial slicing on unordered `.DatetimeIndex` objects with keys that are not in the index is deprecated and will be removed in a future version (`18531`)
- The ``how`` keyword in `PeriodIndex.astype` is deprecated and will be removed in a future version, use ``index.to_timestamp(how=how)`` instead (`37982`)
- Deprecated `Index.asi8` for `Index` subclasses other than `.DatetimeIndex`, `.TimedeltaIndex`, and `PeriodIndex` (`37877`)
- The ``inplace`` parameter of `Categorical.remove_unused_categories` is deprecated and will be removed in a future version (`37643`)
- The ``null_counts`` parameter of `DataFrame.info` is deprecated and replaced by ``show_counts``. It will be removed in a future version (`37999`)  **Calling NumPy ufuncs on non-aligned DataFrames**  Calling NumPy ufuncs on non-aligned DataFrames changed behaviour in pandas
1.2.0 (to align the inputs before calling the ufunc), but this change is
reverted in pandas 1.2.1. The behaviour to not align is now deprecated instead,
see the `the 1.2.1 release notes <whatsnew_121.ufunc_deprecation>` for
more details.  .. ---  .. _whatsnew_120.performance:  Performance improvements
~~~  - Performance improvements when creating DataFrame or Series with dtype ``str`` or `StringDtype` from array with many string elements (`36304`, `36317`, `36325`, `36432`, `37371`)
- Performance improvement in `.DataFrameGroupBy.agg` and `.SeriesGroupBy.agg` with the ``numba`` engine (`35759`)

- Performance improvements when creating `Series.map` from a huge dictionary (`34717`)
- Performance improvement in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` with the ``numba`` engine (`36240`)
- `.Styler` uuid method altered to compress data transmission over web whilst maintaining reasonably low table collision probability (`36345`)
- Performance improvement in `to_datetime` with non-ns time unit for ``float`` ``dtype`` columns (`20445`)
- Performance improvement in setting values on an `IntervalArray` (`36310`)
- The internal index method `~Index._shallow_copy` now makes the new index and original index share cached attributes, avoiding creating these again, if created on either. This can speed up operations that depend on creating copies of existing indexes (`36840`)
- Performance improvement in `.RollingGroupby.count` (`35625`)
- Small performance decrease to `.Rolling.min` and `.Rolling.max` for fixed windows (`36567`)
- Reduced peak memory usage in `DataFrame.to_pickle` when using ``protocol=5`` in python 3.8+ (`34244`)
- Faster ``dir`` calls when the object has many index labels, e.g. ``dir(ser)`` (`37450`)
- Performance improvement in `ExpandingGroupby` (`37064`)
- Performance improvement in `Series.astype` and `DataFrame.astype` for `Categorical` (`8628`)
- Performance improvement in `DataFrame.groupby` for ``float`` ``dtype`` (`28303`), changes of the underlying hash-function can lead to changes in float based indexes sort ordering for ties (e.g. `Index.value_counts`)
- Performance improvement in `pd.isin` for inputs with more than 1e6 elements (`36611`)
- Performance improvement for `DataFrame.__setitem__` with list-like indexers (`37954`)
- `read_json` now avoids reading entire file into memory when chunksize is specified (`34548`)  .. ---  .. _whatsnew_120.bug_fixes:  Bug fixes
^^^
- `Categorical.fillna` will always return a copy, validate a passed fill value regardless of whether there are any NAs to fill, and disallow an ``NaT`` as a fill value for numeric categories (`36530`)
- Bug in `Categorical.__setitem__` that incorrectly raised when trying to set a tuple value (`20439`)
- Bug in `CategoricalIndex.equals` incorrectly casting non-category entries to ``np.nan`` (`37667`)
- Bug in `CategoricalIndex.where` incorrectly setting non-category entries to ``np.nan`` instead of raising ``TypeError`` (`37977`)
- Bug in `Categorical.to_numpy` and ``np.array(categorical)`` with tz-aware ``datetime64`` categories incorrectly dropping the time zone information instead of casting to object dtype (`38136`)  Datetime-like
^^^
- Bug in `DataFrame.combine_first` that would convert datetime-like column on other `DataFrame` to integer when the column is not present in original `DataFrame` (`28481`)
- Bug in `.DatetimeArray.date` where a ``ValueError`` would be raised with a read-only backing array (`33530`)
- Bug in ``NaT`` comparisons failing to raise ``TypeError`` on invalid inequality comparisons (`35046`)
- Bug in `.DateOffset` where attributes reconstructed from pickle files differ from original objects when input values exceed normal ranges (e.g. months=12) (`34511`)
- Bug in `.DatetimeIndex.get_slice_bound` where ``datetime.date`` objects were not accepted or naive `Timestamp` with a tz-aware `.DatetimeIndex` (`35690`)
- Bug in `.DatetimeIndex.slice_locs` where ``datetime.date`` objects were not accepted (`34077`)
- Bug in `.DatetimeIndex.searchsorted`, `.TimedeltaIndex.searchsorted`, `PeriodIndex.searchsorted`, and `Series.searchsorted` with ``datetime64``, ``timedelta64`` or `Period` dtype placement of ``NaT`` values being inconsistent with NumPy (`36176`, `36254`)
- Inconsistency in `.DatetimeArray`, `.TimedeltaArray`, and `.PeriodArray` method ``__setitem__`` casting arrays of strings to datetime-like scalars but not scalar strings (`36261`)

- Bug in `.DatetimeArray.take` incorrectly allowing ``fill_value`` with a mismatched time zone (`37356`)
- Bug in `.DatetimeIndex.shift` incorrectly raising when shifting empty indexes (`14811`)
- `Timestamp` and `.DatetimeIndex` comparisons between tz-aware and tz-naive objects now follow the standard library ``datetime`` behavior, returning ``True``/``False`` for ``!=``/``==`` and raising for inequality comparisons (`28507`)
- Bug in `.DatetimeIndex.equals` and `.TimedeltaIndex.equals` incorrectly considering ``int64`` indexes as equal (`36744`)
- `Series.to_json`, `DataFrame.to_json`, and `read_json` now implement time zone parsing when orient structure is ``table`` (`35973`)
- `astype` now attempts to convert to ``datetime64[ns, tz]`` directly from ``object`` with inferred time zone from string (`35973`)
- Bug in `.TimedeltaIndex.sum` and `Series.sum` with ``timedelta64`` dtype on an empty index or series returning ``NaT`` instead of ``Timedelta(0)`` (`31751`)
- Bug in `.DatetimeArray.shift` incorrectly allowing ``fill_value`` with a mismatched time zone (`37299`)
- Bug in adding a `.BusinessDay` with nonzero ``offset`` to a non-scalar other (`37457`)
- Bug in `to_datetime` with a read-only array incorrectly raising (`34857`)
- Bug in `Series.isin` with ``datetime64[ns]`` dtype and `.DatetimeIndex.isin` incorrectly casting integers to datetimes (`36621`)
- Bug in `Series.isin` with ``datetime64[ns]`` dtype and `.DatetimeIndex.isin` failing to consider tz-aware and tz-naive datetimes as always different (`35728`)
- Bug in `Series.isin` with ``PeriodDtype`` dtype and `PeriodIndex.isin` failing to consider arguments with different ``PeriodDtype`` as always different (`37528`)
- Bug in `Period` constructor now correctly handles nanoseconds in the ``value`` argument (`34621` and `17053`)  Timedelta
^^^
- Bug in `.TimedeltaIndex`, `Series`, and `DataFrame` floor-division with ``timedelta64`` dtypes and ``NaT`` in the denominator (`35529`)
- Bug in parsing of ISO 8601 durations in `Timedelta` and `to_datetime` (`29773`, `36204`)
- Bug in `to_timedelta` with a read-only array incorrectly raising (`34857`)
- Bug in `Timedelta` incorrectly truncating to sub-second portion of a string input when it has precision higher than nanoseconds (`36738`)  Timezones
^^^  - Bug in `date_range` was raising ``AmbiguousTimeError`` for valid input with ``ambiguous=False`` (`35297`)
- Bug in `Timestamp.replace` was losing fold information (`37610`)  Numeric
^^^
- Bug in `to_numeric` where float precision was incorrect (`31364`)
- Bug in `DataFrame.any` with ``axis=1`` and ``bool_only=True`` ignoring the ``bool_only`` keyword (`32432`)
- Bug in `Series.equals` where a ``ValueError`` was raised when NumPy arrays were compared to scalars (`35267`)
- Bug in `Series` where two Series each have a `.DatetimeIndex` with different time zones having those indexes incorrectly changed when performing arithmetic operations (`33671`)
- Bug in `pandas.testing` module functions when used with ``check_exact=False`` on complex numeric types (`28235`)
- Bug in `DataFrame.__rmatmul__` error handling reporting transposed shapes (`21581`)
- Bug in `Series` flex arithmetic methods where the result when operating with a ``list``, ``tuple`` or ``np.ndarray`` would have an incorrect name (`36760`)
- Bug in `.IntegerArray` multiplication with ``timedelta`` and ``np.timedelta64`` objects (`36870`)
- Bug in `MultiIndex` comparison with tuple incorrectly treating tuple as array-like (`21517`)
- Bug in `DataFrame.diff` with ``datetime64`` dtypes including ``NaT`` values failing to fill ``NaT`` results correctly (`32441`)
- Bug in `DataFrame` arithmetic ops incorrectly accepting keyword arguments (`36843`)
- Bug in `.IntervalArray` comparisons with `Series` not returning Series (`36908`)

- Bug in `DataFrame` allowing arithmetic operations with list of array-likes with undefined results. Behavior changed to raising ``ValueError`` (`36702`)
- Bug in `DataFrame.std` with ``timedelta64`` dtype and ``skipna=False`` (`37392`)
- Bug in `DataFrame.min` and `DataFrame.max` with ``datetime64`` dtype and ``skipna=False`` (`36907`)
- Bug in `DataFrame.idxmax` and `DataFrame.idxmin` with mixed dtypes incorrectly raising ``TypeError`` (`38195`)  Conversion
^^^  - Bug in `DataFrame.to_dict` with ``orient='records'`` now returns python native datetime objects for datetime-like columns (`21256`)
- Bug in `Series.astype` conversion from ``string`` to ``float`` raised in presence of ``pd.NA`` values (`37626`)  Strings
^^^
- Bug in `Series.to_string`, `DataFrame.to_string`, and `DataFrame.to_latex` adding a leading space when ``index=False`` (`24980`)
- Bug in `to_numeric` raising a ``TypeError`` when attempting to convert a string dtype Series containing only numeric strings and ``NA`` (`37262`)  Interval
^^^  - Bug in `DataFrame.replace` and `Series.replace` where `Interval` dtypes would be converted to object dtypes (`34871`)
- Bug in `IntervalIndex.take` with negative indices and ``fill_value=None`` (`37330`)
- Bug in `IntervalIndex.putmask` with datetime-like dtype incorrectly casting to object dtype (`37968`)
- Bug in `IntervalArray.astype` incorrectly dropping dtype information with a `CategoricalDtype` object (`37984`)  Indexing
^^^  - Bug in `PeriodIndex.get_loc` incorrectly raising ``ValueError`` on non-datelike strings instead of ``KeyError``, causing similar errors in `Series.__getitem__`, `Series.__contains__`, and `Series.loc.__getitem__` (`34240`)
- Bug in `Index.sort_values` where, when empty values were passed, the method would break by trying to compare missing values instead of pushing them to the end of the sort order (`35584`)
- Bug in `Index.get_indexer` and `Index.get_indexer_non_unique` where ``int64`` arrays are returned instead of ``intp`` (`36359`)
- Bug in `DataFrame.sort_index` where parameter ascending passed as a list on a single level index gives wrong result (`32334`)
- Bug in `DataFrame.reset_index` was incorrectly raising a ``ValueError`` for input with a `MultiIndex` with missing values in a level with ``Categorical`` dtype (`24206`)
- Bug in indexing with boolean masks on datetime-like values sometimes returning a view instead of a copy (`36210`)
- Bug in `DataFrame.__getitem__` and `DataFrame.loc.__getitem__` with `IntervalIndex` columns and a numeric indexer (`26490`)
- Bug in `Series.loc.__getitem__` with a non-unique `MultiIndex` and an empty-list indexer (`13691`)
- Bug in indexing on a `Series` or `DataFrame` with a `MultiIndex` and a level named ``"0"`` (`37194`)
- Bug in `Series.__getitem__` when using an unsigned integer array as an indexer giving incorrect results or segfaulting instead of raising ``KeyError`` (`37218`)
- Bug in `Index.where` incorrectly casting numeric values to strings (`37591`)
- Bug in `DataFrame.loc` returning empty result when indexer is a slice with negative step size (`38071`)
- Bug in `Series.loc` and `DataFrame.loc` raises when the index was of ``object`` dtype and the given numeric label was in the index (`26491`)
- Bug in `DataFrame.loc` returned requested key plus missing values when ``loc`` was applied to single level from a `MultiIndex` (`27104`)
- Bug in indexing on a `Series` or `DataFrame` with a `CategoricalIndex` using a list-like indexer containing NA values (`37722`)
- Bug in `DataFrame.loc.__setitem__` expanding an empty `DataFrame` with mixed dtypes (`37932`)
- Bug in `DataFrame.xs` ignored ``droplevel=False`` for columns (`19056`)
- Bug in `DataFrame.reindex` raising ``IndexingError`` wrongly for empty DataFrame with ``tolerance`` not ``None`` or ``method="nearest"`` (`27315`)

- Bug in indexing on a `Series` or `DataFrame` with a `CategoricalIndex` using list-like indexer that contains elements that are in the index's ``categories`` but not in the index itself failing to raise ``KeyError`` (`37901`)
- Bug on inserting a boolean label into a `DataFrame` with a numeric `Index` columns incorrectly casting to integer (`36319`)
- Bug in `DataFrame.iloc` and `Series.iloc` aligning objects in ``__setitem__`` (`22046`)
- Bug in `MultiIndex.drop` does not raise if labels are partially found (`37820`)
- Bug in `DataFrame.loc` did not raise ``KeyError`` when missing combination was given with ``slice(None)`` for remaining levels (`19556`)
- Bug in `DataFrame.loc` raising ``TypeError`` when non-integer slice was given to select values from `MultiIndex` (`25165`, `24263`)
- Bug in `Series.at` returning `Series` with one element instead of scalar when index is a `MultiIndex` with one level (`38053`)
- Bug in `DataFrame.loc` returning and assigning elements in wrong order when indexer is differently ordered than the `MultiIndex` to filter (`31330`, `34603`)
- Bug in `DataFrame.loc` and `DataFrame.__getitem__`  raising ``KeyError`` when columns were `MultiIndex` with only one level (`29749`)
- Bug in `Series.__getitem__` and `DataFrame.__getitem__` raising blank ``KeyError`` without missing keys for `IntervalIndex` (`27365`)
- Bug in setting a new label on a `DataFrame` or `Series` with a `CategoricalIndex` incorrectly raising ``TypeError`` when the new label is not among the index's categories (`38098`)
- Bug in `Series.loc` and `Series.iloc` raising ``ValueError`` when inserting a list-like ``np.array``, ``list`` or ``tuple`` in an ``object`` Series of equal length (`37748`, `37486`)
- Bug in `Series.loc` and `Series.iloc` setting all the values of an ``object`` Series with those of a list-like ``ExtensionArray`` instead of inserting it (`38271`)  Missing
^^^  - Bug in `.SeriesGroupBy.transform` now correctly handles missing values for ``dropna=False`` (`35014`)
- Bug in `Series.nunique` with ``dropna=True`` was returning incorrect results when both ``NA`` and ``None`` missing values were present (`37566`)
- Bug in `Series.interpolate` where kwarg ``limit_area`` and ``limit_direction`` had no effect when using methods ``pad`` and ``backfill`` (`31048`)  MultiIndex
^^^  - Bug in `DataFrame.xs` when used with `IndexSlice` raises ``TypeError`` with message ``"Expected label or tuple of labels"`` (`35301`)
- Bug in `DataFrame.reset_index` with ``NaT`` values in index raises ``ValueError`` with message ``"cannot convert float NaN to integer"`` (`36541`)
- Bug in `DataFrame.combine_first` when used with `MultiIndex` containing string and ``NaN`` values raises ``TypeError`` (`36562`)
- Bug in `MultiIndex.drop` dropped ``NaN`` values when non existing key was given as input (`18853`)
- Bug in `MultiIndex.drop` dropping more values than expected when index has duplicates and is not sorted (`33494`)  I/O
^^^  - `read_sas` no longer leaks resources on failure (`35566`)
- Bug in `DataFrame.to_csv` and `Series.to_csv` caused a ``ValueError`` when it was called with a filename in combination with ``mode`` containing a ``b`` (`35058`)
- Bug in `read_csv` with ``float_precision='round_trip'`` did not handle ``decimal`` and ``thousands`` parameters (`35365`)
- `to_pickle` and `read_pickle` were closing user-provided file objects (`35679`)
- `to_csv` passes compression arguments for ``'gzip'`` always to ``gzip.GzipFile`` (`28103`)
- `to_csv` did not support zip compression for binary file object not having a filename (`35058`)
- `to_csv` and `read_csv` did not honor ``compression`` and ``encoding`` for path-like objects that are internally converted to file-like objects (`35677`, `26124`, `32392`)

- `DataFrame.to_pickle`, `Series.to_pickle`, and `read_pickle` did not support compression for file-objects (`26237`, `29054`, `29570`)
- Bug in `LongTableBuilder.middle_separator` was duplicating LaTeX longtable entries in the List of Tables of a LaTeX document (`34360`)
- Bug in `read_csv` with ``engine='python'`` truncating data if multiple items present in first row and first element started with BOM (`36343`)
- Removed ``private_key`` and ``verbose`` from `read_gbq` as they are no longer supported in ``pandas-gbq`` (`34654`, `30200`)
- Bumped minimum pytables version to 3.5.1 to avoid a ``ValueError`` in `read_hdf` (`24839`)
- Bug in `read_table` and `read_csv` when ``delim_whitespace=True`` and ``sep=default`` (`36583`)
- Bug in `DataFrame.to_json` and `Series.to_json` when used with ``lines=True`` and ``orient='records'`` the last line of the record is not appended with 'new line character' (`36888`)
- Bug in `read_parquet` with fixed offset time zones. String representation of time zones was not recognized (`35997`, `36004`)
- Bug in `DataFrame.to_html`, `DataFrame.to_string`, and `DataFrame.to_latex` ignoring the ``na_rep`` argument when ``float_format`` was also specified (`9046`, `13828`)
- Bug in output rendering of complex numbers showing too many trailing zeros (`36799`)
- Bug in `HDFStore` threw a ``TypeError`` when exporting an empty DataFrame with ``datetime64[ns, tz]`` dtypes with a fixed HDF5 store (`20594`)
- Bug in `HDFStore` was dropping time zone information when exporting a Series with ``datetime64[ns, tz]`` dtypes with a fixed HDF5 store (`20594`)
- `read_csv` was closing user-provided binary file handles when ``engine="c"`` and an ``encoding`` was requested (`36980`)
- Bug in `DataFrame.to_hdf` was not dropping missing rows with ``dropna=True`` (`35719`)
- Bug in `read_html` was raising a ``TypeError`` when supplying a ``pathlib.Path`` argument to the ``io`` parameter (`37705`)
- `DataFrame.to_excel`, `Series.to_excel`, `DataFrame.to_markdown`, and `Series.to_markdown` now support writing to fsspec URLs such as S3 and Google Cloud Storage (`33987`)
- Bug in `read_fwf` with ``skip_blank_lines=True`` was not skipping blank lines (`37758`)
- Parse missing values using `read_json` with ``dtype=False`` to ``NaN`` instead of ``None`` (`28501`)
- `read_fwf` was inferring compression with ``compression=None`` which was not consistent with the other ``read_*`` functions (`37909`)
- `DataFrame.to_html` was ignoring ``formatters`` argument for ``ExtensionDtype`` columns (`36525`)
- Bumped minimum xarray version to 0.12.3 to avoid reference to the removed ``Panel`` class (`27101`, `37983`)
- `DataFrame.to_csv` was re-opening file-like handles that also implement ``os.PathLike`` (`38125`)
- Bug in the conversion of a sliced ``pyarrow.Table`` with missing values to a DataFrame (`38525`)
- Bug in `read_sql_table` raising a ``sqlalchemy.exc.OperationalError`` when column names contained a percentage sign (`37517`)  Period
^^^  - Bug in `DataFrame.replace` and `Series.replace` where `Period` dtypes would be converted to object dtypes (`34871`)  Plotting
^^^  - Bug in `DataFrame.plot` was rotating xticklabels when ``subplots=True``, even if the x-axis wasn't an irregular time series (`29460`)
- Bug in `DataFrame.plot` where a marker letter in the ``style`` keyword sometimes caused a ``ValueError`` (`21003`)
- Bug in `DataFrame.plot.bar` and `Series.plot.bar` where ticks positions were assigned by value order instead of using the actual value for numeric or a smart ordering for string (`26186`, `11465`). This fix has been reverted in pandas 1.2.1, see `v1.2.1`
- Twinned axes were losing their tick labels which should only happen to all but the last row or column of 'externally' shared axes (`33819`)

- Bug in `Series.plot` and `DataFrame.plot` was throwing a `ValueError` when the Series or DataFrame was  indexed by a `.TimedeltaIndex` with a fixed frequency and the x-axis lower limit was greater than the upper limit (`37454`)
- Bug in `.DataFrameGroupBy.boxplot` when ``subplots=False`` would raise a ``KeyError`` (`16748`)
- Bug in `DataFrame.plot` and `Series.plot` was overwriting matplotlib's shared y axes behavior when no ``sharey`` parameter was passed (`37942`)
- Bug in `DataFrame.plot` was raising a ``TypeError`` with ``ExtensionDtype`` columns (`32073`)  Styler
^^^  - Bug in `Styler.render` HTML was generated incorrectly because of formatting error in ``rowspan`` attribute, it now matches with w3 syntax (`38234`)  Groupby/resample/rolling
^^^  - Bug in `.DataFrameGroupBy.count` and `SeriesGroupBy.sum` returning ``NaN`` for missing categories when grouped on multiple ``Categoricals``. Now returning ``0`` (`35028`)
- Bug in `.DataFrameGroupBy.apply` that would sometimes throw an erroneous ``ValueError`` if the grouping axis had duplicate entries (`16646`)
- Bug in `DataFrame.resample` that would throw a ``ValueError`` when resampling from ``"D"`` to ``"24H"`` over a transition into daylight savings time (DST) (`35219`)
- Bug when combining methods `DataFrame.groupby` with `DataFrame.resample` and `DataFrame.interpolate` raising a ``TypeError`` (`35325`)
- Bug in `.DataFrameGroupBy.apply` where a non-nuisance grouping column would be dropped from the output columns if another groupby method was called before ``.apply`` (`34656`)
- Bug when subsetting columns on a `~pandas.core.groupby.DataFrameGroupBy` (e.g. ``df.groupby('a')[['b']])``) would reset the attributes ``axis``, ``dropna``, ``group_keys``, ``level``, ``mutated``, ``sort``, and ``squeeze`` to their default values (`9959`)
- Bug in `.DataFrameGroupBy.tshift` failing to raise ``ValueError`` when a frequency cannot be inferred for the index of a group (`35937`)
- Bug in `DataFrame.groupby` does not always maintain column index name for ``any``, ``all``, ``bfill``, ``ffill``, ``shift`` (`29764`)
- Bug in `.DataFrameGroupBy.apply` raising error with ``np.nan`` group(s) when ``dropna=False`` (`35889`)
- Bug in `.Rolling.sum` returned wrong values when dtypes where mixed between float and integer and ``axis=1`` (`20649`, `35596`)
- Bug in `.Rolling.count` returned ``np.nan`` with `~pandas.api.indexers.FixedForwardWindowIndexer` as window, ``min_periods=0`` and only missing values in the window (`35579`)
- Bug where `pandas.core.window.Rolling` produces incorrect window sizes when using a ``PeriodIndex`` (`34225`)
- Bug in `.DataFrameGroupBy.ffill` and `.DataFrameGroupBy.bfill` where a ``NaN`` group would return filled values instead of ``NaN`` when ``dropna=True`` (`34725`)
- Bug in `.RollingGroupby.count` where a ``ValueError`` was raised when specifying the ``closed`` parameter (`35869`)
- Bug in `.DataFrameGroupBy.rolling` returning wrong values with partial centered window (`36040`)
- Bug in `.DataFrameGroupBy.rolling` returned wrong values with time aware window containing ``NaN``. Raises ``ValueError`` because windows are not monotonic now (`34617`)
- Bug in `.Rolling.__iter__` where a ``ValueError`` was not raised when ``min_periods`` was larger than ``window`` (`37156`)
- Using `.Rolling.var` instead of `.Rolling.std` avoids numerical issues for `.Rolling.corr` when `.Rolling.var` is still within floating point precision while `.Rolling.std` is not (`31286`)
- Bug in `.DataFrameGroupBy.quantile` and `.Resampler.quantile` raised ``TypeError`` when values were of type ``Timedelta`` (`29485`)
- Bug in `.Rolling.median` and `.Rolling.quantile` returned wrong values for `.BaseIndexer` subclasses with non-monotonic starting or ending points for windows (`37153`)

- Bug in `DataFrame.groupby` dropped ``nan`` groups from result with ``dropna=False`` when grouping over a single column (`35646`, `35542`)
- Bug in `.DataFrameGroupBy.head`, `DataFrameGroupBy.tail`, `SeriesGroupBy.head`, and `SeriesGroupBy.tail` would raise when used with ``axis=1`` (`9772`)
- Bug in `.DataFrameGroupBy.transform` would raise when used with ``axis=1`` and a transformation kernel (e.g. "shift") (`36308`)
- Bug in `.DataFrameGroupBy.resample` using ``.agg`` with sum produced different result than just calling ``.sum`` (`33548`)
- Bug in `.DataFrameGroupBy.apply` dropped values on ``nan`` group when returning the same axes with the original frame (`38227`)
- Bug in `.DataFrameGroupBy.quantile` couldn't handle with arraylike ``q`` when grouping by columns (`33795`)
- Bug in `DataFrameGroupBy.rank` with ``datetime64tz`` or period dtype incorrectly casting results to those dtypes instead of returning ``float64`` dtype (`38187`)  Reshaping
^^^  - Bug in `DataFrame.crosstab` was returning incorrect results on inputs with duplicate row names, duplicate column names or duplicate names between row and column labels (`22529`)
- Bug in `DataFrame.pivot_table` with ``aggfunc='count'`` or ``aggfunc='sum'`` returning ``NaN`` for missing categories when pivoted on a ``Categorical``. Now returning ``0`` (`31422`)
- Bug in `concat` and `DataFrame` constructor where input index names are not preserved in some cases (`13475`)
- Bug in func `crosstab` when using multiple columns with ``margins=True`` and ``normalize=True`` (`35144`)
- Bug in `DataFrame.stack` where an empty DataFrame.stack would raise an error (`36113`). Now returning an empty Series with empty MultiIndex.
- Bug in `Series.unstack`. Now a Series with single level of Index trying to unstack would raise a ``ValueError`` (`36113`)
- Bug in `DataFrame.agg` with ``func={'name':<FUNC>}`` incorrectly raising ``TypeError`` when ``DataFrame.columns==['Name']`` (`36212`)
- Bug in `Series.transform` would give incorrect results or raise when the argument ``func`` was a dictionary (`35811`)
- Bug in `DataFrame.pivot` did not preserve `MultiIndex` level names for columns when rows and columns are both multiindexed (`36360`)
- Bug in `DataFrame.pivot` modified ``index`` argument when ``columns`` was passed but ``values`` was not (`37635`)
- Bug in `DataFrame.join` returned a non deterministic level-order for the resulting `MultiIndex` (`36910`)
- Bug in `DataFrame.combine_first` caused wrong alignment with dtype ``string`` and one level of ``MultiIndex`` containing only ``NA`` (`37591`)
- Fixed regression in `merge` on merging `.DatetimeIndex` with empty DataFrame (`36895`)
- Bug in `DataFrame.apply` not setting index of return value when ``func`` return type is ``dict`` (`37544`)
- Bug in `DataFrame.merge` and `pandas.merge` returning inconsistent ordering in result for ``how=right`` and ``how=left`` (`35382`)
- Bug in `merge_ordered` couldn't handle list-like ``left_by`` or ``right_by`` (`35269`)
- Bug in `merge_ordered` returned wrong join result when length of ``left_by`` or ``right_by`` equals to the rows of ``left`` or ``right`` (`38166`)
- Bug in `merge_ordered` didn't raise when elements in ``left_by`` or ``right_by`` not exist in ``left`` columns or ``right`` columns (`38167`)
- Bug in `DataFrame.drop_duplicates` not validating bool dtype for ``ignore_index`` keyword (`38274`)  ExtensionArray
^^^  - Fixed bug where `DataFrame` column set to scalar extension type via a dict instantiation was considered an object type rather than the extension type (`35965`)
- Fixed bug where ``astype()`` with equal dtype and ``copy=False`` would return a new object (`28488`)
- Fixed bug when applying a NumPy ufunc with multiple outputs to an `.IntegerArray` returning ``None`` (`36913`)

- Fixed an inconsistency in `.PeriodArray`'s ``__init__`` signature to those of `.DatetimeArray` and `.TimedeltaArray` (`37289`)
- Reductions for `.BooleanArray`, `.Categorical`, `.DatetimeArray`, `.FloatingArray`, `.IntegerArray`, `.PeriodArray`, `.TimedeltaArray`, and `.PandasArray` are now keyword-only methods (`37541`)
- Fixed a bug where a  ``TypeError`` was wrongly raised if a membership check was made on an ``ExtensionArray`` containing nan-like values (`37867`)  Other
^^^  - Bug in `DataFrame.replace` and `Series.replace` incorrectly raising an ``AssertionError`` instead of a ``ValueError`` when invalid parameter combinations are passed (`36045`)
- Bug in `DataFrame.replace` and `Series.replace` with numeric values and string ``to_replace`` (`34789`)
- Fixed metadata propagation in `Series.abs` and ufuncs called on Series and DataFrames (`28283`)
- Bug in `DataFrame.replace` and `Series.replace` incorrectly casting from ``PeriodDtype`` to object dtype (`34871`)
- Fixed bug in metadata propagation incorrectly copying DataFrame columns as metadata when the column name overlaps with the metadata name (`37037`)
- Fixed metadata propagation in the `Series.dt`, `Series.str` accessors, `DataFrame.duplicated`, `DataFrame.stack`, `DataFrame.unstack`, `DataFrame.pivot`, `DataFrame.append`, `DataFrame.diff`, `DataFrame.applymap` and `DataFrame.update` methods (`28283`, `37381`)
- Fixed metadata propagation when selecting columns with ``DataFrame.__getitem__`` (`28283`)
- Bug in `Index.intersection` with non-`Index` failing to set the correct name on the returned `Index` (`38111`)
- Bug in `RangeIndex.intersection` failing to set the correct name on the returned `Index` in some corner cases (`38197`)
- Bug in `Index.difference` failing to set the correct name on the returned `Index` in some corner cases (`38268`)
- Bug in `Index.union` behaving differently depending on whether operand is an `Index` or other list-like (`36384`)
- Bug in `Index.intersection` with non-matching numeric dtypes casting to ``object`` dtype instead of minimal common dtype (`38122`)
- Bug in `IntervalIndex.union` returning an incorrectly-typed `Index` when empty (`38282`)
- Passing an array with 2 or more dimensions to the `Series` constructor now raises the more specific ``ValueError`` rather than a bare ``Exception`` (`35744`)
- Bug in ``dir`` where ``dir(obj)`` wouldn't show attributes defined on the instance for pandas objects (`37173`)
- Bug in `Index.drop` raising ``InvalidIndexError`` when index has duplicates (`38051`)
- Bug in `RangeIndex.difference` returning `Int64Index` in some cases where it should return `RangeIndex` (`38028`)
- Fixed bug in `assert_series_equal` when comparing a datetime-like array with an equivalent non extension dtype array (`37609`)
- Bug in `.is_bool_dtype` would raise when passed a valid string such as ``"boolean"`` (`38386`)
- Fixed regression in logical operators raising ``ValueError`` when columns of `DataFrame` are a `CategoricalIndex` with unused categories (`38367`)  .. ---  .. _whatsnew_120.contributors:  Contributors
~~~  .. contributors:: v1.1.5..v1.2.0

These are the changes in pandas 1.2.1. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_121.regressions:  Fixed regressions
~~~
- Fixed regression in `~DataFrame.to_csv` that created corrupted zip files when there were more rows than ``chunksize`` (`38714`)
- Fixed regression in `~DataFrame.to_csv` opening ``codecs.StreamReaderWriter`` in binary mode instead of in text mode (`39247`)
- Fixed regression in `read_csv` and other read functions were the encoding error policy (``errors``) did not default to ``"replace"`` when no encoding was specified (`38989`)
- Fixed regression in `read_excel` with non-rawbyte file handles (`38788`)
- Fixed regression in `DataFrame.to_stata` not removing the created file when an error occurred (`39202`)
- Fixed regression in ``DataFrame.__setitem__`` raising ``ValueError`` when expanding `DataFrame` and new column is from type ``"0 - name"`` (`39010`)
- Fixed regression in setting with `DataFrame.loc`  raising ``ValueError`` when `DataFrame` has unsorted `MultiIndex` columns and indexer is a scalar (`38601`)
- Fixed regression in setting with `DataFrame.loc` raising ``KeyError`` with `MultiIndex` and list-like columns indexer enlarging `DataFrame` (`39147`)
- Fixed regression in `~DataFrame.groupby()` with `Categorical` grouping column not showing unused categories for ``grouped.indices`` (`38642`)
- Fixed regression in `.DataFrameGroupBy.sem` and `.SeriesGroupBy.sem` where the presence of non-numeric columns would cause an error instead of being dropped (`38774`)
- Fixed regression in `.DataFrameGroupBy.diff` raising for ``int8`` and ``int16`` columns (`39050`)
- Fixed regression in `DataFrame.groupby` when aggregating an ``ExtensionDType`` that could fail for non-numeric values (`38980`)
- Fixed regression in `.Rolling.skew` and `.Rolling.kurt` modifying the object inplace (`38908`)
- Fixed regression in `DataFrame.any` and `DataFrame.all` not returning a result for tz-aware ``datetime64`` columns (`38723`)
- Fixed regression in `DataFrame.apply` with ``axis=1`` using str accessor in apply function (`38979`)
- Fixed regression in `DataFrame.replace` raising ``ValueError`` when `DataFrame` has dtype ``bytes`` (`38900`)
- Fixed regression in `Series.fillna` that raised ``RecursionError`` with ``datetime64[ns, UTC]`` dtype (`38851`)
- Fixed regression in comparisons between ``NaT`` and ``datetime.date`` objects incorrectly returning ``True`` (`39151`)
- Fixed regression in calling NumPy `~numpy.ufunc.accumulate` ufuncs on DataFrames, e.g. ``np.maximum.accumulate(df)`` (`39259`)
- Fixed regression in repr of float-like strings of an ``object`` dtype having trailing 0's truncated after the decimal (`38708`)
- Fixed regression that raised ``AttributeError`` with PyArrow versions [0.16.0, 1.0.0) (`38801`)
- Fixed regression in `pandas.testing.assert_frame_equal` raising ``TypeError`` with ``check_like=True`` when `Index` or columns have mixed dtype (`39168`)  We have reverted a commit that resulted in several plotting related regressions in pandas 1.2.0 (`38969`, `38736`, `38865`, `38947` and `39126`).
As a result, bugs reported as fixed in pandas 1.2.0 related to inconsistent tick labeling in bar plots are again present (`26186` and `11465`)  .. ---  .. _whatsnew_121.ufunc_deprecation:  Calling NumPy ufuncs on non-aligned DataFrames
~~~  Before pandas 1.2.0, calling a NumPy ufunc on non-aligned DataFrames (or
DataFrame / Series combination) would ignore the indices, only match
the inputs by shape, and use the index/columns of the first DataFrame for
the result:  .. code-block:: ipython  In [1]: df1 = pd.DataFrame({"a": [1, 2], "b": [3, 4]}, index=[0, 1])  In [2]: df2 = pd.DataFrame({"a": [1, 2], "b": [3, 4]}, index=[1, 2])  In [3]: df1  Out[3]:  a  b  0  1  3  1  2  4  In [4]: df2  Out[4]:  a  b  1  1  3  2  2  4  In [5]: np.add(df1, df2)  Out[5]:  a  b  0  2  6  1  4  8  This contrasts with how other pandas operations work, which first align

the inputs:  .. code-block:: ipython  In [6]: df1 + df2  Out[6]:  a  b  0  NaN  NaN  1  3.0  7.0  2  NaN  NaN  In pandas 1.2.0, we refactored how NumPy ufuncs are called on DataFrames, and
this started to align the inputs first (`39184`), as happens in other
pandas operations and as it happens for ufuncs called on Series objects.  For pandas 1.2.1, we restored the previous behaviour to avoid a breaking
change, but the above example of ``np.add(df1, df2)`` with non-aligned inputs
will now to raise a warning, and a future pandas 2.0 release will start
aligning the inputs first (`39184`). Calling a NumPy ufunc on Series
objects (eg ``np.add(s1, s2)``) already aligns and continues to do so.  To avoid the warning and keep the current behaviour of ignoring the indices,
convert one of the arguments to a NumPy array:  .. code-block:: ipython  In [7]: np.add(df1, np.asarray(df2))  Out[7]:  a  b  0  2  6  1  4  8  To obtain the future behaviour and silence the warning, you can align manually
before passing the arguments to the ufunc:  .. code-block:: ipython  In [8]: df1, df2 = df1.align(df2)  In [9]: np.add(df1, df2)  Out[9]:  a  b  0  NaN  NaN  1  3.0  7.0  2  NaN  NaN  .. ---  .. _whatsnew_121.bug_fixes:  Bug fixes
~~~  - Bug in `read_csv` with ``float_precision="high"`` caused segfault or wrong parsing of long exponent strings. This resulted in a regression in some cases as the default for ``float_precision`` was changed in pandas 1.2.0 (`38753`)
- Bug in `read_csv` not closing an opened file handle when a ``csv.Error`` or ``UnicodeDecodeError`` occurred while initializing (`39024`)
- Bug in `pandas.testing.assert_index_equal` raising ``TypeError`` with ``check_order=False`` when `Index` has mixed dtype (`39168`)  .. ---  .. _whatsnew_121.other:  Other
~~~  - The deprecated attributes ``_AXIS_NAMES`` and ``_AXIS_NUMBERS`` of `DataFrame` and `Series` will no longer show up in ``dir`` or ``inspect.getmembers`` calls (`38740`)
- Bumped minimum fastparquet version to 0.4.0 to avoid ``AttributeError`` from numba (`38344`)
- Bumped minimum pymysql version to 0.8.1 to avoid test failures (`38344`)
- Fixed build failure on MacOS 11 in Python 3.9.1 (`38766`)
- Added reference to backwards incompatible ``check_freq`` arg of `testing.assert_frame_equal` and `testing.assert_series_equal` in `pandas 1.1.0 what's new <whatsnew_110.api_breaking.testing.check_freq>` (`34050`)  .. ---  .. _whatsnew_121.contributors:  Contributors
~~~  .. contributors:: v1.2.0..v1.2.1

What's new in 1.2.2 (February 09, 2021)

These are the changes in pandas 1.2.2. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_122.regressions:  Fixed regressions
~~~  - Fixed regression in `read_excel` that caused it to raise ``AttributeError`` when checking version of older xlrd versions (`38955`)
- Fixed regression in `DataFrame` constructor reordering element when construction from datetime ndarray with dtype not ``"datetime64[ns]"`` (`39422`)
- Fixed regression in `DataFrame.astype` and `Series.astype` not casting to bytes dtype (`39474`)
- Fixed regression in `~DataFrame.to_pickle` failing to create bz2/xz compressed pickle files with ``protocol=5`` (`39002`)
- Fixed regression in `pandas.testing.assert_series_equal` and `pandas.testing.assert_frame_equal` always raising ``AssertionError`` when comparing extension dtypes (`39410`)
- Fixed regression in `~DataFrame.to_csv` opening ``codecs.StreamWriter`` in binary mode instead of in text mode and ignoring user-provided ``mode`` (`39247`)
- Fixed regression in `Categorical.astype` casting to incorrect dtype when ``np.int32`` is passed to dtype argument (`39402`)
- Fixed regression in `~DataFrame.to_excel` creating corrupt files when appending (``mode="a"``) to an existing file (`39576`)
- Fixed regression in `DataFrame.transform` failing in case of an empty DataFrame or Series (`39636`)
- Fixed regression in `~DataFrame.groupby` or `~DataFrame.resample` when aggregating an all-NaN or numeric object dtype column (`39329`)
- Fixed regression in `.Rolling.count` where the ``min_periods`` argument would be set to ``0`` after the operation (`39554`)
- Fixed regression in `read_excel` that incorrectly raised when the argument ``io`` was a non-path and non-buffer and the ``engine`` argument was specified (`39528`)  .. ---  .. _whatsnew_122.bug_fixes:  Bug fixes
~~~  - `pandas.read_excel` error message when a specified ``sheetname`` does not exist is now uniform across engines (`39250`)
- Fixed bug in `pandas.read_excel` producing incorrect results when the engine ``openpyxl`` is used and the excel file is missing or has incorrect dimension information; the fix requires ``openpyxl`` >= 3.0.0, prior versions may still fail (`38956`, `39001`)
- Fixed bug in `pandas.read_excel` sometimes producing a ``DataFrame`` with trailing rows of ``np.nan`` when the engine ``openpyxl`` is used (`39181`)  .. ---  .. _whatsnew_122.contributors:  Contributors
~~~  .. contributors:: v1.2.1..v1.2.2

What's new in 1.2.3 (March 02, 2021)

These are the changes in pandas 1.2.3. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_123.regressions:  Fixed regressions
~~~  - Fixed regression in `~DataFrame.to_excel` raising ``KeyError`` when giving duplicate columns with ``columns`` attribute (`39695`)
- Fixed regression in nullable integer unary ops propagating mask on assignment (`39943`)
- Fixed regression in `DataFrame.__setitem__` not aligning `DataFrame` on right-hand side for boolean indexer (`39931`)
- Fixed regression in `~DataFrame.to_json` failing to use ``compression`` with URL-like paths that are internally opened in binary mode or with user-provided file objects that are opened in binary mode (`39985`)
- Fixed regression in `Series.sort_index` and `DataFrame.sort_index`, which exited with an ungraceful error when having kwarg ``ascending=None`` passed. Passing ``ascending=None`` is still considered invalid, and the improved error message suggests a proper usage (``ascending`` must be a boolean or a list-like of boolean) (`39434`)
- Fixed regression in `DataFrame.transform` and `Series.transform` giving incorrect column labels when passed a dictionary with a mix of list and non-list values (`40018`)  .. ---  .. _whatsnew_123.contributors:  Contributors
~~~  .. contributors:: v1.2.2..v1.2.3

What's new in 1.2.4 (April 12, 2021)

These are the changes in pandas 1.2.4. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_124.regressions:  Fixed regressions
~~~  - Fixed regression in `DataFrame.sum` when ``min_count`` greater than the `DataFrame` shape was passed resulted in a ``ValueError`` (`39738`)
- Fixed regression in `DataFrame.to_json` raising ``AttributeError`` when run on PyPy (`39837`)
- Fixed regression in (in)equality comparison of ``pd.NaT`` with a non-datetimelike numpy array returning a scalar instead of an array (`40722`)
- Fixed regression in `DataFrame.where` not returning a copy in the case of an all True condition (`39595`)
- Fixed regression in `DataFrame.replace` raising ``IndexError`` when ``regex`` was a multi-key dictionary (`39338`)
- Fixed regression in repr of floats in an ``object`` column not respecting ``float_format`` when printed in the console or outputted through `DataFrame.to_string`, `DataFrame.to_html`, and `DataFrame.to_latex` (`40024`)
- Fixed regression in NumPy ufuncs such as ``np.add`` not passing through all arguments for `DataFrame` (`40662`)  .. ---  .. _whatsnew_124.contributors:  Contributors
~~~  .. contributors:: v1.2.3..v1.2.4

What's new in 1.2.5 (June 22, 2021)

These are the changes in pandas 1.2.5. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_125.regressions:  Fixed regressions
~~~
- Fixed regression in `concat` between two `DataFrame` where one has an `Index` that is all-None and the other is `DatetimeIndex` incorrectly raising (`40841`)
- Fixed regression in `DataFrame.sum` and `DataFrame.prod` when ``min_count`` and ``numeric_only`` are both given (`41074`)
- Fixed regression in `read_csv` when using ``memory_map=True`` with an non-UTF8 encoding (`40986`)
- Fixed regression in `DataFrame.replace` and `Series.replace` when the values to replace is a NumPy float array (`40371`)
- Fixed regression in `ExcelFile` when a corrupt file is opened but not closed (`41778`)
- Fixed regression in `DataFrame.astype` with ``dtype=str`` failing to convert ``NaN`` in categorical columns (`41797`)  .. ---  .. _whatsnew_125.contributors:  Contributors
~~~  .. contributors:: v1.2.4..v1.2.5|HEAD

These are the changes in pandas 1.3.0. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. warning::  When reading new Excel 2007+ (``.xlsx``) files, the default argument  ``engine=None`` to `read_excel` will now result in using the  openpyxl engine in all cases  when the option `io.excel.xlsx.reader` is set to ``"auto"``.  Previously, some cases would use the  xlrd engine instead. See  `What's new 1.2.0 <whatsnew_120>` for background on this change.  .. ---  .. _whatsnew_130.enhancements:  Enhancements
~~~  .. _whatsnew_130.enhancements.read_csv_json_http_headers:  Custom HTTP(s) headers when reading csv or json files
^^^  When reading from a remote URL that is not handled by fsspec (e.g. HTTP and
HTTPS) the dictionary passed to ``storage_options`` will be used to create the
headers included in the request.  This can be used to control the User-Agent
header or send other custom headers (`36688`).
For example:  .. ipython:: python  headers = {"User-Agent": "pandas"}  df = pd.read_csv(  "https://download.bls.gov/pub/time.series/cu/cu.item",  sep="\t",  storage_options=headers  )  .. _whatsnew_130.enhancements.read_to_xml:  Read and write XML documents
^^^  We added I/O support to read and render shallow versions of `XML`_ documents with
`read_xml` and `DataFrame.to_xml`. Using `lxml`_ as parser,
both XPath 1.0 and XSLT 1.0 are available. (`27554`)  .. _XML: https://www.w3.org/standards/xml/core
.. _lxml: https://lxml.de  .. code-block:: ipython  In [1]: xml = """<?xml version='1.0' encoding='utf-8'?>  ...: <data>  ...:  <row>  ...:  <shape>square</shape>  ...:  <degrees>360</degrees>  ...:  <sides>4.0</sides>  ...:  </row>  ...:  <row>  ...:  <shape>circle</shape>  ...:  <degrees>360</degrees>  ...:  <sides/>  ...:  </row>  ...:  <row>  ...:  <shape>triangle</shape>  ...:  <degrees>180</degrees>  ...:  <sides>3.0</sides>  ...:  </row>  ...:  </data>"""  In [2]: df = pd.read_xml(xml)  In [3]: df  Out[3]:  shape  degrees  sides  0  square  360  4.0  1  circle  360  NaN  2  triangle  180  3.0  In [4]: df.to_xml()  Out[4]:  <?xml version='1.0' encoding='utf-8'?>  <data>  <row>  <index>0</index>  <shape>square</shape>  <degrees>360</degrees>  <sides>4.0</sides>  </row>  <row>  <index>1</index>  <shape>circle</shape>  <degrees>360</degrees>  <sides/>  </row>  <row>  <index>2</index>  <shape>triangle</shape>  <degrees>180</degrees>  <sides>3.0</sides>  </row>  </data>  For more, see `io.xml` in the user guide on IO tools.  .. _whatsnew_130.enhancements.styler:  Styler enhancements
^^^  We provided some focused development on .Styler`. See also the `Styler documentation

which has been revised and improved (`39720`, `39317`, `40493`).  - The method `.Styler.set_table_styles` can now accept more natural CSS language for arguments, such as ``'color:red;'`` instead of ``[('color', 'red')]`` (`39563`)  - The methods `.Styler.highlight_null`, `.Styler.highlight_min`, and `.Styler.highlight_max` now allow custom CSS highlighting instead of the default background coloring (`40242`)  - `.Styler.apply` now accepts functions that return an ``ndarray`` when ``axis=None``, making it now consistent with the ``axis=0`` and ``axis=1`` behavior (`39359`)  - When incorrectly formatted CSS is given via `.Styler.apply` or `.Styler.applymap`, an error is now raised upon rendering (`39660`)  - `.Styler.format` now accepts the keyword argument ``escape`` for optional HTML and LaTeX escaping (`40388`, `41619`)  - `.Styler.background_gradient` has gained the argument ``gmap`` to supply a specific gradient map for shading (`22727`)  - `.Styler.clear` now clears `Styler.hidden_index` and `Styler.hidden_columns` as well (`40484`)  - Added the method `.Styler.highlight_between` (`39821`)  - Added the method `.Styler.highlight_quantile` (`40926`)  - Added the method `.Styler.text_gradient` (`41098`)  - Added the method `.Styler.set_tooltips` to allow hover tooltips; this can be used enhance interactive displays (`21266`, `40284`)  - Added the parameter ``precision`` to the method `.Styler.format` to control the display of floating point numbers (`40134`)  - .Styler` rendered HTML output now follows the `w3 HTML Style Guide (`39626`)  - Many features of the `.Styler` class are now either partially or fully usable on a DataFrame with a non-unique indexes or columns (`41143`)  - One has greater control of the display through separate sparsification of the index or columns using the `new styler options <options.available>`, which are also usable via `option_context` (`41142`)  - Added the option ``styler.render.max_elements`` to avoid browser overload when styling large DataFrames (`40712`)  - Added the method `.Styler.to_latex` (`21673`, `42320`), which also allows some limited CSS conversion (`40731`)  - Added the method `.Styler.to_html` (`13379`)  - Added the method `.Styler.set_sticky` to make index and column headers permanently visible in scrolling HTML frames (`29072`)  .. _whatsnew_130.enhancements.dataframe_honors_copy_with_dict:  DataFrame constructor honors ``copy=False`` with dict
^^^  When passing a dictionary to `DataFrame` with ``copy=False``,
a copy will no longer be made (`32960`).  .. ipython:: python  arr = np.array([1, 2, 3])  df = pd.DataFrame({"A": arr, "B": arr.copy()}, copy=False)  df  ``df["A"]`` remains a view on ``arr``:  .. ipython:: python  arr[0] = 0  assert df.iloc[0, 0] == 0  The default behavior when not passing ``copy`` will remain unchanged, i.e.
a copy will be made.  .. _whatsnew_130.enhancements.arrow_string:  PyArrow backed string data type
^^^  We've enhanced the `StringDtype`, an extension type dedicated to string data.
(`39908`)  It is now possible to specify a ``storage`` keyword option to `StringDtype`. Use
pandas options or specify the dtype using ``dtype='string[pyarrow]'`` to allow the

StringArray to be backed by a PyArrow array instead of a NumPy array of Python objects.  The PyArrow backed StringArray requires pyarrow 1.0.0 or greater to be installed.  .. warning::  ``string[pyarrow]`` is currently considered experimental. The implementation  and parts of the API may change without warning.  .. ipython:: python  pd.Series(['abc', None, 'def'], dtype=pd.StringDtype(storage="pyarrow"))  You can use the alias ``"string[pyarrow]"`` as well.  .. ipython:: python  s = pd.Series(['abc', None, 'def'], dtype="string[pyarrow]")  s  You can also create a PyArrow backed string array using pandas options.  .. ipython:: python  with pd.option_context("string_storage", "pyarrow"):  s = pd.Series(['abc', None, 'def'], dtype="string")  s  The usual string accessor methods work. Where appropriate, the return type of the Series
or columns of a DataFrame will also have string dtype.  .. ipython:: python  s.str.upper()  s.str.split('b', expand=True).dtypes  String accessor methods returning integers will return a value with `Int64Dtype`  .. ipython:: python  s.str.count("a")  .. _whatsnew_130.enhancements.centered_datetimelike_rolling_window:  Centered datetime-like rolling windows
^^^  When performing rolling calculations on DataFrame and Series
objects with a datetime-like index, a centered datetime-like window can now be
used (`38780`).
For example:  .. ipython:: python  df = pd.DataFrame(  {"A": [0, 1, 2, 3, 4]}, index=pd.date_range("2020", periods=5, freq="1D")  )  df  df.rolling("2D", center=True).mean()  .. _whatsnew_130.enhancements.other:  Other enhancements
^^^  - `DataFrame.rolling`, `Series.rolling`, `DataFrame.expanding`, and `Series.expanding` now support a ``method`` argument with a ``'table'`` option that performs the windowing operation over an entire `DataFrame`. See `Window Overview <window.overview>` for performance and functional benefits (`15095`, `38995`)
- `.ExponentialMovingWindow` now support a ``online`` method that can perform ``mean`` calculations in an online fashion. See `Window Overview <window.overview>` (`41673`)
- Added `MultiIndex.dtypes` (`37062`)
- Added ``end`` and ``end_day`` options for the ``origin`` argument in `DataFrame.resample` (`37804`)
- Improved error message when ``usecols`` and ``names`` do not match for `read_csv` and ``engine="c"`` (`29042`)
- Improved consistency of error messages when passing an invalid ``win_type`` argument in `Window methods <api.window>` (`15969`)
- `read_sql_query` now accepts a ``dtype`` argument to cast the columnar data from the SQL database based on user input (`10285`)
- `read_csv` now raising ``ParserWarning`` if length of header or given names does not match length of data when ``usecols`` is not specified (`21768`)
- Improved integer type mapping from pandas to SQLAlchemy when using `DataFrame.to_sql` (`35076`)
- `to_numeric` now supports downcasting of nullable ``ExtensionDtype`` objects (`33013`)
- Added support for dict-like names in `MultiIndex.set_names` and `MultiIndex.rename` (`20421`)
- `read_excel` can now auto-detect .xlsb files and older .xls files (`35416`, `41225`)
- `ExcelWriter` now accepts an ``if_sheet_exists`` parameter to control the behavior of append mode when writing to existing sheets (`40230`)
- .Rolling.sum`, `.Expanding.sum`, `.Rolling.mean`, `.Expanding.mean`, `.ExponentialMovingWindow.mean`, `.Rolling.median`, `.Expanding.median`, `.Rolling.max`, `.Expanding.max`, `.Rolling.min`, and `.Expanding.min` now support `Numba execution with the ``engine`` keyword (`38895`, `41267`)
- `DataFrame.apply` can now accept NumPy unary operators as strings, e.g. ``df.apply("sqrt")``, which was already the case for `Series.apply` (`39116`)

- `DataFrame.apply` can now accept non-callable DataFrame properties as strings, e.g. ``df.apply("size")``, which was already the case for `Series.apply` (`39116`)
- `DataFrame.applymap` can now accept kwargs to pass on to the user-provided ``func`` (`39987`)
- Passing a `DataFrame` indexer to ``iloc`` is now disallowed for `Series.__getitem__` and `DataFrame.__getitem__` (`39004`)
- `Series.apply` can now accept list-like or dictionary-like arguments that aren't lists or dictionaries, e.g. ``ser.apply(np.array(["sum", "mean"]))``, which was already the case for `DataFrame.apply` (`39140`)
- `DataFrame.plot.scatter` can now accept a categorical column for the argument ``c`` (`12380`, `31357`)
- `Series.loc` now raises a helpful error message when the Series has a `MultiIndex` and the indexer has too many dimensions (`35349`)
- `read_stata` now supports reading data from compressed files (`26599`)
- Added support for parsing ``ISO 8601``-like timestamps with negative signs to `Timedelta` (`37172`)
- Added support for unary operators in `FloatingArray` (`38749`)
- `RangeIndex` can now be constructed by passing a ``range`` object directly e.g. ``pd.RangeIndex(range(3))`` (`12067`)
- `Series.round` and `DataFrame.round` now work with nullable integer and floating dtypes (`38844`)
- `read_csv` and `read_json` expose the argument ``encoding_errors`` to control how encoding errors are handled (`39450`)
- `.DataFrameGroupBy.any`, `.SeriesGroupBy.any`, `.DataFrameGroupBy.all`, and `.SeriesGroupBy.all` use Kleene logic with nullable data types (`37506`)
- `.DataFrameGroupBy.any`, `.SeriesGroupBy.any`, `.DataFrameGroupBy.all`, and `.SeriesGroupBy.all` return a ``BooleanDtype`` for columns with nullable data types (`33449`)
- `.DataFrameGroupBy.any`, `.SeriesGroupBy.any`, `.DataFrameGroupBy.all`, and `.SeriesGroupBy.all` raising with ``object`` data containing ``pd.NA`` even when ``skipna=True`` (`37501`)
- `.DataFrameGroupBy.rank` and `.SeriesGroupBy.rank` now supports object-dtype data (`38278`)
- Constructing a `DataFrame` or `Series` with the ``data`` argument being a Python iterable that is *not* a NumPy ``ndarray`` consisting of NumPy scalars will now result in a dtype with a precision the maximum of the NumPy scalars; this was already the case when ``data`` is a NumPy ``ndarray`` (`40908`)
- Add keyword ``sort`` to `pivot_table` to allow non-sorting of the result (`39143`)
- Add keyword ``dropna`` to `DataFrame.value_counts` to allow counting rows that include ``NA`` values (`41325`)
- `Series.replace` will now cast results to ``PeriodDtype`` where possible instead of ``object`` dtype (`41526`)
- Improved error message in ``corr`` and ``cov`` methods on `.Rolling`, `.Expanding`, and `.ExponentialMovingWindow` when ``other`` is not a `DataFrame` or `Series` (`41741`)
- `Series.between` can now accept ``left`` or ``right`` as arguments to ``inclusive`` to include only the left or right boundary (`40245`)
- `DataFrame.explode` now supports exploding multiple columns. Its ``column`` argument now also accepts a list of str or tuples for exploding on multiple columns at the same time (`39240`)
- `DataFrame.sample` now accepts the ``ignore_index`` argument to reset the index after sampling, similar to `DataFrame.drop_duplicates` and `DataFrame.sort_values` (`38581`)  .. ---  .. _whatsnew_130.notable_bug_fixes:  Notable bug fixes
~~~  These are bug fixes that might have notable behavior changes.  .. _whatsnew_130.notable_bug_fixes.categorical_unique_maintains_dtype:  ``Categorical.unique`` now always maintains same dtype as original
^^^  Previously, when calling `Categorical.unique` with categorical data, unused categories in the new array
would be removed, making the dtype of the new array different than the

original (`18291`)  As an example of this, given:  .. ipython:: python  dtype = pd.CategoricalDtype(['bad', 'neutral', 'good'], ordered=True)  cat = pd.Categorical(['good', 'good', 'bad', 'bad'], dtype=dtype)  original = pd.Series(cat)  unique = original.unique()  *Previous behavior*:  .. code-block:: ipython  In [1]: unique  ['good', 'bad']  Categories (2, object): ['bad' < 'good']  In [2]: original.dtype == unique.dtype  False  *New behavior*:  .. ipython:: python  unique  original.dtype == unique.dtype  .. _whatsnew_130.notable_bug_fixes.combine_first_preserves_dtype:  Preserve dtypes in `DataFrame.combine_first`
^^^  `DataFrame.combine_first` will now preserve dtypes (`7509`)  .. ipython:: python  df1 = pd.DataFrame({"A": [1, 2, 3], "B": [1, 2, 3]}, index=[0, 1, 2])  df1  df2 = pd.DataFrame({"B": [4, 5, 6], "C": [1, 2, 3]}, index=[2, 3, 4])  df2  combined = df1.combine_first(df2)  *Previous behavior*:  .. code-block:: ipython  In [1]: combined.dtypes  Out[2]:  A  float64  B  float64  C  float64  dtype: object  *New behavior*:  .. ipython:: python  combined.dtypes  .. _whatsnew_130.notable_bug_fixes.groupby_preserves_dtype:  Groupby methods agg and transform no longer changes return dtype for callables
^^^  Previously the methods `.DataFrameGroupBy.aggregate`,
`.SeriesGroupBy.aggregate`, `.DataFrameGroupBy.transform`, and
`.SeriesGroupBy.transform` might cast the result dtype when the argument ``func``
is callable, possibly leading to undesirable results (`21240`). The cast would
occur if the result is numeric and casting back to the input dtype does not change any
values as measured by ``np.allclose``. Now no such casting occurs.  .. ipython:: python  df = pd.DataFrame({'key': [1, 1], 'a': [True, False], 'b': [True, True]})  df  *Previous behavior*:  .. code-block:: ipython  In [5]: df.groupby('key').agg(lambda x: x.sum())  Out[5]:  a  b  key  1  True  2  *New behavior*:  .. ipython:: python  df.groupby('key').agg(lambda x: x.sum())  .. _whatsnew_130.notable_bug_fixes.groupby_reductions_float_result:  ``float`` result for `.DataFrameGroupBy.mean`, `.DataFrameGroupBy.median`, and `.GDataFrameGroupBy.var`, `.SeriesGroupBy.mean`, `.SeriesGroupBy.median`, and `.SeriesGroupBy.var`
^^^  Previously, these methods could result in different dtypes depending on the input values.
Now, these methods will always return a float dtype. (`41137`)  .. ipython:: python  df = pd.DataFrame({'a': [True], 'b': [1], 'c': [1.0]})  *Previous behavior*:  .. code-block:: ipython  In [5]: df.groupby(df.index).mean()  Out[5]:  a  b  c  0  True  1  1.0  *New behavior*:  .. ipython:: python  df.groupby(df.index).mean()  .. _whatsnew_130.notable_bug_fixes.setitem_column_try_inplace:  Try operating inplace when setting values with ``loc`` and ``iloc``
^^^  When setting an entire column using ``loc`` or ``iloc``, pandas will try to
insert the values into the existing data rather than create an entirely new array.  .. ipython:: python  df = pd.DataFrame(range(3), columns=["A"], dtype="float64")  values = df.values  new = np.array([5, 6, 7], dtype="int64")  df.loc[[0, 1, 2], "A"] = new  In both the new and old behavior, the data in ``values`` is overwritten, but in
the old behavior the dtype of ``df["A"]`` changed to ``int64``.  *Previous behavior*:  .. code-block:: ipython  In [1]: df.dtypes  Out[1]:  A  int64  dtype: object  In [2]: np.shares_memory(df["A"].values, new)  Out[2]: False  In [3]: np.shares_memory(df["A"].values, values)  Out[3]: False  In pandas 1.3.0, ``df`` continues to share data with ``values``  *New behavior*:  .. ipython:: python  df.dtypes  np.shares_memory(df["A"], new)  np.shares_memory(df["A"], values)  .. _whatsnew_130.notable_bug_fixes.setitem_never_inplace:  Never operate inplace when setting ``frame[keys] = values``

^^^  When setting multiple columns using ``frame[keys] = values`` new arrays will
replace pre-existing arrays for these keys, which will *not* be over-written
(`39510`).  As a result, the columns will retain the dtype(s) of ``values``,
never casting to the dtypes of the existing arrays.  .. ipython:: python  df = pd.DataFrame(range(3), columns=["A"], dtype="float64")  df[["A"]] = 5  In the old behavior, ``5`` was cast to ``float64`` and inserted into the existing
array backing ``df``:  *Previous behavior*:  .. code-block:: ipython  In [1]: df.dtypes  Out[1]:  A  float64  In the new behavior, we get a new array, and retain an integer-dtyped ``5``:  *New behavior*:  .. ipython:: python  df.dtypes  .. _whatsnew_130.notable_bug_fixes.setitem_with_bool_casting:  Consistent casting with setting into Boolean Series
^^^  Setting non-boolean values into a `Series` with ``dtype=bool`` now consistently
casts to ``dtype=object`` (`38709`)  .. ipython:: python  orig = pd.Series([True, False])  ser = orig.copy()  ser.iloc[1] = np.nan  ser2 = orig.copy()  ser2.iloc[1] = 2.0  *Previous behavior*:  .. code-block:: ipython  In [1]: ser  Out [1]:  0  1.0  1  NaN  dtype: float64  In [2]:ser2  Out [2]:  0  True  1  2.0  dtype: object  *New behavior*:  .. ipython:: python  ser  ser2  .. _whatsnew_130.notable_bug_fixes.rolling_groupby_column:  DataFrameGroupBy.rolling and SeriesGroupBy.rolling no longer return grouped-by column in values
^^^  The group-by column will now be dropped from the result of a
``groupby.rolling`` operation (`32262`)  .. ipython:: python  df = pd.DataFrame({"A": [1, 1, 2, 3], "B": [0, 1, 2, 3]})  df  *Previous behavior*:  .. code-block:: ipython  In [1]: df.groupby("A").rolling(2).sum()  Out[1]:  A  B  A  1 0  NaN  NaN  1  2.0  1.0  2 2  NaN  NaN  3 3  NaN  NaN  *New behavior*:  .. ipython:: python  df.groupby("A").rolling(2).sum()  .. _whatsnew_130.notable_bug_fixes.rolling_var_precision:  Removed artificial truncation in rolling variance and standard deviation
^^^  `.Rolling.std` and `.Rolling.var` will no longer
artificially truncate results that are less than ``~1e-8`` and ``~1e-15`` respectively to
zero (`37051`, `40448`, `39872`).  However, floating point artifacts may now exist in the results when rolling over larger values.  .. ipython:: python  s = pd.Series([7, 5, 5, 5])  s.rolling(3).var()  .. _whatsnew_130.notable_bug_fixes.rolling_groupby_multiindex:  DataFrameGroupBy.rolling and SeriesGroupBy.rolling with MultiIndex no longer drop levels in the result
^^^  `DataFrameGroupBy.rolling` and `SeriesGroupBy.rolling` will no longer drop levels of a `DataFrame`
with a `MultiIndex` in the result. This can lead to a perceived duplication of levels in the resulting
`MultiIndex`, but this change restores the behavior that was present in version 1.1.3 (`38787`, `38523`).  .. ipython:: python  index = pd.MultiIndex.from_tuples([('idx1', 'idx2')], names=['label1', 'label2'])  df = pd.DataFrame({'a': [1], 'b': [2]}, index=index)  df  *Previous behavior*:  .. code-block:: ipython  In [1]: df.groupby('label1').rolling(1).sum()  Out[1]:  a  b  label1  idx1  1.0  2.0  *New behavior*:  .. ipython:: python  df.groupby('label1').rolling(1).sum()  .. ---  .. _whatsnew_130.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_130.api_breaking.deps:  Increased minimum versions for dependencies
^^^
Some minimum supported versions of dependencies were updated.
| numpy  | 1.17.3  |  X  |  X  |
| pytz  | 2017.3  |  X  |  |
| numexpr  | 2.7.0  |  |  X  |
| pytest (dev)  | 6.0  |  |  X  |
| mypy (dev)  | 0.812  |  |  X  |
| setuptools  | 38.6.0  |  |  X  |

| fastparquet  | 0.4.0  |  X  |
| lxml  | 4.3.0  |  |
| matplotlib  | 2.2.3  |  |
| openpyxl  | 3.0.0  |  X  |
| pyarrow  | 0.17.0  |  X  |
| pymysql  | 0.8.1  |  X  |
| pytables  | 3.5.1  |  |
| sqlalchemy  | 1.3.0  |  X  |
| tabulate  | 0.8.7  |  X  |
| xarray  | 0.12.0  |  |
| xlrd  | 1.2.0  |  |
| xlsxwriter  | 1.0.2  |  |
| xlwt  | 1.3.0  |  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  .. _whatsnew_130.api_breaking.other:  Other API changes
^^^
- Partially initialized `CategoricalDtype` objects (i.e. those with ``categories=None``) will no longer compare as equal to fully initialized dtype objects (`38516`)
- Accessing ``_constructor_expanddim`` on a `DataFrame` and ``_constructor_sliced`` on a `Series` now raise an ``AttributeError``. Previously a ``NotImplementedError`` was raised (`38782`)
- Added new `engine`` and ``**engine_kwargs`` parameters to `DataFrame.to_sql` to support other future "SQL engines". Currently we still only use ``SQLAlchemy`` under the hood, but more engines are planned to be supported such as `turbodbc (`36893`)
- Removed redundant ``freq`` from `PeriodIndex` string representation (`41653`)
- `ExtensionDtype.construct_array_type` is now a required method instead of an optional one for `ExtensionDtype` subclasses (`24860`)
- Calling ``hash`` on non-hashable pandas objects will now raise ``TypeError`` with the built-in error message (e.g. ``unhashable type: 'Series'``). Previously it would raise a custom message such as ``'Series' objects are mutable, thus they cannot be hashed``. Furthermore, ``isinstance(<Series>, abc.collections.Hashable)`` will now return ``False`` (`40013`)
- `.Styler.from_custom_template` now has two new arguments for template names, and removed the old ``name``, due to template inheritance having been introducing for better parsing (`42053`). Subclassing modifications to Styler attributes are also needed.  .. _whatsnew_130.api_breaking.build:  Build
^^^
- Documentation in ``.pptx`` and ``.pdf`` formats are no longer included in wheels or source distributions. (`30741`)  .. ---  .. _whatsnew_130.deprecations:  Deprecations
~~~  .. _whatsnew_130.deprecations.nuisance_columns:  Deprecated dropping nuisance columns in DataFrame reductions and DataFrameGroupBy operations
^^^
Calling a reduction (e.g. ``.min``, ``.max``, ``.sum``) on a `DataFrame` with
``numeric_only=None`` (the default), columns where the reduction raises a ``TypeError``
are silently ignored and dropped from the result.  This behavior is deprecated. In a future version, the ``TypeError`` will be raised,

and users will need to select only valid columns before calling the function.  For example:  .. ipython:: python  df = pd.DataFrame({"A": [1, 2, 3, 4], "B": pd.date_range("2016-01-01", periods=4)})  df  *Old behavior*:  .. code-block:: ipython  In [3]: df.prod()  Out[3]:  Out[3]:  A  24  dtype: int64  *Future behavior*:  .. code-block:: ipython  In [4]: df.prod()  ...  TypeError: 'DatetimeArray' does not implement reduction 'prod'  In [5]: df[["A"]].prod()  Out[5]:  A  24  dtype: int64  Similarly, when applying a function to `DataFrameGroupBy`, columns on which
the function raises ``TypeError`` are currently silently ignored and dropped
from the result.  This behavior is deprecated.  In a future version, the ``TypeError``
will be raised, and users will need to select only valid columns before calling
the function.  For example:  .. ipython:: python  df = pd.DataFrame({"A": [1, 2, 3, 4], "B": pd.date_range("2016-01-01", periods=4)})  gb = df.groupby([1, 1, 2, 2])  *Old behavior*:  .. code-block:: ipython  In [4]: gb.prod(numeric_only=False)  Out[4]:  A  1  2  2  12  *Future behavior*:  .. code-block:: ipython  In [5]: gb.prod(numeric_only=False)  ...  TypeError: datetime64 type does not support prod operations  In [6]: gb[["A"]].prod(numeric_only=False)  Out[6]:  A  1  2  2  12  .. _whatsnew_130.deprecations.other:  Other Deprecations
^^^
- Deprecated allowing scalars to be passed to the `Categorical` constructor (`38433`)
- Deprecated constructing `CategoricalIndex` without passing list-like data (`38944`)
- Deprecated allowing subclass-specific keyword arguments in the `Index` constructor, use the specific subclass directly instead (`14093`, `21311`, `22315`, `26974`)
- Deprecated the `astype` method of datetimelike (``timedelta64[ns]``, ``datetime64[ns]``, ``Datetime64TZDtype``, ``PeriodDtype``) to convert to integer dtypes, use ``values.view(...)`` instead (`38544`). This deprecation was later reverted in pandas 1.4.0.
- Deprecated `MultiIndex.is_lexsorted` and `MultiIndex.lexsort_depth`, use `MultiIndex.is_monotonic_increasing` instead (`32259`)
- Deprecated keyword ``try_cast`` in `Series.where`, `Series.mask`, `DataFrame.where`, `DataFrame.mask`; cast results manually if desired (`38836`)
- Deprecated comparison of `Timestamp` objects with ``datetime.date`` objects.  Instead of e.g. ``ts <= mydate`` use ``ts <= pd.Timestamp(mydate)`` or ``ts.date() <= mydate`` (`36131`)
- Deprecated `Rolling.win_type` returning ``"freq"`` (`38963`)
- Deprecated `Rolling.is_datetimelike` (`38963`)
- Deprecated `DataFrame` indexer for `Series.__setitem__` and `DataFrame.__setitem__` (`39004`)
- Deprecated `ExponentialMovingWindow.vol` (`39220`)
- Using ``.astype`` to convert between ``datetime64[ns]`` dtype and `DatetimeTZDtype` is deprecated and will raise in a future version, use ``obj.tz_localize`` or ``obj.dt.tz_localize`` instead (`38622`)
- Deprecated casting ``datetime.date`` objects to ``datetime64`` when used as ``fill_value`` in `DataFrame.unstack`, `DataFrame.shift`, `Series.shift`, and `DataFrame.reindex`, pass ``pd.Timestamp(dateobj)`` instead (`39767`)
- Deprecated `.Styler.set_na_rep` and `.Styler.set_precision` in favor of `.Styler.format` with ``na_rep`` and ``precision`` as existing and new input arguments respectively (`40134`, `40425`)
- Deprecated `.Styler.where` in favor of using an alternative formulation with `Styler.applymap` (`40821`)
- Deprecated allowing partial failure in `Series.transform` and `DataFrame.transform` when ``func`` is list-like or dict-like and raises anything but ``TypeError``; ``func`` raising anything but a ``TypeError`` will raise in a future version (`40211`)
- Deprecated arguments ``error_bad_lines`` and ``warn_bad_lines`` in `read_csv` and `read_table` in favor of argument ``on_bad_lines`` (`15122`)

- Deprecated support for ``np.ma.mrecords.MaskedRecords`` in the `DataFrame` constructor, pass ``{name: data[name] for name in data.dtype.names}`` instead (`40363`)
- Deprecated using `merge`, `DataFrame.merge`, and `DataFrame.join` on a different number of levels (`34862`)
- Deprecated the use of ``**kwargs`` in `.ExcelWriter`; use the keyword argument ``engine_kwargs`` instead (`40430`)
- Deprecated the ``level`` keyword for `DataFrame` and `Series` aggregations; use groupby instead (`39983`)
- Deprecated the ``inplace`` parameter of `Categorical.remove_categories`, `Categorical.add_categories`, `Categorical.reorder_categories`, `Categorical.rename_categories`, `Categorical.set_categories` and will be removed in a future version (`37643`)
- Deprecated `merge` producing duplicated columns through the ``suffixes`` keyword  and already existing columns (`22818`)
- Deprecated setting `Categorical._codes`, create a new `Categorical` with the desired codes instead (`40606`)
- Deprecated the ``convert_float`` optional argument in `read_excel` and `ExcelFile.parse` (`41127`)
- Deprecated behavior of `DatetimeIndex.union` with mixed timezones; in a future version both will be cast to UTC instead of object dtype (`39328`)
- Deprecated using ``usecols`` with out of bounds indices for `read_csv` with ``engine="c"`` (`25623`)
- Deprecated special treatment of lists with first element a Categorical in the `DataFrame` constructor; pass as ``pd.DataFrame({col: categorical, ...})`` instead (`38845`)
- Deprecated behavior of `DataFrame` constructor when a ``dtype`` is passed and the data cannot be cast to that dtype. In a future version, this will raise instead of being silently ignored (`24435`)
- Deprecated the `Timestamp.freq` attribute.  For the properties that use it (``is_month_start``, ``is_month_end``, ``is_quarter_start``, ``is_quarter_end``, ``is_year_start``, ``is_year_end``), when you have a ``freq``, use e.g. ``freq.is_month_start(ts)`` (`15146`)
- Deprecated construction of `Series` or `DataFrame` with ``DatetimeTZDtype`` data and ``datetime64[ns]`` dtype.  Use ``Series(data).dt.tz_localize(None)`` instead (`41555`, `33401`)
- Deprecated behavior of `Series` construction with large-integer values and small-integer dtype silently overflowing; use ``Series(data).astype(dtype)`` instead (`41734`)
- Deprecated behavior of `DataFrame` construction with floating data and integer dtype casting even when lossy; in a future version this will remain floating, matching `Series` behavior (`41770`)
- Deprecated inference of ``timedelta64[ns]``, ``datetime64[ns]``, or ``DatetimeTZDtype`` dtypes in `Series` construction when data containing strings is passed and no ``dtype`` is passed (`33558`)
- In a future version, constructing `Series` or `DataFrame` with ``datetime64[ns]`` data and ``DatetimeTZDtype`` will treat the data as wall-times instead of as UTC times (matching DatetimeIndex behavior). To treat the data as UTC times, use ``pd.Series(data).dt.tz_localize("UTC").dt.tz_convert(dtype.tz)`` or ``pd.Series(data.view("int64"), dtype=dtype)`` (`33401`)
- Deprecated passing lists as ``key`` to `DataFrame.xs` and `Series.xs` (`41760`)
- Deprecated boolean arguments of ``inclusive`` in `Series.between` to have ``{"left", "right", "neither", "both"}`` as standard argument values (`40628`)

- Deprecated passing arguments as positional for all of the following, with exceptions noted (`41485`):  - `concat` (other than ``objs``)  - `read_csv` (other than ``filepath_or_buffer``)  - `read_table` (other than ``filepath_or_buffer``)  - `DataFrame.clip` and `Series.clip` (other than ``upper`` and ``lower``)  - `DataFrame.drop_duplicates` (except for ``subset``), `Series.drop_duplicates`, `Index.drop_duplicates` and `MultiIndex.drop_duplicates`  - `DataFrame.drop` (other than ``labels``) and `Series.drop`  - `DataFrame.dropna` and `Series.dropna`  - `DataFrame.ffill`, `Series.ffill`, `DataFrame.bfill`, and `Series.bfill`  - `DataFrame.fillna` and `Series.fillna` (apart from ``value``)  - `DataFrame.interpolate` and `Series.interpolate` (other than ``method``)  - `DataFrame.mask` and `Series.mask` (other than ``cond`` and ``other``)  - `DataFrame.reset_index` (other than ``level``) and `Series.reset_index`  - `DataFrame.set_axis` and `Series.set_axis` (other than ``labels``)  - `DataFrame.set_index` (other than ``keys``)  - `DataFrame.sort_index` and `Series.sort_index`  - `DataFrame.sort_values` (other than ``by``) and `Series.sort_values`  - `DataFrame.where` and `Series.where` (other than ``cond`` and ``other``)  - `Index.set_names` and `MultiIndex.set_names` (except for ``names``)  - `MultiIndex.codes` (except for ``codes``)  - `MultiIndex.set_levels` (except for ``levels``)  - `Resampler.interpolate` (other than ``method``)  .. ---  .. _whatsnew_130.performance:  Performance improvements
~~~
- Performance improvement in `IntervalIndex.isin` (`38353`)
- Performance improvement in `Series.mean` for nullable data types (`34814`)
- Performance improvement in `Series.isin` for nullable data types (`38340`)
- Performance improvement in `DataFrame.fillna` with ``method="pad"`` or ``method="backfill"`` for nullable floating and nullable integer dtypes (`39953`)
- Performance improvement in `DataFrame.corr` for ``method=kendall`` (`28329`)
- Performance improvement in `DataFrame.corr` for ``method=spearman`` (`40956`, `41885`)
- Performance improvement in `.Rolling.corr` and `.Rolling.cov` (`39388`)
- Performance improvement in `.RollingGroupby.corr`, `.ExpandingGroupby.corr`, `.ExpandingGroupby.corr` and `.ExpandingGroupby.cov` (`39591`)
- Performance improvement in `unique` for object data type (`37615`)
- Performance improvement in `json_normalize` for basic cases (including separators) (`40035` `15621`)
- Performance improvement in `.ExpandingGroupby` aggregation methods (`39664`)
- Performance improvement in `.Styler` where render times are more than 50% reduced and now matches `DataFrame.to_html` (`39972` `39952`, `40425`)
- The method `.Styler.set_td_classes` is now as performant as `.Styler.apply` and `.Styler.applymap`, and even more so in some cases (`40453`)
- Performance improvement in `.ExponentialMovingWindow.mean` with ``times`` (`39784`)
- Performance improvement in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` when requiring the Python fallback implementation (`40176`)
- Performance improvement in the conversion of a PyArrow Boolean array to a pandas nullable Boolean array (`41051`)
- Performance improvement for concatenation of data with type `CategoricalDtype` (`40193`)
- Performance improvement in `.DataFrameGroupBy.cummin`, `.SeriesGroupBy.cummin`, `.DataFrameGroupBy.cummax`, and `.SeriesGroupBy.cummax` with nullable data types (`37493`)
- Performance improvement in `Series.nunique` with nan values (`40865`)
- Performance improvement in `DataFrame.transpose`, `Series.unstack` with ``DatetimeTZDtype`` (`40149`)
- Performance improvement in `Series.plot` and `DataFrame.plot` with entry point lazy loading (`41492`)  .. ---  .. _whatsnew_130.bug_fixes:  Bug fixes
^^^

^^^
- Bug in `CategoricalIndex` incorrectly failing to raise ``TypeError`` when scalar data is passed (`38614`)
- Bug in ``CategoricalIndex.reindex`` failed when the `Index` passed was not categorical but whose values were all labels in the category (`28690`)
- Bug where constructing a `Categorical` from an object-dtype array of ``date`` objects did not round-trip correctly with ``astype`` (`38552`)
- Bug in constructing a `DataFrame` from an ``ndarray`` and a `CategoricalDtype` (`38857`)
- Bug in setting categorical values into an object-dtype column in a `DataFrame` (`39136`)
- Bug in `DataFrame.reindex` was raising an ``IndexError`` when the new index contained duplicates and the old index was a `CategoricalIndex` (`38906`)
- Bug in `Categorical.fillna` with a tuple-like category raising ``NotImplementedError`` instead of ``ValueError`` when filling with a non-category tuple (`41914`)  Datetimelike
^^^
- Bug in `DataFrame` and `Series` constructors sometimes dropping nanoseconds from `Timestamp` (resp. `Timedelta`) ``data``, with ``dtype=datetime64[ns]`` (resp. ``timedelta64[ns]``) (`38032`)
- Bug in `DataFrame.first` and `Series.first` with an offset of one month returning an incorrect result when the first day is the last day of a month (`29623`)
- Bug in constructing a `DataFrame` or `Series` with mismatched ``datetime64`` data and ``timedelta64`` dtype, or vice-versa, failing to raise a ``TypeError`` (`38575`, `38764`, `38792`)
- Bug in constructing a `Series` or `DataFrame` with a ``datetime`` object out of bounds for ``datetime64[ns]`` dtype or a ``timedelta`` object out of bounds for ``timedelta64[ns]`` dtype (`38792`, `38965`)
- Bug in `DatetimeIndex.intersection`, `DatetimeIndex.symmetric_difference`, `PeriodIndex.intersection`, `PeriodIndex.symmetric_difference` always returning object-dtype when operating with `CategoricalIndex` (`38741`)
- Bug in `DatetimeIndex.intersection` giving incorrect results with non-Tick frequencies with ``n != 1`` (`42104`)
- Bug in `Series.where` incorrectly casting ``datetime64`` values to ``int64`` (`37682`)
- Bug in `Categorical` incorrectly typecasting ``datetime`` object to ``Timestamp`` (`38878`)
- Bug in comparisons between `Timestamp` object and ``datetime64`` objects just outside the implementation bounds for nanosecond ``datetime64`` (`39221`)
- Bug in `Timestamp.round`, `Timestamp.floor`, `Timestamp.ceil` for values near the implementation bounds of `Timestamp` (`39244`)
- Bug in `Timedelta.round`, `Timedelta.floor`, `Timedelta.ceil` for values near the implementation bounds of `Timedelta` (`38964`)
- Bug in `date_range` incorrectly creating `DatetimeIndex` containing ``NaT`` instead of raising ``OutOfBoundsDatetime`` in corner cases (`24124`)
- Bug in `infer_freq` incorrectly fails to infer 'H' frequency of `DatetimeIndex` if the latter has a timezone and crosses DST boundaries (`39556`)
- Bug in `Series` backed by `DatetimeArray` or `TimedeltaArray` sometimes failing to set the array's ``freq`` to ``None`` (`41425`)  Timedelta
^^^
- Bug in constructing `Timedelta` from ``np.timedelta64`` objects with non-nanosecond units that are out of bounds for ``timedelta64[ns]`` (`38965`)
- Bug in constructing a `TimedeltaIndex` incorrectly accepting ``np.datetime64("NaT")`` objects (`39462`)
- Bug in constructing `Timedelta` from an input string with only symbols and no digits failed to raise an error (`39710`)
- Bug in `TimedeltaIndex` and `to_timedelta` failing to raise when passed non-nanosecond ``timedelta64`` arrays that overflow when converting to ``timedelta64[ns]`` (`40008`)  Timezones
^^^
- Bug in different ``tzinfo`` objects representing UTC not being treated as equivalent (`39216`)
- Bug in ``dateutil.tz.gettz("UTC")`` not being recognized as equivalent to other UTC-representing tzinfos (`39276`)  Numeric
^^^

^^^
- Bug in `DataFrame.quantile`, `DataFrame.sort_values` causing incorrect subsequent indexing behavior (`38351`)
- Bug in `DataFrame.sort_values` raising an `IndexError` for empty ``by`` (`40258`)
- Bug in `DataFrame.select_dtypes` with ``include=np.number`` would drop numeric ``ExtensionDtype`` columns (`35340`)
- Bug in `DataFrame.mode` and `Series.mode` not keeping consistent integer `Index` for empty input (`33321`)
- Bug in `DataFrame.rank` when the DataFrame contained ``np.inf`` (`32593`)
- Bug in `DataFrame.rank` with ``axis=0`` and columns holding incomparable types raising an ``IndexError`` (`38932`)
- Bug in `Series.rank`, `DataFrame.rank`, `.DataFrameGroupBy.rank`, and `.SeriesGroupBy.rank` treating the most negative ``int64`` value as missing (`32859`)
- Bug in `DataFrame.select_dtypes` different behavior between Windows and Linux with ``include="int"`` (`36596`)
- Bug in `DataFrame.apply` and `DataFrame.agg` when passed the argument ``func="size"`` would operate on the entire ``DataFrame`` instead of rows or columns (`39934`)
- Bug in `DataFrame.transform` would raise a ``SpecificationError`` when passed a dictionary and columns were missing; will now raise a ``KeyError`` instead (`40004`)
- Bug in `.DataFrameGroupBy.rank` and `.SeriesGroupBy.rank` giving incorrect results with ``pct=True`` and equal values between consecutive groups (`40518`)
- Bug in `Series.count` would result in an ``int32`` result on 32-bit platforms when argument ``level=None`` (`40908`)
- Bug in `Series` and `DataFrame` reductions with methods ``any`` and ``all`` not returning Boolean results for object data (`12863`, `35450`, `27709`)
- Bug in `Series.clip` would fail if the Series contains NA values and has nullable int or float as a data type (`40851`)
- Bug in `UInt64Index.where` and `UInt64Index.putmask` with an ``np.int64`` dtype ``other`` incorrectly raising ``TypeError`` (`41974`)
- Bug in `DataFrame.agg()` not sorting the aggregated axis in the order of the provided aggregation functions when one or more aggregation function fails to produce results (`33634`)
- Bug in `DataFrame.clip` not interpreting missing values as no threshold (`40420`)  Conversion
^^^
- Bug in `Series.to_dict` with ``orient='records'`` now returns Python native types (`25969`)
- Bug in `Series.view` and `Index.view` when converting between datetime-like (``datetime64[ns]``, ``datetime64[ns, tz]``, ``timedelta64``, ``period``) dtypes (`39788`)
- Bug in creating a `DataFrame` from an empty ``np.recarray`` not retaining the original dtypes (`40121`)
- Bug in `DataFrame` failing to raise a ``TypeError`` when constructing from a ``frozenset`` (`40163`)
- Bug in `Index` construction silently ignoring a passed ``dtype`` when the data cannot be cast to that dtype (`21311`)
- Bug in `StringArray.astype` falling back to NumPy and raising when converting to ``dtype='categorical'`` (`40450`)
- Bug in `factorize` where, when given an array with a numeric NumPy dtype lower than int64, uint64 and float64, the unique values did not keep their original dtype (`41132`)
- Bug in `DataFrame` construction with a dictionary containing an array-like with ``ExtensionDtype`` and ``copy=True`` failing to make a copy (`38939`)
- Bug in `qcut` raising error when taking ``Float64DType`` as input (`40730`)
- Bug in `DataFrame` and `Series` construction with ``datetime64[ns]`` data and ``dtype=object`` resulting in ``datetime`` objects instead of `Timestamp` objects (`41599`)
- Bug in `DataFrame` and `Series` construction with ``timedelta64[ns]`` data and ``dtype=object`` resulting in ``np.timedelta64`` objects instead of `Timedelta` objects (`41599`)
- Bug in `DataFrame` construction when given a two-dimensional object-dtype ``np.ndarray`` of `Period` or `Interval` objects failing to cast to `PeriodDtype` or `IntervalDtype`, respectively (`41812`)

- Bug in constructing a `Series` from a list and a `PandasDtype` (`39357`)
- Bug in creating a `Series` from a ``range`` object that does not fit in the bounds of ``int64`` dtype (`30173`)
- Bug in creating a `Series` from a ``dict`` with all-tuple keys and an `Index` that requires reindexing (`41707`)
- Bug in `.infer_dtype` not recognizing Series, Index, or array with a Period dtype (`23553`)
- Bug in `.infer_dtype` raising an error for general `.ExtensionArray` objects. It will now return ``"unknown-array"`` instead of raising (`37367`)
- Bug in `DataFrame.convert_dtypes` incorrectly raised a ``ValueError`` when called on an empty DataFrame (`40393`)  Strings
^^^
- Bug in the conversion from ``pyarrow.ChunkedArray`` to `~arrays.StringArray` when the original had zero chunks (`41040`)
- Bug in `Series.replace` and `DataFrame.replace` ignoring replacements with ``regex=True`` for ``StringDType`` data (`41333`, `35977`)
- Bug in `Series.str.extract` with `~arrays.StringArray` returning object dtype for an empty `DataFrame` (`41441`)
- Bug in `Series.str.replace` where the ``case`` argument was ignored when ``regex=False`` (`41602`)  Interval
^^^
- Bug in `IntervalIndex.intersection` and `IntervalIndex.symmetric_difference` always returning object-dtype when operating with `CategoricalIndex` (`38653`, `38741`)
- Bug in `IntervalIndex.intersection` returning duplicates when at least one of the `Index` objects have duplicates which are present in the other (`38743`)
- `IntervalIndex.union`, `IntervalIndex.intersection`, `IntervalIndex.difference`, and `IntervalIndex.symmetric_difference` now cast to the appropriate dtype instead of raising a ``TypeError`` when operating with another `IntervalIndex` with incompatible dtype (`39267`)
- `PeriodIndex.union`, `PeriodIndex.intersection`, `PeriodIndex.symmetric_difference`, `PeriodIndex.difference` now cast to object dtype instead of raising ``IncompatibleFrequency`` when operating with another `PeriodIndex` with incompatible dtype (`39306`)
- Bug in `IntervalIndex.is_monotonic`, `IntervalIndex.get_loc`, `IntervalIndex.get_indexer_for`, and `IntervalIndex.__contains__` when NA values are present (`41831`)  Indexing
^^^
- Bug in `Index.union` and `MultiIndex.union` dropping duplicate ``Index`` values when ``Index`` was not monotonic or ``sort`` was set to ``False`` (`36289`, `31326`, `40862`)
- Bug in `CategoricalIndex.get_indexer` failing to raise ``InvalidIndexError`` when non-unique (`38372`)
- Bug in `IntervalIndex.get_indexer` when ``target`` has ``CategoricalDtype`` and both the index and the target contain NA values (`41934`)
- Bug in `Series.loc` raising a ``ValueError`` when input was filtered with a Boolean list and values to set were a list with lower dimension (`20438`)
- Bug in inserting many new columns into a `DataFrame` causing incorrect subsequent indexing behavior (`38380`)
- Bug in `DataFrame.__setitem__` raising a ``ValueError`` when setting multiple values to duplicate columns (`15695`)
- Bug in `DataFrame.loc`, `Series.loc`, `DataFrame.__getitem__` and `Series.__getitem__` returning incorrect elements for non-monotonic `DatetimeIndex` for string slices (`33146`)
- Bug in `DataFrame.reindex` and `Series.reindex` with timezone aware indexes raising a ``TypeError`` for ``method="ffill"`` and ``method="bfill"`` and specified ``tolerance`` (`38566`)
- Bug in `DataFrame.reindex` with ``datetime64[ns]`` or ``timedelta64[ns]`` incorrectly casting to integers when the ``fill_value`` requires casting to object dtype (`39755`)
- Bug in `DataFrame.__setitem__` raising a ``ValueError`` when setting on an empty `DataFrame` using specified columns and a nonempty `DataFrame` value (`38831`)

- Bug in `DataFrame.loc.__setitem__` raising a ``ValueError`` when operating on a unique column when the `DataFrame` has duplicate columns (`38521`)
- Bug in `DataFrame.iloc.__setitem__` and `DataFrame.loc.__setitem__` with mixed dtypes when setting with a dictionary value (`38335`)
- Bug in `Series.loc.__setitem__` and `DataFrame.loc.__setitem__` raising ``KeyError`` when provided a Boolean generator (`39614`)
- Bug in `Series.iloc` and `DataFrame.iloc` raising a ``KeyError`` when provided a generator (`39614`)
- Bug in `DataFrame.__setitem__` not raising a ``ValueError`` when the right hand side is a `DataFrame` with wrong number of columns (`38604`)
- Bug in `Series.__setitem__` raising a ``ValueError`` when setting a `Series` with a scalar indexer (`38303`)
- Bug in `DataFrame.loc` dropping levels of a `MultiIndex` when the `DataFrame` used as input has only one row (`10521`)
- Bug in `DataFrame.__getitem__` and `Series.__getitem__` always raising ``KeyError`` when slicing with existing strings where the `Index` has milliseconds (`33589`)
- Bug in setting ``timedelta64`` or ``datetime64`` values into numeric `Series` failing to cast to object dtype (`39086`, `39619`)
- Bug in setting `Interval` values into a `Series` or `DataFrame` with mismatched `IntervalDtype` incorrectly casting the new values to the existing dtype (`39120`)
- Bug in setting ``datetime64`` values into a `Series` with integer-dtype incorrectly casting the datetime64 values to integers (`39266`)
- Bug in setting ``np.datetime64("NaT")`` into a `Series` with `Datetime64TZDtype` incorrectly treating the timezone-naive value as timezone-aware (`39769`)
- Bug in `Index.get_loc` not raising ``KeyError`` when ``key=NaN`` and ``method`` is specified but ``NaN`` is not in the `Index` (`39382`)
- Bug in `DatetimeIndex.insert` when inserting ``np.datetime64("NaT")`` into a timezone-aware index incorrectly treating the timezone-naive value as timezone-aware (`39769`)
- Bug in incorrectly raising in `Index.insert`, when setting a new column that cannot be held in the existing ``frame.columns``, or in `Series.reset_index` or `DataFrame.reset_index` instead of casting to a compatible dtype (`39068`)
- Bug in `RangeIndex.append` where a single object of length 1 was concatenated incorrectly (`39401`)
- Bug in `RangeIndex.astype` where when converting to `CategoricalIndex`, the categories became a `Int64Index` instead of a `RangeIndex` (`41263`)
- Bug in setting ``numpy.timedelta64`` values into an object-dtype `Series` using a Boolean indexer (`39488`)
- Bug in setting numeric values into a into a boolean-dtypes `Series` using ``at`` or ``iat`` failing to cast to object-dtype (`39582`)
- Bug in `DataFrame.__setitem__` and `DataFrame.iloc.__setitem__` raising ``ValueError`` when trying to index with a row-slice and setting a list as values (`40440`)
- Bug in `DataFrame.loc` not raising ``KeyError`` when the key was not found in `MultiIndex` and the levels were not fully specified (`41170`)
- Bug in `DataFrame.loc.__setitem__` when setting-with-expansion incorrectly raising when the index in the expanding axis contained duplicates (`40096`)
- Bug in `DataFrame.loc.__getitem__` with `MultiIndex` casting to float when at least one index column has float dtype and we retrieve a scalar (`41369`)
- Bug in `DataFrame.loc` incorrectly matching non-Boolean index elements (`20432`)
- Bug in indexing with ``np.nan`` on a `Series` or `DataFrame` with a `CategoricalIndex` incorrectly raising ``KeyError`` when ``np.nan`` keys are present (`41933`)
- Bug in `Series.__delitem__` with ``ExtensionDtype`` incorrectly casting to ``ndarray`` (`40386`)
- Bug in `DataFrame.at` with a `CategoricalIndex` returning incorrect results when passed integer keys (`41846`)

- Bug in `DataFrame.loc` returning a `MultiIndex` in the wrong order if an indexer has duplicates (`40978`)
- Bug in `DataFrame.__setitem__` raising a ``TypeError`` when using a ``str`` subclass as the column name with a `DatetimeIndex` (`37366`)
- Bug in `PeriodIndex.get_loc` failing to raise a ``KeyError`` when given a `Period` with a mismatched ``freq`` (`41670`)
- Bug ``.loc.__getitem__`` with a `UInt64Index` and negative-integer keys raising ``OverflowError`` instead of ``KeyError`` in some cases, wrapping around to positive integers in others (`41777`)
- Bug in `Index.get_indexer` failing to raise ``ValueError`` in some cases with invalid ``method``, ``limit``, or ``tolerance`` arguments (`41918`)
- Bug when slicing a `Series` or `DataFrame` with a `TimedeltaIndex` when passing an invalid string raising ``ValueError`` instead of a ``TypeError`` (`41821`)
- Bug in `Index` constructor sometimes silently ignoring a specified ``dtype`` (`38879`)
- `Index.where` behavior now mirrors `Index.putmask` behavior, i.e. ``index.where(mask, other)`` matches ``index.putmask(~mask, other)`` (`39412`)  Missing
^^^
- Bug in `Grouper` did not correctly propagate the ``dropna`` argument; `.DataFrameGroupBy.transform` now correctly handles missing values for ``dropna=True`` (`35612`)
- Bug in `isna`, `Series.isna`, `Index.isna`, `DataFrame.isna`, and the corresponding ``notna`` functions not recognizing ``Decimal("NaN")`` objects (`39409`)
- Bug in `DataFrame.fillna` not accepting a dictionary for the ``downcast`` keyword (`40809`)
- Bug in `isna` not returning a copy of the mask for nullable types, causing any subsequent mask modification to change the original array (`40935`)
- Bug in `DataFrame` construction with float data containing ``NaN`` and an integer ``dtype`` casting instead of retaining the ``NaN`` (`26919`)
- Bug in `Series.isin` and `MultiIndex.isin` didn't treat all nans as equivalent if they were in tuples (`41836`)  MultiIndex
^^^
- Bug in `DataFrame.drop` raising a ``TypeError`` when the `MultiIndex` is non-unique and ``level`` is not provided (`36293`)
- Bug in `MultiIndex.intersection` duplicating ``NaN`` in the result (`38623`)
- Bug in `MultiIndex.equals` incorrectly returning ``True`` when the `MultiIndex` contained ``NaN`` even when they are differently ordered (`38439`)
- Bug in `MultiIndex.intersection` always returning an empty result when intersecting with `CategoricalIndex` (`38653`)
- Bug in `MultiIndex.difference` incorrectly raising ``TypeError`` when indexes contain non-sortable entries (`41915`)
- Bug in `MultiIndex.reindex` raising a ``ValueError`` when used on an empty `MultiIndex` and indexing only a specific level (`41170`)
- Bug in `MultiIndex.reindex` raising ``TypeError`` when reindexing against a flat `Index` (`41707`)  I/O
^^^
- Bug in `Index.__repr__` when ``display.max_seq_items=1`` (`38415`)
- Bug in `read_csv` not recognizing scientific notation if the argument ``decimal`` is set and ``engine="python"`` (`31920`)
- Bug in `read_csv` interpreting ``NA`` value as comment, when ``NA`` does contain the comment string fixed for ``engine="python"`` (`34002`)
- Bug in `read_csv` raising an ``IndexError`` with multiple header columns and ``index_col`` is specified when the file has no data rows (`38292`)
- Bug in `read_csv` not accepting ``usecols`` with a different length than ``names`` for ``engine="python"`` (`16469`)
- Bug in `read_csv` returning object dtype when ``delimiter=","`` with ``usecols`` and ``parse_dates`` specified for ``engine="python"`` (`35873`)
- Bug in `read_csv` raising a ``TypeError`` when ``names`` and ``parse_dates`` is specified for ``engine="c"`` (`33699`)
- Bug in `read_clipboard` and `DataFrame.to_clipboard` not working in WSL (`38527`)
- Allow custom error values for the ``parse_dates`` argument of `read_sql`, `read_sql_query` and `read_sql_table` (`35185`)

- Bug in `DataFrame.to_hdf` and `Series.to_hdf` raising a ``KeyError`` when trying to apply for subclasses of ``DataFrame`` or ``Series`` (`33748`)
- Bug in `.HDFStore.put` raising a wrong ``TypeError`` when saving a DataFrame with non-string dtype (`34274`)
- Bug in `json_normalize` resulting in the first element of a generator object not being included in the returned DataFrame (`35923`)
- Bug in `read_csv` applying the thousands separator to date columns when the column should be parsed for dates and ``usecols`` is specified for ``engine="python"`` (`39365`)
- Bug in `read_excel` forward filling `MultiIndex` names when multiple header and index columns are specified (`34673`)
- Bug in `read_excel` not respecting `set_option` (`34252`)
- Bug in `read_csv` not switching ``true_values`` and ``false_values`` for nullable Boolean dtype (`34655`)
- Bug in `read_json` when ``orient="split"`` not maintaining a numeric string index (`28556`)
- `read_sql` returned an empty generator if ``chunksize`` was non-zero and the query returned no results. Now returns a generator with a single empty DataFrame (`34411`)
- Bug in `read_hdf` returning unexpected records when filtering on categorical string columns using the ``where`` parameter (`39189`)
- Bug in `read_sas` raising a ``ValueError`` when ``datetimes`` were null (`39725`)
- Bug in `read_excel` dropping empty values from single-column spreadsheets (`39808`)
- Bug in `read_excel` loading trailing empty rows/columns for some filetypes (`41167`)
- Bug in `read_excel` raising an ``AttributeError`` when the excel file had a ``MultiIndex`` header followed by two empty rows and no index (`40442`)
- Bug in `read_excel`, `read_csv`, `read_table`, `read_fwf`, and `read_clipboard` where one blank row after a ``MultiIndex`` header with no index would be dropped (`40442`)
- Bug in `DataFrame.to_string` misplacing the truncation column when ``index=False`` (`40904`)
- Bug in `DataFrame.to_string` adding an extra dot and misaligning the truncation row when ``index=False`` (`40904`)
- Bug in `read_orc` always raising an ``AttributeError`` (`40918`)
- Bug in `read_csv` and `read_table` silently ignoring ``prefix`` if ``names`` and ``prefix`` are defined, now raising a ``ValueError`` (`39123`)
- Bug in `read_csv` and `read_excel` not respecting the dtype for a duplicated column name when ``mangle_dupe_cols`` is set to ``True`` (`35211`)
- Bug in `read_csv` silently ignoring ``sep`` if ``delimiter`` and ``sep`` are defined, now raising a ``ValueError`` (`39823`)
- Bug in `read_csv` and `read_table` misinterpreting arguments when ``sys.setprofile`` had been previously called (`41069`)
- Bug in the conversion from PyArrow to pandas (e.g. for reading Parquet) with nullable dtypes and a PyArrow array whose data buffer size is not a multiple of the dtype size (`40896`)
- Bug in `read_excel` would raise an error when pandas could not determine the file type even though the user specified the ``engine`` argument (`41225`)
- Bug in `read_clipboard` copying from an excel file shifts values into the wrong column if there are null values in first column (`41108`)
- Bug in `DataFrame.to_hdf` and `Series.to_hdf` raising a ``TypeError`` when trying to append a string column to an incompatible column (`41897`)  Period
^^^
- Comparisons of `Period` objects or `Index`, `Series`, or `DataFrame` with mismatched ``PeriodDtype`` now behave like other mismatched-type comparisons, returning ``False`` for equals, ``True`` for not-equal, and raising ``TypeError`` for inequality checks (`39274`)  Plotting
^^^
- Bug in `plotting.scatter_matrix` raising when 2d ``ax`` argument passed (`16253`)
- Prevent warnings when Matplotlib's ``constrained_layout`` is enabled (`25261`)
- Bug in `DataFrame.plot` was showing the wrong colors in the legend if the function was called repeatedly and some calls used ``yerr`` while others didn't (`39522`)

- Bug in `DataFrame.plot` was showing the wrong colors in the legend if the function was called repeatedly and some calls used ``secondary_y`` and others use ``legend=False`` (`40044`)
- Bug in `DataFrame.plot.box` when ``dark_background`` theme was selected, caps or min/max markers for the plot were not visible (`40769`)  Groupby/resample/rolling
^^^
- Bug in `.DataFrameGroupBy.agg` and `.SeriesGroupBy.agg` with `PeriodDtype` columns incorrectly casting results too aggressively (`38254`)
- Bug in `.SeriesGroupBy.value_counts` where unobserved categories in a grouped categorical Series were not tallied (`38672`)
- Bug in `.SeriesGroupBy.value_counts` where an error was raised on an empty Series (`39172`)
- Bug in `.GroupBy.indices` would contain non-existent indices when null values were present in the groupby keys (`9304`)
- Fixed bug in `.DataFrameGroupBy.sum` and `.SeriesGroupBy.sum` causing a loss of precision by now using Kahan summation (`38778`)
- Fixed bug in `.DataFrameGroupBy.cumsum`, `.SeriesGroupBy.cumsum`, `.DataFrameGroupBy.mean`, and `.SeriesGroupBy.mean` causing loss of precision through using Kahan summation (`38934`)
- Bug in `.Resampler.aggregate` and `DataFrame.transform` raising a ``TypeError`` instead of ``SpecificationError`` when missing keys had mixed dtypes (`39025`)
- Bug in `.DataFrameGroupBy.idxmin` and `.DataFrameGroupBy.idxmax` with ``ExtensionDtype`` columns (`38733`)
- Bug in `Series.resample` would raise when the index was a `PeriodIndex` consisting of ``NaT`` (`39227`)
- Bug in `.RollingGroupby.corr` and `.ExpandingGroupby.corr` where the groupby column would return ``0`` instead of ``np.nan`` when providing ``other`` that was longer than each group (`39591`)
- Bug in `.ExpandingGroupby.corr` and `.ExpandingGroupby.cov` where ``1`` would be returned instead of ``np.nan`` when providing ``other`` that was longer than each group (`39591`)
- Bug in `.DataFrameGroupBy.mean`, `.SeriesGroupBy.mean`, `.DataFrameGroupBy.median`, `.SeriesGroupBy.median`, and `DataFrame.pivot_table` not propagating metadata (`28283`)
- Bug in `Series.rolling` and `DataFrame.rolling` not calculating window bounds correctly when window is an offset and dates are in descending order (`40002`)
- Bug in `Series.groupby` and `DataFrame.groupby` on an empty ``Series`` or ``DataFrame`` would lose index, columns, and/or data types when directly using the methods ``idxmax``, ``idxmin``, ``mad``, ``min``, ``max``, ``sum``, ``prod``, and ``skew`` or using them through ``apply``, ``aggregate``, or ``resample`` (`26411`)
- Bug in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` where a `MultiIndex` would be created instead of an `Index` when used on a `.RollingGroupby` object (`39732`)
- Bug in `.DataFrameGroupBy.sample` where an error was raised when ``weights`` was specified and the index was an `Int64Index` (`39927`)
- Bug in `.DataFrameGroupBy.aggregate` and `.Resampler.aggregate` would sometimes raise a ``SpecificationError`` when passed a dictionary and columns were missing; will now always raise a ``KeyError`` instead (`40004`)
- Bug in `.DataFrameGroupBy.sample` where column selection was not applied before computing the result (`39928`)
- Bug in `.ExponentialMovingWindow` when calling ``__getitem__`` would incorrectly raise a ``ValueError`` when providing ``times`` (`40164`)
- Bug in `.ExponentialMovingWindow` when calling ``__getitem__`` would not retain ``com``, ``span``, ``alpha`` or ``halflife`` attributes  (`40164`)
- `.ExponentialMovingWindow` now raises a ``NotImplementedError`` when specifying ``times`` with ``adjust=False`` due to an incorrect calculation (`40098`)
- Bug in `.ExponentialMovingWindowGroupby.mean` where the ``times`` argument was ignored when ``engine='numba'`` (`40951`)

- Bug in `.ExponentialMovingWindowGroupby.mean` where the wrong times were used the in case of multiple groups (`40951`)
- Bug in `.ExponentialMovingWindowGroupby` where the times vector and values became out of sync for non-trivial groups (`40951`)
- Bug in `Series.asfreq` and `DataFrame.asfreq` dropping rows when the index was not sorted (`39805`)
- Bug in aggregation functions for `DataFrame` not respecting ``numeric_only`` argument when ``level`` keyword was given (`40660`)
- Bug in `.SeriesGroupBy.aggregate` where using a user-defined function to aggregate a Series with an object-typed `Index` causes an incorrect `Index` shape (`40014`)
- Bug in `.RollingGroupby` where ``as_index=False`` argument in ``groupby`` was ignored (`39433`)
- Bug in `.DataFrameGroupBy.any`, `.SeriesGroupBy.any`, `.DataFrameGroupBy.all` and `.SeriesGroupBy.all` raising a ``ValueError`` when using with nullable type columns holding ``NA`` even with ``skipna=True`` (`40585`)
- Bug in `.DataFrameGroupBy.cummin`, `.SeriesGroupBy.cummin`, `.DataFrameGroupBy.cummax` and `.SeriesGroupBy.cummax` incorrectly rounding integer values near the ``int64`` implementations bounds (`40767`)
- Bug in `.DataFrameGroupBy.rank` and `.SeriesGroupBy.rank` with nullable dtypes incorrectly raising a ``TypeError`` (`41010`)
- Bug in `.DataFrameGroupBy.cummin`, `.SeriesGroupBy.cummin`, `.DataFrameGroupBy.cummax` and `.SeriesGroupBy.cummax` computing wrong result with nullable data types too large to roundtrip when casting to float (`37493`)
- Bug in `DataFrame.rolling` returning mean zero for all ``NaN`` window with ``min_periods=0`` if calculation is not numerical stable (`41053`)
- Bug in `DataFrame.rolling` returning sum not zero for all ``NaN`` window with ``min_periods=0`` if calculation is not numerical stable (`41053`)
- Bug in `.SeriesGroupBy.agg` failing to retain ordered `CategoricalDtype` on order-preserving aggregations (`41147`)
- Bug in `.DataFrameGroupBy.min`, `.SeriesGroupBy.min`, `.DataFrameGroupBy.max` and `.SeriesGroupBy.max` with multiple object-dtype columns and ``numeric_only=False`` incorrectly raising a ``ValueError`` (`4111`)
- Bug in `.DataFrameGroupBy.rank` with the GroupBy object's ``axis=0`` and the ``rank`` method's keyword ``axis=1`` (`41320`)
- Bug in `DataFrameGroupBy.__getitem__` with non-unique columns incorrectly returning a malformed `SeriesGroupBy` instead of `DataFrameGroupBy` (`41427`)
- Bug in `.DataFrameGroupBy.transform` with non-unique columns incorrectly raising an ``AttributeError`` (`41427`)
- Bug in `.Resampler.apply` with non-unique columns incorrectly dropping duplicated columns (`41445`)
- Bug in `Series.groupby` aggregations incorrectly returning empty `Series` instead of raising ``TypeError`` on aggregations that are invalid for its dtype, e.g. ``.prod`` with ``datetime64[ns]`` dtype (`41342`)
- Bug in `DataFrameGroupBy` aggregations incorrectly failing to drop columns with invalid dtypes for that aggregation when there are no valid columns (`41291`)
- Bug in `DataFrame.rolling.__iter__` where ``on`` was not assigned to the index of the resulting objects (`40373`)
- Bug in `.DataFrameGroupBy.transform` and `.DataFrameGroupBy.agg` with ``engine="numba"`` where ``*args`` were being cached with the user passed function (`41647`)
- Bug in `DataFrameGroupBy` methods ``agg``, ``transform``, ``sum``, ``bfill``, ``ffill``, ``pad``, ``pct_change``, ``shift``, ``ohlc`` dropping ``.columns.names`` (`41497`)  Reshaping
^^^
- Bug in `merge` raising error when performing an inner join with partial index and ``right_index=True`` when there was no overlap between indices (`33814`)
- Bug in `DataFrame.unstack` with missing levels led to incorrect index names (`37510`)
- Bug in `merge_asof` propagating the right Index with ``left_index=True`` and ``right_on`` specification instead of left Index (`33463`)

- Bug in `DataFrame.join` on a DataFrame with a `MultiIndex` returned the wrong result when one of both indexes had only one level (`36909`)
- `merge_asof` now raises a ``ValueError`` instead of a cryptic ``TypeError`` in case of non-numerical merge columns (`29130`)
- Bug in `DataFrame.join` not assigning values correctly when the DataFrame had a `MultiIndex` where at least one dimension had dtype ``Categorical`` with non-alphabetically sorted categories (`38502`)
- `Series.value_counts` and `Series.mode` now return consistent keys in original order (`12679`, `11227` and `39007`)
- Bug in `DataFrame.stack` not handling ``NaN`` in `MultiIndex` columns correctly (`39481`)
- Bug in `DataFrame.apply` would give incorrect results when the argument ``func`` was a string, ``axis=1``, and the axis argument was not supported; now raises a ``ValueError`` instead (`39211`)
- Bug in `DataFrame.sort_values` not reshaping the index correctly after sorting on columns when ``ignore_index=True`` (`39464`)
- Bug in `DataFrame.append` returning incorrect dtypes with combinations of ``ExtensionDtype`` dtypes (`39454`)
- Bug in `DataFrame.append` returning incorrect dtypes when used with combinations of ``datetime64`` and ``timedelta64`` dtypes (`39574`)
- Bug in `DataFrame.append` with a `DataFrame` with a `MultiIndex` and appending a `Series` whose `Index` is not a `MultiIndex` (`41707`)
- Bug in `DataFrame.pivot_table` returning a `MultiIndex` for a single value when operating on an empty DataFrame (`13483`)
- `Index` can now be passed to the `numpy.all` function (`40180`)
- Bug in `DataFrame.stack` not preserving ``CategoricalDtype`` in a `MultiIndex` (`36991`)
- Bug in `to_datetime` raising an error when the input sequence contained unhashable items (`39756`)
- Bug in `Series.explode` preserving the index when ``ignore_index`` was ``True`` and values were scalars (`40487`)
- Bug in `to_datetime` raising a ``ValueError`` when `Series` contains ``None`` and ``NaT`` and has more than 50 elements (`39882`)
- Bug in `Series.unstack` and `DataFrame.unstack` with object-dtype values containing timezone-aware datetime objects incorrectly raising ``TypeError`` (`41875`)
- Bug in `DataFrame.melt` raising ``InvalidIndexError`` when `DataFrame` has duplicate columns used as ``value_vars`` (`41951`)  Sparse
^^^
- Bug in `DataFrame.sparse.to_coo` raising a ``KeyError`` with columns that are a numeric `Index` without a ``0`` (`18414`)
- Bug in `SparseArray.astype` with ``copy=False`` producing incorrect results when going from integer dtype to floating dtype (`34456`)
- Bug in `SparseArray.max` and `SparseArray.min` would always return an empty result (`40921`)  ExtensionArray
^^^
- Bug in `DataFrame.where` when ``other`` is a Series with an `ExtensionDtype` (`38729`)
- Fixed bug where `Series.idxmax`, `Series.idxmin`, `Series.argmax`, and `Series.argmin` would fail when the underlying data is an `ExtensionArray` (`32749`, `33719`, `36566`)
- Fixed bug where some properties of subclasses of `PandasExtensionDtype` where improperly cached (`40329`)
- Bug in `DataFrame.mask` where masking a DataFrame with an `ExtensionDtype` raises a ``ValueError`` (`40941`)  Styler
^^^
- Bug in `.Styler` where the ``subset`` argument in methods raised an error for some valid MultiIndex slices (`33562`)
- `.Styler` rendered HTML output has seen minor alterations to support w3 good code standards (`39626`)
- Bug in `.Styler` where rendered HTML was missing a column class identifier for certain header cells (`39716`)
- Bug in `.Styler.background_gradient` where text-color was not determined correctly (`39888`)
- Bug in `.Styler.set_table_styles` where multiple elements in CSS-selectors of the ``table_styles`` argument were not correctly added (`34061`)
- Bug in `.Styler` where copying from Jupyter dropped the top left cell and misaligned headers (`12147`)

- Bug in `Styler.where` where ``kwargs`` were not passed to the applicable callable (`40845`)
- Bug in `.Styler` causing CSS to duplicate on multiple renders (`39395`, `40334`)  Other
^^^
- ``inspect.getmembers(Series)`` no longer raises an ``AbstractMethodError`` (`38782`)
- Bug in `Series.where` with numeric dtype and ``other=None`` not casting to ``nan`` (`39761`)
- Bug in `.assert_series_equal`, `.assert_frame_equal`, `.assert_index_equal` and `.assert_extension_array_equal` incorrectly raising when an attribute has an unrecognized NA type (`39461`)
- Bug in `.assert_index_equal` with ``exact=True`` not raising when comparing `CategoricalIndex` instances with ``Int64Index`` and ``RangeIndex`` categories (`41263`)
- Bug in `DataFrame.equals`, `Series.equals`, and `Index.equals` with object-dtype containing ``np.datetime64("NaT")`` or ``np.timedelta64("NaT")`` (`39650`)
- Bug in `show_versions` where console JSON output was not proper JSON (`39701`)
- pandas can now compile on z/OS when using xlc (`35826`)
- Bug in `pandas.util.hash_pandas_object` not recognizing ``hash_key``, ``encoding`` and ``categorize`` when the input object type is a `DataFrame` (`41404`)  .. ---  .. _whatsnew_130.contributors:  Contributors
~~~  .. contributors:: v1.2.5..v1.3.0

What's new in 1.3.1 (July 25, 2021)

These are the changes in pandas 1.3.1. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_131.regressions:  Fixed regressions
~~~
- Pandas could not be built on PyPy (`42355`)
- `DataFrame` constructed with an older version of pandas could not be unpickled (`42345`)
- Performance regression in constructing a `DataFrame` from a dictionary of dictionaries (`42248`)
- Fixed regression in `DataFrame.agg` dropping values when the DataFrame had an Extension Array dtype, a duplicate index, and ``axis=1`` (`42380`)
- Fixed regression in `DataFrame.astype` changing the order of noncontiguous data (`42396`)
- Performance regression in `DataFrame` in reduction operations requiring casting such as `DataFrame.mean` on integer data (`38592`)
- Performance regression in `DataFrame.to_dict` and `Series.to_dict` when ``orient`` argument one of "records", "dict", or "split" (`42352`)
- Fixed regression in indexing with a ``list`` subclass incorrectly raising ``TypeError`` (`42433`, `42461`)
- Fixed regression in `DataFrame.isin` and `Series.isin` raising ``TypeError`` with nullable data containing at least one missing value (`42405`)
- Regression in `concat` between objects with bool dtype and integer dtype casting to object instead of to integer (`42092`)
- Bug in `Series` constructor not accepting a ``dask.Array`` (`38645`)
- Fixed regression for ``SettingWithCopyWarning`` displaying incorrect stacklevel (`42570`)
- Fixed regression for `merge_asof` raising ``KeyError`` when one of the ``by`` columns is in the index (`34488`)
- Fixed regression in `to_datetime` returning pd.NaT for inputs that produce duplicated values, when ``cache=True`` (`42259`)
- Fixed regression in `SeriesGroupBy.value_counts` that resulted in an ``IndexError`` when called on a Series with one row (`42618`)  .. ---  .. _whatsnew_131.bug_fixes:  Bug fixes
~~~
- Fixed bug in `DataFrame.transpose` dropping values when the DataFrame had an Extension Array dtype and a duplicate index (`42380`)
- Fixed bug in `DataFrame.to_xml` raising ``KeyError`` when called with ``index=False`` and an offset index (`42458`)
- Fixed bug in `.Styler.set_sticky` not handling index names correctly for single index columns case (`42537`)
- Fixed bug in `DataFrame.copy` failing to consolidate blocks in the result (`42579`)  .. ---  .. _whatsnew_131.contributors:  Contributors
~~~  .. contributors:: v1.3.0..v1.3.1

What's new in 1.3.2 (August 15, 2021)

These are the changes in pandas 1.3.2. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_132.regressions:  Fixed regressions
~~~
- Performance regression in `DataFrame.isin` and `Series.isin` for nullable data types (`42714`)
- Regression in updating values of `Series` using boolean index, created by using `DataFrame.pop` (`42530`)
- Regression in `DataFrame.from_records` with empty records (`42456`)
- Fixed regression in `DataFrame.shift` where ``TypeError`` occurred when shifting DataFrame created by concatenation of slices and fills with values (`42719`)
- Regression in `DataFrame.agg` when the ``func`` argument returned lists and ``axis=1`` (`42727`)
- Regression in `DataFrame.drop` does nothing if `MultiIndex` has duplicates and indexer is a tuple or list of tuples (`42771`)
- Fixed regression where `read_csv` raised a ``ValueError`` when parameters ``names`` and ``prefix`` were both set to ``None`` (`42387`)
- Fixed regression in comparisons between `Timestamp` object and ``datetime64`` objects outside the implementation bounds for nanosecond ``datetime64`` (`42794`)
- Fixed regression in `.Styler.highlight_min` and `.Styler.highlight_max` where ``pandas.NA`` was not successfully ignored (`42650`)
- Fixed regression in `concat` where ``copy=False`` was not honored in ``axis=1`` Series concatenation (`42501`)
- Regression in `Series.nlargest` and `Series.nsmallest` with nullable integer or float dtype (`42816`)
- Fixed regression in `Series.quantile` with `Int64Dtype` (`42626`)
- Fixed regression in `Series.groupby` and `DataFrame.groupby` where supplying the ``by`` argument with a Series named with a tuple would incorrectly raise (`42731`)  .. ---  .. _whatsnew_132.bug_fixes:  Bug fixes
~~~
- Bug in `read_excel` modifies the dtypes dictionary when reading a file with duplicate columns (`42462`)
- 1D slices over extension types turn into N-dimensional slices over ExtensionArrays (`42430`)
- Fixed bug in `Series.rolling` and `DataFrame.rolling` not calculating window bounds correctly for the first row when ``center=True`` and ``window`` is an offset that covers all the rows (`42753`)
- `.Styler.hide_columns` now hides the index name header row as well as column headers (`42101`)
- `.Styler.set_sticky` has amended CSS to control the column/index names and ensure the correct sticky positions (`42537`)
- Bug in de-serializing datetime indexes in PYTHONOPTIMIZED mode (`42866`)  .. ---  .. _whatsnew_132.contributors:  Contributors
~~~  .. contributors:: v1.3.1..v1.3.2

What's new in 1.3.3 (September 12, 2021)

These are the changes in pandas 1.3.3. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_133.regressions:  Fixed regressions
~~~
- Fixed regression in `DataFrame` constructor failing to broadcast for defined `Index` and len one list of `Timestamp` (`42810`)
- Fixed regression in `.DataFrameGroupBy.agg` and `.SeriesGroupBy.agg` incorrectly raising in some cases (`42390`)
- Fixed regression in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` where ``nan`` values were dropped even with ``dropna=False`` (`43205`)
- Fixed regression in `.DataFrameGroupBy.quantile` and `.SeriesGroupBy.quantile` which were failing with ``pandas.NA`` (`42849`)
- Fixed regression in `merge` where ``on`` columns with ``ExtensionDtype`` or ``bool`` data types were cast to ``object`` in ``right`` and ``outer`` merge (`40073`)
- Fixed regression in `RangeIndex.where` and `RangeIndex.putmask` raising ``AssertionError`` when result did not represent a `RangeIndex` (`43240`)
- Fixed regression in `read_parquet` where the ``fastparquet`` engine would not work properly with fastparquet 0.7.0 (`43075`)
- Fixed regression in `DataFrame.loc.__setitem__` raising ``ValueError`` when setting array as cell value (`43422`)
- Fixed regression in `is_list_like` where objects with ``__iter__`` set to ``None`` would be identified as iterable (`43373`)
- Fixed regression in `DataFrame.__getitem__` raising error for slice of `DatetimeIndex` when index is non monotonic (`43223`)
- Fixed regression in `.Resampler.aggregate` when used after column selection would raise if ``func`` is a list of aggregation functions (`42905`)
- Fixed regression in `DataFrame.corr` where Kendall correlation would produce incorrect results for columns with repeated values (`43401`)
- Fixed regression in `DataFrame.groupby` where aggregation on columns with object types dropped results on those columns (`42395`, `43108`)
- Fixed regression in `Series.fillna` raising ``TypeError`` when filling ``float`` ``Series`` with list-like fill value having a dtype which couldn't cast lostlessly (like ``float32`` filled with ``float64``) (`43424`)
- Fixed regression in `read_csv` raising ``AttributeError`` when the file handle is an ``tempfile.SpooledTemporaryFile`` object (`43439`)
- Fixed performance regression in `core.window.ewm.ExponentialMovingWindow.mean` (`42333`)  .. ---  .. _whatsnew_133.performance:  Performance improvements
~~~
- Performance improvement for `DataFrame.__setitem__` when the key or value is not a `DataFrame`, or key is not list-like (`43274`)  .. ---  .. _whatsnew_133.bug_fixes:  Bug fixes
~~~
- Fixed bug in `.DataFrameGroupBy.agg` and `.DataFrameGroupBy.transform` with ``engine="numba"`` where ``index`` data was not being correctly passed into ``func`` (`43133`)  .. ---  .. _whatsnew_133.contributors:  Contributors
~~~  .. contributors:: v1.3.2..v1.3.3

What's new in 1.3.4 (October 17, 2021)

These are the changes in pandas 1.3.4. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_134.regressions:  Fixed regressions
~~~
- Fixed regression in `DataFrame.convert_dtypes` incorrectly converts byte strings to strings (`43183`)
- Fixed regression in `.DataFrameGroupBy.agg` and `.SeriesGroupBy.agg` were failing silently with mixed data types along ``axis=1`` and `MultiIndex` (`43209`)
- Fixed regression in `merge` with integer and ``NaN`` keys failing with ``outer`` merge (`43550`)
- Fixed regression in `DataFrame.corr` raising ``ValueError`` with ``method="spearman"`` on 32-bit platforms (`43588`)
- Fixed performance regression in `MultiIndex.equals` (`43549`)
- Fixed performance regression in `.DataFrameGroupBy.first`, `.SeriesGroupBy.first`, `.DataFrameGroupBy.last`, and `.SeriesGroupBy.last` with `StringDtype` (`41596`)
- Fixed regression in `Series.cat.reorder_categories` failing to update the categories on the ``Series`` (`43232`)
- Fixed regression in `Series.cat.categories` setter failing to update the categories on the ``Series`` (`43334`)
- Fixed regression in `read_csv` raising ``UnicodeDecodeError`` exception when ``memory_map=True`` (`43540`)
- Fixed regression in `DataFrame.explode` raising ``AssertionError`` when ``column`` is any scalar which is not a string (`43314`)
- Fixed regression in `Series.aggregate` attempting to pass ``args`` and ``kwargs`` multiple times to the user supplied ``func`` in certain cases (`43357`)
- Fixed regression when iterating over a `DataFrame.groupby.rolling` object causing the resulting DataFrames to have an incorrect index if the input groupings were not sorted (`43386`)
- Fixed regression in `DataFrame.groupby.rolling.cov` and `DataFrame.groupby.rolling.corr` computing incorrect results if the input groupings were not sorted (`43386`)  .. ---  .. _whatsnew_134.bug_fixes:  Bug fixes
~~~
- Fixed bug in `pandas.DataFrame.groupby.rolling` and `pandas.api.indexers.FixedForwardWindowIndexer` leading to segfaults and window endpoints being mixed across groups (`43267`)
- Fixed bug in `.DataFrameGroupBy.mean` and `.SeriesGroupBy.mean` with datetimelike values including ``NaT`` values returning incorrect results (`43132`)
- Fixed bug in `Series.aggregate` not passing the first ``args`` to the user supplied ``func`` in certain cases (`43357`)
- Fixed memory leaks in `Series.rolling.quantile` and `Series.rolling.median` (`43339`)  .. ---  .. _whatsnew_134.other:  Other
~~~
- The minimum version of Cython needed to compile pandas is now ``0.29.24`` (`43729`)  .. ---  .. _whatsnew_134.contributors:  Contributors
~~~  .. contributors:: v1.3.3..v1.3.4

What's new in 1.3.5 (December 12, 2021)

These are the changes in pandas 1.3.5. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_135.regressions:  Fixed regressions
~~~
- Fixed regression in `Series.equals` when comparing floats with dtype object to None (`44190`)
- Fixed regression in `merge_asof` raising error when array was supplied as join key (`42844`)
- Fixed regression when resampling `DataFrame` with `DateTimeIndex` with empty groups and ``uint8``, ``uint16`` or ``uint32`` columns incorrectly raising ``RuntimeError`` (`43329`)
- Fixed regression in creating a `DataFrame` from a timezone-aware `Timestamp` scalar near a Daylight Savings Time transition (`42505`)
- Fixed performance regression in `read_csv` (`44106`)
- Fixed regression in `Series.duplicated` and `Series.drop_duplicates` when Series has `Categorical` dtype with boolean categories (`44351`)
- Fixed regression in `.DataFrameGroupBy.sum` and `.SeriesGroupBy.sum` with ``timedelta64[ns]`` dtype containing ``NaT`` failing to treat that value as NA (`42659`)
- Fixed regression in `.RollingGroupby.cov` and `.RollingGroupby.corr` when ``other`` had the same shape as each group would incorrectly return superfluous groups in the result (`42915`)  .. ---  .. _whatsnew_135.contributors:  Contributors
~~~  .. contributors:: v1.3.4..v1.3.5|HEAD

These are the changes in pandas 1.4.0. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_140.enhancements:  Enhancements
~~~  .. _whatsnew_140.enhancements.warning_lineno:  Improved warning messages
^^^  Previously, warning messages may have pointed to lines within the pandas
library. Running the script ``setting_with_copy_warning.py``  .. code-block:: python  import pandas as pd  df = pd.DataFrame({'a': [1, 2, 3]})  df[:2].loc[:, 'a'] = 5  with pandas 1.3 resulted in::  .../site-packages/pandas/core/indexing.py SettingWithCopyWarning:  A value is trying to be set on a copy of a slice from a DataFrame.  This made it difficult to determine where the warning was being generated from.
Now pandas will inspect the call stack, reporting the first line outside of the
pandas library that gave rise to the warning. The output of the above script is
now::  setting_with_copy_warning.py SettingWithCopyWarning:  A value is trying to be set on a copy of a slice from a DataFrame.  .. _whatsnew_140.enhancements.ExtensionIndex:  Index can hold arbitrary ExtensionArrays
^^^  Until now, passing a custom `ExtensionArray` to ``pd.Index`` would cast
the array to ``object`` dtype. Now `Index` can directly hold arbitrary
ExtensionArrays (`43930`).  *Previous behavior*:  .. ipython:: python  arr = pd.array([1, 2, pd.NA])  idx = pd.Index(arr)  In the old behavior, ``idx`` would be object-dtype:  *Previous behavior*:  .. code-block:: ipython  In [1]: idx  Out[1]: Index([1, 2, <NA>], dtype='object')  With the new behavior, we keep the original dtype:  *New behavior*:  .. ipython:: python  idx  One exception to this is ``SparseArray``, which will continue to cast to numpy
dtype until pandas 2.0. At that point it will retain its dtype like other
ExtensionArrays.  .. _whatsnew_140.enhancements.styler:  Styler

^^^  `.Styler` has been further developed in 1.4.0. The following general enhancements have been made:  - Styling and formatting of indexes has been added, with `.Styler.apply_index`, `.Styler.applymap_index` and `.Styler.format_index`. These mirror the signature of the methods already used to style and format data values, and work with both HTML, LaTeX and Excel format (`41893`, `43101`, `41993`, `41995`)  - The new method `.Styler.hide` deprecates `.Styler.hide_index` and `.Styler.hide_columns` (`43758`)  - The keyword arguments ``level`` and ``names`` have been added to `.Styler.hide` (and implicitly to the deprecated methods `.Styler.hide_index` and `.Styler.hide_columns`) for additional control of visibility of MultiIndexes and of Index names (`25475`, `43404`, `43346`)  - The `.Styler.export` and `.Styler.use` have been updated to address all of the added functionality from v1.2.0 and v1.3.0 (`40675`)  - Global options under the category ``pd.options.styler`` have been extended to configure default ``Styler`` properties which address formatting, encoding, and HTML and LaTeX rendering. Note that formerly ``Styler`` relied on ``display.html.use_mathjax``, which has now been replaced by ``styler.html.mathjax`` (`41395`)  - Validation of certain keyword arguments, e.g. ``caption`` (`43368`)  - Various bug fixes as recorded below  Additionally there are specific enhancements to the HTML specific rendering:  - `.Styler.bar` introduces additional arguments to control alignment and display (`26070`, `36419`), and it also validates the input arguments ``width`` and ``height`` (`42511`)  - `.Styler.to_html` introduces keyword arguments ``sparse_index``, ``sparse_columns``, ``bold_headers``, ``caption``, ``max_rows`` and ``max_columns`` (`41946`, `43149`, `42972`)  - `.Styler.to_html` omits CSSStyle rules for hidden table elements as a performance enhancement (`43619`)  - Custom CSS classes can now be directly specified without string replacement (`43686`)  - Ability to render hyperlinks automatically via a new ``hyperlinks`` formatting keyword argument (`45058`)  There are also some LaTeX specific enhancements:  - `.Styler.to_latex` introduces keyword argument ``environment``, which also allows a specific "longtable" entry through a separate jinja2 template (`41866`)  - Naive sparsification is now possible for LaTeX without the necessity of including the multirow package (`43369`)  - *cline* support has been added for `MultiIndex` row sparsification through a keyword argument (`45138`)  .. _whatsnew_140.enhancements.pyarrow_csv_engine:  Multi-threaded CSV reading with a new CSV Engine based on pyarrow
^^^  `pandas.read_csv` now accepts ``engine="pyarrow"`` (requires at least
``pyarrow`` 1.0.1) as an argument, allowing for faster csv parsing on multicore
machines with pyarrow installed. See the `I/O docs </user_guide/io>` for
more info. (`23697`, `43706`)  .. _whatsnew_140.enhancements.window_rank:  Rank function for rolling and expanding windows
^^^  Added ``rank`` function to `Rolling` and `Expanding`. The new
function supports the ``method``, ``ascending``, and ``pct`` flags of
`DataFrame.rank`. The ``method`` argument supports ``min``, ``max``, and
``average`` ranking methods.
Example:  .. ipython:: python  s = pd.Series([1, 4, 2, 3, 5, 3])  s.rolling(3).rank()  s.rolling(3).rank(method="max")  .. _whatsnew_140.enhancements.groupby_indexing:  Groupby positional indexing
^^^  It is now possible to specify positional ranges relative to the ends of each
group.  Negative arguments for `.DataFrameGroupBy.head`, `.SeriesGroupBy.head`, `.DataFrameGroupBy.tail`, and `.SeriesGroupBy.tail` now work
correctly and result in ranges relative to the end and start of each group,

respectively. Previously, negative arguments returned empty frames.  .. ipython:: python  df = pd.DataFrame([["g", "g0"], ["g", "g1"], ["g", "g2"], ["g", "g3"],  ["h", "h0"], ["h", "h1"]], columns=["A", "B"])  df.groupby("A").head(-1)  `.DataFrameGroupBy.nth` and `.SeriesGroupBy.nth` now accept a slice or list of integers and slices.  .. ipython:: python  df.groupby("A").nth(slice(1, -1))  df.groupby("A").nth([slice(None, 1), slice(-1, None)])  `.DataFrameGroupBy.nth` and `.SeriesGroupBy.nth` now accept index notation.  .. ipython:: python  df.groupby("A").nth[1, -1]  df.groupby("A").nth[1:-1]  df.groupby("A").nth[:1, -1:]  .. _whatsnew_140.dict_tight:  DataFrame.from_dict and DataFrame.to_dict have new ``'tight'`` option
^^^  A new ``'tight'`` dictionary format that preserves `MultiIndex` entries
and names is now available with the `DataFrame.from_dict` and
`DataFrame.to_dict` methods and can be used with the standard ``json``
library to produce a tight representation of `DataFrame` objects
(`4889`).  .. ipython:: python  df = pd.DataFrame.from_records(  [[1, 3], [2, 4]],  index=pd.MultiIndex.from_tuples([("a", "b"), ("a", "c")],  names=["n1", "n2"]),  columns=pd.MultiIndex.from_tuples([("x", 1), ("y", 2)],  names=["z1", "z2"]),  )  df  df.to_dict(orient='tight')  .. _whatsnew_140.enhancements.other:  Other enhancements
^^^
- `concat` will preserve the ``attrs`` when it is the same for all objects and discard the ``attrs`` when they are different (`41828`)
- `DataFrameGroupBy` operations with ``as_index=False`` now correctly retain ``ExtensionDtype`` dtypes for columns being grouped on (`41373`)
- Add support for assigning values to ``by`` argument in `DataFrame.plot.hist` and `DataFrame.plot.box` (`15079`)
- `Series.sample`, `DataFrame.sample`, `.DataFrameGroupBy.sample`, and `.SeriesGroupBy.sample` now accept a ``np.random.Generator`` as input to ``random_state``. A generator will be more performant, especially with ``replace=False`` (`38100`)
- `Series.ewm` and `DataFrame.ewm` now support a ``method`` argument with a ``'table'`` option that performs the windowing operation over an entire `DataFrame`. See `Window Overview <window.overview>` for performance and functional benefits (`42273`)
- `.DataFrameGroupBy.cummin`, `.SeriesGroupBy.cummin`, `.DataFrameGroupBy.cummax`, and `.SeriesGroupBy.cummax` now support the argument ``skipna`` (`34047`)
- `read_table` now supports the argument ``storage_options`` (`39167`)
- `DataFrame.to_stata` and `StataWriter` now accept the keyword only argument ``value_labels`` to save labels for non-categorical columns (`38454`)
- Methods that relied on hashmap based algos such as `DataFrameGroupBy.value_counts`, `DataFrameGroupBy.count` and `factorize` ignored imaginary component for complex numbers (`17927`)
- Add `Series.str.removeprefix` and `Series.str.removesuffix` introduced in Python 3.9 to remove pre-/suffixes from string-type `Series` (`36944`)
- Attempting to write into a file in missing parent directory with `DataFrame.to_csv`, `DataFrame.to_html`, `DataFrame.to_excel`, `DataFrame.to_feather`, `DataFrame.to_parquet`, `DataFrame.to_stata`, `DataFrame.to_json`, `DataFrame.to_pickle`, and `DataFrame.to_xml` now explicitly mentions missing parent directory, the same is true for `Series` counterparts (`24306`)
- Indexing with ``.loc`` and ``.iloc`` now supports ``Ellipsis`` (`37750`)
- `IntegerArray.all` , `IntegerArray.any`, `FloatingArray.any`, and `FloatingArray.all` use Kleene logic (`41967`)
- Added support for nullable boolean and integer types in `DataFrame.to_stata`, `~pandas.io.stata.StataWriter`, `~pandas.io.stata.StataWriter117`, and `~pandas.io.stata.StataWriterUTF8` (`40855`)
- `DataFrame.__pos__` and `DataFrame.__neg__` now retain ``ExtensionDtype`` dtypes (`43883`)
- The error raised when an optional dependency can't be imported now includes the original exception, for easier investigation (`43882`)

- Added `.ExponentialMovingWindow.sum` (`13297`)
- `Series.str.split` now supports a ``regex`` argument that explicitly specifies whether the pattern is a regular expression. Default is ``None`` (`43563`, `32835`, `25549`)
- `DataFrame.dropna` now accepts a single label as ``subset`` along with array-like (`41021`)
- Added `DataFrameGroupBy.value_counts` (`43564`)
- `read_csv` now accepts a ``callable`` function in ``on_bad_lines`` when ``engine="python"`` for custom handling of bad lines (`5686`)
- `ExcelWriter` argument ``if_sheet_exists="overlay"`` option added (`40231`)
- `read_excel` now accepts a ``decimal`` argument that allow the user to specify the decimal point when parsing string columns to numeric (`14403`)
- .DataFrameGroupBy.mean`, `.SeriesGroupBy.mean`, `.DataFrameGroupBy.std`, `.SeriesGroupBy.std`, `.DataFrameGroupBy.var`, `.SeriesGroupBy.var`, `.DataFrameGroupBy.sum`, and `.SeriesGroupBy.sum` now support `Numba execution with the ``engine`` keyword (`43731`, `44862`, `44939`)
- `Timestamp.isoformat` now handles the ``timespec`` argument from the base ``datetime`` class (`26131`)
- `NaT.to_numpy` ``dtype`` argument is now respected, so ``np.timedelta64`` can be returned (`44460`)
- New option ``display.max_dir_items`` customizes the number of columns added to `Dataframe.__dir__` and suggested for tab completion (`37996`)
- Added "Juneteenth National Independence Day" to ``USFederalHolidayCalendar`` (`44574`)
- .Rolling.var`, `.Expanding.var`, `.Rolling.std`, and `.Expanding.std` now support `Numba execution with the ``engine`` keyword (`44461`)
- `Series.info` has been added, for compatibility with `DataFrame.info` (`5167`)
- Implemented `IntervalArray.min` and `IntervalArray.max`, as a result of which ``min`` and ``max`` now work for `IntervalIndex`, `Series` and `DataFrame` with ``IntervalDtype`` (`44746`)
- `UInt64Index.map` now retains ``dtype`` where possible (`44609`)
- `read_json` can now parse unsigned long long integers (`26068`)
- `DataFrame.take` now raises a ``TypeError`` when passed a scalar for the indexer (`42875`)
- `is_list_like` now identifies duck-arrays as list-like unless ``.ndim == 0`` (`35131`)
- `ExtensionDtype` and `ExtensionArray` are now (de)serialized when exporting a `DataFrame` with `DataFrame.to_json` using ``orient='table'`` (`20612`, `44705`)
- Add support for Zstandard compression to `DataFrame.to_pickle`/`read_pickle` and friends (`43925`)
- `DataFrame.to_sql` now returns an ``int`` of the number of written rows (`23998`)  .. ---  .. _whatsnew_140.notable_bug_fixes:  Notable bug fixes
~~~  These are bug fixes that might have notable behavior changes.  .. _whatsnew_140.notable_bug_fixes.inconsistent_date_string_parsing:  Inconsistent date string parsing
^^^  The ``dayfirst`` option of `to_datetime` isn't strict, and this can lead
to surprising behavior:  .. ipython:: python  pd.to_datetime(["31-12-2021"], dayfirst=False)  Now, a warning will be raised if a date string cannot be parsed accordance to
the given ``dayfirst`` value when the value is a delimited date string (e.g.
``31-12-2012``).  .. _whatsnew_140.notable_bug_fixes.concat_with_empty_or_all_na:  Ignoring dtypes in concat with empty or all-NA columns
^^^  .. note::  This behaviour change has been reverted in pandas 1.4.3.  When using `concat` to concatenate two or more `DataFrame` objects,
if one of the DataFrames was empty or had all-NA values, its dtype was
*sometimes* ignored when finding the concatenated dtype.  These are now
consistently *not* ignored (`43507`).  .. ipython:: python  df1 = pd.DataFrame({"bar": [pd.Timestamp("2013-01-01")]}, index=range(1))  df2 = pd.DataFrame({"bar": np.nan}, index=range(1, 2))  res = pd.concat([df1, df2])  Previously, the float-dtype in ``df2`` would be ignored so the result dtype
would be ``datetime64[ns]``. As a result, the ``np.nan`` would be cast to

``NaT``.  *Previous behavior*:  .. code-block:: ipython  In [4]: res  Out[4]:  bar  0 2013-01-01  1  NaT  Now the float-dtype is respected. Since the common dtype for these DataFrames is
object, the ``np.nan`` is retained.  *New behavior*:  .. code-block:: ipython  In [4]: res  Out[4]:  bar  0  2013-01-01 000  1  NaN  .. _whatsnew_140.notable_bug_fixes.value_counts_and_mode_do_not_coerce_to_nan:  Null-values are no longer coerced to NaN-value in value_counts and mode
^^^  `Series.value_counts` and `Series.mode` no longer coerce ``None``,
``NaT`` and other null-values to a NaN-value for ``np.object_``-dtype. This
behavior is now consistent with ``unique``, ``isin`` and others
(`42688`).  .. ipython:: python  s = pd.Series([True, None, pd.NaT, None, pd.NaT, None])  res = s.value_counts(dropna=False)  Previously, all null-values were replaced by a NaN-value.  *Previous behavior*:  .. code-block:: ipython  In [3]: res  Out[3]:  NaN  5  True  1  dtype: int64  Now null-values are no longer mangled.  *New behavior*:  .. ipython:: python  res  .. _whatsnew_140.notable_bug_fixes.read_csv_mangle_dup_cols:  mangle_dupe_cols in read_csv no longer renames unique columns conflicting with target names
^^^  `read_csv` no longer renames unique column labels which conflict with the target
names of duplicated columns. Already existing columns are skipped, i.e. the next
available index is used for the target column name (`14704`).  .. ipython:: python  import io  data = "a,a,a.1\n1,2,3"  res = pd.read_csv(io.StringIO(data))  Previously, the second column was called ``a.1``, while the third column was
also renamed to ``a.1.1``.  *Previous behavior*:  .. code-block:: ipython  In [3]: res  Out[3]:  a  a.1  a.1.1  0  1  2  3  Now the renaming checks if ``a.1`` already exists when changing the name of the
second column and jumps this index. The second column is instead renamed to
``a.2``.  *New behavior*:  .. ipython:: python  res  .. _whatsnew_140.notable_bug_fixes.unstack_pivot_int32_limit:  unstack and pivot_table no longer raises ValueError for result that would exceed int32 limit
^^^  Previously `DataFrame.pivot_table` and `DataFrame.unstack` would
raise a ``ValueError`` if the operation could produce a result with more than
``2**31 - 1`` elements. This operation now raises a
`errors.PerformanceWarning` instead (`26314`).  *Previous behavior*:  .. code-block:: ipython  In [3]: df = DataFrame({"ind1": np.arange(2 ** 16), "ind2": np.arange(2 ** 16), "count": 0})  In [4]: df.pivot_table(index="ind1", columns="ind2", values="count", aggfunc="count")  ValueError: Unstacked DataFrame is too big, causing int32 overflow  *New behavior*:  .. code-block:: python  In [4]: df.pivot_table(index="ind1", columns="ind2", values="count", aggfunc="count")  PerformanceWarning: The following operation may generate 4294967296 cells in the resulting pandas object.  .. ---  .. _whatsnew_140.notable_bug_fixes.groupby_apply_mutation:  groupby.apply consistent transform detection
^^^  `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` are designed to be flexible, allowing users to perform
aggregations, transformations, filters, and use it with user-defined functions
that might not fall into any of these categories. As part of this, apply will
attempt to detect when an operation is a transform, and in such a case, the
result will have the same index as the input. In order to determine if the
operation is a transform, pandas compares the input's index to the result's and
determines if it has been mutated. Previously in pandas 1.3, different code
paths used different definitions of "mutated": some would use Python's ``is``

whereas others would test only up to equality.  This inconsistency has been removed, pandas now tests up to equality.  .. ipython:: python  def func(x):  return x.copy()  df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'c': [5, 6]})  df  *Previous behavior*:  .. code-block:: ipython  In [3]: df.groupby(['a']).apply(func)  Out[3]:  a  b  c  a  1 0  1  3  5  2 1  2  4  6  In [4]: df.set_index(['a', 'b']).groupby(['a']).apply(func)  Out[4]:  c  a b  1 3  5  2 4  6  In the examples above, the first uses a code path where pandas uses ``is`` and
determines that ``func`` is not a transform whereas the second tests up to
equality and determines that ``func`` is a transform. In the first case, the
result's index is not the same as the input's.  *New behavior*:  .. code-block:: ipython  In [5]: df.groupby(['a']).apply(func)  Out[5]:  a  b  c  0  1  3  5  1  2  4  6  In [6]: df.set_index(['a', 'b']).groupby(['a']).apply(func)  Out[6]:  c  a b  1 3  5  2 4  6  Now in both cases it is determined that ``func`` is a transform. In each case,
the result has the same index as the input.  .. _whatsnew_140.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_140.api_breaking.python:  Increased minimum version for Python
^^^  pandas 1.4.0 supports Python 3.8 and higher.  .. _whatsnew_140.api_breaking.deps:  Increased minimum versions for dependencies
^^^
| numpy  | 1.18.5  |  X  |  X  |
| pytz  | 2020.1  |  X  |  X  |
| python-dateutil | 2.8.1  |  X  |  X  |
| bottleneck  | 1.3.1  |  |  X  |
| numexpr  | 2.7.1  |  |  X  |
| pytest (dev)  | 6.0  |  |  |
| mypy (dev)  | 0.930  |  |  X  |
+---+---+---+---+  For `optional libraries
<https://pandas.pydata.org/docs/getting_started/install.html>`_ the general
recommendation is to use the latest version. The following table lists the
lowest version per library that is currently being tested throughout the
development of pandas. Optional libraries below the lowest tested version may
still work, but are not considered supported.  +---+---+---+
| beautifulsoup4  | 4.8.2  |  X  |
| fastparquet  | 0.4.0  |  |
| lxml  | 4.5.0  |  X  |
| matplotlib  | 3.3.2  |  X  |
| numba  | 0.50.1  |  X  |
| openpyxl  | 3.0.3  |  X  |
| pandas-gbq  | 0.14.0  |  X  |
| pyarrow  | 1.0.1  |  X  |
| pymysql  | 0.10.1  |  X  |
| pytables  | 3.6.1  |  X  |
| scipy  | 1.4.1  |  X  |
| sqlalchemy  | 1.4.0  |  X  |
| tabulate  | 0.8.7  |  |
| xarray  | 0.15.1  |  X  |
| xlrd  | 2.0.1  |  X  |
| xlsxwriter  | 1.2.2  |  X  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  .. _whatsnew_140.api_breaking.other:  Other API changes
^^^
- `Index.get_indexer_for` no longer accepts keyword arguments (other than ``target``); in the past these would be silently ignored if the index was not unique (`42310`)
- Change in the position of the ``min_rows`` argument in `DataFrame.to_string` due to change in the docstring (`44304`)
- Reduction operations for `DataFrame` or `Series` now raising a ``ValueError`` when ``None`` is passed for ``skipna`` (`44178`)
- `read_csv` and `read_html` no longer raising an error when one of the header rows consists only of ``Unnamed:`` columns (`13054`)

- Changed the ``name`` attribute of several holidays in  ``USFederalHolidayCalendar`` to match `official federal holiday  names <https://www.opm.gov/policy-data-oversight/pay-leave/federal-holidays/>`_  specifically:  - "New Year's Day" gains the possessive apostrophe  - "Presidents Day" becomes "Washington's Birthday"  - "Martin Luther King Jr. Day" is now "Birthday of Martin Luther King, Jr."  - "July 4th" is now "Independence Day"  - "Thanksgiving" is now "Thanksgiving Day"  - "Christmas" is now "Christmas Day"  - Added "Juneteenth National Independence Day"  .. ---  .. _whatsnew_140.deprecations:  Deprecations
~~~  .. _whatsnew_140.deprecations.int64_uint64_float64index:  Deprecated Int64Index, UInt64Index & Float64Index
^^^  `Int64Index`, `UInt64Index` and `Float64Index` have been
deprecated in favor of the base `Index` class and will be removed in
Pandas 2.0 (`43028`).  For constructing a numeric index, you can use the base `Index` class
instead specifying the data type (which will also work on older pandas
releases):  .. code-block:: python  # replace  pd.Int64Index([1, 2, 3])  # with  pd.Index([1, 2, 3], dtype="int64")  For checking the data type of an index object, you can replace ``isinstance``
checks with checking the ``dtype``:  .. code-block:: python  # replace  isinstance(idx, pd.Int64Index)  # with  idx.dtype == "int64"  Currently, in order to maintain backward compatibility, calls to `Index`
will continue to return `Int64Index`, `UInt64Index` and
`Float64Index` when given numeric data, but in the future, an
`Index` will be returned.  *Current behavior*:  .. code-block:: ipython  In [1]: pd.Index([1, 2, 3], dtype="int32")  Out [1]: Int64Index([1, 2, 3], dtype='int64')  In [1]: pd.Index([1, 2, 3], dtype="uint64")  Out [1]: UInt64Index([1, 2, 3], dtype='uint64')  *Future behavior*:  .. code-block:: ipython  In [3]: pd.Index([1, 2, 3], dtype="int32")  Out [3]: Index([1, 2, 3], dtype='int32')  In [4]: pd.Index([1, 2, 3], dtype="uint64")  Out [4]: Index([1, 2, 3], dtype='uint64')  .. _whatsnew_140.deprecations.frame_series_append:  Deprecated DataFrame.append and Series.append
^^^  `DataFrame.append` and `Series.append` have been deprecated and will
be removed in a future version. Use `pandas.concat` instead (`35407`).  *Deprecated syntax*  .. code-block:: ipython  In [1]: pd.Series([1, 2]).append(pd.Series([3, 4])  Out [1]:  <stdin> FutureWarning: The series.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.  0  1  1  2  0  3  1  4  dtype: int64  In [2]: df1 = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))  In [3]: df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))  In [4]: df1.append(df2)  Out [4]:  <stdin> FutureWarning: The series.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.  A  B  0  1  2  1  3  4  0  5  6  1  7  8  *Recommended syntax*  .. ipython:: python  pd.concat([pd.Series([1, 2]), pd.Series([3, 4])])  df1 = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))  df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))  pd.concat([df1, df2])  .. _whatsnew_140.deprecations.other:  Other Deprecations
^^^
- Deprecated `Index.is_type_compatible` (`42113`)
- Deprecated ``method`` argument in `Index.get_loc`, use ``index.get_indexer([label], method=...)`` instead (`42269`)
- Deprecated treating integer keys in `Series.__setitem__` as positional when the index is a `Float64Index` not containing the key, a `IntervalIndex` with no entries containing the key, or a `MultiIndex` with leading `Float64Index` level not containing the key (`33469`)

- Deprecated treating ``numpy.datetime64`` objects as UTC times when passed to the `Timestamp` constructor along with a timezone. In a future version, these will be treated as wall-times. To retain the old behavior, use ``Timestamp(dt64).tz_localize("UTC").tz_convert(tz)`` (`24559`)
- Deprecated ignoring missing labels when indexing with a sequence of labels on a level of a `MultiIndex` (`42351`)
- Creating an empty `Series` without a ``dtype`` will now raise a more visible ``FutureWarning`` instead of a ``DeprecationWarning`` (`30017`)
- Deprecated the ``kind`` argument in `Index.get_slice_bound`, `Index.slice_indexer`, and `Index.slice_locs`; in a future version passing ``kind`` will raise (`42857`)
- Deprecated dropping of nuisance columns in `Rolling`, `Expanding`, and `EWM` aggregations (`42738`)
- Deprecated `Index.reindex` with a non-unique `Index` (`42568`)
- Deprecated `.Styler.render` in favor of `.Styler.to_html` (`42140`)
- Deprecated `.Styler.hide_index` and `.Styler.hide_columns` in favor of `.Styler.hide` (`43758`)
- Deprecated passing in a string column label into ``times`` in `DataFrame.ewm` (`43265`)
- Deprecated the ``include_start`` and ``include_end`` arguments in `DataFrame.between_time`; in a future version passing ``include_start`` or ``include_end`` will raise (`40245`)
- Deprecated the ``squeeze`` argument to `read_csv`, `read_table`, and `read_excel`. Users should squeeze the `DataFrame` afterwards with ``.squeeze("columns")`` instead (`43242`)
- Deprecated the ``index`` argument to `SparseArray` construction (`23089`)
- Deprecated the ``closed`` argument in `date_range` and `bdate_range` in favor of ``inclusive`` argument; In a future version passing ``closed`` will raise (`40245`)
- Deprecated `.Rolling.validate`, `.Expanding.validate`, and `.ExponentialMovingWindow.validate` (`43665`)
- Deprecated silent dropping of columns that raised a ``TypeError`` in `Series.transform` and `DataFrame.transform` when used with a dictionary (`43740`)
- Deprecated silent dropping of columns that raised a ``TypeError``, ``DataError``, and some cases of ``ValueError`` in `Series.aggregate`, `DataFrame.aggregate`, `Series.groupby.aggregate`, and `DataFrame.groupby.aggregate` when used with a list (`43740`)
- Deprecated casting behavior when setting timezone-aware value(s) into a timezone-aware `Series` or `DataFrame` column when the timezones do not match. Previously this cast to object dtype. In a future version, the values being inserted will be converted to the series or column's existing timezone (`37605`)
- Deprecated casting behavior when passing an item with mismatched-timezone to `DatetimeIndex.insert`, `DatetimeIndex.putmask`, `DatetimeIndex.where` `DatetimeIndex.fillna`, `Series.mask`, `Series.where`, `Series.fillna`, `Series.shift`, `Series.replace`, `Series.reindex` (and `DataFrame` column analogues). In the past this has cast to object ``dtype``. In a future version, these will cast the passed item to the index or series's timezone (`37605`, `44940`)
- Deprecated the ``prefix`` keyword argument in `read_csv` and `read_table`, in a future version the argument will be removed (`43396`)
- Deprecated passing non boolean argument to ``sort`` in `concat` (`41518`)
- Deprecated passing arguments as positional for `read_fwf` other than ``filepath_or_buffer`` (`41485`)
- Deprecated passing arguments as positional for `read_xml` other than ``path_or_buffer`` (`45133`)
- Deprecated passing ``skipna=None`` for `DataFrame.mad` and `Series.mad`, pass ``skipna=True`` instead (`44580`)
- Deprecated the behavior of `to_datetime` with the string "now" with ``utc=False``; in a future version this will match ``Timestamp("now")``, which in turn matches `Timestamp.now` returning the local time (`18705`)
- Deprecated `DateOffset.apply`, use ``offset + other`` instead (`44522`)
- Deprecated parameter ``names`` in `Index.copy` (`44916`)

- A deprecation warning is now shown for `DataFrame.to_latex` indicating the arguments signature may change and emulate more the arguments to `.Styler.to_latex` in future versions (`44411`)
- Deprecated behavior of `concat` between objects with bool-dtype and numeric-dtypes; in a future version these will cast to object dtype instead of coercing bools to numeric values (`39817`)
- Deprecated `Categorical.replace`, use `Series.replace` instead (`44929`)
- Deprecated passing ``set`` or ``dict`` as indexer for `DataFrame.loc.__setitem__`, `DataFrame.loc.__getitem__`, `Series.loc.__setitem__`, `Series.loc.__getitem__`, `DataFrame.__getitem__`, `Series.__getitem__` and `Series.__setitem__` (`42825`)
- Deprecated `Index.__getitem__` with a bool key; use ``index.values[key]`` to get the old behavior (`44051`)
- Deprecated downcasting column-by-column in `DataFrame.where` with integer-dtypes (`44597`)
- Deprecated `DatetimeIndex.union_many`, use `DatetimeIndex.union` instead (`44091`)
- Deprecated `.Groupby.pad` in favor of `.Groupby.ffill` (`33396`)
- Deprecated `.Groupby.backfill` in favor of `.Groupby.bfill` (`33396`)
- Deprecated `.Resample.pad` in favor of `.Resample.ffill` (`33396`)
- Deprecated `.Resample.backfill` in favor of `.Resample.bfill` (`33396`)
- Deprecated ``numeric_only=None`` in `DataFrame.rank`; in a future version ``numeric_only`` must be either ``True`` or ``False`` (the default) (`45036`)
- Deprecated the behavior of `Timestamp.utcfromtimestamp`, in the future it will return a timezone-aware UTC `Timestamp` (`22451`)
- Deprecated `NaT.freq` (`45071`)
- Deprecated behavior of `Series` and `DataFrame` construction when passed float-dtype data containing ``NaN`` and an integer dtype ignoring the dtype argument; in a future version this will raise (`40110`)
- Deprecated the behaviour of `Series.to_frame` and `Index.to_frame` to ignore the ``name`` argument when ``name=None``. Currently, this means to preserve the existing name, but in the future explicitly passing ``name=None`` will set ``None`` as the name of the column in the resulting DataFrame (`44212`)  .. ---  .. _whatsnew_140.performance:  Performance improvements
~~~
- Performance improvement in `.DataFrameGroupBy.sample` and `.SeriesGroupBy.sample`, especially when ``weights`` argument provided (`34483`)
- Performance improvement when converting non-string arrays to string arrays (`34483`)
- Performance improvement in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` for user-defined functions (`41598`)
- Performance improvement in constructing `DataFrame` objects (`42631`, `43142`, `43147`, `43307`, `43144`, `44826`)
- Performance improvement in `.DataFrameGroupBy.shift` and `.SeriesGroupBy.shift` when ``fill_value`` argument is provided (`26615`)
- Performance improvement in `DataFrame.corr` for ``method=pearson`` on data without missing values (`40956`)
- Performance improvement in some `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` operations (`42992`, `43578`)
- Performance improvement in `read_stata` (`43059`, `43227`)
- Performance improvement in `read_sas` (`4333`)
- Performance improvement in `to_datetime` with ``uint`` dtypes (`42606`)
- Performance improvement in `to_datetime` with ``infer_datetime_format`` set to ``True`` (`43901`)
- Performance improvement in `Series.sparse.to_coo` (`42880`)
- Performance improvement in indexing with a `UInt64Index` (`43862`)
- Performance improvement in indexing with a `Float64Index` (`43705`)
- Performance improvement in indexing with a non-unique `Index` (`43792`)
- Performance improvement in indexing with a listlike indexer on a `MultiIndex` (`43370`)
- Performance improvement in indexing with a `MultiIndex` indexer on another `MultiIndex` (`43370`)
- Performance improvement in `.DataFrameGroupBy.quantile` and `.SeriesGroupBy.quantile` (`43469`, `43725`)

- Performance improvement in `.DataFrameGroupBy.count` and `.SeriesGroupBy.count` (`43730`, `43694`)
- Performance improvement in `.DataFrameGroupBy.any`, `.SeriesGroupBy.any`, `.DataFrameGroupBy.all`, and `.SeriesGroupBy.all` (`43675`, `42841`)
- Performance improvement in `.DataFrameGroupBy.std` and `.SeriesGroupBy.std` (`43115`, `43576`)
- Performance improvement in `.DataFrameGroupBy.cumsum` and `.SeriesGroupBy.cumsum` (`43309`)
- `SparseArray.min` and `SparseArray.max` no longer require converting to a dense array (`43526`)
- Indexing into a `SparseArray` with a ``slice`` with ``step=1`` no longer requires converting to a dense array (`43777`)
- Performance improvement in `SparseArray.take` with ``allow_fill=False`` (`43654`)
- Performance improvement in `.Rolling.mean`, `.Expanding.mean`, `.Rolling.sum`, `.Expanding.sum`, `.Rolling.max`, `.Expanding.max`, `.Rolling.min` and `.Expanding.min` with ``engine="numba"`` (`43612`, `44176`, `45170`)
- Improved performance of `pandas.read_csv` with ``memory_map=True`` when file encoding is UTF-8 (`43787`)
- Performance improvement in `RangeIndex.sort_values` overriding `Index.sort_values` (`43666`)
- Performance improvement in `RangeIndex.insert` (`43988`)
- Performance improvement in `Index.insert` (`43953`)
- Performance improvement in `DatetimeIndex.tolist` (`43823`)
- Performance improvement in `DatetimeIndex.union` (`42353`)
- Performance improvement in `Series.nsmallest` (`43696`)
- Performance improvement in `DataFrame.insert` (`42998`)
- Performance improvement in `DataFrame.dropna` (`43683`)
- Performance improvement in `DataFrame.fillna` (`43316`)
- Performance improvement in `DataFrame.values` (`43160`)
- Performance improvement in `DataFrame.select_dtypes` (`42611`)
- Performance improvement in `DataFrame` reductions (`43185`, `43243`, `43311`, `43609`)
- Performance improvement in `Series.unstack` and `DataFrame.unstack` (`43335`, `43352`, `42704`, `43025`)
- Performance improvement in `Series.to_frame` (`43558`)
- Performance improvement in `Series.mad` (`43010`)
- Performance improvement in `merge` (`43332`)
- Performance improvement in `to_csv` when index column is a datetime and is formatted (`39413`)
- Performance improvement in `to_csv` when `MultiIndex` contains a lot of unused levels (`37484`)
- Performance improvement in `read_csv` when ``index_col`` was set with a numeric column (`44158`)
- Performance improvement in `concat` (`43354`)
- Performance improvement in `SparseArray.__getitem__` (`23122`)
- Performance improvement in constructing a `DataFrame` from array-like objects like a ``Pytorch`` tensor (`44616`)  .. ---  .. _whatsnew_140.bug_fixes:  Bug fixes
^^^
- Bug in setting dtype-incompatible values into a `Categorical` (or ``Series`` or ``DataFrame`` backed by ``Categorical``) raising ``ValueError`` instead of ``TypeError`` (`41919`)
- Bug in `Categorical.searchsorted` when passing a dtype-incompatible value raising ``KeyError`` instead of ``TypeError`` (`41919`)
- Bug in `Categorical.astype` casting datetimes and `Timestamp` to int for dtype ``object`` (`44930`)
- Bug in `Series.where` with ``CategoricalDtype`` when passing a dtype-incompatible value raising ``ValueError`` instead of ``TypeError`` (`41919`)
- Bug in `Categorical.fillna` when passing a dtype-incompatible value raising ``ValueError`` instead of ``TypeError`` (`41919`)
- Bug in `Categorical.fillna` with a tuple-like category raising ``ValueError`` instead of ``TypeError`` when filling with a non-category tuple (`41919`)  Datetimelike
^^^
- Bug in `DataFrame` constructor unnecessarily copying non-datetimelike 2D object arrays (`39272`)
- Bug in `to_datetime` with ``format`` and ``pandas.NA`` was raising ``ValueError`` (`42957`)

^^^
- `to_datetime` would silently swap ``MM/DD/YYY`` and ``DD/MM/YYY`` formats if the given ``dayfirst`` option could not be respected - now, a warning is raised in the case of delimited date strings (e.g. ``31-12-2012``) (`12585`)
- Bug in `date_range` and `bdate_range` do not return right bound when ``start`` = ``end`` and set is closed on one side (`43394`)
- Bug in inplace addition and subtraction of `DatetimeIndex` or `TimedeltaIndex` with `DatetimeArray` or `TimedeltaArray` (`43904`)
- Bug in calling ``np.isnan``, ``np.isfinite``, or ``np.isinf`` on a timezone-aware `DatetimeIndex` incorrectly raising ``TypeError`` (`43917`)
- Bug in constructing a `Series` from datetime-like strings with mixed timezones incorrectly partially-inferring datetime values (`40111`)
- Bug in addition of a `Tick` object and a ``np.timedelta64`` object incorrectly raising instead of returning `Timedelta` (`44474`)
- ``np.maximum.reduce`` and ``np.minimum.reduce`` now correctly return `Timestamp` and `Timedelta` objects when operating on `Series`, `DataFrame`, or `Index` with ``datetime64[ns]`` or ``timedelta64[ns]`` dtype (`43923`)
- Bug in adding a ``np.timedelta64`` object to a `BusinessDay` or `CustomBusinessDay` object incorrectly raising (`44532`)
- Bug in `Index.insert` for inserting ``np.datetime64``, ``np.timedelta64`` or ``tuple`` into `Index` with ``dtype='object'`` with negative loc adding ``None`` and replacing existing value (`44509`)
- Bug in `Timestamp.to_pydatetime` failing to retain the ``fold`` attribute (`45087`)
- Bug in `Series.mode` with ``DatetimeTZDtype`` incorrectly returning timezone-naive and ``PeriodDtype`` incorrectly raising (`41927`)
- Fixed regression in `~Series.reindex` raising an error when using an incompatible fill value with a datetime-like dtype (or not raising a deprecation warning for using a ``datetime.date`` as fill value) (`42921`)
- Bug in `DateOffset` addition with `Timestamp` where ``offset.nanoseconds`` would not be included in the result (`43968`, `36589`)
- Bug in `Timestamp.fromtimestamp` not supporting the ``tz`` argument (`45083`)
- Bug in `DataFrame` construction from dict of `Series` with mismatched index dtypes sometimes raising depending on the ordering of the passed dict (`44091`)
- Bug in `Timestamp` hashing during some DST transitions caused a segmentation fault (`33931` and `40817`)  Timedelta
^^^
- Bug in division of all-``NaT`` `TimeDeltaIndex`, `Series` or `DataFrame` column with object-dtype array like of numbers failing to infer the result as timedelta64-dtype (`39750`)
- Bug in floor division of ``timedelta64[ns]`` data with a scalar returning garbage values (`44466`)
- Bug in `Timedelta` now properly taking into account any nanoseconds contribution of any kwarg (`43764`, `45227`)  Time Zones
^^^
- Bug in `to_datetime` with ``infer_datetime_format=True`` failing to parse zero UTC offset (``Z``) correctly (`41047`)
- Bug in `Series.dt.tz_convert` resetting index in a `Series` with `CategoricalIndex` (`43080`)
- Bug in ``Timestamp`` and ``DatetimeIndex`` incorrectly raising a ``TypeError`` when subtracting two timezone-aware objects with mismatched timezones (`31793`)  Numeric
^^^
- Bug in floor-dividing a list or tuple of integers by a `Series` incorrectly raising (`44674`)
- Bug in `DataFrame.rank` raising ``ValueError`` with ``object`` columns and ``method="first"`` (`41931`)
- Bug in `DataFrame.rank` treating missing values and extreme values as equal (for example ``np.nan`` and ``np.inf``), causing incorrect results when ``na_option="bottom"`` or ``na_option="top`` used (`41931`)
- Bug in ``numexpr`` engine still being used when the option ``compute.use_numexpr`` is set to ``False`` (`32556`)

- Bug in `DataFrame` arithmetic ops with a subclass whose `_constructor` attribute is a callable other than the subclass itself (`43201`)
- Bug in arithmetic operations involving `RangeIndex` where the result would have the incorrect ``name`` (`43962`)
- Bug in arithmetic operations involving `Series` where the result could have the incorrect ``name`` when the operands having matching NA or matching tuple names (`44459`)
- Bug in division with ``IntegerDtype`` or ``BooleanDtype`` array and NA scalar incorrectly raising (`44685`)
- Bug in multiplying a `Series` with ``FloatingDtype`` with a timedelta-like scalar incorrectly raising (`44772`)  Conversion
^^^
- Bug in `UInt64Index` constructor when passing a list containing both positive integers small enough to cast to int64 and integers too large to hold in int64 (`42201`)
- Bug in `Series` constructor returning 0 for missing values with dtype ``int64`` and ``False`` for dtype ``bool`` (`43017`, `43018`)
- Bug in constructing a `DataFrame` from a `PandasArray` containing `Series` objects behaving differently than an equivalent ``np.ndarray`` (`43986`)
- Bug in `IntegerDtype` not allowing coercion from string dtype (`25472`)
- Bug in `to_datetime` with ``arg:xr.DataArray`` and ``unit="ns"`` specified raises ``TypeError`` (`44053`)
- Bug in `DataFrame.convert_dtypes` not returning the correct type when a subclass does not overload `_constructor_sliced` (`43201`)
- Bug in `DataFrame.astype` not propagating ``attrs`` from the original `DataFrame` (`44414`)
- Bug in `DataFrame.convert_dtypes` result losing ``columns.names`` (`41435`)
- Bug in constructing a ``IntegerArray`` from pyarrow data failing to validate dtypes (`44891`)
- Bug in `Series.astype` not allowing converting from a ``PeriodDtype`` to ``datetime64`` dtype, inconsistent with the `PeriodIndex` behavior (`45038`)  Strings
^^^
- Bug in checking for ``string[pyarrow]`` dtype incorrectly raising an ``ImportError`` when pyarrow is not installed (`44276`)  Interval
^^^
- Bug in `Series.where` with ``IntervalDtype`` incorrectly raising when the ``where`` call should not replace anything (`44181`)  Indexing
^^^
- Bug in `Series.rename` with `MultiIndex` and ``level`` is provided (`43659`)
- Bug in `DataFrame.truncate` and `Series.truncate` when the object's `Index` has a length greater than one but only one unique value (`42365`)
- Bug in `Series.loc` and `DataFrame.loc` with a `MultiIndex` when indexing with a tuple in which one of the levels is also a tuple (`27591`)
- Bug in `Series.loc` with a `MultiIndex` whose first level contains only ``np.nan`` values (`42055`)
- Bug in indexing on a `Series` or `DataFrame` with a `DatetimeIndex` when passing a string, the return type depended on whether the index was monotonic (`24892`)
- Bug in indexing on a `MultiIndex` failing to drop scalar levels when the indexer is a tuple containing a datetime-like string (`42476`)
- Bug in `DataFrame.sort_values` and `Series.sort_values` when passing an ascending value, failed to raise or incorrectly raising ``ValueError`` (`41634`)
- Bug in updating values of `pandas.Series` using boolean index, created by using `pandas.DataFrame.pop` (`42530`)
- Bug in `Index.get_indexer_non_unique` when index contains multiple ``np.nan`` (`35392`)
- Bug in `DataFrame.query` did not handle the degree sign in a backticked column name, such as \`Temp(°C)\`, used in an expression to query a `DataFrame` (`42826`)
- Bug in `DataFrame.drop` where the error message did not show missing labels with commas when raising ``KeyError`` (`42881`)
- Bug in `DataFrame.query` where method calls in query strings led to errors when the ``numexpr`` package was installed (`22435`)
- Bug in `DataFrame.nlargest` and `Series.nlargest` where sorted result did not count indexes containing ``np.nan`` (`28984`)

- Bug in indexing on a non-unique object-dtype `Index` with an NA scalar (e.g. ``np.nan``) (`43711`)
- Bug in `DataFrame.__setitem__` incorrectly writing into an existing column's array rather than setting a new array when the new dtype and the old dtype match (`43406`)
- Bug in setting floating-dtype values into a `Series` with integer dtype failing to set inplace when those values can be losslessly converted to integers (`44316`)
- Bug in `Series.__setitem__` with object dtype when setting an array with matching size and dtype='datetime64[ns]' or dtype='timedelta64[ns]' incorrectly converting the datetime/timedeltas to integers (`43868`)
- Bug in `DataFrame.sort_index` where ``ignore_index=True`` was not being respected when the index was already sorted (`43591`)
- Bug in `Index.get_indexer_non_unique` when index contains multiple ``np.datetime64("NaT")`` and ``np.timedelta64("NaT")`` (`43869`)
- Bug in setting a scalar `Interval` value into a `Series` with ``IntervalDtype`` when the scalar's sides are floats and the values' sides are integers (`44201`)
- Bug when setting string-backed `Categorical` values that can be parsed to datetimes into a `DatetimeArray` or `Series` or `DataFrame` column backed by `DatetimeArray` failing to parse these strings (`44236`)
- Bug in `Series.__setitem__` with an integer dtype other than ``int64`` setting with a ``range`` object unnecessarily upcasting to ``int64`` (`44261`)
- Bug in `Series.__setitem__` with a boolean mask indexer setting a listlike value of length 1 incorrectly broadcasting that value (`44265`)
- Bug in `Series.reset_index` not ignoring ``name`` argument when ``drop`` and ``inplace`` are set to ``True`` (`44575`)
- Bug in `DataFrame.loc.__setitem__` and `DataFrame.iloc.__setitem__` with mixed dtypes sometimes failing to operate in-place (`44345`)
- Bug in `DataFrame.loc.__getitem__` incorrectly raising ``KeyError`` when selecting a single column with a boolean key (`44322`).
- Bug in setting `DataFrame.iloc` with a single ``ExtensionDtype`` column and setting 2D values e.g. ``df.iloc[:] = df.values`` incorrectly raising (`44514`)
- Bug in setting values with `DataFrame.iloc` with a single ``ExtensionDtype`` column and a tuple of arrays as the indexer (`44703`)
- Bug in indexing on columns with ``loc`` or ``iloc`` using a slice with a negative step with ``ExtensionDtype`` columns incorrectly raising (`44551`)
- Bug in `DataFrame.loc.__setitem__` changing dtype when indexer was completely ``False`` (`37550`)
- Bug in `IntervalIndex.get_indexer_non_unique` returning boolean mask instead of array of integers for a non unique and non monotonic index (`44084`)
- Bug in `IntervalIndex.get_indexer_non_unique` not handling targets of ``dtype`` 'object' with NaNs correctly (`44482`)
- Fixed regression where a single column ``np.matrix`` was no longer coerced to a 1d ``np.ndarray`` when added to a `DataFrame` (`42376`)
- Bug in `Series.__getitem__` with a `CategoricalIndex` of integers treating lists of integers as positional indexers, inconsistent with the behavior with a single scalar integer (`15470`, `14865`)
- Bug in `Series.__setitem__` when setting floats or integers into integer-dtype `Series` failing to upcast when necessary to retain precision (`45121`)
- Bug in `DataFrame.iloc.__setitem__` ignores axis argument (`45032`)  Missing
^^^
- Bug in `DataFrame.fillna` with ``limit`` and no ``method`` ignores ``axis='columns'`` or ``axis = 1`` (`40989`, `17399`)
- Bug in `DataFrame.fillna` not replacing missing values when using a dict-like ``value`` and duplicate column names (`43476`)
- Bug in constructing a `DataFrame` with a dictionary ``np.datetime64`` as a value and ``dtype='timedelta64[ns]'``, or vice-versa, incorrectly casting instead of raising (`44428`)

- Bug in `Series.interpolate` and `DataFrame.interpolate` with ``inplace=True`` not writing to the underlying array(s) in-place (`44749`)
- Bug in `Index.fillna` incorrectly returning an unfilled `Index` when NA values are present and ``downcast`` argument is specified. This now raises ``NotImplementedError`` instead; do not pass ``downcast`` argument (`44873`)
- Bug in `DataFrame.dropna` changing `Index` even if no entries were dropped (`41965`)
- Bug in `Series.fillna` with an object-dtype incorrectly ignoring ``downcast="infer"`` (`44241`)  MultiIndex
^^^
- Bug in `MultiIndex.get_loc` where the first level is a `DatetimeIndex` and a string key is passed (`42465`)
- Bug in `MultiIndex.reindex` when passing a ``level`` that corresponds to an ``ExtensionDtype`` level (`42043`)
- Bug in `MultiIndex.get_loc` raising ``TypeError`` instead of ``KeyError`` on nested tuple (`42440`)
- Bug in `MultiIndex.union` setting wrong ``sortorder`` causing errors in subsequent indexing operations with slices (`44752`)
- Bug in `MultiIndex.putmask` where the other value was also a `MultiIndex` (`43212`)
- Bug in `MultiIndex.dtypes` duplicate level names returned only one dtype per name (`45174`)  I/O
^^^
- Bug in `read_excel` attempting to read chart sheets from .xlsx files (`41448`)
- Bug in `json_normalize` where ``errors=ignore`` could fail to ignore missing values of ``meta`` when ``record_path`` has a length greater than one (`41876`)
- Bug in `read_csv` with multi-header input and arguments referencing column names as tuples (`42446`)
- Bug in `read_fwf`, where difference in lengths of ``colspecs`` and ``names`` was not raising ``ValueError`` (`40830`)
- Bug in `Series.to_json` and `DataFrame.to_json` where some attributes were skipped when serializing plain Python objects to JSON (`42768`, `33043`)
- Column headers are dropped when constructing a `DataFrame` from a sqlalchemy's ``Row`` object (`40682`)
- Bug in unpickling an `Index` with object dtype incorrectly inferring numeric dtypes (`43188`)
- Bug in `read_csv` where reading multi-header input with unequal lengths incorrectly raised ``IndexError`` (`43102`)
- Bug in `read_csv` raising ``ParserError`` when reading file in chunks and some chunk blocks have fewer columns than header for ``engine="c"`` (`21211`)
- Bug in `read_csv`, changed exception class when expecting a file path name or file-like object from ``OSError`` to ``TypeError`` (`43366`)
- Bug in `read_csv` and `read_fwf` ignoring all ``skiprows`` except first when ``nrows`` is specified for ``engine='python'`` (`44021`, `10261`)
- Bug in `read_csv` keeping the original column in object format when ``keep_date_col=True`` is set (`13378`)
- Bug in `read_json` not handling non-numpy dtypes correctly (especially ``category``) (`21892`, `33205`)
- Bug in `json_normalize` where multi-character ``sep`` parameter is incorrectly prefixed to every key (`43831`)
- Bug in `json_normalize` where reading data with missing multi-level metadata would not respect ``errors="ignore"`` (`44312`)
- Bug in `read_csv` used second row to guess implicit index if ``header`` was set to ``None`` for ``engine="python"`` (`22144`)
- Bug in `read_csv` not recognizing bad lines when ``names`` were given for ``engine="c"`` (`22144`)
- Bug in `read_csv` with `float_precision="round_trip"` which did not skip initial/trailing whitespace (`43713`)
- Bug when Python is built without the lzma module: a warning was raised at the pandas import time, even if the lzma capability isn't used (`43495`)
- Bug in `read_csv` not applying dtype for ``index_col`` (`9435`)
- Bug in dumping/loading a `DataFrame` with ``yaml.dump(frame)`` (`42748`)
- Bug in `read_csv` raising ``ValueError`` when ``names`` was longer than ``header`` but equal to data rows for ``engine="python"`` (`38453`)

- Bug in `ExcelWriter`, where ``engine_kwargs`` were not passed through to all engines (`43442`)
- Bug in `read_csv` raising ``ValueError`` when ``parse_dates`` was used with `MultiIndex` columns (`8991`)
- Bug in `read_csv` not raising an ``ValueError`` when ``\n`` was specified as ``delimiter`` or ``sep`` which conflicts with ``lineterminator`` (`43528`)
- Bug in `to_csv` converting datetimes in categorical `Series` to integers (`40754`)
- Bug in `read_csv` converting columns to numeric after date parsing failed (`11019`)
- Bug in `read_csv` not replacing ``NaN`` values with ``np.nan`` before attempting date conversion (`26203`)
- Bug in `read_csv` raising ``AttributeError`` when attempting to read a .csv file and infer index column dtype from an nullable integer type (`44079`)
- Bug in `to_csv` always coercing datetime columns with different formats to the same format (`21734`)
- `DataFrame.to_csv` and `Series.to_csv` with ``compression`` set to ``'zip'`` no longer create a zip file containing a file ending with ".zip". Instead, they try to infer the inner file name more smartly (`39465`)
- Bug in `read_csv` where reading a mixed column of booleans and missing values to a float type results in the missing values becoming 1.0 rather than NaN (`42808`, `34120`)
- Bug in `to_xml` raising error for ``pd.NA`` with extension array dtype (`43903`)
- Bug in `read_csv` when passing simultaneously a parser in ``date_parser`` and ``parse_dates=False``, the parsing was still called (`44366`)
- Bug in `read_csv` not setting name of `MultiIndex` columns correctly when ``index_col`` is not the first column (`38549`)
- Bug in `read_csv` silently ignoring errors when failing to create a memory-mapped file (`44766`)
- Bug in `read_csv` when passing a ``tempfile.SpooledTemporaryFile`` opened in binary mode (`44748`)
- Bug in `read_json` raising ``ValueError`` when attempting to parse json strings containing "://" (`36271`)
- Bug in `read_csv` when ``engine="c"`` and ``encoding_errors=None`` which caused a segfault (`45180`)
- Bug in `read_csv` an invalid value of ``usecols`` leading to an unclosed file handle (`45384`)
- Bug in `DataFrame.to_json` fix memory leak (`43877`)  Period
^^^
- Bug in adding a `Period` object to a ``np.timedelta64`` object incorrectly raising ``TypeError`` (`44182`)
- Bug in `PeriodIndex.to_timestamp` when the index has ``freq="B"`` inferring ``freq="D"`` for its result instead of ``freq="B"`` (`44105`)
- Bug in `Period` constructor incorrectly allowing ``np.timedelta64("NaT")`` (`44507`)
- Bug in `PeriodIndex.to_timestamp` giving incorrect values for indexes with non-contiguous data (`44100`)
- Bug in `Series.where` with ``PeriodDtype`` incorrectly raising when the ``where`` call should not replace anything (`45135`)  Plotting
^^^
- When given non-numeric data, `DataFrame.boxplot` now raises a ``ValueError`` rather than a cryptic ``KeyError`` or ``ZeroDivisionError``, in line with other plotting functions like `DataFrame.hist` (`43480`)  Groupby/resample/rolling
^^^
- Bug in `SeriesGroupBy.apply` where passing an unrecognized string argument failed to raise ``TypeError`` when the underlying ``Series`` is empty (`42021`)
- Bug in `Series.rolling.apply`, `DataFrame.rolling.apply`, `Series.expanding.apply` and `DataFrame.expanding.apply` with ``engine="numba"`` where ``*args`` were being cached with the user passed function (`42287`)
- Bug in `.DataFrameGroupBy.max`, `.SeriesGroupBy.max`, `.DataFrameGroupBy.min`, and `.SeriesGroupBy.min` with nullable integer dtypes losing precision (`41743`)
- Bug in `DataFrame.groupby.rolling.var` would calculate the rolling variance only on the first group (`42442`)
- Bug in `.DataFrameGroupBy.shift` and `.SeriesGroupBy.shift` that would return the grouping columns if ``fill_value`` was not ``None`` (`41556`)

- Bug in `SeriesGroupBy.nlargest` and `SeriesGroupBy.nsmallest` would have an inconsistent index when the input `Series` was sorted and ``n`` was greater than or equal to all group sizes (`15272`, `16345`, `29129`)
- Bug in `pandas.DataFrame.ewm`, where non-float64 dtypes were silently failing (`42452`)
- Bug in `pandas.DataFrame.rolling` operation along rows (``axis=1``) incorrectly omits columns containing ``float16`` and ``float32`` (`41779`)
- Bug in `Resampler.aggregate` did not allow the use of Named Aggregation (`32803`)
- Bug in `Series.rolling` when the `Series` ``dtype`` was ``Int64`` (`43016`)
- Bug in `DataFrame.rolling.corr` when the `DataFrame` columns was a `MultiIndex` (`21157`)
- Bug in `DataFrame.groupby.rolling` when specifying ``on`` and calling ``__getitem__`` would subsequently return incorrect results (`43355`)
- Bug in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` with time-based `Grouper` objects incorrectly raising ``ValueError`` in corner cases where the grouping vector contains a ``NaT`` (`43500`, `43515`)
- Bug in `.DataFrameGroupBy.mean` and `.SeriesGroupBy.mean` failing with ``complex`` dtype (`43701`)
- Bug in `Series.rolling` and `DataFrame.rolling` not calculating window bounds correctly for the first row when ``center=True`` and index is decreasing (`43927`)
- Bug in `Series.rolling` and `DataFrame.rolling` for centered datetimelike windows with uneven nanosecond (`43997`)
- Bug in `.DataFrameGroupBy.mean` and `.SeriesGroupBy.mean` raising ``KeyError`` when column was selected at least twice (`44924`)
- Bug in `.DataFrameGroupBy.nth` and `.SeriesGroupBy.nth` failing on ``axis=1`` (`43926`)
- Bug in `Series.rolling` and `DataFrame.rolling` not respecting right bound on centered datetime-like windows, if the index contain duplicates (`3944`)
- Bug in `Series.rolling` and `DataFrame.rolling` when using a `pandas.api.indexers.BaseIndexer` subclass that returned unequal start and end arrays would segfault instead of raising a ``ValueError`` (`44470`)
- Bug in `Groupby.nunique` not respecting ``observed=True`` for ``categorical`` grouping columns (`45128`)
- Bug in `.DataFrameGroupBy.head`, `.SeriesGroupBy.head`, `.DataFrameGroupBy.tail`, and `.SeriesGroupBy.tail` not dropping groups with ``NaN`` when ``dropna=True`` (`45089`)
- Bug in `GroupBy.__iter__` after selecting a subset of columns in a `GroupBy` object, which returned all columns instead of the chosen subset (`44821`)
- Bug in `Groupby.rolling` when non-monotonic data passed, fails to correctly raise ``ValueError`` (`43909`)
- Bug where grouping by a `Series` that has a ``categorical`` data type and length unequal to the axis of grouping raised ``ValueError`` (`44179`)  Reshaping
^^^
- Improved error message when creating a `DataFrame` column from a multi-dimensional `numpy.ndarray` (`42463`)
- Bug in `concat` creating `MultiIndex` with duplicate level entries when concatenating a `DataFrame` with duplicates in `Index` and multiple keys (`42651`)
- Bug in `pandas.cut` on `Series` with duplicate indices and non-exact `pandas.CategoricalIndex` (`42185`, `42425`)
- Bug in `DataFrame.append` failing to retain dtypes when appended columns do not match (`43392`)
- Bug in `concat` of ``bool`` and ``boolean`` dtypes resulting in ``object`` dtype instead of ``boolean`` dtype (`42800`)
- Bug in `crosstab` when inputs are categorical `Series`, there are categories that are not present in one or both of the `Series`, and ``margins=True``. Previously the margin value for missing categories was ``NaN``. It is now correctly reported as 0 (`43505`)
- Bug in `concat` would fail when the ``objs`` argument all had the same index and the ``keys`` argument contained duplicates (`43595`)
- Bug in `concat` which ignored the ``sort`` parameter (`43375`)

- Bug in `merge` with `MultiIndex` as column index for the ``on`` argument returning an error when assigning a column internally (`43734`)
- Bug in `crosstab` would fail when inputs are lists or tuples (`44076`)
- Bug in `DataFrame.append` failing to retain ``index.name`` when appending a list of `Series` objects (`44109`)
- Fixed metadata propagation in `Dataframe.apply` method, consequently fixing the same issue for `Dataframe.transform`, `Dataframe.nunique` and `Dataframe.mode` (`28283`)
- Bug in `concat` casting levels of `MultiIndex` to float if all levels only consist of missing values (`44900`)
- Bug in `DataFrame.stack` with ``ExtensionDtype`` columns incorrectly raising (`43561`)
- Bug in `merge` raising ``KeyError`` when joining over differently named indexes with on keywords (`45094`)
- Bug in `Series.unstack` with object doing unwanted type inference on resulting columns (`44595`)
- Bug in `MultiIndex.join()` with overlapping ``IntervalIndex`` levels (`44096`)
- Bug in `DataFrame.replace` and `Series.replace` results is different ``dtype`` based on ``regex`` parameter (`44864`)
- Bug in `DataFrame.pivot` with ``index=None`` when the `DataFrame` index was a `MultiIndex` (`23955`)  Sparse
^^^
- Bug in `DataFrame.sparse.to_coo` raising ``AttributeError`` when column names are not unique (`29564`)
- Bug in `SparseArray.max` and `SparseArray.min` raising ``ValueError`` for arrays with 0 non-null elements (`43527`)
- Bug in `DataFrame.sparse.to_coo` silently converting non-zero fill values to zero (`24817`)
- Bug in `SparseArray` comparison methods with an array-like operand of mismatched length raising ``AssertionError`` or unclear ``ValueError`` depending on the input (`43863`)
- Bug in `SparseArray` arithmetic methods ``floordiv`` and ``mod`` behaviors when dividing by zero not matching the non-sparse `Series` behavior (`38172`)
- Bug in `SparseArray` unary methods as well as `SparseArray.isna` doesn't recalculate indexes (`44955`)  ExtensionArray
^^^
- Bug in `array` failing to preserve `PandasArray` (`43887`)
- NumPy ufuncs ``np.abs``, ``np.positive``, ``np.negative`` now correctly preserve dtype when called on ExtensionArrays that implement ``__abs__, __pos__, __neg__``, respectively. In particular this is fixed for `TimedeltaArray` (`43899`, `23316`)
- NumPy ufuncs ``np.minimum.reduce`` ``np.maximum.reduce``, ``np.add.reduce``, and ``np.prod.reduce`` now work correctly instead of raising ``NotImplementedError`` on `Series` with ``IntegerDtype`` or ``FloatDtype`` (`43923`, `44793`)
- NumPy ufuncs with ``out`` keyword are now supported by arrays with ``IntegerDtype`` and ``FloatingDtype`` (`45122`)
- Avoid raising ``PerformanceWarning`` about fragmented `DataFrame` when using many columns with an extension dtype (`44098`)
- Bug in `IntegerArray` and `FloatingArray` construction incorrectly coercing mismatched NA values (e.g. ``np.timedelta64("NaT")``) to numeric NA (`44514`)
- Bug in `BooleanArray.__eq__` and `BooleanArray.__ne__` raising ``TypeError`` on comparison with an incompatible type (like a string). This caused `DataFrame.replace` to sometimes raise a ``TypeError`` if a nullable boolean column was included (`44499`)
- Bug in `array` incorrectly raising when passed a ``ndarray`` with ``float16`` dtype (`44715`)
- Bug in calling ``np.sqrt`` on `BooleanArray` returning a malformed `FloatingArray` (`44715`)
- Bug in `Series.where` with ``ExtensionDtype`` when ``other`` is a NA scalar incompatible with the `Series` dtype (e.g. ``NaT`` with a numeric dtype) incorrectly casting to a compatible NA value (`44697`)
- Bug in `Series.replace` where explicitly passing ``value=None`` is treated as if no ``value`` was passed, and ``None`` not being in the result (`36984`, `19998`)

- Bug in `Series.replace` with unwanted downcasting being done in no-op replacements (`44498`)
- Bug in `Series.replace` with ``FloatDtype``, ``string[python]``, or ``string[pyarrow]`` dtype not being preserved when possible (`33484`, `40732`, `31644`, `41215`, `25438`)  Styler
^^^
- Bug in `.Styler` where the ``uuid`` at initialization maintained a floating underscore (`43037`)
- Bug in `.Styler.to_html` where the ``Styler`` object was updated if the ``to_html`` method was called with some args (`43034`)
- Bug in `.Styler.copy` where ``uuid`` was not previously copied (`40675`)
- Bug in `Styler.apply` where functions which returned `Series` objects were not correctly handled in terms of aligning their index labels (`13657`, `42014`)
- Bug when rendering an empty `DataFrame` with a named `Index` (`43305`)
- Bug when rendering a single level `MultiIndex` (`43383`)
- Bug when combining non-sparse rendering and `.Styler.hide_columns` or `.Styler.hide_index` (`43464`)
- Bug setting a table style when using multiple selectors in `.Styler` (`44011`)
- Bugs where row trimming and column trimming failed to reflect hidden rows (`43703`, `44247`)  Other
^^^
- Bug in `DataFrame.astype` with non-unique columns and a `Series` ``dtype`` argument (`44417`)
- Bug in `CustomBusinessMonthBegin.__add__` (`CustomBusinessMonthEnd.__add__`) not applying the extra ``offset`` parameter when beginning (end) of the target month is already a business day (`41356`)
- Bug in `RangeIndex.union` with another ``RangeIndex`` with matching (even) ``step`` and starts differing by strictly less than ``step / 2`` (`44019`)
- Bug in `RangeIndex.difference` with ``sort=None`` and ``step<0`` failing to sort (`44085`)
- Bug in `Series.replace` and `DataFrame.replace` with ``value=None`` and ExtensionDtypes (`44270`, `37899`)
- Bug in `FloatingArray.equals` failing to consider two arrays equal if they contain ``np.nan`` values (`44382`)
- Bug in `DataFrame.shift` with ``axis=1`` and ``ExtensionDtype`` columns incorrectly raising when an incompatible ``fill_value`` is passed (`44564`)
- Bug in `DataFrame.shift` with ``axis=1`` and ``periods`` larger than ``len(frame.columns)`` producing an invalid `DataFrame` (`44978`)
- Bug in `DataFrame.diff` when passing a NumPy integer object instead of an ``int`` object (`44572`)
- Bug in `Series.replace` raising ``ValueError`` when using ``regex=True`` with a `Series` containing ``np.nan`` values (`43344`)
- Bug in `DataFrame.to_records` where an incorrect ``n`` was used when missing names were replaced by ``level_n`` (`44818`)
- Bug in `DataFrame.eval` where ``resolvers`` argument was overriding the default resolvers (`34966`)
- `Series.__repr__` and `DataFrame.__repr__` no longer replace all null-values in indexes with "NaN" but use their real string-representations. "NaN" is used only for ``float("nan")`` (`45263`)  .. ---  .. _whatsnew_140.contributors:  Contributors
~~~  .. contributors:: v1.3.5..v1.4.0

What's new in 1.4.1 (February 12, 2022)

These are the changes in pandas 1.4.1. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_141.regressions:  Fixed regressions
~~~
- Regression in `Series.mask` with ``inplace=True`` and ``PeriodDtype`` and an incompatible ``other`` coercing to a common dtype instead of raising (`45546`)
- Regression in `.assert_frame_equal` not respecting ``check_flags=False`` (`45554`)
- Regression in `DataFrame.loc` raising ``ValueError`` when indexing (getting values) on a `MultiIndex` with one level (`45779`)
- Regression in `Series.fillna` with ``downcast=False`` incorrectly downcasting ``object`` dtype (`45603`)
- Regression in `api.types.is_bool_dtype` raising an ``AttributeError`` when evaluating a categorical `Series` (`45615`)
- Regression in `DataFrame.iat` setting values leading to not propagating correctly in subsequent lookups (`45684`)
- Regression when setting values with `DataFrame.loc` losing `Index` name if `DataFrame` was empty before (`45621`)
- Regression in `~Index.join` with overlapping `IntervalIndex` raising an ``InvalidIndexError`` (`45661`)
- Regression when setting values with `Series.loc` raising with all ``False`` indexer and `Series` on the right hand side (`45778`)
- Regression in `read_sql` with a DBAPI2 connection that is not an instance of ``sqlite3.Connection`` incorrectly requiring SQLAlchemy be installed (`45660`)
- Regression in `DateOffset` when constructing with an integer argument with no keywords (e.g. ``pd.DateOffset(n)``) would behave like ``datetime.timedelta(days=0)`` (`45643`, `45890`)  .. ---  .. _whatsnew_141.bug_fixes:  Bug fixes
~~~
- Fixed segfault in `DataFrame.to_json` when dumping tz-aware datetimes in Python 3.10 (`42130`)
- Stopped emitting unnecessary ``FutureWarning`` in `DataFrame.sort_values` with sparse columns (`45618`)
- Fixed window aggregations in `DataFrame.rolling` and `Series.rolling` to skip over unused elements (`45647`)
- Fixed builtin highlighters in `.Styler` to be responsive to ``NA`` with nullable dtypes (`45804`)
- Bug in `~Rolling.apply` with ``axis=1`` raising an erroneous ``ValueError`` (`45912`)  .. ---  .. _whatsnew_141.other:  Other
~~~
- Reverted performance speedup of `DataFrame.corr` for ``method=pearson`` to fix precision regression (`45640`, `42761`)  .. ---  .. _whatsnew_141.contributors:  Contributors
~~~  .. contributors:: v1.4.0..v1.4.1

What's new in 1.4.2 (April 2, 2022)

These are the changes in pandas 1.4.2. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_142.regressions:  Fixed regressions
~~~
- Fixed regression in `DataFrame.drop` and `Series.drop` when `Index` had extension dtype and duplicates (`45860`)
- Fixed regression in `read_csv` killing python process when invalid file input was given for ``engine="c"`` (`45957`)
- Fixed memory performance regression in `Series.fillna` when called on a `DataFrame` column with ``inplace=True`` (`46149`)
- Provided an alternative solution for passing custom Excel formats in `.Styler.to_excel`, which was a regression based on stricter CSS validation. Examples available in the documentation for `.Styler.format` (`46152`)
- Fixed regression in `DataFrame.replace` when a replacement value was also a target for replacement (`46306`)
- Fixed regression in `DataFrame.replace` when the replacement value was explicitly ``None`` when passed in a dictionary to ``to_replace`` (`45601`, `45836`)
- Fixed regression when setting values with `DataFrame.loc` losing `MultiIndex` names if `DataFrame`  was empty before (`46317`)
- Fixed regression when rendering boolean datatype columns with `.Styler` (`46384`)
- Fixed regression in `Groupby.rolling` with a frequency window that would raise a ``ValueError`` even if the datetimes within each group were monotonic (`46061`)  .. ---  .. _whatsnew_142.bug_fixes:  Bug fixes
~~~
- Fix some cases for subclasses that define their ``_constructor`` properties as general callables (`46018`)
- Fixed "longtable" formatting in `.Styler.to_latex` when ``column_format`` is given in extended format (`46037`)
- Fixed incorrect rendering in `.Styler.format` with ``hyperlinks="html"`` when the url contains a colon or other special characters (`46389`)
- Improved error message in `~pandas.core.window.Rolling` when ``window`` is a frequency and ``NaT`` is in the rolling axis (`46087`)  .. ---  .. _whatsnew_142.contributors:  Contributors
~~~  .. contributors:: v1.4.1..v1.4.2

What's new in 1.4.3 (June 23, 2022)

These are the changes in pandas 1.4.3. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_143.concat:  Behavior of ``concat`` with empty or all-NA DataFrame columns
~~~  The behavior change in version 1.4.0 to stop ignoring the data type
of empty or all-NA columns with float or object dtype in `concat`
(`whatsnew_140.notable_bug_fixes.concat_with_empty_or_all_na`) has been
reverted (`45637`).  .. _whatsnew_143.regressions:  Fixed regressions
~~~
- Fixed regression in `DataFrame.replace` when the replacement value was explicitly ``None`` when passed in a dictionary to ``to_replace`` also casting other columns to object dtype even when there were no values to replace (`46634`)
- Fixed regression in `DataFrame.to_csv` raising error when `DataFrame` contains extension dtype categorical column (`46297`, `46812`)
- Fixed regression in representation of ``dtypes`` attribute of `MultiIndex` (`46900`)
- Fixed regression when setting values with `DataFrame.loc` updating `RangeIndex` when index was set as new column and column was updated afterwards (`47128`)
- Fixed regression in `DataFrame.fillna` and `DataFrame.update` creating a copy when updating inplace (`47188`)
- Fixed regression in `DataFrame.nsmallest` led to wrong results when the sorting column has ``np.nan`` values (`46589`)
- Fixed regression in `read_fwf` raising ``ValueError`` when ``widths`` was specified with ``usecols`` (`46580`)
- Fixed regression in `concat` not sorting columns for mixed column names (`47127`)
- Fixed regression in `.Groupby.transform` and `.Groupby.agg` failing with ``engine="numba"`` when the index was a `MultiIndex` (`46867`)
- Fixed regression in ``NaN`` comparison for `Index` operations where the same object was compared (`47105`)
- Fixed regression is `.Styler.to_latex` and `.Styler.to_html` where ``buf`` failed in combination with ``encoding`` (`47053`)
- Fixed regression in `read_csv` with ``index_col=False`` identifying first row as index names when ``header=None`` (`46955`)
- Fixed regression in `.DataFrameGroupBy.agg` when used with list-likes or dict-likes and ``axis=1`` that would give incorrect results; now raises ``NotImplementedError`` (`46995`)
- Fixed regression in `DataFrame.resample` and `DataFrame.rolling` when used with list-likes or dict-likes and ``axis=1`` that would raise an unintuitive error message; now raises ``NotImplementedError`` (`46904`)
- Fixed regression in `testing.assert_index_equal` when ``check_order=False`` and `Index` has extension or object dtype (`47207`)
- Fixed regression in `read_excel` returning ints as floats on certain input sheets (`46988`)
- Fixed regression in `DataFrame.shift` when ``axis`` is ``columns`` and ``fill_value`` is absent, ``freq`` is ignored (`47039`)
- Fixed regression in `DataFrame.to_json` causing a segmentation violation when `DataFrame` is created with an ``index`` parameter of the type `PeriodIndex` (`46683`)  .. ---  .. _whatsnew_143.bug_fixes:  Bug fixes
~~~
- Bug in `pandas.eval`, `DataFrame.eval` and `DataFrame.query` where passing empty ``local_dict`` or ``global_dict`` was treated as passing ``None`` (`47084`)
- Most I/O methods no longer suppress ``OSError`` and ``ValueError`` when closing file handles (`47136`)
- Improving error message raised by `DataFrame.from_dict` when passing an invalid ``orient`` parameter (`47450`)  .. ---  .. _whatsnew_143.other:  Other
~~~
- The minimum version of Cython needed to compile pandas is now ``0.29.30`` (`41935`)  .. ---  .. _whatsnew_143.contributors:  Contributors
~~~  .. contributors:: v1.4.2..v1.4.3

What's new in 1.4.4 (August 31, 2022)

These are the changes in pandas 1.4.4. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_144.regressions:  Fixed regressions
~~~
- Fixed regression in `DataFrame.fillna` not working on a `DataFrame` with a `MultiIndex` (`47649`)
- Fixed regression in taking NULL `objects` from a `DataFrame` causing a segmentation violation. These NULL values are created by `numpy.empty_like` (`46848`)
- Fixed regression in `concat` materializing the `Index` during sorting even if the `Index` was already sorted (`47501`)
- Fixed regression in `concat` or `merge` handling of all-NaN ExtensionArrays with custom attributes (`47762`)
- Fixed regression in calling bitwise numpy ufuncs (for example, ``np.bitwise_and``) on Index objects (`46769`)
- Fixed regression in `cut` when using a ``datetime64`` IntervalIndex as bins (`46218`)
- Fixed regression in `DataFrame.select_dtypes` where ``include="number"`` included `BooleanDtype` (`46870`)
- Fixed regression in `DataFrame.loc` raising error when indexing with a ``NamedTuple`` (`48124`)
- Fixed regression in `DataFrame.loc` not updating the cache correctly after values were set (`47867`)
- Fixed regression in `DataFrame.loc` not aligning index in some cases when setting a `DataFrame` (`47578`)
- Fixed regression in `DataFrame.loc` setting a length-1 array like value to a single value in the DataFrame (`46268`)
- Fixed regression when slicing with `DataFrame.loc` with `DatetimeIndex` with a `.DateOffset` object for its ``freq`` (`46671`)
- Fixed regression in setting ``None`` or non-string value into a ``string``-dtype Series using a mask (`47628`)
- Fixed regression in updating a DataFrame column through Series ``__setitem__`` (using chained assignment) not updating column values inplace and using too much memory (`47172`)
- Fixed regression in `DataFrame.select_dtypes` returning a view on the original DataFrame (`48090`)
- Fixed regression using custom Index subclasses (for example, used in xarray) with `~DataFrame.reset_index` or `Index.insert` (`47071`)
- Fixed regression in `~Index.intersection` when the `DatetimeIndex` has dates crossing daylight savings time (`46702`)
- Fixed regression in `merge` throwing an error when passing a `Series` with a multi-level name (`47946`)
- Fixed regression in `DataFrame.eval` creating a copy when updating inplace (`47449`)
- Fixed regression where getting a row using `DataFrame.iloc` with `SparseDtype` would raise (`46406`)  .. ---  .. _whatsnew_144.bug_fixes:  Bug fixes
~~~
- The ``FutureWarning`` raised when passing arguments (other than ``filepath_or_buffer``) as positional in `read_csv` is now raised at the correct stacklevel (`47385`)
- Bug in `DataFrame.to_sql` when ``method`` was a ``callable`` that did not return an ``int`` and would raise a ``TypeError`` (`46891`)
- Bug in `.DataFrameGroupBy.value_counts` where ``subset`` had no effect (`46383`)
- Bug when getting values with `DataFrame.loc` with a list of keys causing an internal inconsistency that could lead to a disconnect between ``frame.at[x, y]`` vs ``frame[y].loc[x]`` (`22372`)
- Bug in the `Series.dt.strftime` accessor return a float instead of object dtype Series for all-NaT input, which also causes a spurious deprecation warning (`45858`)  .. ---  .. _whatsnew_144.other:  Other
~~~
- The minimum version of Cython needed to compile pandas is now ``0.29.32`` (`47978`)  .. ---  .. _whatsnew_144.contributors:  Contributors
~~~  .. contributors:: v1.4.3..v1.4.4|HEAD

These are the changes in pandas 1.5.0. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---
.. _whatsnew_150.enhancements:  Enhancements
~~~  .. _whatsnew_150.enhancements.pandas-stubs:  ``pandas-stubs``
^^^  The ``pandas-stubs`` library is now supported by the pandas development team, providing type stubs for the pandas API. Please visit
https://github.com/pandas-dev/pandas-stubs for more information.  We thank VirtusLab and Microsoft for their initial, significant contributions to ``pandas-stubs``  .. _whatsnew_150.enhancements.arrow:  Native PyArrow-backed ExtensionArray
^^^  With Pyarrow_ installed, users can now create pandas objects
that are backed by a ``pyarrow.ChunkedArray`` and ``pyarrow.DataType``.  The `dtype`` argument can accept a string of a `pyarrow data type_
with ``pyarrow`` in brackets e.g. ``"int64[pyarrow]"`` or, for pyarrow data types that take parameters, a `ArrowDtype`
initialized with a ``pyarrow.DataType``.  .. ipython:: python  import pyarrow as pa  ser_float = pd.Series([1.0, 2.0, None], dtype="float32[pyarrow]")  ser_float  list_of_int_type = pd.ArrowDtype(pa.list_(pa.int64()))  ser_list = pd.Series([[1, 2], [3, None]], dtype=list_of_int_type)  ser_list  ser_list.take([1, 0])  ser_float * 5  ser_float.mean()  ser_float.dropna()  Most operations are supported and have been implemented using pyarrow compute_ functions.
We recommend installing the latest version of PyArrow to access the most recently implemented compute functions.  .. warning::  This feature is experimental, and the API can change in a future release without warning.  .. _whatsnew_150.enhancements.dataframe_interchange:  DataFrame interchange protocol implementation
^^^  Pandas now implement the DataFrame interchange API spec.
See the full details on the API at https://data-apis.org/dataframe-protocol/latest/index.html  The protocol consists of two parts:  - New method `DataFrame.__dataframe__` which produces the interchange object.  It effectively "exports" the pandas dataframe as an interchange object so  any other library which has the protocol implemented can "import" that dataframe  without knowing anything about the producer except that it makes an interchange object.
- New function `pandas.api.interchange.from_dataframe` which can take  an arbitrary interchange object from any conformant library and construct a  pandas DataFrame out of it.  .. _whatsnew_150.enhancements.styler:  Styler
^^^  The most notable development is the new method `.Styler.concat` which
allows adding customised footer rows to visualise additional calculations on the data,
e.g. totals and counts etc. (`43875`, `46186`)  Additionally there is an alternative output method `.Styler.to_string`,
which allows using the Styler's formatting methods to create, for example, CSVs (`44502`).  A new feature `.Styler.relabel_index` is also made available to provide full customisation of the display of
index or column headers (`47864`)  Minor feature improvements are:  - Adding the ability to render ``border`` and ``border-{side}`` CSS properties in Excel (`42276`)  - Making keyword arguments consist: `.Styler.highlight_null` now accepts ``color`` and deprecates ``null_color`` although this remains backwards compatible (`45907`)  .. _whatsnew_150.enhancements.resample_group_keys:  Control of index with ``group_keys`` in `DataFrame.resample`
^^^  The argument ``group_keys`` has been added to the method `DataFrame.resample`.
As with `DataFrame.groupby`, this argument controls the whether each group is added

to the index in the resample when `.Resampler.apply` is used.  .. warning::  Not specifying the ``group_keys`` argument will retain the  previous behavior and emit a warning if the result will change  by specifying ``group_keys=False``. In a future version  of pandas, not specifying ``group_keys`` will default to  the same behavior as ``group_keys=False``.  .. ipython:: python  df = pd.DataFrame(  {'a': range(6)},  index=pd.date_range("2021-01-01", periods=6, freq="8H")  )  df.resample("D", group_keys=True).apply(lambda x: x)  df.resample("D", group_keys=False).apply(lambda x: x)  Previously, the resulting index would depend upon the values returned by ``apply``,
as seen in the following example.  .. code-block:: ipython  In [1]: # pandas 1.3  In [2]: df.resample("D").apply(lambda x: x)  Out[2]:  a  2021-01-01 000  0  2021-01-01 0800  1  2021-01-01 1600  2  2021-01-02 000  3  2021-01-02 0800  4  2021-01-02 1600  5  In [3]: df.resample("D").apply(lambda x: x.reset_index())  Out[3]:  index  a  2021-01-01 0 2021-01-01 000  0  1 2021-01-01 0800  1  2 2021-01-01 1600  2  2021-01-02 0 2021-01-02 000  3  1 2021-01-02 0800  4  2 2021-01-02 1600  5  .. _whatsnew_150.enhancements.from_dummies:  from_dummies
^^^  Added new function `~pandas.from_dummies` to convert a dummy coded `DataFrame` into a categorical `DataFrame`.  .. ipython:: python  import pandas as pd  df = pd.DataFrame({"col1_a": [1, 0, 1], "col1_b": [0, 1, 0],  "col2_a": [0, 1, 0], "col2_b": [1, 0, 0],  "col2_c": [0, 0, 1]})  pd.from_dummies(df, sep="_")  .. _whatsnew_150.enhancements.orc:  Writing to ORC files
^^^  The new method `DataFrame.to_orc` allows writing to ORC files (`43864`).  This functionality depends the pyarrow_ library. For more details, see `the IO docs on ORC <io.orc>`.  .. warning::  * It is *highly recommended* to install pyarrow using conda due to some issues occurred by pyarrow.  * `~pandas.DataFrame.to_orc` requires pyarrow>=7.0.0.  * `~pandas.DataFrame.to_orc` is not supported on Windows yet, you can find valid environments on `install optional dependencies <install.warn_orc>`.  * For supported dtypes please refer to supported ORC features in Arrow_.  * Currently timezones in datetime columns are not preserved when a dataframe is converted into ORC files.  .. code-block:: python  df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]})  df.to_orc("./out.orc")  .. _whatsnew_150.enhancements.tar:  Reading directly from TAR archives
^^^  I/O methods like `read_csv` or `DataFrame.to_json` now allow reading and writing
directly on TAR archives (`44787`).  .. code-block:: python  df = pd.read_csv("./movement.tar.gz")  # ...  df.to_csv("./out.tar.gz")  This supports ``.tar``, ``.tar.gz``, ``.tar.bz`` and ``.tar.xz2`` archives.
The used compression method is inferred from the filename.
If the compression method cannot be inferred, use the ``compression`` argument:  .. code-block:: python  df = pd.read_csv(some_file_obj, compression={"method": "tar", "mode": "r:gz"}) # noqa F821  (``mode`` being one of ``tarfile.open``'s modes: https://docs.python.org/3/library/tarfile.html#tarfile.open)  .. _whatsnew_150.enhancements.read_xml_dtypes:  read_xml now supports ``dtype``, ``converters``, and ``parse_dates``
^^^  Similar to other IO methods, `pandas.read_xml` now supports assigning specific dtypes to columns,

apply converter methods, and parse dates (`43567`).  .. ipython:: python  xml_dates = """<?xml version='1.0' encoding='utf-8'?>  <data>  <row>  <shape>square</shape>  <degrees>00360</degrees>  <sides>4.0</sides>  <date>2020-01-01</date>  </row>  <row>  <shape>circle</shape>  <degrees>00360</degrees>  <sides/>  <date>2021-01-01</date>  </row>  <row>  <shape>triangle</shape>  <degrees>00180</degrees>  <sides>3.0</sides>  <date>2022-01-01</date>  </row>  </data>"""  df = pd.read_xml(  xml_dates,  dtype={'sides': 'Int64'},  converters={'degrees': str},  parse_dates=['date']  )  df  df.dtypes  .. _whatsnew_150.enhancements.read_xml_iterparse:  read_xml now supports large XML using ``iterparse``
^^^  For very large XML files that can range in hundreds of megabytes to gigabytes, `pandas.read_xml`
now supports parsing such sizeable files using `lxml's iterparse`_ and `etree's iterparse`_
which are memory-efficient methods to iterate through XML trees and extract specific elements
and attributes without holding entire tree in memory (`45442`).  .. code-block:: ipython  In [1]: df = pd.read_xml(  ...  "/path/to/downloaded/enwikisource-latest-pages-articles.xml",  ...  iterparse = {"page": ["title", "ns", "id"]})  ...  )  df  Out[2]:  title  ns  id  0  Gettysburg Address  0  21450  1  Main Page  0  42950  2  Declaration by United Nations  0  8435  3  Constitution of the United States of America  0  8435  4  Declaration of Independence (Israel)  0  17858  ...  ...  ...  ...  3578760  Page:Black cat 1897 07 v2 n10.pdf/17  104  219649  3578761  Page:Black cat 1897 07 v2 n10.pdf/43  104  219649  3578762  Page:Black cat 1897 07 v2 n10.pdf/44  104  219649  3578763  The History of Tom Jones, a Foundling/Book IX  0  12084291  3578764  Page:Shakespeare of Stratford (1926) Yale.djvu/91  104  21450  [3578765 rows x 3 columns]  .. _`lxml's iterparse`: https://lxml.de/3.2/parsing.html#iterparse-and-iterwalk
.. _`etree's iterparse`: https://docs.python.org/3/library/xml.etree.elementtree.html#xml.etree.ElementTree.iterparse  .. _whatsnew_150.enhancements.copy_on_write:  Copy on Write
^^^  A new feature ``copy_on_write`` was added (`46958`). Copy on write ensures that
any DataFrame or Series derived from another in any way always behaves as a copy.
Copy on write disallows updating any other object than the object the method
was applied to.  Copy on write can be enabled through:  .. code-block:: python  pd.set_option("mode.copy_on_write", True)  pd.options.mode.copy_on_write = True  Alternatively, copy on write can be enabled locally through:  .. code-block:: python  with pd.option_context("mode.copy_on_write", True):  ...  Without copy on write, the parent `DataFrame` is updated when updating a child
`DataFrame` that was derived from this `DataFrame`.  .. ipython:: python  df = pd.DataFrame({"foo": [1, 2, 3], "bar": 1})  view = df["foo"]  view.iloc[0]  df  With copy on write enabled, df won't be updated anymore:  .. ipython:: python  with pd.option_context("mode.copy_on_write", True):  df = pd.DataFrame({"foo": [1, 2, 3], "bar": 1})  view = df["foo"]  view.iloc[0]  df  A more detailed explanation can be found here.  .. _whatsnew_150.enhancements.other:  Other enhancements
^^^
- `Series.map` now raises when ``arg`` is dict but ``na_action`` is not either ``None`` or ``'ignore'`` (`46588`)
- `MultiIndex.to_frame` now supports the argument ``allow_duplicates`` and raises on duplicate labels if it is missing or False (`45245`)
- `.StringArray` now accepts array-likes containing nan-likes (``None``, ``np.nan``) for the ``values`` parameter in its constructor in addition to strings and `pandas.NA`. (`40839`)
- Improved the rendering of ``categories`` in `CategoricalIndex` (`45218`)
- `DataFrame.plot` will now allow the ``subplots`` parameter to be a list of iterables specifying column groups, so that columns may be grouped together in the same subplot (`29688`).

- `to_numeric` now preserves float64 arrays when downcasting would generate values not representable in float32 (`43693`)
- `Series.reset_index` and `DataFrame.reset_index` now support the argument ``allow_duplicates`` (`44410`)
- .DataFrameGroupBy.min`, `.SeriesGroupBy.min`, `.DataFrameGroupBy.max`, and `.SeriesGroupBy.max` now supports `Numba execution with the ``engine`` keyword (`45428`)
- `read_csv` now supports ``defaultdict`` as a ``dtype`` parameter (`41574`)
- `DataFrame.rolling` and `Series.rolling` now support a ``step`` parameter with fixed-length windows (`15354`)
- Implemented a ``bool``-dtype `Index`, passing a bool-dtype array-like to ``pd.Index`` will now retain ``bool`` dtype instead of casting to ``object`` (`45061`)
- Implemented a complex-dtype `Index`, passing a complex-dtype array-like to ``pd.Index`` will now retain complex dtype instead of casting to ``object`` (`45845`)
- `Series` and `DataFrame` with `IntegerDtype` now supports bitwise operations (`34463`)
- Add ``milliseconds`` field support for `.DateOffset` (`43371`)
- `DataFrame.where` tries to maintain dtype of `DataFrame` if fill value can be cast without loss of precision (`45582`)
- `DataFrame.reset_index` now accepts a ``names`` argument which renames the index names (`6878`)
- `concat` now raises when ``levels`` is given but ``keys`` is None (`46653`)
- `concat` now raises when ``levels`` contains duplicate values (`46653`)
- Added ``numeric_only`` argument to `DataFrame.corr`, `DataFrame.corrwith`, `DataFrame.cov`, `DataFrame.idxmin`, `DataFrame.idxmax`, `.DataFrameGroupBy.idxmin`, `.DataFrameGroupBy.idxmax`, `.DataFrameGroupBy.var`, `.SeriesGroupBy.var`, `.DataFrameGroupBy.std`, `.SeriesGroupBy.std`, `.DataFrameGroupBy.sem`, `.SeriesGroupBy.sem`, and `.DataFrameGroupBy.quantile` (`46560`)
- A `errors.PerformanceWarning` is now thrown when using ``string[pyarrow]`` dtype with methods that don't dispatch to ``pyarrow.compute`` methods (`42613`, `46725`)
- Added ``validate`` argument to `DataFrame.join` (`46622`)
- Added ``numeric_only`` argument to `.Resampler.sum`, `.Resampler.prod`, `.Resampler.min`, `.Resampler.max`, `.Resampler.first`, and `.Resampler.last` (`46442`)
- ``times`` argument in `.ExponentialMovingWindow` now accepts ``np.timedelta64`` (`47003`)
- `.DataError`, `.SpecificationError`, `.SettingWithCopyError`, `.SettingWithCopyWarning`, `.NumExprClobberingError`, `.UndefinedVariableError`, `.IndexingError`, `.PyperclipException`, `.PyperclipWindowsException`, `.CSSWarning`, `.PossibleDataLossError`, `.ClosedFileError`, `.IncompatibilityWarning`, `.AttributeConflictWarning`, `.DatabaseError`, `.PossiblePrecisionLoss`, `.ValueLabelTypeMismatch`, `.InvalidColumnName`, and `.CategoricalConversionWarning` are now exposed in ``pandas.errors`` (`27656`)
- Added ``check_like`` argument to `testing.assert_series_equal` (`47247`)
- Add support for `.DataFrameGroupBy.ohlc` and `.SeriesGroupBy.ohlc` for extension array dtypes (`37493`)
- Allow reading compressed SAS files with `read_sas` (e.g., ``.sas7bdat.gz`` files)
- `pandas.read_html` now supports extracting links from table cells (`13141`)
- `DatetimeIndex.astype` now supports casting timezone-naive indexes to ``datetime64[s]``, ``datetime64[ms]``, and ``datetime64[us]``, and timezone-aware indexes to the corresponding ``datetime64[unit, tzname]`` dtypes (`47579`)
- `Series` reducers (e.g. ``min``, ``max``, ``sum``, ``mean``) will now successfully operate when the dtype is numeric and ``numeric_only=True`` is provided; previously this would raise a ``NotImplementedError`` (`47500`)
- `RangeIndex.union` now can return a `RangeIndex` instead of a `Int64Index` if the resulting values are equally spaced (`47557`, `43885`)

- `DataFrame.compare` now accepts an argument ``result_names`` to allow the user to specify the result's names of both left and right DataFrame which are being compared. This is by default ``'self'`` and ``'other'`` (`44354`)
- `DataFrame.quantile` gained a ``method`` argument that can accept ``table`` to evaluate multi-column quantiles (`43881`)
- `Interval` now supports checking whether one interval is contained by another interval (`46613`)
- Added ``copy`` keyword to `Series.set_axis` and `DataFrame.set_axis` to allow user to set axis on a new object without necessarily copying the underlying data (`47932`)
- The method `.ExtensionArray.factorize` accepts ``use_na_sentinel=False`` for determining how null values are to be treated (`46601`)
- The ``Dockerfile`` now installs a dedicated ``pandas-dev`` virtual environment for pandas development instead of using the ``base`` environment (`48427`)  .. ---
.. _whatsnew_150.notable_bug_fixes:  Notable bug fixes
~~~  These are bug fixes that might have notable behavior changes.  .. _whatsnew_150.notable_bug_fixes.groupby_transform_dropna:  Using ``dropna=True`` with ``groupby`` transforms
^^^  A transform is an operation whose result has the same size as its input. When the
result is a `DataFrame` or `Series`, it is also required that the
index of the result matches that of the input. In pandas 1.4, using
`.DataFrameGroupBy.transform` or `.SeriesGroupBy.transform` with null
values in the groups and ``dropna=True`` gave incorrect results. Demonstrated by the
examples below, the incorrect results either contained incorrect values, or the result
did not have the same index as the input.  .. ipython:: python  df = pd.DataFrame({'a': [1, 1, np.nan], 'b': [2, 3, 4]})  *Old behavior*:  .. code-block:: ipython  In [3]: # Value in the last row should be np.nan  df.groupby('a', dropna=True).transform('sum')  Out[3]:  b  0  5  1  5  2  5  In [3]: # Should have one additional row with the value np.nan  df.groupby('a', dropna=True).transform(lambda x: x.sum())  Out[3]:  b  0  5  1  5  In [3]: # The value in the last row is np.nan interpreted as an integer  df.groupby('a', dropna=True).transform('ffill')  Out[3]:  b  0  2  1  3  2 -9223372036854775808  In [3]: # Should have one additional row with the value np.nan  df.groupby('a', dropna=True).transform(lambda x: x)  Out[3]:  b  0  2  1  3  *New behavior*:  .. ipython:: python  df.groupby('a', dropna=True).transform('sum')  df.groupby('a', dropna=True).transform(lambda x: x.sum())  df.groupby('a', dropna=True).transform('ffill')  df.groupby('a', dropna=True).transform(lambda x: x)  .. _whatsnew_150.notable_bug_fixes.to_json_incorrectly_localizing_naive_timestamps:  Serializing tz-naive Timestamps with to_json() with ``iso_dates=True``
^^^  `DataFrame.to_json`, `Series.to_json`, and `Index.to_json`
would incorrectly localize DatetimeArrays/DatetimeIndexes with tz-naive Timestamps
to UTC. (`38760`)  Note that this patch does not fix the localization of tz-aware Timestamps to UTC
upon serialization. (Related issue `12997`)  *Old Behavior*  .. ipython:: python  index = pd.date_range(  start='2020-12-28 000',  end='2020-12-28 0200',  freq='1H',  )  a = pd.Series(  data=range(3),  index=index,  )  .. code-block:: ipython  In [4]: a.to_json(date_format='iso')  Out[4]: '{"2020-12-28T000.000Z":0,"2020-12-28T0100.000Z":1,"2020-12-28T0200.000Z":2}'  In [5]: pd.read_json(a.to_json(date_format='iso'), typ="series").index == a.index  Out[5]: array([False, False, False])  *New Behavior*  .. ipython:: python  a.to_json(date_format='iso')  # Roundtripping now works  pd.read_json(a.to_json(date_format='iso'), typ="series").index == a.index  .. _whatsnew_150.notable_bug_fixes.groupby_value_counts_categorical:  DataFrameGroupBy.value_counts with non-grouping categorical columns and ``observed=True``

^^^  Calling `.DataFrameGroupBy.value_counts` with ``observed=True`` would incorrectly drop non-observed categories of non-grouping columns (`46357`).  .. code-block:: ipython  In [6]: df = pd.DataFrame(["a", "b", "c"], dtype="category").iloc[0:2]  In [7]: df  Out[7]:  0  0  a  1  b  *Old Behavior*  .. code-block:: ipython  In [8]: df.groupby(level=0, observed=True).value_counts()  Out[8]:  0  a  1  1  b  1  dtype: int64  *New Behavior*  .. code-block:: ipython  In [9]: df.groupby(level=0, observed=True).value_counts()  Out[9]:  0  a  1  1  a  0  b  1  0  b  0  c  0  1  c  0  dtype: int64  .. ---
.. _whatsnew_150.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_150.api_breaking.deps:  Increased minimum versions for dependencies
^^^
| numpy  | 1.20.3  |  X  |  X  |
| mypy (dev)  | 0.971  |  |  X  |
| beautifulsoup4  | 4.9.3  |  |  X  |
| blosc  | 1.21.0  |  |  X  |
| bottleneck  | 1.3.2  |  |  X  |
| fsspec  | 2021.07.0  |  |  X  |
| hypothesis  | 6.13.0  |  |  X  |
| gcsfs  | 2021.07.0  |  |  X  |
| jinja2  | 3.0.0  |  |  X  |
| lxml  | 4.6.3  |  |  X  |
| numba  | 0.53.1  |  |  X  |
| numexpr  | 2.7.3  |  |  X  |
| openpyxl  | 3.0.7  |  |  X  |
| pandas-gbq  | 0.15.0  |  |  X  |
| psycopg2  | 2.8.6  |  |  X  |
| pymysql  | 1.0.2  |  |  X  |
| pyreadstat  | 1.1.2  |  |  X  |
| pyxlsb  | 1.0.8  |  |  X  |
| s3fs  | 2021.08.0  |  |  X  |
| scipy  | 1.7.1  |  |  X  |
| sqlalchemy  | 1.4.16  |  |  X  |
| tabulate  | 0.8.9  |  |  X  |
| xarray  | 0.19.0  |  |  X  |
| xlsxwriter  | 1.4.3  |  |  X  |
| beautifulsoup4  |4.9.3  |  X  |
| blosc  |1.21.0  |  X  |
| bottleneck  |1.3.2  |  X  |
| brotlipy  |0.7.0  |  |
| fastparquet  |0.4.0  |  |
| fsspec  |2021.08.0  |  X  |
| html5lib  |1.1  |  |
| hypothesis  |6.13.0  |  X  |
| gcsfs  |2021.08.0  |  X  |
| jinja2  |3.0.0  |  X  |
| lxml  |4.6.3  |  X  |
| matplotlib  |3.3.2  |  |
| numba  |0.53.1  |  X  |
| numexpr  |2.7.3  |  X  |
| odfpy  |1.4.1  |  |
| openpyxl  |3.0.7  |  X  |
| pandas-gbq  |0.15.0  |  X  |
| psycopg2  |2.8.6  |  X  |
| pyarrow  |1.0.1  |  |
| pymysql  |1.0.2  |  X  |
| pyreadstat  |1.1.2  |  X  |
| pytables  |3.6.1  |  |
| python-snappy  |0.6.0  |  |
| pyxlsb  |1.0.8  |  X  |
| s3fs  |2021.08.0  |  X  |
| scipy  |1.7.1  |  X  |
| sqlalchemy  |1.4.16  |  X  |
| tabulate  |0.8.9  |  X  |
| tzdata  |2022a  |  |
| xarray  |0.19.0  |  X  |
| xlrd  |2.0.1  |  |
| xlsxwriter  |1.4.3  |  X  |
| xlwt  |1.3.0  |  |
| zstandard  |0.15.2  |  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  .. _whatsnew_150.api_breaking.other:  Other API changes

^^^  - BigQuery I/O methods `read_gbq` and `DataFrame.to_gbq` default to  ``auth_local_webserver = True``. Google has deprecated the  ``auth_local_webserver = False`` `"out of band" (copy-paste) flow  <https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html?m=1#disallowed-oob>`_.  The ``auth_local_webserver = False`` option is planned to stop working in  October 2022. (`46312`)
- `read_json` now raises ``FileNotFoundError`` (previously ``ValueError``) when input is a string ending in ``.json``, ``.json.gz``, ``.json.bz2``, etc. but no such file exists. (`29102`)
- Operations with `Timestamp` or `Timedelta` that would previously raise ``OverflowError`` instead raise ``OutOfBoundsDatetime`` or ``OutOfBoundsTimedelta`` where appropriate (`47268`)
- When `read_sas` previously returned ``None``, it now returns an empty `DataFrame` (`47410`)
- `DataFrame` constructor raises if ``index`` or ``columns`` arguments are sets (`47215`)  .. ---
.. _whatsnew_150.deprecations:  Deprecations
~~~  .. warning::  In the next major version release, 2.0, several larger API changes are being considered without a formal deprecation such as  making the standard library zoneinfo the default timezone implementation instead of ``pytz``,  having the `Index` support all data types instead of having multiple subclasses (`CategoricalIndex`, `Int64Index`, etc.), and more.  The changes under consideration are logged in this GitHub issue, and any  feedback or concerns are welcome.  .. _whatsnew_150.deprecations.int_slicing_series:  Label-based integer slicing on a Series with an Int64Index or RangeIndex
^^^  In a future version, integer slicing on a `Series` with a `Int64Index` or `RangeIndex` will be treated as *label-based*, not positional. This will make the behavior consistent with other `Series.__getitem__` and `Series.__setitem__` behaviors (`45162`).  For example:  .. ipython:: python  ser = pd.Series([1, 2, 3, 4, 5], index=[2, 3, 5, 7, 11])  In the old behavior, ``ser[2:4]`` treats the slice as positional:  *Old behavior*:  .. code-block:: ipython  In [3]: ser[2:4]  Out[3]:  5  3  7  4  dtype: int64  In a future version, this will be treated as label-based:  *Future behavior*:  .. code-block:: ipython  In [4]: ser.loc[2:4]  Out[4]:  2  1  3  2  dtype: int64  To retain the old behavior, use ``series.iloc[i:j]``. To get the future behavior,
use ``series.loc[i:j]``.  Slicing on a `DataFrame` will not be affected.  .. _whatsnew_150.deprecations.excel_writer_attributes:  `ExcelWriter` attributes
^^^  All attributes of `ExcelWriter` were previously documented as not
public. However some third party Excel engines documented accessing
``ExcelWriter.book`` or ``ExcelWriter.sheets``, and users were utilizing these
and possibly other attributes. Previously these attributes were not safe to use;
e.g. modifications to ``ExcelWriter.book`` would not update ``ExcelWriter.sheets``
and conversely. In order to support this, pandas has made some attributes public
and improved their implementations so that they may now be safely used. (`45572`)  The following attributes are now public and considered safe to access.  - ``book``  - ``check_extension``  - ``close``  - ``date_format``  - ``datetime_format``  - ``engine``  - ``if_sheet_exists``  - ``sheets``  - ``supported_extensions``  The following attributes have been deprecated. They now raise a ``FutureWarning``
when accessed and will be removed in a future version. Users should be aware
that their usage is considered unsafe, and can lead to unexpected results.  - ``cur_sheet``  - ``handles``  - ``path``  - ``save``  - ``write_cells``  See the documentation of `ExcelWriter` for further details.  .. _whatsnew_150.deprecations.group_keys_in_apply:  Using ``group_keys`` with transformers in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply`

^^^  In previous versions of pandas, if it was inferred that the function passed to
`.DataFrameGroupBy.apply` or `.SeriesGroupBy.apply` was a transformer (i.e. the resulting index was equal to
the input index), the ``group_keys`` argument of `DataFrame.groupby` and
`Series.groupby` was ignored and the group keys would never be added to
the index of the result. In the future, the group keys will be added to the index
when the user specifies ``group_keys=True``.  As ``group_keys=True`` is the default value of `DataFrame.groupby` and
`Series.groupby`, not specifying ``group_keys`` with a transformer will
raise a ``FutureWarning``. This can be silenced and the previous behavior
retained by specifying ``group_keys=False``.  .. _whatsnew_150.deprecations.setitem_column_try_inplace:  _ see also _whatsnew_130.notable_bug_fixes.setitem_column_try_inplace  Inplace operation when setting values with ``loc`` and ``iloc``
^^^
Most of the time setting values with `DataFrame.iloc` attempts to set values
inplace, only falling back to inserting a new array if necessary. There are
some cases where this rule is not followed, for example when setting an entire
column from an array with different dtype:  .. ipython:: python  df = pd.DataFrame({'price': [11.1, 12.2]}, index=['book1', 'book2'])  original_prices = df['price']  new_prices = np.array([98, 99])  *Old behavior*:  .. code-block:: ipython  In [3]: df.iloc[:, 0] = new_prices  In [4]: df.iloc[:, 0]  Out[4]:  book1  98  book2  99  Name: price, dtype: int64  In [5]: original_prices  Out[5]:  book1  11.1  book2  12.2  Name: price, float: 64  This behavior is deprecated. In a future version, setting an entire column with
iloc will attempt to operate inplace.  *Future behavior*:  .. code-block:: ipython  In [3]: df.iloc[:, 0] = new_prices  In [4]: df.iloc[:, 0]  Out[4]:  book1  98.0  book2  99.0  Name: price, dtype: float64  In [5]: original_prices  Out[5]:  book1  98.0  book2  99.0  Name: price, dtype: float64  To get the old behavior, use `DataFrame.__setitem__` directly:  .. code-block:: ipython  In [3]: df[df.columns[0]] = new_prices  In [4]: df.iloc[:, 0]  Out[4]  book1  98  book2  99  Name: price, dtype: int64  In [5]: original_prices  Out[5]:  book1  11.1  book2  12.2  Name: price, dtype: float64  To get the old behaviour when ``df.columns`` is not unique and you want to
change a single column by index, you can use `DataFrame.isetitem`, which
has been added in pandas 1.5:  .. code-block:: ipython  In [3]: df_with_duplicated_cols = pd.concat([df, df], axis='columns')  In [3]: df_with_duplicated_cols.isetitem(0, new_prices)  In [4]: df_with_duplicated_cols.iloc[:, 0]  Out[4]:  book1  98  book2  99  Name: price, dtype: int64  In [5]: original_prices  Out[5]:  book1  11.1  book2  12.2  Name: 0, dtype: float64  .. _whatsnew_150.deprecations.numeric_only_default:  ``numeric_only`` default value
^^^  Across the `DataFrame`, `.DataFrameGroupBy`, and `.Resampler` operations such as
``min``, ``sum``, and ``idxmax``, the default
value of the ``numeric_only`` argument, if it exists at all, was inconsistent.
Furthermore, operations with the default value ``None`` can lead to surprising
results. (`46560`)  .. code-block:: ipython  In [1]: df = pd.DataFrame({"a": [1, 2], "b": ["x", "y"]})  In [2]: # Reading the next line without knowing the contents of df, one would  # expect the result to contain the products for both columns a and b.  df[["a", "b"]].prod()  Out[2]:  a  2  dtype: int64  To avoid this behavior, the specifying the value ``numeric_only=None`` has been
deprecated, and will be removed in a future version of pandas. In the future,
all operations with a ``numeric_only`` argument will default to ``False``. Users
should either call the operation only with columns that can be operated on, or
specify ``numeric_only=True`` to operate only on Boolean, integer, and float columns.  In order to support the transition to the new behavior, the following methods have
gained the ``numeric_only`` argument.  - `DataFrame.corr`

- `DataFrame.corrwith`
- `DataFrame.cov`
- `DataFrame.idxmin`
- `DataFrame.idxmax`
- `.DataFrameGroupBy.cummin`
- `.DataFrameGroupBy.cummax`
- `.DataFrameGroupBy.idxmin`
- `.DataFrameGroupBy.idxmax`
- `.DataFrameGroupBy.var`
- `.DataFrameGroupBy.std`
- `.DataFrameGroupBy.sem`
- `.DataFrameGroupBy.quantile`
- `.Resampler.mean`
- `.Resampler.median`
- `.Resampler.sem`
- `.Resampler.std`
- `.Resampler.var`
- `DataFrame.rolling` operations
- `DataFrame.expanding` operations
- `DataFrame.ewm` operations  .. _whatsnew_150.deprecations.other:  Other Deprecations
^^^
- Deprecated the keyword ``line_terminator`` in `DataFrame.to_csv` and `Series.to_csv`, use ``lineterminator`` instead; this is for consistency with `read_csv` and the standard library 'csv' module (`9568`)
- Deprecated behavior of `SparseArray.astype`, `Series.astype`, and `DataFrame.astype` with `SparseDtype` when passing a non-sparse ``dtype``. In a future version, this will cast to that non-sparse dtype instead of wrapping it in a `SparseDtype` (`34457`)
- Deprecated behavior of `DatetimeIndex.intersection` and `DatetimeIndex.symmetric_difference` (``union`` behavior was already deprecated in version 1.3.0) with mixed time zones; in a future version both will be cast to UTC instead of object dtype (`39328`, `45357`)
- Deprecated `DataFrame.iteritems`, `Series.iteritems`, `HDFStore.iteritems` in favor of `DataFrame.items`, `Series.items`, `HDFStore.items`  (`45321`)
- Deprecated `Series.is_monotonic` and `Index.is_monotonic` in favor of `Series.is_monotonic_increasing` and `Index.is_monotonic_increasing` (`45422`, `21335`)
- Deprecated behavior of `DatetimeIndex.astype`, `TimedeltaIndex.astype`, `PeriodIndex.astype` when converting to an integer dtype other than ``int64``. In a future version, these will convert to exactly the specified dtype (instead of always ``int64``) and will raise if the conversion overflows (`45034`)
- Deprecated the ``__array_wrap__`` method of DataFrame and Series, rely on standard numpy ufuncs instead (`45451`)
- Deprecated treating float-dtype data as wall-times when passed with a timezone to `Series` or `DatetimeIndex` (`45573`)
- Deprecated the behavior of `Series.fillna` and `DataFrame.fillna` with ``timedelta64[ns]`` dtype and incompatible fill value; in a future version this will cast to a common dtype (usually object) instead of raising, matching the behavior of other dtypes (`45746`)
- Deprecated the ``warn`` parameter in `infer_freq` (`45947`)
- Deprecated allowing non-keyword arguments in `.ExtensionArray.argsort` (`46134`)
- Deprecated treating all-bool ``object``-dtype columns as bool-like in `DataFrame.any` and `DataFrame.all` with ``bool_only=True``, explicitly cast to bool instead (`46188`)
- Deprecated behavior of method `DataFrame.quantile`, attribute ``numeric_only`` will default False. Including datetime/timedelta columns in the result (`7308`).
- Deprecated `Timedelta.freq` and `Timedelta.is_populated` (`46430`)
- Deprecated `Timedelta.delta` (`46476`)
- Deprecated passing arguments as positional in `DataFrame.any` and `Series.any` (`44802`)
- Deprecated passing positional arguments to `DataFrame.pivot` and `pivot` except ``data`` (`30228`)
- Deprecated the methods `DataFrame.mad`, `Series.mad`, and the corresponding groupby methods (`11787`)
- Deprecated positional arguments to `Index.join` except for ``other``, use keyword-only arguments instead of positional arguments (`46518`)
- Deprecated positional arguments to `StringMethods.rsplit` and `StringMethods.split` except for ``pat``, use keyword-only arguments instead of positional arguments (`47423`)
- Deprecated indexing on a timezone-naive `DatetimeIndex` using a string representing a timezone-aware datetime (`46903`, `36148`)
- Deprecated allowing ``unit="M"`` or ``unit="Y"`` in `Timestamp` constructor with a non-round float value (`47267`)
- Deprecated the ``display.column_space`` global configuration option (`7576`)

- Deprecated the argument ``na_sentinel`` in `factorize`, `Index.factorize`, and `.ExtensionArray.factorize`; pass ``use_na_sentinel=True`` instead to use the sentinel ``-1`` for NaN values and ``use_na_sentinel=False`` instead of ``na_sentinel=None`` to encode NaN values (`46910`)
- Deprecated `.DataFrameGroupBy.transform` not aligning the result when the UDF returned DataFrame (`45648`)
- Clarified warning from `to_datetime` when delimited dates can't be parsed in accordance to specified ``dayfirst`` argument (`46210`)
- Emit warning from `to_datetime` when delimited dates can't be parsed in accordance to specified ``dayfirst`` argument even for dates where leading zero is omitted (e.g. ``31/1/2001``) (`47880`)
- Deprecated `Series` and `Resampler` reducers (e.g. ``min``, ``max``, ``sum``, ``mean``) raising a ``NotImplementedError`` when the dtype is non-numric and ``numeric_only=True`` is provided; this will raise a ``TypeError`` in a future version (`47500`)
- Deprecated `Series.rank` returning an empty result when the dtype is non-numeric and ``numeric_only=True`` is provided; this will raise a ``TypeError`` in a future version (`47500`)
- Deprecated argument ``errors`` for `Series.mask`, `Series.where`, `DataFrame.mask`, and `DataFrame.where` as ``errors`` had no effect on this methods (`47728`)
- Deprecated arguments ``*args`` and ``**kwargs`` in `Rolling`, `Expanding`, and `ExponentialMovingWindow` ops. (`47836`)
- Deprecated the ``inplace`` keyword in `Categorical.set_ordered`, `Categorical.as_ordered`, and `Categorical.as_unordered` (`37643`)
- Deprecated setting a categorical's categories with ``cat.categories = ['a', 'b', 'c']``, use `Categorical.rename_categories` instead (`37643`)
- Deprecated unused arguments ``encoding`` and ``verbose`` in `Series.to_excel` and `DataFrame.to_excel` (`47912`)
- Deprecated the ``inplace`` keyword in `DataFrame.set_axis` and `Series.set_axis`, use ``obj = obj.set_axis(..., copy=False)`` instead (`48130`)
- Deprecated producing a single element when iterating over a `DataFrameGroupBy` or a `SeriesGroupBy` that has been grouped by a list of length 1; A tuple of length one will be returned instead (`42795`)
- Fixed up warning message of deprecation of `MultiIndex.lesort_depth` as public method, as the message previously referred to `MultiIndex.is_lexsorted` instead (`38701`)
- Deprecated the ``sort_columns`` argument in `DataFrame.plot` and `Series.plot` (`47563`).
- Deprecated positional arguments for all but the first argument of `DataFrame.to_stata` and `read_stata`, use keyword arguments instead (`48128`).
- Deprecated the ``mangle_dupe_cols`` argument in `read_csv`, `read_fwf`, `read_table` and `read_excel`. The argument was never implemented, and a new argument where the renaming pattern can be specified will be added instead (`47718`)
- Deprecated allowing ``dtype='datetime64'`` or ``dtype=np.datetime64`` in `Series.astype`, use "datetime64[ns]" instead (`47844`)  .. ---
.. _whatsnew_150.performance:  Performance improvements
~~~
- Performance improvement in `DataFrame.corrwith` for column-wise (axis=0) Pearson and Spearman correlation when other is a `Series` (`46174`)
- Performance improvement in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` for some user-defined DataFrame -> Series functions (`45387`)
- Performance improvement in `DataFrame.duplicated` when subset consists of only one column (`45236`)
- Performance improvement in `.DataFrameGroupBy.diff` and `.SeriesGroupBy.diff` (`16706`)
- Performance improvement in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` when broadcasting values for user-defined functions (`45708`)

- Performance improvement in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` for user-defined functions when only a single group exists (`44977`)
- Performance improvement in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` when grouping on a non-unique unsorted index (`46527`)
- Performance improvement in `DataFrame.loc` and `Series.loc` for tuple-based indexing of a `MultiIndex` (`45681`, `46040`, `46330`)
- Performance improvement in `.DataFrameGroupBy.var` and `.SeriesGroupBy.var` with ``ddof`` other than one (`48152`)
- Performance improvement in `DataFrame.to_records` when the index is a `MultiIndex` (`47263`)
- Performance improvement in `MultiIndex.values` when the MultiIndex contains levels of type DatetimeIndex, TimedeltaIndex or ExtensionDtypes (`46288`)
- Performance improvement in `merge` when left and/or right are empty (`45838`)
- Performance improvement in `DataFrame.join` when left and/or right are empty (`46015`)
- Performance improvement in `DataFrame.reindex` and `Series.reindex` when target is a `MultiIndex` (`46235`)
- Performance improvement when setting values in a pyarrow backed string array (`46400`)
- Performance improvement in `factorize` (`46109`)
- Performance improvement in `DataFrame` and `Series` constructors for extension dtype scalars (`45854`)
- Performance improvement in `read_excel` when ``nrows`` argument provided (`32727`)
- Performance improvement in `.Styler.to_excel` when applying repeated CSS formats (`47371`)
- Performance improvement in `MultiIndex.is_monotonic_increasing`  (`47458`)
- Performance improvement in `BusinessHour` ``str`` and ``repr`` (`44764`)
- Performance improvement in datetime arrays string formatting when one of the default strftime formats ``"%Y-%m-%d %H:%M:%S"`` or ``"%Y-%m-%d %H:%M:%S.%f"`` is used. (`44764`)
- Performance improvement in `Series.to_sql` and `DataFrame.to_sql` (`SQLiteTable`) when processing time arrays. (`44764`)
- Performance improvement to `read_sas` (`47404`)
- Performance improvement in ``argmax`` and ``argmin`` for `arrays.SparseArray` (`34197`)  .. ---
.. _whatsnew_150.bug_fixes:  Bug fixes
^^^
- Bug in `.Categorical.view` not accepting integer dtypes (`25464`)
- Bug in `.CategoricalIndex.union` when the index's categories are integer-dtype and the index contains ``NaN`` values incorrectly raising instead of casting to ``float64`` (`45362`)
- Bug in `concat` when concatenating two (or more) unordered `CategoricalIndex` variables, whose categories are permutations, yields incorrect index values (`24845`)  Datetimelike
^^^
- Bug in `DataFrame.quantile` with datetime-like dtypes and no rows incorrectly returning ``float64`` dtype instead of retaining datetime-like dtype (`41544`)
- Bug in `to_datetime` with sequences of ``np.str_`` objects incorrectly raising (`32264`)
- Bug in `Timestamp` construction when passing datetime components as positional arguments and ``tzinfo`` as a keyword argument incorrectly raising (`31929`)
- Bug in `Index.astype` when casting from object dtype to ``timedelta64[ns]`` dtype incorrectly casting ``np.datetime64("NaT")`` values to ``np.timedelta64("NaT")`` instead of raising (`45722`)
- Bug in `.SeriesGroupBy.value_counts` index when passing categorical column (`44324`)
- Bug in `DatetimeIndex.tz_localize` localizing to UTC failing to make a copy of the underlying data (`46460`)
- Bug in `DatetimeIndex.resolution` incorrectly returning "day" instead of "nanosecond" for nanosecond-resolution indexes (`46903`)
- Bug in `Timestamp` with an integer or float value and ``unit="Y"`` or ``unit="M"`` giving slightly-wrong results (`47266`)
- Bug in `.DatetimeArray` construction when passed another `.DatetimeArray` and ``freq=None`` incorrectly inferring the freq from the given array (`47296`)

- Bug in `to_datetime` where ``OutOfBoundsDatetime`` would be thrown even if ``errors=coerce`` if there were more than 50 rows (`45319`)
- Bug when adding a `DateOffset` to a `Series` would not add the ``nanoseconds`` field (`47856`)  Timedelta
^^^
- Bug in `astype_nansafe` astype("timedelta64[ns]") fails when np.nan is included (`45798`)
- Bug in constructing a `Timedelta` with a ``np.timedelta64`` object and a ``unit`` sometimes silently overflowing and returning incorrect results instead of raising ``OutOfBoundsTimedelta`` (`46827`)
- Bug in constructing a `Timedelta` from a large integer or float with ``unit="W"`` silently overflowing and returning incorrect results instead of raising ``OutOfBoundsTimedelta`` (`47268`)  Time Zones
^^^
- Bug in `Timestamp` constructor raising when passed a ``ZoneInfo`` tzinfo object (`46425`)  Numeric
^^^
- Bug in operations with array-likes with ``dtype="boolean"`` and `NA` incorrectly altering the array in-place (`45421`)
- Bug in arithmetic operations with nullable types without `NA` values not matching the same operation with non-nullable types (`48223`)
- Bug in ``floordiv`` when dividing by ``IntegerDtype`` ``0`` would return ``0`` instead of ``inf`` (`48223`)
- Bug in division, ``pow`` and ``mod`` operations on array-likes with ``dtype="boolean"`` not being like their ``np.bool_`` counterparts (`46063`)
- Bug in multiplying a `Series` with ``IntegerDtype`` or ``FloatingDtype`` by an array-like with ``timedelta64[ns]`` dtype incorrectly raising (`45622`)
- Bug in `mean` where the optional dependency ``bottleneck`` causes precision loss linear in the length of the array. ``bottleneck`` has been disabled for `mean` improving the loss to log-linear but may result in a performance decrease. (`42878`)  Conversion
^^^
- Bug in `DataFrame.astype` not preserving subclasses (`40810`)
- Bug in constructing a `Series` from a float-containing list or a floating-dtype ndarray-like (e.g. ``dask.Array``) and an integer dtype raising instead of casting like we would with an ``np.ndarray`` (`40110`)
- Bug in `Float64Index.astype` to unsigned integer dtype incorrectly casting to ``np.int64`` dtype (`45309`)
- Bug in `Series.astype` and `DataFrame.astype` from floating dtype to unsigned integer dtype failing to raise in the presence of negative values (`45151`)
- Bug in `array` with ``FloatingDtype`` and values containing float-castable strings incorrectly raising (`45424`)
- Bug when comparing string and datetime64ns objects causing ``OverflowError`` exception. (`45506`)
- Bug in metaclass of generic abstract dtypes causing `DataFrame.apply` and `Series.apply` to raise for the built-in function ``type`` (`46684`)
- Bug in `DataFrame.to_records` returning inconsistent numpy types if the index was a `MultiIndex` (`47263`)
- Bug in `DataFrame.to_dict` for ``orient="list"`` or ``orient="index"`` was not returning native types (`46751`)
- Bug in `DataFrame.apply` that returns a `DataFrame` instead of a `Series` when applied to an empty `DataFrame` and ``axis=1`` (`39111`)
- Bug when inferring the dtype from an iterable that is *not* a NumPy ``ndarray`` consisting of all NumPy unsigned integer scalars did not result in an unsigned integer dtype (`47294`)
- Bug in `DataFrame.eval` when pandas objects (e.g. ``'Timestamp'``) were column names (`44603`)  Strings
^^^
- Bug in `str.startswith` and `str.endswith` when using other series as parameter _pat_. Now raises ``TypeError`` (`3485`)
- Bug in `Series.str.zfill` when strings contain leading signs, padding '0' before the sign character rather than after as ``str.zfill`` from standard library (`20868`)  Interval
^^^
- Bug in `IntervalArray.__setitem__` when setting ``np.nan`` into an integer-backed array raising ``ValueError`` instead of ``TypeError`` (`45484`)

^^^
- Bug in `IntervalDtype` when using datetime64[ns, tz] as a dtype string (`46999`)  Indexing
^^^
- Bug in `DataFrame.iloc` where indexing a single row on a `DataFrame` with a single ExtensionDtype column gave a copy instead of a view on the underlying data (`45241`)
- Bug in `DataFrame.__getitem__` returning copy when `DataFrame` has duplicated columns even if a unique column is selected (`45316`, `41062`)
- Bug in `Series.align` does not create `MultiIndex` with union of levels when both MultiIndexes intersections are identical (`45224`)
- Bug in setting a NA value (``None`` or ``np.nan``) into a `Series` with int-based `IntervalDtype` incorrectly casting to object dtype instead of a float-based `IntervalDtype` (`45568`)
- Bug in indexing setting values into an ``ExtensionDtype`` column with ``df.iloc[:, i] = values`` with ``values`` having the same dtype as ``df.iloc[:, i]`` incorrectly inserting a new array instead of setting in-place (`33457`)
- Bug in `Series.__setitem__` with a non-integer `Index` when using an integer key to set a value that cannot be set inplace where a ``ValueError`` was raised instead of casting to a common dtype (`45070`)
- Bug in `DataFrame.loc` not casting ``None`` to ``NA`` when setting value as a list into `DataFrame` (`47987`)
- Bug in `Series.__setitem__` when setting incompatible values into a ``PeriodDtype`` or ``IntervalDtype`` `Series` raising when indexing with a boolean mask but coercing when indexing with otherwise-equivalent indexers; these now consistently coerce, along with `Series.mask` and `Series.where` (`45768`)
- Bug in `DataFrame.where` with multiple columns with datetime-like dtypes failing to downcast results consistent with other dtypes (`45837`)
- Bug in `isin` upcasting to ``float64`` with unsigned integer dtype and list-like argument without a dtype (`46485`)
- Bug in `Series.loc.__setitem__` and `Series.loc.__getitem__` not raising when using multiple keys without using a `MultiIndex` (`13831`)
- Bug in `Index.reindex` raising ``AssertionError`` when ``level`` was specified but no `MultiIndex` was given; level is ignored now (`35132`)
- Bug when setting a value too large for a `Series` dtype failing to coerce to a common type (`26049`, `32878`)
- Bug in `loc.__setitem__` treating ``range`` keys as positional instead of label-based (`45479`)
- Bug in `DataFrame.__setitem__` casting extension array dtypes to object when setting with a scalar key and `DataFrame` as value (`46896`)
- Bug in `Series.__setitem__` when setting a scalar to a nullable pandas dtype would not raise a ``TypeError`` if the scalar could not be cast (losslessly) to the nullable type (`45404`)
- Bug in `Series.__setitem__` when setting ``boolean`` dtype values containing ``NA`` incorrectly raising instead of casting to ``boolean`` dtype (`45462`)
- Bug in `Series.loc` raising with boolean indexer containing ``NA`` when `Index` did not match (`46551`)
- Bug in `Series.__setitem__` where setting `NA` into a numeric-dtype `Series` would incorrectly upcast to object-dtype rather than treating the value as ``np.nan`` (`44199`)
- Bug in `DataFrame.loc` when setting values to a column and right hand side is a dictionary (`47216`)
- Bug in `Series.__setitem__` with ``datetime64[ns]`` dtype, an all-``False`` boolean mask, and an incompatible value incorrectly casting to ``object`` instead of retaining ``datetime64[ns]`` dtype (`45967`)
- Bug in `Index.__getitem__`  raising ``ValueError`` when indexer is from boolean dtype with ``NA`` (`45806`)
- Bug in `Series.__setitem__` losing precision when enlarging `Series` with scalar (`32346`)
- Bug in `Series.mask` with ``inplace=True`` or setting values with a boolean mask with small integer dtypes incorrectly raising (`45750`)
- Bug in `DataFrame.mask` with ``inplace=True`` and ``ExtensionDtype`` columns incorrectly raising (`45577`)

- Bug in getting a column from a DataFrame with an object-dtype row index with datetime-like values: the resulting Series now preserves the exact object-dtype Index from the parent DataFrame (`42950`)
- Bug in `DataFrame.__getattribute__` raising ``AttributeError`` if columns have ``"string"`` dtype (`46185`)
- Bug in `DataFrame.compare` returning all ``NaN`` column when comparing extension array dtype and numpy dtype (`44014`)
- Bug in `DataFrame.where` setting wrong values with ``"boolean"`` mask for numpy dtype (`44014`)
- Bug in indexing on a `DatetimeIndex` with a ``np.str_`` key incorrectly raising (`45580`)
- Bug in `CategoricalIndex.get_indexer` when index contains ``NaN`` values, resulting in elements that are in target but not present in the index to be mapped to the index of the NaN element, instead of -1 (`45361`)
- Bug in setting large integer values into `Series` with ``float32`` or ``float16`` dtype incorrectly altering these values instead of coercing to ``float64`` dtype (`45844`)
- Bug in `Series.asof` and `DataFrame.asof` incorrectly casting bool-dtype results to ``float64`` dtype (`16063`)
- Bug in `NDFrame.xs`, `DataFrame.iterrows`, `DataFrame.loc` and `DataFrame.iloc` not always propagating metadata (`28283`)
- Bug in `DataFrame.sum` min_count changes dtype if input contains NaNs (`46947`)
- Bug in `IntervalTree` that lead to an infinite recursion. (`46658`)
- Bug in `PeriodIndex` raising ``AttributeError`` when indexing on ``NA``, rather than putting ``NaT`` in its place. (`46673`)
- Bug in `DataFrame.at` would allow the modification of multiple columns (`48296`)  Missing
^^^
- Bug in `Series.fillna` and `DataFrame.fillna` with ``downcast`` keyword not being respected in some cases where there are no NA values present (`45423`)
- Bug in `Series.fillna` and `DataFrame.fillna` with `IntervalDtype` and incompatible value raising instead of casting to a common (usually object) dtype (`45796`)
- Bug in `Series.map` not respecting ``na_action`` argument if mapper is a ``dict`` or `Series` (`47527`)
- Bug in `DataFrame.interpolate` with object-dtype column not returning a copy with ``inplace=False`` (`45791`)
- Bug in `DataFrame.dropna` allows to set both ``how`` and ``thresh`` incompatible arguments (`46575`)
- Bug in `DataFrame.fillna` ignored ``axis`` when `DataFrame` is single block (`47713`)  MultiIndex
^^^
- Bug in `DataFrame.loc` returning empty result when slicing a `MultiIndex` with a negative step size and non-null start/stop values (`46156`)
- Bug in `DataFrame.loc` raising when slicing a `MultiIndex` with a negative step size other than -1 (`46156`)
- Bug in `DataFrame.loc` raising when slicing a `MultiIndex` with a negative step size and slicing a non-int labeled index level (`46156`)
- Bug in `Series.to_numpy` where multiindexed Series could not be converted to numpy arrays when an ``na_value`` was supplied (`45774`)
- Bug in `MultiIndex.equals` not commutative when only one side has extension array dtype (`46026`)
- Bug in `MultiIndex.from_tuples` cannot construct Index of empty tuples (`45608`)  I/O
^^^
- Bug in `DataFrame.to_stata` where no error is raised if the `DataFrame` contains ``-np.inf`` (`45350`)
- Bug in `read_excel` results in an infinite loop with certain ``skiprows`` callables (`45585`)
- Bug in `DataFrame.info` where a new line at the end of the output is omitted when called on an empty `DataFrame` (`45494`)
- Bug in `read_csv` not recognizing line break for ``on_bad_lines="warn"`` for ``engine="c"`` (`41710`)
- Bug in `DataFrame.to_csv` not respecting ``float_format`` for ``Float64`` dtype (`45991`)
- Bug in `read_csv` not respecting a specified converter to index columns in all cases (`40589`)
- Bug in `read_csv` interpreting second row as `Index` names even when ``index_col=False`` (`46569`)

- Bug in `read_parquet` when ``engine="pyarrow"`` which caused partial write to disk when column of unsupported datatype was passed (`44914`)
- Bug in `DataFrame.to_excel` and `ExcelWriter` would raise when writing an empty DataFrame to a ``.ods`` file (`45793`)
- Bug in `read_csv` ignoring non-existing header row for ``engine="python"`` (`47400`)
- Bug in `read_excel` raising uncontrolled ``IndexError`` when ``header`` references non-existing rows (`43143`)
- Bug in `read_html` where elements surrounding ``<br>`` were joined without a space between them (`29528`)
- Bug in `read_csv` when data is longer than header leading to issues with callables in ``usecols`` expecting strings (`46997`)
- Bug in Parquet roundtrip for Interval dtype with ``datetime64[ns]`` subtype (`45881`)
- Bug in `read_excel` when reading a ``.ods`` file with newlines between xml elements (`45598`)
- Bug in `read_parquet` when ``engine="fastparquet"`` where the file was not closed on error (`46555`)
- `DataFrame.to_html` now excludes the ``border`` attribute from ``<table>`` elements when ``border`` keyword is set to ``False``.
- Bug in `read_sas` with certain types of compressed SAS7BDAT files (`35545`)
- Bug in `read_excel` not forward filling `MultiIndex` when no names were given (`47487`)
- Bug in `read_sas` returned ``None`` rather than an empty DataFrame for SAS7BDAT files with zero rows (`18198`)
- Bug in `DataFrame.to_string` using wrong missing value with extension arrays in `MultiIndex` (`47986`)
- Bug in `StataWriter` where value labels were always written with default encoding (`46750`)
- Bug in `StataWriterUTF8` where some valid characters were removed from variable names (`47276`)
- Bug in `DataFrame.to_excel` when writing an empty dataframe with `MultiIndex` (`19543`)
- Bug in `read_sas` with RLE-compressed SAS7BDAT files that contain 0x40 control bytes (`31243`)
- Bug in `read_sas` that scrambled column names (`31243`)
- Bug in `read_sas` with RLE-compressed SAS7BDAT files that contain 0x00 control bytes (`47099`)
- Bug in `read_parquet` with ``use_nullable_dtypes=True`` where ``float64`` dtype was returned instead of nullable ``Float64`` dtype (`45694`)
- Bug in `DataFrame.to_json` where ``PeriodDtype`` would not make the serialization roundtrip when read back with `read_json` (`44720`)
- Bug in `read_xml` when reading XML files with Chinese character tags and would raise ``XMLSyntaxError`` (`47902`)  Period
^^^
- Bug in subtraction of `Period` from `.PeriodArray` returning wrong results (`45999`)
- Bug in `Period.strftime` and `PeriodIndex.strftime`, directives ``%l`` and ``%u`` were giving wrong results (`46252`)
- Bug in inferring an incorrect ``freq`` when passing a string to `Period` microseconds that are a multiple of 1000 (`46811`)
- Bug in constructing a `Period` from a `Timestamp` or ``np.datetime64`` object with non-zero nanoseconds and ``freq="ns"`` incorrectly truncating the nanoseconds (`46811`)
- Bug in adding ``np.timedelta64("NaT", "ns")`` to a `Period` with a timedelta-like freq incorrectly raising ``IncompatibleFrequency`` instead of returning ``NaT`` (`47196`)
- Bug in adding an array of integers to an array with `PeriodDtype` giving incorrect results when ``dtype.freq.n > 1`` (`47209`)
- Bug in subtracting a `Period` from an array with `PeriodDtype` returning incorrect results instead of raising ``OverflowError`` when the operation overflows (`47538`)  Plotting
^^^
- Bug in `DataFrame.plot.barh` that prevented labeling the x-axis and ``xlabel`` updating the y-axis label (`45144`)
- Bug in `DataFrame.plot.box` that prevented labeling the x-axis (`45463`)
- Bug in `DataFrame.boxplot` that prevented passing in ``xlabel`` and ``ylabel`` (`45463`)
- Bug in `DataFrame.boxplot` that prevented specifying ``vert=False`` (`36918`)

- Bug in `DataFrame.plot.scatter` that prevented specifying ``norm`` (`45809`)
- Fix showing "None" as ylabel in `Series.plot` when not setting ylabel (`46129`)
- Bug in `DataFrame.plot` that led to xticks and vertical grids being improperly placed when plotting a quarterly series (`47602`)
- Bug in `DataFrame.plot` that prevented setting y-axis label, limits and ticks for a secondary y-axis (`47753`)  Groupby/resample/rolling
^^^
- Bug in `DataFrame.resample` ignoring ``closed="right"`` on `TimedeltaIndex` (`45414`)
- Bug in `.DataFrameGroupBy.transform` fails when ``func="size"`` and the input DataFrame has multiple columns (`27469`)
- Bug in `.DataFrameGroupBy.size` and `.DataFrameGroupBy.transform` with ``func="size"`` produced incorrect results when ``axis=1`` (`45715`)
- Bug in `.ExponentialMovingWindow.mean` with ``axis=1`` and ``engine='numba'`` when the `DataFrame` has more columns than rows (`46086`)
- Bug when using ``engine="numba"`` would return the same jitted function when modifying ``engine_kwargs`` (`46086`)
- Bug in `.DataFrameGroupBy.transform` fails when ``axis=1`` and ``func`` is ``"first"`` or ``"last"`` (`45986`)
- Bug in `.DataFrameGroupBy.cumsum` with ``skipna=False`` giving incorrect results (`46216`)
- Bug in `.DataFrameGroupBy.sum`, `.SeriesGroupBy.sum`, `.DataFrameGroupBy.prod`, `.SeriesGroupBy.prod, `.DataFrameGroupBy.cumsum`, and `.SeriesGroupBy.cumsum` with integer dtypes losing precision (`37493`)
- Bug in `.DataFrameGroupBy.cumsum` and `.SeriesGroupBy.cumsum` with ``timedelta64[ns]`` dtype failing to recognize ``NaT`` as a null value (`46216`)
- Bug in `.DataFrameGroupBy.cumsum` and `.SeriesGroupBy.cumsum` with integer dtypes causing overflows when sum was bigger than maximum of dtype (`37493`)
- Bug in `.DataFrameGroupBy.cummin`, `.SeriesGroupBy.cummin`, `.DataFrameGroupBy.cummax` and `.SeriesGroupBy.cummax` with nullable dtypes incorrectly altering the original data in place (`46220`)
- Bug in `DataFrame.groupby` raising error when ``None`` is in first level of `MultiIndex` (`47348`)
- Bug in `.DataFrameGroupBy.cummax` and `.SeriesGroupBy.cummax` with ``int64`` dtype with leading value being the smallest possible int64 (`46382`)
- Bug in `.DataFrameGroupBy.cumprod` and `.SeriesGroupBy.cumprod` ``NaN`` influences calculation in different columns with ``skipna=False`` (`48064`)
- Bug in `.DataFrameGroupBy.max` and `.SeriesGroupBy.max` with empty groups and ``uint64`` dtype incorrectly raising ``RuntimeError`` (`46408`)
- Bug in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` would fail when ``func`` was a string and args or kwargs were supplied (`46479`)
- Bug in `SeriesGroupBy.apply` would incorrectly name its result when there was a unique group (`46369`)
- Bug in `.Rolling.sum` and `.Rolling.mean` would give incorrect result with window of same values (`42064`, `46431`)
- Bug in `.Rolling.var` and `.Rolling.std` would give non-zero result with window of same values (`42064`)
- Bug in `.Rolling.skew` and `.Rolling.kurt` would give NaN with window of same values (`30993`)
- Bug in `.Rolling.var` would segfault calculating weighted variance when window size was larger than data size (`46760`)
- Bug in `Grouper.__repr__` where ``dropna`` was not included. Now it is (`46754`)
- Bug in `DataFrame.rolling` gives ValueError when center=True, axis=1 and win_type is specified (`46135`)
- Bug in `.DataFrameGroupBy.describe` and `.SeriesGroupBy.describe` produces inconsistent results for empty datasets (`41575`)
- Bug in `DataFrame.resample` reduction methods when used with ``on`` would attempt to aggregate the provided column (`47079`)
- Bug in `DataFrame.groupby` and `Series.groupby` would not respect ``dropna=False`` when the input DataFrame/Series had a NaN values in a `MultiIndex` (`46783`)

- Bug in `DataFrameGroupBy.resample` raises ``KeyError`` when getting the result from a key list which misses the resample key (`47362`)
- Bug in `DataFrame.groupby` would lose index columns when the DataFrame is empty for transforms, like fillna (`47787`)
- Bug in `DataFrame.groupby` and `Series.groupby` with ``dropna=False`` and ``sort=False`` would put any null groups at the end instead the order that they are encountered (`46584`)  Reshaping
^^^
- Bug in `concat` between a `Series` with integer dtype and another with `CategoricalDtype` with integer categories and containing ``NaN`` values casting to object dtype instead of ``float64`` (`45359`)
- Bug in `get_dummies` that selected object and categorical dtypes but not string (`44965`)
- Bug in `DataFrame.align` when aligning a `MultiIndex` to a `Series` with another `MultiIndex` (`46001`)
- Bug in concatenation with ``IntegerDtype``, or ``FloatingDtype`` arrays where the resulting dtype did not mirror the behavior of the non-nullable dtypes (`46379`)
- Bug in `concat` losing dtype of columns when ``join="outer"`` and ``sort=True`` (`47329`)
- Bug in `concat` not sorting the column names when ``None`` is included (`47331`)
- Bug in `concat` with identical key leads to error when indexing `MultiIndex` (`46519`)
- Bug in `pivot_table` raising ``TypeError`` when ``dropna=True`` and aggregation column has extension array dtype (`47477`)
- Bug in `merge` raising error for ``how="cross"`` when using ``FIPS`` mode in ssl library (`48024`)
- Bug in `DataFrame.join` with a list when using suffixes to join DataFrames with duplicate column names (`46396`)
- Bug in `DataFrame.pivot_table` with ``sort=False`` results in sorted index (`17041`)
- Bug in `concat` when ``axis=1`` and ``sort=False`` where the resulting Index was a `Int64Index` instead of a `RangeIndex` (`46675`)
- Bug in `wide_to_long` raises when ``stubnames`` is missing in columns and ``i`` contains string dtype column (`46044`)
- Bug in `DataFrame.join` with categorical index results in unexpected reordering (`47812`)  Sparse
^^^
- Bug in `Series.where` and `DataFrame.where` with ``SparseDtype`` failing to retain the array's ``fill_value`` (`45691`)
- Bug in `SparseArray.unique` fails to keep original elements order (`47809`)  ExtensionArray
^^^
- Bug in `IntegerArray.searchsorted` and `FloatingArray.searchsorted` returning inconsistent results when acting on ``np.nan`` (`45255`)  Styler
^^^
- Bug when attempting to apply styling functions to an empty DataFrame subset (`45313`)
- Bug in `CSSToExcelConverter` leading to ``TypeError`` when border color provided without border style for ``xlsxwriter`` engine (`42276`)
- Bug in `Styler.set_sticky` leading to white text on white background in dark mode (`46984`)
- Bug in `Styler.to_latex` causing ``UnboundLocalError`` when ``clines="all;data"`` and the ``DataFrame`` has no rows. (`47203`)
- Bug in `Styler.to_excel` when using ``vertical-align: middle;`` with ``xlsxwriter`` engine (`30107`)
- Bug when applying styles to a DataFrame with boolean column labels (`47838`)  Metadata
^^^
- Fixed metadata propagation in `DataFrame.melt` (`28283`)
- Fixed metadata propagation in `DataFrame.explode` (`28283`)  Other
^^^  .. ***DO NOT USE THIS SECTION***  - Bug in `.assert_index_equal` with ``names=True`` and ``check_order=False`` not checking names (`47328`)  .. ---
.. _whatsnew_150.contributors:  Contributors
~~~  .. contributors:: v1.4.4..v1.5.0

These are the changes in pandas 1.5.1. See `release` for a full changelog
including other versions of pandas.  {{ header }}  .. ---  .. _whatsnew_151.groupby_categorical_regr:  Behavior of ``groupby`` with categorical groupers (`48645`)
~~~  In versions of pandas prior to 1.5, ``groupby`` with ``dropna=False`` would still drop
NA values when the grouper was a categorical dtype. A fix for this was attempted in
1.5, however it introduced a regression where passing ``observed=False`` and
``dropna=False`` to ``groupby`` would result in only observed categories. It was found
that the patch fixing the ``dropna=False`` bug is incompatible with ``observed=False``,
and decided that the best resolution is to restore the correct ``observed=False``
behavior at the cost of reintroducing the ``dropna=False`` bug.  .. ipython:: python  df = pd.DataFrame(  {  "x": pd.Categorical([1, None], categories=[1, 2, 3]),  "y": [3, 4],  }  )  df  *1.5.0 behavior*:  .. code-block:: ipython  In [3]: # Correct behavior, NA values are not dropped  df.groupby("x", observed=True, dropna=False).sum()  Out[3]:  y  x  1  3  NaN  4  In [4]: # Incorrect behavior, only observed categories present  df.groupby("x", observed=False, dropna=False).sum()  Out[4]:  y  x  1  3  NaN  4  *1.5.1 behavior*:  .. ipython:: python  # Incorrect behavior, NA values are dropped  df.groupby("x", observed=True, dropna=False).sum()  # Correct behavior, unobserved categories present (NA values still dropped)  df.groupby("x", observed=False, dropna=False).sum()  .. _whatsnew_151.regressions:  Fixed regressions
~~~
- Fixed Regression in `Series.__setitem__` casting ``None`` to ``NaN`` for object dtype (`48665`)
- Fixed Regression in `DataFrame.loc` when setting values as a `DataFrame` with all ``True`` indexer (`48701`)
- Regression in `.read_csv` causing an ``EmptyDataError`` when using an UTF-8 file handle that was already read from (`48646`)
- Regression in `to_datetime` when ``utc=True`` and ``arg`` contained timezone naive and aware arguments raised a ``ValueError`` (`48678`)
- Fixed regression in `DataFrame.loc` raising ``FutureWarning`` when setting an empty `DataFrame` (`48480`)
- Fixed regression in `DataFrame.describe` raising ``TypeError`` when result contains ``NA`` (`48778`)
- Fixed regression in `DataFrame.plot` ignoring invalid ``colormap`` for ``kind="scatter"`` (`48726`)
- Fixed regression in `MultiIndex.values` resetting ``freq`` attribute of underlying `Index` object (`49054`)
- Fixed performance regression in `factorize` when ``na_sentinel`` is not ``None`` and ``sort=False`` (`48620`)
- Fixed regression causing an ``AttributeError`` during warning emitted if the provided table name in `DataFrame.to_sql` and the table name actually used in the database do not match (`48733`)
- Fixed regression in `to_datetime` when ``arg`` was a date string with nanosecond and ``format`` contained ``%f`` would raise a ``ValueError`` (`48767`)
- Fixed regression in `testing.assert_frame_equal` raising for `MultiIndex` with `Categorical` and ``check_like=True`` (`48975`)
- Fixed regression in `DataFrame.fillna` replacing wrong values for ``datetime64[ns]`` dtype and ``inplace=True`` (`48863`)
- Fixed `.DataFrameGroupBy.size` not returning a Series when ``axis=1`` (`48738`)
- Fixed Regression in `.DataFrameGroupBy.apply` when user defined function is called on an empty dataframe (`47985`)
- Fixed regression in `DataFrame.apply` when passing non-zero ``axis`` via keyword argument (`48656`)
- Fixed regression in `Series.groupby` and `DataFrame.groupby` when the grouper is a nullable data type (e.g. `Int64`) or a PyArrow-backed string array, contains null values, and ``dropna=False`` (`48794`)
- Fixed performance regression in `Series.isin` with mismatching dtypes (`49162`)
- Fixed regression in `DataFrame.to_parquet` raising when file name was specified as ``bytes`` (`48944`)

- Fixed regression in `ExcelWriter` where the ``book`` attribute could no longer be set; however setting this attribute is now deprecated and this ability will be removed in a future version of pandas (`48780`)
- Fixed regression in `DataFrame.corrwith` when computing correlation on tied data with ``method="spearman"`` (`48826`)  .. ---  .. _whatsnew_151.bug_fixes:  Bug fixes
~~~
- Bug in `Series.__getitem__` not falling back to positional for integer keys and boolean `Index` (`48653`)
- Bug in `DataFrame.to_hdf` raising ``AssertionError`` with boolean index (`48667`)
- Bug in `testing.assert_index_equal` for extension arrays with non matching ``NA`` raising ``ValueError`` (`48608`)
- Bug in `DataFrame.pivot_table` raising unexpected ``FutureWarning`` when setting datetime column as index (`48683`)
- Bug in `DataFrame.sort_values` emitting unnecessary ``FutureWarning`` when called on `DataFrame` with boolean sparse columns (`48784`)
- Bug in `.arrays.ArrowExtensionArray` with a comparison operator to an invalid object would not raise a ``NotImplementedError`` (`48833`)  .. ---  .. _whatsnew_151.other:  Other
~~~
- Avoid showing deprecated signatures when introspecting functions with warnings about arguments becoming keyword-only (`48692`)  .. ---  .. _whatsnew_151.contributors:  Contributors
~~~  .. contributors:: v1.5.0..v1.5.1

What's new in 1.5.2 (November 21, 2022)

These are the changes in pandas 1.5.2. See `release` for a full changelog
.. _whatsnew_152.regressions:  Fixed regressions
~~~
- Fixed regression in `MultiIndex.join` for extension array dtypes (`49277`)
- Fixed regression in `Series.replace` raising ``RecursionError`` with numeric dtype and when specifying ``value=None`` (`45725`)
- Fixed regression in arithmetic operations for `DataFrame` with `MultiIndex` columns with different dtypes (`49769`)
- Fixed regression in `DataFrame.plot` preventing `~matplotlib.colors.Colormap` instance  from being passed using the ``colormap`` argument if Matplotlib 3.6+ is used (`49374`)
- Fixed regression in `date_range` returning an invalid set of periods for ``CustomBusinessDay`` frequency and ``start`` date with timezone (`49441`)
- Fixed performance regression in groupby operations (`49676`)
- Fixed regression in `Timedelta` constructor returning object of wrong type when subclassing ``Timedelta`` (`49579`)  .. ---
.. _whatsnew_152.bug_fixes:  Bug fixes
~~~
- Bug in the Copy-on-Write implementation losing track of views in certain chained indexing cases (`48996`)
- Fixed memory leak in `.Styler.to_excel` (`49751`)  .. ---
.. _whatsnew_152.other:  Other
~~~
- Reverted ``color`` as an alias for ``c`` and ``size`` as an alias for ``s`` in function `DataFrame.plot.scatter` (`49732`)  .. ---
.. _whatsnew_152.contributors:  Contributors
~~~  .. contributors:: v1.5.1..v1.5.2

What's new in 1.5.3 (January 18, 2023)

These are the changes in pandas 1.5.3. See `release` for a full changelog
.. _whatsnew_153.regressions:  Fixed regressions
~~~
- Fixed performance regression in `Series.isin` when ``values`` is empty (`49839`)
- Fixed regression in `DataFrame.memory_usage` showing unnecessary ``FutureWarning`` when `DataFrame` is empty (`50066`)
- Fixed regression in `.DataFrameGroupBy.transform` when used with ``as_index=False`` (`49834`)
- Enforced reversion of ``color`` as an alias for ``c`` and ``size`` as an alias for ``s`` in function `DataFrame.plot.scatter` (`49732`)
- Fixed regression in `.SeriesGroupBy.apply` setting a ``name`` attribute on the result if the result was a `DataFrame` (`49907`)
- Fixed performance regression in setting with the `~DataFrame.at` indexer (`49771`)
- Fixed regression in `to_datetime` raising ``ValueError`` when parsing array of ``float`` containing ``np.nan`` (`50237`)  .. ---
.. _whatsnew_153.bug_fixes:  Bug fixes
~~~
- Bug in the Copy-on-Write implementation losing track of views when indexing a `DataFrame` with another `DataFrame` (`50630`)
- Bug in `.Styler.to_excel` leading to error when unrecognized ``border-style`` (e.g. ``"hair"``) provided to Excel writers (`48649`)
- Bug in `Series.quantile` emitting warning from NumPy when `Series` has only ``NA`` values (`50681`)
- Bug when chaining several `.Styler.concat` calls, only the last styler was concatenated (`49207`)
- Fixed bug when instantiating a `DataFrame` subclass inheriting from ``typing.Generic`` that triggered a ``UserWarning`` on python 3.11 (`49649`)
- Bug in `pivot_table` with NumPy 1.24 or greater when the `DataFrame` columns has nested elements (`50342`)
- Bug in `pandas.testing.assert_series_equal` (and equivalent ``assert_`` functions) when having nested data and using numpy >= 1.25 (`50360`)  .. ---
.. _whatsnew_153.other:  Other
~~~  .. note::  If you are using `DataFrame.to_sql`, `read_sql`, `read_sql_table`, or `read_sql_query` with SQLAlchemy 1.4.46 or greater,  you may see a ``sqlalchemy.exc.RemovedIn20Warning``. These warnings can be safely ignored for the SQLAlchemy 1.4.x releases  as pandas works toward compatibility with SQLAlchemy 2.0.  - Reverted deprecation (`45324`) of behavior of `Series.__getitem__` and `Series.__setitem__` slicing with an integer `Index`; this will remain positional (`49612`)
- A ``FutureWarning`` raised when attempting to set values inplace with `DataFrame.loc` or `DataFrame.iloc` has been changed to a ``DeprecationWarning`` (`48673`)  .. ---
.. _whatsnew_153.contributors:  Contributors
~~~  .. contributors:: v1.5.2..v1.5.3

What's new in 1.5.4 (March XX, 2023)

These are the changes in pandas 1.5.4. See `release` for a full changelog
.. _whatsnew_154.regressions:  Fixed regressions

~~~~~~~~~~~~~~~~~

.. ---
.. _whatsnew_154.bug_fixes:  Bug fixes

~~~~~

.. ---
.. _whatsnew_154.contributors:  Contributors
~~~  .. contributors:: v1.5.3..v1.5.4|HEAD

These are the changes in pandas 2.0.0. See `release` for a full changelog
.. _whatsnew_200.enhancements:  Enhancements
~~~  .. _whatsnew_200.enhancements.optional_dependency_management_pip:  Installing optional dependencies with pip extras
^^^
When installing pandas using pip, sets of optional dependencies can also be installed by specifying extras.  .. code-block:: bash  pip install "pandas[performance, aws]>=2.0.0"  The available extras, found in the `installation guide<install.dependencies>`, are
``[all, performance, computation, timezone, fss, aws, gcp, excel, parquet, feather, hdf5, spss, postgresql, mysql,
sql-other, html, xml, plot, output_formatting, clipboard, compression, test]`` (`39164`).  .. _whatsnew_200.enhancements.index_can_hold_numpy_numeric_dtypes:  `Index` can now hold numpy numeric dtypes
^^^  It is now possible to use any numpy numeric dtype in a `Index` (`42717`).  Previously it was only possible to use ``int64``, ``uint64`` & ``float64`` dtypes:  .. code-block:: ipython  In [1]: pd.Index([1, 2, 3], dtype=np.int8)  Out[1]: Int64Index([1, 2, 3], dtype="int64")  In [2]: pd.Index([1, 2, 3], dtype=np.uint16)  Out[2]: UInt64Index([1, 2, 3], dtype="uint64")  In [3]: pd.Index([1, 2, 3], dtype=np.float32)  Out[3]: Float64Index([1.0, 2.0, 3.0], dtype="float64")  `Int64Index`, `UInt64Index` & `Float64Index` were deprecated in pandas
version 1.4 and have now been removed. Instead `Index` should be used directly, and
can it now take all numpy numeric dtypes, i.e.
``int8``/ ``int16``/``int32``/``int64``/``uint8``/``uint16``/``uint32``/``uint64``/``float32``/``float64`` dtypes:  .. ipython:: python  pd.Index([1, 2, 3], dtype=np.int8)  pd.Index([1, 2, 3], dtype=np.uint16)  pd.Index([1, 2, 3], dtype=np.float32)  The ability for `Index` to hold the numpy numeric dtypes has meant some changes in Pandas
functionality. In particular, operations that previously were forced to create 64-bit indexes,
can now create indexes with lower bit sizes, e.g. 32-bit indexes.  Below is a possibly non-exhaustive list of changes:  1. Instantiating using a numpy numeric array now follows the dtype of the numpy array.  Previously, all indexes created from numpy numeric arrays were forced to 64-bit. Now,  for example, ``Index(np.array([1, 2, 3]))`` will be ``int32`` on 32-bit systems, where  it previously would have been ``int64``` even on 32-bit systems.  Instantiating `Index` using a list of numbers will still return 64bit dtypes,  e.g. ``Index([1, 2, 3])`` will have a ``int64`` dtype, which is the same as previously.
2. The various numeric datetime attributes of `DatetimeIndex` (`~DatetimeIndex.day`,  `~DatetimeIndex.month`, `~DatetimeIndex.year` etc.) were previously in of  dtype ``int64``, while they were ``int32`` for `arrays.DatetimeArray`. They are now  ``int32`` on ``DatetimeIndex`` also:  .. ipython:: python  idx = pd.date_range(start='1/1/2018', periods=3, freq='M')  idx.array.year  idx.year  3. Level dtypes on Indexes from `Series.sparse.from_coo` are now of dtype ``int32``,  the same as they are on the ``rows``/``cols`` on a scipy sparse matrix. Previously they  were of dtype ``int64``.  .. ipython:: python  from scipy import sparse  A = sparse.coo_matrix(  ([3.0, 1.0, 2.0], ([1, 0, 0], [0, 2, 3])), shape=(3, 4)  )  ser = pd.Series.sparse.from_coo(A)  ser.index.dtypes  4. `Index` cannot be instantiated using a float16 dtype. Previously instantiating  an `Index` using dtype ``float16`` resulted in a `Float64Index` with a  ``float64`` dtype. It row raises a ``NotImplementedError``:  .. ipython:: python  pd.Index([1, 2, 3], dtype=np.float16)  .. _whatsnew_200.enhancements.io_use_nullable_dtypes_and_dtype_backend:  Configuration option, ``mode.dtype_backend``, to return pyarrow-backed dtypes
^^^  The ``use_nullable_dtypes`` keyword argument has been expanded to the following functions to enable automatic conversion to nullable dtypes (`36712`)  * `read_csv`
* `read_clipboard`
* `read_fwf`

* `read_excel`
* `read_html`
* `read_xml`
* `read_json`
* `read_sql`
* `read_sql_query`
* `read_sql_table`
* `read_orc`
* `read_feather`
* `read_spss`
* `to_numeric`  To simplify opting-in to nullable dtypes for these functions, a new option ``nullable_dtypes`` was added that allows setting
the keyword argument globally to ``True`` if not specified directly. The option can be enabled
through:  .. ipython:: python  pd.options.mode.nullable_dtypes = True  The option will only work for functions with the keyword ``use_nullable_dtypes``.  Additionally a new global configuration, ``mode.dtype_backend`` can now be used in conjunction with the parameter ``use_nullable_dtypes=True`` in the following functions
to select the nullable dtypes implementation.  * `read_csv`
* `read_parquet`
* `to_numeric`  And the following methods will also utilize the ``mode.dtype_backend`` option.  * `DataFrame.convert_dtypes`
* `Series.convert_dtypes`  By default, ``mode.dtype_backend`` is set to ``"pandas"`` to return existing, numpy-backed nullable dtypes, but it can also
be set to ``"pyarrow"`` to return pyarrow-backed, nullable `ArrowDtype` (`48957`, `49997`).  .. ipython:: python  import io  data = io.StringIO("""a,b,c,d,e,f,g,h,i  1,2.5,True,a,,,  3,4.5,False,b,6,7.5,True,a,  """)  with pd.option_context("mode.dtype_backend", "pandas"):  df = pd.read_csv(data, use_nullable_dtypes=True)  df.dtypes  data.seek(0)  with pd.option_context("mode.dtype_backend", "pyarrow"):  df_pyarrow = pd.read_csv(data, use_nullable_dtypes=True, engine="pyarrow")  df_pyarrow.dtypes  Copy-on-Write improvements

^^^  - A new lazy copy mechanism that defers the copy until the object in question is modified  was added to the following methods:  - `DataFrame.reset_index` / `Series.reset_index`  - `DataFrame.set_index`  - `DataFrame.set_axis` / `Series.set_axis`  - `DataFrame.set_flags` / `Series.set_flags`  - `DataFrame.rename_axis` / `Series.rename_axis`  - `DataFrame.reindex` / `Series.reindex`  - `DataFrame.reindex_like` / `Series.reindex_like`  - `DataFrame.assign`  - `DataFrame.drop`  - `DataFrame.dropna` / `Series.dropna`  - `DataFrame.select_dtypes`  - `DataFrame.align` / `Series.align`  - `Series.to_frame`  - `DataFrame.rename` / `Series.rename`  - `DataFrame.add_prefix` / `Series.add_prefix`  - `DataFrame.add_suffix` / `Series.add_suffix`  - `DataFrame.drop_duplicates` / `Series.drop_duplicates`  - `DataFrame.droplevel` / `Series.droplevel`  - `DataFrame.reorder_levels` / `Series.reorder_levels`  - `DataFrame.between_time` / `Series.between_time`  - `DataFrame.filter` / `Series.filter`  - `DataFrame.head` / `Series.head`  - `DataFrame.tail` / `Series.tail`  - `DataFrame.isetitem`  - `DataFrame.pipe` / `Series.pipe`  - `DataFrame.pop` / `Series.pop`  - `DataFrame.replace` / `Series.replace`  - `DataFrame.shift` / `Series.shift`  - `DataFrame.sort_index` / `Series.sort_index`  - `DataFrame.sort_values` / `Series.sort_values`  - `DataFrame.squeeze` / `Series.squeeze`  - `DataFrame.swapaxes`  - `DataFrame.swaplevel` / `Series.swaplevel`  - `DataFrame.take` / `Series.take`  - `DataFrame.to_timestamp` / `Series.to_timestamp`  - `DataFrame.to_period` / `Series.to_period`  - `DataFrame.truncate`  - `DataFrame.tz_convert` / `Series.tz_localize`  - `DataFrame.fillna` / `Series.fillna`  - `DataFrame.interpolate` / `Series.interpolate`  - `DataFrame.ffill` / `Series.ffill`  - `DataFrame.bfill` / `Series.bfill`  - `DataFrame.infer_objects` / `Series.infer_objects`  - `DataFrame.astype` / `Series.astype`  - `DataFrame.convert_dtypes` / `Series.convert_dtypes`  - `concat`  These methods return views when Copy-on-Write is enabled, which provides a significant  performance improvement compared to the regular execution (`49473`).  - Accessing a single column of a DataFrame as a Series (e.g. ``df["col"]``) now always  returns a new object every time it is constructed when Copy-on-Write is enabled (not  returning multiple times an identical, cached Series object). This ensures that those  Series objects correctly follow the Copy-on-Write rules (`49450`)  - The `Series` constructor will now create a lazy copy (deferring the copy until  a modification to the data happens) when constructing a Series from an existing  Series with the default of ``copy=False`` (`50471`)  - The `DataFrame` constructor, when constructing a DataFrame from a dictionary  of Series objects and specifying ``copy=False``, will now use a lazy copy  of those Series objects for the columns of the DataFrame (`50777`)  - Trying to set values using chained assignment (for example, ``df["a"][1:3] = 0``)  will now always raise an exception when Copy-on-Write is enabled. In this mode,  chained assignment can never work because we are always setting into a temporary  object that is the result of an indexing operation (getitem), which under  Copy-on-Write always behaves as a copy. Thus, assigning through a chain  can never update the original Series or DataFrame. Therefore, an informative  error is raised to the user instead of silently doing nothing (`49467`)  - `DataFrame.replace` will now respect the Copy-on-Write mechanism  when ``inplace=True``.  Copy-on-Write can be enabled through one of  .. code-block:: python  pd.set_option("mode.copy_on_write", True)  .. code-block:: python  pd.options.mode.copy_on_write = True  Alternatively, copy on write can be enabled locally through:  .. code-block:: python  with pd.option_context("mode.copy_on_write", True):  ...  .. _whatsnew_200.enhancements.other:  Other enhancements
^^^

^^^
- Added support for ``dt`` accessor methods when using `ArrowDtype` with a ``pyarrow.timestamp`` type (`50954`)
- `read_sas` now supports using ``encoding='infer'`` to correctly read and use the encoding specified by the sas file. (`48048`)
- `.DataFrameGroupBy.quantile`, `.SeriesGroupBy.quantile` and `.DataFrameGroupBy.std` now preserve nullable dtypes instead of casting to numpy dtypes (`37493`)
- `.DataFrameGroupBy.std`, `.SeriesGroupBy.std` now support datetime64, timedelta64, and `DatetimeTZDtype` dtypes (`48481`)
- `Series.add_suffix`, `DataFrame.add_suffix`, `Series.add_prefix` and `DataFrame.add_prefix` support an ``axis`` argument. If ``axis`` is set, the default behaviour of which axis to consider can be overwritten (`47819`)
- `.testing.assert_frame_equal` now shows the first element where the DataFrames differ, analogously to ``pytest``'s output (`47910`)
- Added ``index`` parameter to `DataFrame.to_dict` (`46398`)
- Added support for extension array dtypes in `merge` (`44240`)
- Added metadata propagation for binary operators on `DataFrame` (`28283`)
- Added ``cumsum``, ``cumprod``, ``cummin`` and ``cummax`` to the ``ExtensionArray`` interface via ``_accumulate`` (`28385`)
- `.CategoricalConversionWarning`, `.InvalidComparison`, `.InvalidVersion`, `.LossySetitemError`, and `.NoBufferPresent` are now exposed in ``pandas.errors`` (`27656`)
- Fix ``test`` optional_extra by adding missing test package ``pytest-asyncio`` (`48361`)
- `DataFrame.astype` exception message thrown improved to include column name when type conversion is not possible. (`47571`)
- `date_range` now supports a ``unit`` keyword ("s", "ms", "us", or "ns") to specify the desired resolution of the output index (`49106`)
- `timedelta_range` now supports a ``unit`` keyword ("s", "ms", "us", or "ns") to specify the desired resolution of the output index (`49824`)
- `DataFrame.to_json` now supports a ``mode`` keyword with supported inputs 'w' and 'a'. Defaulting to 'w', 'a' can be used when lines=True and orient='records' to append record oriented json lines to an existing json file. (`35849`)
- Added ``name`` parameter to `IntervalIndex.from_breaks`, `IntervalIndex.from_arrays` and `IntervalIndex.from_tuples` (`48911`)
- Improve exception message when using `.testing.assert_frame_equal` on a `DataFrame` to include the column that is compared (`50323`)
- Improved error message for `merge_asof` when join-columns were duplicated (`50102`)
- Added support for extension array dtypes to `get_dummies` (`32430`)
- Added `Index.infer_objects` analogous to `Series.infer_objects` (`50034`)
- Added ``copy`` parameter to `Series.infer_objects` and `DataFrame.infer_objects`, passing ``False`` will avoid making copies for series or columns that are already non-object or where no better dtype can be inferred (`50096`)
- `DataFrame.plot.hist` now recognizes ``xlabel`` and ``ylabel`` arguments (`49793`)
- `Series.drop_duplicates` has gained ``ignore_index`` keyword to reset index (`48304`)
- `Series.dropna` and `DataFrame.dropna` has gained ``ignore_index`` keyword to reset index (`31725`)
- Improved error message in `to_datetime` for non-ISO8601 formats, informing users about the position of the first error (`50361`)
- Improved error message when trying to align `DataFrame` objects (for example, in `DataFrame.compare`) to clarify that "identically labelled" refers to both index and columns (`50083`)
- Added `DatetimeIndex.as_unit` and `TimedeltaIndex.as_unit` to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (`50616`)
- Added `Series.dt.unit` and `Series.dt.as_unit` to convert to different resolutions; supported resolutions are "s", "ms", "us", and "ns" (`51223`)
- Added new argument ``dtype`` to `read_sql` to be consistent with `read_sql_query` (`50797`)
- `to_datetime` now accepts ``"ISO8601"`` as an argument to ``format``, which will match any ISO8601 string (but possibly not identically-formatted) (`50411`)

- `to_datetime` now accepts ``"mixed"`` as an argument to ``format``, which will infer the format for each element individually (`50972`)
- Added new argument ``engine`` to `read_json` to support parsing JSON with pyarrow by specifying ``engine="pyarrow"`` (`48893`)
- Added support for SQLAlchemy 2.0 (`40686`)

- `Index` set operations `Index.union`, `Index.intersection`, `Index.difference`, and `Index.symmetric_difference` now support ``sort=True``, which will always return a sorted result, unlike the default ``sort=None`` which does not sort in some cases (`25151`)

.. ---
.. _whatsnew_200.notable_bug_fixes:  Notable bug fixes
~~~  These are bug fixes that might have notable behavior changes.  .. _whatsnew_200.notable_bug_fixes.cumsum_cumprod_overflow:  `.DataFrameGroupBy.cumsum` and `.DataFrameGroupBy.cumprod` overflow instead of lossy casting to float
^^^  In previous versions we cast to float when applying ``cumsum`` and ``cumprod`` which
lead to incorrect results even if the result could be hold by ``int64`` dtype.
Additionally, the aggregation overflows consistent with numpy and the regular
`DataFrame.cumprod` and `DataFrame.cumsum` methods when the limit of
``int64`` is reached (`37493`).  *Old Behavior*  .. code-block:: ipython  In [1]: df = pd.DataFrame({"key": ["b"] * 7, "value": 625})  In [2]: df.groupby("key")["value"].cumprod()[5]  Out[2]: 5.960464477539062e+16  We return incorrect results with the 6th value.  *New Behavior*  .. ipython:: python  df = pd.DataFrame({"key": ["b"] * 7, "value": 625})  df.groupby("key")["value"].cumprod()  We overflow with the 7th value, but the 6th value is still correct.  .. _whatsnew_200.notable_bug_fixes.groupby_nth_filter:  `.DataFrameGroupBy.nth` and `.SeriesGroupBy.nth` now behave as filtrations
^^^  In previous versions of pandas, `.DataFrameGroupBy.nth` and
`.SeriesGroupBy.nth` acted as if they were aggregations. However, for most
inputs ``n``, they may return either zero or multiple rows per group. This means
that they are filtrations, similar to e.g. `.DataFrameGroupBy.head`. pandas
now treats them as filtrations (`13666`).  .. ipython:: python  df = pd.DataFrame({"a": [1, 1, 2, 1, 2], "b": [np.nan, 2.0, 3.0, 4.0, 5.0]})  gb = df.groupby("a")  *Old Behavior*  .. code-block:: ipython  In [5]: gb.nth(n=1)  Out[5]:  A  B  1  1  2.0  4  2  5.0  *New Behavior*  .. ipython:: python  gb.nth(n=1)  In particular, the index of the result is derived from the input by selecting
the appropriate rows. Also, when ``n`` is larger than the group, no rows instead of
``NaN`` is returned.  *Old Behavior*  .. code-block:: ipython  In [5]: gb.nth(n=3, dropna="any")  Out[5]:  B  A  1 NaN  2 NaN  *New Behavior*  .. ipython:: python  gb.nth(n=3, dropna="any")  .. ---
.. _whatsnew_200.api_breaking:  Backwards incompatible API changes
~~~  .. _whatsnew_200.api_breaking.unsupported_datetimelike_dtype_arg:  Construction with datetime64 or timedelta64 dtype with unsupported resolution
^^^
In past versions, when constructing a `Series` or `DataFrame` and
passing a "datetime64" or "timedelta64" dtype with unsupported resolution
(i.e. anything other than "ns"), pandas would silently replace the given dtype
with its nanosecond analogue:  *Previous behavior*:  .. code-block:: ipython  In [5]: pd.Series(["2016-01-01"], dtype="datetime64[s]")  Out[5]:  0  2016-01-01  dtype: datetime64[ns]  In [6] pd.Series(["2016-01-01"], dtype="datetime64[D]")  Out[6]:  0  2016-01-01  dtype: datetime64[ns]  In pandas 2.0 we support resolutions "s", "ms", "us", and "ns". When passing
a supported dtype (e.g. "datetime64[s]"), the result now has exactly
the requested dtype:  *New behavior*:  .. ipython:: python  pd.Series(["2016-01-01"], dtype="datetime64[s]")  With an un-supported dtype, pandas now raises instead of silently swapping in
a supported dtype:  *New behavior*:  .. ipython:: python  pd.Series(["2016-01-01"], dtype="datetime64[D]")  .. _whatsnew_200.api_breaking.value_counts:  Value counts sets the resulting name to ``count``
^^^
In past versions, when running `Series.value_counts`, the result would inherit
the original object's name, and the result index would be nameless. This would cause
confusion when resetting the index, and the column names would not correspond with the
column values.
Now, the result name will be ``'count'`` (or ``'proportion'`` if ``normalize=True`` was passed),

and the index will be named after the original object (`49497`).  *Previous behavior*:  .. code-block:: ipython  In [8]: pd.Series(['quetzal', 'quetzal', 'elk'], name='animal').value_counts()  Out[2]:  quetzal  2  elk  1  Name: animal, dtype: int64  *New behavior*:  .. ipython:: python  pd.Series(['quetzal', 'quetzal', 'elk'], name='animal').value_counts()  Likewise for other ``value_counts`` methods (for example, `DataFrame.value_counts`).  .. _whatsnew_200.api_breaking.astype_to_unsupported_datetimelike:  Disallow astype conversion to non-supported datetime64/timedelta64 dtypes
^^^
In previous versions, converting a `Series` or `DataFrame`
from ``datetime64[ns]`` to a different ``datetime64[X]`` dtype would return
with ``datetime64[ns]`` dtype instead of the requested dtype. In pandas 2.0,
support is added for "datetime64[s]", "datetime64[ms]", and "datetime64[us]" dtypes,
so converting to those dtypes gives exactly the requested dtype:  *Previous behavior*:  .. ipython:: python  idx = pd.date_range("2016-01-01", periods=3)  ser = pd.Series(idx)  *Previous behavior*:  .. code-block:: ipython  In [4]: ser.astype("datetime64[s]")  Out[4]:  0  2016-01-01  1  2016-01-02  2  2016-01-03  dtype: datetime64[ns]  With the new behavior, we get exactly the requested dtype:  *New behavior*:  .. ipython:: python  ser.astype("datetime64[s]")  For non-supported resolutions e.g. "datetime64[D]", we raise instead of silently
ignoring the requested dtype:  *New behavior*:  .. ipython:: python  ser.astype("datetime64[D]")  For conversion from ``timedelta64[ns]`` dtypes, the old behavior converted
to a floating point format.  *Previous behavior*:  .. ipython:: python  idx = pd.timedelta_range("1 Day", periods=3)  ser = pd.Series(idx)  *Previous behavior*:  .. code-block:: ipython  In [7]: ser.astype("timedelta64[s]")  Out[7]:  0  86400.0  1  172800.0  2  259200.0  dtype: float64  In [8]: ser.astype("timedelta64[D]")  Out[8]:  0  1.0  1  2.0  2  3.0  dtype: float64  The new behavior, as for datetime64, either gives exactly the requested dtype or raises:  *New behavior*:  .. ipython:: python  ser.astype("timedelta64[s]")  ser.astype("timedelta64[D]")  .. _whatsnew_200.api_breaking.default_to_stdlib_tzinfos:  UTC and fixed-offset timezones default to standard-library tzinfo objects
^^^
In previous versions, the default ``tzinfo`` object used to represent UTC
was ``pytz.UTC``. In pandas 2.0, we default to ``datetime.timezone.utc`` instead.
Similarly, for timezones represent fixed UTC offsets, we use ``datetime.timezone``
objects instead of ``pytz.FixedOffset`` objects. See (`34916`)  *Previous behavior*:  .. code-block:: ipython  In [2]: ts = pd.Timestamp("2016-01-01", tz="UTC")  In [3]: type(ts.tzinfo)  Out[3]: pytz.UTC  In [4]: ts2 = pd.Timestamp("2016-01-01 0406-07:00")  In [3]: type(ts2.tzinfo)  Out[5]: pytz._FixedOffset  *New behavior*:  .. ipython:: python  ts = pd.Timestamp("2016-01-01", tz="UTC")  type(ts.tzinfo)  ts2 = pd.Timestamp("2016-01-01 0406-07:00")  type(ts2.tzinfo)  For timezones that are neither UTC nor fixed offsets, e.g. "US/Pacific", we
continue to default to ``pytz`` objects.  .. _whatsnew_200.api_breaking.zero_len_indexes:  Empty DataFrames/Series will now default to have a ``RangeIndex``
^^^  Before, constructing an empty (where ``data`` is ``None`` or an empty list-like argument) `Series` or `DataFrame` without

specifying the axes (``index=None``, ``columns=None``) would return the axes as empty `Index` with object dtype.  Now, the axes return an empty `RangeIndex` (`49572`).  *Previous behavior*:  .. code-block:: ipython  In [8]: pd.Series().index  Out[8]:  Index([], dtype='object')  In [9] pd.DataFrame().axes  Out[9]:  [Index([], dtype='object'), Index([], dtype='object')]  *New behavior*:  .. ipython:: python  pd.Series().index  pd.DataFrame().axes  .. _whatsnew_200.api_breaking.to_latex:  DataFrame to LaTeX has a new render engine
^^^  The existing `DataFrame.to_latex` has been restructured to utilise the
extended implementation previously available under `.Styler.to_latex`.
The arguments signature is similar, albeit ``col_space`` has been removed since
it is ignored by LaTeX engines. This render engine also requires ``jinja2`` as a
dependency which needs to be installed, since rendering is based upon jinja2 templates.  The pandas latex options below are no longer used and have been removed. The generic
max rows and columns arguments remain but for this functionality should be replaced
by the Styler equivalents.
The alternative options giving similar functionality are indicated below:  - ``display.latex.escape``: replaced with ``styler.format.escape``,
- ``display.latex.longtable``: replaced with ``styler.latex.environment``,
- ``display.latex.multicolumn``, ``display.latex.multicolumn_format`` and  ``display.latex.multirow``: replaced with ``styler.sparse.rows``,  ``styler.sparse.columns``, ``styler.latex.multirow_align`` and  ``styler.latex.multicol_align``,
- ``display.latex.repr``: replaced with ``styler.render.repr``,
- ``display.max_rows`` and ``display.max_columns``: replace with  ``styler.render.max_rows``, ``styler.render.max_columns`` and  ``styler.render.max_elements``.  Note that due to this change some defaults have also changed:  - ``multirow`` now defaults to *True*.
- ``multirow_align`` defaults to *"r"* instead of *"l"*.
- ``multicol_align`` defaults to *"r"* instead of *"l"*.
- ``escape`` now defaults to *False*.  Note that the behaviour of ``_repr_latex_`` is also changed. Previously
setting ``display.latex.repr`` would generate LaTeX only when using nbconvert for a
JupyterNotebook, and not when the user is running the notebook. Now the
``styler.render.repr`` option allows control of the specific output
within JupyterNotebooks for operations (not just on nbconvert). See `39911`.  .. _whatsnew_200.api_breaking.deps:  Increased minimum versions for dependencies
^^^
| mypy (dev)  | 1.0  |  |  X  |
| pytest (dev)  | 7.0.0  |  |  X  |
| pytest-xdist (dev)| 2.2.0  |  |  X  |
| hypothesis (dev)  | 6.34.2  |  |  X  |
| python-dateutil  | 2.8.2  |  X  |  X  |
| pyarrow  | 7.0.0  |  X  |
| matplotlib  | 3.6.1  |  X  |
| fastparquet  | 0.6.3  |  X  |
| xarray  | 0.21.0  |  X  |
+---+---+---+  See `install.dependencies` and `install.optional_dependencies` for more.  Datetimes are now parsed with a consistent format

^^^  In the past, `to_datetime` guessed the format for each element independently. This was appropriate for some cases where elements had mixed date formats - however, it would regularly cause problems when users expected a consistent format but the function would switch formats between elements. As of version 2.0.0, parsing will use a consistent format, determined by the first non-NA value (unless the user specifies a format, in which case that is used).  *Old behavior*:  .. code-block:: ipython  In [1]: ser = pd.Series(['13-01-2000', '12-01-2000'])  In [2]: pd.to_datetime(ser)  Out[2]:  0  2000-01-13  1  2000-12-01  dtype: datetime64[ns]  *New behavior*:  .. ipython:: python  ser = pd.Series(['13-01-2000', '12-01-2000'])  pd.to_datetime(ser)  Note that this affects `read_csv` as well.  If you still need to parse dates with inconsistent formats, you can use
``format='mixed`` (possibly alongside ``dayfirst``) ::  ser = pd.Series(['13-01-2000', '12 January 2000'])  pd.to_datetime(ser, format='mixed', dayfirst=True)  or, if your formats are all ISO8601 (but possibly not identically-formatted) ::  ser = pd.Series(['2020-01-01', '2020-01-01 03:00'])  pd.to_datetime(ser, format='ISO8601')  .. _whatsnew_200.api_breaking.other:  Other API changes
^^^
- The ``freq``, ``tz``, ``nanosecond``, and ``unit`` keywords in the `Timestamp` constructor are now keyword-only (`45307`, `32526`)
- Passing ``nanoseconds`` greater than 999 or less than 0 in `Timestamp` now raises a ``ValueError`` (`48538`, `48255`)
- `read_csv`: specifying an incorrect number of columns with ``index_col`` of now raises ``ParserError`` instead of ``IndexError`` when using the c parser.
- Default value of ``dtype`` in `get_dummies` is changed to ``bool`` from ``uint8`` (`45848`)
- `DataFrame.astype`, `Series.astype`, and `DatetimeIndex.astype` casting datetime64 data to any of "datetime64[s]", "datetime64[ms]", "datetime64[us]" will return an object with the given resolution instead of coercing back to "datetime64[ns]" (`48928`)
- `DataFrame.astype`, `Series.astype`, and `DatetimeIndex.astype` casting timedelta64 data to any of "timedelta64[s]", "timedelta64[ms]", "timedelta64[us]" will return an object with the given resolution instead of coercing to "float64" dtype (`48963`)
- `DatetimeIndex.astype`, `TimedeltaIndex.astype`, `PeriodIndex.astype` `Series.astype`, `DataFrame.astype` with ``datetime64``, ``timedelta64`` or `PeriodDtype` dtypes no longer allow converting to integer dtypes other than "int64", do ``obj.astype('int64', copy=False).astype(dtype)`` instead (`49715`)
- `Index.astype` now allows casting from ``float64`` dtype to datetime-like dtypes, matching `Series` behavior (`49660`)
- Passing data with dtype of "timedelta64[s]", "timedelta64[ms]", or "timedelta64[us]" to `TimedeltaIndex`, `Series`, or `DataFrame` constructors will now retain that dtype instead of casting to "timedelta64[ns]"; timedelta64 data with lower resolution will be cast to the lowest supported resolution "timedelta64[s]" (`49014`)
- Passing ``dtype`` of "timedelta64[s]", "timedelta64[ms]", or "timedelta64[us]" to `TimedeltaIndex`, `Series`, or `DataFrame` constructors will now retain that dtype instead of casting to "timedelta64[ns]"; passing a dtype with lower resolution for `Series` or `DataFrame` will be cast to the lowest supported resolution "timedelta64[s]" (`49014`)
- Passing a ``np.datetime64`` object with non-nanosecond resolution to `Timestamp` will retain the input resolution if it is "s", "ms", "us", or "ns"; otherwise it will be cast to the closest supported resolution (`49008`)
- Passing ``datetime64`` values with resolution other than nanosecond to `to_datetime` will retain the input resolution if it is "s", "ms", "us", or "ns"; otherwise it will be cast to the closest supported resolution (`50369`)

- Passing integer values and a non-nanosecond datetime64 dtype (e.g. "datetime64[s]") `DataFrame`, `Series`, or `Index` will treat the values as multiples of the dtype's unit, matching the behavior of e.g. ``Series(np.array(values, dtype="M8[s]"))`` (`51092`)
- Passing a string in ISO-8601 format to `Timestamp` will retain the resolution of the parsed input if it is "s", "ms", "us", or "ns"; otherwise it will be cast to the closest supported resolution (`49737`)
- The ``other`` argument in `DataFrame.mask` and `Series.mask` now defaults to ``no_default`` instead of ``np.nan`` consistent with `DataFrame.where` and `Series.where`. Entries will be filled with the corresponding NULL value (``np.nan`` for numpy dtypes, ``pd.NA`` for extension dtypes). (`49111`)
- Changed behavior of `Series.quantile` and `DataFrame.quantile` with `SparseDtype` to retain sparse dtype (`49583`)
- When creating a `Series` with a object-dtype `Index` of datetime objects, pandas no longer silently converts the index to a `DatetimeIndex` (`39307`, `23598`)
- `pandas.testing.assert_index_equal` with parameter ``exact="equiv"`` now considers two indexes equal when both are either a `RangeIndex` or `Index` with an ``int64`` dtype. Previously it meant either a `RangeIndex` or a `Int64Index` (`51098`)
- `Series.unique` with dtype "timedelta64[ns]" or "datetime64[ns]" now returns `TimedeltaArray` or `DatetimeArray` instead of ``numpy.ndarray`` (`49176`)
- `to_datetime` and `DatetimeIndex` now allow sequences containing both ``datetime`` objects and numeric entries, matching `Series` behavior (`49037`, `50453`)
- `pandas.api.types.is_string_dtype` now only returns ``True`` for array-likes with ``dtype=object`` when the elements are inferred to be strings (`15585`)
- Passing a sequence containing ``datetime`` objects and ``date`` objects to `Series` constructor will return with ``object`` dtype instead of ``datetime64[ns]`` dtype, consistent with `Index` behavior (`49341`)
- Passing strings that cannot be parsed as datetimes to `Series` or `DataFrame` with ``dtype="datetime64[ns]"`` will raise instead of silently ignoring the keyword and returning ``object`` dtype (`24435`)
- Passing a sequence containing a type that cannot be converted to `Timedelta` to `to_timedelta` or to the `Series` or `DataFrame` constructor with ``dtype="timedelta64[ns]"`` or to `TimedeltaIndex` now raises ``TypeError`` instead of ``ValueError`` (`49525`)
- Changed behavior of `Index` constructor with sequence containing at least one ``NaT`` and everything else either ``None`` or ``NaN`` to infer ``datetime64[ns]`` dtype instead of ``object``, matching `Series` behavior (`49340`)
- `read_stata` with parameter ``index_col`` set to ``None`` (the default) will now set the index on the returned `DataFrame` to a `RangeIndex` instead of a `Int64Index` (`49745`)
- Changed behavior of `Index`, `Series`, and `DataFrame` arithmetic methods when working with object-dtypes, the results no longer do type inference on the result of the array operations, use ``result.infer_objects(copy=False)`` to do type inference on the result (`4999`, `49714`)
- Changed behavior of `Index` constructor with an object-dtype ``numpy.ndarray`` containing all-``bool`` values or all-complex values, this will now retain object dtype, consistent with the `Series` behavior (`49594`)
- Added ``"None"`` to default ``na_values`` in `read_csv` (`50286`)
- Changed behavior of `Series` and `DataFrame` constructors when given an integer dtype and floating-point data that is not round numbers, this now raises ``ValueError`` instead of silently retaining the float dtype; do ``Series(data)`` or ``DataFrame(data)`` to get the old behavior, and ``Series(data).astype(dtype)`` or ``DataFrame(data).astype(dtype)`` to get the specified dtype (`49599`)

- Changed behavior of `DataFrame.shift` with ``axis=1``, an integer ``fill_value``, and homogeneous datetime-like dtype, this now fills new columns with integer dtypes instead of casting to datetimelike (`49842`)
- Files are now closed when encountering an exception in `read_json` (`49921`)
- Changed behavior of `read_csv`, `read_json` & `read_fwf`, where the index will now always be a `RangeIndex`, when no index is specified. Previously the index would be a `Index` with dtype ``object`` if the new DataFrame/Series has length 0 (`49572`)
- `DataFrame.values`, `DataFrame.to_numpy`, `DataFrame.xs`, `DataFrame.reindex`, `DataFrame.fillna`, and `DataFrame.replace` no longer silently consolidate the underlying arrays; do ``df = df.copy()`` to ensure consolidation (`49356`)
- Creating a new DataFrame using a full slice on both axes with `~DataFrame.loc`  or `~DataFrame.iloc` (thus, ``df.loc[:, :]`` or ``df.iloc[:, :]``) now returns a  new DataFrame (shallow copy) instead of the original DataFrame, consistent with other  methods to get a full slice (for example ``df.loc[:]`` or ``df[:]``) (`49469`)
- The `Series` and `DataFrame` constructors will now return a shallow copy  (i.e. share data, but not attributes) when passed a Series and DataFrame,  respectively, and with the default of ``copy=False`` (and if no other keyword triggers  a copy). Previously, the new Series or DataFrame would share the index attribute (e.g.  ``df.index = ...`` would also update the index of the parent or child) (`49523`)
- Disallow computing ``cumprod`` for `Timedelta` object; previously this returned incorrect values (`50246`)
- `DataFrame` objects read from a `HDFStore` file without an index now have a `RangeIndex` instead of an ``int64`` index (`51076`)
- Instantiating an `Index` with an numeric numpy dtype with data containing `NA` and/or `NaT` now raises a ``ValueError``. Previously a ``TypeError`` was raised (`51050`)
- Loading a JSON file with duplicate columns using ``read_json(orient='split')`` renames columns to avoid duplicates, as `read_csv` and the other readers do (`50370`)
- The levels of the index of the `Series` returned from ``Series.sparse.from_coo`` now always have dtype ``int32``. Previously they had dtype ``int64`` (`50926`)
- `to_datetime` with ``unit`` of either "Y" or "M" will now raise if a sequence contains a non-round ``float`` value, matching the ``Timestamp`` behavior (`50301`)
- The methods `Series.round`, `DataFrame.__invert__`, `Series.__invert__`, `DataFrame.swapaxes`, `DataFrame.first`, `DataFrame.last`, `Series.first`, `Series.last` and `DataFrame.align` will now always return new objects (`51032`)
- `DataFrameGroupBy` aggregations (e.g. "sum") with object-dtype columns no longer infer non-object dtypes for their results, explicitly call ``result.infer_objects(copy=False)`` on the result to obtain the old behavior (`51205`)

- Added `pandas.api.types.is_any_real_numeric_dtype` to check for real numeric dtypes (`51152`)

.. ---
.. _whatsnew_200.deprecations:  Deprecations
~~~
- Deprecated parsing datetime strings with system-local timezone to ``tzlocal``, pass a ``tz`` keyword or explicitly call ``tz_localize`` instead (`50791`)
- Deprecated argument ``infer_datetime_format`` in `to_datetime` and `read_csv`, as a strict version of it is now the default (`48621`)
- Deprecated behavior of `to_datetime` with ``unit`` when parsing strings, in a future version these will be parsed as datetimes (matching unit-less behavior) instead of cast to floats. To retain the old behavior, cast strings to numeric types before calling `to_datetime` (`50735`)
- Deprecated `pandas.io.sql.execute` (`50185`)
- `Index.is_boolean` has been deprecated. Use `pandas.api.types.is_bool_dtype` instead (`50042`)
- `Index.is_integer` has been deprecated. Use `pandas.api.types.is_integer_dtype` instead (`50042`)
- `Index.is_floating` has been deprecated. Use `pandas.api.types.is_float_dtype` instead (`50042`)
- `Index.holds_integer` has been deprecated. Use `pandas.api.types.infer_dtype` instead (`50243`)
- `Index.is_numeric` has been deprecated. Use `pandas.api.types.is_any_real_numeric_dtype` instead (`50042`,`51152`)
- `Index.is_categorical` has been deprecated. Use `pandas.api.types.is_categorical_dtype` instead (`50042`)
- `Index.is_object` has been deprecated. Use `pandas.api.types.is_object_dtype` instead (`50042`)
- `Index.is_interval` has been deprecated. Use `pandas.api.types.is_interval_dtype` instead (`50042`)
- Deprecated ``all`` and ``any`` reductions with ``datetime64`` and `DatetimeTZDtype` dtypes, use e.g. ``(obj != pd.Timestamp(0), tz=obj.tz).all()`` instead (`34479`)
- Deprecated unused arguments ``*args`` and ``**kwargs`` in `Resampler` (`50977`)
- Deprecated calling ``float`` or ``int`` on a single element `Series` to return a ``float`` or ``int`` respectively. Extract the element before calling ``float`` or ``int`` instead (`51101`)
- Deprecated `Grouper.groups`, use `Groupby.groups` instead (`51182`)
- Deprecated `Grouper.grouper`, use `Groupby.grouper` instead (`51182`)
- Deprecated `Grouper.obj`, use `Groupby.obj` instead (`51206`)
- Deprecated `Grouper.indexer`, use `Resampler.indexer` instead (`51206`)
- Deprecated `Grouper.ax`, use `Resampler.ax` instead (`51206`)
- Deprecated `Series.pad` in favor of `Series.ffill` (`33396`)
- Deprecated `Series.backfill` in favor of `Series.bfill` (`33396`)
- Deprecated `DataFrame.pad` in favor of `DataFrame.ffill` (`33396`)

- Deprecated `DataFrame.backfill` in favor of `DataFrame.bfill` (`33396`)

.. ---
.. _whatsnew_200.prior_deprecations:  Removal of prior version deprecations/changes
~~~
- Removed `Int64Index`, `UInt64Index` and `Float64Index`. See also `here <whatsnew_200.enhancements.index_can_hold_numpy_numeric_dtypes>` for more information (`42717`)
- Removed deprecated `Timestamp.freq`, `Timestamp.freqstr` and argument ``freq`` from the `Timestamp` constructor and `Timestamp.fromordinal` (`14146`)
- Removed deprecated `CategoricalBlock`, `Block.is_categorical`, require datetime64 and timedelta64 values to be wrapped in `DatetimeArray` or `TimedeltaArray` before passing to `Block.make_block_same_class`, require ``DatetimeTZBlock.values`` to have the correct ndim when passing to the `BlockManager` constructor, and removed the "fastpath" keyword from the `SingleBlockManager` constructor (`40226`, `40571`)
- Removed deprecated global option ``use_inf_as_null`` in favor of ``use_inf_as_na`` (`17126`)
- Removed deprecated module ``pandas.core.index`` (`30193`)
- Removed deprecated alias ``pandas.core.tools.datetimes.to_time``, import the function directly from ``pandas.core.tools.times`` instead (`34145`)
- Removed deprecated alias ``pandas.io.json.json_normalize``, import the function directly from ``pandas.json_normalize`` instead (`27615`)
- Removed deprecated `Categorical.to_dense`, use ``np.asarray(cat)`` instead (`32639`)
- Removed deprecated `Categorical.take_nd` (`27745`)
- Removed deprecated `Categorical.mode`, use ``Series(cat).mode()`` instead (`45033`)
- Removed deprecated `Categorical.is_dtype_equal` and `CategoricalIndex.is_dtype_equal` (`37545`)
- Removed deprecated `CategoricalIndex.take_nd` (`30702`)
- Removed deprecated `Index.is_type_compatible` (`42113`)
- Removed deprecated `Index.is_mixed`, check ``index.inferred_type`` directly instead (`32922`)
- Removed deprecated `pandas.api.types.is_categorical`; use `pandas.api.types.is_categorical_dtype` instead  (`33385`)
- Removed deprecated `Index.asi8` (`37877`)
- Enforced deprecation changing behavior when passing ``datetime64[ns]`` dtype data and timezone-aware dtype to `Series`, interpreting the values as wall-times instead of UTC times, matching `DatetimeIndex` behavior (`41662`)
- Enforced deprecation changing behavior when applying a numpy ufunc on multiple non-aligned (on the index or columns) `DataFrame` that will now align the inputs first (`39239`)
- Removed deprecated `DataFrame._AXIS_NUMBERS`, `DataFrame._AXIS_NAMES`, `Series._AXIS_NUMBERS`, `Series._AXIS_NAMES` (`33637`)
- Removed deprecated `Index.to_native_types`, use ``obj.astype(str)`` instead (`36418`)
- Removed deprecated `Series.iteritems`, `DataFrame.iteritems`, use ``obj.items`` instead (`45321`)
- Removed deprecated `DataFrame.lookup` (`35224`)
- Removed deprecated `Series.append`, `DataFrame.append`, use `concat` instead (`35407`)
- Removed deprecated `Series.iteritems`, `DataFrame.iteritems` and `HDFStore.iteritems` use ``obj.items`` instead (`45321`)
- Removed deprecated `DatetimeIndex.union_many` (`45018`)
- Removed deprecated ``weekofyear`` and ``week`` attributes of `DatetimeArray`, `DatetimeIndex` and ``dt`` accessor in favor of ``isocalendar().week`` (`33595`)
- Removed deprecated `RangeIndex._start`, `RangeIndex._stop`, `RangeIndex._step`, use ``start``, ``stop``, ``step`` instead (`30482`)
- Removed deprecated `DatetimeIndex.to_perioddelta`, Use ``dtindex - dtindex.to_period(freq).to_timestamp()`` instead (`34853`)
- Removed deprecated `.Styler.hide_index` and `.Styler.hide_columns` (`49397`)
- Removed deprecated `.Styler.set_na_rep` and `.Styler.set_precision` (`49397`)
- Removed deprecated `.Styler.where` (`49397`)
- Removed deprecated `.Styler.render` (`49397`)
- Removed deprecated argument ``col_space`` in `DataFrame.to_latex` (`47970`)
- Removed deprecated argument ``null_color`` in `.Styler.highlight_null` (`49397`)

- Removed deprecated argument ``check_less_precise`` in `.testing.assert_frame_equal`, `.testing.assert_extension_array_equal`, `.testing.assert_series_equal`,  `.testing.assert_index_equal` (`30562`)
- Removed deprecated ``null_counts`` argument in `DataFrame.info`. Use ``show_counts`` instead (`37999`)
- Removed deprecated `Index.is_monotonic`, and `Series.is_monotonic`; use ``obj.is_monotonic_increasing`` instead (`45422`)
- Removed deprecated `Index.is_all_dates` (`36697`)
- Enforced deprecation disallowing passing a timezone-aware `Timestamp` and ``dtype="datetime64[ns]"`` to `Series` or `DataFrame` constructors (`41555`)
- Enforced deprecation disallowing passing a sequence of timezone-aware values and ``dtype="datetime64[ns]"`` to to `Series` or `DataFrame` constructors (`41555`)
- Enforced deprecation disallowing ``numpy.ma.mrecords.MaskedRecords`` in the `DataFrame` constructor; pass ``"{name: data[name] for name in data.dtype.names}`` instead (`40363`)
- Enforced deprecation disallowing unit-less "datetime64" dtype in `Series.astype` and `DataFrame.astype` (`47844`)
- Enforced deprecation disallowing using ``.astype`` to convert a ``datetime64[ns]`` `Series`, `DataFrame`, or `DatetimeIndex` to timezone-aware dtype, use ``obj.tz_localize`` or ``ser.dt.tz_localize`` instead (`39258`)
- Enforced deprecation disallowing using ``.astype`` to convert a timezone-aware `Series`, `DataFrame`, or `DatetimeIndex` to timezone-naive ``datetime64[ns]`` dtype, use ``obj.tz_localize(None)`` or ``obj.tz_convert("UTC").tz_localize(None)`` instead (`39258`)
- Enforced deprecation disallowing passing non boolean argument to sort in `concat` (`44629`)
- Removed Date parser functions `~pandas.io.date_converters.parse_date_time`,  `~pandas.io.date_converters.parse_date_fields`, `~pandas.io.date_converters.parse_all_fields`  and `~pandas.io.date_converters.generic_parser` (`24518`)
- Removed argument ``index`` from the `core.arrays.SparseArray` constructor (`43523`)
- Remove argument ``squeeze`` from `DataFrame.groupby` and `Series.groupby` (`32380`)
- Removed deprecated ``apply``, ``apply_index``, ``__call__``, ``onOffset``, and ``isAnchored`` attributes from `DateOffset` (`34171`)
- Removed ``keep_tz`` argument in `DatetimeIndex.to_series` (`29731`)
- Remove arguments ``names`` and ``dtype`` from `Index.copy` and ``levels`` and ``codes`` from `MultiIndex.copy` (`35853`, `36685`)
- Remove argument ``inplace`` from `MultiIndex.set_levels` and `MultiIndex.set_codes` (`35626`)
- Removed arguments ``verbose`` and ``encoding`` from `DataFrame.to_excel` and `Series.to_excel` (`47912`)
- Removed argument ``line_terminator`` from `DataFrame.to_csv` and `Series.to_csv`, use ``lineterminator`` instead (`45302`)
- Removed argument ``inplace`` from `DataFrame.set_axis` and `Series.set_axis`, use ``obj = obj.set_axis(..., copy=False)`` instead (`48130`)
- Disallow passing positional arguments to `MultiIndex.set_levels` and `MultiIndex.set_codes` (`41485`)
- Disallow parsing to Timedelta strings with components with units "Y", "y", or "M", as these do not represent unambiguous durations (`36838`)
- Removed `MultiIndex.is_lexsorted` and `MultiIndex.lexsort_depth` (`38701`)
- Removed argument ``how`` from `PeriodIndex.astype`, use `PeriodIndex.to_timestamp` instead (`37982`)
- Removed argument ``try_cast`` from `DataFrame.mask`, `DataFrame.where`, `Series.mask` and `Series.where` (`38836`)
- Removed argument ``tz`` from `Period.to_timestamp`, use ``obj.to_timestamp(...).tz_localize(tz)`` instead (`34522`)
- Removed argument ``sort_columns`` in `DataFrame.plot` and `Series.plot` (`47563`)
- Removed argument ``is_copy`` from `DataFrame.take` and `Series.take` (`30615`)
- Removed argument ``kind`` from `Index.get_slice_bound`, `Index.slice_indexer` and `Index.slice_locs` (`41378`)

- Removed arguments ``prefix``, ``squeeze``, ``error_bad_lines`` and ``warn_bad_lines`` from `read_csv` (`40413`, `43427`)
- Removed argument ``datetime_is_numeric`` from `DataFrame.describe` and `Series.describe` as datetime data will always be summarized as numeric data (`34798`)
- Disallow passing list ``key`` to `Series.xs` and `DataFrame.xs`, pass a tuple instead (`41789`)
- Disallow subclass-specific keywords (e.g. "freq", "tz", "names", "closed") in the `Index` constructor (`38597`)
- Removed argument ``inplace`` from `Categorical.remove_unused_categories` (`37918`)
- Disallow passing non-round floats to `Timestamp` with ``unit="M"`` or ``unit="Y"`` (`47266`)
- Remove keywords ``convert_float`` and ``mangle_dupe_cols`` from `read_excel` (`41176`)
- Remove keyword ``mangle_dupe_cols`` from `read_csv` and `read_table` (`48137`)
- Removed ``errors`` keyword from `DataFrame.where`, `Series.where`, `DataFrame.mask` and `Series.mask` (`47728`)
- Disallow passing non-keyword arguments to `read_excel` except ``io`` and ``sheet_name`` (`34418`)
- Disallow passing non-keyword arguments to `DataFrame.drop` and `Series.drop` except ``labels`` (`41486`)
- Disallow passing non-keyword arguments to `DataFrame.fillna` and `Series.fillna` except ``value`` (`41485`)
- Disallow passing non-keyword arguments to `StringMethods.split` and `StringMethods.rsplit` except for ``pat`` (`47448`)
- Disallow passing non-keyword arguments to `DataFrame.set_index` except ``keys`` (`41495`)
- Disallow passing non-keyword arguments to `Resampler.interpolate` except ``method`` (`41699`)
- Disallow passing non-keyword arguments to `DataFrame.reset_index` and `Series.reset_index` except ``level`` (`41496`)
- Disallow passing non-keyword arguments to `DataFrame.dropna` and `Series.dropna` (`41504`)
- Disallow passing non-keyword arguments to `ExtensionArray.argsort` (`46134`)
- Disallow passing non-keyword arguments to `Categorical.sort_values` (`47618`)
- Disallow passing non-keyword arguments to `Index.drop_duplicates` and `Series.drop_duplicates` (`41485`)
- Disallow passing non-keyword arguments to `DataFrame.drop_duplicates` except for ``subset`` (`41485`)
- Disallow passing non-keyword arguments to `DataFrame.sort_index` and `Series.sort_index` (`41506`)
- Disallow passing non-keyword arguments to `DataFrame.interpolate` and `Series.interpolate` except for ``method`` (`41510`)
- Disallow passing non-keyword arguments to `DataFrame.any` and `Series.any` (`44896`)
- Disallow passing non-keyword arguments to `Index.set_names` except for ``names`` (`41551`)
- Disallow passing non-keyword arguments to `Index.join` except for ``other`` (`46518`)
- Disallow passing non-keyword arguments to `concat` except for ``objs`` (`41485`)
- Disallow passing non-keyword arguments to `pivot` except for ``data`` (`48301`)
- Disallow passing non-keyword arguments to `DataFrame.pivot` (`48301`)
- Disallow passing non-keyword arguments to `read_html` except for ``io`` (`27573`)
- Disallow passing non-keyword arguments to `read_json` except for ``path_or_buf`` (`27573`)
- Disallow passing non-keyword arguments to `read_sas` except for ``filepath_or_buffer`` (`47154`)
- Disallow passing non-keyword arguments to `read_stata` except for ``filepath_or_buffer`` (`48128`)
- Disallow passing non-keyword arguments to `read_csv` except ``filepath_or_buffer`` (`41485`)
- Disallow passing non-keyword arguments to `read_table` except ``filepath_or_buffer`` (`41485`)
- Disallow passing non-keyword arguments to `read_fwf` except ``filepath_or_buffer`` (`44710`)
- Disallow passing non-keyword arguments to `read_xml` except for ``path_or_buffer`` (`45133`)
- Disallow passing non-keyword arguments to `Series.mask` and `DataFrame.mask` except ``cond`` and ``other`` (`41580`)

- Disallow passing non-keyword arguments to `DataFrame.to_stata` except for ``path`` (`48128`)
- Disallow passing non-keyword arguments to `DataFrame.where` and `Series.where` except for ``cond`` and ``other`` (`41523`)
- Disallow passing non-keyword arguments to `Series.set_axis` and `DataFrame.set_axis` except for ``labels`` (`41491`)
- Disallow passing non-keyword arguments to `Series.rename_axis` and `DataFrame.rename_axis` except for ``mapper`` (`47587`)
- Disallow passing non-keyword arguments to `Series.clip` and `DataFrame.clip` (`41511`)
- Disallow passing non-keyword arguments to `Series.bfill`, `Series.ffill`, `DataFrame.bfill` and `DataFrame.ffill` (`41508`)
- Disallow passing non-keyword arguments to `DataFrame.replace`, `Series.replace` except for ``to_replace`` and ``value`` (`47587`)
- Disallow passing non-keyword arguments to `DataFrame.sort_values` except for ``by`` (`41505`)
- Disallow passing non-keyword arguments to `Series.sort_values` (`41505`)
- Disallow passing non-keyword arguments to `DataFrame.reindex` except for ``labels`` (`17966`)
- Disallow `Index.reindex` with non-unique `Index` objects (`42568`)
- Disallowed constructing `Categorical` with scalar ``data`` (`38433`)
- Disallowed constructing `CategoricalIndex` without passing ``data`` (`38944`)
- Removed `.Rolling.validate`, `.Expanding.validate`, and `.ExponentialMovingWindow.validate` (`43665`)
- Removed `Rolling.win_type` returning ``"freq"`` (`38963`)
- Removed `Rolling.is_datetimelike` (`38963`)
- Removed the ``level`` keyword in `DataFrame` and `Series` aggregations; use ``groupby`` instead (`39983`)
- Removed deprecated `Timedelta.delta`, `Timedelta.is_populated`, and `Timedelta.freq` (`46430`, `46476`)
- Removed deprecated `NaT.freq` (`45071`)
- Removed deprecated `Categorical.replace`, use `Series.replace` instead (`44929`)
- Removed the ``numeric_only`` keyword from `Categorical.min` and `Categorical.max` in favor of ``skipna`` (`48821`)
- Changed behavior of `DataFrame.median` and `DataFrame.mean` with ``numeric_only=None`` to not exclude datetime-like columns THIS NOTE WILL BE IRRELEVANT ONCE ``numeric_only=None`` DEPRECATION IS ENFORCED (`29941`)
- Removed `is_extension_type` in favor of `is_extension_array_dtype` (`29457`)
- Removed ``.ExponentialMovingWindow.vol`` (`39220`)
- Removed `Index.get_value` and `Index.set_value` (`33907`, `28621`)
- Removed `Series.slice_shift` and `DataFrame.slice_shift` (`37601`)
- Remove `DataFrameGroupBy.pad` and `DataFrameGroupBy.backfill` (`45076`)
- Remove ``numpy`` argument from `read_json` (`30636`)
- Disallow passing abbreviations for ``orient`` in `DataFrame.to_dict` (`32516`)
- Disallow partial slicing on an non-monotonic `DatetimeIndex` with keys which are not in Index. This now raises a ``KeyError`` (`18531`)
- Removed ``get_offset`` in favor of `to_offset` (`30340`)
- Removed the ``warn`` keyword in `infer_freq` (`45947`)
- Removed the ``include_start`` and ``include_end`` arguments in `DataFrame.between_time` in favor of ``inclusive`` (`43248`)
- Removed the ``closed`` argument in `date_range` and `bdate_range` in favor of ``inclusive`` argument (`40245`)
- Removed the ``center`` keyword in `DataFrame.expanding` (`20647`)
- Removed the ``truediv`` keyword from `eval` (`29812`)
- Removed the ``method`` and ``tolerance`` arguments in `Index.get_loc`. Use ``index.get_indexer([label], method=..., tolerance=...)`` instead (`42269`)
- Removed the ``pandas.datetime`` submodule (`30489`)
- Removed the ``pandas.np`` submodule (`30296`)
- Removed ``pandas.util.testing`` in favor of ``pandas.testing`` (`30745`)
- Removed `Series.str.__iter__` (`28277`)
- Removed ``pandas.SparseArray`` in favor of `arrays.SparseArray` (`30642`)
- Removed ``pandas.SparseSeries`` and ``pandas.SparseDataFrame``, including pickle support. (`30642`)

- Enforced disallowing passing an integer ``fill_value`` to `DataFrame.shift` and `Series.shift`` with datetime64, timedelta64, or period dtypes (`32591`)
- Enforced disallowing a string column label into ``times`` in `DataFrame.ewm` (`43265`)
- Enforced disallowing passing ``True`` and ``False`` into ``inclusive`` in `Series.between` in favor of ``"both"`` and ``"neither"`` respectively (`40628`)
- Enforced disallowing using ``usecols`` with out of bounds indices for ``read_csv`` with ``engine="c"`` (`25623`)
- Enforced disallowing the use of ``**kwargs`` in `.ExcelWriter`; use the keyword argument ``engine_kwargs`` instead (`40430`)
- Enforced disallowing a tuple of column labels into `.DataFrameGroupBy.__getitem__` (`30546`)
- Enforced disallowing missing labels when indexing with a sequence of labels on a level of a `MultiIndex`. This now raises a ``KeyError`` (`42351`)
- Enforced disallowing setting values with ``.loc`` using a positional slice. Use ``.loc`` with labels or ``.iloc`` with positions instead (`31840`)
- Enforced disallowing positional indexing with a ``float`` key even if that key is a round number, manually cast to integer instead (`34193`)
- Enforced disallowing using a `DataFrame` indexer with ``.iloc``, use ``.loc`` instead for automatic alignment (`39022`)
- Enforced disallowing ``set`` or ``dict`` indexers in ``__getitem__`` and ``__setitem__`` methods (`42825`)
- Enforced disallowing indexing on a `Index` or positional indexing on a `Series` producing multi-dimensional objects e.g. ``obj[:, None]``, convert to numpy before indexing instead (`35141`)
- Enforced disallowing ``dict`` or ``set`` objects in ``suffixes`` in `merge` (`34810`)
- Enforced disallowing `merge` to produce duplicated columns through the ``suffixes`` keyword and already existing columns (`22818`)
- Enforced disallowing using `merge` or `join` on a different number of levels (`34862`)
- Enforced disallowing ``value_name`` argument in `DataFrame.melt` to match an element in the `DataFrame` columns (`35003`)
- Enforced disallowing passing ``showindex`` into ``**kwargs`` in `DataFrame.to_markdown` and `Series.to_markdown` in favor of ``index`` (`33091`)
- Removed setting Categorical._codes directly (`41429`)
- Removed setting Categorical.categories directly (`47834`)
- Removed argument ``inplace`` from `Categorical.add_categories`, `Categorical.remove_categories`, `Categorical.set_categories`, `Categorical.rename_categories`, `Categorical.reorder_categories`, `Categorical.set_ordered`, `Categorical.as_ordered`, `Categorical.as_unordered` (`37981`, `41118`, `41133`, `47834`)
- Enforced `Rolling.count` with ``min_periods=None`` to default to the size of the window (`31302`)
- Renamed ``fname`` to ``path`` in `DataFrame.to_parquet`, `DataFrame.to_stata` and `DataFrame.to_feather` (`30338`)
- Enforced disallowing indexing a `Series` with a single item list with a slice (e.g. ``ser[[slice(0, 2)]]``). Either convert the list to tuple, or pass the slice directly instead (`31333`)
- Changed behavior indexing on a `DataFrame` with a `DatetimeIndex` index using a string indexer, previously this operated as a slice on rows, now it operates like any other column key; use ``frame.loc[key]`` for the old behavior (`36179`)
- Enforced the ``display.max_colwidth`` option to not accept negative integers (`31569`)
- Removed the ``display.column_space`` option in favor of ``df.to_string(col_space=...)`` (`47280`)
- Removed the deprecated method ``mad`` from pandas classes (`11787`)
- Removed the deprecated method ``tshift`` from pandas classes (`11631`)
- Changed behavior of empty data passed into `Series`; the default dtype will be ``object`` instead of ``float64`` (`29405`)

- Changed the behavior of `DatetimeIndex.union`, `DatetimeIndex.intersection`, and `DatetimeIndex.symmetric_difference` with mismatched timezones to convert to UTC instead of casting to object dtype (`39328`)
- Changed the behavior of `to_datetime` with argument "now" with ``utc=False`` to match ``Timestamp("now")`` (`18705`)
- Changed the behavior of indexing on a timezone-aware `DatetimeIndex` with a timezone-naive ``datetime`` object or vice-versa; these now behave like any other non-comparable type by raising ``KeyError`` (`36148`)
- Changed the behavior of `Index.reindex`, `Series.reindex`, and `DataFrame.reindex` with a ``datetime64`` dtype and a ``datetime.date`` object for ``fill_value``; these are no longer considered equivalent to ``datetime.datetime`` objects so the reindex casts to object dtype (`39767`)
- Changed behavior of `SparseArray.astype` when given a dtype that is not explicitly ``SparseDtype``, cast to the exact requested dtype rather than silently using a ``SparseDtype`` instead (`34457`)
- Changed behavior of `Index.ravel` to return a view on the original `Index` instead of a ``np.ndarray`` (`36900`)
- Changed behavior of `Series.to_frame` and `Index.to_frame` with explicit ``name=None`` to use ``None`` for the column name instead of the index's name or default ``0`` (`45523`)
- Changed behavior of `concat` with one array of ``bool``-dtype and another of integer dtype, this now returns ``object`` dtype instead of integer dtype; explicitly cast the bool object to integer before concatenating to get the old behavior (`45101`)
- Changed behavior of `DataFrame` constructor given floating-point ``data`` and an integer ``dtype``, when the data cannot be cast losslessly, the floating point dtype is retained, matching `Series` behavior (`41170`)
- Changed behavior of `Index` constructor when given a ``np.ndarray`` with object-dtype containing numeric entries; this now retains object dtype rather than inferring a numeric dtype, consistent with `Series` behavior (`42870`)
- Changed behavior of `Index.__and__`, `Index.__or__` and `Index.__xor__` to behave as logical operations (matching `Series` behavior) instead of aliases for set operations (`37374`)
- Changed behavior of `DataFrame` constructor when passed a list whose first element is a `Categorical`, this now treats the elements as rows casting to ``object`` dtype, consistent with behavior for other types (`38845`)
- Changed behavior of `DataFrame` constructor when passed a ``dtype`` (other than int) that the data cannot be cast to; it now raises instead of silently ignoring the dtype (`41733`)
- Changed the behavior of `Series` constructor, it will no longer infer a datetime64 or timedelta64 dtype from string entries (`41731`)
- Changed behavior of `Timestamp` constructor with a ``np.datetime64`` object and a ``tz`` passed to interpret the input as a wall-time as opposed to a UTC time (`42288`)
- Changed behavior of `Timestamp.utcfromtimestamp` to return a timezone-aware object satisfying ``Timestamp.utcfromtimestamp(val).timestamp() == val`` (`45083`)
- Changed behavior of `Index` constructor when passed a ``SparseArray`` or ``SparseDtype`` to retain that dtype instead of casting to ``numpy.ndarray`` (`43930`)
- Changed behavior of setitem-like operations (``__setitem__``, ``fillna``, ``where``, ``mask``, ``replace``, ``insert``, fill_value for ``shift``) on an object with `DatetimeTZDtype` when using a value with a non-matching timezone, the value will be cast to the object's timezone instead of casting both to object-dtype (`44243`)
- Changed behavior of `Index`, `Series`, `DataFrame` constructors with floating-dtype data and a `DatetimeTZDtype`, the data are now interpreted as UTC-times instead of wall-times, consistent with how integer-dtype data are treated (`45573`)

- Changed behavior of `Series` and `DataFrame` constructors with integer dtype and floating-point data containing ``NaN``, this now raises ``IntCastingNaNError`` (`40110`)
- Changed behavior of `Series` and `DataFrame` constructors with an integer ``dtype`` and values that are too large to losslessly cast to this dtype, this now raises ``ValueError`` (`41734`)
- Changed behavior of `Series` and `DataFrame` constructors with an integer ``dtype`` and values having either ``datetime64`` or ``timedelta64`` dtypes, this now raises ``TypeError``, use ``values.view("int64")`` instead (`41770`)
- Removed the deprecated ``base`` and ``loffset`` arguments from `pandas.DataFrame.resample`, `pandas.Series.resample` and `pandas.Grouper`. Use ``offset`` or ``origin`` instead (`31809`)
- Changed behavior of `Series.fillna` and `DataFrame.fillna` with ``timedelta64[ns]`` dtype and an incompatible ``fill_value``; this now casts to ``object`` dtype instead of raising, consistent with the behavior with other dtypes (`45746`)
- Change the default argument of ``regex`` for `Series.str.replace` from ``True`` to ``False``. Additionally, a single character ``pat`` with ``regex=True`` is now treated as a regular expression instead of a string literal. (`36695`, `24804`)
- Changed behavior of `DataFrame.any` and `DataFrame.all` with ``bool_only=True``; object-dtype columns with all-bool values will no longer be included, manually cast to ``bool`` dtype first (`46188`)
- Changed behavior of `DataFrame.max`, `DataFrame.min`, `DataFrame.mean`, `DataFrame.median`, `DataFrame.skew`, `DataFrame.kurt` with ``axis=None`` to return a scalar applying the aggregation across both axes (`45072`)
- Changed behavior of comparison of a `Timestamp` with a ``datetime.date`` object; these now compare as un-equal and raise on inequality comparisons, matching the ``datetime.datetime`` behavior (`36131`)
- Changed behavior of comparison of ``NaT`` with a ``datetime.date`` object; these now raise on inequality comparisons (`39196`)
- Enforced deprecation of silently dropping columns that raised a ``TypeError`` in `Series.transform` and `DataFrame.transform` when used with a list or dictionary (`43740`)
- Changed behavior of `DataFrame.apply` with list-like so that any partial failure will raise an error (`43740`)
- Changed behaviour of `DataFrame.to_latex` to now use the Styler implementation via `.Styler.to_latex` (`47970`)
- Changed behavior of `Series.__setitem__` with an integer key and a `Float64Index` when the key is not present in the index; previously we treated the key as positional (behaving like ``series.iloc[key] = val``), now we treat it is a label (behaving like ``series.loc[key] = val``), consistent with `Series.__getitem__`` behavior (`33469`)
- Removed ``na_sentinel`` argument from `factorize`, `.Index.factorize`, and `.ExtensionArray.factorize` (`47157`)
- Changed behavior of `Series.diff` and `DataFrame.diff` with `ExtensionDtype` dtypes whose arrays do not implement ``diff``, these now raise ``TypeError`` rather than casting to numpy (`31025`)
- Enforced deprecation of calling numpy "ufunc"s on `DataFrame` with ``method="outer"``; this now raises ``NotImplementedError`` (`36955`)
- Enforced deprecation disallowing passing ``numeric_only=True`` to `Series` reductions (``rank``, ``any``, ``all``, ...) with non-numeric dtype (`47500`)
- Changed behavior of `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` so that ``group_keys`` is respected even if a transformer is detected (`34998`)
- Comparisons between a `DataFrame` and a `Series` where the frame's columns do not match the series's index raise ``ValueError`` instead of automatically aligning, do ``left, right = left.align(right, axis=1, copy=False)`` before comparing (`36795`)
- Enforced deprecation ``numeric_only=None`` (the default) in DataFrame reductions that would silently drop columns that raised; ``numeric_only`` now defaults to ``False`` (`41480`)

- Changed default of ``numeric_only`` to ``False`` in all DataFrame methods with that argument (`46096`, `46906`)
- Changed default of ``numeric_only`` to ``False`` in `Series.rank` (`47561`)
- Enforced deprecation of silently dropping nuisance columns in groupby and resample operations when ``numeric_only=False`` (`41475`)
- Enforced deprecation of silently dropping nuisance columns in `Rolling`, `Expanding`, and `ExponentialMovingWindow` ops. This will now raise a `.errors.DataError` (`42834`)
- Changed behavior in setting values with ``df.loc[:, foo] = bar`` or ``df.iloc[:, foo] = bar``, these now always attempt to set values inplace before falling back to casting (`45333`)
- Changed default of ``numeric_only`` in various `.DataFrameGroupBy` methods; all methods now default to ``numeric_only=False`` (`46072`)
- Changed default of ``numeric_only`` to ``False`` in `.Resampler` methods (`47177`)
- Using the method `DataFrameGroupBy.transform` with a callable that returns DataFrames will align to the input's index (`47244`)
- When providing a list of columns of length one to `DataFrame.groupby`, the keys that are returned by iterating over the resulting `DataFrameGroupBy` object will now be tuples of length one (`47761`)
- Removed deprecated methods `ExcelWriter.write_cells`, `ExcelWriter.save`, `ExcelWriter.cur_sheet`, `ExcelWriter.handles`, `ExcelWriter.path` (`45795`)
- The `ExcelWriter` attribute ``book`` can no longer be set; it is still available to be accessed and mutated (`48943`)
- Removed unused ``*args`` and ``**kwargs`` in `Rolling`, `Expanding`, and `ExponentialMovingWindow` ops (`47851`)
- Removed the deprecated argument ``line_terminator`` from `DataFrame.to_csv` (`45302`)
- Removed the deprecated argument ``label`` from `lreshape` (`30219`)
- Arguments after ``expr`` in `DataFrame.eval` and `DataFrame.query` are keyword-only (`47587`)
- Removed `Index._get_attributes_dict` (`50648`)
- Removed `Series.__array_wrap__` (`50648`)  .. ---
.. _whatsnew_200.performance:  Performance improvements
~~~
- Performance improvement in `.DataFrameGroupBy.median` and `.SeriesGroupBy.median` and `.DataFrameGroupBy.cumprod` for nullable dtypes (`37493`)
- Performance improvement in `.DataFrameGroupBy.all`, `.DataFrameGroupBy.any`, `.SeriesGroupBy.all`, and `.SeriesGroupBy.any` for object dtype (`50623`)
- Performance improvement in `MultiIndex.argsort` and `MultiIndex.sort_values` (`48406`)
- Performance improvement in `MultiIndex.size` (`48723`)
- Performance improvement in `MultiIndex.union` without missing values and without duplicates (`48505`, `48752`)
- Performance improvement in `MultiIndex.difference` (`48606`)
- Performance improvement in `MultiIndex` set operations with sort=None (`49010`)
- Performance improvement in `.DataFrameGroupBy.mean`, `.SeriesGroupBy.mean`, `.DataFrameGroupBy.var`, and `.SeriesGroupBy.var` for extension array dtypes (`37493`)
- Performance improvement in `MultiIndex.isin` when ``level=None`` (`48622`, `49577`)
- Performance improvement in `MultiIndex.putmask` (`49830`)
- Performance improvement in `Index.union` and `MultiIndex.union` when index contains duplicates (`48900`)
- Performance improvement in `Series.rank` for pyarrow-backed dtypes (`50264`)
- Performance improvement in `Series.searchsorted` for pyarrow-backed dtypes (`50447`)
- Performance improvement in `Series.fillna` for extension array dtypes (`49722`, `50078`)
- Performance improvement in `Index.join`, `Index.intersection` and `Index.union` for masked dtypes when `Index` is monotonic (`50310`)
- Performance improvement for `Series.value_counts` with nullable dtype (`48338`)
- Performance improvement for `Series` constructor passing integer numpy array with nullable dtype (`48338`)
- Performance improvement for `DatetimeIndex` constructor passing a list (`48609`)

- Performance improvement in `merge` and `DataFrame.join` when joining on a sorted `MultiIndex` (`48504`)
- Performance improvement in `to_datetime` when parsing strings with timezone offsets (`50107`)
- Performance improvement in `DataFrame.loc` and `Series.loc` for tuple-based indexing of a `MultiIndex` (`48384`)
- Performance improvement for `Series.replace` with categorical dtype (`49404`)
- Performance improvement for `MultiIndex.unique` (`48335`)
- Performance improvement for indexing operations with nullable dtypes (`49420`)
- Performance improvement for `concat` with extension array backed indexes (`49128`, `49178`)
- Performance improvement for `api.types.infer_dtype` (`51054`)
- Reduce memory usage of `DataFrame.to_pickle`/`Series.to_pickle` when using BZ2 or LZMA (`49068`)
- Performance improvement for `~arrays.StringArray` constructor passing a numpy array with type ``np.str_`` (`49109`)
- Performance improvement in `~arrays.IntervalArray.from_tuples` (`50620`)
- Performance improvement in `~arrays.ArrowExtensionArray.factorize` (`49177`)
- Performance improvement in `~arrays.ArrowExtensionArray.__setitem__` (`50248`, `50632`)
- Performance improvement in `~arrays.ArrowExtensionArray` comparison methods when array contains NA (`50524`)
- Performance improvement in `~arrays.ArrowExtensionArray.to_numpy` (`49973`)
- Performance improvement when parsing strings to `BooleanDtype` (`50613`)
- Performance improvement in `DataFrame.join` when joining on a subset of a `MultiIndex` (`48611`)
- Performance improvement for `MultiIndex.intersection` (`48604`)
- Performance improvement in `DataFrame.__setitem__` (`46267`)
- Performance improvement in ``var`` and ``std`` for nullable dtypes (`48379`).
- Performance improvement when iterating over pyarrow and nullable dtypes (`49825`, `49851`)
- Performance improvements to `read_sas` (`47403`, `47405`, `47656`, `48502`)
- Memory improvement in `RangeIndex.sort_values` (`48801`)
- Performance improvement in `Series.to_numpy` if ``copy=True`` by avoiding copying twice (`24345`)
- Performance improvement in `Series.rename` with `MultiIndex` (`21055`)
- Performance improvement in `DataFrameGroupBy` and `SeriesGroupBy` when ``by`` is a categorical type and ``sort=False`` (`48976`)
- Performance improvement in `DataFrameGroupBy` and `SeriesGroupBy` when ``by`` is a categorical type and ``observed=False`` (`49596`)
- Performance improvement in `read_stata` with parameter ``index_col`` set to ``None`` (the default). Now the index will be a `RangeIndex` instead of `Int64Index` (`49745`)
- Performance improvement in `merge` when not merging on the index - the new index will now be `RangeIndex` instead of `Int64Index` (`49478`)
- Performance improvement in `DataFrame.to_dict` and `Series.to_dict` when using any non-object dtypes (`46470`)
- Performance improvement in `read_html` when there are multiple tables (`49929`)
- Performance improvement in `Period` constructor when constructing from a string or integer (`38312`)
- Performance improvement in `to_datetime` when using ``'%Y%m%d'`` format (`17410`)
- Performance improvement in `to_datetime` when format is given or can be inferred (`50465`)
- Performance improvement in `Series.median` for nullable dtypes (`50838`)
- Performance improvement in `read_csv` when passing `to_datetime` lambda-function to ``date_parser`` and inputs have mixed timezone offsetes (`35296`)
- Performance improvement in `isna` and `isnull` (`50658`)
- Performance improvement in `.SeriesGroupBy.value_counts` with categorical dtype (`46202`)
- Fixed a reference leak in `read_hdf` (`37441`)
- Fixed a memory leak in `DataFrame.to_json` and `Series.to_json` when serializing datetimes and timedeltas (`40443`)

- Decreased memory usage in many `DataFrameGroupBy` methods (`51090`)

.. ---
.. _whatsnew_200.bug_fixes:  Bug fixes
^^^
- Bug in `Categorical.set_categories` losing dtype information (`48812`)
- Bug in `Series.replace` with categorical dtype when ``to_replace`` values overlap with new values (`49404`)
- Bug in `Series.replace` with categorical dtype losing nullable dtypes of underlying categories (`49404`)
- Bug in `DataFrame.groupby` and `Series.groupby` would reorder categories when used as a grouper (`48749`)
- Bug in `Categorical` constructor when constructing from a `Categorical` object and ``dtype="category"`` losing ordered-ness (`49309`)

- Bug in `.SeriesGroupBy.min`, `.SeriesGroupBy.max`, `.DataFrameGroupBy.min`, and `.DataFrameGroupBy.max` with unordered `CategoricalDtype` with no groups failing to raise ``TypeError`` (`51034`)

Datetimelike
^^^
- Bug in `pandas.infer_freq`, raising ``TypeError`` when inferred on `RangeIndex` (`47084`)
- Bug in `to_datetime` incorrectly raising ``OverflowError`` with string arguments corresponding to large integers (`50533`)
- Bug in `to_datetime` was raising on invalid offsets with ``errors='coerce'`` and ``infer_datetime_format=True`` (`48633`)
- Bug in `DatetimeIndex` constructor failing to raise when ``tz=None`` is explicitly specified in conjunction with timezone-aware ``dtype`` or data (`48659`)
- Bug in subtracting a ``datetime`` scalar from `DatetimeIndex` failing to retain the original ``freq`` attribute (`48818`)
- Bug in ``pandas.tseries.holiday.Holiday`` where a half-open date interval causes inconsistent return types from `USFederalHolidayCalendar.holidays` (`49075`)
- Bug in rendering `DatetimeIndex` and `Series` and `DataFrame` with timezone-aware dtypes with ``dateutil`` or ``zoneinfo`` timezones near daylight-savings transitions (`49684`)
- Bug in `to_datetime` was raising ``ValueError`` when parsing `Timestamp`, ``datetime.datetime``, ``datetime.date``, or ``np.datetime64`` objects when non-ISO8601 ``format`` was passed (`49298`, `50036`)
- Bug in `to_datetime` was raising ``ValueError`` when parsing empty string and non-ISO8601 format was passed. Now, empty strings will be parsed as `NaT`, for compatibility with how is done for ISO8601 formats (`50251`)
- Bug in `Timestamp` was showing ``UserWarning``, which was not actionable by users, when parsing non-ISO8601 delimited date strings (`50232`)
- Bug in `to_datetime` was showing misleading ``ValueError`` when parsing dates with format containing ISO week directive and ISO weekday directive (`50308`)
- Bug in `Timestamp.round` when the ``freq`` argument has zero-duration (e.g. "0ns") returning incorrect results instead of raising (`49737`)
- Bug in `to_datetime` was not raising ``ValueError`` when invalid format was passed and ``errors`` was ``'ignore'`` or ``'coerce'`` (`50266`)
- Bug in `DateOffset` was throwing ``TypeError`` when constructing with milliseconds and another super-daily argument (`49897`)
- Bug in `to_datetime` was not raising ``ValueError`` when parsing string with decimal date with format ``'%Y%m%d'`` (`50051`)
- Bug in `to_datetime` was not converting ``None`` to ``NaT`` when parsing mixed-offset date strings with ISO8601 format (`50071`)
- Bug in `to_datetime` was not returning input when parsing out-of-bounds date string with ``errors='ignore'`` and ``format='%Y%m%d'`` (`14487`)
- Bug in `to_datetime` was converting timezone-naive ``datetime.datetime`` to timezone-aware when parsing with timezone-aware strings, ISO8601 format, and ``utc=False`` (`50254`)
- Bug in `to_datetime` was throwing ``ValueError`` when parsing dates with ISO8601 format where some values were not zero-padded (`21422`)
- Bug in `to_datetime` was giving incorrect results when using ``format='%Y%m%d'`` and ``errors='ignore'`` (`26493`)
- Bug in `to_datetime` was failing to parse date strings ``'today'`` and ``'now'`` if ``format`` was not ISO8601 (`50359`)
- Bug in `Timestamp.utctimetuple` raising a ``TypeError`` (`32174`)
- Bug in `to_datetime` was raising ``ValueError`` when parsing mixed-offset `Timestamp` with ``errors='ignore'`` (`50585`)
- Bug in `to_datetime` was incorrectly handling floating-point inputs within 1 ``unit`` of the overflow boundaries (`50183`)
- Bug in `to_datetime` with unit of "Y" or "M" giving incorrect results, not matching pointwise `Timestamp` results (`50870`)
- Bug in `Series.interpolate` and `DataFrame.interpolate` with datetime or timedelta dtypes incorrectly raising ``ValueError`` (`11312`)
- Bug in `to_datetime` was not returning input with ``errors='ignore'`` when input was out-of-bounds (`50587`)
- Bug in `DataFrame.from_records` when given a `DataFrame` input with timezone-aware datetime64 columns incorrectly dropping the timezone-awareness (`51162`)

- Bug in `to_datetime` was raising ``decimal.InvalidOperation`` when parsing date strings with ``errors='coerce'`` (`51084`)
- Bug in `to_datetime` with both ``unit`` and ``origin`` specified returning incorrect results (`42624`)
- Bug in `GroupBy.quantile` with datetime or timedelta dtypes giving incorrect results for groups containing ``NaT`` (`51373`)

- Bug in `Groupby.quantile` incorrectly raising with `PeriodDtype` or `DatetimeTZDtype` (`51373`)

Timedelta
^^^
- Bug in `to_timedelta` raising error when input has nullable dtype ``Float64`` (`48796`)
- Bug in `Timedelta` constructor incorrectly raising instead of returning ``NaT`` when given a ``np.timedelta64("nat")`` (`48898`)

- Bug in `Timedelta` constructor failing to raise when passed both a `Timedelta` object and keywords (e.g. days, seconds) (`48898`)

Timezones
^^^
- Bug in `Series.astype` and `DataFrame.astype` with object-dtype containing multiple timezone-aware ``datetime`` objects with heterogeneous timezones to a `DatetimeTZDtype` incorrectly raising (`32581`)
- Bug in `to_datetime` was failing to parse date strings with timezone name when ``format`` was specified with ``%Z`` (`49748`)
- Better error message when passing invalid values to ``ambiguous`` parameter in `Timestamp.tz_localize` (`49565`)

- Bug in string parsing incorrectly allowing a `Timestamp` to be constructed with an invalid timezone, which would raise when trying to print (`50668`)

Numeric
^^^
- Bug in `DataFrame.add` cannot apply ufunc when inputs contain mixed DataFrame type and Series type (`39853`)
- Bug in arithmetic operations on `Series` not propagating mask when combining masked dtypes and numpy dtypes (`45810`, `42630`)
- Bug in DataFrame reduction methods (e.g. `DataFrame.sum`) with object dtype, ``axis=1`` and ``numeric_only=False`` would not be coerced to float (`49551`)
- Bug in `DataFrame.sem` and `Series.sem` where an erroneous ``TypeError`` would always raise when using data backed by an `ArrowDtype` (`49759`)
- Bug in `Series.__add__` casting to object for list and masked `Series` (`22962`)
- Bug in `DataFrame.query` with ``engine="numexpr"`` and column names are ``min`` or ``max`` would raise a ``TypeError`` (`50937`)  Conversion
^^^
- Bug in constructing `Series` with ``int64`` dtype from a string list raising instead of casting (`44923`)
- Bug in constructing `Series` with masked dtype and boolean values with ``NA`` raising (`42137`)
- Bug in `DataFrame.eval` incorrectly raising an ``AttributeError`` when there are negative values in function call (`46471`)
- Bug in `Series.convert_dtypes` not converting dtype to nullable dtype when `Series` contains ``NA`` and has dtype ``object`` (`48791`)
- Bug where any `ExtensionDtype` subclass with ``kind="M"`` would be interpreted as a timezone type (`34986`)
- Bug in `.arrays.ArrowExtensionArray` that would raise ``NotImplementedError`` when passed a sequence of strings or binary (`49172`)
- Bug in `Series.astype` raising ``pyarrow.ArrowInvalid`` when converting from a non-pyarrow string dtype to a pyarrow numeric type (`50430`)
- Bug in `DataFrame.astype` modifying input array inplace when converting to ``string`` and ``copy=False`` (`51073`)
- Bug in `Series.to_numpy` converting to NumPy array before applying ``na_value`` (`48951`)
- Bug in `DataFrame.astype` not copying data when converting to pyarrow dtype (`50984`)
- Bug in `to_datetime` was not respecting ``exact`` argument when ``format`` was an ISO8601 format (`12649`)
- Bug in `TimedeltaArray.astype` raising ``TypeError`` when converting to a pyarrow duration type (`49795`)

- Bug in `DataFrame.eval` and `DataFrame.query` raising for extension array dtypes (`29618`, `50261`, `31913`)

Strings
^^^
- Bug in `pandas.api.types.is_string_dtype` that would not return ``True`` for `StringDtype` or `ArrowDtype` with ``pyarrow.string()`` (`15585`)
- Bug in converting string dtypes to "datetime64[ns]" or "timedelta64[ns]" incorrectly raising ``TypeError`` (`36153`)

- Bug in setting values in a string-dtype column with an array, mutating the array as side effect when it contains missing values (`51299`)

Interval
^^^
- Bug in `IntervalIndex.is_overlapping` incorrect output if interval has duplicate left boundaries (`49581`)
- Bug in `Series.infer_objects` failing to infer `IntervalDtype` for an object series of `Interval` objects (`50090`)

- Bug in `Series.shift` with `IntervalDtype` and invalid null ``fill_value`` failing to raise ``TypeError`` (`51258`)

Indexing
^^^
- Bug in `DataFrame.__setitem__` raising when indexer is a `DataFrame` with ``boolean`` dtype (`47125`)
- Bug in `DataFrame.reindex` filling with wrong values when indexing columns and index for ``uint`` dtypes (`48184`)
- Bug in `DataFrame.loc` when setting `DataFrame` with different dtypes coercing values to single dtype (`50467`)
- Bug in `DataFrame.sort_values` where ``None`` was not returned when ``by`` is empty list and ``inplace=True`` (`50643`)
- Bug in `DataFrame.loc` coercing dtypes when setting values with a list indexer (`49159`)
- Bug in `Series.loc` raising error for out of bounds end of slice indexer (`50161`)
- Bug in `DataFrame.loc` raising ``ValueError`` with ``bool`` indexer and `MultiIndex` (`47687`)
- Bug in `DataFrame.loc` raising ``IndexError`` when setting values for a pyarrow-backed column with a non-scalar indexer (`50085`)
- Bug in `DataFrame.__getitem__`, `Series.__getitem__`, `DataFrame.__setitem__` and `Series.__setitem__`  when indexing on indexes with extension float dtypes (`Float64` & `Float64`) or complex dtypes using integers (`51053`)
- Bug in `DataFrame.loc` modifying object when setting incompatible value with an empty indexer (`45981`)
- Bug in `DataFrame.__setitem__` raising ``ValueError`` when right hand side is `DataFrame` with `MultiIndex` columns (`49121`)
- Bug in `DataFrame.reindex` casting dtype to ``object`` when `DataFrame` has single extension array column when re-indexing ``columns`` and ``index`` (`48190`)
- Bug in `DataFrame.iloc` raising ``IndexError`` when indexer is a `Series` with numeric extension array dtype (`49521`)
- Bug in `~DataFrame.describe` when formatting percentiles in the resulting index showed more decimals than needed (`46362`)
- Bug in `DataFrame.compare` does not recognize differences when comparing ``NA`` with value in nullable dtypes (`48939`)
- Bug in `Series.rename` with `MultiIndex` losing extension array dtypes (`21055`)
- Bug in `DataFrame.isetitem` coercing extension array dtypes in `DataFrame` to object (`49922`)

- Bug in `BusinessHour` would cause creation of `DatetimeIndex` to fail when no opening hour was included in the index (`49835`)

Missing
^^^
- Bug in `Index.equals` raising ``TypeError`` when `Index` consists of tuples that contain ``NA`` (`48446`)
- Bug in `Series.map` caused incorrect result when data has NaNs and defaultdict mapping was used (`48813`)
- Bug in `NA` raising a ``TypeError`` instead of return `NA` when performing a binary operation with a ``bytes`` object (`49108`)
- Bug in `DataFrame.update` with ``overwrite=False`` raising ``TypeError`` when ``self`` has column with ``NaT`` values and column not present in ``other`` (`16713`)
- Bug in `Series.replace` raising ``RecursionError`` when replacing value in object-dtype `Series` containing ``NA`` (`47480`)
- Bug in `Series.replace` raising ``RecursionError`` when replacing value in numeric `Series` with ``NA`` (`50758`)  MultiIndex
^^^
- Bug in `MultiIndex.get_indexer` not matching ``NaN`` values (`29252`, `37222`, `38623`, `42883`, `43222`, `46173`, `48905`)
- Bug in `MultiIndex.argsort` raising ``TypeError`` when index contains `NA` (`48495`)
- Bug in `MultiIndex.difference` losing extension array dtype (`48606`)
- Bug in `MultiIndex.set_levels` raising ``IndexError`` when setting empty level (`48636`)
- Bug in `MultiIndex.unique` losing extension array dtype (`48335`)
- Bug in `MultiIndex.intersection` losing extension array (`48604`)
- Bug in `MultiIndex.union` losing extension array (`48498`, `48505`, `48900`)
- Bug in `MultiIndex.union` not sorting when sort=None and index contains missing values (`49010`)
- Bug in `MultiIndex.append` not checking names for equality (`48288`)
- Bug in `MultiIndex.symmetric_difference` losing extension array (`48607`)
- Bug in `MultiIndex.join` losing dtypes when `MultiIndex` has duplicates (`49830`)
- Bug in `MultiIndex.putmask` losing extension array (`49830`)

- Bug in `MultiIndex.value_counts` returning a `Series` indexed by flat index of tuples instead of a `MultiIndex` (`49558`)

I/O
^^^
- Bug in `read_sas` caused fragmentation of `DataFrame` and raised `.errors.PerformanceWarning` (`48595`)
- Improved error message in `read_excel` by including the offending sheet name when an exception is raised while reading a file (`48706`)
- Bug when a pickling a subset PyArrow-backed data that would serialize the entire data instead of the subset (`42600`)
- Bug in `read_sql_query` ignoring ``dtype`` argument when ``chunksize`` is specified and result is empty (`50245`)
- Bug in `read_csv` for a single-line csv with fewer columns than ``names`` raised `.errors.ParserError` with ``engine="c"`` (`47566`)
- Bug in `read_json` raising with ``orient="table"`` and ``NA`` value (`40255`)
- Bug in displaying ``string`` dtypes not showing storage option (`50099`)
- Bug in `DataFrame.to_string` with ``header=False`` that printed the index name on the same line as the first row of the data (`49230`)
- Bug in `DataFrame.to_string` ignoring float formatter for extension arrays (`39336`)
- Fixed memory leak which stemmed from the initialization of the internal JSON module (`49222`)
- Fixed issue where `json_normalize` would incorrectly remove leading characters from column names that matched the ``sep`` argument (`49861`)
- Bug in `read_csv` unnecessarily overflowing for extension array dtype when containing ``NA`` (`32134`)
- Bug in `DataFrame.to_dict` not converting ``NA`` to ``None`` (`50795`)
- Bug in `DataFrame.to_json` where it would segfault when failing to encode a string (`50307`)
- Bug in `read_xml` where file-like objects failed when iterparse is used (`50641`)
- Bug in `read_xml` ignored repeated elements when iterparse is used (`51183`)  Period
^^^
- Bug in `Period.strftime` and `PeriodIndex.strftime`, raising ``UnicodeDecodeError`` when a locale-specific directive was passed (`46319`)
- Bug in adding a `Period` object to an array of `DateOffset` objects incorrectly raising ``TypeError`` (`50162`)
- Bug in `Period` where passing a string with finer resolution than nanosecond would result in a ``KeyError`` instead of dropping the extra precision (`50417`)
- Bug in parsing strings representing Week-periods e.g. "2017-01-23/2017-01-29" as minute-frequency instead of week-frequency (`50803`)
- Bug in `.GroupBy.sum`, `.GroupBy.cumsum`, `.GroupBy.prod`, `.GroupBy.cumprod` with `PeriodDtype` failing to raise ``TypeError`` (`51040`)

- Bug in parsing empty string with `Period` incorrectly raising ``ValueError`` instead of returning ``NaT`` (`51349`)

Plotting
^^^
- Bug in `DataFrame.plot.hist`, not dropping elements of ``weights`` corresponding to ``NaN`` values in ``data`` (`48884`)

- ``ax.set_xlim`` was sometimes raising ``UserWarning`` which users couldn't address due to ``set_xlim`` not accepting parsing arguments - the converter now uses `Timestamp` instead (`49148`)

Groupby/resample/rolling
^^^
- Bug in `.ExponentialMovingWindow` with ``online`` not raising a ``NotImplementedError`` for unsupported operations (`48834`)
- Bug in `.DataFrameGroupBy.sample` raises ``ValueError`` when the object is empty (`48459`)
- Bug in `Series.groupby` raises ``ValueError`` when an entry of the index is equal to the name of the index (`48567`)
- Bug in `.DataFrameGroupBy.resample` produces inconsistent results when passing empty DataFrame (`47705`)
- Bug in `.DataFrameGroupBy` and `.SeriesGroupBy` would not include unobserved categories in result when grouping by categorical indexes (`49354`)
- Bug in `.DataFrameGroupBy` and `.SeriesGroupBy` would change result order depending on the input index when grouping by categoricals (`49223`)
- Bug in `.DataFrameGroupBy` and `.SeriesGroupBy` when grouping on categorical data would sort result values even when used with ``sort=False`` (`42482`)
- Bug in `.DataFrameGroupBy.apply` and `.SeriesGroupBy.apply` with ``as_index=False`` would not attempt the computation without using the grouping keys when using them failed with a ``TypeError`` (`49256`)
- Bug in `.DataFrameGroupBy.describe` would describe the group keys (`49256`)
- Bug in `.SeriesGroupBy.describe` with ``as_index=False`` would have the incorrect shape (`49256`)
- Bug in `.DataFrameGroupBy` and `.SeriesGroupBy` with ``dropna=False`` would drop NA values when the grouper was categorical (`36327`)
- Bug in `.SeriesGroupBy.nunique` would incorrectly raise when the grouper was an empty categorical and ``observed=True`` (`21334`)
- Bug in `.SeriesGroupBy.nth` would raise when grouper contained NA values after subsetting from a `DataFrameGroupBy` (`26454`)
- Bug in `DataFrame.groupby` would not include a `.Grouper` specified by ``key`` in the result when ``as_index=False`` (`50413`)
- Bug in `.DataFrameGroupBy.value_counts` would raise when used with a `.TimeGrouper` (`50486`)
- Bug in `.Resampler.size` caused a wide `DataFrame` to be returned instead of a `Series` with `MultiIndex` (`46826`)
- Bug in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` would raise incorrectly when grouper had ``axis=1`` for ``"idxmin"`` and ``"idxmax"`` arguments (`45986`)
- Bug in `.DataFrameGroupBy` would raise when used with an empty DataFrame, categorical grouper, and ``dropna=False`` (`50634`)
- Bug in `.SeriesGroupBy.value_counts` did not respect ``sort=False`` (`50482`)
- Bug in `.DataFrameGroupBy.resample` raises ``KeyError`` when getting the result from a key list when resampling on time index (`50840`)
- Bug in `.DataFrameGroupBy.transform` and `.SeriesGroupBy.transform` would raise incorrectly when grouper had ``axis=1`` for ``"ngroup"`` argument (`45986`)
- Bug in `.DataFrameGroupBy.describe` produced incorrect results when data had duplicate columns (`50806`)
- Bug in `.DataFrameGroupBy.agg` with ``engine="numba"`` failing to respect ``as_index=False`` (`51228`)

- Bug in `DataFrameGroupBy.agg`, `SeriesGroupBy.agg`, and `Resampler.agg` would ignore arguments when passed a list of functions (`50863`)

Reshaping
^^^
- Bug in `DataFrame.pivot_table` raising ``TypeError`` for nullable dtype and ``margins=True`` (`48681`)
- Bug in `DataFrame.unstack` and `Series.unstack` unstacking wrong level of `MultiIndex` when `MultiIndex` has mixed names (`48763`)
- Bug in `DataFrame.melt` losing extension array dtype (`41570`)
- Bug in `DataFrame.pivot` not respecting ``None`` as column name (`48293`)
- Bug in `DataFrame.join` when ``left_on`` or ``right_on`` is or includes a `CategoricalIndex` incorrectly raising ``AttributeError`` (`48464`)
- Bug in `DataFrame.pivot_table` raising ``ValueError`` with parameter ``margins=True`` when result is an empty `DataFrame` (`49240`)
- Clarified error message in `merge` when passing invalid ``validate`` option (`49417`)
- Bug in `DataFrame.explode` raising ``ValueError`` on multiple columns with ``NaN`` values or empty lists (`46084`)
- Bug in `DataFrame.transpose` with ``IntervalDtype`` column with ``timedelta64[ns]`` endpoints (`44917`)

- Bug in `DataFrame.agg` and `Series.agg` would ignore arguments when passed a list of functions (`50863`)

Sparse
^^^
- Bug in `Series.astype` when converting a ``SparseDtype`` with ``datetime64[ns]`` subtype to ``int64`` dtype raising, inconsistent with the non-sparse behavior (`49631`,`50087`)
- Bug in `Series.astype` when converting a from ``datetime64[ns]`` to ``Sparse[datetime64[ns]]`` incorrectly raising (`50082`)
- Bug in `Series.sparse.to_coo` raising ``SystemError`` when `MultiIndex` contains a ``ExtensionArray`` (`50996`)  ExtensionArray
^^^
- Bug in `Series.mean` overflowing unnecessarily with nullable integers (`48378`)
- Bug in `Series.tolist` for nullable dtypes returning numpy scalars instead of python scalars (`49890`)
- Bug in `Series.round` for pyarrow-backed dtypes raising ``AttributeError`` (`50437`)
- Bug when concatenating an empty DataFrame with an ExtensionDtype to another DataFrame with the same ExtensionDtype, the resulting dtype turned into object (`48510`)
- Bug in `array.PandasArray.to_numpy` raising with ``NA`` value when ``na_value`` is specified (`40638`)
- Bug in `api.types.is_numeric_dtype` where a custom `ExtensionDtype` would not return ``True`` if ``_is_numeric`` returned ``True`` (`50563`)
- Bug in `api.types.is_integer_dtype`, `api.types.is_unsigned_integer_dtype`, `api.types.is_signed_integer_dtype`, `api.types.is_float_dtype` where a custom `ExtensionDtype` would not return ``True`` if ``kind`` returned the corresponding NumPy type (`50667`)
- Bug in `Series` constructor unnecessarily overflowing for nullable unsigned integer dtypes (`38798`, `25880`)
- Bug in setting non-string value into ``StringArray`` raising ``ValueError`` instead of ``TypeError`` (`49632`)
- Bug in `DataFrame.reindex` not honoring the default ``copy=True`` keyword in case of columns with ExtensionDtype (and as a result also selecting multiple columns with getitem (``[]``) didn't correctly result in a copy) (`51197`)  Styler
^^^

- Fix `~pandas.io.formats.style.Styler.background_gradient` for nullable dtype `Series` with ``NA`` values (`50712`)

Metadata
^^^

- Fixed metadata propagation in `DataFrame.corr` and `DataFrame.cov` (`28283`)

Other
^^^
- Bug in `Series.searchsorted` inconsistent behavior when accepting `DataFrame` as parameter ``value`` (`49620`)

- Bug in `array` failing to raise on `DataFrame` inputs (`51167`)

.. ***DO NOT USE THIS SECTION***

sphinxext

This directory contains custom sphinx extensions in use in the pandas
documentation.