Scikit-learn governance and decision-making
*******************************************

The purpose of this document is to formalize the governance process
used by the scikit-learn project, to clarify how decisions are made
and how the various elements of our community interact. This document
establishes a decision-making structure that takes into account
feedback from all members of the community and strives to find
consensus, while avoiding any deadlocks.

This is a meritocratic, consensus-based community project. Anyone with
an interest in the project can join the community, contribute to the
project design and participate in the decision making process. This
document describes how that participation takes place and how to set
about earning merit within the project community.

Roles And Responsibilities
==========================

Contributors
------------

Contributors are community members who contribute in concrete ways to
the project. Anyone can become a contributor, and contributions can
take many forms – not only code – as detailed in the contributors
guide.

Contributor Experience Team
---------------------------

The contributor experience team is composed of community members who
have permission on github to label and close issues. Their work is
crucial to improve the communication in the project and limit the
crowding of the issue tracker.

Similarly to what has been decided in the python project, any
contributor may become a member of the scikit-learn contributor

experience team, after showing some continuity in participating to
scikit-learn development (with pull requests and reviews). Any core
developer or member of the contributor experience team is welcome to
propose a scikit-learn contributor to join the contributor experience
team. Other core developers are then consulted: while it is expected
that most acceptances will be unanimous, a two-thirds majority is
enough. Every new member of the contributor experience team will be
announced in the mailing list. Members of the team are welcome to
participate in monthly core developer meetings.

Communication team
------------------

Members of the communication team help with outreach and communication
for scikit-learn. The goal of the team is to develop public awareness
of scikit-learn, of its features and usage, as well as branding.

For this, they can operate the scikit-learn accounts on various social
networks and produce materials.

Every new communicator will be announced in the mailing list.
Communicators are welcome to participate in monthly core developer
meetings.

Core developers
---------------

Core developers are community members who have shown that they are
dedicated to the continued development of the project through ongoing
engagement with the community. They have shown they can be trusted to

maintain scikit-learn with care. Being a core developer allows
contributors to more easily carry on with their project related
activities by giving them direct access to the project's repository
and is represented as being an organization member on the scikit-learn
GitHub organization. Core developers are expected to review code
contributions, can merge approved pull requests, can cast votes for
and against merging a pull-request, and can be involved in deciding
major changes to the API.

New core developers can be nominated by any existing core developers.
Once they have been nominated, there will be a vote by the current
core developers. Voting on new core developers is one of the few
activities that takes place on the project's private management list.
While it is expected that most votes will be unanimous, a two-thirds
majority of the cast votes is enough. The vote needs to be open for at
least 1 week.

Core developers that have not contributed to the project (commits or
GitHub comments) in the past 12 months will be asked if they want to
become emeritus core developers and recant their commit and voting
rights until they become active again. The list of core developers,
active and emeritus (with dates at which they became active) is public
on the scikit-learn website.

Technical Committee
-------------------

The Technical Committee (TC) members are core developers who have
additional responsibilities to ensure the smooth running of the
project. TC members are expected to participate in strategic planning,
and approve changes to the governance model. The purpose of the TC is
to ensure a smooth progress from the big-picture perspective. Indeed
changes that impact the full project require a synthetic analysis and
a consensus that is both explicit and informed. In cases that the core
developer community (which includes the TC members) fails to reach
such a consensus in the required time frame, the TC is the entity to
resolve the issue. Membership of the TC is by nomination by a core
developer. A nomination will result in discussion which cannot take
more than a month and then a vote by the core developers which will
stay open for a week. TC membership votes are subject to a two-third
majority of all cast votes as well as a simple majority approval of
all the current TC members. TC members who do not actively engage with
the TC duties are expected to resign.

The Technical Committee of scikit-learn consists of >>:user:`Thomas
Fan <thomasjpfan>`<<, >>:user:`Alexandre Gramfort <agramfort>`<<,
>>:user:`Olivier Grisel <ogrisel>`<<, >>:user:`Adrin Jalali
<adrinjalali>`<<, >>:user:`Andreas Müller <amueller>`<<, >>:user:`Joel

Nothman <jnothman>`<< and >>:user:`Gaël Varoquaux <GaelVaroquaux>`<<.

Decision Making Process
=======================

Decisions about the future of the project are made through discussion
with all members of the community. All non-sensitive project
management discussion takes place on the project contributors' mailing
list and the issue tracker. Occasionally, sensitive discussion occurs
on a private list.

Scikit-learn uses a "consensus seeking" process for making decisions.
The group tries to find a resolution that has no open objections among
core developers. At any point during the discussion, any core-
developer can call for a vote, which will conclude one month from the
call for the vote. Any vote must be backed by a SLEP. If no option can
gather two thirds of the votes cast, the decision is escalated to the
TC, which in turn will use consensus seeking with the fallback option
of a simple majority vote if no consensus can be found within a month.
This is what we hereafter may refer to as “the decision making
process”.

Decisions (in addition to adding core developers and TC membership as
above) are made according to the following rules:

* **Minor Documentation changes**, such as typo fixes, or addition /
  correction of a sentence, but no change of the scikit-learn.org

  landing page or the “about” page: Requires +1 by a core developer,
  no -1 by a core developer (lazy consensus), happens on the issue or
  pull request page. Core developers are expected to give “reasonable
  time” to others to give their opinion on the pull request if they're
  not confident others would agree.

* **Code changes and major documentation changes** require +1 by two
  core developers, no -1 by a core developer (lazy consensus), happens
  on the issue of pull-request page.

* **Changes to the API principles and changes to dependencies or
  supported versions** happen via a Enhancement proposals (SLEPs) and
  follows the decision-making process outlined above.

* **Changes to the governance model** use the same decision process
  outlined above.

If a veto -1 vote is cast on a lazy consensus, the proposer can appeal
to the community and core developers and the change can be approved or
rejected using the decision making procedure outlined above.

Enhancement proposals (SLEPs)
=============================

For all votes, a proposal must have been made public and discussed
before the vote. Such proposal must be a consolidated document, in the
form of a "Scikit-Learn Enhancement Proposal" (SLEP), rather than a
long discussion on an issue. A SLEP must be submitted as a pull-

request to enhancement proposals using the SLEP template.

5. Visualizations
*****************

Scikit-learn defines a simple API for creating visualizations for
machine learning. The key feature of this API is to allow for quick
plotting and visual adjustments without recalculation. We provide
*Display* classes that expose two methods for creating plots:
*from_estimator* and *from_predictions*. The *from_estimator* method
will take a fitted estimator and some data (*X* and *y*) and create a
*Display* object. Sometimes, we would like to only compute the
predictions once and one should use *from_predictions* instead. In the
following example, we plot a ROC curve for a fitted support vector
machine:

The returned *svc_disp* object allows us to continue using the already
computed ROC curve for SVC in future plots. In this case, the
*svc_disp* is a "RocCurveDisplay" that stores the computed values as
attributes called *roc_auc*, *fpr*, and *tpr*. Be aware that we could
get the predictions from the support vector machine and then use
*from_predictions* instead of *from_estimator*. Next, we train a
random forest classifier and plot the previously computed roc curve
again by using the *plot* method of the *Display* object.

Notice that we pass *alpha=0.8* to the plot functions to adjust the
alpha values of the curves.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_miscellaneous_plot_roc_curve_visualization_
  api.py*

* *sphx_glr_auto_examples_miscellaneous_plot_partial_dependence_visua
  lization_api.py*

Examples:
^^^^^^^^^

  api.py*

* *sphx_glr_auto_examples_miscellaneous_plot_display_object_visualiza
  tion.py*

* *sphx_glr_auto_examples_calibration_plot_compare_calibration.py*

5.1. Available Plotting Utilities
=================================

5.1.1. Display Objects
----------------------

3. Model selection and evaluation
*********************************

* 3.1. Cross-validation: evaluating estimator performance

  * 3.1.1. Computing cross-validated metrics

  * 3.1.2. Cross validation iterators

  * 3.1.3. A note on shuffling

  * 3.1.4. Cross validation and model selection

  * 3.1.5. Permutation test score

* 3.2. Tuning the hyper-parameters of an estimator

  * 3.2.1. Exhaustive Grid Search

  * 3.2.2. Randomized Parameter Optimization

  * 3.2.3. Searching for optimal parameters with successive halving

  * 3.2.4. Tips for parameter search

  * 3.2.5. Alternatives to brute force parameter search

* 3.3. Metrics and scoring: quantifying the quality of predictions

  * 3.3.1. The "scoring" parameter: defining model evaluation rules

  * 3.3.2. Classification metrics

  * 3.3.3. Multilabel ranking metrics

  * 3.3.4. Regression metrics

  * 3.3.5. Clustering metrics

  * 3.3.6. Dummy estimators

* 3.4. Validation curves: plotting scores to evaluate models

  * 3.4.1. Validation curve

  * 3.4.2. Learning curve

11. Dispatching
***************

* 11.1. Array API support (experimental)

  * 11.1.1. Example usage

  * 11.1.2. Estimators with support for *Array API*-compatible inputs

9. Model persistence
********************

After training a scikit-learn model, it is desirable to have a way to
persist the model for future use without having to retrain. The
following sections give you some hints on how to persist a scikit-
learn model.

9.1. Python specific serialization
==================================

It is possible to save a model in scikit-learn by using Python's
built-in persistence model, namely pickle:

   >>> from sklearn import svm
   >>> from sklearn import datasets
   >>> clf = svm.SVC()
   >>> X, y= datasets.load_iris(return_X_y=True)
   >>> clf.fit(X, y)
   SVC()

   >>> import pickle
   >>> s = pickle.dumps(clf)
   >>> clf2 = pickle.loads(s)
   >>> clf2.predict(X[0:1])
   array([0])
   >>> y[0]
   0

In the specific case of scikit-learn, it may be better to use joblib's
replacement of pickle ("dump" & "load"), which is more efficient on
objects that carry large numpy arrays internally as is often the case
for fitted scikit-learn estimators, but can only pickle to the disk
and not to a string:

   >>> from joblib import dump, load
   >>> dump(clf, 'filename.joblib') 

Later you can load back the pickled model (possibly in another Python
process) with:

   >>> clf = load('filename.joblib') 

Note:

  "dump" and "load" functions also accept file-like object instead of
  filenames. More information on data persistence with Joblib is
  available here.

When an estimator is unpickled with a scikit-learn version that is
inconsistent with the version the estimator was pickled with, a

available here.

"InconsistentVersionWarning" is raised. This warning can be caught to
obtain the original version the estimator was pickled with:

   from sklearn.exceptions import InconsistentVersionWarning
   warnings.simplefilter("error", InconsistentVersionWarning)

   try:
      est = pickle.loads("model_from_prevision_version.pickle")

   except InconsistentVersionWarning as w:
      print(w.original_sklearn_version)

9.1.1. Security & maintainability limitations
---------------------------------------------

pickle (and joblib by extension), has some issues regarding
maintainability and security. Because of this,

* Never unpickle untrusted data as it could lead to malicious code
  being executed upon loading.

* While models saved using one version of scikit-learn might load in
  other versions, this is entirely unsupported and inadvisable. It
  should also be kept in mind that operations performed on such data
  could give different and unexpected results.

In order to rebuild a similar model with future versions of scikit-
learn, additional metadata should be saved along the pickled model:

* The training data, e.g. a reference to an immutable snapshot

* The python source code used to generate the model

* The versions of scikit-learn and its dependencies

* The cross validation score obtained on the training data

This should make it possible to check that the cross-validation score
is in the same range as before.

Aside for a few exceptions, pickled models should be portable across
architectures assuming the same versions of dependencies and Python
are used. If you encounter an estimator that is not portable please
open an issue on GitHub. Pickled models are often deployed in
production using containers, like Docker, in order to freeze the
environment and dependencies.

If you want to know more about these issues and explore other possible
serialization methods, please refer to this talk by Alex Gaynor.

9.1.2. A more secure format: *skops*
------------------------------------

skops provides a more secure format via the "skops.io" module. It
avoids using "pickle" and only loads files which have types and
references to functions which are trusted either by default or by the
user. The API is very similar to "pickle", and you can persist your
models as explain in the docs using "skops.io.dump()" and
"skops.io.dumps()":

   import skops.io as sio
   obj = sio.dumps(clf)

And you can load them back using "skops.io.load()" and
"skops.io.loads()". However, you need to specify the types which are
trusted by you. You can get existing unknown types in a dumped object
/ file using "skops.io.get_untrusted_types()", and after checking its

contents, pass it to the load function:

   unknown_types = sio.get_untrusted_types(obj)
   clf = sio.loads(obj, trusted=unknown_types)

If you trust the source of the file / object, you can pass
"trusted=True":

   clf = sio.loads(obj, trusted=True)

Please report issues and feature requests related to this format on
the skops issue tracker.

9.2. Interoperable formats

For reproducibility and quality control needs, when different
architectures and environments should be taken into account, exporting
the model in Open Neural Network Exchange format or Predictive Model
Markup Language (PMML) format might be a better approach than using
*pickle* alone. These are helpful where you may want to use your model
for prediction in a different environment from where the model was
trained.

ONNX is a binary serialization of the model. It has been developed to
improve the usability of the interoperable representation of data
models. It aims to facilitate the conversion of the data models
between different machine learning frameworks, and to improve their
portability on different computing architectures. More details are
available from the ONNX tutorial. To convert scikit-learn model to
ONNX a specific tool sklearn-onnx has been developed.

PMML is an implementation of the XML document standard defined to

represent data models together with the data used to generate them.
Being human and machine readable, PMML is a good option for model
validation on different platforms and long term archiving. On the
other hand, as XML in general, its verbosity does not help in
production when performance is critical. To convert scikit-learn model
to PMML you can use for example sklearn2pmml distributed under the
Affero GPLv3 license.

* Mathieu Blondel

* Matthieu Brucher

* Lars Buitinck

* David Cournapeau

* Noel Dawe

* Vincent Dubourg

* Edouard Duchesnay

* Alexander Fabisch

* Virgile Fritsch

* Satrajit Ghosh

* Angel Soler Gollonet

* Chris Gorgolewski

* Jaques Grobler

* Brian Holt

* Arnaud Joly

* Thouis (Ray) Jones

* Kyle Kastner

* manoj kumar

* Robert Layton

* Wei Li

* Paolo Losi

* Gilles Louppe

* Vincent Michel

* Jarrod Millman

* Alexandre Passos

* Fabian Pedregosa

* Peter Prettenhofer

* (Venkat) Raghav, Rajagopalan

* Jacob Schreiber

* 杜世橋 Du Shiqiao

* Jake Vanderplas

* David Warde-Farley

* Ron Weiss

Roadmap
*******

Purpose of this document
========================

This document list general directions that core contributors are
interested to see developed in scikit-learn. The fact that an item is
listed here is in no way a promise that it will happen, as resources
are limited. Rather, it is an indication that help is welcomed on this
topic.

Statement of purpose: Scikit-learn in 2018
==========================================

Eleven years after the inception of Scikit-learn, much has changed in
the world of machine learning. Key changes include:

* Computational tools: The exploitation of GPUs, distributed
  programming frameworks like Scala/Spark, etc.

* High-level Python libraries for experimentation, processing and data
  management: Jupyter notebook, Cython, Pandas, Dask, Numba...

* Changes in the focus of machine learning research: artificial
  intelligence applications (where input structure is key) with deep
  learning, representation learning, reinforcement learning, domain
  transfer, etc.

A more subtle change over the last decade is that, due to changing
interests in ML, PhD students in machine learning are more likely to
contribute to PyTorch, Dask, etc. than to Scikit-learn, so our
contributor pool is very different to a decade ago.

Scikit-learn remains very popular in practice for trying out canonical
machine learning techniques, particularly for applications in
experimental science and in data science. A lot of what we provide is

now very mature. But it can be costly to maintain, and we cannot
therefore include arbitrary new implementations. Yet Scikit-learn is
also essential in defining an API framework for the development of
interoperable machine learning components external to the core
library.

**Thus our main goals in this era are to**:

* continue maintaining a high-quality, well-documented collection of
  canonical tools for data processing and machine learning within the
  current scope (i.e. rectangular data largely invariant to column and
  row order; predicting targets with simple structure)

* improve the ease for users to develop and publish external
  components

* improve interoperability with modern data science tools (e.g.
  Pandas, Dask) and infrastructures (e.g. distributed processing)

Many of the more fine-grained goals can be found under the API tag on
the issue tracker.

Architectural / general goals

The list is numbered not as an indication of the order of priority,
but to make referring to specific points easier. Please add new
entries only at the bottom. Note that the crossed out entries are
already done, and we try to keep the document up to date as we work on
these issues.

1. Improved handling of Pandas DataFrames

   * document current handling

   * column reordering issue >>:issue:`7242`<<

   * avoiding unnecessary conversion to ndarray  >>:issue:`12147`<<

   * returning DataFrames from transformers >>:issue:`5523`<<

   * getting DataFrames from dataset loaders  >>:issue:`10733`<< ,
     >>:issue:`13902`<<

   * Sparse currently not considered  >>:issue:`12800`<<

2. Improved handling of categorical features

   * Tree-based models should be able to handle both continuous and
     categorical features >>:issue:`12866`<< and  >>:issue:`15550`<< .

   *  In dataset loaders >>:issue:`13902`<<

   * As generic transformers to be used with ColumnTransforms (e.g.
     ordinal encoding supervised by correlation with target variable)
     >>:issue:`5853`<<, >>:issue:`11805`<<

   * Handling mixtures of categorical and continuous variables

3. Improved handling of missing data

   * Making sure meta-estimators are lenient towards missing data,
     >>:issue:`15319`<<

   * Non-trivial imputers  >>:issue:`11977`<<, >>:issue:`12852`<<

   * Learners directly handling missing data  >>:issue:`13911`<<

   * An amputation sample generator to make parts of a dataset go
     missing >>:issue:`6284`<<

4. More didactic documentation

   * More and more options have been added to scikit-learn. As a
     result, the documentation is crowded which makes it hard for

missing >>:issue:`6284`<<

     beginners to get the big picture. Some work could be done in
     prioritizing the information.

5. Passing around information that is not (X, y): Sample properties

   * We need to be able to pass sample weights to scorers in cross
     validation.

   * We should have standard/generalised ways of passing sample-wise
     properties around in meta-estimators. >>:issue:`4497`<<
     >>:issue:`7646`<<

6. Passing around information that is not (X, y): Feature properties

   * Feature names or descriptions should ideally be available to fit
     for, e.g. . >>:issue:`6425`<< >>:issue:`6424`<<

   * Per-feature handling (e.g. "is this a nominal / ordinal / English
     language text?") should also not need to be provided to estimator
     constructors, ideally, but should be available as metadata
     alongside X. >>:issue:`8480`<<

7. Passing around information that is not (X, y): Target information

   * We have problems getting the full set of classes to all
     components when the data is split/sampled. >>:issue:`6231`<<
     >>:issue:`8100`<<

   * We have no way to handle a mixture of categorical and continuous
     targets.

8. Make it easier for external users to write Scikit-learn-compatible
   components

   * More flexible estimator checks that do not select by estimator

targets.

     name  >>:issue:`6599`<<  >>:issue:`6715`<<

   * Example of how to develop an estimator or a meta-estimator,
     >>:issue:`14582`<<

   * More self-sufficient running of scikit-learn-contrib or a similar
     resource

9. Support resampling and sample reduction

   * Allow subsampling of majority classes (in a pipeline?)
     >>:issue:`3855`<<

   * Implement random forests with resampling >>:issue:`13227`<<

10. Better interfaces for interactive development

    *  __repr__ and HTML visualisations of estimators
      >>:issue:`6323`<< and >>:pr:`14180`<< .

    * Include plotting tools, not just as examples. >>:issue:`9173`<<

11. Improved tools for model diagnostics and basic inference

    *  alternative feature importances implementations,
      >>:issue:`13146`<<

    * better ways to handle validation sets when fitting

    * better ways to find thresholds / create decision rules
      >>:issue:`8614`<<

12. Better tools for selecting hyperparameters with transductive
    estimators

    * Grid search and cross validation are not applicable to most
      clustering tasks. Stability-based selection is more relevant.

13. Better support for manual and automatic pipeline building

    * Easier way to construct complex pipelines and valid search
      spaces >>:issue:`7608`<< >>:issue:`5082`<< >>:issue:`8243`<<

    * provide search ranges for common estimators??

    * cf. searchgrid

14. Improved tracking of fitting

    * Verbose is not very friendly and should use a standard logging
      library >>:issue:`6929`<<, >>:issue:`78`<<

    * Callbacks or a similar system would facilitate logging and early
      stopping

15. Distributed parallelism

    * Accept data which complies with "__array_function__"

16. A way forward for more out of core

    * Dask enables easy out-of-core computation. While the Dask model
      probably cannot be adaptable to all machine-learning algorithms,
      most machine learning is on smaller data than ETL, hence we can
      maybe adapt to very large scale while supporting only a fraction
      of the patterns.

17. Support for working with pre-trained models

    * Estimator "freezing". In particular, right now it's impossible
      to clone a *CalibratedClassifierCV* with prefit.
      >>:issue:`8370`<<. >>:issue:`6451`<<

18. Backwards-compatible de/serialization of some estimators

    * Currently serialization (with pickle) breaks across versions.
      While we may not be able to get around other limitations of
      pickle re security etc, it would be great to offer cross-version
      safety from version 1.0. Note: Gael and Olivier think that this

pickle re security etc, it would be great to offer cross-version
      can cause heavy maintenance burden and we should manage the
      trade-offs. A possible alternative is presented in the following
      point.

19. Documentation and tooling for model lifecycle management

    * Document good practices for model deployments and lifecycle:
      before deploying a model: snapshot the code versions (numpy,
      scipy, scikit-learn, custom code repo), the training script and
      an alias on how to retrieve historical training data + snapshot
      a copy of a small validation set + snapshot of the predictions
      (predicted probabilities for classifiers) on that validation
      set.

    * Document and tools to make it easy to manage upgrade of scikit-
      learn versions:

      * Try to load the old pickle, if it works, use the validation
        set prediction snapshot to detect that the serialized model
        still behave the same;

      * If joblib.load / pickle.load not work, use the versioned
        control training script + historical training set to retrain
        the model and use the validation set prediction snapshot to
        assert that it is possible to recover the previous predictive
        performance: if this is not the case there is probably a bug
        in scikit-learn that needs to be reported.

20. Everything in Scikit-learn should probably conform to our API

performance: if this is not the case there is probably a bug

    contract. We are still in the process of making decisions on some
    of these related issues.

    * *Pipeline <pipeline.Pipeline>* and *FeatureUnion* modify their
      input parameters in fit. Fixing this requires making sure we
      have a good grasp of their use cases to make sure all current
      functionality is maintained. >>:issue:`8157`<< >>:issue:`7382`<<

21. (Optional) Improve scikit-learn common tests suite to make sure
    that (at least for frequently used) models have stable predictions
    across-versions (to be discussed);

    * Extend documentation to mention how to deploy models in Python-
      free environments for instance ONNX. and use the above best
      practices to assess predictive consistency between scikit-learn
      and ONNX prediction functions on validation set.

    * Document good practices to detect temporal distribution drift
      for deployed model and good practices for re-training on fresh
      data without causing catastrophic predictive performance
      regressions.

Subpackage-specific goals
=========================

"sklearn.ensemble"

*  a stacking implementation, >>:issue:`11047`<<

"sklearn.cluster"

* kmeans variants for non-Euclidean distances, if we can show these
  have benefits beyond hierarchical clustering.

"sklearn.model_selection"

*  multi-metric scoring is slow >>:issue:`9326`<<

* perhaps we want to be able to get back more than multiple metrics

* the handling of random states in CV splitters is a poor design and
  contradicts the validation of similar parameters in estimators,
  SLEP011

* exploit warm-starting and path algorithms so the benefits of
  *EstimatorCV* objects can be accessed via *GridSearchCV* and used in
  Pipelines. >>:issue:`1626`<<

* Cross-validation should be able to be replaced by OOB estimates
  whenever a cross-validation iterator is used.

* Redundant computations in pipelines should be avoided (related to
  point above) cf dask-ml

"sklearn.neighbors"

*  Ability to substitute a custom/approximate/precomputed nearest
  neighbors implementation for ours in all/most contexts that nearest
  neighbors are used for learning. >>:issue:`10463`<<

"sklearn.pipeline"

* Performance issues with *Pipeline.memory*

* see "Everything in Scikit-learn should conform to our API contract"
  above

Support
*******

There are several ways to get in touch with the developers.

Mailing List
============

* The main mailing list is scikit-learn.

* There is also a commit list scikit-learn-commits, where updates to
  the main repository and test failures get notified.

User questions
==============

* Some scikit-learn developers support users on StackOverflow using
  the [scikit-learn] tag.

* For general theoretical or methodological Machine Learning questions
  stack exchange is probably a more suitable venue.

In both cases please use a descriptive question in the title field
(e.g. no "Please help with scikit-learn!" as this is not a question)
and put details on what you tried to achieve, what were the expected
results and what you observed instead in the details field.

Code and data snippets are welcome. Minimalistic (up to ~20 lines
long) reproduction script very helpful.

Please describe the nature of your data and how you preprocessed it:
what is the number of samples, what is the number and type of features
(i.d. categorical or numerical) and for supervised learning tasks,
what target are your trying to predict: binary, multiclass (1 out of
"n_classes") or multilabel ("k" out of "n_classes") classification or
continuous variable regression.

User questions should **not be asked on the bug tracker**, as it
crowds the list of issues and makes the development of the project
harder.

Bug tracker
===========

If you think you've encountered a bug, please report it to the issue
tracker:

harder.

tracker:

https://github.com/scikit-learn/scikit-learn/issues

Don't forget to include:

   * steps (or better script) to reproduce,

   * expected outcome,

   * observed outcome or Python (or gdb) tracebacks

To help developers fix your bug faster, please link to a
https://gist.github.com holding a standalone minimalistic python
script that reproduces your bug and optionally a minimalistic
subsample of your dataset (for instance, exported as CSV files using
"numpy.savetxt").

Note: Gists are Git cloneable repositories and thus you can use Git to
push datafiles to them.

Gitter
======

Some developers like to hang out on scikit-learn Gitter room:
https://gitter.im/scikit-learn/scikit-learn.

Documentation resources

This documentation is relative to . Documentation for other versions
can be found here.

Printable pdf documentation for old versions can be found here.

4. Inspection
*************

Predictive performance is often the main goal of developing machine
learning models. Yet summarizing performance with an evaluation metric
is often insufficient: it assumes that the evaluation metric and test
dataset perfectly reflect the target domain, which is rarely true. In
certain domains, a model needs a certain level of interpretability
before it can be deployed. A model that is exhibiting performance
issues needs to be debugged for one to understand the model's
underlying issue. The "sklearn.inspection" module provides tools to
help understand the predictions from a model and what affects them.
This can be used to evaluate assumptions and biases of a model, design
a better model, or to diagnose issues with model performance.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_inspection_plot_linear_model_coefficient_in
  terpretation.py*

* 4.1. Partial Dependence and Individual Conditional Expectation plots

  * 4.1.1. Partial dependence plots

  * 4.1.2. Individual conditional expectation (ICE) plot

  * 4.1.3. Mathematical Definition

  * 4.1.4. Computation methods

* 4.2. Permutation feature importance

  * 4.2.1. Outline of the permutation importance algorithm

  * 4.2.2. Relation to impurity-based importance in trees

  * 4.2.3. Misleading values on strongly correlated features

Table Of Contents

* Welcome to scikit-learn

  * Installing scikit-learn

  * Frequently Asked Questions

  * Support

  * Related Projects

  * About us

  * Who is using scikit-learn?

  * Release History

  * Roadmap

  * Scikit-learn governance and decision-making

* scikit-learn Tutorials

  * An introduction to machine learning with scikit-learn

  * A tutorial on statistical-learning for scientific data processing

  * Working With Text Data

  * Choosing the right estimator

  * External Resources, Videos and Talks

* Getting Started

  * Fitting and predicting: estimator basics

  * Transformers and pre-processors

  * Pipelines: chaining pre-processors and estimators

  * Model evaluation

  * Automatic parameter searches

  * Next steps

* User Guide

  * 1. Supervised learning

  * 2. Unsupervised learning

  * 3. Model selection and evaluation

  * 4. Inspection

  * 5. Visualizations

  * 6. Dataset transformations

  * 7. Dataset loading utilities

  * 8. Computing with scikit-learn

  * 9. Model persistence

  * 10. Common pitfalls and recommended practices

  * 11. Dispatching

* Glossary of Common Terms and API Elements

  * General Concepts

  * Class APIs and Estimator Types

  * Target Types

  * Methods

  * Parameters

  * Attributes

  * Data and sample properties

* API Reference

  * "sklearn.base": Base classes and utility functions

  * "sklearn.calibration": Probability Calibration

  * "sklearn.cluster": Clustering

  * "sklearn.compose": Composite Estimators

  * "sklearn.covariance": Covariance Estimators

* "sklearn.calibration": Probability Calibration

  * "sklearn.cross_decomposition": Cross decomposition

  * "sklearn.datasets": Datasets

  * "sklearn.decomposition": Matrix Decomposition

  * "sklearn.discriminant_analysis": Discriminant Analysis

  * "sklearn.dummy": Dummy estimators

  * "sklearn.ensemble": Ensemble Methods

  * "sklearn.exceptions": Exceptions and warnings

  * "sklearn.experimental": Experimental

  * "sklearn.feature_extraction": Feature Extraction

  * "sklearn.feature_selection": Feature Selection

  * "sklearn.gaussian_process": Gaussian Processes

  * "sklearn.impute": Impute

  * "sklearn.inspection": Inspection

  * "sklearn.isotonic": Isotonic regression

  * "sklearn.kernel_approximation": Kernel Approximation

  * "sklearn.kernel_ridge": Kernel Ridge Regression

  * "sklearn.linear_model": Linear Models

  * "sklearn.manifold": Manifold Learning

  * "sklearn.metrics": Metrics

  * "sklearn.mixture": Gaussian Mixture Models

  * "sklearn.model_selection": Model Selection

  * "sklearn.multiclass": Multiclass classification

  * "sklearn.multioutput": Multioutput regression and classification

  * "sklearn.naive_bayes": Naive Bayes

  * "sklearn.neighbors": Nearest Neighbors

  * "sklearn.neural_network": Neural network models

  * "sklearn.pipeline": Pipeline

  * "sklearn.preprocessing": Preprocessing and Normalization

  * "sklearn.random_projection": Random projection

* "sklearn.neural_network": Neural network models

  * "sklearn.semi_supervised": Semi-Supervised Learning

  * "sklearn.svm": Support Vector Machines

  * "sklearn.tree": Decision Trees

  * "sklearn.utils": Utilities

  * Recently deprecated

* Developer's Guide

  * Contributing

  * Crafting a minimal reproducer for scikit-learn

  * Developing scikit-learn estimators

  * Developers' Tips and Tricks

  * Utilities for Developers

  * How to optimize for speed

  * Installing the development version of scikit-learn

  * Bug triaging and issue curation

  * Maintainer / core-developer information

  * Developing with the Plotting API

Installing scikit-learn
***********************

There are different ways to install scikit-learn:

   * Install the latest official release. This is the best approach
     for most users. It will provide a stable version and pre-built
     packages are available for most platforms.

   * Install the version of scikit-learn provided by your operating
     system or Python distribution. This is a quick option for those
     who have operating systems or Python distributions that
     distribute scikit-learn. It might not provide the latest release
     version.

   * Building the package from source. This is best for users who want
     the latest-and-greatest features and aren't afraid of running
     brand-new code. This is also needed for users who wish to
     contribute to the project.

Installing the latest release

Then run:

In order to check your installation you can use

Note that in order to avoid potential conflicts with other packages it
is strongly recommended to use a virtual environment (venv) or a conda
environment.

Using such an isolated environment makes it possible to install a
specific version of scikit-learn with pip or conda and its
dependencies independently of any previously installed Python
packages. In particular under Linux is it discouraged to install pip
packages alongside the packages managed by the package manager of the
distribution (apt, dnf, pacman...).

Note that you should always remember to activate the environment of

your choice prior to running any Python command whenever you start a
new terminal session.

If you have not installed NumPy or SciPy yet, you can also install
these using conda or pip. When using pip, please ensure that *binary
wheels* are used, and NumPy and SciPy are not recompiled from source,
which can happen when using particular configurations of operating
system and hardware (such as Linux on a Raspberry Pi).

Scikit-learn plotting capabilities (i.e., functions start with "plot_"
and classes end with "Display") require Matplotlib. The examples
require Matplotlib and some examples require scikit-image, pandas, or
seaborn. The minimum version of Scikit-learn dependencies are listed
below along with its purpose.

Warning:

  Scikit-learn 0.20 was the last version to support Python 2.7 and
  Python 3.4. Scikit-learn 0.21 supported Python 3.5-3.7. Scikit-learn
  0.22 supported Python 3.5-3.8. Scikit-learn 0.23 - 0.24 require
  Python 3.6 or newer. Scikit-learn 1.0 supported Python 3.7-3.10.
  Scikit-learn 1.1 and later requires Python 3.8 or newer.

Note:

  For installing on PyPy, PyPy3-v5.10+, Numpy 1.14.0+, and scipy
  1.1.0+ are required.

Installing on Apple Silicon M1 hardware
=======================================

The recently introduced *macos/arm64* platform (sometimes also known

1.1.0+ are required.

as *macos/aarch64*) requires the open source community to upgrade the
build configuration and automation to properly support it.

At the time of writing (January 2021), the only way to get a working
installation of scikit-learn on this hardware is to install scikit-
learn and its dependencies from the conda-forge distribution, for
instance using the miniforge installers:

https://github.com/conda-forge/miniforge

The following issue tracks progress on making it possible to install
scikit-learn from PyPI with pip:

https://github.com/scikit-learn/scikit-learn/issues/19137

Third party distributions of scikit-learn
=========================================

Some third-party distributions provide versions of scikit-learn
integrated with their package-management systems.

These can make installation and upgrading much easier for users since
the integration includes the ability to automatically install
dependencies (numpy, scipy) that scikit-learn requires.

The following is an incomplete list of OS and python distributions
that provide their own version of scikit-learn.

Alpine Linux

Alpine Linux's package is provided through the official repositories
as "py3-scikit-learn" for Python. It can be installed by typing the
following command:

Arch Linux
----------

Arch Linux's package is provided through the official repositories as

Arch Linux

"python-scikit-learn" for Python. It can be installed by typing the

Debian/Ubuntu
-------------

The Debian/Ubuntu package is split in three different packages called
"python3-sklearn" (python modules), "python3-sklearn-lib" (low-level
implementations and bindings), "python3-sklearn-doc" (documentation).
Only the Python 3 version is available in the Debian Buster (the more
recent Debian distribution). Packages can be installed using "apt-
get":

Fedora
------

The Fedora package is called "python3-scikit-learn" for the python 3
version, the only one available in Fedora30. It can be installed using
"dnf":

NetBSD
------

scikit-learn is available via pkgsrc-wip:

   https://pkgsrc.se/math/py-scikit-learn

MacPorts for Mac OSX
--------------------

The MacPorts package is named "py<XY>-scikits-learn", where "XY"
denotes the Python version. It can be installed by typing the

Anaconda and Enthought Deployment Manager for all supported platforms
---------------------------------------------------------------------

Anaconda and Enthought Deployment Manager both ship with scikit-learn
in addition to a large set of scientific python library for Windows,
Mac OSX and Linux.

Anaconda offers scikit-learn as part of its free distribution.

Intel conda channel

Intel maintains a dedicated conda channel that ships scikit-learn:

This version of scikit-learn comes with alternative solvers for some
common estimators. Those solvers come from the DAAL C++ library and
are optimized for multi-core Intel CPUs.

Note that those solvers are not enabled by default, please refer to
the daal4py documentation for more details.

Compatibility with the standard scikit-learn solvers is checked by
running the full scikit-learn test suite via automated continuous
integration as reported on https://github.com/IntelPython/daal4py.

WinPython for Windows
---------------------

The WinPython project distributes scikit-learn as an additional
plugin.

Troubleshooting
===============

Error caused by file path length limit on Windows
-------------------------------------------------

It can happen that pip fails to install packages when reaching the
default path size limit of Windows if Python is installed in a nested
location such as the *AppData* folder structure under the user home
directory, for instance:

   C:\Users\username>C:\Users\username\AppData\Local\Microsoft\WindowsApps\python.exe -m pip install scikit-learn
   Collecting scikit-learn
   ...
   Installing collected packages: scikit-learn

C:\Users\username>C:\Users\username\AppData\Local\Microsoft\WindowsApps\python.exe -m pip install scikit-learn
   ...
   ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: 'C:\\Users\\username\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python37\\site-packages\\sklearn\\datasets\\tests\\data\\openml\\292\\api-v1-json-data-list-data_name-australian-limit-2-data_version-1-status-deactivated.json.gz'

In this case it is possible to lift that limit in the Windows registry
by using the "regedit" tool:

1. Type "regedit" in the Windows start menu to launch "regedit".

2. Go to the "Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Co
   ntrol\FileSystem" key.

3. Edit the value of the "LongPathsEnabled" property of that key and
   set it to 1.

4. Reinstall scikit-learn (ignoring the previous broken installation):

Getting Started

The purpose of this guide is to illustrate some of the main features
that "scikit-learn" provides. It assumes a very basic working
knowledge of machine learning practices (model fitting, predicting,
cross-validation, etc.). Please refer to our installation instructions
for installing "scikit-learn".

"Scikit-learn" is an open source machine learning library that
supports supervised and unsupervised learning. It also provides
various tools for model fitting, data preprocessing, model selection,
model evaluation, and many other utilities.

Fitting and predicting: estimator basics
========================================

"Scikit-learn" provides dozens of built-in machine learning algorithms
and models, called *estimators*. Each estimator can be fitted to some
data using its *fit* method.

Here is a simple example where we fit a "RandomForestClassifier" to
some very basic data:

   >>> from sklearn.ensemble import RandomForestClassifier
   >>> clf = RandomForestClassifier(random_state=0)
   >>> X = [[ 1,  2,  3],  # 2 samples, 3 features
   ...      [11, 12, 13]]
   >>> y = [0, 1]  # classes of each sample
   RandomForestClassifier(random_state=0)

The *fit* method generally accepts 2 inputs:

* The samples matrix (or design matrix) *X*. The size of "X" is
  typically "(n_samples, n_features)", which means that samples are
  represented as rows and features are represented as columns.

* The target values *y* which are real numbers for regression tasks,

typically "(n_samples, n_features)", which means that samples are

  or integers for classification (or any other discrete set of
  values). For unsupervized learning tasks, "y" does not need to be
  specified. "y" is usually 1d array where the "i" th entry
  corresponds to the target of the "i" th sample (row) of "X".

Both "X" and "y" are usually expected to be numpy arrays or equivalent
*array-like* data types, though some estimators work with other
formats such as sparse matrices.

Once the estimator is fitted, it can be used for predicting target
values of new data. You don't need to re-train the estimator:

   >>> clf.predict(X)  # predict classes of the training data
   array([0, 1])
   >>> clf.predict([[4, 5, 6], [14, 15, 16]])  # predict classes of new data

Transformers and pre-processors
===============================

Machine learning workflows are often composed of different parts. A
typical pipeline consists of a pre-processing step that transforms or
imputes the data, and a final predictor that predicts target values.

In "scikit-learn", pre-processors and transformers follow the same API
as the estimator objects (they actually all inherit from the same
"BaseEstimator" class). The transformer objects don't have a *predict*
method but rather a *transform* method that outputs a newly
transformed sample matrix "X":

   >>> from sklearn.preprocessing import StandardScaler
   >>> X = [[0, 15],
   ...      [1, -10]]
   >>> # scale data according to computed scaling values
   >>> StandardScaler().fit(X).transform(X)
   array([[-1.,  1.],
          [ 1., -1.]])

Sometimes, you want to apply different transformations to different
features: the ColumnTransformer is designed for these use-cases.

Pipelines: chaining pre-processors and estimators
=================================================

Transformers and estimators (predictors) can be combined together into
a single unifying object: a "Pipeline". The pipeline offers the same
API as a regular estimator: it can be fitted and used for prediction
with "fit" and "predict". As we will see later, using a pipeline will
also prevent you from data leakage, i.e. disclosing some testing data
in your training data.

In the following example, we load the Iris dataset, split it into
train and test sets, and compute the accuracy score of a pipeline on
the test data:

   >>> from sklearn.linear_model import LogisticRegression
   >>> from sklearn.pipeline import make_pipeline
   >>> from sklearn.datasets import load_iris
   >>> from sklearn.model_selection import train_test_split
   >>> from sklearn.metrics import accuracy_score
   ...
   >>> # create a pipeline object

>>> from sklearn.datasets import load_iris
   ...
   >>> pipe = make_pipeline(
   ...     StandardScaler(),
   ...     LogisticRegression()
   ... )
   ...
   >>> # load the iris dataset and split it into train and test sets
   >>> X, y = load_iris(return_X_y=True)
   >>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
   ...
   >>> # fit the whole pipeline
   >>> pipe.fit(X_train, y_train)
   Pipeline(steps=[('standardscaler', StandardScaler()),
                   ('logisticregression', LogisticRegression())])
   >>> # we can now use it like any other estimator
   >>> accuracy_score(pipe.predict(X_test), y_test)
   0.97...

Model evaluation
================

Fitting a model to some data does not entail that it will predict well
on unseen data. This needs to be directly evaluated. We have just seen
the "train_test_split()" helper that splits a dataset into train and
test sets, but "scikit-learn" provides many other tools for model
evaluation, in particular for cross-validation.

We here briefly show how to perform a 5-fold cross-validation
procedure, using the "cross_validate()" helper. Note that it is also
possible to manually iterate over the folds, use different data
splitting strategies, and use custom scoring functions. Please refer
to our User Guide for more details:

   >>> from sklearn.datasets import make_regression

   >>> from sklearn.linear_model import LinearRegression
   >>> from sklearn.model_selection import cross_validate
   ...
   >>> X, y = make_regression(n_samples=1000, random_state=0)
   >>> lr = LinearRegression()
   ...
   >>> result = cross_validate(lr, X, y)  # defaults to 5-fold CV
   >>> result['test_score']  # r_squared score is high because dataset is easy
   array([1., 1., 1., 1., 1.])

Automatic parameter searches
============================

All estimators have parameters (often called hyper-parameters in the
literature) that can be tuned. The generalization power of an
estimator often critically depends on a few parameters. For example a
"RandomForestRegressor" has a "n_estimators" parameter that determines
the number of trees in the forest, and a "max_depth" parameter that
determines the maximum depth of each tree. Quite often, it is not
clear what the exact values of these parameters should be since they
depend on the data at hand.

"Scikit-learn" provides tools to automatically find the best parameter
combinations (via cross-validation). In the following example, we
randomly search over the parameter space of a random forest with a
"RandomizedSearchCV" object. When the search is over, the
"RandomizedSearchCV" behaves as a "RandomForestRegressor" that has
been fitted with the best set of parameters. Read more in the User
Guide:

Guide:

   >>> from sklearn.datasets import fetch_california_housing
   >>> from sklearn.ensemble import RandomForestRegressor
   >>> from sklearn.model_selection import RandomizedSearchCV
   >>> from scipy.stats import randint
   ...
   >>> X, y = fetch_california_housing(return_X_y=True)
   ...
   >>> # define the parameter space that will be searched over
   >>> param_distributions = {'n_estimators': randint(1, 5),
   ...                        'max_depth': randint(5, 10)}
   ...
   >>> # now create a searchCV object and fit it to the data
   >>> search = RandomizedSearchCV(estimator=RandomForestRegressor(random_state=0),
   ...                             n_iter=5,
   ...                             param_distributions=param_distributions,
   ...                             random_state=0)
   >>> search.fit(X_train, y_train)
   RandomizedSearchCV(estimator=RandomForestRegressor(random_state=0), n_iter=5,
                      param_distributions={'max_depth': ...,
                                           'n_estimators': ...},
                      random_state=0)
   >>> search.best_params_
   {'max_depth': 9, 'n_estimators': 4}

'n_estimators': ...},

   >>> # the search object now acts like a normal random forest estimator
   >>> # with max_depth=9 and n_estimators=4
   >>> search.score(X_test, y_test)
   0.73...

Note:

  In practice, you almost always want to search over a pipeline,
  instead of a single estimator. One of the main reasons is that if
  you apply a pre-processing step to the whole dataset without using a
  pipeline, and then perform any kind of cross-validation, you would
  be breaking the fundamental assumption of independence between
  training and testing data. Indeed, since you pre-processed the data
  using the whole dataset, some information about the test sets are
  available to the train sets. This will lead to over-estimating the
  generalization power of the estimator (you can read more in this
  Kaggle post).Using a pipeline for cross-validation and searching
  will largely keep you from this common pitfall.

Next steps
==========

We have briefly covered estimator fitting and predicting, pre-
processing steps, pipelines, cross-validation tools and automatic
hyper-parameter searches. This guide should give you an overview of
some of the main features of the library, but there is much more to
"scikit-learn"!

Please refer to our User Guide for details on all the tools that we
provide. You can also find an exhaustive list of the public API in the
API Reference.

You can also look at our numerous *examples* that illustrate the use
of "scikit-learn" in many different contexts.

The tutorials also contain additional learning resources.

10. Common pitfalls and recommended practices
*********************************************

The purpose of this chapter is to illustrate some common pitfalls and
anti-patterns that occur when using scikit-learn. It provides examples
of what **not** to do, along with a corresponding correct example.

10.1. Inconsistent preprocessing
================================

scikit-learn provides a library of Dataset transformations, which may
clean (see Preprocessing data), reduce (see Unsupervised
dimensionality reduction), expand (see Kernel Approximation) or
generate (see Feature extraction) feature representations. If these
data transforms are used when training a model, they also must be used
on subsequent datasets, whether it's test data or data in a production
system. Otherwise, the feature space will change, and the model will
not be able to perform effectively.

For the following example, let's create a synthetic dataset with a
single feature:

   >>> random_state = 42
   >>> X, y = make_regression(random_state=random_state, n_features=1, noise=1)
   >>> X_train, X_test, y_train, y_test = train_test_split(
   ...     X, y, test_size=0.4, random_state=random_state)

**Wrong**

The train dataset is scaled, but not the test dataset, so model
performance on the test dataset is worse than expected:

   >>> from sklearn.metrics import mean_squared_error

**Wrong**

   >>> scaler = StandardScaler()
   >>> X_train_transformed = scaler.fit_transform(X_train)
   >>> model = LinearRegression().fit(X_train_transformed, y_train)
   >>> mean_squared_error(y_test, model.predict(X_test))
   62.80...

**Right**

Instead of passing the non-transformed *X_test* to *predict*, we
should transform the test data, the same way we transformed the
training data:

   >>> X_test_transformed = scaler.transform(X_test)
   >>> mean_squared_error(y_test, model.predict(X_test_transformed))
   0.90...

Alternatively, we recommend using a "Pipeline", which makes it easier
to chain transformations with estimators, and reduces the possibility
of forgetting a transformation:

   >>> model = make_pipeline(StandardScaler(), LinearRegression())
   >>> model.fit(X_train, y_train)
                   ('linearregression', LinearRegression())])
   0.90...

Pipelines also help avoiding another common pitfall: leaking the test
data into the training data.

10.2. Data leakage
==================

Data leakage occurs when information that would not be available at

prediction time is used when building the model. This results in
overly optimistic performance estimates, for example from cross-
validation, and thus poorer performance when the model is used on
actually novel data, for example during production.

A common cause is not keeping the test and train data subsets
separate. Test data should never be used to make choices about the
model. **The general rule is to never call** *fit* **on the test
data**. While this may sound obvious, this is easy to miss in some
cases, for example when applying certain pre-processing steps.

Although both train and test data subsets should receive the same
preprocessing transformation (as described in the previous section),
it is important that these transformations are only learnt from the
training data. For example, if you have a normalization step where you
divide by the average value, the average should be the average of the
train subset, **not** the average of all the data. If the test subset
is included in the average calculation, information from the test
subset is influencing the model.

An example of data leakage during preprocessing is detailed below.

10.2.1. Data leakage during pre-processing
------------------------------------------

Note:

  We here choose to illustrate data leakage with a feature selection

Note:

  step. This risk of leakage is however relevant with almost all
  transformations in scikit-learn, including (but not limited to)
  "StandardScaler", "SimpleImputer", and "PCA".

A number of Feature selection functions are available in scikit-learn.
They can help remove irrelevant, redundant and noisy features as well
as improve your model build time and performance. As with any other
type of preprocessing, feature selection should **only** use the
training data. Including the test data in feature selection will
optimistically bias your model.

To demonstrate we will create this binary classification problem with
10,000 randomly generated features:

   >>> import numpy as np
   >>> n_samples, n_features, n_classes = 200, 10000, 2
   >>> rng = np.random.RandomState(42)
   >>> X = rng.standard_normal((n_samples, n_features))
   >>> y = rng.choice(n_classes, n_samples)

**Wrong**

Using all the data to perform feature selection results in an accuracy
score much higher than chance, even though our targets are completely
random. This randomness means that our *X* and *y* are independent and
we thus expect the accuracy to be around 0.5. However, since the
feature selection step 'sees' the test data, the model has an unfair
advantage. In the incorrect example below we first use all the data

for feature selection and then split the data into training and test
subsets for model fitting. The result is a much higher than expected
accuracy score:

   >>> from sklearn.feature_selection import SelectKBest
   >>> from sklearn.ensemble import GradientBoostingClassifier

   >>> # Incorrect preprocessing: the entire data is transformed
   >>> X_selected = SelectKBest(k=25).fit_transform(X, y)

   ...     X_selected, y, random_state=42)
   >>> gbc = GradientBoostingClassifier(random_state=1)
   >>> gbc.fit(X_train, y_train)
   GradientBoostingClassifier(random_state=1)

   >>> y_pred = gbc.predict(X_test)
   >>> accuracy_score(y_test, y_pred)
   0.76

**Right**

To prevent data leakage, it is good practice to split your data into
train and test subsets **first**. Feature selection can then be formed
using just the train dataset. Notice that whenever we use *fit* or
*fit_transform*, we only use the train dataset. The score is now what
we would expect for the data, close to chance:

   ...     X, y, random_state=42)
   >>> select = SelectKBest(k=25)

   >>> X_train_selected = select.fit_transform(X_train, y_train)

   >>> gbc.fit(X_train_selected, y_train)

   >>> X_test_selected = select.transform(X_test)
   >>> y_pred = gbc.predict(X_test_selected)
   0.46

Here again, we recommend using a "Pipeline" to chain together the
feature selection and model estimators. The pipeline ensures that only
the training data is used when performing *fit* and the test data is
used only for calculating the accuracy score:

   >>> pipeline = make_pipeline(SelectKBest(k=25),
   ...                          GradientBoostingClassifier(random_state=1))
   >>> pipeline.fit(X_train, y_train)
   Pipeline(steps=[('selectkbest', SelectKBest(k=25)),
                   ('gradientboostingclassifier',
                   GradientBoostingClassifier(random_state=1))])

   >>> y_pred = pipeline.predict(X_test)
   0.46

The pipeline can also be fed into a cross-validation function such as
"cross_val_score()". Again, the pipeline ensures that the correct data

>>> accuracy_score(y_test, y_pred)
   0.46

subset and estimator method is used during fitting and predicting:

   >>> from sklearn.model_selection import cross_val_score
   >>> scores = cross_val_score(pipeline, X, y)
   >>> print(f"Mean accuracy: {scores.mean():.2f}+/-{scores.std():.2f}")
   Mean accuracy: 0.45+/-0.07

10.2.2. How to avoid data leakage
---------------------------------

Below are some tips on avoiding data leakage:

* Always split the data into train and test subsets first,
  particularly before any preprocessing steps.

* Never include test data when using the *fit* and *fit_transform*
  methods. Using all the data, e.g., *fit(X)*, can result in overly
  optimistic scores.

  Conversely, the *transform* method should be used on both train and
  test subsets as the same preprocessing should be applied to all the
  data. This can be achieved by using *fit_transform* on the train
  subset and *transform* on the test subset.

* The scikit-learn pipeline is a great way to prevent data leakage as
  it ensures that the appropriate method is performed on the correct
  data subset. The pipeline is ideal for use in cross-validation and
  hyper-parameter tuning functions.

10.3. Controlling randomness

Some scikit-learn objects are inherently random. These are usually
estimators (e.g. "RandomForestClassifier") and cross-validation

splitters (e.g. "KFold"). The randomness of these objects is
controlled via their *random_state* parameter, as described in the
*Glossary*. This section expands on the glossary entry, and describes
good practices and common pitfalls w.r.t. to this subtle parameter.

Note:

  Recommendation summaryFor an optimal robustness of cross-validation
  (CV) results, pass *RandomState* instances when creating estimators,
  or leave *random_state* to *None*. Passing integers to CV splitters
  is usually the safest option and is preferable; passing
  *RandomState* instances to splitters may sometimes be useful to
  achieve very specific use-cases. For both estimators and splitters,
  passing an integer vs passing an instance (or *None*) leads to
  subtle but significant differences, especially for CV procedures.
  These differences are important to understand when reporting
  results.For reproducible results across executions, remove any use
  of *random_state=None*.

10.3.1. Using *None* or *RandomState* instances, and repeated calls to *fit* and *split*
----------------------------------------------------------------------------------------

The *random_state* parameter determines whether multiple calls to
*fit* (for estimators) or to *split* (for CV splitters) will produce
the same results, according to these rules:

* If an integer is passed, calling *fit* or *split* multiple times
  always yields the same results.

* If *None* or a *RandomState* instance is passed: *fit* and *split*
  will yield different results each time they are called, and the
  succession of calls explores all sources of entropy. *None* is the
  default value for all *random_state* parameters.

We here illustrate these rules for both estimators and CV splitters.

Note:

  Since passing *random_state=None* is equivalent to passing the
  global *RandomState* instance from *numpy*
  (*random_state=np.random.mtrand._rand*), we will not explicitly
  mention *None* here. Everything that applies to instances also
  applies to using *None*.

10.3.1.1. Estimators
~~~~~~~~~~~~~~~~~~~~

Passing instances means that calling *fit* multiple times will not
yield the same results, even if the estimator is fitted on the same
data and with the same hyper-parameters:

   >>> from sklearn.linear_model import SGDClassifier
   >>> from sklearn.datasets import make_classification

   >>> rng = np.random.RandomState(0)
   >>> X, y = make_classification(n_features=5, random_state=rng)
   >>> sgd = SGDClassifier(random_state=rng)

   >>> sgd.fit(X, y).coef_
   array([[ 8.85418642,  4.79084103, -3.13077794,  8.11915045, -0.56479934]])

>>> sgd = SGDClassifier(random_state=rng)

   array([[ 6.70814003,  5.25291366, -7.55212743,  5.18197458,  1.37845099]])

We can see from the snippet above that repeatedly calling *sgd.fit*
has produced different models, even if the data was the same. This is
because the Random Number Generator (RNG) of the estimator is consumed
(i.e. mutated) when *fit* is called, and this mutated RNG will be used
in the subsequent calls to *fit*. In addition, the *rng* object is
shared across all objects that use it, and as a consequence, these
objects become somewhat inter-dependent. For example, two estimators
that share the same *RandomState* instance will influence each other,
as we will see later when we discuss cloning. This point is important
to keep in mind when debugging.

If we had passed an integer to the *random_state* parameter of the
"SGDClassifier", we would have obtained the same models, and thus the
same scores each time. When we pass an integer, the same RNG is used
across all calls to *fit*. What internally happens is that even though
the RNG is consumed when *fit* is called, it is always reset to its
original state at the beginning of *fit*.

10.3.1.2. CV splitters
~~~~~~~~~~~~~~~~~~~~~~

Randomized CV splitters have a similar behavior when a *RandomState*
instance is passed; calling *split* multiple times yields different
data splits:

   >>> from sklearn.model_selection import KFold

   >>> X = y = np.arange(10)
   >>> cv = KFold(n_splits=2, shuffle=True, random_state=rng)

   >>> for train, test in cv.split(X, y):
   ...     print(train, test)
   [0 3 5 6 7] [1 2 4 8 9]
   [1 2 4 8 9] [0 3 5 6 7]

   [0 4 6 7 8] [1 2 3 5 9]
   [1 2 3 5 9] [0 4 6 7 8]

We can see that the splits are different from the second time *split*
is called. This may lead to unexpected results if you compare the
performance of multiple estimators by calling *split* many times, as
we will see in the next section.

10.3.2. Common pitfalls and subtleties
--------------------------------------

While the rules that govern the *random_state* parameter are seemingly
simple, they do however have some subtle implications. In some cases,
this can even lead to wrong conclusions.

10.3.2.1. Estimators

**Different `random_state` types lead to different cross-validation
procedures**

Depending on the type of the *random_state* parameter, estimators will
behave differently, especially in cross-validation procedures.
Consider the following snippet:

   >>> X, y = make_classification(random_state=0)

   >>> rf_123 = RandomForestClassifier(random_state=123)
   >>> cross_val_score(rf_123, X, y)
   array([0.85, 0.95, 0.95, 0.9 , 0.9 ])

   >>> rf_inst = RandomForestClassifier(random_state=np.random.RandomState(0))
   >>> cross_val_score(rf_inst, X, y)
   array([0.9 , 0.95, 0.95, 0.9 , 0.9 ])

We see that the cross-validated scores of *rf_123* and *rf_inst* are
different, as should be expected since we didn't pass the same
*random_state* parameter. However, the difference between these scores
is more subtle than it looks, and **the cross-validation procedures
that were performed by** "cross_val_score()" **significantly differ in
each case**:

* Since *rf_123* was passed an integer, every call to *fit* uses the
  same RNG: this means that all random characteristics of the random
  forest estimator will be the same for each of the 5 folds of the CV
  procedure. In particular, the (randomly chosen) subset of features
  of the estimator will be the same across all folds.

* Since *rf_inst* was passed a *RandomState* instance, each call to
  *fit* starts from a different RNG. As a result, the random subset of
  features will be different for each folds.

While having a constant estimator RNG across folds isn't inherently
wrong, we usually want CV results that are robust w.r.t. the
estimator's randomness. As a result, passing an instance instead of an
integer may be preferable, since it will allow the estimator RNG to
vary for each fold.

Note:

  Here, "cross_val_score()" will use a non-randomized CV splitter (as
  is the default), so both estimators will be evaluated on the same
  splits. This section is not about variability in the splits. Also,
  whether we pass an integer or an instance to "make_classification()"
  isn't relevant for our illustration purpose: what matters is what we
  pass to the "RandomForestClassifier" estimator.

**Cloning**

Another subtle side effect of passing *RandomState* instances is how
"clone()" will work:

   >>> from sklearn import clone

   >>> a = RandomForestClassifier(random_state=rng)
   >>> b = clone(a)

Since a *RandomState* instance was passed to *a*, *a* and *b* are not
clones in the strict sense, but rather clones in the statistical
sense: *a* and *b* will still be different models, even when calling
*fit(X, y)* on the same data. Moreover, *a* and *b* will influence

each-other since they share the same internal RNG: calling *a.fit*
will consume *b*'s RNG, and calling *b.fit* will consume *a*'s RNG,
since they are the same. This bit is true for any estimators that
share a *random_state* parameter; it is not specific to clones.

If an integer were passed, *a* and *b* would be exact clones and they
would not influence each other.

Warning:

  Even though "clone()" is rarely used in user code, it is called
  pervasively throughout scikit-learn codebase: in particular, most
  meta-estimators that accept non-fitted estimators call "clone()"
  internally ("GridSearchCV", "StackingClassifier",
  "CalibratedClassifierCV", etc.).

10.3.2.2. CV splitters

When passed a *RandomState* instance, CV splitters yield different
splits each time *split* is called. When comparing different
estimators, this can lead to overestimating the variance of the
difference in performance between the estimators:

   >>> from sklearn.naive_bayes import GaussianNB
   >>> from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

>>> from sklearn.model_selection import KFold

   >>> X, y = make_classification(random_state=rng)
   >>> cv = KFold(shuffle=True, random_state=rng)
   >>> lda = LinearDiscriminantAnalysis()
   >>> nb = GaussianNB()

   >>> for est in (lda, nb):
   ...     print(cross_val_score(est, X, y, cv=cv))
   [0.8  0.75 0.75 0.7  0.85]
   [0.85 0.95 0.95 0.85 0.95]

Directly comparing the performance of the "LinearDiscriminantAnalysis"
estimator vs the "GaussianNB" estimator **on each fold** would be a
mistake: **the splits on which the estimators are evaluated are
different**. Indeed, "cross_val_score()" will internally call
*cv.split* on the same "KFold" instance, but the splits will be
different each time. This is also true for any tool that performs
model selection via cross-validation, e.g. "GridSearchCV" and
"RandomizedSearchCV": scores are not comparable fold-to-fold across
different calls to *search.fit*, since *cv.split* would have been
called multiple times. Within a single call to *search.fit*, however,
fold-to-fold comparison is possible since the search estimator only
calls *cv.split* once.

For comparable fold-to-fold results in all scenarios, one should pass
an integer to the CV splitter: *cv = KFold(shuffle=True,
random_state=0)*.

Note:

  While fold-to-fold comparison is not advisable with *RandomState*
  instances, one can however expect that average scores allow to

Note:

  conclude whether one estimator is better than another, as long as
  enough folds and data are used.

Note:

  What matters in this example is what was passed to "KFold". Whether
  we pass a *RandomState* instance or an integer to
  "make_classification()" is not relevant for our illustration
  purpose. Also, neither "LinearDiscriminantAnalysis" nor "GaussianNB"
  are randomized estimators.

10.3.3. General recommendations
-------------------------------

10.3.3.1. Getting reproducible results across multiple executions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to obtain reproducible (i.e. constant) results across
multiple *program executions*, we need to remove all uses of
*random_state=None*, which is the default. The recommended way is to
declare a *rng* variable at the top of the program, and pass it down
to any object that accepts a *random_state* parameter:

   >>> rf = RandomForestClassifier(random_state=rng)
   >>> X_train, X_test, y_train, y_test = train_test_split(X, y,

>>> X, y = make_classification(random_state=rng)
   ...                                                     random_state=rng)
   >>> rf.fit(X_train, y_train).score(X_test, y_test)
   0.84

We are now guaranteed that the result of this script will always be
0.84, no matter how many times we run it. Changing the global *rng*
variable to a different value should affect the results, as expected.

It is also possible to declare the *rng* variable as an integer. This
may however lead to less robust cross-validation results, as we will
see in the next section.

Note:

  We do not recommend setting the global *numpy* seed by calling
  *np.random.seed(0)*. See here for a discussion.

10.3.3.2. Robustness of cross-validation results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When we evaluate a randomized estimator performance by cross-
validation, we want to make sure that the estimator can yield accurate
predictions for new data, but we also want to make sure that the
estimator is robust w.r.t. its random initialization. For example, we
would like the random weights initialization of a "SGDCLassifier" to
be consistently good across all folds: otherwise, when we train that
estimator on new data, we might get unlucky and the random
initialization may lead to bad performance. Similarly, we want a
random forest to be robust w.r.t the set of randomly selected features

that each tree will be using.

For these reasons, it is preferable to evaluate the cross-validation
performance by letting the estimator use a different RNG on each fold.
This is done by passing a *RandomState* instance (or *None*) to the
estimator initialization.

When we pass an integer, the estimator will use the same RNG on each
fold: if the estimator performs well (or bad), as evaluated by CV, it
might just be because we got lucky (or unlucky) with that specific
seed. Passing instances leads to more robust CV results, and makes the
comparison between various algorithms fairer. It also helps limiting
the temptation to treat the estimator's RNG as a hyper-parameter that
can be tuned.

Whether we pass *RandomState* instances or integers to CV splitters
has no impact on robustness, as long as *split* is only called once.
When *split* is called multiple times, fold-to-fold comparison isn't
possible anymore. As a result, passing integer to CV splitters is
usually safer and covers most use-cases.

Frequently Asked Questions
**************************

Here we try to give some answers to questions that regularly pop up on
the mailing list.

Table of Contents
^^^^^^^^^^^^^^^^^

* About the project

  * What is the project name (a lot of people get it wrong)?

  * How do you pronounce the project name?

  * Why scikit?

  * Do you support PyPy?

* Implementation decisions

  * Why is there no support for deep or reinforcement learning / Will
    there be support for deep or reinforcement learning in scikit-
    learn?

  * Will you add graphical models or sequence prediction to scikit-
    learn?

  * Why did you remove HMMs from scikit-learn?

  * Will you add GPU support?

  * Why do categorical variables need preprocessing in scikit-learn,
    compared to other tools?

  * Why does Scikit-learn not directly work with, for example,
    pandas.DataFrame?

  * Do you plan to implement transform for target y in a pipeline?

  * Why are there so many different estimators for linear models?

* Contributing

  * How can I contribute to scikit-learn?

  * Why is my pull request not getting any attention?

  * What are the inclusion criteria for new algorithms ?

  * Why are you so selective on what algorithms you include in scikit-
    learn?

* Using scikit-learn

  * What's the best way to get help on scikit-learn usage?

  * How should I save, export or deploy estimators for production?

  * How can I create a bunch object?

  * How can I load my own datasets into a format usable by scikit-
    learn?

* How should I save, export or deploy estimators for production?

    learn?

  * How do I deal with string data (or trees, graphs...)?

  * Why do I sometime get a crash/freeze with n_jobs > 1 under OSX or
    Linux?

  * Why does my job use more cores than specified with n_jobs?

  * How do I set a "random_state" for an entire execution?

About the project
=================

What is the project name (a lot of people get it wrong)?
--------------------------------------------------------

scikit-learn, but not scikit or SciKit nor sci-kit learn. Also not
scikits.learn or scikits-learn, which were previously used.

How do you pronounce the project name?

sy-kit learn. sci stands for science!

Why scikit?
-----------

There are multiple scikits, which are scientific toolboxes built
around SciPy. Apart from scikit-learn, another popular one is scikit-
image.

Do you support PyPy?

scikit-learn is regularly tested and maintained to work with PyPy (an
alternative Python implementation with a built-in just-in-time
compiler).

Note however that this support is still considered experimental and
specific components might behave slightly differently. Please refer to
the test suite of the specific module of interest for more details.

Implementation decisions

Why is there no support for deep or reinforcement learning / Will there be support for deep or reinforcement learning in scikit-learn?
--------------------------------------------------------------------------------------------------------------------------------------

Deep learning and reinforcement learning both require a rich
vocabulary to define an architecture, with deep learning additionally
requiring GPUs for efficient computing. However, neither of these fit
within the design constraints of scikit-learn; as a result, deep
learning and reinforcement learning are currently out of scope for
what scikit-learn seeks to achieve.

You can find more information about addition of gpu support at Will
you add GPU support?.

Note that scikit-learn currently implements a simple multilayer
perceptron in "sklearn.neural_network". We will only accept bug fixes
for this module. If you want to implement more complex deep learning
models, please turn to popular deep learning frameworks such as
tensorflow, keras and pytorch.

Will you add graphical models or sequence prediction to scikit-learn?

Not in the foreseeable future. scikit-learn tries to provide a unified
API for the basic tasks in machine learning, with pipelines and meta-

algorithms like grid search to tie everything together. The required
concepts, APIs, algorithms and expertise required for structured
learning are different from what scikit-learn has to offer. If we
started doing arbitrary structured learning, we'd need to redesign the
whole package and the project would likely collapse under its own
weight.

There are two projects with API similar to scikit-learn that do
structured prediction:

* pystruct handles general structured learning (focuses on SSVMs on
  arbitrary graph structures with approximate inference; defines the
  notion of sample as an instance of the graph structure)

* seqlearn handles sequences only (focuses on exact inference; has
  HMMs, but mostly for the sake of completeness; treats a feature
  vector as a sample and uses an offset encoding for the dependencies
  between feature vectors)

Why did you remove HMMs from scikit-learn?

See Will you add graphical models or sequence prediction to scikit-
learn?.

Will you add GPU support?
-------------------------

No, or at least not in the near future. The main reason is that GPU
support will introduce many software dependencies and introduce
platform specific issues. scikit-learn is designed to be easy to
install on a wide variety of platforms. Outside of neural networks,

GPUs don't play a large role in machine learning today, and much
larger gains in speed can often be achieved by a careful choice of
algorithms.

Why do categorical variables need preprocessing in scikit-learn, compared to other tools?
-----------------------------------------------------------------------------------------

Most of scikit-learn assumes data is in NumPy arrays or SciPy sparse
matrices of a single numeric dtype. These do not explicitly represent
categorical variables at present. Thus, unlike R's data.frames or
pandas.DataFrame, we require explicit conversion of categorical
features to numeric values, as discussed in Encoding categorical
features. See also *sphx_glr_auto_examples_compose_plot_column_transf
ormer_mixed_types.py* for an example of working with heterogeneous
(e.g. categorical and numeric) data.

Why does Scikit-learn not directly work with, for example, pandas.DataFrame?
----------------------------------------------------------------------------

The homogeneous NumPy and SciPy data objects currently expected are
most efficient to process for most operations. Extensive work would
also be needed to support Pandas categorical types. Restricting input
to homogeneous types therefore reduces maintenance cost and encourages
usage of efficient data structures.

Note however that "ColumnTransformer" makes it convenient to handle
heterogeneous pandas dataframes by mapping homogeneous subsets of
dataframe columns selected by name or dtype to dedicated scikit-learn
transformers.

Therefore "ColumnTransformer" are often used in the first step of
scikit-learn pipelines when dealing with heterogeneous dataframes (see
Pipeline: chaining estimators for more details).

See also *sphx_glr_auto_examples_compose_plot_column_transformer_mixe
d_types.py* for an example of working with heterogeneous (e.g.
categorical and numeric) data.

Do you plan to implement transform for target y in a pipeline?
--------------------------------------------------------------

Currently transform only works for features X in a pipeline. There's a
long-standing discussion about not being able to transform y in a
pipeline. Follow on github issue #4143. Meanwhile check out
"TransformedTargetRegressor", pipegraph, imbalanced-learn. Note that
Scikit-learn solved for the case where y has an invertible
transformation applied before training and inverted after prediction.
Scikit-learn intends to solve for use cases where y should be
transformed at training time and not at test time, for resampling and
similar uses, like at *imbalanced-learn*. In general, these use cases

can be solved with a custom meta estimator rather than a Pipeline

Why are there so many different estimators for linear models?
-------------------------------------------------------------

Usually, there is one classifier and one regressor per model type,
e.g. "GradientBoostingClassifier" and "GradientBoostingRegressor".
Both have similar options and both have the parameter *loss*, which is
especially useful in the regression case as it enables the estimation
of conditional mean as well as conditional quantiles.

For linear models, there are many estimator classes which are very
close to each other. Let us have a look at

* "LinearRegression", no penalty

* "Ridge", L2 penalty

* "Lasso", L1 penalty (sparse models)

* "ElasticNet", L1 + L2 penalty (less sparse models)

* "SGDRegressor" with *loss='squared_loss'*

**Maintainer perspective:** They all do in principle the same and are
different only by the penalty they impose. This, however, has a large
impact on the way the underlying optimization problem is solved. In
the end, this amounts to usage of different methods and tricks from
linear algebra. A special case is *SGDRegressor* which comprises all 4
previous models and is different by the optimization procedure. A
further side effect is that the different estimators favor different

data layouts (*X* c-contiguous or f-contiguous, sparse csr or csc).
This complexity of the seemingly simple linear models is the reason
for having different estimator classes for different penalties.

**User perspective:** First, the current design is inspired by the
scientific literature where linear regression models with different
regularization/penalty were given different names, e.g. *ridge
regression*. Having different model classes with according names makes
it easier for users to find those regression models. Secondly, if all
the 5 above mentioned linear models were unified into a single class,
there would be parameters with a lot of options like the "solver"
parameter. On top of that, there would be a lot of exclusive
interactions between different parameters. For example, the possible
options of the parameters "solver", "precompute" and "selection" would
depend on the chosen values of the penalty parameters "alpha" and
"l1_ratio".

Contributing

How can I contribute to scikit-learn?
-------------------------------------

See Contributing. Before wanting to add a new algorithm, which is
usually a major and lengthy undertaking, it is recommended to start
with known issues. Please do not contact the contributors of scikit-
learn directly regarding contributing to scikit-learn.

Why is my pull request not getting any attention?

The scikit-learn review process takes a significant amount of time,
and contributors should not be discouraged by a lack of activity or
review on their pull request. We care a lot about getting things right
the first time, as maintenance and later change comes at a high cost.
We rarely release any "experimental" code, so all of our contributions
will be subject to high use immediately and should be of the highest
quality possible initially.

Beyond that, scikit-learn is limited in its reviewing bandwidth; many
of the reviewers and core developers are working on scikit-learn on
their own time. If a review of your pull request comes slowly, it is
likely because the reviewers are busy. We ask for your understanding
and request that you not close your pull request or discontinue your
work solely because of this reason.

What are the inclusion criteria for new algorithms ?
----------------------------------------------------

We only consider well-established algorithms for inclusion. A rule of
thumb is at least 3 years since publication, 200+ citations, and wide
use and usefulness. A technique that provides a clear-cut improvement
(e.g. an enhanced data structure or a more efficient approximation
technique) on a widely-used method will also be considered for
inclusion.

From the algorithms or techniques that meet the above criteria, only
those which fit well within the current API of scikit-learn, that is a
"fit", "predict/transform" interface and ordinarily having
input/output that is a numpy array or sparse matrix, are accepted.

The contributor should support the importance of the proposed addition
with research papers and/or implementations in other similar packages,
demonstrate its usefulness via common use-cases/applications and
corroborate performance improvements, if any, with benchmarks and/or
plots. It is expected that the proposed algorithm should outperform
the methods that are already implemented in scikit-learn at least in
some areas.

Inclusion of a new algorithm speeding up an existing model is easier
if:

* it does not introduce new hyper-parameters (as it makes the library
  more future-proof),

* it is easy to document clearly when the contribution improves the
  speed and when it does not, for instance "when n_features >>
  n_samples",

* benchmarks clearly show a speed up.

Also, note that your implementation need not be in scikit-learn to be
used together with scikit-learn tools. You can implement your favorite
algorithm in a scikit-learn compatible way, upload it to GitHub and
let us know. We will be happy to list it under Related Projects. If
you already have a package on GitHub following the scikit-learn API,

you may also be interested to look at scikit-learn-contrib.

Why are you so selective on what algorithms you include in scikit-learn?
------------------------------------------------------------------------

Code comes with maintenance cost, and we need to balance the amount of
code we have with the size of the team (and add to this the fact that
complexity scales non linearly with the number of features). The
package relies on core developers using their free time to fix bugs,
maintain code and review contributions. Any algorithm that is added
needs future attention by the developers, at which point the original
author might long have lost interest. See also What are the inclusion
criteria for new algorithms ?. For a great read about long-term
maintenance issues in open-source software, look at the Executive
Summary of Roads and Bridges

Using scikit-learn

What's the best way to get help on scikit-learn usage?
------------------------------------------------------

**For general machine learning questions**, please use Cross Validated
with the "[machine-learning]" tag.

**For scikit-learn usage questions**, please use Stack Overflow with
the "[scikit-learn]" and "[python]" tags. You can alternatively use

Please make sure to include a minimal reproduction code snippet

(ideally shorter than 10 lines) that highlights your problem on a toy
dataset (for instance from "sklearn.datasets" or randomly generated
with functions of "numpy.random" with a fixed random seed). Please
remove any line of code that is not necessary to reproduce your
problem.

The problem should be reproducible by simply copy-pasting your code
snippet in a Python shell with scikit-learn installed. Do not forget
to include the import statements.

More guidance to write good reproduction code snippets can be found
at:

https://stackoverflow.com/help/mcve

If your problem raises an exception that you do not understand (even
after googling it), please make sure to include the full traceback
that you obtain when running the reproduction script.

For bug reports or feature requests, please make use of the issue
tracker on GitHub.

There is also a scikit-learn Gitter channel where some users and
developers might be found.

**Please do not email any authors directly to ask for assistance,
report bugs, or for any other issue related to scikit-learn.**

How should I save, export or deploy estimators for production?

See Model persistence.

How can I create a bunch object?
--------------------------------

Bunch objects are sometimes used as an output for functions and

methods. They extend dictionaries by enabling values to be accessed by
key, *bunch["value_key"]*, or by an attribute, *bunch.value_key*.

They should not be used as an input; therefore you almost never need
to create a "Bunch" object, unless you are extending the scikit-
learn's API.

How can I load my own datasets into a format usable by scikit-learn?
--------------------------------------------------------------------

Generally, scikit-learn works on any numeric data stored as numpy
arrays or scipy sparse matrices. Other types that are convertible to
numeric arrays such as pandas DataFrame are also acceptable.

For more information on loading your data files into these usable data
structures, please refer to loading external datasets.

How do I deal with string data (or trees, graphs...)?
-----------------------------------------------------

scikit-learn estimators assume you'll feed them real-valued feature
vectors. This assumption is hard-coded in pretty much all of the
library. However, you can feed non-numerical inputs to estimators in
several ways.

If you have text documents, you can use a term frequency features; see
Text feature extraction for the built-in *text vectorizers*. For more
general feature extraction from any kind of data, see Loading features
from dicts and Feature hashing.

Another common case is when you have non-numerical data and a custom
distance (or similarity) metric on these data. Examples include
strings with edit distance (aka. Levenshtein distance; e.g., DNA or
RNA sequences). These can be encoded as numbers, but doing so is
painful and error-prone. Working with distance metrics on arbitrary
data can be done in two ways.

Firstly, many estimators take precomputed distance/similarity
matrices, so if the dataset is not too large, you can compute
distances for all pairs of inputs. If the dataset is large, you can
use feature vectors with only one "feature", which is an index into a
separate data structure, and supply a custom metric function that
looks up the actual data in this data structure. E.g., to use DBSCAN
with Levenshtein distances:

   >>> from leven import levenshtein       
   >>> from sklearn.cluster import dbscan
   >>> data = ["ACCTCCTAGAAG", "ACCTACTAGAAGTT", "GAATATTAGGCCGA"]
   >>> def lev_metric(x, y):
   ...     i, j = int(x[0]), int(y[0])     # extract indices
   ...     return levenshtein(data[i], data[j])
   ...
   >>> X = np.arange(len(data)).reshape(-1, 1)
   >>> X
   array([[0],
          [1],
          [2]])
   >>> # We need to specify algoritum='brute' as the default assumes
   >>> # a continuous feature space.

>>> X
          [1],
          [2]])
   >>> dbscan(X, metric=lev_metric, eps=5, min_samples=2, algorithm='brute')
   ... 
   ([0, 1], array([ 0,  0, -1]))

(This uses the third-party edit distance package "leven".)

Similar tricks can be used, with some care, for tree kernels, graph
kernels, etc.

Why do I sometime get a crash/freeze with n_jobs > 1 under OSX or Linux?

Several scikit-learn tools such as "GridSearchCV" and
"cross_val_score" rely internally on Python's *multiprocessing* module
to parallelize execution onto several Python processes by passing
"n_jobs > 1" as an argument.

The problem is that Python "multiprocessing" does a "fork" system call
without following it with an "exec" system call for performance
reasons. Many libraries like (some versions of) Accelerate / vecLib
under OSX, (some versions of) MKL, the OpenMP runtime of GCC, nvidia's
Cuda (and probably many others), manage their own internal thread
pool. Upon a call to *fork*, the thread pool state in the child
process is corrupted: the thread pool believes it has many threads
while only the main thread state has been forked. It is possible to
change the libraries to make them detect when a fork happens and
reinitialize the thread pool in that case: we did that for OpenBLAS
(merged upstream in main since 0.2.10) and we contributed a patch to

GCC's OpenMP runtime (not yet reviewed).

But in the end the real culprit is Python's "multiprocessing" that
does "fork" without "exec" to reduce the overhead of starting and
using new Python processes for parallel computing. Unfortunately this
is a violation of the POSIX standard and therefore some software
editors like Apple refuse to consider the lack of fork-safety in
Accelerate / vecLib as a bug.

In Python 3.4+ it is now possible to configure "multiprocessing" to
use the 'forkserver' or 'spawn' start methods (instead of the default
'fork') to manage the process pools. To work around this issue when
using scikit-learn, you can set the "JOBLIB_START_METHOD" environment
variable to 'forkserver'. However the user should be aware that using
the 'forkserver' method prevents joblib.Parallel to call function
interactively defined in a shell session.

If you have custom code that uses "multiprocessing" directly instead
of using it via joblib you can enable the 'forkserver' mode globally
for your program: Insert the following instructions in your main
script:

   import multiprocessing

   # other imports, custom code, load data, define model...

   if __name__ == '__main__':
       multiprocessing.set_start_method('forkserver')

       # call scikit-learn utils with n_jobs > 1 here

# other imports, custom code, load data, define model...

You can find more default on the new start methods in the
multiprocessing documentation.

Why does my job use more cores than specified with n_jobs?
----------------------------------------------------------

This is because "n_jobs" only controls the number of jobs for routines
that are parallelized with "joblib", but parallel code can come from
other sources:

* some routines may be parallelized with OpenMP (for code written in C
  or Cython).

* scikit-learn relies a lot on numpy, which in turn may rely on
  numerical libraries like MKL, OpenBLAS or BLIS which can provide
  parallel implementations.

For more details, please refer to our Parallelism notes.

How do I set a "random_state" for an entire execution?

Please refer to Controlling randomness.

Release History

Release notes for all scikit-learn releases are linked in this page.

**Tip:** Subscribe to scikit-learn releases on libraries.io to be
notified when new versions are released.

* Version 1.3

* Version 1.2.2

* Version 1.2.1

* Version 1.2.0

* Version 1.1.3

* Version 1.1.2

* Version 1.1.1

* Version 1.1.0

* Version 1.0.2

* Version 1.0.1

* Version 1.0.0

* Version 0.24.2

* Version 0.24.1

* Version 0.24.0

* Version 0.23.2

* Version 0.23.1

* Version 0.23.0

* Version 0.22.2.post1

* Version 0.22.1

* Version 0.22.0

* Version 0.21.3

* Version 0.21.2

* Version 0.21.1

* Version 0.21.0

* Version 0.20.4

* Version 0.20.3

* Version 0.20.2

* Version 0.20.1

* Version 0.20.0

* Version 0.19.2

* Version 0.19.1

* Version 0.19

* Version 0.18.2

* Version 0.18.1

* Version 0.18

* Version 0.17.1

* Version 0.17

* Version 0.16.1

* Version 0.16

* Version 0.15.2

* Version 0.15.1

* Version 0.15

* Version 0.14

* Version 0.13.1

* Version 0.13

* Version 0.12.1

* Version 0.12

* Version 0.11

* Version 0.10

* Version 0.9

* Version 0.8

* Version 0.7

* Version 0.6

* Version 0.5

* Version 0.4

* Earlier versions

About us
********

History
=======

This project was started in 2007 as a Google Summer of Code project by
David Cournapeau. Later that year, Matthieu Brucher started work on
this project as part of his thesis.

In 2010 Fabian Pedregosa, Gael Varoquaux, Alexandre Gramfort and
Vincent Michel of INRIA took leadership of the project and made the
first public release, February the 1st 2010. Since then, several
releases have appeared following a ~ 3-month cycle, and a thriving
international community has been leading the development.

Governance

The decision making process and governance structure of scikit-learn
is laid out in the governance document.

Authors
=======

The following people are currently core contributors to scikit-learn's
development and maintenance:

Please do not email the authors directly to ask for assistance or
report issues. Instead, please see What's the best way to ask
questions about scikit-learn in the FAQ.

See also: How you can contribute to the project

===========================

The following people are active contributors who also help with
triaging issues, PRs, and general maintenance:

Communication Team

The following people help with communication around scikit-learn.

Emeritus Core Developers

The following people have been active contributors in the past, but
are no longer active in the project:

* Noel Dawe

* Noel Dawe

* Wei Li

* Ron Weiss

Emeritus Communication Team

The following people have been active in the communication team in the
past, but no longer have communication responsibilities:

* Reshama Shaikh

Citing scikit-learn
===================

If you use scikit-learn in a scientific publication, we would
appreciate citations to the following paper:

   Scikit-learn: Machine Learning in Python, Pedregosa *et al.*, JMLR
   12, pp. 2825-2830, 2011.

   Bibtex entry:

      @article{scikit-learn,
       title={Scikit-learn: Machine Learning in {P}ython},
       author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
               and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
               and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and

and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
               Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
       journal={Journal of Machine Learning Research},
       volume={12},
       pages={2825--2830},
       year={2011}
      }

If you want to cite scikit-learn for its API or design, you may also
want to consider the following paper:

   >>:arxiv:`API design for machine learning software: experiences
   from the scikit-learn project <1309.0238>`<<, Buitinck *et al.*,
   2013.

      @inproceedings{sklearn_api,
        author    = {Lars Buitinck and Gilles Louppe and Mathieu Blondel and
                     Fabian Pedregosa and Andreas Mueller and Olivier Grisel and
                     Vlad Niculae and Peter Prettenhofer and Alexandre Gramfort
                     and Jaques Grobler and Robert Layton and Jake VanderPlas and
                     Arnaud Joly and Brian Holt and Ga{\"{e}}l Varoquaux},
        title     = {{API} design for machine learning software: experiences from the scikit-learn
                     project},
        booktitle = {ECML PKDD Workshop: Languages for Data Mining and Machine Learning},
        year      = {2013},
        pages = {108--122},
      }

Artwork
=======

High quality PNG and SVG logos are available in the doc/logos/ source
directory.

[image]

Funding
=======

year      = {2013},
      }

Artwork
=======

[image]

Funding
=======

Scikit-Learn is a community driven project, however institutional and
private grants help to assure its sustainability.

The project would like to thank the following funders.

======================================================================

The Members of the Scikit-Learn Consortium at Inria Foundation  fund
Olivier Grisel, Guillaume Lemaitre, and Jérémie du Boisberranger.

+-------------+-------------+
| [image: bcg][image]       |
|                           |
| [image:     | [image: b   |
| axa][ima    | np][image]  |
| ge]         |             |
| [image:     | [image: m   |
| fujitsu]    | sn][image]  |
| [image]     |             |
|                           |
| [image:                   |
| dataiku][image]           |
| [image: aphp][image]      |
|                           |
| [image: inria][image]     |

Hugging Face funds Adrin Jalali since 2022.

[image]

Microsoft funds Andreas Müller since 2020.

[image]

[image]

[image]

Quansight Labs funds Thomas J. Fan since 2021.

[image]

Past Sponsors

Columbia University funded Andreas Müller (2016-2020).

[image]

The University of Sydney funded Joel Nothman (2017-2021).

[image]

Andreas Müller received a grant to improve scikit-learn from the
Alfred P. Sloan Foundation . This grant supported the position of
Nicolas Hug and Thomas J. Fan.

[image]

INRIA actively supports this project. It has provided funding for
Fabian Pedregosa (2010-2012), Jaques Grobler (2012-2013) and Olivier
Grisel (2013-2017) to work on this project full-time. It also hosts
coding sprints and other events.

[image]

Paris-Saclay Center for Data Science funded one year for a developer
to work on the project full-time (2014-2015), 50% of the time of
Guillaume Lemaitre (2016-2017) and 50% of the time of Joris van den
Bossche (2017-2018).

[image]

NYU Moore-Sloan Data Science Environment funded Andreas Mueller

[image]

(2014-2016) to work on this project. The Moore-Sloan Data Science
Environment also funds several students to work on the project part-
time.

[image]

Télécom Paristech funded Manoj Kumar (2014), Tom Dupré la Tour (2015),
Raghav RV (2015-2017), Thierry Guillemot (2016-2017) and Albert Thomas
(2017) to work on scikit-learn.

[image]

The Labex DigiCosme funded Nicolas Goix (2015-2016), Tom Dupré la Tour
(2015-2016 and 2017-2018), Mathurin Massias (2018-2019) to work part
time on scikit-learn during their PhDs. It also funded a scikit-learn
coding sprint in 2015.

[image]

The Chan-Zuckerberg Initiative funded Nicolas Hug to work full-time on
scikit-learn in 2020.

[image]

The following students were sponsored by Google to work on scikit-
learn through the Google Summer of Code program.

* 2007 - David Cournapeau

* 2011 - Vlad Niculae

* 2012 - Vlad Niculae, Immanuel Bayer.

* 2013 - Kemal Eren, Nicolas Trésegnie

* 2014 - Hamzeh Alsalhi, Issam Laradji, Maheshakya Wijewardena, Manoj
  Kumar.

* 2015 - Raghav RV, Wei Xue

* 2016 - Nelson Liu, YenChen Lin

  Kumar.

The NeuroDebian project providing Debian packaging and contributions
is supported by Dr. James V. Haxby (Dartmouth College).

Sprints
=======

The International 2019 Paris sprint was kindly hosted by AXA. Also
some participants could attend thanks to the support of the Alfred P.
Sloan Foundation, the Python Software Foundation (PSF) and the DATAIA
Institute.

The 2013 International Paris Sprint was made possible thanks to the
support of Télécom Paristech, tinyclues, the French Python Association
and the Fonds de la Recherche Scientifique.

The 2011 International Granada sprint was made possible thanks to the
support of the PSF and tinyclues.

Donating to the project
-----------------------

If you are interested in donating to the project or to one of our
code-sprints, please donate via the NumFOCUS Donations Page.

All donations will be handled by NumFOCUS, a non-profit-organization
which is managed by a board of Scipy community members. NumFOCUS's
mission is to foster scientific computing software, in particular in
Python. As a fiscal home of scikit-learn, it ensures that money is

available when needed to keep the project funded and available while
in compliance with tax regulations.

The received donations for the scikit-learn project mostly will go
towards covering travel-expenses for code sprints, as well as towards
the organization budget of the project [1].

-[ Notes ]-

[1] Regarding the organization budget, in particular, we might use
    some of the donated funds to pay for other project expenses such
    as DNS, hosting or continuous integration services.

Infrastructure support
======================

* We would also like to thank Microsoft Azure, Travis Cl, CircleCl for
  free CPU time on their Continuous Integration servers, and Anaconda
  Inc. for the storage they provide for our staging and nightly
  builds.

Glossary of Common Terms and API Elements
*****************************************

This glossary hopes to definitively represent the tacit and explicit
conventions applied in Scikit-learn and its API, while providing a
reference for users and contributors. It aims to describe the concepts
and either detail their corresponding API or link to other relevant
parts of the documentation which do so. By linking to glossary entries
from the API Reference and User Guide, we may minimize redundancy and
inconsistency.

We begin by listing general concepts (and any that didn't fit
elsewhere), but more specific sets of related terms are listed below:
Class APIs and Estimator Types, Target Types, Methods, Parameters,
Attributes, Data and sample properties.

General Concepts

1d
1d array
   One-dimensional array. A NumPy array whose ".shape" has length 1. A
   vector.

2d
2d array
   Two-dimensional array. A NumPy array whose ".shape" has length 2.
   Often represents a matrix.

API
   Refers to both the *specific* interfaces for estimators implemented
   in Scikit-learn and the *generalized* conventions across types of
   estimators as described in this glossary and overviewed in the
   contributor documentation.

   The specific interfaces that constitute Scikit-learn's public API
   are largely documented in API Reference. However, we less formally
   consider anything as public API if none of the identifiers required
   to access it begins with "_".  We generally try to maintain

consider anything as public API if none of the identifiers required
   *backwards compatibility* for all objects in the public API.

   Private API, including functions, modules and methods beginning "_"
   are not assured to be stable.

array-like
   The most common data format for *input* to Scikit-learn estimators
   and functions, array-like is any type object for which
   "numpy.asarray()" will produce an array of appropriate shape
   (usually 1 or 2-dimensional) of appropriate dtype (usually
   numeric).

   This includes:

   * a numpy array

   * a list of numbers

   * a list of length-k lists of numbers for some fixed length k

   * a "pandas.DataFrame" with all columns numeric

   * a numeric "pandas.Series"

   It excludes:

   * a *sparse matrix*

   * an iterator

   * a generator

   Note that *output* from scikit-learn estimators and functions (e.g.
   predictions) should generally be arrays or sparse matrices, or
   lists thereof (as in multi-output "tree.DecisionTreeClassifier"'s
   "predict_proba"). An estimator where "predict()" returns a list or
   a *pandas.Series* is not valid.

attribute
attributes
   We mostly use attribute to refer to how model information is stored
   on an estimator during fitting.  Any public attribute stored on an
   estimator instance is required to begin with an alphabetic
   character and end in a single underscore if it is set in *fit* or

on an estimator during fitting.  Any public attribute stored on an
   *partial_fit*.  These are what is documented under an estimator's
   *Attributes* documentation.  The information stored in attributes
   is usually either: sufficient statistics used for prediction or
   transformation; *transductive* outputs such as *labels_* or
   *embedding_*; or diagnostic data, such as *feature_importances_*.
   Common attributes are listed below.

   A public attribute may have the same name as a constructor
   *parameter*, with a "_" appended.  This is used to store a
   validated or estimated version of the user's input. For example,
   "decomposition.PCA" is constructed with an "n_components"
   parameter. From this, together with other parameters and the data,
   PCA estimates the attribute "n_components_".

   Further private attributes used in prediction/transformation/etc.
   may also be set when fitting.  These begin with a single underscore
   and are not assured to be stable for public access.

   A public attribute on an estimator instance that does not end in an
   underscore should be the stored, unmodified value of an "__init__"
   *parameter* of the same name.  Because of this equivalence, these
   are documented under an estimator's *Parameters* documentation.

backwards compatibility
   We generally try to maintain backward compatibility (i.e.

are documented under an estimator's *Parameters* documentation.

   interfaces and behaviors may be extended but not changed or
   removed) from release to release but this comes with some
   exceptions:

   Public API only
      The behavior of objects accessed through private identifiers
      (those beginning "_") may be changed arbitrarily between
      versions.

   As documented
      We will generally assume that the users have adhered to the
      documented parameter types and ranges. If the documentation asks
      for a list and the user gives a tuple, we do not assure
      consistent behavior from version to version.

   Deprecation
      Behaviors may change following a *deprecation* period (usually
      two releases long).  Warnings are issued using Python's
      "warnings" module.

   Keyword arguments
      We may sometimes assume that all optional parameters (other than
      X and y to *fit* and similar methods) are passed as keyword
      arguments only and may be positionally reordered.

   Bug fixes and enhancements
      Bug fixes and -- less often -- enhancements may change the
      behavior of estimators, including the predictions of an
      estimator trained on the same data and *random_state*.  When
      this happens, we attempt to note it clearly in the changelog.

   Serialization
      We make no assurances that pickling an estimator in one version

this happens, we attempt to note it clearly in the changelog.

      will allow it to be unpickled to an equivalent model in the
      subsequent version.  (For estimators in the sklearn package, we
      issue a warning when this unpickling is attempted, even if it
      may happen to work.)  See Security & maintainability
      limitations.

   "utils.estimator_checks.check_estimator()"
      We provide limited backwards compatibility assurances for the
      estimator checks: we may add extra requirements on estimators
      tested with this function, usually when these were informally
      assumed but not formally tested.

   Despite this informal contract with our users, the software is
   provided as is, as stated in the license.  When a release
   inadvertently introduces changes that are not backward compatible,
   these are known as software regressions.

callable
   A function, class or an object which implements the "__call__"
   method; anything that returns True when the argument of callable().

categorical feature
   A categorical or nominal *feature* is one that has a finite set of
   discrete values across the population of data. These are commonly
   represented as columns of integers or strings. Strings will be
   rejected by most scikit-learn estimators, and integers will be
   treated as ordinal or count-valued. For the use with most

represented as columns of integers or strings. Strings will be
   estimators, categorical variables should be one-hot encoded.
   Notable exceptions include tree-based models such as random forests
   and gradient boosting models that often work better and faster with
   integer-coded categorical variables. "OrdinalEncoder" helps
   encoding string-valued categorical features as ordinal integers,
   and "OneHotEncoder" can be used to one-hot encode categorical
   features. See also Encoding categorical features and the
   categorical-encoding package for tools related to encoding
   categorical features.

clone
cloned
   To copy an *estimator instance* and create a new one with identical
   *parameters*, but without any fitted *attributes*, using "clone()".

   When "fit" is called, a *meta-estimator* usually clones a wrapped
   estimator instance before fitting the cloned instance. (Exceptions,
   for legacy reasons, include "Pipeline" and "FeatureUnion".)

   If the estimator's *random_state* parameter is an integer (or if
   the estimator doesn't have a *random_state* parameter), an *exact
   clone* is returned: the clone and the original estimator will give
   the exact same results. Otherwise, *statistical clone* is returned:
   the clone might yield different results from the original
   estimator. More details can be found in Controlling randomness.

common tests

the clone might yield different results from the original

   This refers to the tests run on almost every estimator class in
   Scikit-learn to check they comply with basic API conventions.  They
   are available for external use through
   "utils.estimator_checks.check_estimator()", with most of the
   implementation in "sklearn/utils/estimator_checks.py".

   Note: Some exceptions to the common testing regime are currently
   hard-coded into the library, but we hope to replace this by marking
   exceptional behaviours on the estimator using semantic *estimator
   tags*.

deprecation
   We use deprecation to slowly violate our *backwards compatibility*
   assurances, usually to:

   * change the default value of a parameter; or

   * remove a parameter, attribute, method, class, etc.

   We will ordinarily issue a warning when a deprecated element is
   used, although there may be limitations to this.  For instance, we
   will raise a warning when someone sets a parameter that has been
   deprecated, but may not when they access that parameter's attribute
   on the estimator instance.

   See the Contributors' Guide.

dimensionality
   May be used to refer to the number of *features* (i.e.
   *n_features*), or columns in a 2d feature matrix. Dimensions are,
   however, also used to refer to the length of a NumPy array's shape,
   distinguishing a 1d array from a 2d matrix.

docstring

*n_features*), or columns in a 2d feature matrix. Dimensions are,

docstring
   The embedded documentation for a module, class, function, etc.,
   usually in code as a string at the beginning of the object's
   definition, and accessible as the object's "__doc__" attribute.

   We try to adhere to PEP257, and follow NumpyDoc conventions.

double underscore
double underscore notation
   When specifying parameter names for nested estimators, "__" may be
   used to separate between parent and child in some contexts. The
   most common use is when setting parameters through a meta-estimator
   with *set_params* and hence in specifying a search grid in
   parameter search. See *parameter*. It is also used in
   "pipeline.Pipeline.fit()" for passing *sample properties* to the
   "fit" methods of estimators in the pipeline.

dtype
data type
   NumPy arrays assume a homogeneous data type throughout, available
   in the ".dtype" attribute of an array (or sparse matrix). We
   generally assume simple data types for scikit-learn data: float or
   integer. We may support object or string data types for arrays
   before encoding or vectorizing.  Our estimators do not work with
   struct arrays, for instance.

   Our documentation can sometimes give information about the dtype
   precision, e.g. *np.int32*, *np.int64*, etc. When the precision is

struct arrays, for instance.

   provided, it refers to the NumPy dtype. If an arbitrary precision
   is used, the documentation will refer to dtype *integer* or
   *floating*. Note that in this case, the precision can be platform
   dependent. The *numeric* dtype refers to accepting both *integer*
   and *floating*.

   TODO: Mention efficiency and precision issues; casting policy.

duck typing
   We try to apply duck typing to determine how to handle some input
   values (e.g. checking whether a given estimator is a classifier).
   That is, we avoid using "isinstance" where possible, and rely on
   the presence or absence of attributes to determine an object's
   behaviour.  Some nuance is required when following this approach:

   * For some estimators, an attribute may only be available once it
     is *fitted*.  For instance, we cannot a priori determine if
     *predict_proba* is available in a grid search where the grid
     includes alternating between a probabilistic and a non-
     probabilistic predictor in the final step of the pipeline.  In
     the following, we can only determine if "clf" is probabilistic
     after fitting it on some data:

        >>> from sklearn.model_selection import GridSearchCV
        >>> from sklearn.linear_model import SGDClassifier
        >>> clf = GridSearchCV(SGDClassifier(),

after fitting it on some data:

        ...                    param_grid={'loss': ['log_loss', 'hinge']})

     This means that we can only check for duck-typed attributes after
     fitting, and that we must be careful to make *meta-estimators*
     only present attributes according to the state of the underlying
     estimator after fitting.

   * Checking if an attribute is present (using "hasattr") is in
     general just as expensive as getting the attribute ("getattr" or
     dot notation).  In some cases, getting the attribute may indeed
     be expensive (e.g. for some implementations of
     *feature_importances_*, which may suggest this is an API design
     flaw).  So code which does "hasattr" followed by "getattr" should
     be avoided; "getattr" within a try-except block is preferred.

   * For determining some aspects of an estimator's expectations or
     support for some feature, we use *estimator tags* instead of duck
     typing.

early stopping
   This consists in stopping an iterative optimization method before
   the convergence of the training loss, to avoid over-fitting. This
   is generally done by monitoring the generalization score on a
   validation set. When available, it is activated through the
   parameter "early_stopping" or by setting a positive
   *n_iter_no_change*.

estimator instance

validation set. When available, it is activated through the

   We sometimes use this terminology to distinguish an *estimator*
   class from a constructed instance. For example, in the following,
   "cls" is an estimator class, while "est1" and "est2" are instances:

      cls = RandomForestClassifier
      est1 = cls()
      est2 = RandomForestClassifier()

examples
   We try to give examples of basic usage for most functions and
   classes in the API:

   * as doctests in their docstrings (i.e. within the "sklearn/"
     library code itself).

   * as examples in the *example gallery* rendered (using sphinx-
     gallery) from scripts in the "examples/" directory, exemplifying
     key features or parameters of the estimator/function.  These
     should also be referenced from the User Guide.

   * sometimes in the User Guide (built from "doc/") alongside a
     technical description of the estimator.

experimental
   An experimental tool is already usable but its public API, such as
   default parameter values or fitted attributes, is still subject to
   change in future versions without the usual *deprecation* warning
   policy.

evaluation metric
evaluation metrics
   Evaluation metrics give a measure of how well a model performs.  We
   may use this term specifically to refer to the functions in
   "metrics" (disregarding "metrics.pairwise"), as distinct from the

Evaluation metrics give a measure of how well a model performs.  We
   *score* method and the *scoring* API used in cross validation. See
   Metrics and scoring: quantifying the quality of predictions.

   These functions usually accept a ground truth (or the raw data
   where the metric evaluates clustering without a ground truth) and a
   prediction, be it the output of *predict* ("y_pred"), of
   *predict_proba* ("y_proba"), or of an arbitrary score function
   including *decision_function* ("y_score"). Functions are usually
   named to end with "_score" if a greater score indicates a better
   model, and "_loss" if a lesser score indicates a better model.
   This diversity of interface motivates the scoring API.

   Note that some estimators can calculate metrics that are not
   included in "metrics" and are estimator-specific, notably model
   likelihoods.

estimator tags
   A proposed feature (e.g. >>:issue:`8022`<<) by which the
   capabilities of an estimator are described through a set of
   semantic tags.  This would enable some runtime behaviors based on
   estimator inspection, but it also allows each estimator to be
   tested for appropriate invariances while being excepted from other
   *common tests*.

   Some aspects of estimator tags are currently determined through the
   *duck typing* of methods like "predict_proba" and through some

*common tests*.

   special attributes on estimator objects:

   "_estimator_type"
      This string-valued attribute identifies an estimator as being a
      classifier, regressor, etc. It is set by mixins such as
      "base.ClassifierMixin", but needs to be more explicitly adopted
      on a *meta-estimator*.  Its value should usually be checked by
      way of a helper such as "base.is_classifier()".

   For more detailed info, see Estimator Tags.

feature
features
feature vector
   In the abstract, a feature is a function (in its mathematical
   sense) mapping a sampled object to a numeric or categorical
   quantity. "Feature" is also commonly used to refer to these
   quantities, being the individual elements of a vector representing
   a sample. In a data matrix, features are represented as columns:
   each column contains the result of applying a feature function to a
   set of samples.

   Elsewhere features are known as attributes, predictors, regressors,
   or independent variables.

   Nearly all estimators in scikit-learn assume that features are
   numeric, finite and not missing, even when they have semantically
   distinct domains and distributions (categorical, ordinal, count-
   valued, real-valued, interval). See also *categorical feature* and
   *missing values*.

   "n_features" indicates the number of features in a dataset.

fitting

valued, real-valued, interval). See also *categorical feature* and

fitting
   Calling *fit* (or *fit_transform*, *fit_predict*, etc.) on an
   estimator.

fitted
   The state of an estimator after *fitting*.

   There is no conventional procedure for checking if an estimator is
   fitted.  However, an estimator that is not fitted:

   * should raise "exceptions.NotFittedError" when a prediction method
     (*predict*, *transform*, etc.) is called.
     ("utils.validation.check_is_fitted()" is used internally for this
     purpose.)

   * should not have any *attributes* beginning with an alphabetic
     character and ending with an underscore. (Note that a descriptor
     for the attribute may still be present on the class, but hasattr
     should return False)

function
   We provide ad hoc function interfaces for many algorithms, while
   *estimator* classes provide a more consistent interface.

   In particular, Scikit-learn may provide a function interface that
   fits a model to some data and returns the learnt model parameters,
   as in "linear_model.enet_path()".  For transductive models, this
   also returns the embedding or cluster labels, as in
   "manifold.spectral_embedding()" or "cluster.dbscan()".  Many
   preprocessing transformers also provide a function interface, akin
   to calling *fit_transform*, as in "preprocessing.maxabs_scale()".
   Users should be careful to avoid *data leakage* when making use of

to calling *fit_transform*, as in "preprocessing.maxabs_scale()".
   these "fit_transform"-equivalent functions.

   We do not have a strict policy about when to or when not to provide
   function forms of estimators, but maintainers should consider
   consistency with existing interfaces, and whether providing a
   function would lead users astray from best practices (as regards
   data leakage, etc.)

gallery
   See *examples*.

hyperparameter
hyper-parameter
   See *parameter*.

impute
imputation
   Most machine learning algorithms require that their inputs have no
   *missing values*, and will not work if this requirement is
   violated. Algorithms that attempt to fill in (or impute) missing
   values are referred to as imputation algorithms.

indexable
   An *array-like*, *sparse matrix*, pandas DataFrame or sequence
   (usually a list).

induction
inductive
   Inductive (contrasted with *transductive*) machine learning builds
   a model of some data that can then be applied to new instances.
   Most estimators in Scikit-learn are inductive, having *predict*
   and/or *transform* methods.

joblib
   A Python library (https://joblib.readthedocs.io) used in Scikit-
   learn to facilite simple parallelism and caching.  Joblib is
   oriented towards efficiently working with numpy arrays, such as
   through use of *memory mapping*. See Parallelism for more
   information.

label indicator matrix
multilabel indicator matrix

oriented towards efficiently working with numpy arrays, such as

multilabel indicator matrices
   The format used to represent multilabel data, where each row of a
   2d array or sparse matrix corresponds to a sample, each column
   corresponds to a class, and each element is 1 if the sample is
   labeled with the class and 0 if not.

leakage
data leakage
   A problem in cross validation where generalization performance can
   be over-estimated since knowledge of the test data was
   inadvertently included in training a model.  This is a risk, for
   instance, when applying a *transformer* to the entirety of a
   dataset rather than each training portion in a cross validation
   split.

   We aim to provide interfaces (such as "pipeline" and
   "model_selection") that shield the user from data leakage.

memmapping
memory map
memory mapping
   A memory efficiency strategy that keeps data on disk rather than
   copying it into main memory.  Memory maps can be created for arrays
   that can be read, written, or both, using "numpy.memmap". When
   using *joblib* to parallelize operations in Scikit-learn, it may
   automatically memmap large arrays to reduce memory duplication
   overhead in multiprocessing.

missing values
   Most Scikit-learn estimators do not work with missing values. When
   they do (e.g. in "impute.SimpleImputer"), NaN is the preferred

overhead in multiprocessing.

   representation of missing values in float arrays.  If the array has
   integer dtype, NaN cannot be represented. For this reason, we
   support specifying another "missing_values" value when *imputation*
   or learning can be performed in integer space. *Unlabeled data* is
   a special case of missing values in the *target*.

"n_features"
   The number of *features*.

"n_outputs"
   The number of *outputs* in the *target*.

"n_samples"
   The number of *samples*.

"n_targets"
   Synonym for *n_outputs*.

narrative docs
narrative documentation
   An alias for User Guide, i.e. documentation written in
   "doc/modules/". Unlike the API reference provided through
   docstrings, the User Guide aims to:

   * group tools provided by Scikit-learn together thematically or in
     terms of usage;

   * motivate why someone would use each particular tool, often
     through comparison;

   * provide both intuitive and technical descriptions of tools;

   * provide or link to *examples* of using key features of a tool.

np
   A shorthand for Numpy due to the conventional import statement:

      import numpy as np

online learning
   Where a model is iteratively updated by receiving each batch of
   ground truth *targets* soon after making predictions on
   corresponding batch of data.  Intrinsically, the model must be

Where a model is iteratively updated by receiving each batch of
   usable for prediction after each batch. See *partial_fit*.

out-of-core
   An efficiency strategy where not all the data is stored in main
   memory at once, usually by performing learning on batches of data.
   See *partial_fit*.

outputs
   Individual scalar/categorical variables per sample in the *target*.
   For example, in multilabel classification each possible label
   corresponds to a binary output. Also called *responses*, *tasks* or
   *targets*. See *multiclass multioutput* and *continuous
   multioutput*.

pair
   A tuple of length two.

parameter
parameters
param
params
   We mostly use *parameter* to refer to the aspects of an estimator
   that can be specified in its construction. For example, "max_depth"
   and "random_state" are parameters of "RandomForestClassifier".
   Parameters to an estimator's constructor are stored unmodified as
   attributes on the estimator instance, and conventionally start with
   an alphabetic character and end with an alphanumeric character.
   Each estimator's constructor parameters are described in the
   estimator's docstring.

   We do not use parameters in the statistical sense, where parameters
   are values that specify a model and can be estimated from data.
   What we call parameters might be what statisticians call

We do not use parameters in the statistical sense, where parameters
   hyperparameters to the model: aspects for configuring model
   structure that are often not directly learnt from data.  However,
   our parameters are also used to prescribe modeling operations that
   do not affect the learnt model, such as *n_jobs* for controlling
   parallelism.

   When talking about the parameters of a *meta-estimator*, we may
   also be including the parameters of the estimators wrapped by the
   meta-estimator.  Ordinarily, these nested parameters are denoted by
   using a *double underscore* ("__") to separate between the
   estimator-as-parameter and its parameter.  Thus "clf =
   BaggingClassifier(estimator=DecisionTreeClassifier(max_depth=3))"
   has a deep parameter "estimator__max_depth" with value "3", which
   is accessible with "clf.estimator.max_depth" or
   "clf.get_params()['estimator__max_depth']".

   The list of parameters and their current values can be retrieved
   from an *estimator instance* using its *get_params* method.

   Between construction and fitting, parameters may be modified using
   *set_params*.  To enable this, parameters are not ordinarily
   validated or altered when the estimator is constructed, or when
   each parameter is set. Parameter validation is performed when *fit*
   is called.

   Common parameters are listed below.

pairwise metric

each parameter is set. Parameter validation is performed when *fit*
   is called.

pairwise metrics
   In its broad sense, a pairwise metric defines a function for
   measuring similarity or dissimilarity between two samples (with
   each ordinarily represented as a *feature vector*).  We
   particularly provide implementations of distance metrics (as well
   as improper metrics like Cosine Distance) through
   "metrics.pairwise_distances()", and of kernel functions (a
   constrained class of similarity functions) in
   "metrics.pairwise_kernels()".  These can compute pairwise distance
   matrices that are symmetric and hence store data redundantly.

   See also *precomputed* and *metric*.

   Note that for most distance metrics, we rely on implementations
   from "scipy.spatial.distance", but may reimplement for efficiency
   in our context. The "metrics.DistanceMetric" interface is used to
   implement distance metrics for integration with efficient neighbors
   search.

pd
   A shorthand for Pandas due to the conventional import statement:

      import pandas as pd

precomputed
   Where algorithms rely on *pairwise metrics*, and can be computed
   from pairwise metrics alone, we often allow the user to specify
   that the *X* provided is already in the pairwise (dis)similarity
   space, rather than in a feature space.  That is, when passed to
   *fit*, it is a square, symmetric matrix, with each vector

that the *X* provided is already in the pairwise (dis)similarity
   indicating (dis)similarity to every sample, and when passed to
   prediction/transformation methods, each row corresponds to a
   testing sample and each column to a training sample.

   Use of precomputed X is usually indicated by setting a "metric",
   "affinity" or "kernel" parameter to the string 'precomputed'. If
   this is the case, then the estimator should set the *pairwise*
   estimator tag as True.

rectangular
   Data that can be represented as a matrix with *samples* on the
   first axis and a fixed, finite set of *features* on the second is
   called rectangular.

   This term excludes samples with non-vectorial structures, such as
   text, an image of arbitrary size, a time series of arbitrary
   length, a set of vectors, etc. The purpose of a *vectorizer* is to
   produce rectangular forms of such data.

sample
samples
   We usually use this term as a noun to indicate a single feature
   vector. Elsewhere a sample is called an instance, data point, or
   observation. "n_samples" indicates the number of samples in a
   dataset, being the number of rows in a data array *X*.

sample property
sample properties
   A sample property is data for each sample (e.g. an array of length
   n_samples) passed to an estimator method or a similar function,

   alongside but distinct from the *features* ("X") and *target*
   ("y"). The most prominent example is *sample_weight*; see others at
   Data and sample properties.

   As of version 0.19 we do not have a consistent approach to handling
   sample properties and their routing in *meta-estimators*, though a
   "fit_params" parameter is often used.

scikit-learn-contrib
   A venue for publishing Scikit-learn-compatible libraries that are
   broadly authorized by the core developers and the contrib
   community, but not maintained by the core developer team. See https
   ://scikit-learn-contrib.github.io.

scikit-learn enhancement proposals
SLEP
SLEPs
   Changes to the API principles and changes to dependencies or
   supported versions happen via a SLEP and follows the decision-
   making process outlined in Scikit-learn governance and decision-
   making. For all votes, a proposal must have been made public and
   discussed before the vote. Such a proposal must be a consolidated
   document, in the form of a "Scikit-Learn Enhancement Proposal"
   (SLEP), rather than a long discussion on an issue. A SLEP must be
   submitted as a pull-request to enhancement proposals using the SLEP
   template.

semi-supervised
semi-supervised learning
semisupervised
   Learning where the expected prediction (label or ground truth) is

template.

   only available for some samples provided as training data when
   *fitting* the model.  We conventionally apply the label "-1" to
   *unlabeled* samples in semi-supervised classification.

sparse matrix
sparse graph
   A representation of two-dimensional numeric data that is more
   memory efficient the corresponding dense numpy array where almost
   all elements are zero. We use the "scipy.sparse" framework, which
   provides several underlying sparse data representations, or
   *formats*. Some formats are more efficient than others for
   particular tasks, and when a particular format provides especial
   benefit, we try to document this fact in Scikit-learn parameter
   descriptions.

   Some sparse matrix formats (notably CSR, CSC, COO and LIL)
   distinguish between *implicit* and *explicit* zeros. Explicit zeros
   are stored (i.e. they consume memory in a "data" array) in the data
   structure, while implicit zeros correspond to every element not
   otherwise defined in explicit storage.

   Two semantics for sparse matrices are used in Scikit-learn:

   matrix semantics
      The sparse matrix is interpreted as an array with implicit and
      explicit zeros being interpreted as the number 0.  This is the
      interpretation most often adopted, e.g. when sparse matrices are
      used for feature matrices or *multilabel indicator matrices*.

   graph semantics

interpretation most often adopted, e.g. when sparse matrices are

      As with "scipy.sparse.csgraph", explicit zeros are interpreted
      as the number 0, but implicit zeros indicate a masked or absent
      value, such as the absence of an edge between two vertices of a
      graph, where an explicit value indicates an edge's weight. This
      interpretation is adopted to represent connectivity in
      clustering, in representations of nearest neighborhoods (e.g.
      "neighbors.kneighbors_graph()"), and for precomputed distance
      representation where only distances in the neighborhood of each
      point are required.

   When working with sparse matrices, we assume that it is sparse for
   a good reason, and avoid writing code that densifies a user-
   provided sparse matrix, instead maintaining sparsity or raising an
   error if not possible (i.e. if an estimator does not / cannot
   support sparse matrices).

stateless
   An estimator is stateless if it does not store any information that
   is obtained during *fit*. This information can be either parameters
   learned during *fit* or statistics computed from the training data.
   An estimator is stateless if it has no *attributes* apart from ones
   set in *__init__*. Calling *fit* for these estimators will only
   validate the public *attributes* passed in *__init__*.

supervised
supervised learning

set in *__init__*. Calling *fit* for these estimators will only

   available for each sample when *fitting* the model, provided as
   *y*.  This is the approach taken in a *classifier* or *regressor*
   among other estimators.

target
targets
   The *dependent variable* in *supervised* (and *semisupervised*)
   learning, passed as *y* to an estimator's *fit* method.  Also known
   as *dependent variable*, *outcome variable*, *response variable*,
   *ground truth* or *label*. Scikit-learn works with targets that
   have minimal structure: a class from a finite set, a finite real-
   valued number, multiple classes, or multiple numbers. See Target
   Types.

transduction
transductive
   A transductive (contrasted with *inductive*) machine learning
   method is designed to model a specific dataset, but not to apply
   that model to unseen data.  Examples include "manifold.TSNE",
   "cluster.AgglomerativeClustering" and
   "neighbors.LocalOutlierFactor".

unlabeled
unlabeled data
   Samples with an unknown ground truth when fitting; equivalently,
   *missing values* in the *target*.  See also *semisupervised* and
   *unsupervised* learning.

unsupervised
unsupervised learning
   not available for each sample when *fitting* the model, as in

*unsupervised* learning.

   *clusterers* and *outlier detectors*.  Unsupervised estimators
   ignore any *y* passed to *fit*.

Class APIs and Estimator Types
==============================

classifier
classifiers
   A *supervised* (or *semi-supervised*) *predictor* with a finite set
   of discrete possible output values.

   A classifier supports modeling some of *binary*, *multiclass*,
   *multilabel*, or *multiclass multioutput* targets.  Within scikit-
   learn, all classifiers support multi-class classification,
   defaulting to using a one-vs-rest strategy over the binary
   classification problem.

   Classifiers must store a *classes_* attribute after fitting, and
   usually inherit from "base.ClassifierMixin", which sets their
   *_estimator_type* attribute.

   A classifier can be distinguished from other estimators with
   "is_classifier()".

   A classifier must implement:

   * *fit*

   * *predict*

   * *score*

   It may also be appropriate to implement *decision_function*,
   *predict_proba* and *predict_log_proba*.

clusterer
clusterers
   A *unsupervised* *predictor* with a finite set of discrete output
   values.

   A clusterer usually stores *labels_* after fitting, and must do so
   if it is *transductive*.

   A clusterer must implement:

   * *fit*

   * *fit_predict* if *transductive*

   * *predict* if *inductive*

density estimator

if it is *transductive*.

   * *fit*

   An *unsupervised* estimation of input probability density function.
   Commonly used techniques are:

   * Kernel Density Estimation - uses a kernel function, controlled by
     the bandwidth parameter to represent density;

   * Gaussian mixture - uses mixture of Gaussian models to represent
     density.

estimator
estimators
   An object which manages the estimation and decoding of a model. The
   model is estimated as a deterministic function of:

   * *parameters* provided in object construction or with
     *set_params*;

   * the global "numpy.random" random state if the estimator's
     *random_state* parameter is set to None; and

   * any data or *sample properties* passed to the most recent call to
     *fit*, *fit_transform* or *fit_predict*, or data similarly passed
     in a sequence of calls to *partial_fit*.

   The estimated model is stored in public and private *attributes* on
   the estimator instance, facilitating decoding through prediction
   and transformation methods.

   Estimators must provide a *fit* method, and should provide
   *set_params* and *get_params*, although these are usually provided
   by inheritance from "base.BaseEstimator".

   The core functionality of some estimators may also be available as
   a *function*.

feature extractor
feature extractors
   A *transformer* which takes input where each sample is not

The core functionality of some estimators may also be available as

   represented as an *array-like* object of fixed length, and produces
   an *array-like* object of *features* for each sample (and thus a
   2-dimensional array-like for a set of samples).  In other words, it
   (lossily) maps a non-rectangular data representation into
   *rectangular* data.

   Feature extractors must implement at least:

   * *fit*

   * *transform*

   * *get_feature_names_out*

meta-estimator
meta-estimators
metaestimator
metaestimators
   An *estimator* which takes another estimator as a parameter.
   Examples include "pipeline.Pipeline",
   "model_selection.GridSearchCV", "feature_selection.SelectFromModel"
   and "ensemble.BaggingClassifier".

   In a meta-estimator's *fit* method, any contained estimators should
   be *cloned* before they are fit (although FIXME: Pipeline and
   FeatureUnion do not do this currently). An exception to this is
   that an estimator may explicitly document that it accepts a pre-
   fitted estimator (e.g. using "prefit=True" in
   "feature_selection.SelectFromModel"). One known issue with this is
   that the pre-fitted estimator will lose its model if the meta-
   estimator is cloned.  A meta-estimator should have "fit" called
   before prediction, even if all contained estimators are pre-fitted.

   In cases where a meta-estimator's primary behaviors (e.g. *predict*

before prediction, even if all contained estimators are pre-fitted.

   or *transform* implementation) are functions of
   prediction/transformation methods of the provided *base estimator*
   (or multiple base estimators), a meta-estimator should provide at
   least the standard methods provided by the base estimator.  It may
   not be possible to identify which methods are provided by the
   underlying estimator until the meta-estimator has been *fitted*
   (see also *duck typing*), for which
   "utils.metaestimators.available_if()" may help.  It should also
   provide (or modify) the *estimator tags* and *classes_* attribute
   provided by the base estimator.

   Meta-estimators should be careful to validate data as minimally as
   possible before passing it to an underlying estimator. This saves
   computation time, and may, for instance, allow the underlying
   estimator to easily work with data that is not *rectangular*.

outlier detector
outlier detectors
   An *unsupervised* binary *predictor* which models the distinction
   between core and outlying samples.

   Outlier detectors must implement:

   * *fit*

   Inductive outlier detectors may also implement *decision_function*
   to give a normalized inlier score where outliers have score below
   0.  *score_samples* may provide an unnormalized score per sample.

predictor
predictors

to give a normalized inlier score where outliers have score below

predictor
   An *estimator* supporting *predict* and/or *fit_predict*. This
   encompasses *classifier*, *regressor*, *outlier detector* and
   *clusterer*.

   In statistics, "predictors" refers to *features*.

regressor
regressors
   A *supervised* (or *semi-supervised*) *predictor* with *continuous*
   output values.

   Regressors usually inherit from "base.RegressorMixin", which sets
   their *_estimator_type* attribute.

   A regressor can be distinguished from other estimators with
   "is_regressor()".

   A regressor must implement:

   * *fit*

   * *score*

transformer
transformers
   An estimator supporting *transform* and/or *fit_transform*. A
   purely *transductive* transformer, such as "manifold.TSNE", may not
   implement "transform".

vectorizer
vectorizers
   See *feature extractor*.

There are further APIs specifically related to a small family of
estimators, such as:

cross-validation splitter
CV splitter
cross-validation generator
   A non-estimator family of classes used to split a dataset into a
   sequence of train and test portions (see Cross-validation:
   evaluating estimator performance), by providing *split* and
   *get_n_splits* methods. Note that unlike estimators, these do not
   have *fit* methods and do not provide *set_params* or *get_params*.
   Parameter validation may be performed in "__init__".

cross-validation estimator

have *fit* methods and do not provide *set_params* or *get_params*.

   An estimator that has built-in cross-validation capabilities to
   automatically select the best hyper-parameters (see the User
   Guide). Some example of cross-validation estimators are
   "ElasticNetCV" and "LogisticRegressionCV". Cross-validation
   estimators are named *EstimatorCV* and tend to be roughly
   equivalent to *GridSearchCV(Estimator(), ...)*. The advantage of
   using a cross-validation estimator over the canonical *estimator*
   class along with grid search is that they can take advantage of
   warm-starting by reusing precomputed results in the previous steps
   of the cross-validation process. This generally leads to speed
   improvements. An exception is the "RidgeCV" class, which can
   instead perform efficient Leave-One-Out (LOO) CV. By default, all
   these estimators, apart from "RidgeCV" with an LOO-CV, will be
   refitted on the full training dataset after finding the best
   combination of hyper-parameters.

scorer
   A non-estimator callable object which evaluates an estimator on
   given test data, returning a number. Unlike *evaluation metrics*, a
   greater returned number must correspond with a *better* score. See
   The scoring parameter: defining model evaluation rules.

Further examples:

* "metrics.DistanceMetric"

* "gaussian_process.kernels.Kernel"

* "tree.Criterion"

Target Types

binary

The scoring parameter: defining model evaluation rules.

binary
   A classification problem consisting of two classes.  A binary
   target may  be represented as for a *multiclass* problem but with
   only two labels.  A binary decision function is represented as a 1d
   array.

   Semantically, one class is often considered the "positive" class.
   Unless otherwise specified (e.g. using *pos_label* in *evaluation
   metrics*), we consider the class label with the greater value
   (numerically or lexicographically) as the positive class: of labels
   [0, 1], 1 is the positive class; of [1, 2], 2 is the positive
   class; of ['no', 'yes'], 'yes' is the positive class; of ['no',
   'YES'], 'no' is the positive class.  This affects the output of
   *decision_function*, for instance.

   Note that a dataset sampled from a multiclass "y" or a continuous
   "y" may appear to be binary.

   "type_of_target()" will return 'binary' for binary input, or a
   similar array with only a single class present.

continuous
   A regression problem where each sample's target is a finite
   floating point number represented as a 1-dimensional array of
   floats (or sometimes ints).

   "type_of_target()" will return 'continuous' for continuous input,
   but if the data is all integers, it will be identified as
   'multiclass'.

continuous multioutput
continuous multi-output
multioutput continuous

but if the data is all integers, it will be identified as

multi-output continuous
   A regression problem where each sample's target consists of
   "n_outputs" *outputs*, each one a finite floating point number, for
   a fixed int "n_outputs > 1" in a particular dataset.

   Continuous multioutput targets are represented as multiple
   *continuous* targets, horizontally stacked into an array of shape
   "(n_samples, n_outputs)".

   "type_of_target()" will return 'continuous-multioutput' for
   continuous multioutput input, but if the data is all integers, it
   will be identified as 'multiclass-multioutput'.

multiclass
multi-class
   A classification problem consisting of more than two classes.  A
   multiclass target may be represented as a 1-dimensional array of
   strings or integers.  A 2d column vector of integers (i.e. a single
   output in *multioutput* terms) is also accepted.

   We do not officially support other orderable, hashable objects as
   class labels, even if estimators may happen to work when given
   classification targets of such type.

   For semi-supervised classification, *unlabeled* samples should have
   the special label -1 in "y".

   Within scikit-learn, all estimators supporting binary
   classification also support multiclass classification, using One-
   vs-Rest by default.

   A "preprocessing.LabelEncoder" helps to canonicalize multiclass
   targets as integers.

classification also support multiclass classification, using One-

   "type_of_target()" will return 'multiclass' for multiclass input.
   The user may also want to handle 'binary' input identically to

multiclass multioutput
multi-class multi-output
multioutput multiclass
multi-output multi-class
   A classification problem where each sample's target consists of
   "n_outputs" *outputs*, each a class label, for a fixed int
   "n_outputs > 1" in a particular dataset.  Each output has a fixed
   set of available classes, and each sample is labeled with a class
   for each output. An output may be binary or multiclass, and in the
   case where all outputs are binary, the target is *multilabel*.

   Multiclass multioutput targets are represented as multiple
   *multiclass* targets, horizontally stacked into an array of shape

   XXX: For simplicity, we may not always support string class labels
   for multiclass multioutput, and integer class labels should be
   used.

   "multioutput" provides estimators which estimate multi-output
   problems using multiple single-output estimators.  This may not
   fully account for dependencies among the different outputs, which
   methods natively handling the multioutput case (e.g. decision
   trees, nearest neighbors, neural networks) may do better.

fully account for dependencies among the different outputs, which

   "type_of_target()" will return 'multiclass-multioutput' for
   multiclass multioutput input.

multilabel
multi-label
   A *multiclass multioutput* target where each output is *binary*.
   This may be represented as a 2d (dense) array or sparse matrix of
   integers, such that each column is a separate binary target, where
   positive labels are indicated with 1 and negative labels are
   usually -1 or 0.  Sparse multilabel targets are not supported
   everywhere that dense multilabel targets are supported.

   Semantically, a multilabel target can be thought of as a set of
   labels for each sample.  While not used internally,
   "preprocessing.MultiLabelBinarizer" is provided as a utility to
   convert from a list of sets representation to a 2d array or sparse
   matrix. One-hot encoding a multiclass target with
   "preprocessing.LabelBinarizer" turns it into a multilabel problem.

   "type_of_target()" will return 'multilabel-indicator' for
   multilabel input, whether sparse or dense.

multioutput
multi-output
   A target where each sample has multiple classification/regression
   labels. See *multiclass multioutput* and *continuous multioutput*.
   We do not currently support modelling mixed classification and
   regression targets.

Methods
=======

"decision_function"

labels. See *multiclass multioutput* and *continuous multioutput*.

Methods
=======

   In a fitted *classifier* or *outlier detector*, predicts a "soft"
   score for each sample in relation to each class, rather than the
   "hard" categorical prediction produced by *predict*.  Its input is
   usually only some observed data, *X*.

   If the estimator was not already *fitted*, calling this method
   should raise a "exceptions.NotFittedError".

   Output conventions:

   binary classification
      A 1-dimensional array, where values strictly greater than zero
      indicate the positive class (i.e. the last class in *classes_*).

   multiclass classification
      A 2-dimensional array, where the row-wise arg-maximum is the
      predicted class.  Columns are ordered according to *classes_*.

   multilabel classification
      Scikit-learn is inconsistent in its representation of
      *multilabel* decision functions. It may be represented one of
      two ways:

      * List of 2d arrays, each array of shape: (*n_samples*, 2), like
        in multiclass multioutput. List is of length *n_labels*.

      * Single 2d array of shape (*n_samples*, *n_labels*), with each
        'column' in the array corresponding to the individual binary
        classification decisions. This is identical to the multiclass
        classification format, though its semantics differ: it should

classification decisions. This is identical to the multiclass
        be interpreted, like in the binary case, by thresholding at 0.

   multioutput classification
      A list of 2d arrays, corresponding to each multiclass decision
      function.

   outlier detection
      A 1-dimensional array, where a value greater than or equal to
      zero indicates an inlier.

"fit"
   The "fit" method is provided on every estimator. It usually takes
   some *samples* "X", *targets* "y" if the model is supervised, and
   potentially other *sample properties* such as *sample_weight*.  It
   should:

   * clear any prior *attributes* stored on the estimator, unless
     *warm_start* is used;

   * validate and interpret any *parameters*, ideally raising an error
     if invalid;

   * validate the input data;

   * estimate and store model attributes from the estimated parameters
     and provided data; and

   * return the now *fitted* estimator to facilitate method chaining.

   Target Types describes possible formats for "y".

"fit_predict"
   Used especially for *unsupervised*, *transductive* estimators, this
   fits the model and returns the predictions (similar to *predict*)
   on the training data. In clusterers, these predictions are also
   stored in the *labels_* attribute, and the output of
   ".fit_predict(X)" is usually equivalent to ".fit(X).predict(X)".
   The parameters to "fit_predict" are the same as those to "fit".

stored in the *labels_* attribute, and the output of

"fit_transform"
   A method on *transformers* which fits the estimator and returns the
   transformed training data. It takes parameters as in *fit* and its
   output should have the same shape as calling ".fit(X,
   ...).transform(X)". There are nonetheless rare cases where
   ".fit_transform(X, ...)" and ".fit(X, ...).transform(X)" do not
   return the same value, wherein training data needs to be handled
   differently (due to model blending in stacked ensembles, for
   instance; such cases should be clearly documented). *Transductive*
   transformers may also provide "fit_transform" but not *transform*.

   One reason to implement "fit_transform" is that performing "fit"
   and "transform" separately would be less efficient than together.
   "base.TransformerMixin" provides a default implementation,
   providing a consistent interface across transformers where
   "fit_transform" is or is not specialized.

   In *inductive* learning -- where the goal is to learn a generalized
   model that can be applied to new data -- users should be careful
   not to apply "fit_transform" to the entirety of a dataset (i.e.
   training and test data together) before further modelling, as this
   results in *data leakage*.

"get_feature_names_out"
   Primarily for *feature extractors*, but also used for other

training and test data together) before further modelling, as this

   transformers to provide string names for each column in the output
   of the estimator's *transform* method.  It outputs an array of
   strings and may take an array-like of strings as input,
   corresponding to the names of input columns from which output
   column names can be generated.  If *input_features* is not passed
   in, then the *feature_names_in_* attribute will be used. If the
   *feature_names_in_* attribute is not defined, then the input names
   are named *[x0, x1, ..., x(n_features_in_ - 1)]*.

"get_n_splits"
   On a *CV splitter* (not an estimator), returns the number of
   elements one would get if iterating through the return value of
   *split* given the same parameters.  Takes the same parameters as
   split.

"get_params"
   Gets all *parameters*, and their values, that can be set using
   *set_params*.  A parameter "deep" can be used, when set to False to
   only return those parameters not including "__", i.e.  not due to
   indirection via contained estimators.

   Most estimators adopt the definition from "base.BaseEstimator",
   which simply adopts the parameters defined for "__init__".
   "pipeline.Pipeline", among others, reimplements "get_params" to
   declare the estimators named in its "steps" parameters as
   themselves being parameters.

"partial_fit"

"pipeline.Pipeline", among others, reimplements "get_params" to

   Facilitates fitting an estimator in an online fashion.  Unlike
   "fit", repeatedly calling "partial_fit" does not clear the model,
   but updates it with the data provided. The portion of data provided
   to "partial_fit" may be called a mini-batch. Each mini-batch must
   be of consistent shape, etc. In iterative estimators, "partial_fit"
   often only performs a single iteration.

   "partial_fit" may also be used for *out-of-core* learning, although
   usually limited to the case where learning can be performed online,
   i.e. the model is usable after each "partial_fit" and there is no
   separate processing needed to finalize the model. "cluster.Birch"
   introduces the convention that calling "partial_fit(X)" will
   produce a model that is not finalized, but the model can be
   finalized by calling "partial_fit()" i.e. without passing a further
   mini-batch.

   Generally, estimator parameters should not be modified between
   calls to "partial_fit", although "partial_fit" should validate them
   as well as the new mini-batch of data.  In contrast, "warm_start"
   is used to repeatedly fit the same estimator with the same data but
   varying parameters.

   Like "fit", "partial_fit" should return the estimator object.

   To clear the model, a new estimator should be constructed, for
   instance with "base.clone()".

varying parameters.

   NOTE: Using "partial_fit" after "fit" results in undefined
   behavior.

"predict"
   Makes a prediction for each sample, usually only taking *X* as
   input (but see under regressor output conventions below). In a
   *classifier* or *regressor*, this prediction is in the same target
   space used in fitting (e.g. one of {'red', 'amber', 'green'} if the
   "y" in fitting consisted of these strings).  Despite this, even
   when "y" passed to *fit* is a list or other array-like, the output
   of "predict" should always be an array or sparse matrix. In a
   *clusterer* or *outlier detector* the prediction is an integer.

   classifier
      An array of shape "(n_samples,)" "(n_samples, n_outputs)".
      *Multilabel* data may be represented as a sparse matrix if a
      sparse matrix was used in fitting. Each element should be one of
      the values in the classifier's *classes_* attribute.

   clusterer
      An array of shape "(n_samples,)" where each value is from 0 to
      "n_clusters - 1" if the corresponding sample is clustered, and
      -1 if the sample is not clustered, as in "cluster.dbscan()".

   outlier detector

"n_clusters - 1" if the corresponding sample is clustered, and

      An array of shape "(n_samples,)" where each value is -1 for an
      outlier and 1 otherwise.

   regressor
      A numeric array of shape "(n_samples,)", usually float64. Some
      regressors have extra options in their "predict" method,
      allowing them to return standard deviation ("return_std=True")
      or covariance ("return_cov=True") relative to the predicted
      value.  In this case, the return value is a tuple of arrays
      corresponding to (prediction mean, std, cov) as required.

"predict_log_proba"
   The natural logarithm of the output of *predict_proba*, provided to
   facilitate numerical stability.

"predict_proba"
   A method in *classifiers* and *clusterers* that can return
   probability estimates for each class/cluster.  Its input is usually
   only some observed data, *X*.

   Output conventions are like those for *decision_function* except in
   the *binary* classification case, where one column is output for
   each class (while "decision_function" outputs a 1d array). For
   binary and multiclass predictions, each row should add to 1.

   Like other methods, "predict_proba" should only be present when the
   estimator can make probabilistic predictions (see *duck typing*).

Like other methods, "predict_proba" should only be present when the
   This means that the presence of the method may depend on estimator
   parameters (e.g. in "linear_model.SGDClassifier") or training data
   (e.g. in "model_selection.GridSearchCV") and may only appear after
   fitting.

"score"
   A method on an estimator, usually a *predictor*, which evaluates
   its predictions on a given dataset, and returns a single numerical
   score.  A greater return value should indicate better predictions;
   accuracy is used for classifiers and R^2 for regressors by default.

   Some estimators implement a custom, estimator-specific score
   function, often the likelihood of the data under the model.

"score_samples"
   A method that returns a score for each given sample. The exact
   definition of *score* varies from one class to another. In the case
   of density estimation, it can be the log density model on the data,
   and in the case of outlier detection, it can be the opposite of the
   outlier factor of the data.

"set_params"
   Available in any estimator, takes keyword arguments corresponding
   to keys in *get_params*.  Each is provided a new value to assign

should raise a "exceptions.NotFittedError".

   such that calling "get_params" after "set_params" will reflect the
   changed *parameters*.  Most estimators use the implementation in
   "base.BaseEstimator", which handles nested parameters and otherwise
   sets the parameter as an attribute on the estimator. The method is
   overridden in "pipeline.Pipeline" and related estimators.

"split"
   On a *CV splitter* (not an estimator), this method accepts
   parameters (*X*, *y*, *groups*), where all may be optional, and
   returns an iterator over "(train_idx, test_idx)" pairs.  Each of
   {train,test}_idx is a 1d integer array, with values from 0 from
   "X.shape[0] - 1" of any length, such that no values appear in both
   some "train_idx" and its corresponding "test_idx".

"transform"
   In a *transformer*, transforms the input, usually only *X*, into
   some transformed space (conventionally notated as *Xt*). Output is
   an array or sparse matrix of length *n_samples* and with the number
   of columns fixed after *fitting*.

Parameters

These common parameter names, specifically used in estimator
construction (see concept *parameter*), sometimes also appear as
parameters of functions or non-estimator constructors.

"class_weight"

   Used to specify sample weights when fitting classifiers as a
   function of the *target* class.  Where *sample_weight* is also
   supported and given, it is multiplied by the "class_weight"
   contribution. Similarly, where "class_weight" is used in a
   *multioutput* (including *multilabel*) tasks, the weights are
   multiplied across outputs (i.e. columns of "y").

   By default, all samples have equal weight such that classes are
   effectively weighted by their prevalence in the training data. This
   could be achieved explicitly with "class_weight={label1: 1, label2:
   1, ...}" for all class labels.

   More generally, "class_weight" is specified as a dict mapping class
   labels to weights ("{class_label: weight}"), such that each sample
   of the named class is given that weight.

   "class_weight='balanced'" can be used to give all classes equal
   weight by giving each sample a weight inversely related to its
   class's prevalence in the training data: "n_samples / (n_classes *
   np.bincount(y))". Class weights will be used differently depending
   on the algorithm: for linear models (such as linear SVM or logistic
   regression), the class weights will alter the loss function by
   weighting the loss of each sample by its class weight. For tree-

regression), the class weights will alter the loss function by
   based algorithms, the class weights will be used for reweighting
   the splitting criterion. **Note** however that this rebalancing
   does not take the weight of samples in each class into account.

   For multioutput classification, a list of dicts is used to specify
   weights for each output. For example, for four-class multilabel
   classification weights should be "[{0: 1, 1: 1}, {0: 1, 1: 5}, {0:
   1, 1: 1}, {0: 1, 1: 1}]" instead of "[{1:1}, {2:5}, {3:1}, {4:1}]".

   The "class_weight" parameter is validated and interpreted with
   "utils.compute_class_weight()".

"cv"
   Determines a cross validation splitting strategy, as used in cross-
   validation based routines. "cv" is also available in estimators
   such as "multioutput.ClassifierChain" or
   "calibration.CalibratedClassifierCV" which use the predictions of
   one estimator as training data for another, to not overfit the
   training supervision.

   Possible inputs for "cv" are usually:

   * An integer, specifying the number of folds in K-fold cross
     validation. K-fold will be stratified over classes if the
     estimator is a classifier (determined by "base.is_classifier()")
     and the *targets* may represent a binary or multiclass (but not
     multioutput) classification problem (determined by
     "utils.multiclass.type_of_target()").

and the *targets* may represent a binary or multiclass (but not

   * A *cross-validation splitter* instance. Refer to the User Guide
     for splitters available within Scikit-learn.

   * An iterable yielding train/test splits.

   With some exceptions (especially where not using cross validation
   at all is an option), the default is 5-fold.

   "cv" values are validated and interpreted with "utils.check_cv()".

"kernel"
   Specifies the kernel function to be used by Kernel Method
   algorithms. For example, the estimators "SVC" and
   "GaussianProcessClassifier" both have a "kernel" parameter that
   takes the name of the kernel to use as string or a callable kernel
   function used to compute the kernel matrix. For more reference, see
   the Kernel Approximation and the Gaussian Processes user guides.

"max_iter"
   For estimators involving iterative optimization, this determines
   the maximum number of iterations to be performed in *fit*.  If
   "max_iter" iterations are run without convergence, a
   "exceptions.ConvergenceWarning" should be raised.  Note that the
   interpretation of "a single iteration" is inconsistent across
   estimators: some, but not all, use it to mean a single epoch (i.e.
   a pass over every sample in the data).

   FIXME perhaps we should have some common tests about the
   relationship between ConvergenceWarning and max_iter.

"memory"

a pass over every sample in the data).

"memory"
   Some estimators make use of "joblib.Memory" to store partial
   solutions during fitting. Thus when "fit" is called again, those
   partial solutions have been memoized and can be reused.

   A "memory" parameter can be specified as a string with a path to a
   directory, or a "joblib.Memory" instance (or an object with a
   similar interface, i.e. a "cache" method) can be used.

   "memory" values are validated and interpreted with
   "utils.validation.check_memory()".

"metric"
   As a parameter, this is the scheme for determining the distance
   between two data points.  See "metrics.pairwise_distances()".  In
   practice, for some algorithms, an improper distance metric (one
   that does not obey the triangle inequality, such as Cosine
   Distance) may be used.

   XXX: hierarchical clustering uses "affinity" with this meaning.

   We also use *metric* to refer to *evaluation metrics*, but avoid
   using this sense as a parameter name.

"n_components"
   The number of features which a *transformer* should transform the
   input into. See *components_* for the special case of affine
   projection.

"n_iter_no_change"
   Number of iterations with no improvement to wait before stopping
   the iterative procedure. This is also known as a *patience*
   parameter. It is typically used with *early stopping* to avoid
   stopping too early.

the iterative procedure. This is also known as a *patience*

"n_jobs"
   This parameter is used to specify how many concurrent processes or
   threads should be used for routines that are parallelized with
   *joblib*.

   "n_jobs" is an integer, specifying the maximum number of
   concurrently running workers. If 1 is given, no joblib parallelism
   is used at all, which is useful for debugging. If set to -1, all
   CPUs are used. For "n_jobs" below -1, (n_cpus + 1 + n_jobs) are
   used. For example with "n_jobs=-2", all CPUs but one are used.

   "n_jobs" is "None" by default, which means *unset*; it will
   generally be interpreted as "n_jobs=1", unless the current
   "joblib.Parallel" backend context specifies otherwise.

   Note that even if "n_jobs=1", low-level parallelism (via Numpy and
   OpenMP) might be used in some configuration.

   For more details on the use of "joblib" and its interactions with
   scikit-learn, please refer to our parallelism notes.

"pos_label"
   Value with which positive labels must be encoded in binary
   classification problems in which the positive class is not assumed.
   This value is typically required to compute asymmetric evaluation
   metrics such as precision and recall.

"random_state"
   Whenever randomization is part of a Scikit-learn algorithm, a
   "random_state" parameter may be provided to control the random
   number generator used.  Note that the mere presence of

   "random_state" doesn't mean that randomization is always used, as
   it may be dependent on another parameter, e.g. "shuffle", being
   set.

   The passed value will have an effect on the reproducibility of the
   results returned by the function (*fit*, *split*, or any other
   function like "k_means()"). *random_state*'s value may be:

   None (default)
      Use the global random state instance from "numpy.random".
      Calling the function multiple times will reuse the same
      instance, and will produce different results.

   An integer
      Use a new random number generator seeded by the given integer.
      Using an int will produce the same results across different
      calls. However, it may be worthwhile checking that your results
      are stable across a number of different distinct random seeds.
      Popular integer random seeds are 0 and 42. Integer values must
      be in the range *[0, 2**32 - 1]*.

   A "numpy.random.RandomState" instance
      Use the provided random state, only affecting other users of
      that same random state instance. Calling the function multiple
      times will reuse the same instance, and will produce different
      results.

   "utils.check_random_state()" is used internally to validate the

times will reuse the same instance, and will produce different
      results.

   input "random_state" and return a "RandomState" instance.

   For more details on how to control the randomness of scikit-learn
   objects and avoid common pitfalls, you may refer to Controlling
   randomness.

"scoring"
   Specifies the score function to be maximized (usually by cross
   validation), or -- in some cases -- multiple score functions to be
   reported. The score function can be a string accepted by
   "metrics.get_scorer()" or a callable *scorer*, not to be confused
   with an *evaluation metric*, as the latter have a more diverse API.
   "scoring" may also be set to None, in which case the estimator's
   *score* method is used.  See The scoring parameter: defining model
   evaluation rules in the User Guide.

   Where multiple metrics can be evaluated, "scoring" may be given
   either as a list of unique strings, a dictionary with names as keys
   and callables as values or a callable that returns a dictionary.
   Note that this does *not* specify which score function is to be
   maximized, and another parameter such as "refit" maybe used for
   this purpose.

   The "scoring" parameter is validated and interpreted using
   "metrics.check_scoring()".

"verbose"
   Logging is not handled very consistently in Scikit-learn at
   present, but when it is provided as an option, the "verbose"

"metrics.check_scoring()".

"verbose"
   parameter is usually available to choose no logging (set to False).
   Any True value should enable some logging, but larger integers
   (e.g. above 10) may be needed for full verbosity.  Verbose logs are
   usually printed to Standard Output. Estimators should not produce
   any output on Standard Output with the default "verbose" setting.

"warm_start"
   When fitting an estimator repeatedly on the same dataset, but for
   multiple parameter values (such as to find the value maximizing
   performance as in grid search), it may be possible to reuse aspects
   of the model learned from the previous parameter value, saving
   time.  When "warm_start" is true, the existing *fitted* model
   *attributes* are used to initialize the new model in a subsequent
   call to *fit*.

   Note that this is only applicable for some models and some
   parameters, and even some orders of parameter values. For example,
   "warm_start" may be used when building random forests to add more
   trees to the forest (increasing "n_estimators") but not to reduce
   their number.

   *partial_fit* also retains the model between calls, but differs:
   with "warm_start" the parameters change and the data is (more-or-
   less) constant across calls to "fit"; with "partial_fit", the mini-
   batch of data changes and model parameters stay fixed.

with "warm_start" the parameters change and the data is (more-or-

   There are cases where you want to use "warm_start" to fit on
   different, but closely related data. For example, one may initially
   fit to a subset of the data, then fine-tune the parameter search on
   the full dataset. For classification, all data in a sequence of
   "warm_start" calls to "fit" must include samples from each class.

Attributes

See concept *attribute*.

"classes_"
   A list of class labels known to the *classifier*, mapping each
   label to a numerical index used in the model representation our
   output. For instance, the array output from *predict_proba* has
   columns aligned with "classes_". For *multi-output* classifiers,
   "classes_" should be a list of lists, with one class listing for
   each output.  For each output, the classes should be sorted
   (numerically, or lexicographically for strings).

   "classes_" and the mapping to indices is often managed with
   "preprocessing.LabelEncoder".

"components_"
   An affine transformation matrix of shape "(n_components,
   n_features)" used in many linear *transformers* where
   *n_components* is the number of output features and *n_features* is
   the number of input features.

   See also *components_* which is a similar attribute for linear
   predictors.

"coef_"

*n_components* is the number of output features and *n_features* is

"coef_"
   The weight/coefficient matrix of a generalized linear model
   *predictor*, of shape "(n_features,)" for binary classification and
   single-output regression, "(n_classes, n_features)" for multiclass
   classification and "(n_targets, n_features)" for multi-output
   regression. Note this does not include the intercept (or bias)
   term, which is stored in "intercept_".

   When available, "feature_importances_" is not usually provided as
   well, but can be calculated as the  norm of each feature's entry in
   "coef_".

   transformers.

"embedding_"
   An embedding of the training data in manifold learning estimators,
   with shape "(n_samples, n_components)", identical to the output of
   *fit_transform*.  See also *labels_*.

"n_iter_"
   The number of iterations actually performed when fitting an
   iterative estimator that may stop upon convergence. See also
   *max_iter*.

"feature_importances_"
   A vector of shape "(n_features,)" available in some *predictors* to
   provide a relative measure of the importance of each feature in the
   predictions of the model.

"labels_"
   A vector containing a cluster label for each sample of the training
   data in *clusterers*, identical to the output of *fit_predict*.
   See also *embedding_*.

predictions of the model.

"labels_"

Data and sample properties

See concept *sample property*.

"groups"
   Used in cross-validation routines to identify samples that are
   correlated. Each value is an identifier such that, in a supporting
   *CV splitter*, samples from some "groups" value may not appear in
   both a training set and its corresponding test set. See Cross-
   validation iterators for grouped data.

"sample_weight"
   A relative weight for each sample.  Intuitively, if all weights are
   integers, a weighted model or score should be equivalent to that
   calculated when repeating the sample the number of times specified
   in the weight.  Weights may be specified as floats, so that sample
   weights are usually equivalent up to a constant positive scaling
   factor.

   FIXME  Is this interpretation always the case in practice? We have
   no common tests.

   Some estimators, such as decision trees, support negative weights.
   FIXME: This feature or its absence may not be tested or documented
   in many estimators.

   This is not entirely the case where other parameters of the model
   consider the number of samples in a region, as with "min_samples"
   in "cluster.DBSCAN".  In this case, a count of samples becomes to a
   sum of their weights.

consider the number of samples in a region, as with "min_samples"

   In classification, sample weights can also be specified as a
   function of class with the *class_weight* estimator *parameter*.

"X"
   Denotes data that is observed at training and prediction time, used
   as independent variables in learning.  The notation is uppercase to
   denote that it is ordinarily a matrix (see *rectangular*). When a
   matrix, each sample may be represented by a *feature* vector, or a
   vector of *precomputed* (dis)similarity with each training sample.
   "X" may also not be a matrix, and may require a *feature extractor*
   or a *pairwise metric* to turn it into one before learning a model.

"Xt"
   Shorthand for "transformed *X*".

"y"
"Y"
   Denotes data that may be observed at training time as the dependent
   variable in learning, but which is unavailable at prediction time,
   and is usually the *target* of prediction.  The notation may be
   uppercase to denote that it is a matrix, representing *multi-
   output* targets, for instance; but usually we use "y" and sometimes
   do so even when multiple outputs are assumed.

8. Computing with scikit-learn
******************************

* 8.1. Strategies to scale computationally: bigger data

  * 8.1.1. Scaling with instances using out-of-core learning

* 8.2. Computational Performance

  * 8.2.1. Prediction Latency

  * 8.2.2. Prediction Throughput

  * 8.2.3. Tips and Tricks

* 8.3. Parallelism, resource management, and configuration

  * 8.3.1. Parallelism

  * 8.3.2. Configuration switches

Welcome to scikit-learn

* Installing scikit-learn

  * Installing the latest release

  * Installing on Apple Silicon M1 hardware

  * Third party distributions of scikit-learn

  * Troubleshooting

* Frequently Asked Questions

  * About the project

  * Implementation decisions

  * Using scikit-learn

* Support

  * Mailing List

  * User questions

  * Bug tracker

  * Gitter

  * Documentation resources

* Related Projects

  * Interoperability and framework enhancements

  * Other estimators and tasks

  * Statistical learning with Python

  * Translations of scikit-learn documentation

* About us

  * History

  * Governance

  * Authors

  * Contributor Experience Team

  * Communication Team

  * Emeritus Core Developers

  * Emeritus Communication Team

  * Citing scikit-learn

  * Artwork

  * Funding

  * Sprints

  * Infrastructure support

* Who is using scikit-learn?

  * J.P.Morgan

  * Spotify

  * Inria

  * betaworks

  * Hugging Face

  * Evernote

  * Télécom ParisTech

  * Booking.com

  * AWeber

  * Yhat

  * Rangespan

  * Birchbox

  * Bestofmedia Group

  * Change.org

  * PHIMECA Engineering

  * HowAboutWe

  * PeerIndex

  * DataRobot

  * OkCupid

  * Lovely

  * Data Publica

  * Machinalis

  * solido

  * INFONEA

  * Dataiku

  * Otto Group

  * Zopa

  * MARS

  * BNP Paribas Cardif

* Release History

  * Version 1.3

  * Version 1.2.2

  * Version 1.2.1

  * Version 1.2.0

  * Version 1.1.3

  * Version 1.1.2

  * Version 1.1.1

  * Version 1.1.0

  * Version 1.0.2

  * Version 1.0.1

  * Version 1.0.0

  * Version 0.24.2

  * Version 0.24.1

  * Version 0.24.0

  * Version 0.23.2

  * Version 0.23.1

  * Version 0.23.0

  * Version 0.22.2.post1

  * Version 0.22.1

  * Version 0.22.0

  * Version 0.21.3

  * Version 0.21.2

  * Version 0.21.1

  * Version 0.21.0

  * Version 0.20.4

  * Version 0.20.3

  * Version 0.20.2

  * Version 0.20.1

  * Version 0.20.0

  * Version 0.19.2

  * Version 0.19.1

  * Version 0.19

  * Version 0.18.2

  * Version 0.18.1

  * Version 0.18

  * Version 0.17.1

  * Version 0.17

  * Version 0.16.1

  * Version 0.16

  * Version 0.15.2

  * Version 0.15.1

  * Version 0.15

  * Version 0.14

  * Version 0.13.1

  * Version 0.13

  * Version 0.12.1

  * Version 0.12

  * Version 0.11

  * Version 0.10

  * Version 0.9

  * Version 0.8

  * Version 0.7

  * Version 0.6

  * Version 0.5

  * Version 0.4

  * Earlier versions

* Roadmap

  * Purpose of this document

  * Statement of purpose: Scikit-learn in 2018

  * Architectural / general goals

  * Subpackage-specific goals

* Scikit-learn governance and decision-making

  * Roles And Responsibilities

  * Decision Making Process

  * Enhancement proposals (SLEPs)

7. Dataset loading utilities
****************************

The "sklearn.datasets" package embeds some small toy datasets as
introduced in the Getting Started section.

This package also features helpers to fetch larger datasets commonly
used by the machine learning community to benchmark algorithms on data
that comes from the 'real world'.

To evaluate the impact of the scale of the dataset ("n_samples" and
"n_features") while controlling the statistical properties of the data
(typically the correlation and informativeness of the features), it is
also possible to generate synthetic data.

**General dataset API.** There are three main kinds of dataset
interfaces that can be used to get datasets depending on the desired
type of dataset.

**The dataset loaders.** They can be used to load small standard
datasets, described in the Toy datasets section.

**The dataset fetchers.** They can be used to download and load larger
datasets, described in the Real world datasets section.

Both loaders and fetchers functions return a "Bunch" object holding at
least two items: an array of shape "n_samples" * "n_features" with key
"data" (except for 20newsgroups) and a numpy array of length
"n_samples", containing the target values, with key "target".

The Bunch object is a dictionary that exposes its keys as attributes.
For more information about Bunch object, see "Bunch".

It's also possible for almost all of these function to constrain the
output to be a tuple containing only the data and the target, by

setting the "return_X_y" parameter to "True".

The datasets also contain a full description in their "DESCR"
attribute and some contain "feature_names" and "target_names". See the
dataset descriptions below for details.

**The dataset generation functions.** They can be used to generate
controlled synthetic datasets, described in the Generated datasets
section.

These functions return a tuple "(X, y)" consisting of a "n_samples" *
"n_features" numpy array "X" and an array of length "n_samples"
containing the targets "y".

In addition, there are also miscellaneous tools to load datasets of
other formats or from other locations, described in the Loading other
datasets section.

* 7.1. Toy datasets

* 7.2. Real world datasets

* 7.3. Generated datasets

  * 7.3.1. Generators for classification and clustering

  * 7.3.2. Generators for regression

  * 7.3.3. Generators for manifold learning

  * 7.3.4. Generators for decomposition

* 7.4. Loading other datasets

  * 7.4.1. Sample images

  * 7.4.2. Datasets in svmlight / libsvm format

  * 7.4.3. Downloading datasets from the openml.org repository

  * 7.4.4. Loading from external datasets

6. Dataset transformations

scikit-learn provides a library of transformers, which may clean (see
Preprocessing data), reduce (see Unsupervised dimensionality
reduction), expand (see Kernel Approximation) or generate (see Feature
extraction) feature representations.

Like other estimators, these are represented by classes with a "fit"
method, which learns model parameters (e.g. mean and standard
deviation for normalization) from a training set, and a "transform"
method which applies this transformation model to unseen data.
"fit_transform" may be more convenient and efficient for modelling and
transforming the training data simultaneously.

Combining such transformers, either in parallel or series is covered
in Pipelines and composite estimators. Pairwise metrics, Affinities
and Kernels covers transforming feature spaces into affinity matrices,
while Transforming the prediction target (y) considers transformations
of the target space (e.g. categorical labels) for use in scikit-learn.

* 6.1. Pipelines and composite estimators

  * 6.1.1. Pipeline: chaining estimators

  * 6.1.2. Transforming target in regression

  * 6.1.3. FeatureUnion: composite feature spaces

  * 6.1.4. ColumnTransformer for heterogeneous data

  * 6.1.5. Visualizing Composite Estimators

* 6.2. Feature extraction

  * 6.2.1. Loading features from dicts

  * 6.2.2. Feature hashing

  * 6.2.3. Text feature extraction

  * 6.2.4. Image feature extraction

* 6.3. Preprocessing data

  * 6.3.1. Standardization, or mean removal and variance scaling

  * 6.3.2. Non-linear transformation

  * 6.3.3. Normalization

  * 6.3.4. Encoding categorical features

  * 6.3.5. Discretization

  * 6.3.6. Imputation of missing values

  * 6.3.7. Generating polynomial features

  * 6.3.8. Custom transformers

* 6.4. Imputation of missing values

  * 6.4.1. Univariate vs. Multivariate Imputation

  * 6.4.2. Univariate feature imputation

  * 6.4.3. Multivariate feature imputation

  * 6.4.4. Nearest neighbors imputation

  * 6.4.5. Keeping the number of features constants

  * 6.4.6. Marking imputed values

  * 6.4.7. Estimators that handle NaN values

* 6.5. Unsupervised dimensionality reduction

  * 6.5.1. PCA: principal component analysis

  * 6.5.2. Random projections

  * 6.5.3. Feature agglomeration

* 6.6. Random Projection

  * 6.6.1. The Johnson-Lindenstrauss lemma

  * 6.6.2. Gaussian random projection

  * 6.6.3. Sparse random projection

  * 6.6.4. Inverse Transform

* 6.7. Kernel Approximation

  * 6.7.1. Nystroem Method for Kernel Approximation

  * 6.7.2. Radial Basis Function Kernel

  * 6.7.3. Additive Chi Squared Kernel

  * 6.7.4. Skewed Chi Squared Kernel

  * 6.7.5. Polynomial Kernel Approximation via Tensor Sketch

  * 6.7.6. Mathematical Details

* 6.8. Pairwise metrics, Affinities and Kernels

  * 6.8.1. Cosine similarity

* 6.7.5. Polynomial Kernel Approximation via Tensor Sketch

  * 6.8.2. Linear kernel

  * 6.8.3. Polynomial kernel

  * 6.8.4. Sigmoid kernel

  * 6.8.5. RBF kernel

  * 6.8.6. Laplacian kernel

  * 6.8.7. Chi-squared kernel

* 6.9. Transforming the prediction target ("y")

  * 6.9.1. Label binarization

  * 6.9.2. Label encoding

2. Unsupervised learning
************************

* 2.1. Gaussian mixture models

  * 2.1.1. Gaussian Mixture

  * 2.1.2. Variational Bayesian Gaussian Mixture

* 2.2. Manifold learning

  * 2.2.1. Introduction

  * 2.2.2. Isomap

  * 2.2.3. Locally Linear Embedding

  * 2.2.4. Modified Locally Linear Embedding

  * 2.2.5. Hessian Eigenmapping

  * 2.2.6. Spectral Embedding

  * 2.2.7. Local Tangent Space Alignment

  * 2.2.8. Multi-dimensional Scaling (MDS)

  * 2.2.9. t-distributed Stochastic Neighbor Embedding (t-SNE)

  * 2.2.10. Tips on practical use

* 2.3. Clustering

  * 2.3.1. Overview of clustering methods

  * 2.3.2. K-means

  * 2.3.3. Affinity Propagation

  * 2.3.4. Mean Shift

  * 2.3.5. Spectral clustering

  * 2.3.6. Hierarchical clustering

  * 2.3.7. DBSCAN

  * 2.3.8. OPTICS

  * 2.3.9. BIRCH

  * 2.3.10. Clustering performance evaluation

* 2.4. Biclustering

  * 2.4.1. Spectral Co-Clustering

  * 2.4.2. Spectral Biclustering

  * 2.4.3. Biclustering evaluation

* 2.5. Decomposing signals in components (matrix factorization
  problems)

  * 2.5.1. Principal component analysis (PCA)

  * 2.5.2. Kernel Principal Component Analysis (kPCA)

  * 2.5.3. Truncated singular value decomposition and latent semantic
    analysis

  * 2.5.4. Dictionary Learning

  * 2.5.5. Factor Analysis

  * 2.5.6. Independent component analysis (ICA)

  * 2.5.7. Non-negative matrix factorization (NMF or NNMF)

  * 2.5.8. Latent Dirichlet Allocation (LDA)

* 2.6. Covariance estimation

  * 2.6.1. Empirical covariance

  * 2.6.2. Shrunk Covariance

* 2.5.7. Non-negative matrix factorization (NMF or NNMF)

  * 2.6.3. Sparse inverse covariance

  * 2.6.4. Robust Covariance Estimation

* 2.7. Novelty and Outlier Detection

  * 2.7.1. Overview of outlier detection methods

  * 2.7.2. Novelty Detection

  * 2.7.3. Outlier Detection

  * 2.7.4. Novelty detection with Local Outlier Factor

* 2.8. Density Estimation

  * 2.8.1. Density Estimation: Histograms

  * 2.8.2. Kernel Density Estimation

* 2.9. Neural network models (unsupervised)

  * 2.9.1. Restricted Boltzmann machines

User Guide
**********

* 1. Supervised learning

  * 1.1. Linear Models

    * 1.1.1. Ordinary Least Squares

    * 1.1.2. Ridge regression and classification

    * 1.1.3. Lasso

    * 1.1.4. Multi-task Lasso

    * 1.1.5. Elastic-Net

    * 1.1.6. Multi-task Elastic-Net

    * 1.1.7. Least Angle Regression

    * 1.1.8. LARS Lasso

    * 1.1.9. Orthogonal Matching Pursuit (OMP)

    * 1.1.10. Bayesian Regression

    * 1.1.11. Logistic regression

    * 1.1.12. Generalized Linear Models

    * 1.1.13. Stochastic Gradient Descent - SGD

    * 1.1.14. Perceptron

    * 1.1.15. Passive Aggressive Algorithms

    * 1.1.16. Robustness regression: outliers and modeling errors

    * 1.1.17. Quantile Regression

    * 1.1.18. Polynomial regression: extending linear models with
      basis functions

  * 1.2. Linear and Quadratic Discriminant Analysis

    * 1.2.1. Dimensionality reduction using Linear Discriminant
      Analysis

    * 1.2.2. Mathematical formulation of the LDA and QDA classifiers

    * 1.2.3. Mathematical formulation of LDA dimensionality reduction

    * 1.2.4. Shrinkage and Covariance Estimator

    * 1.2.5. Estimation algorithms

  * 1.3. Kernel ridge regression

  * 1.4. Support Vector Machines

    * 1.4.1. Classification

    * 1.4.2. Regression

    * 1.4.3. Density estimation, novelty detection

    * 1.4.4. Complexity

    * 1.4.5. Tips on Practical Use

    * 1.4.6. Kernel functions

    * 1.4.7. Mathematical formulation

    * 1.4.8. Implementation details

  * 1.5. Stochastic Gradient Descent

    * 1.5.1. Classification

* 1.4.5. Tips on Practical Use

    * 1.5.2. Regression

    * 1.5.3. Online One-Class SVM

    * 1.5.4. Stochastic Gradient Descent for sparse data

    * 1.5.5. Complexity

    * 1.5.6. Stopping criterion

    * 1.5.7. Tips on Practical Use

    * 1.5.8. Mathematical formulation

    * 1.5.9. Implementation details

  * 1.6. Nearest Neighbors

    * 1.6.1. Unsupervised Nearest Neighbors

    * 1.6.2. Nearest Neighbors Classification

    * 1.6.3. Nearest Neighbors Regression

    * 1.6.4. Nearest Neighbor Algorithms

    * 1.6.5. Nearest Centroid Classifier

    * 1.6.6. Nearest Neighbors Transformer

    * 1.6.7. Neighborhood Components Analysis

  * 1.7. Gaussian Processes

    * 1.7.1. Gaussian Process Regression (GPR)

    * 1.7.2. GPR examples

    * 1.7.3. Gaussian Process Classification (GPC)

    * 1.7.4. GPC examples

    * 1.7.5. Kernels for Gaussian Processes

  * 1.8. Cross decomposition

    * 1.8.1. PLSCanonical

    * 1.8.2. PLSSVD

    * 1.8.3. PLSRegression

    * 1.8.4. Canonical Correlation Analysis

  * 1.9. Naive Bayes

    * 1.9.1. Gaussian Naive Bayes

    * 1.9.2. Multinomial Naive Bayes

    * 1.9.3. Complement Naive Bayes

    * 1.9.4. Bernoulli Naive Bayes

    * 1.9.5. Categorical Naive Bayes

    * 1.9.6. Out-of-core naive Bayes model fitting

  * 1.10. Decision Trees

    * 1.10.1. Classification

    * 1.10.2. Regression

* 1.9.4. Bernoulli Naive Bayes

    * 1.10.3. Multi-output problems

    * 1.10.4. Complexity

    * 1.10.5. Tips on practical use

    * 1.10.6. Tree algorithms: ID3, C4.5, C5.0 and CART

    * 1.10.7. Mathematical formulation

    * 1.10.8. Minimal Cost-Complexity Pruning

  * 1.11. Ensemble methods

    * 1.11.1. Bagging meta-estimator

    * 1.11.2. Forests of randomized trees

    * 1.11.3. AdaBoost

    * 1.11.4. Gradient Tree Boosting

    * 1.11.5. Histogram-Based Gradient Boosting

    * 1.11.6. Voting Classifier

    * 1.11.7. Voting Regressor

    * 1.11.8. Stacked generalization

  * 1.12. Multiclass and multioutput algorithms

    * 1.12.1. Multiclass classification

    * 1.12.2. Multilabel classification

    * 1.12.3. Multiclass-multioutput classification

    * 1.12.4. Multioutput regression

  * 1.13. Feature selection

    * 1.13.1. Removing features with low variance

    * 1.13.2. Univariate feature selection

    * 1.13.3. Recursive feature elimination

    * 1.13.4. Feature selection using SelectFromModel

    * 1.13.5. Sequential Feature Selection

    * 1.13.6. Feature selection as part of a pipeline

  * 1.14. Semi-supervised learning

    * 1.14.1. Self Training

    * 1.14.2. Label Propagation

  * 1.15. Isotonic regression

  * 1.16. Probability calibration

    * 1.16.1. Calibration curves

    * 1.16.2. Calibrating a classifier

    * 1.16.3. Usage

* 1.14.2. Label Propagation

  * 1.17. Neural network models (supervised)

    * 1.17.1. Multi-layer Perceptron

    * 1.17.2. Classification

    * 1.17.3. Regression

    * 1.17.4. Regularization

    * 1.17.5. Algorithms

    * 1.17.6. Complexity

    * 1.17.7. Mathematical formulation

    * 1.17.8. Tips on Practical Use

    * 1.17.9. More control with warm_start

* 2. Unsupervised learning

  * 2.1. Gaussian mixture models

    * 2.1.1. Gaussian Mixture

    * 2.1.2. Variational Bayesian Gaussian Mixture

  * 2.2. Manifold learning

    * 2.2.1. Introduction

    * 2.2.2. Isomap

    * 2.2.3. Locally Linear Embedding

    * 2.2.4. Modified Locally Linear Embedding

    * 2.2.5. Hessian Eigenmapping

    * 2.2.6. Spectral Embedding

    * 2.2.7. Local Tangent Space Alignment

    * 2.2.8. Multi-dimensional Scaling (MDS)

    * 2.2.9. t-distributed Stochastic Neighbor Embedding (t-SNE)

    * 2.2.10. Tips on practical use

  * 2.3. Clustering

    * 2.3.1. Overview of clustering methods

    * 2.3.2. K-means

    * 2.3.3. Affinity Propagation

    * 2.3.4. Mean Shift

    * 2.3.5. Spectral clustering

    * 2.3.6. Hierarchical clustering

    * 2.3.7. DBSCAN

    * 2.3.8. OPTICS

    * 2.3.9. BIRCH

    * 2.3.10. Clustering performance evaluation

  * 2.4. Biclustering

    * 2.4.1. Spectral Co-Clustering

    * 2.4.2. Spectral Biclustering

    * 2.4.3. Biclustering evaluation

* 2.3.9. BIRCH

  * 2.5. Decomposing signals in components (matrix factorization
    problems)

    * 2.5.1. Principal component analysis (PCA)

    * 2.5.2. Kernel Principal Component Analysis (kPCA)

    * 2.5.3. Truncated singular value decomposition and latent
      semantic analysis

    * 2.5.4. Dictionary Learning

    * 2.5.5. Factor Analysis

    * 2.5.6. Independent component analysis (ICA)

    * 2.5.7. Non-negative matrix factorization (NMF or NNMF)

    * 2.5.8. Latent Dirichlet Allocation (LDA)

  * 2.6. Covariance estimation

    * 2.6.1. Empirical covariance

    * 2.6.2. Shrunk Covariance

    * 2.6.3. Sparse inverse covariance

    * 2.6.4. Robust Covariance Estimation

  * 2.7. Novelty and Outlier Detection

    * 2.7.1. Overview of outlier detection methods

    * 2.7.2. Novelty Detection

    * 2.7.3. Outlier Detection

    * 2.7.4. Novelty detection with Local Outlier Factor

  * 2.8. Density Estimation

    * 2.8.1. Density Estimation: Histograms

    * 2.8.2. Kernel Density Estimation

  * 2.9. Neural network models (unsupervised)

    * 2.9.1. Restricted Boltzmann machines

* 3. Model selection and evaluation

  * 3.1. Cross-validation: evaluating estimator performance

    * 3.1.1. Computing cross-validated metrics

    * 3.1.2. Cross validation iterators

    * 3.1.3. A note on shuffling

    * 3.1.4. Cross validation and model selection

    * 3.1.5. Permutation test score

  * 3.2. Tuning the hyper-parameters of an estimator

    * 3.2.1. Exhaustive Grid Search

    * 3.2.2. Randomized Parameter Optimization

    * 3.2.3. Searching for optimal parameters with successive halving

    * 3.2.4. Tips for parameter search

    * 3.2.5. Alternatives to brute force parameter search

  * 3.3. Metrics and scoring: quantifying the quality of predictions

    * 3.3.1. The "scoring" parameter: defining model evaluation rules

    * 3.3.2. Classification metrics

    * 3.3.3. Multilabel ranking metrics

    * 3.3.4. Regression metrics

    * 3.3.5. Clustering metrics

    * 3.3.6. Dummy estimators

  * 3.4. Validation curves: plotting scores to evaluate models

    * 3.4.1. Validation curve

    * 3.4.2. Learning curve

* 4. Inspection

  * 4.1. Partial Dependence and Individual Conditional Expectation
    plots

    * 4.1.1. Partial dependence plots

    * 4.1.2. Individual conditional expectation (ICE) plot

    * 4.1.3. Mathematical Definition

    * 4.1.4. Computation methods

  * 4.2. Permutation feature importance

    * 4.2.1. Outline of the permutation importance algorithm

    * 4.2.2. Relation to impurity-based importance in trees

    * 4.2.3. Misleading values on strongly correlated features

* 5. Visualizations

* 4.2.1. Outline of the permutation importance algorithm

  * 5.1. Available Plotting Utilities

    * 5.1.1. Display Objects

* 6. Dataset transformations

  * 6.1. Pipelines and composite estimators

    * 6.1.1. Pipeline: chaining estimators

    * 6.1.2. Transforming target in regression

    * 6.1.3. FeatureUnion: composite feature spaces

    * 6.1.4. ColumnTransformer for heterogeneous data

    * 6.1.5. Visualizing Composite Estimators

  * 6.2. Feature extraction

    * 6.2.1. Loading features from dicts

    * 6.2.2. Feature hashing

    * 6.2.3. Text feature extraction

    * 6.2.4. Image feature extraction

  * 6.3. Preprocessing data

    * 6.3.1. Standardization, or mean removal and variance scaling

    * 6.3.2. Non-linear transformation

    * 6.3.3. Normalization

    * 6.3.4. Encoding categorical features

    * 6.3.5. Discretization

    * 6.3.6. Imputation of missing values

    * 6.3.7. Generating polynomial features

    * 6.3.8. Custom transformers

  * 6.4. Imputation of missing values

    * 6.4.1. Univariate vs. Multivariate Imputation

    * 6.4.2. Univariate feature imputation

    * 6.4.3. Multivariate feature imputation

    * 6.4.4. Nearest neighbors imputation

    * 6.4.5. Keeping the number of features constants

    * 6.4.6. Marking imputed values

    * 6.4.7. Estimators that handle NaN values

  * 6.5. Unsupervised dimensionality reduction

* 6.4.5. Keeping the number of features constants

    * 6.5.1. PCA: principal component analysis

    * 6.5.2. Random projections

    * 6.5.3. Feature agglomeration

  * 6.6. Random Projection

    * 6.6.1. The Johnson-Lindenstrauss lemma

    * 6.6.2. Gaussian random projection

    * 6.6.3. Sparse random projection

    * 6.6.4. Inverse Transform

  * 6.7. Kernel Approximation

    * 6.7.1. Nystroem Method for Kernel Approximation

    * 6.7.2. Radial Basis Function Kernel

    * 6.7.3. Additive Chi Squared Kernel

    * 6.7.4. Skewed Chi Squared Kernel

    * 6.7.5. Polynomial Kernel Approximation via Tensor Sketch

    * 6.7.6. Mathematical Details

  * 6.8. Pairwise metrics, Affinities and Kernels

    * 6.8.1. Cosine similarity

    * 6.8.2. Linear kernel

    * 6.8.3. Polynomial kernel

    * 6.8.4. Sigmoid kernel

    * 6.8.5. RBF kernel

    * 6.8.6. Laplacian kernel

    * 6.8.7. Chi-squared kernel

  * 6.9. Transforming the prediction target ("y")

    * 6.9.1. Label binarization

    * 6.9.2. Label encoding

* 7. Dataset loading utilities

  * 7.1. Toy datasets

  * 7.2. Real world datasets

  * 7.3. Generated datasets

    * 7.3.1. Generators for classification and clustering

    * 7.3.2. Generators for regression

    * 7.3.3. Generators for manifold learning

    * 7.3.4. Generators for decomposition

  * 7.4. Loading other datasets

    * 7.4.1. Sample images

* 7.3.2. Generators for regression

    * 7.4.2. Datasets in svmlight / libsvm format

    * 7.4.3. Downloading datasets from the openml.org repository

    * 7.4.4. Loading from external datasets

* 8. Computing with scikit-learn

  * 8.1. Strategies to scale computationally: bigger data

    * 8.1.1. Scaling with instances using out-of-core learning

  * 8.2. Computational Performance

    * 8.2.1. Prediction Latency

    * 8.2.2. Prediction Throughput

    * 8.2.3. Tips and Tricks

  * 8.3. Parallelism, resource management, and configuration

    * 8.3.1. Parallelism

    * 8.3.2. Configuration switches

* 9. Model persistence

  * 9.1. Python specific serialization

    * 9.1.1. Security & maintainability limitations

    * 9.1.2. A more secure format: *skops*

  * 9.2. Interoperable formats

* 10. Common pitfalls and recommended practices

  * 10.1. Inconsistent preprocessing

  * 10.2. Data leakage

    * 10.2.1. Data leakage during pre-processing

    * 10.2.2. How to avoid data leakage

  * 10.3. Controlling randomness

    * 10.3.1. Using *None* or *RandomState* instances, and repeated
      calls to *fit* and *split*

    * 10.3.2. Common pitfalls and subtleties

    * 10.3.3. General recommendations

* 11. Dispatching

  * 11.1. Array API support (experimental)

    * 11.1.1. Example usage

    * 11.1.2. Estimators with support for *Array API*-compatible
      inputs

External Resources, Videos and Talks
************************************

For written tutorials, see the Tutorial section of the documentation.

New to Scientific Python?

For those that are still new to the scientific Python ecosystem, we
highly recommend the Python Scientific Lecture Notes. This will help
you find your footing a bit and will definitely improve your scikit-
learn experience.  A basic understanding of NumPy arrays is
recommended to make the most of scikit-learn.

External Tutorials

There are several online tutorials available which are geared toward
specific subject areas:

* Machine Learning for NeuroImaging in Python

* Machine Learning for Astronomical Data Analysis

Videos
======

* An introduction to scikit-learn Part I and Part II at Scipy 2013 by
  Gael Varoquaux, Jake Vanderplas  and Olivier Grisel. Notebooks on
  github.

* Introduction to scikit-learn by Gael Varoquaux at ICML 2010

     A three minute video from a very early stage of scikit-learn,
     explaining the basic idea and approach we are following.

* Introduction to statistical learning with scikit-learn by Gael
  Varoquaux at SciPy 2011

     An extensive tutorial, consisting of four sessions of one hour.
     The tutorial covers the basics of machine learning, many
     algorithms and how to apply them using scikit-learn. The material
     corresponding is now in the scikit-learn documentation section A
     tutorial on statistical-learning for scientific data processing.

corresponding is now in the scikit-learn documentation section A

* Statistical Learning for Text Classification with scikit-learn and
  NLTK (and slides) by Olivier Grisel at PyCon 2011

     Thirty minute introduction to text classification. Explains how
     to use NLTK and scikit-learn to solve real-world text
     classification tasks and compares against cloud-based solutions.

* Introduction to Interactive Predictive Analytics in Python with
  scikit-learn by Olivier Grisel at PyCon 2012

     3-hours long introduction to prediction tasks using scikit-learn.

* scikit-learn - Machine Learning in Python by Jake Vanderplas at the
  2012 PyData workshop at Google

     Interactive demonstration of some scikit-learn features. 75
     minutes.

* scikit-learn tutorial by Jake Vanderplas at PyData NYC 2012

     Presentation using the online tutorial, 45 minutes.

1. Supervised learning
**********************

* 1.1. Linear Models

  * 1.1.1. Ordinary Least Squares

  * 1.1.2. Ridge regression and classification

  * 1.1.3. Lasso

  * 1.1.4. Multi-task Lasso

  * 1.1.5. Elastic-Net

  * 1.1.6. Multi-task Elastic-Net

  * 1.1.7. Least Angle Regression

  * 1.1.8. LARS Lasso

  * 1.1.9. Orthogonal Matching Pursuit (OMP)

  * 1.1.10. Bayesian Regression

  * 1.1.11. Logistic regression

  * 1.1.12. Generalized Linear Models

  * 1.1.13. Stochastic Gradient Descent - SGD

  * 1.1.14. Perceptron

  * 1.1.15. Passive Aggressive Algorithms

  * 1.1.16. Robustness regression: outliers and modeling errors

  * 1.1.17. Quantile Regression

  * 1.1.18. Polynomial regression: extending linear models with basis
    functions

* 1.2. Linear and Quadratic Discriminant Analysis

  * 1.2.1. Dimensionality reduction using Linear Discriminant Analysis

  * 1.2.2. Mathematical formulation of the LDA and QDA classifiers

  * 1.2.3. Mathematical formulation of LDA dimensionality reduction

  * 1.2.4. Shrinkage and Covariance Estimator

  * 1.2.5. Estimation algorithms

* 1.3. Kernel ridge regression

* 1.4. Support Vector Machines

  * 1.4.1. Classification

  * 1.4.2. Regression

  * 1.4.3. Density estimation, novelty detection

  * 1.4.4. Complexity

  * 1.4.5. Tips on Practical Use

  * 1.4.6. Kernel functions

  * 1.4.7. Mathematical formulation

  * 1.4.8. Implementation details

* 1.5. Stochastic Gradient Descent

  * 1.5.1. Classification

  * 1.5.2. Regression

  * 1.5.3. Online One-Class SVM

* 1.4.7. Mathematical formulation

  * 1.5.4. Stochastic Gradient Descent for sparse data

  * 1.5.5. Complexity

  * 1.5.6. Stopping criterion

  * 1.5.7. Tips on Practical Use

  * 1.5.8. Mathematical formulation

  * 1.5.9. Implementation details

* 1.6. Nearest Neighbors

  * 1.6.1. Unsupervised Nearest Neighbors

  * 1.6.2. Nearest Neighbors Classification

  * 1.6.3. Nearest Neighbors Regression

  * 1.6.4. Nearest Neighbor Algorithms

  * 1.6.5. Nearest Centroid Classifier

  * 1.6.6. Nearest Neighbors Transformer

  * 1.6.7. Neighborhood Components Analysis

* 1.7. Gaussian Processes

  * 1.7.1. Gaussian Process Regression (GPR)

  * 1.7.2. GPR examples

  * 1.7.3. Gaussian Process Classification (GPC)

  * 1.7.4. GPC examples

  * 1.7.5. Kernels for Gaussian Processes

* 1.8. Cross decomposition

  * 1.8.1. PLSCanonical

  * 1.8.2. PLSSVD

  * 1.8.3. PLSRegression

  * 1.8.4. Canonical Correlation Analysis

* 1.9. Naive Bayes

  * 1.9.1. Gaussian Naive Bayes

  * 1.9.2. Multinomial Naive Bayes

  * 1.9.3. Complement Naive Bayes

  * 1.9.4. Bernoulli Naive Bayes

  * 1.9.5. Categorical Naive Bayes

  * 1.9.6. Out-of-core naive Bayes model fitting

* 1.10. Decision Trees

  * 1.10.1. Classification

  * 1.10.2. Regression

  * 1.10.3. Multi-output problems

  * 1.10.4. Complexity

  * 1.10.5. Tips on practical use

  * 1.10.6. Tree algorithms: ID3, C4.5, C5.0 and CART

* 1.10.1. Classification

  * 1.10.7. Mathematical formulation

  * 1.10.8. Minimal Cost-Complexity Pruning

* 1.11. Ensemble methods

  * 1.11.1. Bagging meta-estimator

  * 1.11.2. Forests of randomized trees

  * 1.11.3. AdaBoost

  * 1.11.4. Gradient Tree Boosting

  * 1.11.5. Histogram-Based Gradient Boosting

  * 1.11.6. Voting Classifier

  * 1.11.7. Voting Regressor

  * 1.11.8. Stacked generalization

* 1.12. Multiclass and multioutput algorithms

  * 1.12.1. Multiclass classification

  * 1.12.2. Multilabel classification

  * 1.12.3. Multiclass-multioutput classification

  * 1.12.4. Multioutput regression

* 1.13. Feature selection

  * 1.13.1. Removing features with low variance

  * 1.13.2. Univariate feature selection

  * 1.13.3. Recursive feature elimination

  * 1.13.4. Feature selection using SelectFromModel

  * 1.13.5. Sequential Feature Selection

  * 1.13.6. Feature selection as part of a pipeline

* 1.14. Semi-supervised learning

  * 1.14.1. Self Training

  * 1.14.2. Label Propagation

* 1.15. Isotonic regression

* 1.16. Probability calibration

  * 1.16.1. Calibration curves

  * 1.16.2. Calibrating a classifier

  * 1.16.3. Usage

* 1.17. Neural network models (supervised)

  * 1.17.1. Multi-layer Perceptron

  * 1.17.2. Classification

  * 1.17.3. Regression

  * 1.17.4. Regularization

  * 1.17.5. Algorithms

  * 1.17.6. Complexity

  * 1.17.7. Mathematical formulation

  * 1.17.8. Tips on Practical Use

  * 1.17.9. More control with warm_start

Related Projects
****************

Projects implementing the scikit-learn estimator API are encouraged to
use the scikit-learn-contrib template which facilitates best practices
for testing and documenting estimators. The scikit-learn-contrib
GitHub organization also accepts high-quality contributions of
repositories conforming to this template.

Below is a list of sister-projects, extensions and domain specific
packages.

Interoperability and framework enhancements
===========================================

These tools adapt scikit-learn for use with other technologies or
otherwise enhance the functionality of scikit-learn's estimators.

**Data formats**

* Fast svmlight / libsvm file loader Fast and memory-efficient
  svmlight / libsvm file loader for Python.

* sklearn_pandas bridge for scikit-learn pipelines and pandas data
  frame with dedicated transformers.

* sklearn_xarray provides compatibility of scikit-learn estimators
  with xarray data structures.

**Auto-ML**

* auto-sklearn An automated machine learning toolkit and a drop-in
  replacement for a scikit-learn estimator

* autoviml Automatically Build Multiple Machine Learning Models with a
  Single Line of Code. Designed as a faster way to use scikit-learn
  models without having to preprocess data.

* TPOT An automated machine learning toolkit that optimizes a series
  of scikit-learn operators to design a machine learning pipeline,
  including data and feature preprocessors as well as the estimators.

of scikit-learn operators to design a machine learning pipeline,
  Works as a drop-in replacement for a scikit-learn estimator.

* Featuretools A framework to perform automated feature engineering.
  It can be used for transforming temporal and relational datasets
  into feature matrices for machine learning.

* Neuraxle A library for building neat pipelines, providing the right
  abstractions to both ease research, development, and deployment of
  machine learning applications. Compatible with deep learning
  frameworks and scikit-learn API, it can stream minibatches, use data
  checkpoints, build funky pipelines, and serialize models with custom
  per-step savers.

* EvalML EvalML is an AutoML library which builds, optimizes, and
  evaluates machine learning pipelines using domain-specific objective
  functions. It incorporates multiple modeling libraries under one
  API, and the objects that EvalML creates use an sklearn-compatible
  API.

**Experimentation frameworks**

* Neptune Metadata store for MLOps, built for teams that run a lot of
  experiments.‌ It gives you a single place to log, store, display,
  organize, compare, and query all your model building metadata.

* Sacred Tool to help you configure, organize, log and reproduce
  experiments

* REP Environment for conducting data-driven research in a consistent
  and reproducible way

* Scikit-Learn Laboratory  A command-line wrapper around scikit-learn

experiments

  that makes it easy to run machine learning experiments with multiple
  learners and large feature sets.

**Model inspection and visualization**

* dtreeviz A python library for decision tree visualization and model
  interpretation.

* eli5 A library for debugging/inspecting machine learning models and
  explaining their predictions.

* mlxtend Includes model visualization utilities.

* sklearn-evaluation Machine learning model evaluation made easy:
  plots, tables, HTML reports, experiment tracking and Jupyter
  notebook analysis. Visual analysis, model selection, evaluation and
  diagnostics.

* yellowbrick A suite of custom matplotlib visualizers for scikit-
  learn estimators to support visual feature analysis, model
  selection, evaluation, and diagnostics.

**Model selection**

* scikit-optimize A library to minimize (very) expensive and noisy
  black-box functions. It implements several methods for sequential
  model-based optimization, and includes a replacement for
  "GridSearchCV" or "RandomizedSearchCV" to do cross-validated
  parameter search using any of these strategies.

* sklearn-deap Use evolutionary algorithms instead of gridsearch in
  scikit-learn.

**Model export for production**

* sklearn-onnx Serialization of many Scikit-learn pipelines to ONNX
  for interchange and prediction.

` skops.io A

scikit-learn.

   persistence model more secure than pickle, which can be used
   instead of pickle in most common cases.

* sklearn2pmml Serialization of a wide variety of scikit-learn
  estimators and transformers into PMML with the help of JPMML-SkLearn
  library.

* sklearn-porter Transpile trained scikit-learn models to C, Java,
  Javascript and others.

* m2cgen A lightweight library which allows to transpile trained
  machine learning models including many scikit-learn estimators into
  a native code of C, Java, Go, R, PHP, Dart, Haskell, Rust and many
  other programming languages.

* treelite Compiles tree-based ensemble models into C code for
  minimizing prediction latency.

Other estimators and tasks

Not everything belongs or is mature enough for the central scikit-
learn project. The following are projects providing interfaces similar
to scikit-learn for additional learning algorithms, infrastructures
and tasks.

**Structured learning**

* tslearn A machine learning library for time series that offers tools
  for pre-processing and feature extraction as well as dedicated
  models for clustering, classification and regression.

* sktime A scikit-learn compatible toolbox for machine learning with
  time series including time series classification/regression and
  (supervised/panel) forecasting.

* HMMLearn Implementation of hidden markov models that was previously
  part of scikit-learn.

* PyStruct General conditional random fields and structured
  prediction.

* pomegranate Probabilistic modelling for Python, with an emphasis on
  hidden Markov models.

* sklearn-crfsuite Linear-chain conditional random fields (CRFsuite
  wrapper with sklearn-like API).

**Deep neural networks etc.**

* nolearn A number of wrappers and abstractions around existing neural
  network libraries

* Keras High-level API for TensorFlow with a scikit-learn inspired
  API.

* lasagne A lightweight library to build and train neural networks in
  Theano.

* skorch A scikit-learn compatible neural network library that wraps
  PyTorch.

* scikeras provides a wrapper around Keras to interface it with
  scikit-learn. SciKeras is the successor of
  *tf.keras.wrappers.scikit_learn*.

**Federated Learning**

* Flower A friendly federated learning framework with a unified
  approach that can federate any workload, any ML framework, and any
  programming language.

**Broad scope**

* mlxtend Includes a number of additional estimators as well as model
  visualization utilities.

* scikit-lego A number of scikit-learn compatible custom transformers,
  models and metrics, focusing on solving practical industry tasks.

**Other regression and classification**

* xgboost Optimised gradient boosted decision tree library.

* ML-Ensemble Generalized ensemble learning (stacking, blending,
  subsemble, deep ensembles, etc.).

* lightning Fast state-of-the-art linear model solvers (SDCA, AdaGrad,
  SVRG, SAG, etc...).

* py-earth Multivariate adaptive regression splines

* Kernel Regression Implementation of Nadaraya-Watson kernel
  regression with automatic bandwidth selection

* gplearn Genetic Programming for symbolic regression tasks.

* scikit-multilearn Multi-label classification with focus on label
  space manipulation.

* seglearn Time series and sequence learning using sliding window
  segmentation.

* libOPF Optimal path forest classifier

* fastFM Fast factorization machine implementation compatible with
  scikit-learn

**Decomposition and clustering**

* lda: Fast implementation of latent Dirichlet allocation in Cython
  which uses Gibbs sampling to sample from the true posterior
  distribution. (scikit-learn's "LatentDirichletAllocation"
  implementation uses variational inference to sample from a tractable
  approximation of a topic model's posterior distribution.)

* kmodes k-modes clustering algorithm for categorical data, and
  several of its variations.

* hdbscan HDBSCAN and Robust Single Linkage clustering algorithms for
  robust variable density clustering.

* spherecluster Spherical K-means and mixture of von Mises Fisher
  clustering routines for data on the unit hypersphere.

**Pre-processing**

* categorical-encoding A library of sklearn compatible categorical
  variable encoders.

* imbalanced-learn Various methods to under- and over-sample datasets.

* Feature-engine A library of sklearn compatible transformers for
  missing data imputation, categorical encoding, variable
  transformation, discretization, outlier handling and more. Feature-
  engine allows the application of preprocessing steps to selected
  groups of variables and it is fully compatible with the Scikit-learn
  Pipeline.

**Topological Data Analysis**

* giotto-tda A library for Topological Data Analysis aiming to provide
  a scikit-learn compatible API. It offers tools to transform data
  inputs (point clouds, graphs, time series, images) into forms
  suitable for computations of topological summaries, and components
  dedicated to extracting sets of scalar features of topological
  origin, which can be used alongside other feature extraction methods
  in scikit-learn.

Statistical learning with Python

Other packages useful for data analysis and machine learning.

* Pandas Tools for working with heterogeneous and columnar data,

  relational queries, time series and basic statistics.

* statsmodels Estimating and analysing statistical models. More
  focused on statistical tests and less on prediction than scikit-
  learn.

* PyMC Bayesian statistical models and fitting algorithms.

* Seaborn Visualization library based on matplotlib. It provides a
  high-level interface for drawing attractive statistical graphics.

* scikit-survival A library implementing models to learn from censored
  time-to-event data (also called survival analysis). Models are fully
  compatible with scikit-learn.

Recommendation Engine packages
------------------------------

* implicit, Library for implicit feedback datasets.

* lightfm A Python/Cython implementation of a hybrid recommender
  system.

* OpenRec TensorFlow-based neural-network inspired recommendation
  algorithms.

* Spotlight Pytorch-based implementation of deep recommender models.

* Surprise Lib Library for explicit feedback datasets.

Domain specific packages
------------------------

* scikit-network Machine learning on graphs.

* scikit-image Image processing and computer vision in python.

* Natural language toolkit (nltk) Natural language processing and some
  machine learning.

* gensim  A library for topic modelling, document indexing and
  similarity retrieval

* NiLearn Machine learning for neuro-imaging.

machine learning.

* AstroML  Machine learning for astronomy.

* MSMBuilder  Machine learning for protein conformational dynamics
  time series.

Translations of scikit-learn documentation

Translation's purpose is to ease reading and understanding in
languages other than English. Its aim is to help people who do not
understand English or have doubts about its interpretation.
Additionally, some people prefer to read documentation in their native
language, but please bear in mind that the only official documentation
is the English one [1].

Those translation efforts are community initiatives and we have no
control on them. If you want to contribute or report an issue with the
translation, please contact the authors of the translation. Some
available translations are linked here to improve their dissemination
and promote community efforts.

* Chinese translation (source)

* Persian translation (source)

* Spanish translation (source)

* Korean translation (source)

-[ Footnotes ]-

[1] following linux documentation Disclaimer

Version 1.2.2

**In development**

The following estimators and functions, when fit with the same data
and parameters, may produce different models from the previous
version. This often occurs due to changes in the modelling logic (bug
fixes or enhancements), or in random sampling procedures.

Changed models

Changes impacting all modules

Changelog
=========

"sklearn.isotonic"

*   Fixes a bug in "isotonic.IsotonicRegression" where
  "isotonic.IsotonicRegression.predict()" would return a pandas
  DataFrame when the global configuration sets
  *transform_output="pandas"*. >>:pr:`25500`<< by >>:user:`Guillaume
  Lemaitre <glemaitre>`<<.

Version 1.2.1

**January 2023**

*   The fitted components in "MiniBatchDictionaryLearning" might
  differ. The online updates of the sufficient statistics now properly
  take the sizes of the batches into account. >>:pr:`25354`<< by
  >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<.

*   The *categories_* attribute of "preprocessing.OneHotEncoder" now
  always contains an array of *object`s when using predefined
  categories that are strings. Predefined categories encoded as bytes

  will no longer work with `X* encoded as strings. >>:pr:`25174`<< by
  >>:user:`Tim Head <betatim>`<<.

*   Support *pandas.Int64* dtyped *y* for classifiers and regressors.
  >>:pr:`25089`<< by >>:user:`Tim Head <betatim>`<<.

*   Remove spurious warnings for estimators internally using neighbors
  search methods. >>:pr:`25129`<< by >>:user:`Julien Jerphanion
  <jjerphan>`<<.

*   Fix a bug where the current configuration was ignored in
  estimators using *n_jobs > 1*. This bug was triggered for tasks
  dispatched by the auxillary thread of *joblib* as
  "sklearn.get_config()" used to access an empty thread local
  configuration instead of the configuration visible from the thread
  where *joblib.Parallel* was first called. >>:pr:`25363`<< by
  >>:user:`Guillaume Lemaitre <glemaitre>`<<.

Changelog
=========

"sklearn.base"
--------------

*   Fix a regression in *BaseEstimator.__getstate__* that would
  prevent certain estimators to be pickled when using Python 3.11.
  >>:pr:`25188`<< by >>:user:`Benjamin Bossan <BenjaminBossan>`<<.

*   Inheriting from "base.TransformerMixin" will only wrap the
  *transform* method if the class defines *transform* itself.
  >>:pr:`25295`<< by Thomas Fan.

"sklearn.datasets"

*   Fix an inconsistency in "datasets.fetch_openml()" between liac-
  arff and pandas parser when a leading space is introduced after the
  delimiter. The ARFF specs requires to ignore the leading space.
  >>:pr:`25312`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

"sklearn.decomposition"

*   Fixed a bug in "decomposition.MiniBatchDictionaryLearning" where
  the online updates of the sufficient statistics where not correct
  when calling *partial_fit* on batches of different sizes.
  >>:pr:`25354`<< by >>:user:`Jérémie du Boisberranger
  <jeremiedbb>`<<.

*   "decomposition.DictionaryLearning" better supports readonly NumPy
  arrays. In particular, it better supports large datasets which are
  memory-mapped when it is used with coordinate descent algorithms
  (i.e. when *fit_algorithm='cd'*). >>:pr:`25172`<< by >>:user:`Julien
  Jerphanion <jjerphan>`<<.

*   "ensemble.RandomForestClassifier",
  "ensemble.RandomForestRegressor" "ensemble.ExtraTreesClassifier" and
  "ensemble.ExtraTreesRegressor" now support sparse readonly datasets.
  >>:pr:`25341`<< by >>:user:`Julien Jerphanion <jjerphan>`<<

"sklearn.feature_extraction"
----------------------------

*   "feature_extraction.FeatureHasher" raises an informative error

>>:pr:`25341`<< by >>:user:`Julien Jerphanion <jjerphan>`<<

  when the input is a list of strings. >>:pr:`25094`<< by Thomas Fan.

"sklearn.linear_model"

*   Fix a regression in "linear_model.SGDClassifier" and
  "linear_model.SGDRegressor" that makes them unusable with the
  *verbose* parameter set to a value greater than 0. >>:pr:`25250`<<
  by >>:user:`Jérémie Du Boisberranger <jeremiedbb>`<<.

"sklearn.manifold"

*   "manifold.TSNE" now works correctly when output type is set to
  pandas >>:pr:`25370`<< by >>:user:`Tim Head <betatim>`<<.

*   "model_selection.cross_validate()" with multimetric scoring in
  case of some failing scorers the non-failing scorers now returns
  proper scores instead of *error_score* values. >>:pr:`23101`<< by
  >>:user:`András Simon <simonandras>`<< and Thomas Fan.

"sklearn.neural_network"

*   "neural_network.MLPClassifier" and "neural_network.MLPRegressor"
  no longer raise warnings when fitting data with feature names.
  >>:pr:`24873`<< by >>:user:`Tim Head <betatim>`<<.

"sklearn.preprocessing"

*   "preprocessing.FunctionTransformer.inverse_transform()" correctly
  supports DataFrames that are all numerical when
  *check_inverse=True*. >>:pr:`25274`<< by Thomas Fan.

*   "preprocessing.SplineTransformer.get_feature_names_out()"
  correctly returns feature names when *extrapolations="periodic"*.
  >>:pr:`25296`<< by Thomas Fan.

"sklearn.tree"

*   "tree.DecisionTreeClassifier", "tree.DecisionTreeRegressor"
  "tree.ExtraTreeClassifier" and "tree.ExtraTreeRegressor" now support
  sparse readonly datasets. >>:pr:`25341`<< by >>:user:`Julien
  Jerphanion <jjerphan>`<<

"sklearn.utils"

*   Restore "utils.check_array()"'s behaviour for pandas Series of
  type boolean. The type is maintained, instead of converting to
  *float64.* >>:pr:`25147`<< by >>:user:`Tim Head <betatim>`<<.

*   "utils.fixes.delayed()" is deprecated in 1.2.1 and will be removed
  in 1.5. Instead, import "utils.parallel.delayed()" and use it in
  conjunction with the newly introduced "utils.parallel.Parallel()" to
  ensure proper propagation of the scikit-learn configuration to the
  workers. >>:pr:`25363`<< by >>:user:`Guillaume Lemaitre
  <glemaitre>`<<.

Version 1.2.0

**December 2022**

For a short description of the main highlights of the release, please
refer to *sphx_glr_auto_examples_release_highlights_plot_release_high
lights_1_2_0.py*.

Legend for changelogs
=====================

*  : something big that you couldn't do before.

*  : something that you couldn't do before.

*  : an existing feature now may not require as much computation or
  memory.

*  : a miscellaneous minor improvement.

*  : something that previously didn't work as documentated -- or
  according to reasonable expectations -- should now work.

*  : you will need to change your code to have the same effect in the
  future; or a feature will be removed in the future.

*   The default *eigen_tol* for "cluster.SpectralClustering",
  "manifold.SpectralEmbedding", "cluster.spectral_clustering()", and
  "manifold.spectral_embedding()" is now *None* when using the *'amg'*
  or *'lobpcg'* solvers. This change improves numerical stability of
  the solver, but may result in a different model.

*   "linear_model.GammaRegressor", "linear_model.PoissonRegressor" and
  "linear_model.TweedieRegressor" can reach higher precision with the
  lbfgs solver, in particular when *tol* is set to a tiny value.
  Moreover, *verbose* is now properly propagated to L-BFGS-B.

"linear_model.TweedieRegressor" can reach higher precision with the
  >>:pr:`23619`<< by >>:user:`Christian Lorentzen <lorentzenchr>`<<.

*   The default value for *eps* "metrics.logloss()" has changed from
  *1e-15* to *"auto"*. *"auto"* sets *eps* to
  *np.finfo(y_pred.dtype).eps*. >>:pr:`24354`<< by >>:user:`Safiuddin
  Khaja <Safikh>`<< and >>:user:`gsiisg <gsiisg>`<<.

*   Make sign of *components_* deterministic in
  "decomposition.SparsePCA". >>:pr:`23935`<< by >>:user:`Guillaume

*   The *components_* signs in "decomposition.FastICA" might differ.
  It is now consistent and deterministic with all SVD solvers.
  >>:pr:`22527`<< by >>:user:`Meekail Zain <micky774>`<< and Thomas
  Fan.

*   The condition for early stopping has now been changed in
  "linear_model._sgd_fast._plain_sgd()" which is used by
  "linear_model.SGDRegressor" and "linear_model.SGDClassifier". The
  old condition did not disambiguate between training and validation
  set and had an effect of overscaling the error tolerance. This has
  been fixed in >>:pr:`23798`<< by >>:user:`Harsh Agrawal
  <Harsh14901>`<<.

*   For "model_selection.GridSearchCV" and
  "model_selection.RandomizedSearchCV" ranks corresponding to nan
  scores will all be set to the maximum possible rank. >>:pr:`24543`<<
  by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

"model_selection.RandomizedSearchCV" ranks corresponding to nan

*   The default value of *tol* was changed from *1e-3* to *1e-4* for
  "linear_model.ridge_regression()", "linear_model.Ridge" and
  "linear_model.`RidgeClassifier". >>:pr:`24465`<< by
  >>:user:`Christian Lorentzen <lorentzenchr>`<<.

*   The *set_output* API has been adopted by all transformers. Meta-
  estimators that contain transformers such as "pipeline.Pipeline" or
  "compose.ColumnTransformer" also define a *set_output*. For details,
  see SLEP018. >>:pr:`23734`<< and >>:pr:`24699`<< by Thomas Fan.

*   Low-level routines for reductions on pairwise distances for dense
  float32 datasets have been refactored. The following functions and
  estimators now benefit from improved performances in terms of
  hardware scalability and speed-ups:

  * "sklearn.metrics.pairwise_distances_argmin()"

  * "sklearn.metrics.pairwise_distances_argmin_min()"

  * "sklearn.cluster.AffinityPropagation"

  * "sklearn.cluster.Birch"

  * "sklearn.cluster.MeanShift"

  * "sklearn.cluster.OPTICS"

  * "sklearn.cluster.SpectralClustering"

  * "sklearn.feature_selection.mutual_info_regression()"

  * "sklearn.neighbors.KNeighborsClassifier"

  * "sklearn.neighbors.KNeighborsRegressor"

  * "sklearn.neighbors.RadiusNeighborsClassifier"

  * "sklearn.neighbors.RadiusNeighborsRegressor"

* "sklearn.neighbors.KNeighborsClassifier"

  * "sklearn.neighbors.LocalOutlierFactor"

  * "sklearn.neighbors.NearestNeighbors"

  * "sklearn.manifold.Isomap"

  * "sklearn.manifold.LocallyLinearEmbedding"

  * "sklearn.manifold.TSNE"

  * "sklearn.manifold.trustworthiness()"

  * "sklearn.semi_supervised.LabelPropagation"

  * "sklearn.semi_supervised.LabelSpreading"

  For instance "sklearn.neighbors.NearestNeighbors.kneighbors" and
  "sklearn.neighbors.NearestNeighbors.radius_neighbors" can
  respectively be up to ×20 and ×5 faster than previously on a laptop.

  Moreover, implementations of those two algorithms are now suitable
  for machine with many cores, making them usable for datasets
  consisting of millions of samples.

  >>:pr:`23865`<< by >>:user:`Julien Jerphanion <jjerphan>`<<.

*   Finiteness checks (detection of NaN and infinite values) in all
  estimators are now significantly more efficient for float32 data by
  leveraging NumPy's SIMD optimized primitives. >>:pr:`23446`<< by
  >>:user:`Meekail Zain <micky774>`<<

  estimators are now faster by utilizing a more efficient stop-on-
  first second-pass algorithm. >>:pr:`23197`<< by >>:user:`Meekail
  Zain <micky774>`<<

*   Support for combinations of dense and sparse datasets pairs for

first second-pass algorithm. >>:pr:`23197`<< by >>:user:`Meekail

  all distance metrics and for float32 and float64 datasets has been
  added or has seen its performance improved for the following
  estimators:

  >>:pr:`23604`<< and >>:pr:`23585`<< by >>:user:`Julien Jerphanion
  <jjerphan>`<<, >>:user:`Olivier Grisel <ogrisel>`<<, and Thomas Fan,
  >>:pr:`24556`<< by >>:user:`Vincent Maladière <Vincent-
  Maladiere>`<<.

*   Systematically check the sha256 digest of dataset tarballs used in
  code examples in the documentation. >>:pr:`24617`<< by
  >>:user:`Olivier Grisel <ogrisel>`<< and Thomas Fan. Thanks to
  Sim4n6 for the report.

Changelog
=========

*   Introduces "base.ClassNamePrefixFeaturesOutMixin" and
  "base.ClassNamePrefixFeaturesOutMixin" mixins that defines

Sim4n6 for the report.

Changelog
=========

  *get_feature_names_out* for common transformer uses cases.
  >>:pr:`24688`<< by Thomas Fan.

"sklearn.calibration"

*   Rename *base_estimator* to *estimator* in
  "calibration.CalibratedClassifierCV" to improve readability and
  consistency. The parameter *base_estimator* is deprecated and will
  be removed in 1.4. >>:pr:`22054`<< by >>:user:`Kevin Roice
  <kevroi>`<<.

-----------------

*   "cluster.KMeans" with *algorithm="lloyd"* is now faster and uses
  less memory. >>:pr:`24264`<< by >>:user:`Vincent Maladiere <Vincent-

*   The *predict* and *fit_predict* methods of "cluster.OPTICS" now
  accept sparse data type for input data. >>:pr:`14736`<< by
  >>:user:`Hunt Zhan <huntzhan>`<<, >>:pr:`20802`<< by
  >>:user:`Brandon Pokorny <Clickedbigfoot>`<<, and >>:pr:`22965`<< by
  >>:user:`Meekail Zain <micky774>`<<.

*   "cluster.Birch" now preserves dtype for *numpy.float32* inputs.
  >>:pr:`22968`<< by *Meekail Zain <micky774>*.

*   "cluster.KMeans" and "cluster.MiniBatchKMeans" now accept a new
  *'auto'* option for *n_init* which changes the number of random
  initializations to one when using *init='k-means++'* for efficiency.
  This begins deprecation for the default values of *n_init* in the
  two classes and both will have their defaults changed to

initializations to one when using *init='k-means++'* for efficiency.
  *n_init='auto'* in 1.4. >>:pr:`23038`<< by >>:user:`Meekail Zain
  <micky774>`<<.

*   "cluster.SpectralClustering" and "cluster.spectral_clustering()"
  now propogates the *eigen_tol* parameter to all choices of
  *eigen_solver*. Includes a new option *eigen_tol="auto"* and begins
  deprecation to change the default from *eigen_tol=0* to
  *eigen_tol="auto"* in version 1.3. >>:pr:`23210`<< by

*   "cluster.KMeans" now supports readonly attributes when predicting.
  >>:pr:`24258`<< by Thomas Fan

*   The *affinity* attribute is now deprecated for
  "cluster.AgglomerativeClustering" and will be renamed to *metric* in
  v1.4. >>:pr:`23470`<< by >>:user:`Meekail Zain <micky774>`<<.

*   Introduce the new parameter *parser* in "datasets.fetch_openml()".
  *parser="pandas"* allows to use the very CPU and memory efficient
  *pandas.read_csv* parser to load dense ARFF formatted dataset files.
  It is possible to pass *parser="liac-arff"* to use the old LIAC
  parser. When *parser="auto"*, dense datasets are loaded with
  "pandas" and sparse datasets are loaded with "liac-arff". Currently,
  *parser="liac-arff"* by default and will change to *parser="auto"*
  in version 1.4 >>:pr:`21938`<< by >>:user:`Guillaume Lemaitre

*parser="liac-arff"* by default and will change to *parser="auto"*

*   "datasets.dump_svmlight_file()" is now accelerated with a Cython
  implementation, providing 2-4x speedups. >>:pr:`23127`<< by

*   Path-like objects, such as those created with pathlib are now
  allowed as paths in "datasets.load_svmlight_file()" and
  "datasets.load_svmlight_files()". >>:pr:`19075`<< by >>:user:`Carlos
  Ramos Carreño <vnmabus>`<<.

*   Make sure that "datasets.fetch_lfw_people()" and
  "datasets.fetch_lfw_pairs()" internally crops images based on the
  *slice_* parameter. >>:pr:`24951`<< by >>:user:`Guillaume Lemaitre

*   "decomposition.FastICA.fit()" has been optimised w.r.t its memory
  footprint and runtime. >>:pr:`22268`<< by >>:user:`MohamedBsh
  <Bsh>`<<.

*   "decomposition.SparsePCA" and "decomposition.MiniBatchSparsePCA"
  now implements an *inverse_transform* function. >>:pr:`23905`<< by

*   "decomposition.FastICA" now allows the user to select how
  whitening is performed through the new *whiten_solver* parameter,
  which supports *svd* and *eigh*. *whiten_solver* defaults to *svd*
  although *eigh* may be faster and more memory efficient in cases
  where *num_features > num_samples*. >>:pr:`11860`<< by
  >>:user:`Pierre Ablin <pierreablin>`<<, >>:pr:`22527`<< by

although *eigh* may be faster and more memory efficient in cases
  >>:user:`Meekail Zain <micky774>`<< and Thomas Fan.

*   "decomposition.LatentDirichletAllocation" now preserves dtype for
  *numpy.float32* input. >>:pr:`24528`<< by >>:user:`Takeshi Oura
  <takoika>`<< and >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<.

*   The *n_iter* parameter of "decomposition.MiniBatchSparsePCA" is
  deprecated and replaced by the parameters *max_iter*, *tol*, and
  *max_no_improvement* to be consistent with
  "decomposition.MiniBatchDictionaryLearning". *n_iter* will be
  removed in version 1.3. >>:pr:`23726`<< by >>:user:`Guillaume

*   The *n_features_* attribute of "decomposition.PCA" is deprecated
  in favor of *n_features_in_* and will be removed in 1.4.
  >>:pr:`24421`<< by >>:user:`Kshitij Mathur <Kshitij68>`<<.

"sklearn.discriminant_analysis"

*   "discriminant_analysis.LinearDiscriminantAnalysis" now supports
  the Array API for *solver="svd"*. Array API support is considered
  experimental and might evolve without being subjected to our usual
  rolling deprecation cycle policy. See Array API support
  (experimental) for more details. >>:pr:`22554`<< by Thomas Fan.

experimental and might evolve without being subjected to our usual

*   Validate parameters only in *fit* and not in *__init__* for
  "discriminant_analysis.QuadraticDiscriminantAnalysis".
  >>:pr:`24218`<< by >>:user:`Stefanie Molin <stefmolin>`<<.

*   "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor" now support interaction
  constraints via the argument *interaction_cst* of their
  constructors. >>:pr:`21020`<< by >>:user:`Christian Lorentzen
  <lorentzenchr>`<<. Using interaction constraints also makes fitting
  faster. >>:pr:`24856`<< by >>:user:`Christian Lorentzen
  <lorentzenchr>`<<.

*   Adds *class_weight* to "ensemble.HistGradientBoostingClassifier".
  >>:pr:`22014`<< by Thomas Fan.

*   Improve runtime performance of "ensemble.IsolationForest" by
  avoiding data copies. >>:pr:`23252`<< by >>:user:`Zhehao Liu
  <MaxwellLZH>`<<.

*   "ensemble.StackingClassifier" now accepts any kind of base
  estimator. >>:pr:`24538`<< by >>:user:`Guillem G Subies
  <GuillemGSubies>`<<.

*   Make it possible to pass the *categorical_features* parameter of
  "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor" as feature names.
  >>:pr:`24889`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   "ensemble.StackingClassifier" now supports multilabel-indicator

"ensemble.HistGradientBoostingRegressor" as feature names.

  target >>:pr:`24146`<< by >>:user:`Nicolas Peretti <nicoperetti>`<<,
  >>:user:`Nestor Navarro <nestornav>`<<, >>:user:`Nati Tomattis
  <natitomattis>`<<, and >>:user:`Vincent Maladiere <Vincent-

  "ensemble.HistGradientBoostingClassifier" now accept their
  *monotonic_cst* parameter to be passed as a dictionary in addition
  to the previously supported array-like format. Such dictionary have
  feature names as keys and one of *-1*, *0*, *1* as value to specify
  monotonicity constraints for each feature. >>:pr:`24855`<< by
  >>:user:`Olivier Grisel <ogrisel>`<<.

*   Interaction constraints for
  "ensemble.HistGradientBoostingRegressor" can now be specified as
  strings for two common cases: "no_interactions" and "pairwise"
  interactions. >>:pr:`24849`<< by >>:user:`Tim Head <betatim>`<<.

*   Fixed the issue where "ensemble.AdaBoostClassifier" outputs NaN in
  feature importance when fitted with very small sample weight.
  >>:pr:`20415`<< by >>:user:`Zhehao Liu <MaxwellLZH>`<<.

  "ensemble.HistGradientBoostingRegressor" no longer error when
  predicting on categories encoded as negative values and instead

  consider them a member of the "missing category". >>:pr:`24283`<< by
  Thomas Fan.

  "ensemble.HistGradientBoostingRegressor", with *verbose>=1*, print
  detailed timing information on computing histograms and finding best
  splits. The time spent in the root node was previously missing and
  is now included in the printed information. >>:pr:`24894`<< by

*   Rename the constructor parameter *base_estimator* to *estimator*
  in the following classes: "ensemble.BaggingClassifier",
  "ensemble.BaggingRegressor", "ensemble.AdaBoostClassifier",
  "ensemble.AdaBoostRegressor". *base_estimator* is deprecated in 1.2
  and will be removed in 1.4. >>:pr:`23819`<< by >>:user:`Adrian
  Trujillo <trujillo9616>`<< and >>:user:`Edoardo Abati <EdAbati>`<<.

*   Rename the fitted attribute *base_estimator_* to *estimator_* in
  the following classes: "ensemble.BaggingClassifier",
  "ensemble.AdaBoostRegressor", "ensemble.RandomForestClassifier",
  "ensemble.RandomForestRegressor", "ensemble.ExtraTreesClassifier",
  "ensemble.ExtraTreesRegressor", "ensemble.RandomTreesEmbedding",
  "ensemble.IsolationForest". *base_estimator_* is deprecated in 1.2

"ensemble.ExtraTreesRegressor", "ensemble.RandomTreesEmbedding",

"sklearn.feature_selection"

*   Fix a bug in "feature_selection.mutual_info_regression()" and
  "feature_selection.mutual_info_classif()", where the continuous
  features in *X* should be scaled to a unit variance independently if
  the target *y* is continuous or discrete. >>:pr:`24747`<< by
  >>:user:`Guillaume Lemaitre <glemaitre>`<<

"sklearn.gaussian_process"
--------------------------

*   Fix "gaussian_process.kernels.Matern" gradient computation with
  *nu=0.5* for PyPy (and possibly other non CPython interpreters).
  >>:pr:`24245`<< by >>:user:`Loïc Estève <lesteve>`<<.

*   The *fit* method of "gaussian_process.GaussianProcessRegressor"
  will not modify the input X in case a custom kernel is used, with a
  *diag* method that returns part of the input X. >>:pr:`24405`<< by
  >>:user:`Omar Salman <OmarManzoor>`<<.

"sklearn.impute"
----------------

*   Added *keep_empty_features* parameter to "impute.SimpleImputer",
  "impute.KNNImputer" and "impute.IterativeImputer", preventing
  removal of features containing only missing values when
  transforming. >>:pr:`16695`<< by >>:user:`Vitor Santa Rosa
  <vitorsrg>`<<.

"sklearn.inspection"

*   Extended "inspection.partial_dependence()" and

transforming. >>:pr:`16695`<< by >>:user:`Vitor Santa Rosa

  "inspection.PartialDependenceDisplay" to handle categorical
  features. >>:pr:`18298`<< by >>:user:`Madhura Jayaratne
  <madhuracj>`<< and >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   "inspection.DecisionBoundaryDisplay" now raises error if input
  data is not 2-dimensional. >>:pr:`25077`<< by >>:user:`Arturo Amor
  <ArturoAmorQ>`<<.

"sklearn.kernel_approximation"

*   "kernel_approximation.RBFSampler" now preserves dtype for
  *numpy.float32* inputs. >>:pr:`24317`<< by *Tim Head <betatim>*.

*   "kernel_approximation.SkewedChi2Sampler" now preserves dtype for
  *numpy.float32* inputs. >>:pr:`24350`<< by >>:user:`Rahil Parikh
  <rprkh>`<<.

*   "kernel_approximation.RBFSampler" now accepts *'scale'* option for
  parameter *gamma*. >>:pr:`24755`<< by >>:user:`Gleb Levitski
  <GLevV>`<<.

*   "linear_model.LogisticRegression",
  "linear_model.LogisticRegressionCV", "linear_model.GammaRegressor",
  "linear_model.PoissonRegressor" and "linear_model.TweedieRegressor"
  got a new solver *solver="newton-cholesky"*. This is a 2nd order
  (Newton) optimisation routine that uses a Cholesky decomposition of
  the hessian matrix. When *n_samples >> n_features*, the *"newton-
  cholesky"* solver has been observed to converge both faster and to a

the hessian matrix. When *n_samples >> n_features*, the *"newton-
  higher precision solution than the *"lbfgs"* solver on problems with
  one-hot encoded categorical variables with some rare categorical
  levels. >>:pr:`24637`<< and >>:pr:`24767`<< by >>:user:`Christian
  Lorentzen <lorentzenchr>`<<.

*   "linear_model.SGDClassifier" and "linear_model.SGDRegressor" will
  raise an error when all the validation samples have zero sample
  weight. >>:pr:`23275`<< by *Zhehao Liu <MaxwellLZH>*.

*   "linear_model.SGDOneClassSVM" no longer performs parameter
  validation in the constructor. All validation is now handled in
  *fit()* and *partial_fit()*. >>:pr:`24433`<< by
  >>:user:`Yogendrasingh <iofall>`<<, >>:user:`Arisa Y. <arisayosh>`<<
  and >>:user:`Tim Head <betatim>`<<.

*   Fix average loss calculation when early stopping is enabled in
  "linear_model.SGDRegressor" and "linear_model.SGDClassifier". Also
  updated the condition for early stopping accordingly.
  >>:pr:`23798`<< by >>:user:`Harsh Agrawal <Harsh14901>`<<.

*   The default value for the *solver* parameter in

updated the condition for early stopping accordingly.

  "linear_model.QuantileRegressor" will change from *"interior-point"*
  to *"highs"* in version 1.4. >>:pr:`23637`<< by >>:user:`Guillaume

*   String option *"none"* is deprecated for *penalty* argument in
  "linear_model.LogisticRegression", and will be removed in version
  1.4. Use *None* instead. >>:pr:`23877`<< by >>:user:`Zhehao Liu

  "linear_model.RidgeClassifier". >>:pr:`24465`<< by

*   Adds option to use the normalized stress in "manifold.MDS". This
  is enabled by setting the new *normalize* parameter to *True*.
  >>:pr:`10168`<< by >>:user:`Łukasz Borchmann <Borchmann>`<<,
  >>:pr:`12285`<< by >>:user:`Matthias Miltenberger <mattmilten>`<<,
  >>:pr:`13042`<< by >>:user:`Matthieu Parizy <matthieu-pa>`<<,
  >>:pr:`18094`<< by >>:user:`Roth E Conrad <rotheconrad>`<< and
  >>:pr:`22562`<< by >>:user:`Meekail Zain <micky774>`<<.

*   Adds *eigen_tol* parameter to "manifold.SpectralEmbedding". Both
  "manifold.spectral_embedding()" and "manifold.SpectralEmbedding" now
  propogate *eigen_tol* to all choices of *eigen_solver*. Includes a

"manifold.spectral_embedding()" and "manifold.SpectralEmbedding" now
  new option *eigen_tol="auto"* and begins deprecation to change the
  default from *eigen_tol=0* to *eigen_tol="auto"* in version 1.3.
  >>:pr:`23210`<< by >>:user:`Meekail Zain <micky774>`<<.

*   "manifold.Isomap" now preserves dtype for *np.float32* inputs.
  >>:pr:`24714`<< by >>:user:`Rahil Parikh <rprkh>`<<.

*   Added an *"auto"* option to the *normalized_stress* argument in
  "manifold.MDS" and "manifold.smacof()". Note that
  *normalized_stress* is only valid for non-metric MDS, therefore the
  *"auto"* option enables *normalized_stress* when *metric=False* and
  disables it when *metric=True*. *"auto"* will become the default
  value foor *normalized_stress* in version 1.4. >>:pr:`23834`<< by

"sklearn.metrics"

*   "metrics.ConfusionMatrixDisplay.from_estimator()",
  "metrics.ConfusionMatrixDisplay.from_predictions()", and
  "metrics.ConfusionMatrixDisplay.plot()" accepts a *text_kw*
  parameter which is passed to matplotlib's *text* function.
  >>:pr:`24051`<< by Thomas Fan.

*   "metrics.class_likelihood_ratios()" is added to compute the
  positive and negative likelihood ratios derived from the confusion
  matrix of a binary classification problem. >>:pr:`22518`<< by
  >>:user:`Arturo Amor <ArturoAmorQ>`<<.

*   Add "metrics.PredictionErrorDisplay" to plot residuals vs

matrix of a binary classification problem. >>:pr:`22518`<< by

  predicted and actual vs predicted to qualitatively assess the
  behavior of a regressor. The display can be created with the class
  methods "metrics.PredictionErrorDisplay.from_estimator()" and
  "metrics.PredictionErrorDisplay.from_predictions()". >>:pr:`18020`<<

*   "metrics.roc_auc_score()" now supports micro-averaging
  (*average="micro"*) for the One-vs-Rest multiclass case
  (*multi_class="ovr"*). >>:pr:`24338`<< by >>:user:`Arturo Amor

*   Adds an *"auto"* option to *eps* in "metrics.logloss()". This
  option will automatically set the *eps* value depending on the data
  type of *y_pred*. In addition, the default value of *eps* is changed
  from *1e-15* to the new *"auto"* option. >>:pr:`24354`<< by
  >>:user:`Safiuddin Khaja <Safikh>`<< and >>:user:`gsiisg
  <gsiisg>`<<.

*   Allows *csr_matrix* as input for parameter: *y_true* of
     the "metrics.label_ranking_average_precision_score()" metric.
     >>:pr:`23442`<< by >>:user:`Sean Atukorala <ShehanAT>`<<

*   "metrics.ndcg_score()" will now trigger a warning when the
  *y_true* value contains a negative value. Users may still use
  negative values, but the result may not be between 0 and 1. Starting
  in v1.4, passing in negative values for *y_true* will raise an

*y_true* value contains a negative value. Users may still use
  error. >>:pr:`22710`<< by >>:user:`Conroy Trinh <trinhcon>`<< and
  >>:pr:`23461`<< by >>:user:`Meekail Zain <micky774>`<<.

*   "metrics.log_loss()" with *eps=0* now returns a correct value of 0
  or *np.inf* instead of *nan* for predictions at the boundaries (0 or
  1). It also accepts integer input. >>:pr:`24365`<< by

*   The parameter *sum_over_features* of
  "metrics.pairwise.manhattan_distances()" is deprecated and will be
  removed in 1.4. >>:pr:`24630`<< by >>:user:`Rushil Desai
  <rusdes>`<<.

*   Added the class "model_selection.LearningCurveDisplay" that allows
  to make easy plotting of learning curves obtained by the function
  "model_selection.learning_curve()". >>:pr:`24084`<< by

*   For all *SearchCV* classes and scipy >= 1.10, rank corresponding
  to a nan score is correctly set to the maximum possible rank, rather
  than *np.iinfo(np.int32).min*. >>:pr:`24141`<< by >>:user:`Loïc
  Estève <lesteve>`<<.

*   In both "model_selection.HalvingGridSearchCV" and
  "model_selection.HalvingRandomSearchCV" parameter combinations with
  a NaN score now share the lowest rank. >>:pr:`24539`<< by

"model_selection.HalvingRandomSearchCV" parameter combinations with

"sklearn.multioutput"

*   Added boolean *verbose* flag to classes:
  "multioutput.ClassifierChain" and "multioutput.RegressorChain".
  >>:pr:`23977`<< by >>:user:`Eric Fiegel <efiegel>`<<,
  >>:user:`Chiara Marmo <cmarmo>`<<, >>:user:`Lucy Liu <lucyleeow>`<<,
  and >>:user:`Guillaume Lemaitre <glemaitre>`<<.

"sklearn.naive_bayes"

*   Add methods *predict_joint_log_proba* to all naive Bayes
  classifiers. >>:pr:`23683`<< by >>:user:`Andrey Melnik <avm19>`<<.

*   A new parameter *force_alpha* was added to
  "naive_bayes.BernoulliNB", "naive_bayes.ComplementNB",
  "naive_bayes.CategoricalNB", and "naive_bayes.MultinomialNB",
  allowing user to set parameter alpha to a very small number, greater
  or equal 0, which was earlier automatically changed to *1e-10*
  instead. >>:pr:`16747`<< by >>:user:`arka204`<<, >>:pr:`18805`<< by
  >>:user:`hongshaoyang`<<, >>:pr:`22269`<< by >>:user:`Meekail Zain

*   Adds new function "neighbors.sort_graph_by_row_values()" to sort a

>>:user:`hongshaoyang`<<, >>:pr:`22269`<< by >>:user:`Meekail Zain

  CSR sparse graph such that each row is stored with increasing
  values. This is useful to improve efficiency when using precomputed
  sparse distance matrices in a variety of estimators and avoid an
  *EfficiencyWarning*. >>:pr:`23139`<< by Tom Dupre la Tour.

*   "neighbors.NearestCentroid" is faster and requires less memory as
  it better leverages CPUs' caches to compute predictions.
  >>:pr:`24645`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   "neighbors.KernelDensity" bandwidth parameter now accepts
  definition using Scott's and Silverman's estimation methods.
  >>:pr:`10468`<< by >>:user:`Ruben <icfly2>`<< and >>:pr:`22993`<< by
  >>:user:`Jovan Stojanovic <jovan-stojanovic>`<<.

*   "neighbors.NeighborsBase" now accepts Minkowski semi-metric (i.e.
  when 0 < p < 1 for *metric="minkowski"*) for *algorithm="auto"* or
  *algorithm="brute"*. >>:pr:`24750`<< by >>:user:`Rudresh Veerkhare
  <RudreshVeerkhare>`<<

*   "neighbors.NearestCentroid" now raises an informative error
  message at fit-time instead of failing with a low-level error
  message at predict-time. >>:pr:`23874`<< by >>:user:`Juan Gomez
  <2357juan>`<<.

*   Set *n_jobs=None* by default (instead of *1*) for
  "neighbors.KNeighborsTransformer" and
  "neighbors.RadiusNeighborsTransformer". >>:pr:`24075`<< by

<2357juan>`<<.

  >>:user:`Valentin Laurent <Valentin-Laurent>`<<.

*   "neighbors.LocalOutlierFactor" now preserves dtype for
  *numpy.float32* inputs. >>:pr:`22665`<< by >>:user:`Julien

  always expose the parameters *best_loss_*, *validation_scores_*, and
  *best_validation_score_*. *best_loss_* is set to *None* when
  *early_stopping=True*, while *validation_scores_* and
  *best_validation_score_* are set to *None* when
  *early_stopping=False*. >>:pr:`24683`<< by >>:user:`Guillaume

*   "pipeline.FeatureUnion.get_feature_names_out()" can now be used
  when one of the transformers in the "pipeline.FeatureUnion" is
  *"passthrough"*. >>:pr:`24058`<< by >>:user:`Diederik Perdok
  <diederikwp>`<<

*   The "pipeline.FeatureUnion" class now has a *named_transformers*
  attribute for accessing transformers by name. >>:pr:`20331`<< by
  >>:user:`Christopher Flynn <crflynn>`<<.

*   "preprocessing.FunctionTransformer" will always try to set
  *n_features_in_* and *feature_names_in_* regardless of the
  *validate* parameter. >>:pr:`23993`<< by Thomas Fan.

*   "preprocessing.LabelEncoder" correctly encodes NaNs in
  *transform*. >>:pr:`22629`<< by Thomas Fan.

*   The *sparse* parameter of "preprocessing.OneHotEncoder" is now
  deprecated and will be removed in version 1.4. Use *sparse_output*
  instead. >>:pr:`24412`<< by >>:user:`Rushil Desai <rusdes>`<<.

"sklearn.svm"

*   The *class_weight_* attribute is now deprecated for "svm.NuSVR",
  "svm.SVR", "svm.OneClassSVM". >>:pr:`22898`<< by >>:user:`Meekail
  Zain <micky774>`<<.

*   "tree.plot_tree()", "tree.export_graphviz()" now uses a lower case
  *x[i]* to represent feature *i*. >>:pr:`23480`<< by Thomas Fan.

*   A new module exposes development tools to discover estimators
  (i.e. "utils.discovery.all_estimators()"), displays (i.e.
  "utils.discovery.all_displays()") and functions (i.e.
  "utils.discovery.all_functions()") in scikit-learn. >>:pr:`21469`<<

*   "utils.extmath.randomized_svd()" now accepts an argument,
  *lapack_svd_driver*, to specify the lapack driver used in the
  internal deterministic SVD used by the randomized SVD algorithm.
  >>:pr:`20617`<< by >>:user:`Srinath Kailasa <skailasa>`<<

*   "utils.validation.column_or_1d()" now accepts a *dtype* parameter

internal deterministic SVD used by the randomized SVD algorithm.

  to specific *y*'s dtype. >>:pr:`22629`<< by Thomas Fan.

*   "utils.extmath.cartesian()" now accepts arrays with different
  *dtype* and will cast the ouptut to the most permissive *dtype*.
  >>:pr:`25067`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   "utils.multiclass.type_of_target()" now properly handles sparse
  matrices. >>:pr:`14862`<< by >>:user:`Léonard Binet
  <leonardbinet>`<<.

*   HTML representation no longer errors when an estimator class is a
  value in *get_params*. >>:pr:`24512`<< by Thomas Fan.

*   "utils.estimator_checks.check_estimator()" now takes into account
  the *requires_positive_X* tag correctly. >>:pr:`24667`<< by Thomas
  Fan.

*   "utils.check_array()" now supports Pandas Series with *pd.NA* by
  raising a better error message or returning a compatible *ndarray*.
  >>:pr:`25080`<< by Thomas Fan.

*   The extra keyword parameters of "utils.extmath.density()" are
  deprecated and will be removed in 1.4. >>:pr:`24523`<< by
  >>:user:`Mia Bajic <clytaemnestra>`<<.

Code and Documentation Contributors
===================================

Thanks to everyone who has contributed to the maintenance and
improvement of the project since version 1.1, including:

2357juan, 3lLobo, Adam J. Stewart, Adam Li, Aditya Anulekh, Adrin

Jalali, Aiko, Akshita Prasanth, Alessandro Miola, Alex, Alexandr,
Alexandre Perez-Lebel, aman kumar, Amit Bera, Andreas Grivas, Andreas
Mueller, András Simon, angela-maennel, Aniket Shirsat, Antony Lee,
anupam, Apostolos Tsetoglou, Aravindh R, Artur Hermano, Arturo Amor,
Ashwin Mathur, avm19, b0rxington, Badr MOUFAD, Bardiya Ak, Bartłomiej
Gońda, BdeGraaff, Benjamin Bossan, Benjamin Carter, berkecanrizai,
Bernd Fritzke, Bhoomika, Biswaroop Mitra, Brandon TH Chen, Brett
Cannon, Bsh, carlo, Carlos Ramos Carreño, ceh, chalulu, Charles
Zablit, Chiara Marmo, Christian Lorentzen, Christian Ritter,
christianwaldmann, Christine P. Chai, Claudio Salvatore Arcidiacono,
Clément Verrier, crispinlogan, Da-Lan, DanGonite57, Daniela Fernandes,
DanielGaerber, darioka, Darren Nguyen, David Gilbertson, David Poznik,
david-cortes, Denis, Dev Khant, Dhanshree Arora, Diadochokinetic,
diederikwp, Dimitri Papadopoulos Orfanos, drewhogg, Duarte OC, Dwight
Lindquist, Eden Brekke, Edoardo Abati, Eleanore Denies, EliaSchiavon,
ErmolaevPA, Fabrizio Damicelli, fcharras, Flynn, francesco-tuveri,
Franck Charras, ftorres16, Gael Varoquaux, Geevarghese George,
GeorgiaMayDay, Gianr Lazz, Gleb Levitski, Glòria Macià Muñoz,
Guillaume Lemaitre, Guillem García Subies, Guitared, gunesbayir,

Hansin Ahuja, Hao Chun Chang, Harsh Agrawal, harshit5674, hasan-yaman,
Henry Sorsky, henrymooresc, Hristo Vrigazov, htsedebenham, humahn,
i-aki-y, Ido M, Iglesys, Iliya Zhechev, Irene, Ivan Sedykh, ivanllt,
jakirkham, Jason G, Jiten Sidhpura, jkarolczak, John Koumentis, John
P, John Pangas, johnthagen, Jordan Fleming, Joshua Choo Yun Keat,
Jovan Stojanovic, João David, Juan Carlos Alfaro Jiménez, Juan Felipe
Arias, juanfe88, Julien Jerphanion, jygerardy, Jérémie du
Boisberranger, Kanishk Sachdev, Kanissh, Kendall, Kenneth Prabakaran,
Kento Nozawa, kernc, Kevin Roice, Kian Eliasi, Kilian Kluge, Kilian
Lieret, Kirandevraj, Kraig, krishna kumar, krishna vamsi, Kshitij
Kapadni, Kshitij Mathur, Lauren Burke, lingyi1110, Lisa Casino, Loic
Esteve, Luciano Mantovani, Lucy Liu, Léonard Binet, m. bou, Maascha,
Madhura Jayaratne, madinak, Maksym, Malte S. Kurz, Mansi Agrawal,
Marco Edward Gorelli, Marco Wurps, Maren Westermann, Maria Telenczuk,
martin-kokos, mathurinm, mauroantonioserrano, Maxi Marufo, Maxim
Smolskiy, Maxwell, Meekail Zain, Mehgarg, mehmetcanakbay, Mia Bajić,
Michael Flaks, Michael Hornstein, Michel de Ruiter, Michelle Paradis,
Misa Ogura, Moritz Wilksch, mrastgoo, Naipawat Poolsawat, Naoise
Holohan, Nass, Nathan Jacobi, Nguyễn Văn Diễn, Nihal Thukarama Rao,

Nikita Jare, Nima Sarajpoor, nima10khodaveisi, nitinramvelraj, Nwanna-
Joseph, Nymark Kho, o-holman, Olivier Grisel, Olle Lukowski, Omar
Hassoun, Omar Salman, osman tamer, Oyindamola Olatunji, PAB, Pandata,
Paulo Sergio  Soares, Petar Mlinarić, Peter Jansson, Peter Steinbach,
Philipp Jung, Piet Brömmel, priyam kakati, puhuk, Rachel Freeland,
Rachit Keerti Das, Rafal Wojdyla, Rahil Parikh, ram vikram singh, Ravi
Makhija, Rehan Guha, Reshama Shaikh, Richard Klima, Rob Crockett,
Robert Hommes, Robert Juergens, Robin Lenz, Rocco Meli, Roman4oo, Ross
Barnowski, Rowan Mankoo, Rudresh Veerkhare, Rushil Desai, Sabri Monaf
Sabri, Safikh, Safiuddin Khaja, Salahuddin, Sam Adam Day, Sandra
Yojana Meneses, Sandro Ephrem, Sangam, SangamSwadik, SarahRemus,
SavkoMax, Scott Gigante, Scott Gustafson, Sean Atukorala, sec65,
SELEE, seljaks, Shane, shellyfung, Shinsuke Mori, Shoaib Khan, Shogo
Hida, Shrankhla Srivastava, Shuangchi He, Simon, Srinath Kailasa,
Stefanie Molin, stellalin7, Steve Schmerler, Steven Van Vaerenbergh,
Stéphane Collot, Sven Stehle, the-syd-sre, TheDevPanda, Thomas Bonald,
Thomas Germer, Thomas J. Fan, Ti-Ion, Tim Head, Timofei Kornev,
toastedyeast, Tobias Pitters, Tom Dupré la Tour, Tom Mathews, Tom
McTiernan, tspeng, Tyler Egashira, Valentin Laurent, Varun Jain, Vera

Komeyer, Vicente Reyes-Puerta, Vincent M, Vishal, wattai, wchathura,
WEN Hao, x110, Xiao Yuan, Xunius, yanhong-zhao-ef, Z Adil Khwaja

Version 0.21.3
**************

  memory.

**July 30, 2019**

* The v0.20.0 release notes failed to mention a backwards
  incompatibility in "metrics.make_scorer()" when *needs_proba=True*
  and *y_true* is binary. Now, the scorer function is supposed to
  accept a 1D *y_pred* (i.e., probability of the positive class, shape
  *(n_samples,)*), instead of a 2D *y_pred* (i.e., shape *(n_samples,
  2)*).

Changelog
=========

*   Fixed a bug in "cluster.KMeans" where computation with
  *init='random'* was single threaded for *n_jobs > 1* or *n_jobs =
  -1*. >>:pr:`12955`<< by >>:user:`Prabakaran Kumaresshan
  <nixphix>`<<.

*   Fixed a bug in "cluster.OPTICS" where users were unable to pass

-1*. >>:pr:`12955`<< by >>:user:`Prabakaran Kumaresshan

  float *min_samples* and *min_cluster_size*. >>:pr:`14496`<< by
  >>:user:`Fabian Klopfer <someusername1>`<< and >>:user:`Hanmin Qin
  <qinhanmin2014>`<<.

*   Fixed a bug in "cluster.KMeans" where KMeans++ initialisation
  could rarely result in an IndexError. >>:issue:`11756`<< by Joel
  Nothman.

"sklearn.compose"

*   Fixed an issue in "compose.ColumnTransformer" where using
  DataFrames whose column order differs between :func:"fit" and
  :func:"transform" could lead to silently passing incorrect columns
  to the "remainder" transformer. >>:pr:`14237`<< by *Andreas
  Schuderer <schuderer>*.

*   "datasets.fetch_california_housing()", "datasets.fetch_covtype()",
  "datasets.fetch_kddcup99()", "datasets.fetch_olivetti_faces()",
  "datasets.fetch_rcv1()", and
  "datasets.fetch_species_distributions()" try to persist the
  previously cache using the new "joblib" if the cached data was
  persisted using the deprecated "sklearn.externals.joblib". This
  behavior is set to be deprecated and removed in v0.23.
  >>:pr:`14197`<< by Adrin Jalali.

*   Fix zero division error in "HistGradientBoostingClassifier()" and
  "HistGradientBoostingRegressor()". >>:pr:`14024`<< by *Nicolas Hug
  <NicolasHug>*.

*   Fixed a bug in "impute.SimpleImputer" and

"HistGradientBoostingRegressor()". >>:pr:`14024`<< by *Nicolas Hug

  "impute.IterativeImputer" so that no errors are thrown when there
  are missing values in training data. >>:pr:`13974`<< by *Frank Hoang
  <fhoang7>*.

*   Fixed a bug in "inspection.plot_partial_dependence()" where
  "target" parameter was not being taken into account for multiclass
  problems. >>:pr:`14393`<< by >>:user:`Guillem G. Subies
  <guillemgsubies>`<<.

*   Fixed a bug in "linear_model.LogisticRegressionCV" where
  "refit=False" would fail depending on the "'multiclass'" and
  "'penalty'" parameters (regression introduced in 0.21).
  >>:pr:`14087`<< by Nicolas Hug.

*   Compatibility fix for "linear_model.ARDRegression" and
  Scipy>=1.3.0. Adapts to upstream changes to the default *pinvh*
  cutoff threshold which otherwise results in poor accuracy in some
  cases. >>:pr:`14067`<< by >>:user:`Tim Staley <timstaley>`<<.

*   Fixed a bug in "neighbors.NeighborhoodComponentsAnalysis" where
  the validation of initial parameters "n_components", "max_iter" and
  "tol" required too strict types. >>:pr:`14092`<< by >>:user:`Jérémie
  du Boisberranger <jeremiedbb>`<<.

*   Fixed bug in "tree.export_text()" when the tree has one feature

"tol" required too strict types. >>:pr:`14092`<< by >>:user:`Jérémie

  and a single feature name is passed in. >>:pr:`14053`<< by *Thomas
  Fan*.

*   Fixed an issue with "plot_tree()" where it displayed entropy
  calculations even for *gini* criterion in DecisionTreeClassifiers.
  >>:pr:`13947`<< by >>:user:`Frank Hoang <fhoang7>`<<.

Version 0.21.2

**24 May 2019**

Changelog
=========

*   Fixed a bug in "cross_decomposition.CCA" improving numerical
  stability when *Y* is close to zero. >>:pr:`13903`<< by Thomas Fan.

*   Fixed a bug in "metrics.pairwise.euclidean_distances()" where a
  part of the distance matrix was left un-instanciated for suffiently
  large float32 datasets (regression introduced in 0.21).
  >>:pr:`13910`<< by >>:user:`Jérémie du Boisberranger

*   Fixed a bug in "preprocessing.OneHotEncoder" where the new *drop*
  parameter was not reflected in *get_feature_names*. >>:pr:`13894`<<
  by >>:user:`James Myatt <jamesmyatt>`<<.

"sklearn.utils.sparsefuncs"

*   Fixed a bug where "min_max_axis()" would fail on 32-bit systems
  for certain large inputs. This affects "preprocessing.MaxAbsScaler",
  "preprocessing.normalize()" and "preprocessing.LabelBinarizer".

for certain large inputs. This affects "preprocessing.MaxAbsScaler",
  >>:pr:`13741`<< by >>:user:`Roddy MacSween <rlms>`<<.

Version 0.21.1

**17 May 2019**

This is a bug-fix release to primarily resolve some packaging issues
in version 0.21.0. It also includes minor documentation improvements
and some bug fixes.

Changelog
=========

*   Fixed a bug in "inspection.partial_dependence()" to only check
  classifier and not regressor for the multiclass-multioutput case.
  >>:pr:`14309`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   Fixed a bug in "metrics.pairwise_distances" where it would raise
  "AttributeError" for boolean metrics when "X" had a boolean dtype
  and "Y == None". >>:issue:`13864`<< by >>:user:`Paresh Mathur
  <rick2047>`<<.

*   Fixed two bugs in "metrics.pairwise_distances" when "n_jobs > 1".
  First it used to return a distance matrix with same dtype as input,
  even for integer dtype. Then the diagonal was not zeros for
  euclidean metric when "Y" is "X". >>:issue:`13877`<< by

*   Fixed a bug in "neighbors.KernelDensity" which could not be
  restored from a pickle if "sample_weight" had been used.
  >>:issue:`13772`<< by >>:user:`Aditya Vyas <aditya1702>`<<.

Version 0.21.0

**May 2019**

restored from a pickle if "sample_weight" had been used.

* "discriminant_analysis.LinearDiscriminantAnalysis" for multiclass
  classification.

* "discriminant_analysis.LinearDiscriminantAnalysis" with 'eigen'
  solver.

* "linear_model.BayesianRidge"

* Decision trees and derived ensembles when both *max_depth* and
  *max_leaf_nodes* are set.

* "linear_model.LogisticRegression" and
  "linear_model.LogisticRegressionCV" with 'saga' solver.

* "ensemble.GradientBoostingClassifier"

* "sklearn.feature_extraction.text.HashingVectorizer",
  "sklearn.feature_extraction.text.TfidfVectorizer", and
  "sklearn.feature_extraction.text.CountVectorizer"

* "neural_network.MLPClassifier"

* "svm.SVC.decision_function()" and
  "multiclass.OneVsOneClassifier.decision_function()".

* "linear_model.SGDClassifier" and any derived classifiers.

* Any model using the "linear_model._sag.sag_solver()" function with a
  *0* seed, including "linear_model.LogisticRegression",
  "linear_model.LogisticRegressionCV", "linear_model.Ridge", and
  "linear_model.RidgeCV" with 'sag' solver.

* "linear_model.RidgeCV" when using leave-one-out cross-validation

"linear_model.LogisticRegressionCV", "linear_model.Ridge", and

  with sparse inputs.

Details are listed in the changelog below.

(While we are trying to better inform users by providing this
information, we cannot assure that this list is complete.)

Known Major Bugs

* The default *max_iter* for "linear_model.LogisticRegression" is too
  small for many solvers given the default *tol*. In particular, we
  accidentally changed the default *max_iter* for the liblinear solver
  from 1000 to 100 iterations in >>:pr:`3591`<< released in version
  0.16. In a future release we hope to choose better default
  *max_iter* and *tol* heuristically depending on the solver (see
  >>:pr:`13317`<<).

Changelog
=========

Support for Python 3.4 and below has been officially dropped.

*   The R2 score used when calling "score" on a regressor will use
  "multioutput='uniform_average'" from version 0.23 to keep consistent
  with "metrics.r2_score()". This will influence the "score" method of
  all the multioutput regressors (except for
  "multioutput.MultiOutputRegressor"). >>:pr:`13157`<< by
  >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Added support to bin the data passed into
  "calibration.calibration_curve" by quantiles instead of uniformly
  between 0 and 1. >>:pr:`13086`<< by >>:user:`Scott Cole <srcole>`<<.

*   Allow n-dimensional arrays as input for
  *calibration.CalibratedClassifierCV*. >>:pr:`13485`<< by
  >>:user:`William de Vazelhes <wdevazelhes>`<<.

*   A new clustering algorithm: "cluster.OPTICS": an algorithm related
  to "cluster.DBSCAN", that has hyperparameters easier to set and that
  scales better, by >>:user:`Shane <espg>`<<, Adrin Jalali,
  >>:user:`Erich Schubert <kno10>`<<, Hanmin Qin, and >>:user:`Assia
  Benbihi <assiaben>`<<.

*   Fixed a bug where "cluster.Birch" could occasionally raise an
  AttributeError. >>:pr:`13651`<< by Joel Nothman.

*   Fixed a bug in "cluster.KMeans" where empty clusters weren't
  correctly relocated when using sample weights. >>:pr:`13486`<< by

*   The "n_components_" attribute in "cluster.AgglomerativeClustering"
  and "cluster.FeatureAgglomeration" has been renamed to
  "n_connected_components_". >>:pr:`13427`<< by >>:user:`Stephane
  Couvreur <scouvreur>`<<.

*   "cluster.AgglomerativeClustering" and
  "cluster.FeatureAgglomeration" now accept a "distance_threshold"
  parameter which can be used to find the clusters instead of
  "n_clusters". >>:issue:`9069`<< by >>:user:`Vathsala Achar
  <VathsalaAchar>`<< and Adrin Jalali.

parameter which can be used to find the clusters instead of

*   "compose.ColumnTransformer" is no longer an experimental feature.
  >>:pr:`13835`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Added support for 64-bit group IDs and pointers in SVMLight files.
  >>:pr:`10727`<< by >>:user:`Bryan K Woods <bryan-woods>`<<.

*   "datasets.load_sample_images()" returns images with a
  deterministic order. >>:pr:`13250`<< by >>:user:`Thomas Fan
  <thomasjpfan>`<<.

*   "decomposition.KernelPCA" now has deterministic output (resolved
  sign ambiguity in eigenvalue decomposition of the kernel matrix).
  >>:pr:`13241`<< by >>:user:`Aurélien Bellet <bellet>`<<.

*   Fixed a bug in "decomposition.KernelPCA", *fit().transform()* now
  produces the correct output (the same as *fit_transform()*) in case
  of non-removed zero eigenvalues (*remove_zero_eig=False*).
  *fit_inverse_transform* was also accelerated by using the same trick
  as *fit_transform* to compute the transform of *X*. >>:pr:`12143`<<
  by >>:user:`Sylvain Marié <smarie>`<<

*   Fixed a bug in "decomposition.NMF" where *init = 'nndsvd'*, *init
  = 'nndsvda'*, and *init = 'nndsvdar'* are allowed when *n_components
  < n_features* instead of *n_components <= min(n_samples,

  n_features)*. >>:pr:`11650`<< by >>:user:`Hossein Pourbozorg
  <hossein-pourbozorg>`<< and >>:user:`Zijie (ZJ) Poh <zjpoh>`<<.

*   The default value of the "init" argument in
  "decomposition.non_negative_factorization()" will change from
  "random" to "None" in version 0.23 to make it consistent with
  "decomposition.NMF". A FutureWarning is raised when the default
  value is used. >>:pr:`12988`<< by >>:user:`Zijie (ZJ) Poh
  <zjpoh>`<<.

*   "discriminant_analysis.LinearDiscriminantAnalysis" now preserves
  "float32" and "float64" dtypes. >>:pr:`8769`<< and >>:pr:`11000`<<
  by >>:user:`Thibault Sejourne <thibsej>`<<

*   A "ChangedBehaviourWarning" is now raised when
  "discriminant_analysis.LinearDiscriminantAnalysis" is given as
  parameter "n_components > min(n_features, n_classes - 1)", and
  "n_components" is changed to "min(n_features, n_classes - 1)" if so.
  Previously the change was made, but silently. >>:pr:`11526`<< by
  >>:user:`William de Vazelhes<wdevazelhes>`<<.

*   Fixed a bug in "discriminant_analysis.LinearDiscriminantAnalysis"
  where the predicted probabilities would be incorrectly computed in
  the multiclass case. >>:pr:`6848`<<, by >>:user:`Agamemnon Krasoulis
  <agamemnonc>`<< and *Guillaume Lemaitre <glemaitre>*.

where the predicted probabilities would be incorrectly computed in

  where the predicted probabilities would be incorrectly computed with
  "eigen" solver. >>:pr:`11727`<<, by >>:user:`Agamemnon Krasoulis
  <agamemnonc>`<<.

"sklearn.dummy"

*   Fixed a bug in "dummy.DummyClassifier" where the "predict_proba"
  method was returning int32 array instead of float64 for the
  "stratified" strategy. >>:pr:`13266`<< by >>:user:`Christos
  Aridas<chkoar>`<<.

*   Fixed a bug in "dummy.DummyClassifier" where it was throwing a
  dimension mismatch error in prediction time if a column vector "y"
  with "shape=(n, 1)" was given at "fit" time. >>:pr:`13545`<< by
  >>:user:`Nick Sorros <nsorros>`<< and Adrin Jalali.

*   Add two new implementations of gradient boosting trees:
  "ensemble.HistGradientBoostingRegressor". The implementation of
  these estimators is inspired by LightGBM and can be orders of
  magnitude faster than "ensemble.GradientBoostingRegressor" and
  "ensemble.GradientBoostingClassifier" when the number of samples is
  larger than tens of thousands of samples. The API of these new
  estimators is slightly different, and some of the features from
  "ensemble.GradientBoostingClassifier" and

larger than tens of thousands of samples. The API of these new
  "ensemble.GradientBoostingRegressor" are not yet supported.

  These new estimators are experimental, which means that their
  results or their API might change without any deprecation cycle. To
  use them, you need to explicitly import
  "enable_hist_gradient_boosting":

     >>> # explicitly require this experimental feature
     >>> from sklearn.experimental import enable_hist_gradient_boosting  # noqa
     >>> # now you can import normally from sklearn.ensemble
     >>> from sklearn.ensemble import HistGradientBoostingClassifier

  Note:

    Update: since version 1.0, these estimators are not experimental
    anymore and you don't need to use *from sklearn.experimental
    import enable_hist_gradient_boosting*.

  >>:pr:`12807`<< by >>:user:`Nicolas Hug<NicolasHug>`<<.

*   Add "ensemble.VotingRegressor" which provides an equivalent of
  "ensemble.VotingClassifier" for regression problems. >>:pr:`12513`<<
  by >>:user:`Ramil Nugmanov <stsouko>`<< and >>:user:`Mohamed Ali
  Jamaoui <mohamed-ali>`<<.

*   Make "ensemble.IsolationForest" prefer threads over processes when
  running with "n_jobs > 1" as the underlying decision tree fit calls
  do release the GIL. This changes reduces memory usage and
  communication overhead. >>:pr:`12543`<< by >>:user:`Isaac Storch
  <istorch>`<< and Olivier Grisel.

do release the GIL. This changes reduces memory usage and

*   Make "ensemble.IsolationForest" more memory efficient by avoiding
  keeping in memory each tree prediction. >>:pr:`13260`<< by Nicolas
  Goix.

*   "ensemble.IsolationForest" now uses chunks of data at prediction
  step, thus capping the memory usage. >>:pr:`13283`<< by Nicolas
  Goix.

*   "sklearn.ensemble.GradientBoostingClassifier" and
  "sklearn.ensemble.GradientBoostingRegressor" now keep the input "y"
  as "float64" to avoid it being copied internally by trees.
  >>:pr:`13524`<< by Adrin Jalali.

*   Minimized the validation of X in "ensemble.AdaBoostClassifier" and
  "ensemble.AdaBoostRegressor" >>:pr:`13174`<< by >>:user:`Christos
  Aridas <chkoar>`<<.

*   "ensemble.IsolationForest" now exposes "warm_start" parameter,
  allowing iterative addition of trees to an isolation forest.
  >>:pr:`13496`<< by >>:user:`Peter Marko <petibear>`<<.

*   The values of "feature_importances_" in all random forest based
  models (i.e. "ensemble.RandomForestClassifier",
  "ensemble.GradientBoostingClassifier", and
  "ensemble.GradientBoostingRegressor") now:

  * sum up to "1"

  * all the single node trees in feature importance calculation are
    ignored

"ensemble.GradientBoostingClassifier", and

    ignored

  * in case all trees have only one single node (i.e. a root node),
    feature importances will be an array of all zeros.

  >>:pr:`13636`<< and >>:pr:`13620`<< by Adrin Jalali.

*   Fixed a bug in "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor", which didn't support scikit-
  learn estimators as the initial estimator. Also added support of
  initial estimator which does not support sample weights.
  >>:pr:`12436`<< by >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<
  and >>:pr:`12983`<< by >>:user:`Nicolas Hug<NicolasHug>`<<.

*   Fixed the output of the average path length computed in
  "ensemble.IsolationForest" when the input is either 0, 1 or 2.
  >>:pr:`13251`<< by >>:user:`Albert Thomas <albertcthomas>`<< and
  >>:user:`joshuakennethjones <joshuakennethjones>`<<.

*   Fixed a bug in "ensemble.GradientBoostingClassifier" where the
  gradients would be incorrectly computed in multiclass classification
  problems. >>:pr:`12715`<< by >>:user:`Nicolas Hug<NicolasHug>`<<.

*   Fixed a bug in "ensemble.GradientBoostingClassifier" where
  validation sets for early stopping were not sampled with
  stratification. >>:pr:`13164`<< by >>:user:`Nicolas
  Hug<NicolasHug>`<<.

validation sets for early stopping were not sampled with

  default initial prediction of a multiclass classifier would predict
  the classes priors instead of the log of the priors. >>:pr:`12983`<<
  by >>:user:`Nicolas Hug<NicolasHug>`<<.

*   Fixed a bug in "ensemble.RandomForestClassifier" where the
  "predict" method would error for multiclass multioutput forests
  models if any targets were strings. >>:pr:`12834`<< by
  >>:user:`Elizabeth Sander <elsander>`<<.

*   Fixed a bug in "ensemble.gradient_boosting.LossFunction" and
  "ensemble.gradient_boosting.LeastSquaresError" where the default
  value of "learning_rate" in "update_terminal_regions" is not
  consistent with the document and the caller functions. Note however
  that directly using these loss functions is deprecated.
  >>:pr:`6463`<< by >>:user:`movelikeriver <movelikeriver>`<<.

*   "ensemble.partial_dependence()" (and consequently the new version
  "sklearn.inspection.partial_dependence()") now takes sample weights
  into account for the partial dependence computation when the
  gradient boosting model has been trained with sample weights.
  >>:pr:`13193`<< by >>:user:`Samuel O. Ronsin <samronsin>`<<.

*   "ensemble.partial_dependence()" and
  "ensemble.plot_partial_dependence()" are now deprecated in favor of
  "inspection.partial_dependence" and

  "inspection.plot_partial_dependence". >>:pr:`12599`<< by
  >>:user:`Trevor Stephens<trevorstephens>`<< and >>:user:`Nicolas

*   "ensemble.VotingClassifier" and "ensemble.VotingRegressor" were
  failing during "fit" in one of the estimators was set to "None" and
  "sample_weight" was not "None". >>:pr:`13779`<< by

*   "ensemble.VotingClassifier" and "ensemble.VotingRegressor" accept
  "'drop'" to disable an estimator in addition to "None" to be
  consistent with other estimators (i.e., "pipeline.FeatureUnion" and
  "compose.ColumnTransformer"). >>:pr:`13780`<< by >>:user:`Guillaume

"sklearn.externals"

*   Deprecated "externals.six" since we have dropped support for
  Python 2.7. >>:pr:`12916`<< by >>:user:`Hanmin Qin

*   If "input='file'" or "input='filename'", and a callable is given
  as the "analyzer",
  "sklearn.feature_extraction.text.HashingVectorizer",
  "sklearn.feature_extraction.text.CountVectorizer" now read the data
  from the file(s) and then pass it to the given "analyzer", instead
  of passing the file name(s) or the file object(s) to the analyzer.
  >>:pr:`13641`<< by Adrin Jalali.

from the file(s) and then pass it to the given "analyzer", instead

*   Added "impute.IterativeImputer", which is a strategy for imputing
  missing values by modeling each feature with missing values as a
  function of other features in a round-robin fashion. >>:pr:`8478`<<
  and >>:pr:`12177`<< by >>:user:`Sergey Feldman <sergeyf>`<< and
  >>:user:`Ben Lawson <benlawson>`<<.

  The API of IterativeImputer is experimental and subject to change
  without any deprecation cycle. To use them, you need to explicitly
  import "enable_iterative_imputer":

     >>> from sklearn.experimental import enable_iterative_imputer  # noqa
     >>> # now you can import normally from sklearn.impute
     >>> from sklearn.impute import IterativeImputer

*   The "impute.SimpleImputer" and "impute.IterativeImputer" have a
  new parameter "'add_indicator'", which simply stacks a
  "impute.MissingIndicator" transform into the output of the imputer's
  transform. That allows a predictive estimator to account for
  missingness. >>:pr:`12583`<<, >>:pr:`13601`<< by >>:user:`Danylo
  Baibak <DanilBaibak>`<<.

*   In "impute.MissingIndicator" avoid implicit densification by
  raising an exception if input is sparse add *missing_values*
  property is set to 0. >>:pr:`13240`<< by >>:user:`Bartosz Telenczuk
  <btel>`<<.

*   Fixed two bugs in "impute.MissingIndicator". First, when "X" is

property is set to 0. >>:pr:`13240`<< by >>:user:`Bartosz Telenczuk

  sparse, all the non-zero non missing values used to become explicit
  False in the transformed data. Then, when "features='missing-only'",
  all features used to be kept if there were no missing values at all.
  >>:pr:`13562`<< by >>:user:`Jérémie du Boisberranger

(new subpackage)

*   Partial dependence plots ("inspection.plot_partial_dependence()")
  are now supported for any regressor or classifier (provided that
  they have a *predict_proba* method). >>:pr:`12599`<< by
  >>:user:`Trevor Stephens <trevorstephens>`<< and >>:user:`Nicolas
  Hug <NicolasHug>`<<.

*   Allow different dtypes (such as float32) in
  "isotonic.IsotonicRegression". >>:pr:`8769`<< by >>:user:`Vlad
  Niculae <vene>`<<

*   "linear_model.Ridge" now preserves "float32" and "float64" dtypes.
  >>:issue:`8769`<< and >>:issue:`11000`<< by >>:user:`Guillaume
  Lemaitre <glemaitre>`<<, and >>:user:`Joan Massich <massich>`<<

*   "linear_model.LogisticRegression" and
  "linear_model.LogisticRegressionCV" now support Elastic-Net penalty,
  with the 'saga' solver. >>:pr:`11646`<< by >>:user:`Nicolas Hug
  <NicolasHug>`<<.

*   Added "linear_model.lars_path_gram", which is
  "linear_model.lars_path" in the sufficient stats mode, allowing

with the 'saga' solver. >>:pr:`11646`<< by >>:user:`Nicolas Hug

  users to compute "linear_model.lars_path" without providing "X" and
  "y". >>:pr:`11699`<< by >>:user:`Kuai Yu <yukuairoy>`<<.

*   "linear_model.make_dataset()" now preserves "float32" and
  "float64" dtypes, reducing memory consumption in stochastic
  gradient, SAG and SAGA solvers. >>:pr:`8769`<< and >>:pr:`11000`<<
  by >>:user:`Nelle Varoquaux <NelleV>`<<, >>:user:`Arthur Imbert
  <Henley13>`<<, >>:user:`Guillaume Lemaitre <glemaitre>`<<, and
  >>:user:`Joan Massich <massich>`<<

*   "linear_model.LogisticRegression" now supports an unregularized
  objective when "penalty='none'" is passed. This is equivalent to
  setting "C=np.inf" with l2 regularization. Not supported by the
  liblinear solver. >>:pr:`12860`<< by >>:user:`Nicolas Hug

*   *sparse_cg* solver in "linear_model.Ridge" now supports fitting
  the intercept (i.e. "fit_intercept=True") when inputs are sparse.
  >>:pr:`13336`<< by >>:user:`Bartosz Telenczuk <btel>`<<.

*   The coordinate descent solver used in *Lasso*, *ElasticNet*, etc.
  now issues a *ConvergenceWarning* when it completes without meeting
  the desired toleranbce. >>:pr:`11754`<< and >>:pr:`13397`<< by
  >>:user:`Brent Fagan <brentfagan>`<< and >>:user:`Adrin Jalali
  <adrinjalali>`<<.

*   Fixed a bug in "linear_model.LogisticRegression" and

>>:user:`Brent Fagan <brentfagan>`<< and >>:user:`Adrin Jalali

  "linear_model.LogisticRegressionCV" with 'saga' solver, where the
  weights would not be correctly updated in some cases.
  >>:pr:`11646`<< by Tom Dupre la Tour.

*   Fixed the posterior mean, posterior covariance and returned
  regularization parameters in "linear_model.BayesianRidge". The
  posterior mean and the posterior covariance were not the ones
  computed with the last update of the regularization parameters and
  the returned regularization parameters were not the final ones. Also
  fixed the formula of the log marginal likelihood used to compute the
  score when *compute_score=True*. >>:pr:`12174`<< by >>:user:`Albert
  Thomas <albertcthomas>`<<.

*   Fixed a bug in "linear_model.LassoLarsIC", where user input
  "copy_X=False" at instance creation would be overridden by default
  parameter value "copy_X=True" in "fit". >>:pr:`12972`<< by
  >>:user:`Lucio Fernandez-Arjona <luk-f-a>`<<

*   Fixed a bug in "linear_model.LinearRegression" that was not
  returning the same coeffecients and intercepts with
  "fit_intercept=True" in sparse and dense case. >>:pr:`13279`<< by
  Alexandre Gramfort

*   Fixed a bug in "linear_model.HuberRegressor" that was broken when
  "X" was of dtype bool. >>:pr:`13328`<< by Alexandre Gramfort.

*   Fixed a performance issue of "saga" and "sag" solvers when called
  in a "joblib.Parallel" setting with "n_jobs > 1" and

"X" was of dtype bool. >>:pr:`13328`<< by Alexandre Gramfort.

  "backend="threading"", causing them to perform worse than in the
  sequential case. >>:pr:`13389`<< by >>:user:`Pierre Glaser
  <pierreglaser>`<<.

*   Fixed a bug in
  "linear_model.stochastic_gradient.BaseSGDClassifier" that was not
  deterministic when trained in a multi-class setting on several
  threads. >>:pr:`13422`<< by >>:user:`Clément Doumouro <ClemDoum>`<<.

*   Fixed bug in "linear_model.ridge_regression()",
  "linear_model.Ridge" and "linear_model.RidgeClassifier" that caused
  unhandled exception for arguments "return_intercept=True" and
  "solver=auto" (default) or any other solver different from "sag".
  >>:pr:`13363`<< by >>:user:`Bartosz Telenczuk <btel>`<<

*   "linear_model.ridge_regression()" will now raise an exception if
  "return_intercept=True" and solver is different from "sag".
  Previously, only warning was issued. >>:pr:`13363`<< by
  >>:user:`Bartosz Telenczuk <btel>`<<

*   "linear_model.ridge_regression()" will choose "sparse_cg" solver
  for sparse inputs when "solver=auto" and "sample_weight" is provided
  (previously *cholesky* solver was selected). >>:pr:`13363`<< by

*    The use of "linear_model.lars_path" with "X=None" while passing
  "Gram" is deprecated in version 0.21 and will be removed in version

>>:user:`Bartosz Telenczuk <btel>`<<

  0.23. Use "linear_model.lars_path_gram" instead. >>:pr:`11699`<< by
  >>:user:`Kuai Yu <yukuairoy>`<<.

*   "linear_model.logistic_regression_path()" is deprecated in version
  0.21 and will be removed in version 0.23. >>:pr:`12821`<< by
  >>:user:`Nicolas Hug <NicolasHug>`<<.

*   "linear_model.RidgeCV" with leave-one-out cross-validation now
  correctly fits an intercept when "fit_intercept=True" and the design
  matrix is sparse. >>:issue:`13350`<< by >>:user:`Jérôme Dockès
  <jeromedockes>`<<

*   Make "manifold.tsne.trustworthiness()" use an inverted index
  instead of an *np.where* lookup to find the rank of neighbors in the
  input space. This improves efficiency in particular when computed
  with lots of neighbors and/or small datasets. >>:pr:`9907`<< by

*   Added the "metrics.max_error()" metric and a corresponding
  "'max_error'" scorer for single output regression. >>:pr:`12232`<<
  by >>:user:`Krishna Sangeeth <whiletruelearn>`<<.

*   Add "metrics.multilabel_confusion_matrix()", which calculates a
  confusion matrix with true positive, false positive, false negative
  and true negative counts for each class. This facilitates the
  calculation of set-wise metrics such as recall, specificity, fall

confusion matrix with true positive, false positive, false negative
  out and miss rate. >>:pr:`11179`<< by >>:user:`Shangwu Yao
  <ShangwuYao>`<< and Joel Nothman.

*   "metrics.jaccard_score()" has been added to calculate the Jaccard
  coefficient as an evaluation metric for binary, multilabel and
  multiclass tasks, with an interface analogous to
  "metrics.f1_score()". >>:pr:`13151`<< by >>:user:`Gaurav Dhingra
  <gxyd>`<< and Joel Nothman.

*   Added "metrics.pairwise.haversine_distances()" which can be
  accessed with *metric='pairwise'* through
  "metrics.pairwise_distances()" and estimators. (Haversine distance
  was previously available for nearest neighbors calculation.)
  >>:pr:`12568`<< by >>:user:`Wei Xue <xuewei4d>`<<, >>:user:`Emmanuel
  Arias <eamanu>`<< and Joel Nothman.

*   Faster "metrics.pairwise_distances()" with *n_jobs* > 1 by using a
  thread-based backend, instead of process-based backends.
  >>:pr:`8216`<< by >>:user:`Pierre Glaser <pierreglaser>`<< and
  >>:user:`Romuald Menuet <zanospi>`<<

*   The pairwise manhattan distances with sparse input now uses the
  BLAS shipped with scipy instead of the bundled BLAS. >>:pr:`12732`<<
  by >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<

*   Use label *accuracy* instead of *micro-average* on
  "metrics.classification_report()" to avoid confusion. *micro-

by >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<

  average* is only shown for multi-label or multi-class with a subset
  of classes because it is otherwise identical to accuracy.
  >>:pr:`12334`<< by >>:user:`Emmanuel Arias <eamanu@eamanu.com>`<<,
  Joel Nothman and Andreas Müller

*   Added *beta* parameter to
  "metrics.homogeneity_completeness_v_measure()" and
  "metrics.v_measure_score()" to configure the tradeoff between
  homogeneity and completeness. >>:pr:`13607`<< by >>:user:`Stephane
  Couvreur <scouvreur>`<< and and >>:user:`Ivan Sanchez <ivsanro1>`<<.

*   The metric "metrics.r2_score()" is degenerate with a single sample
  and now it returns NaN and raises
  "exceptions.UndefinedMetricWarning". >>:pr:`12855`<< by
  >>:user:`Pawel Sendyk <psendyk>`<<.

*   Fixed a bug where "metrics.brier_score_loss()" will sometimes
  return incorrect result when there's only one class in "y_true".
  >>:pr:`13628`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Fixed a bug in "metrics.label_ranking_average_precision_score()"
  where sample_weight wasn't taken into account for samples with
  degenerate labels. >>:pr:`13447`<< by >>:user:`Dan Ellis <dpwe>`<<.

*   The parameter "labels" in "metrics.hamming_loss()" is deprecated
  in version 0.21 and will be removed in version 0.23. >>:pr:`10580`<<
  by >>:user:`Reshama Shaikh <reshamas>`<< and >>:user:`Sandra
  Mitrovic <SandraMNE>`<<.

in version 0.21 and will be removed in version 0.23. >>:pr:`10580`<<

*   The function "metrics.pairwise.euclidean_distances()", and
  therefore several estimators with "metric='euclidean'", suffered
  from numerical precision issues with "float32" features. Precision
  has been increased at the cost of a small drop of performance.
  >>:pr:`13554`<< by >>:user:`Celelibi`<< and >>:user:`Jérémie du
  Boisberranger <jeremiedbb>`<<.

*   "metrics.jaccard_similarity_score()" is deprecated in favour of
  the more consistent "metrics.jaccard_score()". The former behavior
  for binary and multiclass targets is broken. >>:pr:`13151`<< by Joel
  Nothman.

"sklearn.mixture"

*   Fixed a bug in "mixture.BaseMixture" and therefore on estimators
  based on it, i.e. "mixture.GaussianMixture" and
  "mixture.BayesianGaussianMixture", where "fit_predict" and
  "fit.predict" were not equivalent. >>:pr:`13142`<< by

*   Classes "GridSearchCV" and "RandomizedSearchCV" now allow for
  refit=callable to add flexibility in identifying the best estimator.
  See *sphx_glr_auto_examples_model_selection_plot_grid_search_refit_
  callable.py*. >>:pr:`11354`<< by >>:user:`Wenhao Zhang
  <wenhaoz@ucla.edu>`<<, Joel Nothman and >>:user:`Adrin Jalali

callable.py*. >>:pr:`11354`<< by >>:user:`Wenhao Zhang

*   Classes "GridSearchCV", "RandomizedSearchCV", and methods
  "cross_val_score()", "cross_val_predict()", "cross_validate()", now
  print train scores when *return_train_scores* is True and *verbose*
  > 2. For "learning_curve()", and "validation_curve()" only the
  latter is required. >>:pr:`12613`<< and >>:pr:`12669`<< by
  >>:user:`Marc Torrellas <marctorrellas>`<<.

*   Some *CV splitter* classes and *model_selection.train_test_split*
  now raise "ValueError" when the resulting training set is empty.
  >>:pr:`12861`<< by >>:user:`Nicolas Hug <NicolasHug>`<<.

*   Fixed a bug where "model_selection.StratifiedKFold" shuffles each
  class's samples with the same "random_state", making "shuffle=True"
  ineffective. >>:pr:`13124`<< by >>:user:`Hanmin Qin

*   Added ability for "model_selection.cross_val_predict()" to handle
  multi-label (and multioutput-multiclass) targets with
  "predict_proba"-type methods. >>:pr:`8773`<< by >>:user:`Stephen
  Hoover <stephen-hoover>`<<.

*   Fixed an issue in "cross_val_predict()" where
  *method="predict_proba"* returned always *0.0* when one of the
  classes was excluded in a cross-validation fold. >>:pr:`13366`<< by
  >>:user:`Guillaume Fournier <gfournier>`<<

"sklearn.multiclass"

*   Fixed an issue in
  "multiclass.OneVsOneClassifier.decision_function()" where the

>>:user:`Guillaume Fournier <gfournier>`<<

  decision_function value of a given sample was different depending on
  whether the decision_function was evaluated on the sample alone or
  on a batch containing this same sample due to the scaling used in
  decision_function. >>:pr:`10440`<< by >>:user:`Jonathan Ohayon
  <Johayon>`<<.

*   Fixed a bug in "multioutput.MultiOutputClassifier" where the
  *predict_proba* method incorrectly checked for *predict_proba*
  attribute in the estimator object. >>:pr:`12222`<< by
  >>:user:`Rebekah Kim <rebekahkim>`<<

*   Added "neighbors.NeighborhoodComponentsAnalysis" for metric
  learning, which implements the Neighborhood Components Analysis
  algorithm.  >>:pr:`10058`<< by >>:user:`William de Vazelhes
  <wdevazelhes>`<< and >>:user:`John Chiotellis <johny-c>`<<.

*   Methods in "neighbors.NearestNeighbors" : "kneighbors()",
  "radius_neighbors()", "kneighbors_graph()",
  "radius_neighbors_graph()" now raise "NotFittedError", rather than
  "AttributeError", when called before "fit" >>:pr:`12279`<< by
  >>:user:`Krishna Sangeeth <whiletruelearn>`<<.

*   Fixed a bug in "neural_network.MLPClassifier" and
  "neural_network.MLPRegressor" where the option "shuffle=False" was

  being ignored. >>:pr:`12582`<< by >>:user:`Sam Waterbury
  <samwaterbury>`<<.

*   Fixed a bug in "neural_network.MLPClassifier" where validation
  sets for early stopping were not sampled with stratification. In the
  multilabel case however, splits are still not stratified.
  >>:pr:`13164`<< by >>:user:`Nicolas Hug<NicolasHug>`<<.

*   "pipeline.Pipeline" can now use indexing notation (e.g.
  "my_pipeline[0:-1]") to extract a subsequence of steps as another
  Pipeline instance.  A Pipeline can also be indexed directly to
  extract a particular step (e.g. "my_pipeline['svc']"), rather than
  accessing "named_steps". >>:pr:`2568`<< by Joel Nothman.

*   Added optional parameter "verbose" in "pipeline.Pipeline",
  "compose.ColumnTransformer" and "pipeline.FeatureUnion" and
  corresponding "make_" helpers for showing progress and timing of
  each step. >>:pr:`11364`<< by >>:user:`Baze Petrushev
  <petrushev>`<<, >>:user:`Karan Desai <karandesai-96>`<<, Joel
  Nothman, and >>:user:`Thomas Fan <thomasjpfan>`<<.

*   "pipeline.Pipeline" now supports using "'passthrough'" as a
  transformer, with the same effect as "None". >>:pr:`11144`<< by
  >>:user:`Thomas Fan <thomasjpfan>`<<.

*   "pipeline.Pipeline"  implements "__len__" and therefore
  "len(pipeline)" returns the number of steps in the pipeline.

>>:user:`Thomas Fan <thomasjpfan>`<<.

  >>:pr:`13439`<< by >>:user:`Lakshya KD <LakshKD>`<<.

*   "preprocessing.OneHotEncoder" now supports dropping one feature
  per category with a new drop parameter. >>:pr:`12908`<< by
  >>:user:`Drew Johnston <drewmjohnston>`<<.

*   "preprocessing.OneHotEncoder" and "preprocessing.OrdinalEncoder"
  now handle pandas DataFrames more efficiently. >>:pr:`13253`<< by
  >>:user:`maikia`<<.

*   Make "preprocessing.MultiLabelBinarizer" cache class mappings
  instead of calculating it every time on the fly. >>:pr:`12116`<< by
  >>:user:`Ekaterina Krivich <kiote>`<< and Joel Nothman.

*   "preprocessing.PolynomialFeatures" now supports compressed sparse
  row (CSR) matrices as input for degrees 2 and 3. This is typically
  much faster than the dense case as it scales with matrix density and
  expansion degree (on the order of density^degree), and is much, much
  faster than the compressed sparse column (CSC) case. >>:pr:`12197`<<
  by >>:user:`Andrew Nystrom <awnystrom>`<<.

*   Speed improvement in "preprocessing.PolynomialFeatures", in the
  dense case. Also added a new parameter "order" which controls output
  order for further speed performances. >>:pr:`12251`<< by Tom Dupre
  la Tour.

*   Fixed the calculation overflow when using a float16 dtype with

order for further speed performances. >>:pr:`12251`<< by Tom Dupre
  la Tour.

  "preprocessing.StandardScaler". >>:pr:`13007`<< by
  >>:user:`Raffaello Baluyot <baluyotraf>`<<

*   Fixed a bug in "preprocessing.QuantileTransformer" and
  "preprocessing.quantile_transform()" to force n_quantiles to be at
  most equal to n_samples. Values of n_quantiles larger than n_samples
  were either useless or resulting in a wrong approximation of the
  cumulative distribution function estimator. >>:pr:`13333`<< by
  >>:user:`Albert Thomas <albertcthomas>`<<.

*   The default value of *copy* in
  "preprocessing.quantile_transform()" will change from False to True
  in 0.23 in order to make it more consistent with the default *copy*
  values of other functions in "preprocessing" and prevent unexpected
  side effects by modifying the value of *X* inplace. >>:pr:`13459`<<
  by >>:user:`Hunter McGushion <HunterMcGushion>`<<.

*   Fixed an issue in "svm.SVC.decision_function()" when
  "decision_function_shape='ovr'". The decision_function value of a
  given sample was different depending on whether the
  decision_function was evaluated on the sample alone or on a batch
  containing this same sample due to the scaling used in

*   Decision Trees can now be plotted with matplotlib using

decision_function. >>:pr:`10440`<< by >>:user:`Jonathan Ohayon

  "tree.plot_tree()" without relying on the "dot" library, removing a
  hard-to-install dependency. >>:pr:`8508`<< by Andreas Müller.

*   Decision Trees can now be exported in a human readable textual
  format using "tree.export_text()". >>:pr:`6261`<< by *Giuseppe
  Vettigli <JustGlowing>*.

*   "get_n_leaves()" and "get_depth()" have been added to
  "tree.BaseDecisionTree" and consequently all estimators based on it,
  including "tree.DecisionTreeClassifier",
  "tree.DecisionTreeRegressor", "tree.ExtraTreeClassifier", and
  "tree.ExtraTreeRegressor". >>:pr:`12300`<< by >>:user:`Adrin Jalali

*   Trees and forests did not previously *predict* multi-output
  classification targets with string labels, despite accepting them in
  *fit*. >>:pr:`11458`<< by >>:user:`Mitar Milutinovic <mitar>`<<.

*   Fixed an issue with "tree.BaseDecisionTree" and consequently all
  estimators based on it, including "tree.DecisionTreeClassifier",
  "tree.ExtraTreeRegressor", where they used to exceed the given
  "max_depth" by 1 while expanding the tree if "max_leaf_nodes" and
  "max_depth" were both specified by the user. Please note that this
  also affects all ensemble methods using decision trees.
  >>:pr:`12344`<< by >>:user:`Adrin Jalali <adrinjalali>`<<.

"max_depth" were both specified by the user. Please note that this

*   "utils.resample()" now accepts a "stratify" parameter for sampling
  according to class distributions. >>:pr:`13549`<< by

*   Deprecated "warn_on_dtype" parameter from "utils.check_array()"
  and "utils.check_X_y()". Added explicit warning for dtype conversion
  in "check_pairwise_arrays()" if the "metric" being passed is a
  pairwise boolean metric. >>:pr:`13382`<< by >>:user:`Prathmesh
  Savale <praths007>`<<.

Multiple modules

*   The *__repr__()* method of all estimators (used when calling
  *print(estimator)*) has been entirely re-written, building on
  Python's pretty printing standard library. All parameters are
  printed by default, but this can be altered with the
  "print_changed_only" option in "sklearn.set_config()".
  >>:pr:`11705`<< by >>:user:`Nicolas Hug <NicolasHug>`<<.

*   Add estimators tags: these are annotations of estimators that
  allow programmatic inspection of their capabilities, such as sparse
  matrix support, supported output types and supported methods.
  Estimator tags also determine the tests that are run on an estimator
  when *check_estimator* is called. Read more in the User Guide.
  >>:pr:`8022`<< by >>:user:`Andreas Müller <amueller>`<<.

Estimator tags also determine the tests that are run on an estimator

*   Memory copies are avoided when casting arrays to a different dtype
  in multiple estimators. >>:pr:`11973`<< by >>:user:`Roman Yurchak
  <rth>`<<.

*   Fixed a bug in the implementation of the "our_rand_r()" helper
  function that was not behaving consistently across platforms.
  >>:pr:`13422`<< by >>:user:`Madhura Parikh <jdnc>`<< and
  >>:user:`Clément Doumouro <ClemDoum>`<<.

Miscellaneous

*   Joblib is no longer vendored in scikit-learn, and becomes a
  dependency. Minimal supported version is joblib 0.11, however using
  version >= 0.13 is strongly recommended. >>:pr:`13531`<< by
  >>:user:`Roman Yurchak <rth>`<<.

Changes to estimator checks

These changes mostly affect library developers.

* Add "check_fit_idempotent" to "check_estimator()", which checks that
  when *fit* is called twice with the same data, the output of
  *predict*, *predict_proba*, *transform*, and *decision_function*
  does not change. >>:pr:`12328`<< by >>:user:`Nicolas Hug
  <NicolasHug>`<<

* Many checks can now be disabled or configured with Estimator Tags.

>>:pr:`8022`<< by >>:user:`Andreas Müller <amueller>`<<.

improvement of the project since version 0.20, including:

adanhawth, Aditya Vyas, Adrin Jalali, Agamemnon Krasoulis, Albert
Thomas, Alberto Torres, Alexandre Gramfort, amourav, Andrea Navarrete,
Andreas Mueller, Andrew Nystrom, assiaben, Aurélien Bellet, Bartosz
Michałowski, Bartosz Telenczuk, bauks, BenjaStudio, bertrandhaut,
Bharat Raghunathan, brentfagan, Bryan Woods, Cat Chenal, Cheuk Ting
Ho, Chris Choe, Christos Aridas, Clément Doumouro, Cole Smith,
Connossor, Corey Levinson, Dan Ellis, Dan Stine, Danylo Baibak, daten-
kieker, Denis Kataev, Didi Bar-Zev, Dillon Gardner, Dmitry Mottl,
Dmitry Vukolov, Dougal J. Sutherland, Dowon, drewmjohnston, Dror
Atariah, Edward J Brown, Ekaterina Krivich, Elizabeth Sander, Emmanuel
Arias, Eric Chang, Eric Larson, Erich Schubert, esvhd, Falak, Feda
Curic, Federico Caselli, Frank Hoang, Fibinse Xavier`, Finn O'Shea,
Gabriel Marzinotto, Gabriel Vacaliuc, Gabriele Calvo, Gael Varoquaux,
GauravAhlawat, Giuseppe Vettigli, Greg Gandenberger, Guillaume
Fournier, Guillaume Lemaitre, Gustavo De Mari Pereira, Hanmin Qin,
haroldfox, hhu-luqi, Hunter McGushion, Ian Sanders, JackLangerman,
Jacopo Notarstefano, jakirkham, James Bourbeau, Jan Koch, Jan S,
janvanrijn, Jarrod Millman, jdethurens, jeremiedbb, JF, joaak, Joan

Massich, Joel Nothman, Jonathan Ohayon, Joris Van den Bossche,
josephsalmon, Jérémie Méhault, Katrin Leinweber, ken, kms15, Koen,
Kossori Aruku, Krishna Sangeeth, Kuai Yu, Kulbear, Kushal Chauhan,
Kyle Jackson, Lakshya KD, Leandro Hermida, Lee Yi Jie Joel, Lily
Xiong, Lisa Sarah Thomas, Loic Esteve, louib, luk-f-a, maikia, mail-
liam, Manimaran, Manuel López-Ibáñez, Marc Torrellas, Marco Gaido,
Marco Gorelli, MarcoGorelli, marineLM, Mark Hannel, Martin Gubri,
Masstran, mathurinm, Matthew Roeschke, Max Copeland, melsyt,
mferrari3, Mickaël Schoentgen, Ming Li, Mitar, Mohammad Aftab,
Mohammed AbdelAal, Mohammed Ibraheem, Muhammad Hassaan Rafique,
mwestt, Naoya Iijima, Nicholas Smith, Nicolas Goix, Nicolas Hug,
Nikolay Shebanov, Oleksandr Pavlyk, Oliver Rausch, Olivier Grisel,
Orestis, Osman, Owen Flanagan, Paul Paczuski, Pavel Soriano, pavlos
kallis, Pawel Sendyk, peay, Peter, Peter Cock, Peter Hausamann, Peter
Marko, Pierre Glaser, pierretallotte, Pim de Haan, Piotr Szymański,
Prabakaran Kumaresshan, Pradeep Reddy Raamana, Prathmesh Savale,
Pulkit Maloo, Quentin Batista, Radostin Stoyanov, Raf Baluyot, Rajdeep
Dua, Ramil Nugmanov, Raúl García Calvo, Rebekah Kim, Reshama Shaikh,
Rohan Lekhwani, Rohan Singh, Rohan Varma, Rohit Kapoor, Roman

Feldbauer, Roman Yurchak, Romuald M, Roopam Sharma, Ryan, Rüdiger
Busche, Sam Waterbury, Samuel O. Ronsin, SandroCasagrande, Scott Cole,
Scott Lowe, Sebastian Raschka, Shangwu Yao, Shivam Kotwalia, Shiyu
Duan, smarie, Sriharsha Hatwar, Stephen Hoover, Stephen Tierney,
Stéphane Couvreur, surgan12, SylvainLan, TakingItCasual, Tashay Green,
thibsej, Thomas Fan, Thomas J Fan, Thomas Moreau, Tom Dupré la Tour,
Tommy, Tulio Casagrande, Umar Farouk Umar, Utkarsh Upadhyay, Vinayak
Mehta, Vishaal Kapoor, Vivek Kumar, Vlad Niculae, vqean3, Wenhao
Zhang, William de Vazelhes, xhan, Xing Han Lu, xinyuliu12, Yaroslav
Halchenko, Zach Griffith, Zach Miller, Zayd Hammoudeh, Zhuyi Xue,
Zijie (ZJ) Poh, ^__^

Version 0.20.4

This is a bug-fix release with some bug fixes applied to version
0.20.3.

Changelog
=========

The bundled version of joblib was upgraded from 0.13.0 to 0.13.2.

  Nothman.

  ineffective. >>:issue:`13124`<< by >>:user:`Hanmin Qin

Version 0.20.3

**March 1, 2019**

This is a bug-fix release with some minor documentation improvements

>>:issue:`13772`<< by >>:user:`Aditya Vyas <aditya1702>`<<.

and enhancements to features released in 0.20.0.

Changelog
=========

*   Fixed a bug in "cluster.KMeans" where computation was single
  threaded when *n_jobs > 1* or *n_jobs = -1*. >>:issue:`12949`<< by
  >>:user:`Prabakaran Kumaresshan <nixphix>`<<.

*   Fixed a bug in "compose.ColumnTransformer" to handle negative
  indexes in the columns list of the transformers. >>:issue:`12946`<<
  by >>:user:`Pierre Tallotte <pierretallotte>`<<.

"sklearn.covariance"

*   Fixed a regression in "covariance.graphical_lasso()" so that the
  case *n_features=2* is handled correctly. >>:issue:`13276`<< by
  >>:user:`Aurélien Bellet <bellet>`<<.

*   Fixed a bug in "decomposition.sparse_encode()" where computation
  was single threaded when *n_jobs > 1* or *n_jobs = -1*.
  >>:issue:`13005`<< by >>:user:`Prabakaran Kumaresshan <nixphix>`<<.

*   "sklearn.datasets.fetch_openml()" now loads data by streaming,
  avoiding high memory usage.  >>:issue:`13312`<< by Joris Van den
  Bossche.

*   Fixed a bug in "feature_extraction.text.CountVectorizer" which
  would result in the sparse feature matrix having conflicting

Bossche.

  *indptr* and *indices* precisions under very large vocabularies.
  >>:issue:`11295`<< by >>:user:`Gabriel Vacaliuc <gvacaliuc>`<<.

*   add support for non-numeric data in
  "sklearn.impute.MissingIndicator" which was not supported while
  "sklearn.impute.SimpleImputer" was supporting this for some
  imputation strategies. >>:issue:`13046`<< by >>:user:`Guillaume

*   Fixed a bug in "linear_model.MultiTaskElasticNet" and
  "linear_model.MultiTaskLasso" which were breaking when "warm_start =
  True". >>:issue:`12360`<< by >>:user:`Aakanksha Joshi <joaak>`<<.

*   Fixed a bug in "preprocessing.KBinsDiscretizer" where
  "strategy='kmeans'" fails with an error during transformation due to
  unsorted bin edges. >>:issue:`13134`<< by >>:user:`Sandro Casagrande
  <SandroCasagrande>`<<.

*   Fixed a bug in "preprocessing.OneHotEncoder" where the deprecation
  of "categorical_features" was handled incorrectly in combination
  with "handle_unknown='ignore'". >>:issue:`12881`<< by Joris Van den
  Bossche.

*   Bins whose width are too small (i.e., <= 1e-8) are removed with a
  warning in "preprocessing.KBinsDiscretizer". >>:issue:`13165`<< by

Bossche.

*   Fixed a bug in "svm.SVC", "svm.NuSVC", "svm.SVR", "svm.NuSVR" and
  "svm.OneClassSVM" where the "scale" option of parameter "gamma" is
  erroneously defined as "1 / (n_features * X.std())". It's now
  defined as "1 / (n_features * X.var())". >>:issue:`13221`<< by

With thanks to:

Adrin Jalali, Agamemnon Krasoulis, Albert Thomas, Andreas Mueller,
Aurélien Bellet, bertrandhaut, Bharat Raghunathan, Dowon, Emmanuel
Arias, Fibinse Xavier, Finn O'Shea, Gabriel Vacaliuc, Gael Varoquaux,
Guillaume Lemaitre, Hanmin Qin, joaak, Joel Nothman, Joris Van den
Bossche, Jérémie Méhault, kms15, Kossori Aruku, Lakshya KD, maikia,
Manuel López-Ibáñez, Marco Gorelli, MarcoGorelli, mferrari3, Mickaël
Schoentgen, Nicolas Hug, pavlos kallis, Pierre Glaser, pierretallotte,
Prabakaran Kumaresshan, Reshama Shaikh, Rohit Kapoor, Roman Yurchak,
SandroCasagrande, Tashay Green, Thomas Fan, Vishaal Kapoor, Zhuyi Xue,
Zijie (ZJ) Poh

Version 0.20.2

**December 20, 2018**

* "sklearn.neighbors" when "metric=='jaccard'" (bug fix)

* use of "'seuclidean'" or "'mahalanobis'" metrics in some cases (bug
  fix)

Changelog
=========

*   Fixed an issue in "compose.make_column_transformer()" which raises
  unexpected error when columns is pandas Index or pandas Series.
  >>:issue:`12704`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Fixed a bug in "metrics.pairwise_distances()" and
  "metrics.pairwise_distances_chunked()" where parameters "V" of
  ""seuclidean"" and "VI" of ""mahalanobis"" metrics were computed
  after the data was split into chunks instead of being pre-computed
  on whole data. >>:issue:`12701`<< by >>:user:`Jeremie du

*   Fixed "sklearn.neighbors.DistanceMetric" jaccard distance function
  to return 0 when two all-zero vectors are compared.
  >>:issue:`12685`<< by >>:user:`Thomas Fan <thomasjpfan>`<<.

*   Calling "utils.check_array()" on *pandas.Series* with categorical
  data, which raised an error in 0.20.0, now returns the expected

  output again. >>:issue:`12699`<< by Joris Van den Bossche.

adanhawth, Adrin Jalali, Albert Thomas, Andreas Mueller, Dan Stine,
Feda Curic, Hanmin Qin, Jan S, jeremiedbb, Joel Nothman, Joris Van den
Bossche, josephsalmon, Katrin Leinweber, Loic Esteve, Muhammad Hassaan
Rafique, Nicolas Hug, Olivier Grisel, Paul Paczuski, Reshama Shaikh,
Sam Waterbury, Shivam Kotwalia, Thomas Fan

Version 0.20.1

**November 21, 2018**

and enhancements to features released in 0.20.0. Note that we also
include some API changes in this release, so you might get some extra
warnings after updating from 0.20.0 to 0.20.1.

* "decomposition.IncrementalPCA" (bug fix)

Changelog
=========

*   make "cluster.MeanShift" no longer try to do nested parallelism as
  the overhead would hurt performance significantly when "n_jobs > 1".

Changelog
=========

  >>:issue:`12159`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   Fixed a bug in "cluster.DBSCAN" with precomputed sparse neighbors
  graph, which would add explicitly zeros on the diagonal even when
  already present. >>:issue:`12105`<< by Tom Dupre la Tour.

*   Fixed an issue in "compose.ColumnTransformer" when stacking
  columns with types not convertible to a numeric. >>:issue:`11912`<<
  by >>:user:`Adrin Jalali <adrinjalali>`<<.

*   "compose.ColumnTransformer" now applies the "sparse_threshold"
  even if all transformation results are sparse. >>:issue:`12304`<< by
  Andreas Müller.

*   "compose.make_column_transformer()" now expects "(transformer,
  columns)" instead of "(columns, transformer)" to keep consistent
  with "compose.ColumnTransformer". >>:issue:`12339`<< by
  >>:user:`Adrin Jalali <adrinjalali>`<<.

*   "datasets.fetch_openml()" to correctly use the local cache.
  >>:issue:`12246`<< by >>:user:`Jan N. van Rijn <janvanrijn>`<<.

*   "datasets.fetch_openml()" to correctly handle ignore attributes
  and row id attributes. >>:issue:`12330`<< by >>:user:`Jan N. van
  Rijn <janvanrijn>`<<.

*   Fixed integer overflow in "datasets.make_classification()" for
  values of "n_informative" parameter larger than 64.

Rijn <janvanrijn>`<<.

  >>:issue:`10811`<< by >>:user:`Roman Feldbauer <VarIr>`<<.

*   Fixed olivetti faces dataset "DESCR" attribute to point to the
  right location in "datasets.fetch_olivetti_faces()".
  >>:issue:`12441`<< by >>:user:`Jérémie du Boisberranger
  <jeremiedbb>`<<

*   "datasets.fetch_openml()" to retry downloading when reading from
  local cache fails. >>:issue:`12517`<< by >>:user:`Thomas Fan

*   Fixed a regression in "decomposition.IncrementalPCA" where 0.20.0
  raised an error if the number of samples in the final batch for
  fitting IncrementalPCA was smaller than n_components.
  >>:issue:`12234`<< by >>:user:`Ming Li <minggli>`<<.

*   Fixed a bug mostly affecting "ensemble.RandomForestClassifier"
  where "class_weight='balanced_subsample'" failed with more than 32
  classes. >>:issue:`12165`<< by Joel Nothman.

*   Fixed a bug affecting "ensemble.BaggingClassifier",
  "ensemble.BaggingRegressor" and "ensemble.IsolationForest", where
  "max_features" was sometimes rounded down to zero.
  >>:issue:`12388`<< by >>:user:`Connor Tann <Connossor>`<<.

*   Fixed a regression in v0.20.0 where
  "feature_extraction.text.CountVectorizer()" and other text

  vectorizers could error during stop words validation with custom
  preprocessors or tokenizers. >>:issue:`12393`<< by Roman Yurchak.

*   "linear_model.SGDClassifier" and variants with
  "early_stopping=True" would not use a consistent validation split in
  the multiclass case and this would cause a crash when using those
  estimators as part of parallel parameter search or cross-validation.
  >>:issue:`12122`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   Fixed a bug affecting "SGDClassifier" in the multiclass case. Each
  one-versus-all step is run in a "joblib.Parallel" call and mutating
  a common parameter, causing a segmentation fault if called within a
  backend using processes and not threads. We now use
  "require=sharedmem" at the "joblib.Parallel" instance creation.
  >>:issue:`12518`<< by >>:user:`Pierre Glaser <pierreglaser>`<< and

*   Fixed a bug in "metrics.pairwise.pairwise_distances_argmin_min()"
  which returned the square root of the distance when the metric
  parameter was set to "euclidean". >>:issue:`12481`<< by

*   Fixed a bug in "metrics.pairwise.pairwise_distances_chunked()"
  which didn't ensure the diagonal is zero for euclidean distances.

>>:user:`Jérémie du Boisberranger <jeremiedbb>`<<.

  >>:issue:`12612`<< by >>:user:`Andreas Müller <amueller>`<<.

*   The "metrics.calinski_harabaz_score()" has been renamed to
  "metrics.calinski_harabasz_score()" and will be removed in version
  0.23. >>:issue:`12211`<< by >>:user:`Lisa Thomas <LisaThomas9>`<<,
  >>:user:`Mark Hannel <markhannel>`<< and >>:user:`Melissa Ferrari
  <mferrari3>`<<.

*   Ensure that the "fit_predict" method of "mixture.GaussianMixture"
  and "mixture.BayesianGaussianMixture" always yield assignments
  consistent with "fit" followed by "predict" even if the convergence
  criterion is too loose or not met. >>:issue:`12451`<< by

*   force the parallelism backend to "threading" for
  "neighbors.KDTree" and "neighbors.BallTree" in Python 2.7 to avoid
  pickling errors caused by the serialization of their methods.
  >>:issue:`12171`<< by >>:user:`Thomas Moreau <tomMoral>`<<.

*   Fixed bug in "preprocessing.OrdinalEncoder" when passing manually
  specified categories. >>:issue:`12365`<< by Joris Van den Bossche.

*   Fixed bug in "preprocessing.KBinsDiscretizer" where the
  "transform" method mutates the "_encoder" attribute. The "transform"

specified categories. >>:issue:`12365`<< by Joris Van den Bossche.

  method is now thread safe. >>:issue:`12514`<< by >>:user:`Hanmin Qin

*   Fixed a bug in "preprocessing.PowerTransformer" where the Yeo-
  Johnson transform was incorrect for lambda parameters outside of
  *[0, 2]* >>:issue:`12522`<< by >>:user:`Nicolas Hug<NicolasHug>`<<.

*   Fixed a bug in "preprocessing.OneHotEncoder" where transform
  failed when set to ignore unknown numpy strings of different lengths
  >>:issue:`12471`<< by >>:user:`Gabriel Marzinotto<GMarzinotto>`<<.

*   The default value of the "method" argument in
  "preprocessing.power_transform()" will be changed from "box-cox" to
  "yeo-johnson" to match "preprocessing.PowerTransformer" in version
  0.23. A FutureWarning is raised when the default value is used.
  >>:issue:`12317`<< by >>:user:`Eric Chang <chang>`<<.

*   Use float64 for mean accumulator to avoid floating point precision
  issues in "preprocessing.StandardScaler" and
  "decomposition.IncrementalPCA" when using float32 datasets.
  >>:issue:`12338`<< by >>:user:`bauks <bauks>`<<.

*   Calling "utils.check_array()" on *pandas.Series*, which raised an
  error in 0.20.0, now returns the expected output again.
  >>:issue:`12625`<< by Andreas Müller

*   When using site joblib by setting the environment variable
  *SKLEARN_SITE_JOBLIB*, added compatibility with joblib 0.11 in
  addition to 0.12+. >>:issue:`12350`<< by Joel Nothman and Roman
  Yurchak.

*   Make sure to avoid raising "FutureWarning" when calling
  "np.vstack" with numpy 1.16 and later (use list comprehensions
  instead of generator expressions in many locations of the scikit-
  learn code base). >>:issue:`12467`<< by >>:user:`Olivier Grisel
  <ogrisel>`<<.

*   Removed all mentions of "sklearn.externals.joblib", and deprecated
  joblib methods exposed in "sklearn.utils", except for
  "utils.parallel_backend()" and "utils.register_parallel_backend()",
  which allow users to configure parallel computation in scikit-learn.
  Other functionalities are part of joblib. package and should be used
  directly, by installing it. The goal of this change is to prepare
  for unvendoring joblib in future version of scikit-learn.
  >>:issue:`12345`<< by >>:user:`Thomas Moreau <tomMoral>`<<

^__^, Adrin Jalali, Andrea Navarrete, Andreas Mueller, bauks,
BenjaStudio, Cheuk Ting Ho, Connossor, Corey Levinson, Dan Stine,
daten-kieker, Denis Kataev, Dillon Gardner, Dmitry Vukolov, Dougal J.

Sutherland, Edward J Brown, Eric Chang, Federico Caselli, Gabriel
Marzinotto, Gael Varoquaux, GauravAhlawat, Gustavo De Mari Pereira,
Hanmin Qin, haroldfox, JackLangerman, Jacopo Notarstefano, janvanrijn,
jdethurens, jeremiedbb, Joel Nothman, Joris Van den Bossche, Koen,
Kushal Chauhan, Lee Yi Jie Joel, Lily Xiong, mail-liam, Mark Hannel,
melsyt, Ming Li, Nicholas Smith, Nicolas Hug, Nikolay Shebanov,
Oleksandr Pavlyk, Olivier Grisel, Peter Hausamann, Pierre Glaser,
Pulkit Maloo, Quentin Batista, Radostin Stoyanov, Ramil Nugmanov,
Rebekah Kim, Reshama Shaikh, Rohan Singh, Roman Feldbauer, Roman
Yurchak, Roopam Sharma, Sam Waterbury, Scott Lowe, Sebastian Raschka,
Stephen Tierney, SylvainLan, TakingItCasual, Thomas Fan, Thomas
Moreau, Tom Dupré la Tour, Tulio Casagrande, Utkarsh Upadhyay, Xing
Han Lu, Yaroslav Halchenko, Zach Miller

Version 0.20.0

**September 25, 2018**

This release packs in a mountain of bug fixes, features and
enhancements for the Scikit-learn library, and improvements to the
documentation and examples. Thanks to our contributors!

This release is dedicated to the memory of Raghav Rajagopalan.

Warning:

  Version 0.20 is the last version of scikit-learn to support Python
  2.7 and Python 3.4. Scikit-learn 0.21 will require Python 3.5 or
  higher.

Highlights

Warning:

  higher.

We have tried to improve our support for common data-science use-cases
including missing values, categorical variables, heterogeneous data,
and features/targets with unusual distributions. Missing values in
features, represented by NaNs, are now accepted in column-wise
preprocessing such as scalers. Each feature is fitted disregarding
NaNs, and data containing NaNs can be transformed. The new "impute"
module provides estimators for learning despite missing data.

"ColumnTransformer" handles the case where different features or
columns of a pandas.DataFrame need different preprocessing. String or
pandas Categorical columns can now be encoded with "OneHotEncoder" or
"OrdinalEncoder".

"TransformedTargetRegressor" helps when the regression target needs to
be transformed to be modeled. "PowerTransformer" and
"KBinsDiscretizer" join "QuantileTransformer" as non-linear
transformations.

Beyond this, we have added *sample_weight* support to several
estimators (including "KMeans", "BayesianRidge" and "KernelDensity")
and improved stopping criteria in others (including "MLPRegressor",
"GradientBoostingRegressor" and "SGDRegressor").

This release is also the first to be accompanied by a Glossary of
Common Terms and API Elements developed by Joel Nothman. The glossary
is a reference resource to help users and contributors become familiar

with the terminology and conventions used in Scikit-learn.

Sorry if your contribution didn't make it into the highlights. There's
a lot here...

* "cluster.MeanShift" (bug fix)

* "decomposition.IncrementalPCA" in Python 2 (bug fix)

* "decomposition.SparsePCA" (bug fix)

* "ensemble.GradientBoostingClassifier" (bug fix affecting feature
  importances)

* "isotonic.IsotonicRegression" (bug fix)

* "linear_model.ARDRegression" (bug fix)

* "linear_model.LogisticRegressionCV" (bug fix)

* "linear_model.OrthogonalMatchingPursuit" (bug fix)

* "linear_model.PassiveAggressiveClassifier" (bug fix)

* "linear_model.PassiveAggressiveRegressor" (bug fix)

* "linear_model.Perceptron" (bug fix)

* "linear_model.SGDClassifier" (bug fix)

* "linear_model.SGDRegressor" (bug fix)

* "metrics.roc_auc_score" (bug fix)

* "metrics.roc_curve" (bug fix)

* "neural_network.BaseMultilayerPerceptron" (bug fix)

* "neural_network.MLPClassifier" (bug fix)

* "neural_network.MLPRegressor" (bug fix)

* The v0.19.0 release notes failed to mention a backwards
  incompatibility with "model_selection.StratifiedKFold" when

  "shuffle=True" due to >>:issue:`7823`<<.

* >>:issue:`11924`<<: "linear_model.LogisticRegressionCV" with
  *solver='lbfgs'* and *multi_class='multinomial'* may be non-
  deterministic or otherwise broken on macOS. This appears to be the
  case on Travis CI servers, but has not been confirmed on personal
  MacBooks! This issue has been present in previous releases.

* >>:issue:`9354`<<: "metrics.pairwise.euclidean_distances()" (which
  is used several times throughout the library) gives results with
  poor precision, which particularly affects its use with 32-bit float
  inputs. This became more problematic in versions 0.18 and 0.19 when
  some algorithms were changed to avoid casting 32-bit data into
  64-bit.

Changelog
=========

Support for Python 3.3 has been officially dropped.

*   "cluster.AgglomerativeClustering" now supports Single Linkage
  clustering via "linkage='single'". >>:issue:`9372`<< by
  >>:user:`Leland McInnes <lmcinnes>`<< and >>:user:`Steve Astels
  <sastels>`<<.

*   "cluster.KMeans" and "cluster.MiniBatchKMeans" now support sample
  weights via new parameter "sample_weight" in "fit" function.

<sastels>`<<.

  >>:issue:`10933`<< by >>:user:`Johannes Hansen <jnhansen>`<<.

*   "cluster.KMeans", "cluster.MiniBatchKMeans" and
  "cluster.k_means()" passed with "algorithm='full'" now enforces row-
  major ordering, improving runtime. >>:issue:`10471`<< by
  >>:user:`Gaurav Dhingra <gxyd>`<<.

*   "cluster.DBSCAN" now is parallelized according to "n_jobs"
  regardless of "algorithm". >>:issue:`8003`<< by >>:user:`Joël
  Billaud <recamshak>`<<.

*   "cluster.KMeans" now gives a warning if the number of distinct
  clusters found is smaller than "n_clusters". This may occur when the
  number of distinct points in the data set is actually smaller than
  the number of cluster one is looking for. >>:issue:`10059`<< by
  >>:user:`Christian Braune <christianbraune79>`<<.

*   Fixed a bug where the "fit" method of
  "cluster.AffinityPropagation" stored cluster centers as 3d array
  instead of 2d array in case of non-convergence. For the same class,
  fixed undefined and arbitrary behavior in case of training data
  where all samples had equal similarity. >>:issue:`9612`<<. By
  >>:user:`Jonatan Samoocha <jsamoocha>`<<.

*   Fixed a bug in "cluster.spectral_clustering()" where the
  normalization of the spectrum was using a division instead of a
  multiplication. >>:issue:`8129`<< by >>:user:`Jan Margeta
  <jmargeta>`<<, >>:user:`Guillaume Lemaitre <glemaitre>`<<, and

normalization of the spectrum was using a division instead of a
  >>:user:`Devansh D. <devanshdalal>`<<.

*   Fixed a bug in "cluster.k_means_elkan()" where the returned
  "iteration" was 1 less than the correct value. Also added the
  missing "n_iter_" attribute in the docstring of "cluster.KMeans".
  >>:issue:`11353`<< by >>:user:`Jeremie du Boisberranger

*   Fixed a bug in "cluster.mean_shift()" where the assigned labels
  were not deterministic if there were multiple clusters with the same
  intensities. >>:issue:`11901`<< by >>:user:`Adrin Jalali

*   Deprecate "pooling_func" unused parameter in
  "cluster.AgglomerativeClustering". >>:issue:`9875`<< by
  >>:user:`Kumar Ashutosh <thechargedneutron>`<<.

* New module.

*   Added "compose.ColumnTransformer", which allows to apply different
  transformers to different columns of arrays or pandas DataFrames.
  >>:issue:`9012`<< by Andreas Müller and Joris Van den Bossche, and
  >>:issue:`11315`<< by >>:user:`Thomas Fan <thomasjpfan>`<<.

*   Added the "compose.TransformedTargetRegressor" which transforms
  the target y before fitting a regression model. The predictions are
  mapped back to the original space via an inverse transform.
  >>:issue:`9041`<< by Andreas Müller and >>:user:`Guillaume Lemaitre

mapped back to the original space via an inverse transform.

*   Runtime improvements to "covariance.GraphicalLasso".
  >>:issue:`9858`<< by >>:user:`Steven Brown <stevendbrown>`<<.

*   The "covariance.graph_lasso()", "covariance.GraphLasso" and
  "covariance.GraphLassoCV" have been renamed to
  "covariance.graphical_lasso()", "covariance.GraphicalLasso" and
  "covariance.GraphicalLassoCV" respectively and will be removed in
  version 0.22. >>:issue:`9993`<< by >>:user:`Artiem Krinitsyn
  <artiemq>`<<

*   Added "datasets.fetch_openml()" to fetch datasets from OpenML.
  OpenML is a free, open data sharing platform and will be used
  instead of mldata as it provides better service availability.
  >>:issue:`9908`<< by Andreas Müller and >>:user:`Jan N. van Rijn
  <janvanrijn>`<<.

*   In "datasets.make_blobs()", one can now pass a list to the
  "n_samples" parameter to indicate the number of samples to generate
  per cluster. >>:issue:`8617`<< by >>:user:`Maskani Filali Mohamed
  <maskani-moh>`<< and >>:user:`Konstantinos Katrioplas <kkatrio>`<<.

*   Add "filename" attribute to "datasets" that have a CSV file.
  >>:issue:`9101`<< by >>:user:`alex-33 <alex-33>`<< and
  >>:user:`Maskani Filali Mohamed <maskani-moh>`<<.

*   "return_X_y" parameter has been added to several dataset loaders.

>>:issue:`9101`<< by >>:user:`alex-33 <alex-33>`<< and

  >>:issue:`10774`<< by >>:user:`Chris Catalfo <ccatalfo>`<<.

*   Fixed a bug in "datasets.load_boston()" which had a wrong data
  point. >>:issue:`10795`<< by >>:user:`Takeshi Yoshizawa
  <tarcusx>`<<.

*   Fixed a bug in "datasets.load_iris()" which had two wrong data
  points. >>:issue:`11082`<< by >>:user:`Sadhana Srinivasan
  <rotuna>`<< and >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Fixed a bug in "datasets.fetch_kddcup99()", where data were not
  properly shuffled. >>:issue:`9731`<< by Nicolas Goix.

*   Fixed a bug in "datasets.make_circles()", where no odd number of
  data points could be generated. >>:issue:`10045`<< by

*   Deprecated "sklearn.datasets.fetch_mldata()" to be removed in
  version 0.22. mldata.org is no longer operational. Until removal it
  will remain possible to load cached datasets. >>:issue:`11466`<< by
  Joel Nothman.

*   "decomposition.dict_learning()" functions and models now support
  positivity constraints. This applies to the dictionary and sparse
  code. >>:issue:`6374`<< by >>:user:`John Kirkham <jakirkham>`<<.

*     "decomposition.SparsePCA" now exposes "normalize_components".
  When set to True, the train and test data are centered with the

code. >>:issue:`6374`<< by >>:user:`John Kirkham <jakirkham>`<<.

  train mean respectively during the fit phase and the transform
  phase. This fixes the behavior of SparsePCA. When set to False,
  which is the default, the previous abnormal behaviour still holds.
  The False value is for backward compatibility and should not be
  used. >>:issue:`11585`<< by >>:user:`Ivan Panico <FollowKenny>`<<.

*   Efficiency improvements in "decomposition.dict_learning()".
  >>:issue:`11420`<< and others by >>:user:`John Kirkham
  <jakirkham>`<<.

*   Fix for uninformative error in "decomposition.IncrementalPCA": now
  an error is raised if the number of components is larger than the
  chosen batch size. The "n_components=None" case was adapted
  accordingly. >>:issue:`6452`<<. By >>:user:`Wally Gauze
  <wallygauze>`<<.

*   Fixed a bug where the "partial_fit" method of
  "decomposition.IncrementalPCA" used integer division instead of
  float division on Python 2. >>:issue:`9492`<< by >>:user:`James
  Bourbeau <jrbourbeau>`<<.

*   In "decomposition.PCA" selecting a n_components parameter greater
  than the number of samples now raises an error. Similarly, the
  "n_components=None" case now selects the minimum of "n_samples" and
  "n_features". >>:issue:`8484`<< by >>:user:`Wally Gauze

*   Fixed a bug in "decomposition.PCA" where users will get unexpected

"n_features". >>:issue:`8484`<< by >>:user:`Wally Gauze

  error with large datasets when "n_components='mle'" on Python 3
  versions. >>:issue:`9886`<< by >>:user:`Hanmin Qin

*   Fixed an underflow in calculating KL-divergence for
  "decomposition.NMF" >>:issue:`10142`<< by Tom Dupre la Tour.

*   Fixed a bug in "decomposition.SparseCoder" when running OMP sparse
  coding in parallel using read-only memory mapped datastructures.
  >>:issue:`5956`<< by >>:user:`Vighnesh Birodkar
  <vighneshbirodkar>`<< and >>:user:`Olivier Grisel <ogrisel>`<<.

*   Memory usage improvement for "_class_means()" and "_class_cov()"
  in "discriminant_analysis". >>:issue:`10898`<< by >>:user:`Nanxin
  Chen <bobchennan>`<<.

*   "dummy.DummyRegressor" now has a "return_std" option in its
  "predict" method. The returned standard deviations will be zeros.

*   "dummy.DummyClassifier" and "dummy.DummyRegressor" now only
  require X to be an object with finite length or shape.
  >>:issue:`9832`<< by >>:user:`Vrishank Bhardwaj <vrishank97>`<<.

*   "dummy.DummyClassifier" and "dummy.DummyRegressor" can now be
  scored without supplying test samples. >>:issue:`11951`<< by
  >>:user:`Rüdiger Busche <JarnoRFB>`<<.

*   "ensemble.BaggingRegressor" and "ensemble.BaggingClassifier" can

>>:user:`Rüdiger Busche <JarnoRFB>`<<.

  now be fit with missing/non-finite values in X and/or multi-output Y
  to support wrapping pipelines that perform their own imputation.
  >>:issue:`9707`<< by >>:user:`Jimmy Wan <jimmywan>`<<.

*   "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor" now support early stopping via
  "n_iter_no_change", "validation_fraction" and "tol".
  >>:issue:`7071`<< by Raghav RV

*   Added "named_estimators_" parameter in "ensemble.VotingClassifier"
  to access fitted estimators. >>:issue:`9157`<< by
  >>:user:`Herilalaina Rakotoarison <herilalaina>`<<.

*   Fixed a bug when fitting "ensemble.GradientBoostingClassifier" or
  "ensemble.GradientBoostingRegressor" with "warm_start=True" which
  previously raised a segmentation fault due to a non-conversion of
  CSC matrix into CSR format expected by "decision_function".
  Similarly, Fortran-ordered arrays are converted to C-ordered arrays
  in the dense case. >>:issue:`9991`<< by >>:user:`Guillaume Lemaitre

*   Fixed a bug in "ensemble.GradientBoostingRegressor" and
  "ensemble.GradientBoostingClassifier" to have feature importances
  summed and then normalized, rather than normalizing on a per-tree
  basis. The previous behavior over-weighted the Gini importance of
  features that appear in later stages. This issue only affected

summed and then normalized, rather than normalizing on a per-tree
  feature importances. >>:issue:`11176`<< by >>:user:`Gil Forsyth
  <gforsyth>`<<.

*   The default value of the "n_estimators" parameter of
  "ensemble.RandomForestClassifier", "ensemble.RandomForestRegressor",
  "ensemble.ExtraTreesClassifier", "ensemble.ExtraTreesRegressor", and
  "ensemble.RandomTreesEmbedding" will change from 10 in version 0.20
  to 100 in 0.22. A FutureWarning is raised when the default value is
  used. >>:issue:`11542`<< by >>:user:`Anna Ayzenshtat
  <annaayzenshtat>`<<.

*   Classes derived from "ensemble.BaseBagging". The attribute
  "estimators_samples_" will return a list of arrays containing the
  indices selected for each bootstrap instead of a list of arrays
  containing the mask of the samples selected for each bootstrap.
  Indices allows to repeat samples while mask does not allow this
  functionality. >>:issue:`9524`<< by >>:user:`Guillaume Lemaitre

*   "ensemble.BaseBagging" where one could not deterministically
  reproduce "fit" result using the object attributes when
  "random_state" is set. >>:issue:`9723`<< by >>:user:`Guillaume

*   Enable the call to *get_feature_names* in unfitted
  "feature_extraction.text.CountVectorizer" initialized with a

Lemaitre <glemaitre>`<<.

  vocabulary. >>:issue:`10908`<< by >>:user:`Mohamed Maskani <maskani-
  moh>`<<.

*   "idf_" can now be set on a
  "feature_extraction.text.TfidfTransformer". >>:issue:`10899`<< by
  >>:user:`Sergey Melderis <serega>`<<.

*   Fixed a bug in "feature_extraction.image.extract_patches_2d()"
  which would throw an exception if "max_patches" was greater than or
  equal to the number of all possible patches rather than simply
  returning the number of possible patches. >>:issue:`10101`<< by
  >>:user:`Varun Agrawal <varunagrawal>`<<

*   Fixed a bug in "feature_extraction.text.CountVectorizer",
  "feature_extraction.text.TfidfVectorizer",
  "feature_extraction.text.HashingVectorizer" to support 64 bit sparse
  array indexing necessary to process large datasets with more than
  2·10⁹ tokens (words or n-grams). >>:issue:`9147`<< by >>:user
  :`Claes-Fredrik Mannby <mannby>`<< and Roman Yurchak.

*   Fixed bug in "feature_extraction.text.TfidfVectorizer" which was
  ignoring the parameter "dtype". In addition,
  "feature_extraction.text.TfidfTransformer" will preserve "dtype" for
  floating and raise a warning if "dtype" requested is integer.
  >>:issue:`10441`<< by >>:user:`Mayur Kulkarni <maykulkarni>`<< and

>>:issue:`10441`<< by >>:user:`Mayur Kulkarni <maykulkarni>`<< and

*   Added select K best features functionality to
  "feature_selection.SelectFromModel". >>:issue:`6689`<< by
  >>:user:`Nihar Sheth <nsheth12>`<< and >>:user:`Quazi Rahman
  <qmaruf>`<<.

*   Added "min_features_to_select" parameter to
  "feature_selection.RFECV" to bound evaluated features counts.
  >>:issue:`11293`<< by >>:user:`Brent Yi <brentyi>`<<.

*   "feature_selection.RFECV"'s fit method now supports *groups*.
  >>:issue:`9656`<< by >>:user:`Adam Greenhall <adamgreenhall>`<<.

*   Fixed computation of "n_features_to_compute" for edge case with
  tied CV scores in "feature_selection.RFECV". >>:issue:`9222`<< by
  >>:user:`Nick Hoh <nickypie>`<<.

*   In "gaussian_process.GaussianProcessRegressor", method "predict"
  is faster when using "return_std=True" in particular more when
  called several times in a row. >>:issue:`9234`<< by
  >>:user:`andrewww <andrewww>`<< and >>:user:`Minghui Liu <minghui-
  liu>`<<.

* New module, adopting "preprocessing.Imputer" as
  "impute.SimpleImputer" with minor changes (see under preprocessing
  below).

*   Added "impute.MissingIndicator" which generates a binary indicator
  for missing values. >>:issue:`8075`<< by >>:user:`Maniteja Nandana
  <maniteja123>`<< and >>:user:`Guillaume Lemaitre <glemaitre>`<<.

for missing values. >>:issue:`8075`<< by >>:user:`Maniteja Nandana

*   The "impute.SimpleImputer" has a new strategy, "'constant'", to
  complete missing values with a fixed one, given by the "fill_value"
  parameter. This strategy supports numeric and non-numeric data, and
  so does the "'most_frequent'" strategy now. >>:issue:`11211`<< by
  >>:user:`Jeremie du Boisberranger <jeremiedbb>`<<.

*   Fixed a bug in "isotonic.IsotonicRegression" which incorrectly
  combined weights when fitting a model to data involving points with
  identical X values. >>:issue:`9484`<< by >>:user:`Dallas Card
  <dallascard>`<<

*   "linear_model.SGDClassifier", "linear_model.SGDRegressor",
  "linear_model.PassiveAggressiveClassifier",
  "linear_model.PassiveAggressiveRegressor" and
  "linear_model.Perceptron" now expose "early_stopping",
  "validation_fraction" and "n_iter_no_change" parameters, to stop
  optimization monitoring the score on a validation set. A new
  learning rate ""adaptive"" strategy divides the learning rate by 5
  each time "n_iter_no_change" consecutive epochs fail to improve the
  model. >>:issue:`9043`<< by Tom Dupre la Tour.

*   Add *sample_weight* parameter to the fit method of
  "linear_model.BayesianRidge" for weighted linear regression.
  >>:issue:`10112`<< by >>:user:`Peter St. John <pstjohn>`<<.

*   Fixed a bug in "logistic.logistic_regression_path()" to ensure
  that the returned coefficients are correct when
  "multiclass='multinomial'". Previously, some of the coefficients
  would override each other, leading to incorrect results in
  "linear_model.LogisticRegressionCV". >>:issue:`11724`<< by

*   Fixed a bug in "linear_model.LogisticRegression" where when using
  the parameter "multi_class='multinomial'", the "predict_proba"
  method was returning incorrect probabilities in the case of binary
  outcomes. >>:issue:`9939`<< by >>:user:`Roger Westover <rwolst>`<<.

*   Fixed a bug in "linear_model.LogisticRegressionCV" where the
  "score" method always computes accuracy, not the metric given by the
  "scoring" parameter. >>:issue:`10998`<< by >>:user:`Thomas Fan

*   Fixed a bug in "linear_model.LogisticRegressionCV" where the 'ovr'
  strategy was always used to compute cross-validation scores in the
  multiclass setting, even if "'multinomial'" was set.
  >>:issue:`8720`<< by >>:user:`William de Vazelhes <wdevazelhes>`<<.

*   Fixed a bug in "linear_model.OrthogonalMatchingPursuit" that was
  broken when setting "normalize=False". >>:issue:`10071`<< by
  Alexandre Gramfort.

*   Fixed a bug in "linear_model.ARDRegression" which caused

broken when setting "normalize=False". >>:issue:`10071`<< by

  incorrectly updated estimates for the standard deviation and the
  coefficients. >>:issue:`10153`<< by >>:user:`Jörg Döpfert
  <jdoepfert>`<<.

*   Fixed a bug in "linear_model.ARDRegression" and
  "linear_model.BayesianRidge" which caused NaN predictions when
  fitted with a constant target. >>:issue:`10095`<< by >>:user:`Jörg
  Döpfert <jdoepfert>`<<.

*   Fixed a bug in "linear_model.RidgeClassifierCV" where the
  parameter "store_cv_values" was not implemented though it was
  documented in "cv_values" as a way to set up the storage of cross-
  validation values for different alphas. >>:issue:`10297`<< by
  >>:user:`Mabel Villalba-Jiménez <mabelvj>`<<.

*   Fixed a bug in "linear_model.ElasticNet" which caused the input to
  be overridden when using parameter "copy_X=True" and
  "check_input=False". >>:issue:`10581`<< by >>:user:`Yacine Mazari
  <ymazari>`<<.

*   Fixed a bug in "sklearn.linear_model.Lasso" where the coefficient
  had wrong shape when "fit_intercept=False". >>:issue:`10687`<< by
  >>:user:`Martin Hahn <martin-hahn>`<<.

*   Fixed a bug in "sklearn.linear_model.LogisticRegression()" where
  the "multi_class='multinomial'" with binary output "with
  warm_start=True" >>:issue:`10836`<< by >>:user:`Aishwarya Srinivasan
  <aishgrt1>`<<.

*   Fixed a bug in "linear_model.RidgeCV" where using integer "alphas"

warm_start=True" >>:issue:`10836`<< by >>:user:`Aishwarya Srinivasan

  raised an error. >>:issue:`10397`<< by >>:user:`Mabel Villalba-
  Jiménez <mabelvj>`<<.

*   Fixed condition triggering gap computation in "linear_model.Lasso"
  and "linear_model.ElasticNet" when working with sparse matrices.
  >>:issue:`10992`<< by Alexandre Gramfort.

*   Fixed a bug in "linear_model.SGDClassifier",
  "linear_model.SGDRegressor",
  "linear_model.Perceptron", where the stopping criterion was stopping
  the algorithm before convergence. A parameter "n_iter_no_change" was
  added and set by default to 5. Previous behavior is equivalent to
  setting the parameter to 1. >>:issue:`9043`<< by Tom Dupre la Tour.

*   Fixed a bug where liblinear and libsvm-based estimators would
  segfault if passed a scipy.sparse matrix with 64-bit indices. They
  now raise a ValueError. >>:issue:`11327`<< by >>:user:`Karan Dhingra
  <kdhingra307>`<< and Joel Nothman.

*   The default values of the "solver" and "multi_class" parameters of
  "linear_model.LogisticRegression" will change respectively from
  "'liblinear'" and "'ovr'" in version 0.20 to "'lbfgs'" and "'auto'"
  in version 0.22. A FutureWarning is raised when the default values
  are used. >>:issue:`11905`<< by Tom Dupre la Tour and Joel Nothman.

in version 0.22. A FutureWarning is raised when the default values

*   Deprecate "positive=True" option in "linear_model.Lars" as the
  underlying implementation is broken. Use "linear_model.Lasso"
  instead. >>:issue:`9837`<< by Alexandre Gramfort.

*   "n_iter_" may vary from previous releases in
  "linear_model.LogisticRegression" with "solver='lbfgs'" and
  "linear_model.HuberRegressor". For Scipy <= 1.0.0, the optimizer
  could perform more than the requested maximum number of iterations.
  Now both estimators will report at most "max_iter" iterations even
  if more were performed. >>:issue:`10723`<< by Joel Nothman.

*   Speed improvements for both 'exact' and 'barnes_hut' methods in
  "manifold.TSNE". >>:issue:`10593`<< and >>:issue:`10610`<< by Tom
  Dupre la Tour.

*   Support sparse input in "manifold.Isomap.fit()". >>:issue:`8554`<<
  by >>:user:`Leland McInnes <lmcinnes>`<<.

*   "manifold.t_sne.trustworthiness()" accepts metrics other than
  Euclidean. >>:issue:`9775`<< by >>:user:`William de Vazelhes
  <wdevazelhes>`<<.

*   Fixed a bug in "manifold.spectral_embedding()" where the

*     Deprecate "precomputed" parameter in function

<jmargeta>`<<, >>:user:`Guillaume Lemaitre <glemaitre>`<<, and

  "manifold.t_sne.trustworthiness()". Instead, the new parameter
  "metric" should be used with any compatible metric including
  'precomputed', in which case the input matrix "X" should be a matrix
  of pairwise distances or squared distances. >>:issue:`9775`<< by

*   Deprecate "precomputed" parameter in function

*   Added the "metrics.davies_bouldin_score()" metric for evaluation
  of clustering models without a ground truth. >>:issue:`10827`<< by
  >>:user:`Luis Osa <logc>`<<.

*   Added the "metrics.balanced_accuracy_score()" metric and a
  corresponding "'balanced_accuracy'" scorer for binary and multiclass
  classification. >>:issue:`8066`<< by >>:user:`xyguo`<< and
  >>:user:`Aman Dalmia <dalmia>`<<, and >>:issue:`10587`<< by Joel
  Nothman.

*   Partial AUC is available via "max_fpr" parameter in
  "metrics.roc_auc_score()". >>:issue:`3840`<< by >>:user:`Alexander
  Niederbühl <Alexander-N>`<<.

Nothman.

*   A scorer based on "metrics.brier_score_loss()" is also available.
  >>:issue:`9521`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Added control over the normalization in
  "metrics.normalized_mutual_info_score()" and
  "metrics.adjusted_mutual_info_score()" via the "average_method"
  parameter. In version 0.22, the default normalizer for each will
  become the *arithmetic* mean of the entropies of each clustering.
  >>:issue:`11124`<< by >>:user:`Arya McCarthy <aryamccarthy>`<<.

*   Added "output_dict" parameter in "metrics.classification_report()"
  to return classification statistics as dictionary.
  >>:issue:`11160`<< by >>:user:`Dan Barkhorn <danielbarkhorn>`<<.

*   "metrics.classification_report()" now reports all applicable
  averages on the given data, including micro, macro and weighted
  average as well as samples average for multilabel data.
  >>:issue:`11679`<< by >>:user:`Alexander Pacha <apacha>`<<.

*   "metrics.average_precision_score()" now supports binary "y_true"
  other than "{0, 1}" or "{-1, 1}" through "pos_label" parameter.
  >>:issue:`9980`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   "metrics.label_ranking_average_precision_score()" now supports
  "sample_weight". >>:issue:`10845`<< by >>:user:`Jose Perez-Parras
  Toledano <jopepato>`<<.

*   Add "dense_output" parameter to

  "metrics.pairwise.linear_kernel()". When False and both inputs are
  sparse, will return a sparse matrix. >>:issue:`10999`<< by
  >>:user:`Taylor G Smith <tgsmith61591>`<<.

*   "metrics.silhouette_score()" and "metrics.silhouette_samples()"
  are more memory efficient and run faster. This avoids some reported
  freezes and MemoryErrors. >>:issue:`11135`<< by Joel Nothman.

*   Fixed a bug in "metrics.precision_recall_fscore_support()" when
  truncated *range(n_labels)* is passed as value for *labels*.
  >>:issue:`10377`<< by >>:user:`Gaurav Dhingra <gxyd>`<<.

*   Fixed a bug due to floating point error in
  "metrics.roc_auc_score()" with non-integer sample weights.
  >>:issue:`9786`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Fixed a bug where "metrics.roc_curve()" sometimes starts on y-axis
  instead of (0, 0), which is inconsistent with the document and other
  implementations. Note that this will not influence the result from
  "metrics.roc_auc_score()" >>:issue:`10093`<< by >>:user:`alexryndin
  <alexryndin>`<< and >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   Fixed a bug to avoid integer overflow. Casted product to 64 bits
  integer in "metrics.mutual_info_score()". >>:issue:`9772`<< by

*   Fixed a bug where "metrics.average_precision_score()" will
  sometimes return "nan" when "sample_weight" contains 0.

*   Fixed a bug in "metrics.fowlkes_mallows_score()" to avoid integer
  overflow. Casted return value of *contingency_matrix* to *int64* and
  computed product of square roots rather than square root of product.
  >>:issue:`9515`<< by >>:user:`Alan Liddell <aliddell>`<< and
  >>:user:`Manh Dao <manhdao>`<<.

*   Deprecate "reorder" parameter in "metrics.auc()" as it's no longer
  required for "metrics.roc_auc_score()". Moreover using
  "reorder=True" can hide bugs due to floating point error in the
  input. >>:issue:`9851`<< by >>:user:`Hanmin Qin <qinhanmin2014>`<<.

*   In "metrics.normalized_mutual_info_score()" and
  "metrics.adjusted_mutual_info_score()", warn that "average_method"
  will have a new default value. In version 0.22, the default
  normalizer for each will become the *arithmetic* mean of the
  entropies of each clustering. Currently,
  "metrics.normalized_mutual_info_score()" uses the default of
  "average_method='geometric'", and
  "metrics.adjusted_mutual_info_score()" uses the default of
  "average_method='max'" to match their behaviors in version 0.19.

"metrics.adjusted_mutual_info_score()" uses the default of

*   The "batch_size" parameter to
  "metrics.pairwise_distances_argmin_min()" and
  "metrics.pairwise_distances_argmin()" is deprecated to be removed in
  v0.22. It no longer has any effect, as batch size is determined by
  global "working_memory" config. See Limiting Working Memory.
  >>:issue:`10280`<< by Joel Nothman and >>:user:`Aman Dalmia
  <dalmia>`<<.

*   Added function *fit_predict* to "mixture.GaussianMixture" and
  "mixture.GaussianMixture", which is essentially equivalent to
  calling *fit* and *predict*. >>:issue:`10336`<< by >>:user:`Shu
  Haoran <haoranShu>`<< and >>:user:`Andrew Peng <Andrew-peng>`<<.

*   Fixed a bug in "mixture.BaseMixture" where the reported *n_iter_*
  was missing an iteration. It affected "mixture.GaussianMixture" and
  "mixture.BayesianGaussianMixture". >>:issue:`10740`<< by
  >>:user:`Erich Schubert <kno10>`<< and >>:user:`Guillaume Lemaitre

*   Fixed a bug in "mixture.BaseMixture" and its subclasses
  "mixture.GaussianMixture" and "mixture.BayesianGaussianMixture"
  where the "lower_bound_" was not the max lower bound across all
  initializations (when "n_init > 1"), but just the lower bound of the
  last initialization. >>:issue:`10869`<< by >>:user:`Aurélien Géron
  <ageron>`<<.

initializations (when "n_init > 1"), but just the lower bound of the

*   Add *return_estimator* parameter in
  "model_selection.cross_validate()" to return estimators fitted on
  each split. >>:issue:`9686`<< by >>:user:`Aurélien Bellet
  <bellet>`<<.

*   New "refit_time_" attribute will be stored in
  "model_selection.GridSearchCV" and
  "model_selection.RandomizedSearchCV" if "refit" is set to "True".
  This will allow measuring the complete time it takes to perform
  hyperparameter optimization and refitting the best model on the
  whole dataset. >>:issue:`11310`<< by >>:user:`Matthias Feurer
  <mfeurer>`<<.

*   Expose *error_score* parameter in
  "model_selection.cross_validate()",
  "model_selection.cross_val_score()",
  "model_selection.learning_curve()" and
  "model_selection.validation_curve()" to control the behavior
  triggered when an error occurs in
  "model_selection._fit_and_score()". >>:issue:`11576`<< by
  >>:user:`Samuel O. Ronsin <samronsin>`<<.

*   *BaseSearchCV* now has an experimental, private interface to
  support customized parameter search strategies, through its
  "_run_search" method. See the implementations in
  "model_selection.RandomizedSearchCV" and please provide feedback if
  you use this. Note that we do not assure the stability of this API
  beyond version 0.20. >>:issue:`9599`<< by Joel Nothman

"model_selection.RandomizedSearchCV" and please provide feedback if

*   Add improved error message in "model_selection.cross_val_score()"
  when multiple metrics are passed in "scoring" keyword.
  >>:issue:`11006`<< by >>:user:`Ming Li <minggli>`<<.

*   The default number of cross-validation folds "cv" and the default
  number of splits "n_splits" in the "model_selection.KFold"-like
  splitters will change from 3 to 5 in 0.22 as 3-fold has a lot of
  variance. >>:issue:`11557`<< by >>:user:`Alexandre Boucaud
  <aboucaud>`<<.

*   The default of "iid" parameter of "model_selection.GridSearchCV"
  and "model_selection.RandomizedSearchCV" will change from "True" to
  "False" in version 0.22 to correspond to the standard definition of
  cross-validation, and the parameter will be removed in version 0.24
  altogether. This parameter is of greatest practical significance
  where the sizes of different test sets in cross-validation were very
  unequal, i.e. in group-based CV strategies. >>:issue:`9085`<< by
  >>:user:`Laurent Direr <ldirer>`<< and Andreas Müller.

*   The default value of the "error_score" parameter in
  "model_selection.RandomizedSearchCV" will change to "np.NaN" in
  version 0.22. >>:issue:`10677`<< by >>:user:`Kirill Zhdanovich
  <Zhdanovich>`<<.

*   Changed ValueError exception raised in

"model_selection.RandomizedSearchCV" will change to "np.NaN" in

  "model_selection.ParameterSampler" to a UserWarning for case where
  the class is instantiated with a greater value of "n_iter" than the
  total space of parameters in the parameter grid. "n_iter" now acts
  as an upper bound on iterations. >>:issue:`10982`<< by
  >>:user:`Juliet Lawton <julietcl>`<<

*   Invalid input for "model_selection.ParameterGrid" now raises
  TypeError. >>:issue:`10928`<< by >>:user:`Solutus Immensus
  <solutusimmensus>`<<

*   Added "multioutput.RegressorChain" for multi-target regression.
  >>:issue:`9257`<< by >>:user:`Kumar Ashutosh <thechargedneutron>`<<.

*   Added "naive_bayes.ComplementNB", which implements the Complement
  Naive Bayes classifier described in Rennie et al. (2003).
  >>:issue:`8190`<< by >>:user:`Michael A. Alcorn <airalcorn2>`<<.

*   Add *var_smoothing* parameter in "naive_bayes.GaussianNB" to give
  a precise control over variances calculation. >>:issue:`9681`<< by
  >>:user:`Dmitry Mottl <Mottl>`<<.

*   Fixed a bug in "naive_bayes.GaussianNB" which incorrectly raised
  error for prior list which summed to 1. >>:issue:`10005`<< by

*   Fixed a bug in "naive_bayes.MultinomialNB" which did not accept

error for prior list which summed to 1. >>:issue:`10005`<< by

  vector valued pseudocounts (alpha). >>:issue:`10346`<< by
  >>:user:`Tobias Madsen <TobiasMadsen>`<<

*   "neighbors.RadiusNeighborsRegressor" and
  "neighbors.RadiusNeighborsClassifier" are now parallelized according
  to "n_jobs" regardless of "algorithm". >>:issue:`10887`<< by
  >>:user:`Joël Billaud <recamshak>`<<.

*   "Nearest neighbors" query methods are now more memory efficient
  when "algorithm='brute'". >>:issue:`11136`<< by Joel Nothman and
  >>:user:`Aman Dalmia <dalmia>`<<.

*   Add "sample_weight" parameter to the fit method of
  "neighbors.KernelDensity" to enable weighting in kernel density
  estimation. >>:issue:`4394`<< by >>:user:`Samuel O. Ronsin
  <samronsin>`<<.

*   Novelty detection with "neighbors.LocalOutlierFactor": Add a
  "novelty" parameter to "neighbors.LocalOutlierFactor". When
  "novelty" is set to True, "neighbors.LocalOutlierFactor" can then be
  used for novelty detection, i.e. predict on new unseen data.
  Available prediction methods are "predict", "decision_function" and
  "score_samples". By default, "novelty" is set to "False", and only
  the "fit_predict" method is available. By >>:user:`Albert Thomas
  <albertcthomas>`<<.

*   Fixed a bug in "neighbors.NearestNeighbors" where fitting a
  NearestNeighbors model fails when a) the distance metric used is a

<albertcthomas>`<<.

  callable and b) the input to the NearestNeighbors model is sparse.
  >>:issue:`9579`<< by >>:user:`Thomas Kober <tttthomasssss>`<<.

*   Fixed a bug so "predict" in "neighbors.RadiusNeighborsRegressor"
  can handle empty neighbor set when using non uniform weights. Also
  raises a new warning when no neighbors are found for samples.
  >>:issue:`9655`<< by >>:user:`Andreas Bjerre-Nielsen <abjer>`<<.

*     Fixed a bug in "KDTree" construction that results in faster
  construction and querying times. >>:issue:`11556`<< by >>:user:`Jake
  VanderPlas <jakevdp>`<<

*   Fixed a bug in "neighbors.KDTree" and "neighbors.BallTree" where
  pickled tree objects would change their type to the super class
  "BinaryTree". >>:issue:`11774`<< by >>:user:`Nicolas Hug

*   Add *n_iter_no_change* parameter in
  "neural_network.BaseMultilayerPerceptron",
  "neural_network.MLPRegressor", and "neural_network.MLPClassifier" to
  give control over maximum number of epochs to not meet "tol"
  improvement. >>:issue:`9456`<< by >>:user:`Nicholas Nadeau
  <nnadeau>`<<.

*   Fixed a bug in "neural_network.BaseMultilayerPerceptron",
  "neural_network.MLPRegressor", and "neural_network.MLPClassifier"
  with new "n_iter_no_change" parameter now at 10 from previously

  hardcoded 2. >>:issue:`9456`<< by >>:user:`Nicholas Nadeau

*   Fixed a bug in "neural_network.MLPRegressor" where fitting quit
  unexpectedly early due to local minima or fluctuations.
  >>:issue:`9456`<< by >>:user:`Nicholas Nadeau <nnadeau>`<<

*   The "predict" method of "pipeline.Pipeline" now passes keyword
  arguments on to the pipeline's last estimator, enabling the use of
  parameters such as "return_std" in a pipeline with caution.
  >>:issue:`9304`<< by >>:user:`Breno Freitas <brenolf>`<<.

*   "pipeline.FeatureUnion" now supports "'drop'" as a transformer to
  drop features. >>:issue:`11144`<< by >>:user:`Thomas Fan

*   Expanded "preprocessing.OneHotEncoder" to allow to encode
  categorical string features as a numeric array using a one-hot (or
  dummy) encoding scheme, and added "preprocessing.OrdinalEncoder" to
  convert to ordinal integers. Those two classes now handle encoding
  of all feature types (also handles string-valued features) and
  derives the categories based on the unique values in the features
  instead of the maximum value in the features. >>:issue:`9151`<< and
  >>:issue:`10521`<< by >>:user:`Vighnesh Birodkar

derives the categories based on the unique values in the features
  <vighneshbirodkar>`<< and Joris Van den Bossche.

*   Added "preprocessing.KBinsDiscretizer" for turning continuous
  features into categorical or one-hot encoded features.
  >>:issue:`7668`<<, >>:issue:`9647`<<, >>:issue:`10195`<<,
  >>:issue:`10192`<<, >>:issue:`11272`<<, >>:issue:`11467`<< and
  >>:issue:`11505`<<. by >>:user:`Henry Lin <hlin117>`<<, Hanmin Qin,
  Tom Dupre la Tour and >>:user:`Giovanni Giuseppe Costa <ggc87>`<<.

*   Added "preprocessing.PowerTransformer", which implements the Yeo-
  Johnson and Box-Cox power transformations. Power transformations try
  to find a set of feature-wise parametric transformations to
  approximately map data to a Gaussian distribution centered at zero
  and with unit variance. This is useful as a variance-stabilizing
  transformation in situations where normality and homoscedasticity
  are desirable. >>:issue:`10210`<< by >>:user:`Eric Chang <chang>`<<
  and >>:user:`Maniteja Nandana <maniteja123>`<<, and
  >>:issue:`11520`<< by >>:user:`Nicolas Hug <nicolashug>`<<.

*   NaN values are ignored and handled in the following preprocessing
  methods: "preprocessing.MaxAbsScaler", "preprocessing.MinMaxScaler",
  "preprocessing.RobustScaler", "preprocessing.StandardScaler",
  "preprocessing.PowerTransformer",
  "preprocessing.QuantileTransformer" classes and

"preprocessing.RobustScaler", "preprocessing.StandardScaler",
  "preprocessing.maxabs_scale()", "preprocessing.minmax_scale()",
  "preprocessing.robust_scale()", "preprocessing.scale()",
  "preprocessing.power_transform()",
  "preprocessing.quantile_transform()" functions respectively
  addressed in issues >>:issue:`11011`<<, >>:issue:`11005`<<,
  >>:issue:`11308`<<, >>:issue:`11206`<<, >>:issue:`11306`<<, and
  >>:issue:`10437`<<. By >>:user:`Lucija Gregov <LucijaGregov>`<< and

*   "preprocessing.PolynomialFeatures" now supports sparse input.
  >>:issue:`10452`<< by >>:user:`Aman Dalmia <dalmia>`<< and Joel
  Nothman.

*   "preprocessing.RobustScaler" and "preprocessing.robust_scale()"
  can be fitted using sparse matrices. >>:issue:`11308`<< by

*   "preprocessing.OneHotEncoder" now supports the *get_feature_names*
  method to obtain the transformed feature names. >>:issue:`10181`<<
  by >>:user:`Nirvan Anjirbag <Nirvan101>`<< and Joris Van den
  Bossche.

*   A parameter "check_inverse" was added to
  "preprocessing.FunctionTransformer" to ensure that "func" and
  "inverse_func" are the inverse of each other. >>:issue:`9399`<< by

*   The "transform" method of
  "sklearn.preprocessing.MultiLabelBinarizer" now ignores any unknown

>>:user:`Guillaume Lemaitre <glemaitre>`<<.

  classes. A warning is raised stating the unknown classes classes
  found which are ignored. >>:issue:`10913`<< by >>:user:`Rodrigo
  Agundez <rragundez>`<<.

*   Fixed bugs in "preprocessing.LabelEncoder" which would sometimes
  throw errors when "transform" or "inverse_transform" was called with
  empty arrays. >>:issue:`10458`<< by >>:user:`Mayur Kulkarni
  <maykulkarni>`<<.

*   Fix ValueError in "preprocessing.LabelEncoder" when using
  "inverse_transform" on unseen labels. >>:issue:`9816`<< by
  >>:user:`Charlie Newey <newey01c>`<<.

*   Fix bug in "preprocessing.OneHotEncoder" which discarded the
  "dtype" when returning a sparse matrix output. >>:issue:`11042`<< by
  >>:user:`Daniel Morales <DanielMorales9>`<<.

*   Fix "fit" and "partial_fit" in "preprocessing.StandardScaler" in
  the rare case when "with_mean=False" and *with_std=False* which was
  crashing by calling "fit" more than once and giving inconsistent
  results for "mean_" whether the input was a sparse or a dense
  matrix. "mean_" will be set to "None" with both sparse and dense
  inputs. "n_samples_seen_" will be also reported for both input
  types. >>:issue:`11235`<< by >>:user:`Guillaume Lemaitre

*   Deprecate "n_values" and "categorical_features" parameters and
  "active_features_", "feature_indices_" and "n_values_" attributes of

<glemaitre>`<<.

  "preprocessing.OneHotEncoder". The "n_values" parameter can be
  replaced with the new "categories" parameter, and the attributes
  with the new "categories_" attribute. Selecting the categorical
  features with the "categorical_features" parameter is now better
  supported using the "compose.ColumnTransformer". >>:issue:`10521`<<
  by Joris Van den Bossche.

*   Deprecate "preprocessing.Imputer" and move the corresponding
  module to "impute.SimpleImputer". >>:issue:`9726`<< by

*   The "axis" parameter that was in "preprocessing.Imputer" is no
  longer present in "impute.SimpleImputer". The behavior is equivalent
  to "axis=0" (impute along columns). Row-wise imputation can be
  performed with FunctionTransformer (e.g.,
  "FunctionTransformer(lambda X:
  SimpleImputer().fit_transform(X.T).T)"). >>:issue:`10829`<< by
  >>:user:`Guillaume Lemaitre <glemaitre>`<< and >>:user:`Gilberto
  Olimpio <gilbertoolimpio>`<<.

*   The NaN marker for the missing values has been changed between the
  "preprocessing.Imputer" and the "impute.SimpleImputer".
  "missing_values='NaN'" should now be "missing_values=np.nan".
  >>:issue:`11211`<< by >>:user:`Jeremie du Boisberranger

*   In "preprocessing.FunctionTransformer", the default of "validate"

>>:issue:`11211`<< by >>:user:`Jeremie du Boisberranger

  will be from "True" to "False" in 0.22. >>:issue:`10655`<< by

*   Fixed a bug in "svm.SVC" where when the argument "kernel" is
  unicode in Python2, the "predict_proba" method was raising an
  unexpected TypeError given dense inputs. >>:issue:`10412`<< by
  >>:user:`Jiongyan Zhang <qmick>`<<.

*   Deprecate "random_state" parameter in "svm.OneClassSVM" as the
  underlying implementation is not random. >>:issue:`9497`<< by

*   The default value of "gamma" parameter of "svm.SVC", "NuSVC",
  "SVR", "NuSVR", "OneClassSVM" will change from "'auto'" to "'scale'"
  in version 0.22 to account better for unscaled features.
  >>:issue:`8361`<< by >>:user:`Gaurav Dhingra <gxyd>`<< and
  >>:user:`Ting Neo <neokt>`<<.

*   Although private (and hence not assured API stability),
  "tree._criterion.ClassificationCriterion" and
  "tree._criterion.RegressionCriterion" may now be cimported and
  extended. >>:issue:`10325`<< by >>:user:`Camil Staps
  <camilstaps>`<<.

*   Fixed a bug in "tree.BaseDecisionTree" with *splitter="best"*
  where split threshold could become infinite when values in X were
  near infinite. >>:issue:`10536`<< by >>:user:`Jonathan Ohayon

where split threshold could become infinite when values in X were

*   Fixed a bug in "tree.MAE" to ensure sample weights are being used
  during the calculation of tree MAE impurity. Previous behaviour
  could cause suboptimal splits to be chosen since the impurity
  calculation considered all samples to be of equal weight importance.
  >>:issue:`11464`<< by >>:user:`John Stott <JohnStott>`<<.

*   "utils.check_array()" and "utils.check_X_y()" now have
  "accept_large_sparse" to control whether scipy.sparse matrices with
  64-bit indices should be rejected. >>:issue:`11327`<< by
  >>:user:`Karan Dhingra <kdhingra307>`<< and Joel Nothman.

*     Avoid copying the data in "utils.check_array()" when the input
  data is a memmap (and "copy=False"). >>:issue:`10663`<< by
  >>:user:`Arthur Mensch <arthurmensch>`<< and >>:user:`Loïc Estève
  <lesteve>`<<.

*   "utils.check_array()" yield a "FutureWarning" indicating that
  arrays of bytes/strings will be interpreted as decimal numbers
  beginning in version 0.22. >>:issue:`10229`<< by >>:user:`Ryan Lee
  <rtlee9>`<<

*     More consistent outlier detection API: Add a "score_samples"
  method in "svm.OneClassSVM", "ensemble.IsolationForest",
  "neighbors.LocalOutlierFactor", "covariance.EllipticEnvelope". It
  allows to access raw score functions from original papers. A new

method in "svm.OneClassSVM", "ensemble.IsolationForest",
  "offset_" parameter allows to link "score_samples" and
  "decision_function" methods. The "contamination" parameter of
  "ensemble.IsolationForest" and "neighbors.LocalOutlierFactor"
  "decision_function" methods is used to define this "offset_" such
  that outliers (resp. inliers) have negative (resp. positive)
  "decision_function" values. By default, "contamination" is kept
  unchanged to 0.1 for a deprecation period. In 0.22, it will be set
  to "auto", thus using method-specific score offsets. In
  "covariance.EllipticEnvelope" "decision_function" method, the
  "raw_values" parameter is deprecated as the shifted Mahalanobis
  distance will be always returned in 0.22. >>:issue:`9015`<< by
  Nicolas Goix.

*     A "behaviour" parameter has been introduced in
  "ensemble.IsolationForest" to ensure backward compatibility. In the
  old behaviour, the "decision_function" is independent of the
  "contamination" parameter. A threshold attribute depending on the
  "contamination" parameter is thus used. In the new behaviour the
  "decision_function" is dependent on the "contamination" parameter,
  in such a way that 0 becomes its natural threshold to detect
  outliers. Setting behaviour to "old" is deprecated and will not be
  possible in version 0.22. Beside, the behaviour parameter will be

in such a way that 0 becomes its natural threshold to detect
  removed in 0.24. >>:issue:`11553`<< by Nicolas Goix.

*   Added convergence warning to "svm.LinearSVC" and
  "linear_model.LogisticRegression" when "verbose" is set to 0.
  >>:issue:`10881`<< by >>:user:`Alexandre Sevin <AlexandreSev>`<<.

*   Changed warning type from "UserWarning" to
  "exceptions.ConvergenceWarning" for failing convergence in
  "linear_model.logistic_regression_path()",
  "linear_model.RANSACRegressor", "linear_model.ridge_regression()",
  "gaussian_process.GaussianProcessRegressor",
  "gaussian_process.GaussianProcessClassifier",
  "decomposition.fastica()", "cross_decomposition.PLSCanonical",
  "cluster.AffinityPropagation", and "cluster.Birch".
  >>:issue:`10306`<< by >>:user:`Jonathan Siebert <jotasi>`<<.

*   A new configuration parameter, "working_memory" was added to
  control memory consumption limits in chunked operations, such as the
  new "metrics.pairwise_distances_chunked()". See Limiting Working
  Memory. >>:issue:`10280`<< by Joel Nothman and >>:user:`Aman Dalmia

*   The version of "joblib" bundled with Scikit-learn is now 0.12.
  This uses a new default multiprocessing implementation, named loky.
  While this may incur some memory and communication overhead, it
  should provide greater cross-platform stability than relying on

This uses a new default multiprocessing implementation, named loky.
  Python standard library multiprocessing. >>:issue:`11741`<< by the
  Joblib developers, especially >>:user:`Thomas Moreau <tomMoral>`<<
  and Olivier Grisel.

*   An environment variable to use the site joblib instead of the
  vendored one was added (Environment variables). The main API of
  joblib is now exposed in "sklearn.utils". >>:issue:`11166`<< by Gael
  Varoquaux.

*   Add almost complete PyPy 3 support. Known unsupported
  functionalities are "datasets.load_svmlight_file()",
  "feature_extraction.FeatureHasher" and
  "feature_extraction.text.HashingVectorizer". For running on PyPy,
  PyPy3-v5.10+, Numpy 1.14.0+, and scipy 1.1.0+ are required.
  >>:issue:`11010`<< by >>:user:`Ronan Lamy <rlamy>`<< and Roman
  Yurchak.

*   A utility method "sklearn.show_versions()" was added to print out
  information relevant for debugging. It includes the user system, the
  Python executable, the version of the main libraries and BLAS
  binding information. >>:issue:`11596`<< by >>:user:`Alexandre
  Boucaud <aboucaud>`<<

*   Fixed a bug when setting parameters on meta-estimator, involving
  both a wrapped estimator and its parameter. >>:issue:`9999`<< by
  >>:user:`Marcus Voss <marcus-voss>`<< and Joel Nothman.

*   Fixed a bug where calling "sklearn.base.clone()" was not thread

both a wrapped estimator and its parameter. >>:issue:`9999`<< by

  safe and could result in a "pop from empty list" error.
  >>:issue:`9569`<< by Andreas Müller.

*   The default value of "n_jobs" is changed from "1" to "None" in all
  related functions and classes. "n_jobs=None" means "unset". It will
  generally be interpreted as "n_jobs=1", unless the current
  "joblib.Parallel" backend context specifies otherwise (See
  *Glossary* for additional information). Note that this change
  happens immediately (i.e., without a deprecation cycle).
  >>:issue:`11741`<< by Olivier Grisel.

*   Fixed a bug in validation helpers where passing a Dask DataFrame
  results in an error. >>:issue:`12462`<< by >>:user:`Zachariah Miller
  <zwmiller>`<<

* Checks for transformers now apply if the estimator implements
  *transform*, regardless of whether it inherits from
  "sklearn.base.TransformerMixin". >>:issue:`10474`<< by Joel Nothman.

* Classifiers are now checked for consistency between
  *decision_function* and categorical predictions. >>:issue:`10500`<<
  by >>:user:`Narine Kokhlikyan <NarineK>`<<.

* Allow tests in "utils.estimator_checks.check_estimator()" to test
  functions that accept pairwise data. >>:issue:`9701`<< by
  >>:user:`Kyle Johnson <gkjohns>`<<

* Allow "utils.estimator_checks.check_estimator()" to check that there
  is no private settings apart from parameters during estimator
  initialization. >>:issue:`9378`<< by >>:user:`Herilalaina
  Rakotoarison <herilalaina>`<<

* The set of checks in "utils.estimator_checks.check_estimator()" now
  includes a "check_set_params" test which checks that "set_params" is
  equivalent to passing parameters in "__init__" and warns if it
  encounters parameter validation. >>:issue:`7738`<< by >>:user:`Alvin
  Chiang <absolutelyNoWarranty>`<<

* Add invariance tests for clustering metrics. >>:issue:`8102`<< by
  >>:user:`Ankita Sinha <anki08>`<< and >>:user:`Guillaume Lemaitre

* Add "check_methods_subset_invariance" to "check_estimator()", which
  checks that estimator methods are invariant if applied to a data
  subset. >>:issue:`10428`<< by >>:user:`Jonathan Ohayon <Johayon>`<<

* Add tests in "utils.estimator_checks.check_estimator()" to check
  that an estimator can handle read-only memmap input data.
  >>:issue:`10663`<< by >>:user:`Arthur Mensch <arthurmensch>`<< and
  >>:user:`Loïc Estève <lesteve>`<<.

* "check_sample_weights_pandas_series" now uses 8 rather than 6
  samples to accommodate for the default number of clusters in
  "cluster.KMeans". >>:issue:`10933`<< by >>:user:`Johannes Hansen
  <jnhansen>`<<.

samples to accommodate for the default number of clusters in

* Estimators are now checked for whether "sample_weight=None" equates
  to "sample_weight=np.ones(...)". >>:issue:`11558`<< by
  >>:user:`Sergul Aydore <sergulaydore>`<<.

improvement of the project since version 0.19, including:

211217613, Aarshay Jain, absolutelyNoWarranty, Adam Greenhall, Adam
Kleczewski, Adam Richie-Halford, adelr, AdityaDaflapurkar, Adrin
Jalali, Aidan Fitzgerald, aishgrt1, Akash Shivram, Alan Liddell, Alan
Yee, Albert Thomas, Alexander Lenail, Alexander-N, Alexandre Boucaud,
Alexandre Gramfort, Alexandre Sevin, Alex Egg, Alvaro Perez-Diaz,
Amanda, Aman Dalmia, Andreas Bjerre-Nielsen, Andreas Mueller, Andrew
Peng, Angus Williams, Aniruddha Dave, annaayzenshtat, Anthony Gitter,
Antonio Quinonez, Anubhav Marwaha, Arik Pamnani, Arthur Ozga, Artiem
K, Arunava, Arya McCarthy, Attractadore, Aurélien Bellet, Aurélien
Geron, Ayush Gupta, Balakumaran Manoharan, Bangda Sun, Barry Hart,
Bastian Venthur, Ben Lawson, Benn Roth, Breno Freitas, Brent Yi, brett
koonce, Caio Oliveira, Camil Staps, cclauss, Chady Kamar, Charlie
Brummitt, Charlie Newey, chris, Chris, Chris Catalfo, Chris Foster,
Chris Holdgraf, Christian Braune, Christian Hirsch, Christian Hogan,

Christopher Jenness, Clement Joudet, cnx, cwitte, Dallas Card, Dan
Barkhorn, Daniel, Daniel Ferreira, Daniel Gomez, Daniel Klevebring,
Danielle Shwed, Daniel Mohns, Danil Baibak, Darius Morawiec, David
Beach, David Burns, David Kirkby, David Nicholson, David Pickup,
Derek, Didi Bar-Zev, diegodlh, Dillon Gardner, Dillon Niederhut,
dilutedsauce, dlovell, Dmitry Mottl, Dmitry Petrov, Dor Cohen, Douglas
Duhaime, Ekaterina Tuzova, Eric Chang, Eric Dean Sanchez, Erich
Schubert, Eunji, Fang-Chieh Chou, FarahSaeed, felix, Félix Raimundo,
fenx, filipj8, FrankHui, Franz Wompner, Freija Descamps, frsi,
Gabriele Calvo, Gael Varoquaux, Gaurav Dhingra, Georgi Peev, Gil
Forsyth, Giovanni Giuseppe Costa, gkevinyen5418, goncalo-rodrigues,
Gryllos Prokopis, Guillaume Lemaitre, Guillaume "Vermeille" Sanchez,
Gustavo De Mari Pereira, hakaa1, Hanmin Qin, Henry Lin, Hong, Honghe,
Hossein Pourbozorg, Hristo, Hunan Rostomyan, iampat, Ivan PANICO,
Jaewon Chung, Jake VanderPlas, jakirkham, James Bourbeau, James
Malcolm, Jamie Cox, Jan Koch, Jan Margeta, Jan Schlüter, janvanrijn,
Jason Wolosonovich, JC Liu, Jeb Bearer, jeremiedbb, Jimmy Wan, Jinkun
Wang, Jiongyan Zhang, jjabl, jkleint, Joan Massich, Joël Billaud, Joel
Nothman, Johannes Hansen, JohnStott, Jonatan Samoocha, Jonathan

Ohayon, Jörg Döpfert, Joris Van den Bossche, Jose Perez-Parras
Toledano, josephsalmon, jotasi, jschendel, Julian Kuhlmann, Julien
Chaumond, julietcl, Justin Shenk, Karl F, Kasper Primdal Lauritzen,
Katrin Leinweber, Kirill, ksemb, Kuai Yu, Kumar Ashutosh, Kyeongpil
Kang, Kye Taylor, kyledrogo, Leland McInnes, Léo DS, Liam Geron,
Liutong Zhou, Lizao Li, lkjcalc, Loic Esteve, louib, Luciano Viola,
Lucija Gregov, Luis Osa, Luis Pedro Coelho, Luke M Craig, Luke
Persola, Mabel, Mabel Villalba, Maniteja Nandana, MarkIwanchyshyn,
Mark Roth, Markus Müller, MarsGuy, Martin Gubri, martin-hahn, martin-
kokos, mathurinm, Matthias Feurer, Max Copeland, Mayur Kulkarni,
Meghann Agarwal, Melanie Goetz, Michael A. Alcorn, Minghui Liu, Ming
Li, Minh Le, Mohamed Ali Jamaoui, Mohamed Maskani, Mohammad Shahebaz,
Muayyad Alsadi, Nabarun Pal, Nagarjuna Kumar, Naoya Kanai, Narendran
Santhanam, NarineK, Nathaniel Saul, Nathan Suh, Nicholas Nadeau,
P.Eng.,  AVS, Nick Hoh, Nicolas Goix, Nicolas Hug, Nicolau Werneck,
nielsenmarkus11, Nihar Sheth, Nikita Titov, Nilesh Kevlani, Nirvan
Anjirbag, notmatthancock, nzw, Oleksandr Pavlyk, oliblum90, Oliver
Rausch, Olivier Grisel, Oren Milman, Osaid Rehman Nasir, pasbi,
Patrick Fernandes, Patrick Olden, Paul Paczuski, Pedro Morales, Peter,
Peter St. John, pierreablin, pietruh, Pinaki Nath Chowdhury, Piotr

Szymański, Pradeep Reddy Raamana, Pravar D Mahajan, pravarmahajan,
QingYing Chen, Raghav RV, Rajendra arora, RAKOTOARISON Herilalaina,
Rameshwar Bhaskaran, RankyLau, Rasul Kerimov, Reiichiro Nakano, Rob,
Roman Kosobrodov, Roman Yurchak, Ronan Lamy, rragundez, Rüdiger
Busche, Ryan, Sachin Kelkar, Sagnik Bhattacharya, Sailesh Choyal, Sam
Radhakrishnan, Sam Steingold, Samuel Bell, Samuel O. Ronsin, Saqib
Nizam Shamsi, SATISH J, Saurabh Gupta, Scott Gigante, Sebastian
Flennerhag, Sebastian Raschka, Sebastien Dubois, Sébastien Lerique,
Sebastin Santy, Sergey Feldman, Sergey Melderis, Sergul Aydore,
Shahebaz, Shalil Awaley, Shangwu Yao, Sharad Vijalapuram, Sharan
Yalburgi, shenhanc78, Shivam Rastogi, Shu Haoran, siftikha, Sinclert
Pérez, SolutusImmensus, Somya Anand, srajan paliwal, Sriharsha Hatwar,
Sri Krishna, Stefan van der Walt, Stephen McDowell, Steven Brown,
syonekura, Taehoon Lee, Takanori Hayashi, tarcusx, Taylor G Smith,
theriley106, Thomas, Thomas Fan, Thomas Heavey, Tobias Madsen,
tobycheese, Tom Augspurger, Tom Dupré la Tour, Tommy, Trevor Stephens,
Trishnendu Ghorai, Tulio Casagrande, twosigmajab, Umar Farouk Umar,
Urvang Patel, Utkarsh Upadhyay, Vadim Markovtsev, Varun Agrawal,
Vathsala Achar, Vilhelm von Ehrenheim, Vinayak Mehta, Vinit, Vinod

Kumar L, Viraj Mavani, Viraj Navkal, Vivek Kumar, Vlad Niculae,
vqean3, Vrishank Bhardwaj, vufg, wallygauze, Warut Vijitbenjaronk,
wdevazelhes, Wenhao Zhang, Wes Barnett, Will, William de Vazelhes,
Will Rosenfeld, Xin Xiong, Yiming (Paul) Li, ymazari, Yufeng, Zach
Griffith, Zé Vinícius, Zhenqing Hu, Zhiqing Xiao, Zijie (ZJ) Poh

Version 1.3.0

**In Development**

  memory.

*   "multiclass.OutputCodeClassifier.predict()" now uses a more
  efficient pairwise distance reduction. As a consequence, the tie-
  breaking strategy is different and thus the predicted labels may be
  different. >>:pr:`25196`<< by >>:user:`Guillaume Lemaitre

*   The *fit_transform* method of "decomposition.DictionaryLearning"
  is more efficient but may produce different results as in previous
  versions when *transform_algorithm* is not the same as
  *fit_algorithm* and the number of iterations is small.
  >>:pr:`24871`<< by >>:user:`Omar Salman <OmarManzoor>`<<.

*fit_algorithm* and the number of iterations is small.

*   The *get_feature_names_out* method of the following classes now
  raises a *NotFittedError* if the instance is not fitted. This
  ensures the error is consistent in all estimators with the
  *get_feature_names_out* method.

  * "impute.MissingIndicator"

  * "feature_extraction.DictVectorizer"

  * "feature_extraction.text.TfidfTransformer"

  * "feature_selection.GenericUnivariateSelect"

  * "feature_selection.RFE"

  * "feature_selection.RFECV"

  * "feature_selection.SelectFdr"

  * "feature_selection.SelectFpr"

  * "feature_selection.SelectFromModel"

  * "feature_selection.SelectFwe"

  * "feature_selection.SelectKBest"

  * "feature_selection.SelectPercentile"

  * "feature_selection.SequentialFeatureSelector"

  * "feature_selection.VarianceThreshold"

  * "kernel_approximation.AdditiveChi2Sampler"

  * "impute.IterativeImputer"

  * "impute.KNNImputer"

  * "impute.SimpleImputer"

  * "isotonic.IsotonicRegression"

  * "preprocessing.Binarizer"

  * "preprocessing.KBinsDiscretizer"

  * "preprocessing.MaxAbsScaler"

  * "preprocessing.MinMaxScaler"

  * "preprocessing.Normalizer"

  * "preprocessing.OrdinalEncoder"

  * "preprocessing.PowerTransformer"

  * "preprocessing.QuantileTransformer"

  * "preprocessing.RobustScaler"

  * "preprocessing.SplineTransformer"

  * "preprocessing.StandardScaler"

  * "random_projection.GaussianRandomProjection"

* "preprocessing.QuantileTransformer"

  * "random_projection.SparseRandomProjection"

  The *NotFittedError* displays an informative message asking to fit
  the instance with the appropriate arguments.

  >>:pr:`25294`<<, >>:pr:`25308`<<, >>:pr:`25291`<<, >>:pr:`25367`<<,
  >>:pr:`25402`<<, by >>:user:`John Pangas <jpangas>`<<,
  >>:user:`Rahil Parikh <rprkh>`<< , and >>:user:`Alex Buzenet
  <albuzenet>`<<.

Changelog
=========

*   A *__sklearn_clone__* protocol is now available to override the
  default behavior of "base.clone()". >>:pr:`24568`<< by Thomas Fan.

*   The *sample_weight* parameter in *predict* for
  "cluster.KMeans.predict()" and "cluster.MiniBatchKMeans.predict()"
  is now deprecated and will be removed in v1.5. >>:pr:`25251`<< by
  >>:user:`Gleb Levitski <glevv>`<<.

*   "decomposition.DictionaryLearning" now accepts the parameter
  *callback* for consistency with the function
  "decomposition.dict_learning()". >>:pr:`24871`<< by >>:user:`Omar
  Salman <OmarManzoor>`<<.

*   "ensemble.HistGradientBoostingRegressor" now supports the Gamma
  deviance loss via *loss="gamma"*. Using the Gamma deviance as loss
  function comes in handy for modelling skewed distributed, strictly

deviance loss via *loss="gamma"*. Using the Gamma deviance as loss
  positive valued targets. >>:pr:`22409`<< by >>:user:`Christian

*   Compute a custom out-of-bag score by passing a callable to
  "ensemble.ExtraTreesClassifier" and "ensemble.ExtraTreesRegressor".
  >>:pr:`25177`<< by >>:user:`Tim Head <betatim>`<<.

*   "ensemble.GradientBoostingClassifier" now exposes out-of-bag
  scores via the *oob_scores_* or *oob_score_* attributes.
  >>:pr:`24882`<< by >>:user:`Ashwin Mathur <awinml>`<<.

*   "ensemble.IsolationForest" predict time is now faster (typically
  by a factor of 8 or more). Internally, the estimator now precomputes
  decision path lengths per tree at *fit* time. It is therefore not
  possible to load an estimator trained with scikit-learn 1.2 to make
  it predict with scikit-learn 1.3: retraining with scikit-learn 1.3
  is required. >>:pr:`25186`<< by >>:user:`Felipe Breve Siola
  <fsiola>`<<.

*   "ensemble.BaggingClassifier" and "ensemble.BaggingRegressor"
  expose the *allow_nan* tag from the underlying estimator.
  >>:pr:`25506`<< by Thomas Fan.

"sklearn.exception"

*   Added "exception.InconsistentVersionWarning" which is raised when
  a scikit-learn estimator is unpickled with a scikit-learn version
  that is inconsistent with the sckit-learn verion the estimator was

a scikit-learn estimator is unpickled with a scikit-learn version
  pickled with. >>:pr:`25297`<< by Thomas Fan.

*   Added the parameter *fill_value* to "impute.IterativeImputer".
  >>:pr:`25232`<< by >>:user:`Thijs van Weezel
  <ValueInvestorThijs>`<<.

*   "metric.manhattan_distances()" now supports readonly sparse
  datasets. >>:pr:`25432`<< by >>:user:`Julien Jerphanion

*   "naive_bayes.GaussianNB" does not raise anymore a
  *ZeroDivisionError* when the provided *sample_weight* reduces the
  problem to a single class in *fit*. >>:pr:`24140`<< by
  >>:user:`Jonathan Ohayon <Johayon>`<< and >>:user:`Chiara Marmo
  <cmarmo>`<<.

*   Remove support for *KulsinskiDistance* in "neighbors.BallTree".
  This dissimilarity is not a metric and cannot be supported by the
  BallTree. >>:pr:`25417`<< by >>:user:`Guillaume Lemaitre

*   "pipeline.FeatureUnion" can now use indexing notation (e.g.
  *feature_union["scalar"]*) to access transformers by name.
  >>:pr:`25093`<< by Thomas Fan.

*   "pipeline.FeatureUnion" can now access the *feature_names_in_*
  attribute if the *X* value seen during *.fit* has a *columns*
  attribute and all columns are strings. e.g. when *X* is a

  *pandas.DataFrame* >>:pr:`25220`<< by >>:user:`Ian Thompson
  <it176131>`<<.

*   Adds a *feature_name_combiner* parameter to
  "preprocessing.OneHotEncoder". This specifies a custom callable to
  create feature names to be returned by "get_feature_names_out()".
  The callable combines input arguments *(input_feature, category)* to
  a string. >>:pr:`22506`<< by >>:user:`Mario Kostelac
  <mariokostelac>`<<.

*   Added support for *sample_weight* in
  "preprocessing.KBinsDiscretizer". This allows specifying the
  parameter *sample_weight* for each sample to be used while fitting.
  The option is only available when *strategy* is set to *quantile*
  and *kmeans*. >>:pr:`24935`<< by >>:user:`Seladus <seladus>`<<,
  >>:user:`Guillaume Lemaitre <glemaitre>`<<, and >>:user:`Dea María
  Léon <deamarialeon>`<<, >>:pr:`25257`<< by >>:user:`Gleb Levitski
  <glevv>`<<.

*   "AdditiveChi2Sampler" is now stateless. The *sample_interval_*
  attribute is deprecated and will be removed in 1.5. >>:pr:`25190`<<
  by >>:user:`Vincent Maladière <Vincent-Maladiere>`<<.

*   "estimator_checks.check_transformers_unfitted_stateless()" has
  been introduced to ensure stateless transformers don't raise

  *NotFittedError* during *transform* with no prior call to *fit* or
  *fit_transform*. >>:pr:`25190`<< by >>:user:`Vincent Maladière
  <Vincent-Maladiere>`<<.

improvement of the project since version 1.2, including:

TODO: update at the time of the release.

Version 1.1.3

**October 2022**

This bugfix release only includes fixes for compatibility with the
latest SciPy release >= 1.9.2. Notable changes include:

*   Include *msvcp140.dll* in the scikit-learn wheels since it has
  been removed in the latest SciPy wheels. >>:pr:`24631`<< by
  >>:user:`Chiara Marmo <cmarmo>`<<.

*   Create wheels for Python 3.11. >>:pr:`24446`<< by >>:user:`Chiara
  Marmo <cmarmo>`<<.

Other bug fixes will be available in the next 1.2 release, which will
be released in the coming weeks.

Note that support for 32-bit Python on Windows has been dropped in
this release. This is due to the fact that SciPy 1.9.2 also dropped
the support for that platform. Windows users are advised to install
the 64-bit version of Python instead.

Version 1.1.2

**August 2022**

*   "manifold.TSNE" now throws a *ValueError* when fit with
  *perplexity>=n_samples* to ensure mathematical correctness of the
  algorithm. >>:pr:`10805`<< by >>:user:`Mathias Andersen
  <MrMathias>`<< and >>:pr:`23471`<< by >>:user:`Meekail Zain

Changelog
=========

*   A default HTML representation is shown for meta-estimators with
  invalid parameters. >>:pr:`24015`<< by Thomas Fan.

<micky774>`<<.

Changelog
=========

*   Add support for F-contiguous arrays for estimators and functions
  whose back-end have been changed in 1.1. >>:pr:`23990`<< by
  >>:user:`Julien Jerphanion <jjerphan>`<<.

*   Wheels are now available for MacOS 10.9 and greater.
  >>:pr:`23833`<< by Thomas Fan.

*   The *get_params* method of the "BaseEstimator" class now supports
  estimators with *type*-type params that have the *get_params*
  method. >>:pr:`24017`<< by >>:user:`Henry Sorsky <hsorsky>`<<.

*   Fixed a bug in "cluster.Birch" that could trigger an error when
  splitting a node if there are duplicates in the dataset.
  >>:pr:`23395`<< by >>:user:`Jérémie du Boisberranger

*   "feature_selection.SelectFromModel" defaults to selection
  threshold 1e-5 when the estimator is either
  "linear_model.ElasticNet" or "linear_model.ElasticNetCV" with
  *l1_ratio* equals 1 or "linear_model.LassoCV". >>:pr:`23636`<< by
  >>:user:`Hao Chun Chang <haochunchang>`<<.

*   "impute.SimpleImputer" uses the dtype seen in *fit* for
  *transform* when the dtype is object. >>:pr:`22063`<< by Thomas Fan.

*   Use dtype-aware tolerances for the validation of gram matrices

*transform* when the dtype is object. >>:pr:`22063`<< by Thomas Fan.

  (passed by users or precomputed). >>:pr:`22059`<< by >>:user:`Malte
  S. Kurz <MalteKurz>`<<.

*   Fixed an error in "linear_model.LogisticRegression" with *solver
  ="newton-cg"*, *fit_intercept=True*, and a single feature.
  >>:pr:`23608`<< by Tom Dupre la Tour.

*   Fixed error message of "metrics.coverage_error" for 1D array
  input. >>:pr:`23548`<< by >>:user:`Hao Chun Chang <haochunchang>`<<.

*   "preprocessing.OrdinalEncoder.inverse_transform()" correctly
  handles use cases where *unknown_value* or *encoded_missing_value*
  is *nan*. >>:pr:`24087`<< by Thomas Fan.

*   Fixed invalid memory access bug during fit in
  "tree.DecisionTreeRegressor" and "tree.DecisionTreeClassifier".
  >>:pr:`23273`<< by Thomas Fan.

Version 1.1.1

**May 2022**

Changelog
=========

*   The error message is improved when importing
  "model_selection.HalvingGridSearchCV",
  "model_selection.HalvingRandomSearchCV", or

Changelog
=========

  "impute.IterativeImputer" without importing the experimental flag.
  >>:pr:`23194`<< by Thomas Fan.

*   Added an extension in doc/conf.py to automatically generate the
  list of estimators that handle NaN values. >>:pr:`23198`<< by
  >>:user:`Lise Kleiber <lisekleiber>`<<, >>:user:`Zhehao Liu
  <MaxwellLZH>`<< and >>:user:`Chiara Marmo <cmarmo>`<<.

*   Avoid timeouts in "datasets.fetch_openml()" by not passing a
  *timeout* argument, >>:pr:`23358`<< by >>:user:`Loïc Estève

*   Avoid spurious warning in "decomposition.IncrementalPCA" when
  *n_samples == n_components*. >>:pr:`23264`<< by >>:user:`Lucy Liu
  <lucyleeow>`<<.

*   The *partial_fit* method of "feature_selection.SelectFromModel"
  now conducts validation for *max_features* and *feature_names_in*
  parameters. >>:pr:`23299`<< by >>:user:`Long Bao <lorentzbao>`<<.

*   Fixes "metrics.precision_recall_curve()" to compute precision-
  recall at 100% recall. The Precision-Recall curve now displays the
  last point corresponding to a classifier that always predicts the
  positive class: recall=100% and precision=class balance.

recall at 100% recall. The Precision-Recall curve now displays the
  >>:pr:`23214`<< by >>:user:`Stéphane Collot <stephanecollot>`<< and
  >>:user:`Max Baak <mbaak>`<<.

*   "preprocessing.PolynomialFeatures" with "degree" equal to 0 will
  raise error when "include_bias" is set to False, and outputs a
  single constant array when "include_bias" is set to True.
  >>:pr:`23370`<< by >>:user:`Zhehao Liu <MaxwellLZH>`<<.

*   Fixes performance regression with low cardinality features for
  "tree.DecisionTreeClassifier", "tree.DecisionTreeRegressor",
  "ensemble.GradientBoostingRegressor". >>:pr:`23410`<< by

*   "utils.class_weight.compute_sample_weight()" now works with sparse
  *y*. >>:pr:`23115`<< by >>:user:`kernc <kernc>`<<.

Version 1.1.0

lights_1_1_0.py*.

  memory.

Minimal dependencies
====================

Version 1.1.0 of scikit-learn requires python 3.8+, numpy 1.17.3+ and
scipy 1.3.2+. Optional minimal dependency is matplotlib 3.1.2+.

*   "cluster.KMeans" now defaults to "algorithm="lloyd"" instead of
  "algorithm="auto"", which was equivalent to "algorithm="elkan"".
  Lloyd's algorithm and Elkan's algorithm converge to the same
  solution, up to numerical rounding errors, but in general Lloyd's
  algorithm uses much less memory, and it is often faster.

*   Fitting "tree.DecisionTreeClassifier",
  "tree.DecisionTreeRegressor", "ensemble.RandomForestClassifier",
  "ensemble.RandomForestRegressor",

  "ensemble.GradientBoostingRegressor" is on average 15% faster than
  in previous versions thanks to a new sort algorithm to find the best
  split. Models might be different because of a different handling of
  splits with tied criterion values: both the old and the new sorting
  algorithm are unstable sorting algorithms. >>:pr:`22868`<< by Thomas
  Fan.

*   The eigenvectors initialization for "cluster.SpectralClustering"
  and "manifold.SpectralEmbedding" now samples from a Gaussian when
  using the *'amg'* or *'lobpcg'* solver. This change  improves
  numerical stability of the solver, but may result in a different
  model.

*   "feature_selection.f_regression()" and
  "feature_selection.r_regression()" will now returned finite score by
  default instead of *np.nan* and *np.inf* for some corner case. You
  can use *force_finite=False* if you really want to get non-finite
  values and keep the old behavior.

*   Panda's DataFrames with all non-string columns such as a
  MultiIndex no longer warns when passed into an Estimator. Estimators
  will continue to ignore the column names in DataFrames with non-
  string columns. For *feature_names_in_* to be defined, columns must
  be all strings. >>:pr:`22410`<< by Thomas Fan.

*   "preprocessing.KBinsDiscretizer" changed handling of bin edges

string columns. For *feature_names_in_* to be defined, columns must

  slightly, which might result in a different encoding with the same
  data.

*   "calibration.calibration_curve()" changed handling of bin edges
  slightly, which might result in a different output curve given the
  same data.

*   "discriminant_analysis.LinearDiscriminantAnalysis" now uses the
  correct variance-scaling coefficient which may result in different
  model behavior.

*   "feature_selection.SelectFromModel.fit()" and
  "feature_selection.SelectFromModel.partial_fit()" can now be called
  with *prefit=True*. *estimators_* will be a deep copy of *estimator*
  when *prefit=True*. >>:pr:`23271`<< by >>:user:`Guillaume Lemaitre

Changelog
=========

  float64 datasets have been refactored. The following functions and

* "sklearn.cluster.SpectralClustering"

  >>:pr:`21987`<<, >>:pr:`22064`<<, >>:pr:`22065`<<, >>:pr:`22288`<<
  and >>:pr:`22320`<< by >>:user:`Julien Jerphanion <jjerphan>`<<.

*   All scikit-learn models now generate a more informative error
  message when some input contains unexpected *NaN* or infinite
  values. In particular the message contains the input name ("X", "y"
  or "sample_weight") and if an unexpected *NaN* value is found in
  *X*, the error message suggests potential solutions. >>:pr:`21219`<<
  by >>:user:`Olivier Grisel <ogrisel>`<<.

or "sample_weight") and if an unexpected *NaN* value is found in

  message when setting invalid hyper-parameters with *set_params*.
  >>:pr:`21542`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   Removes random unique identifiers in the HTML representation. With
  this change, jupyter notebooks are reproducible as long as the cells
  are run in the same order. >>:pr:`23098`<< by Thomas Fan.

*   Estimators with *non_deterministic* tag set to *True* will skip
  both *check_methods_sample_order_invariance* and
  *check_methods_subset_invariance* tests. >>:pr:`22318`<< by
  >>:user:`Zhehao Liu <MaxwellLZH>`<<.

*   The option for using the log loss, aka binomial or multinomial
  deviance, via the *loss* parameters was made more consistent. The
  preferred way is by setting the value to *"log_loss"*. Old option
  names are still valid and produce the same models, but are
  deprecated and will be removed in version 1.3.

  * For "ensemble.GradientBoostingClassifier", the *loss* parameter
    name "deviance" is deprecated in favor of the new name "log_loss",
    which is now the default. >>:pr:`23036`<< by >>:user:`Christian
    Lorentzen <lorentzenchr>`<<.

  * For "ensemble.HistGradientBoostingClassifier", the *loss*
    parameter names "auto", "binary_crossentropy" and

Lorentzen <lorentzenchr>`<<.

    "categorical_crossentropy" are deprecated in favor of the new name
    "log_loss", which is now the default. >>:pr:`23040`<< by
    >>:user:`Christian Lorentzen <lorentzenchr>`<<.

  * For "linear_model.SGDClassifier", the *loss* parameter name "log"
    is deprecated in favor of the new name "log_loss". >>:pr:`23046`<<
    by >>:user:`Christian Lorentzen <lorentzenchr>`<<.

*   Rich html representation of estimators is now enabled by default
  in Jupyter notebooks. It can be deactivated by setting
  *display='text'* in "sklearn.set_config()". >>:pr:`22856`<< by

*   "calibration.calibration_curve()" accepts a parameter *pos_label*
  to specify the positive class label. >>:pr:`21032`<< by

*   "calibration.CalibratedClassifierCV.fit()" now supports passing
  *fit_params*, which are routed to the *base_estimator*.
  >>:pr:`18170`<< by >>:user:`Benjamin Bossan <BenjaminBossan>`<<.

*   "calibration.CalibrationDisplay" accepts a parameter *pos_label*
  to add this information to the plot. >>:pr:`21038`<< by

*   "calibration.calibration_curve()" handles bin edges more
  consistently now. >>:pr:`14975`<< by Andreas Müller and

  >>:pr:`22526`<< by >>:user:`Meekail Zain <micky774>`<<.

*   "calibration.calibration_curve()"'s *normalize* parameter is now
  deprecated and will be removed in version 1.3. It is recommended
  that a proper probability (i.e. a classifier's *predict_proba*
  positive class) is used for *y_prob*. >>:pr:`23095`<< by
  >>:user:`Jordan Silke <jsilke>`<<.

*   "cluster.BisectingKMeans" introducing Bisecting K-Means algorithm
  >>:pr:`20031`<< by >>:user:`Michal Krawczyk <michalkrawczyk>`<<,
  >>:user:`Tom Dupre la Tour <TomDLT>`<< and >>:user:`Jérémie du

  now include the new *'cluster_qr'* method that clusters samples in
  the embedding space as an alternative to the existing *'kmeans'* and
  *'discrete'* methods. See "cluster.spectral_clustering()" for more
  details. >>:pr:`21148`<< by >>:user:`Andrew Knyazev <lobpcg>`<<.

*   Adds *get_feature_names_out* to "cluster.Birch",
  "cluster.FeatureAgglomeration", "cluster.KMeans",
  "cluster.MiniBatchKMeans". >>:pr:`22255`<< by Thomas Fan.

*   "cluster.SpectralClustering" now raises consistent error messages
  when passed invalid values for *n_clusters*, *n_init*, *gamma*,
  *n_neighbors*, *eigen_tol* or *degree*. >>:pr:`21881`<< by

  >>:user:`Hugo Vassard <hvassard>`<<.

*   "cluster.AffinityPropagation" now returns cluster centers and
  labels if they exist, even if the model has not fully converged.
  When returning these potentially-degenerate cluster centers and
  labels, a new warning message is shown. If no cluster centers were
  constructed, then the cluster centers remain an empty list with
  labels set to *-1* and the original warning message is shown.
  >>:pr:`22217`<< by >>:user:`Meekail Zain <micky774>`<<.

*   In "cluster.KMeans", the default "algorithm" is now ""lloyd""
  which is the full classical EM-style algorithm. Both ""auto"" and
  ""full"" are deprecated and will be removed in version 1.3. They are
  now aliases for ""lloyd"". The previous default was ""auto"", which
  relied on Elkan's algorithm. Lloyd's algorithm uses less memory than
  Elkan's, it is faster on many datasets, and its results are
  identical, hence the change. >>:pr:`21735`<< by >>:user:`Aurélien
  Geron <ageron>`<<.

*   "cluster.KMeans"'s *init* parameter now properly supports array-
  like input and NumPy string scalars. >>:pr:`22154`<< by Thomas Fan.

*   "compose.ColumnTransformer" now removes validation errors from
  *__init__* and *set_params* methods. >>:pr:`22537`<< by

  >>:user:`iofall <iofall>`<< and >>:user:`Arisa Y. <arisayosh>`<<.

*   *get_feature_names_out* functionality in
  "compose.ColumnTransformer" was broken when columns were specified
  using *slice*. This is fixed in >>:pr:`22775`<< and >>:pr:`22913`<<
  by >>:user:`randomgeek78 <randomgeek78>`<<.

*   "covariance.GraphicalLassoCV" now accepts NumPy array for the
  parameter *alphas*. >>:pr:`22493`<< by >>:user:`Guillaume Lemaitre

"sklearn.cross_decomposition"
-----------------------------

*   the *inverse_transform* method of
  "cross_decomposition.PLSRegression",
  "cross_decomposition.PLSCanonical" and "cross_decomposition.CCA" now
  allows reconstruction of a *X* target when a *Y* parameter is given.
  >>:pr:`19680`<< by >>:user:`Robin Thibaut <robinthibaut>`<<.

*   Adds *get_feature_names_out* to all transformers in the
  "cross_decomposition" module: "cross_decomposition.CCA",
  "cross_decomposition.PLSSVD", "cross_decomposition.PLSRegression",
  and "cross_decomposition.PLSCanonical". >>:pr:`22119`<< by Thomas
  Fan.

*   The shape of the *coef_* attribute of "cross_decomposition.CCA",
  "cross_decomposition.PLSCanonical" and
  "cross_decomposition.PLSRegression" will change in version 1.3, from
  *(n_features, n_targets)* to *(n_targets, n_features)*, to be

"cross_decomposition.PLSCanonical" and
  consistent with other linear models and to make it work with
  interface expecting a specific shape for *coef_* (e.g.
  "feature_selection.RFE"). >>:pr:`22016`<< by >>:user:`Guillaume

*   add the fitted attribute *intercept_* to
  "cross_decomposition.PLSCanonical",
  "cross_decomposition.PLSRegression", and "cross_decomposition.CCA".
  The method *predict* is indeed equivalent to *Y = X @ coef_ +
  intercept_*. >>:pr:`22015`<< by >>:user:`Guillaume Lemaitre

*   "datasets.load_files()" now accepts a ignore list and an allow
  list based on file extensions. >>:pr:`19747`<< by >>:user:`Tony
  Attalla <tonyattalla>`<< and >>:pr:`22498`<< by >>:user:`Meekail

*   "datasets.make_swiss_roll()" now supports the optional argument
  hole; when set to True, it returns the swiss-hole dataset.
  >>:pr:`21482`<< by >>:user:`Sebastian Pujalte <pujaltes>`<<.

*   "datasets.make_blobs()" no longer copies data during the
  generation process, therefore uses less memory. >>:pr:`22412`<< by

*   "datasets.load_diabetes()" now accepts the parameter "scaled", to
  allow loading unscaled data. The scaled version of this dataset is
  now computed from the unscaled data, and can produce slightly

  different results that in previous version (within a 1e-4 absolute
  tolerance). >>:pr:`16605`<< by >>:user:`Mandy Gu
  <happilyeverafter95>`<<.

*   "datasets.fetch_openml()" now has two optional arguments
  *n_retries* and *delay*. By default, "datasets.fetch_openml()" will
  retry 3 times in case of a network failure with a delay between each
  try. >>:pr:`21901`<< by >>:user:`Rileran <rileran>`<<.

*   "datasets.fetch_covtype()" is now concurrent-safe: data is
  downloaded to a temporary directory before being moved to the data
  directory. >>:pr:`23113`<< by >>:user:`Ilion Beyst <iasoon>`<<.

*   "datasets.make_sparse_coded_signal()" now accepts a parameter
  *data_transposed* to explicitly specify the shape of matrix *X*. The
  default behavior *True* is to return a transposed matrix *X*
  corresponding to a *(n_features, n_samples)* shape. The default
  value will change to *False* in version 1.3. >>:pr:`21425`<< by
  >>:user:`Gabriel Stefanini Vicente <g4brielvs>`<<.

*   Added a new estimator "decomposition.MiniBatchNMF". It is a faster
  but less accurate version of non-negative matrix factorization,
  better suited for large datasets. >>:pr:`16948`<< by >>:user:`Chiara
  Marmo <cmarmo>`<<, >>:user:`Patricio Cerda <pcerda>`<< and

but less accurate version of non-negative matrix factorization,

*   "decomposition.dict_learning()",
  "decomposition.dict_learning_online()" and
  "decomposition.sparse_encode()" preserve dtype for *numpy.float32*.
  "decomposition.DictionaryLearning",
  "decomposition.MiniBatchDictionaryLearning" and
  "decomposition.SparseCoder" preserve dtype for *numpy.float32*.
  >>:pr:`22002`<< by >>:user:`Takeshi Oura <takoika>`<<.

*   "decomposition.PCA" exposes a parameter *n_oversamples* to tune
  "utils.randomized_svd()" and get accurate results when the number of
  features is large. >>:pr:`21109`<< by >>:user:`Smile <x-shadow-
  man>`<<.

*   The "decomposition.MiniBatchDictionaryLearning" and
  "decomposition.dict_learning_online()" have been refactored and now
  have a stopping criterion based on a small change of the dictionary
  or objective function, controlled by the new *max_iter*, *tol* and
  *max_no_improvement* parameters. In addition, some of their
  parameters and attributes are deprecated.

  * the *n_iter* parameter of both is deprecated. Use *max_iter*
    instead.

  * the *iter_offset*, *return_inner_stats*, *inner_stats* and
    *return_n_iter* parameters of
    "decomposition.dict_learning_online()" serve internal purpose and
    are deprecated.

instead.

  * the *inner_stats_*, *iter_offset_* and *random_state_* attributes
    of "decomposition.MiniBatchDictionaryLearning" serve internal
    purpose and are deprecated.

  * the default value of the *batch_size* parameter of both will
    change from 3 to 256 in version 1.3.

  >>:pr:`18975`<< by >>:user:`Jérémie du Boisberranger

  preserve dtype for *numpy.float32*. >>:pr:`22111`<< by
  >>:user:`Takeshi Oura <takoika>`<<.

*   "decomposition.TruncatedSVD" now allows *n_components ==
  n_features*, if *algorithm='randomized'*. >>:pr:`22181`<< by
  >>:user:`Zach Deane-Mayer <zachmayer>`<<.

  "decomposition" module: "decomposition.DictionaryLearning",
  "decomposition.FactorAnalysis", "decomposition.FastICA",
  "decomposition.IncrementalPCA", "decomposition.KernelPCA",
  "decomposition.LatentDirichletAllocation",
  "decomposition.MiniBatchDictionaryLearning",
  "decomposition.MiniBatchSparsePCA", "decomposition.NMF",
  "decomposition.PCA", "decomposition.SparsePCA", and
  "decomposition.TruncatedSVD". >>:pr:`21334`<< by Thomas Fan.

*   "decomposition.TruncatedSVD" exposes the parameter *n_oversamples*

"decomposition.PCA", "decomposition.SparsePCA", and

  and *power_iteration_normalizer* to tune "utils.randomized_svd()"
  and get accurate results when the number of features is large, the
  rank of the matrix is high, or other features of the matrix make low
  rank approximation difficult. >>:pr:`21705`<< by >>:user:`Jay S.
  Stanley III <stanleyjs>`<<.

*   "decomposition.PCA" exposes the parameter
  *power_iteration_normalizer* to tune "utils.randomized_svd()" and
  get more accurate results when low rank approximation is difficult.
  >>:pr:`21705`<< by >>:user:`Jay S. Stanley III <stanleyjs>`<<.

*   "decomposition.FastICA" now validates input parameters in *fit*
  instead of *__init__*. >>:pr:`21432`<< by >>:user:`Hannah Bohle
  <hhnnhh>`<< and >>:user:`Maren Westermann <marenwestermann>`<<.

*   "decomposition.FastICA" now accepts *np.float32* data without
  silent upcasting. The dtype is preserved by *fit* and
  *fit_transform* and the main fitted attributes use a dtype of the
  same precision as the training data. >>:pr:`22806`<< by
  >>:user:`Jihane Bennis <JihaneBennis>`<< and >>:user:`Olivier Grisel

*   "decomposition.FactorAnalysis" now validates input parameters in
  *fit* instead of *__init__*. >>:pr:`21713`<< by >>:user:`Haya
  <HayaAlmutairi>`<< and >>:user:`Krum Arnaudov <krumeto>`<<.

*   "decomposition.KernelPCA" now validates input parameters in *fit*
  instead of *__init__*. >>:pr:`21567`<< by >>:user:`Maggie Chege
  <MaggieChege>`<<.

*   "decomposition.PCA" and "decomposition.IncrementalPCA" more safely
  calculate precision using the inverse of the covariance matrix if
  *self.noise_variance_* is zero. >>:pr:`22300`<< by >>:user:`Meekail
  Zain <micky774>`<< and >>:pr:`15948`<< by >>:user:`sysuresh`<<.

*   Greatly reduced peak memory usage in "decomposition.PCA" when
  calling *fit* or *fit_transform*. >>:pr:`22553`<< by

*   "decomposition.FastICA()" now supports unit variance for
  whitening. The default value of its *whiten* argument will change
  from *True* (which behaves like *'arbitrary-variance'*) to *'unit-
  variance'* in version 1.3. >>:pr:`19490`<< by >>:user:`Facundo
  Ferrin <fferrin>`<< and >>:user:`Julien Jerphanion <jjerphan>`<<.

*   Adds *get_feature_names_out* to
  "discriminant_analysis.LinearDiscriminantAnalysis". >>:pr:`22120`<<
  by Thomas Fan.

  model behavior. >>:pr:`15984`<< by >>:user:`Okon Samuel

  <OkonSamuel>`<< and >>:pr:`22696`<< by >>:user:`Meekail Zain

*   "dummy.DummyRegressor" no longer overrides the *constant*
  parameter during *fit*. >>:pr:`22486`<< by Thomas Fan.

*   Added additional option *loss="quantile"* to
  "ensemble.HistGradientBoostingRegressor" for modelling quantiles.
  The quantile level can be specified with the new parameter
  *quantile*. >>:pr:`21800`<< and >>:pr:`20567`<< by

*   "fit()" of "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor" now calls "utils.check_array()"
  with parameter *force_all_finite=False* for non initial warm-start
  runs as it has already been checked before. >>:pr:`22159`<< by
  >>:user:`Geoffrey Paris <Geoffrey-Paris>`<<.

*   "ensemble.HistGradientBoostingClassifier" is faster, for binary
  and in particular for multiclass problems thanks to the new private
  loss function module. >>:pr:`20811`<<, >>:pr:`20567`<< and
  >>:pr:`21814`<< by >>:user:`Christian Lorentzen <lorentzenchr>`<<.

*   Adds support to use pre-fit models with *cv="prefit"* in
  "ensemble.StackingClassifier" and "ensemble.StackingRegressor".
  >>:pr:`16748`<< by >>:user:`Siqi He <siqi-he>`<< and >>:pr:`22215`<<

  by >>:user:`Meekail Zain <micky774>`<<.

*   "ensemble.RandomForestClassifier" and
  "ensemble.ExtraTreesClassifier" have the new *criterion="log_loss"*,
  which is equivalent to *criterion="entropy"*. >>:pr:`23047`<< by

*   Adds *get_feature_names_out* to "ensemble.VotingClassifier",
  "ensemble.VotingRegressor", "ensemble.StackingClassifier", and
  "ensemble.StackingRegressor". >>:pr:`22695`<< and >>:pr:`22697`<<

*   "ensemble.RandomTreesEmbedding" now has an informative
  *get_feature_names_out* function that includes both tree index and
  leaf index in the output feature names. >>:pr:`21762`<< by
  >>:user:`Zhehao Liu <MaxwellLZH>`<< and Thomas Fan.

*   Fitting a "ensemble.RandomForestClassifier",
  "ensemble.ExtraTreesRegressor", and "ensemble.RandomTreesEmbedding"
  is now faster in a multiprocessing setting, especially for
  subsequent fits with *warm_start* enabled. >>:pr:`22106`<< by
  >>:user:`Pieter Gijsbers <PGijsbers>`<<.

*   Change the parameter *validation_fraction* in
  "ensemble.GradientBoostingRegressor" so that an error is raised if
  anything other than a float is passed in as an argument.

"ensemble.GradientBoostingClassifier" and
  >>:pr:`21632`<< by >>:user:`Genesis Valencia <genvalen>`<<.

*   Removed a potential source of CPU oversubscription in
  "ensemble.HistGradientBoostingRegressor" when CPU resource usage is
  limited, for instance using cgroups quota in a docker container.
  >>:pr:`22566`<< by >>:user:`Jérémie du Boisberranger

  "ensemble.HistGradientBoostingRegressor" no longer warns when
  fitting on a pandas DataFrame with a non-default *scoring* parameter
  and early_stopping enabled. >>:pr:`22908`<< by Thomas Fan.

*   Fixes HTML repr for "ensemble.StackingClassifier" and
  "ensemble.StackingRegressor". >>:pr:`23097`<< by Thomas Fan.

*   The attribute *loss_* of "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor" has been deprecated and will be
  removed in version 1.3. >>:pr:`23079`<< by >>:user:`Christian

*   Changed the default of *max_features* to 1.0 for
  "ensemble.RandomForestRegressor" and to *"sqrt"* for
  "ensemble.RandomForestClassifier". Note that these give the same fit
  results as before, but are much easier to understand. The old
  default value *"auto"* has been deprecated and will be removed in
  version 1.3. The same changes are also applied for

results as before, but are much easier to understand. The old
  "ensemble.ExtraTreesRegressor" and "ensemble.ExtraTreesClassifier".
  >>:pr:`20803`<< by >>:user:`Brian Sun <bsun94>`<<.

  skipping repetitive input checks. >>:pr:`23149`<< by >>:user:`Zhehao
  Liu <MaxwellLZH>`<<.

*   "feature_extraction.FeatureHasher" now supports PyPy.
  >>:pr:`23023`<< by Thomas Fan.

*   "feature_extraction.FeatureHasher" now validates input parameters
  in *transform* instead of *__init__*. >>:pr:`21573`<< by
  >>:user:`Hannah Bohle <hhnnhh>`<< and >>:user:`Maren Westermann
  <marenwestermann>`<<.

*   "feature_extraction.text.TfidfVectorizer" now does not create a
  "feature_extraction.text.TfidfTransformer" at *__init__* as required
  by our API. >>:pr:`21832`<< by >>:user:`Guillaume Lemaitre

*   Added auto mode to "feature_selection.SequentialFeatureSelector".
  If the argument *n_features_to_select* is *'auto'*, select features
  until the score improvement does not exceed the argument *tol*. The
  default value of *n_features_to_select* changed from *None* to
  *'warn'* in 1.1 and will become *'auto'* in 1.3. *None* and *'warn'*
  will be removed in 1.3. >>:pr:`20145`<< by >>:user:`murata-yu

default value of *n_features_to_select* changed from *None* to
  <murata-yu>`<<.

*   Added the ability to pass callables to the *max_features*
  parameter of "feature_selection.SelectFromModel". Also introduced
  new attribute *max_features_* which is inferred from *max_features*
  and the data during *fit*. If *max_features* is an integer, then
  *max_features_ = max_features*. If *max_features* is a callable,
  then *max_features_ = max_features(X)*. >>:pr:`22356`<< by

*   "feature_selection.GenericUnivariateSelect" preserves float32
  dtype. >>:pr:`18482`<< by >>:user:`Thierry Gameiro <titigmr>`<< and
  >>:user:`Daniel Kharsa <aflatoune>`<< and >>:pr:`22370`<< by

*   Add a parameter *force_finite* to
  "feature_selection.f_regression()" and
  "feature_selection.r_regression()". This parameter allows to force
  the output to be finite in the case where a feature or a the target
  is constant or that the feature and target are perfectly correlated
  (only for the F-statistic). >>:pr:`17819`<< by >>:user:`Juan Carlos
  Alfaro Jiménez <alfaro96>`<<.

*   Improve runtime performance of "feature_selection.chi2()" with
  boolean arrays. >>:pr:`22235`<< by Thomas Fan.

*   Reduced memory usage of "feature_selection.chi2()".
  >>:pr:`21837`<< by >>:user:`Louis Wagner <lrwagner>`<<.

boolean arrays. >>:pr:`22235`<< by Thomas Fan.

*   *predict* and *sample_y* methods of
  "gaussian_process.GaussianProcessRegressor" now return arrays of the
  correct shape in single-target and multi-target cases, and for both
  *normalize_y=False* and *normalize_y=True*. >>:pr:`22199`<< by
  >>:user:`Guillaume Lemaitre <glemaitre>`<<, >>:user:`Aidar
  Shakerimoff <AidarShakerimoff>`<< and >>:user:`Tenavi Nakamura-
  Zimmerer <Tenavi>`<<.

*   "gaussian_process.GaussianProcessClassifier" raises a more
  informative error if *CompoundKernel* is passed via *kernel*.
  >>:pr:`22223`<< by >>:user:`MarcoM <marcozzxx810>`<<.

*   "impute.SimpleImputer" now warns with feature names when features
  which are skipped due to the lack of any observed values in the
  training set. >>:pr:`21617`<< by >>:user:`Christian Ritter
  <chritter>`<<.

*   Added support for *pd.NA* in "impute.SimpleImputer".
  >>:pr:`21114`<< by >>:user:`Ying Xiong <yxiong>`<<.

*   Adds *get_feature_names_out* to "impute.SimpleImputer",
  "impute.KNNImputer", "impute.IterativeImputer", and
  "impute.MissingIndicator". >>:pr:`21078`<< by Thomas Fan.

*   The *verbose* parameter was deprecated for "impute.SimpleImputer".
  A warning will always be raised upon the removal of empty columns.

"impute.MissingIndicator". >>:pr:`21078`<< by Thomas Fan.

  >>:pr:`21448`<< by >>:user:`Oleh Kozynets <OlehKSS>`<< and
  >>:user:`Christian Ritter <chritter>`<<.

*   Add a display to plot the boundary decision of a classifier by
  using the method
  "inspection.DecisionBoundaryDisplay.from_estimator()".
  >>:pr:`16061`<< by Thomas Fan.

*   In "inspection.PartialDependenceDisplay.from_estimator()", allow
  *kind* to accept a list of strings to specify  which type of plot to
  draw for each feature interaction. >>:pr:`19438`<< by

*   "inspection.PartialDependenceDisplay.from_estimator()",
  "inspection.PartialDependenceDisplay.plot()", and
  "inspection.plot_partial_dependence()" now support plotting centered
  Individual Conditional Expectation (cICE) and centered PDP curves
  controlled by setting the parameter *centered*. >>:pr:`18310`<< by
  >>:user:`Johannes Elfner <JoElfner>`<< and >>:user:`Guillaume

*   Adds *get_feature_names_out* to "isotonic.IsotonicRegression".
  >>:pr:`22249`<< by Thomas Fan.

  "kernel_approximation.AdditiveChi2Sampler".
  "kernel_approximation.Nystroem",

  "kernel_approximation.PolynomialCountSketch",
  "kernel_approximation.RBFSampler", and
  "kernel_approximation.SkewedChi2Sampler". >>:pr:`22137`<< and
  >>:pr:`22694`<< by Thomas Fan.

*   "linear_model.ElasticNet", "linear_model.ElasticNetCV",
  "linear_model.Lasso" and "linear_model.LassoCV" support
  *sample_weight* for sparse input *X*. >>:pr:`22808`<< by

*   "linear_model.Ridge" with *solver="lsqr"* now supports to fit
  sparse input with *fit_intercept=True*. >>:pr:`22950`<< by

*   "linear_model.QuantileRegressor" support sparse input for the
  highs based solvers. >>:pr:`21086`<< by >>:user:`Venkatachalam
  Natchiappan <venkyyuvy>`<<. In addition, those solvers now use the
  CSC matrix right from the beginning which speeds up fitting.
  >>:pr:`22206`<< by >>:user:`Christian Lorentzen <lorentzenchr>`<<.

*   "linear_model.LogisticRegression" is faster for "solvers="lbfgs""
  and "solver="newton-cg"", for binary and in particular for
  multiclass problems thanks to the new private loss function module.
  In the multiclass case, the memory consumption has also been reduced
  for these solvers as the target is now label encoded (mapped to

In the multiclass case, the memory consumption has also been reduced
  integers) instead of label binarized (one-hot encoded). The more
  classes, the larger the benefit. >>:pr:`21808`<<, >>:pr:`20567`<<
  and >>:pr:`21814`<< by >>:user:`Christian Lorentzen

  "linear_model.TweedieRegressor" are faster for "solvers="lbfgs"".
  >>:pr:`22548`<<, >>:pr:`21808`<< and >>:pr:`20567`<< by

*   Rename parameter *base_estimator* to *estimator* in
  "linear_model.RANSACRegressor" to improve readability and
  consistency. *base_estimator* is deprecated and will be removed in
  1.3. >>:pr:`22062`<< by >>:user:`Adrian Trujillo <trujillo9616>`<<.

*   "linear_model.ElasticNet()" and and other linear model classes
  using coordinate descent show error messages when non-finite
  parameter weights are produced. >>:pr:`22148`<< by
  >>:user:`Christian Ritter <chritter>`<< and >>:user:`Norbert
  Preining <norbusan>`<<.

*   "linear_model.ElasticNet" and "linear_model.Lasso" now raise
  consistent error messages when passed invalid values for *l1_ratio*,
  *alpha*, *max_iter* and *tol*. >>:pr:`22240`<< by >>:user:`Arturo
  Amor <ArturoAmorQ>`<<.

*   "linear_model.BayesianRidge" and "linear_model.ARDRegression" now
  preserve float32 dtype. >>:pr:`9087`<< by >>:user:`Arthur Imbert

Amor <ArturoAmorQ>`<<.

  <Henley13>`<< and >>:pr:`22525`<< by >>:user:`Meekail Zain

*   "linear_model.RidgeClassifier" is now supporting multilabel
  classification. >>:pr:`19689`<< by >>:user:`Guillaume Lemaitre

*   "linear_model.RidgeCV" and "linear_model.RidgeClassifierCV" now
  raise consistent error message when passed invalid values for
  *alphas*. >>:pr:`21606`<< by >>:user:`Arturo Amor <ArturoAmorQ>`<<.

*   "linear_model.Ridge" and "linear_model.RidgeClassifier" now raise
  consistent error message when passed invalid values for *alpha*,
  *max_iter* and *tol*. >>:pr:`21341`<< by >>:user:`Arturo Amor

*   "linear_model.orthogonal_mp_gram()" preservse dtype for
  *numpy.float32*. >>:pr:`22002`<< by >>:user:`Takeshi Oura
  <takoika>`<<.

*   "linear_model.LassoLarsIC" now correctly computes AIC and BIC. An
  error is now raised when *n_features > n_samples* and when the noise
  variance is not provided. >>:pr:`21481`<< by >>:user:`Guillaume
  Lemaitre <glemaitre>`<< and >>:user:`Andrés Babino <ababino>`<<.

*   "linear_model.TheilSenRegressor" now validates input parameter
  "max_subpopulation" in *fit* instead of *__init__*. >>:pr:`21767`<<
  by >>:user:`Maren Westermann <marenwestermann>`<<.

*   "linear_model.ElasticNetCV" now produces correct warning when

"max_subpopulation" in *fit* instead of *__init__*. >>:pr:`21767`<<

  *l1_ratio=0*. >>:pr:`21724`<< by >>:user:`Yar Khine Phyo
  <yarkhinephyo>`<<.

  "linear_model.LogisticRegressionCV" now set the *n_iter_* attribute
  with a shape that respects the docstring and that is consistent with
  the shape obtained when using the other solvers in the one-vs-rest
  setting. Previously, it would record only the maximum of the number
  of iterations for each binary sub-problem while now all of them are
  recorded. >>:pr:`21998`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   The property *family* of "linear_model.TweedieRegressor" is not
  validated in *__init__* anymore. Instead, this (private) property is
  deprecated in "linear_model.GammaRegressor",
  "linear_model.PoissonRegressor" and "linear_model.TweedieRegressor",
  and will be removed in 1.3. >>:pr:`22548`<< by >>:user:`Christian

*   The *coef_* and *intercept_* attributes of
  "linear_model.LinearRegression" are now correctly computed in the
  presence of sample weights when the input is sparse. >>:pr:`22891`<<
  by >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<.

*   The *coef_* and *intercept_* attributes of "linear_model.Ridge"
  with *solver="sparse_cg"* and *solver="lbfgs"* are now correctly

by >>:user:`Jérémie du Boisberranger <jeremiedbb>`<<.

  computed in the presence of sample weights when the input is sparse.
  >>:pr:`22899`<< by >>:user:`Jérémie du Boisberranger

*   "linear_model.SGDRegressor" and "linear_model.SGDClassifier" now
  computes the validation error correctly when early stopping is
  enabled. >>:pr:`23256`<< by >>:user:`Zhehao Liu <MaxwellLZH>`<<.

*   "linear_model.LassoLarsIC" now exposes *noise_variance* as a
  parameter in order to provide an estimate of the noise variance.
  This is particularly relevant when *n_features > n_samples* and the
  estimator of the noise variance cannot be computed. >>:pr:`21481`<<

*   "manifold.Isomap" now supports radius-based neighbors via the
  *radius* argument. >>:pr:`19794`<< by >>:user:`Zhehao Liu

*   "manifold.spectral_embedding()" and "manifold.SpectralEmbedding"
  supports *np.float32* dtype and will preserve this dtype.
  >>:pr:`21534`<< by >>:user:`Andrew Knyazev <lobpcg>`<<.

*   Adds *get_feature_names_out* to "manifold.Isomap" and
  "manifold.LocallyLinearEmbedding". >>:pr:`22254`<< by Thomas Fan.

*   added *metric_params* to "manifold.TSNE" constructor for
  additional parameters of distance metric to use in optimization.

"manifold.LocallyLinearEmbedding". >>:pr:`22254`<< by Thomas Fan.

  >>:pr:`21805`<< by >>:user:`Jeanne Dionisi <jeannedionisi>`<< and
  >>:pr:`22685`<< by >>:user:`Meekail Zain <micky774>`<<.

*   "manifold.trustworthiness()" raises an error if *n_neighbours >=
  n_samples / 2* to ensure a correct support for the function.
  >>:pr:`18832`<< by >>:user:`Hong Shao Yang <hongshaoyang>`<< and
  >>:pr:`23033`<< by >>:user:`Meekail Zain <micky774>`<<.

*   "manifold.spectral_embedding()" now uses Gaussian instead of the
  previous uniform on [0, 1] random initial approximations to
  eigenvectors in eigen_solvers *lobpcg* and *amg* to improve their
  numerical stability. >>:pr:`21565`<< by >>:user:`Andrew Knyazev
  <lobpcg>`<<.

*   "metrics.r2_score()" and "metrics.explained_variance_score()" have
  a new *force_finite* parameter. Setting this parameter to *False*
  will return the actual non-finite score in case of perfect
  predictions or constant *y_true*, instead of the finite
  approximation (*1.0* and *0.0* respectively) currently returned by
  default. >>:pr:`17266`<< by >>:user:`Sylvain Marié <smarie>`<<.

*   "metrics.d2_pinball_score()" and
  "metrics.d2_absolute_error_score()" calculate the D^2 regression
  score for the pinball loss and the absolute error respectively.
  "metrics.d2_absolute_error_score()" is a special case of

"metrics.d2_absolute_error_score()" calculate the D^2 regression
  "metrics.d2_pinball_score()" with a fixed quantile parameter
  *alpha=0.5* for ease of use and discovery. The D^2 scores are
  generalizations of the *r2_score* and can be interpeted as the
  fraction of deviance explained. >>:pr:`22118`<< by >>:user:`Ohad
  Michel <ohadmich>`<<.

*   "metrics.top_k_accuracy_score()" raises an improved error message
  when *y_true* is binary and *y_score* is 2d. >>:pr:`22284`<< by

*   "metrics.roc_auc_score()" now supports "average=None" in the
  multiclass case when "multiclass='ovr'" which will return the score
  per class. >>:pr:`19158`<< by >>:user:`Nicki Skafte
  <SkafteNicki>`<<.

*   Adds *im_kw* parameter to
  "metrics.ConfusionMatrixDisplay.from_estimator()"
  "metrics.ConfusionMatrixDisplay.plot()". The *im_kw* parameter is
  passed to the *matplotlib.pyplot.imshow* call when plotting the
  confusion matrix. >>:pr:`20753`<< by Thomas Fan.

*   "metrics.silhouette_score()" now supports integer input for
  precomputed distances. >>:pr:`22108`<< by Thomas Fan.

*   Fixed a bug in "metrics.normalized_mutual_info_score()" which
  could return unbounded values. >>:pr:`22635`<< by >>:user:`Jérémie

*   Fixes "metrics.precision_recall_curve()" and

could return unbounded values. >>:pr:`22635`<< by >>:user:`Jérémie

  "metrics.average_precision_score()" when true labels are all
  negative. >>:pr:`19085`<< by >>:user:`Varun Agrawal
  <varunagrawal>`<<.

*   *metrics.SCORERS* is now deprecated and will be removed in 1.3.
  Please use "metrics.get_scorer_names()" to retrieve the names of all
  available scorers. >>:pr:`22866`<< by Adrin Jalali.

*   Parameters "sample_weight" and "multioutput" of
  "metrics.mean_absolute_percentage_error()" are now keyword-only, in
  accordance with SLEP009. A deprecation cycle was introduced.
  >>:pr:`21576`<< by >>:user:`Paul-Emile Dugnat <pedugnat>`<<.

*   The *"wminkowski"* metric of "metrics.DistanceMetric" is
  deprecated and will be removed in version 1.3. Instead the existing
  *"minkowski"* metric now takes in an optional *w* parameter for
  weights. This deprecation aims at remaining consistent with SciPy
  1.8 convention. >>:pr:`21873`<< by >>:user:`Yar Khine Phyo

*   "metrics.DistanceMetric" has been moved from "sklearn.neighbors"
  to "sklearn.metrics". Using *neighbors.DistanceMetric* for imports
  is still valid for backward compatibility, but this alias will be
  removed in 1.3. >>:pr:`21177`<< by >>:user:`Julien Jerphanion

*   "mixture.GaussianMixture" and "mixture.BayesianGaussianMixture"

removed in 1.3. >>:pr:`21177`<< by >>:user:`Julien Jerphanion

  can now be initialized using k-means++ and random data points.
  >>:pr:`20408`<< by >>:user:`Gordon Walsh <g-walsh>`<<,
  >>:user:`Alberto Ceballos<alceballosa>`<< and >>:user:`Andres
  Rios<ariosramirez>`<<.

*   Fix a bug that correctly initialize *precisions_cholesky_* in
  "mixture.GaussianMixture" when providing *precisions_init* by taking
  its square root. >>:pr:`22058`<< by >>:user:`Guillaume Lemaitre

*   "mixture.GaussianMixture" now normalizes *weights_* more safely,
  preventing rounding errors when calling
  "mixture.GaussianMixture.sample()" with *n_components=1*.
  >>:pr:`23034`<< by >>:user:`Meekail Zain <micky774>`<<.

*   it is now possible to pass *scoring="matthews_corrcoef"* to all
  model selection tools with a *scoring* argument to use the Matthews
  correlation coefficient (MCC). >>:pr:`22203`<< by >>:user:`Olivier
  Grisel <ogrisel>`<<.

*   raise an error during cross-validation when the fits for all the
  splits failed. Similarly raise an error during grid-search when the
  fits for all the models and all the splits failed. >>:pr:`21026`<<
  by >>:user:`Loïc Estève <lesteve>`<<.

*   "model_selection.GridSearchCV",
  "model_selection.HalvingGridSearchCV" now validate input parameters

by >>:user:`Loïc Estève <lesteve>`<<.

  in *fit* instead of *__init__*. >>:pr:`21880`<< by >>:user:`Mrinal
  Tyagi <MrinalTyagi>`<<.

*   "model_selection.learning_curve()" now supports *partial_fit* with
  regressors. >>:pr:`22982`<< by Thomas Fan.

*   "multiclass.OneVsRestClassifier" now supports a *verbose*
  parameter so progress on fitting can be seen. >>:pr:`22508`<< by
  >>:user:`Chris Combs <combscCode>`<<.

*   "multiclass.OneVsOneClassifier.predict()" returns correct
  predictions when the inner classifier only has a *predict_proba*.
  >>:pr:`22604`<< by Thomas Fan.

  "neighbors.RadiusNeighborsTransformer",
  "neighbors.NeighborhoodComponentsAnalysis". >>:pr:`22212`<< by

*   "neighbors.KernelDensity" now validates input parameters in *fit*
  instead of *__init__*. >>:pr:`21430`<< by >>:user:`Desislava
  Vasileva <DessyVV>`<< and >>:user:`Lucy Jimenez <LucyJimenez>`<<.

*   "neighbors.KNeighborsRegressor.predict()" now works properly when
  given an array-like input if *KNeighborsRegressor* is first
  constructed with a callable passed to the *weights* parameter.
  >>:pr:`22687`<< by >>:user:`Meekail Zain <micky774>`<<.

constructed with a callable passed to the *weights* parameter.

*   "neural_network.MLPClassifier()" and
  "neural_network.MLPRegressor()" show error messages when optimizers
  produce non-finite parameter weights. >>:pr:`22150`<< by

*   Adds *get_feature_names_out* to "neural_network.BernoulliRBM".
  >>:pr:`22248`<< by Thomas Fan.

*   Added support for "passthrough" in "pipeline.FeatureUnion".
  Setting a transformer to "passthrough" will pass the features
  unchanged. >>:pr:`20860`<< by >>:user:`Shubhraneel Pal
  <shubhraneel>`<<.

*   "pipeline.Pipeline" now does not validate hyper-parameters in
  *__init__* but in *.fit()*. >>:pr:`21888`<< by >>:user:`iofall
  <iofall>`<< and >>:user:`Arisa Y. <arisayosh>`<<.

*   "pipeline.FeatureUnion" does not validate hyper-parameters in
  *__init__*. Validation is now handled in *.fit()* and
  *.fit_transform()*. >>:pr:`21954`<< by >>:user:`iofall <iofall>`<<
  and >>:user:`Arisa Y. <arisayosh>`<<.

*   Defines *__sklearn_is_fitted__* in "pipeline.FeatureUnion" to
  return correct result with "utils.validation.check_is_fitted()".
  >>:pr:`22953`<< by >>:user:`randomgeek78 <randomgeek78>`<<.

*   "preprocessing.OneHotEncoder" now supports grouping infrequent

>>:pr:`22953`<< by >>:user:`randomgeek78 <randomgeek78>`<<.

  categories into a single feature. Grouping infrequent categories is
  enabled by specifying how to select infrequent categories with
  *min_frequency* or *max_categories*. >>:pr:`16018`<< by Thomas Fan.

*   Adds a *subsample* parameter to "preprocessing.KBinsDiscretizer".
  This allows specifying a maximum number of samples to be used while
  fitting the model. The option is only available when *strategy* is
  set to *quantile*. >>:pr:`21445`<< by >>:user:`Felipe Bidu
  <fbidu>`<< and >>:user:`Amanda Dsouza <amy12xx>`<<.

*   Adds *encoded_missing_value* to "preprocessing.OrdinalEncoder" to
  configure the encoded value for missing data. >>:pr:`21988`<< by

*   Added the *get_feature_names_out* method and a new parameter
  *feature_names_out* to "preprocessing.FunctionTransformer". You can
  set *feature_names_out* to 'one-to-one' to use the input features
  names as the output feature names, or you can set it to a callable
  that returns the output feature names. This is especially useful
  when the transformer changes the number of features. If
  *feature_names_out* is None (which is the default), then
  *get_output_feature_names* is not defined. >>:pr:`21569`<< by
  >>:user:`Aurélien Geron <ageron>`<<.

*   Adds *get_feature_names_out* to "preprocessing.Normalizer",

*get_output_feature_names* is not defined. >>:pr:`21569`<< by

  "preprocessing.KernelCenterer", "preprocessing.OrdinalEncoder", and
  "preprocessing.Binarizer". >>:pr:`21079`<< by Thomas Fan.

*   "preprocessing.PowerTransformer" with *method='yeo-johnson'*
  better supports significantly non-Gaussian data when searching for
  an optimal lambda. >>:pr:`20653`<< by Thomas Fan.

*   "preprocessing.LabelBinarizer" now validates input parameters in
  *fit* instead of *__init__*. >>:pr:`21434`<< by >>:user:`Krum
  Arnaudov <krumeto>`<<.

*   "preprocessing.FunctionTransformer" with *check_inverse=True* now
  provides informative error message when input has mixed dtypes.
  >>:pr:`19916`<< by >>:user:`Zhehao Liu <MaxwellLZH>`<<.

*   "preprocessing.KBinsDiscretizer" handles bin edges more

*   Adds "preprocessing.KBinsDiscretizer.get_feature_names_out()"
  support when *encode="ordinal"*. >>:pr:`22735`<< by Thomas Fan.

"sklearn.random_projection"

*   Adds an *inverse_transform* method and a
  *compute_inverse_transform* parameter to
  "random_projection.GaussianRandomProjection" and
  "random_projection.SparseRandomProjection". When the parameter is
  set to True, the pseudo-inverse of the components is computed during

"random_projection.GaussianRandomProjection" and
  *fit* and stored as *inverse_components_*. >>:pr:`21701`<< by

*   "random_projection.SparseRandomProjection" and
  "random_projection.GaussianRandomProjection" preserves dtype for
  *numpy.float32*. >>:pr:`22114`<< by >>:user:`Takeshi Oura

  "sklearn.random_projection" module:
  "random_projection.SparseRandomProjection". >>:pr:`21330`<< by

*   "svm.OneClassSVM", "svm.NuSVC", "svm.NuSVR", "svm.SVC" and
  "svm.SVR" now expose *n_iter_*, the number of iterations of the
  libsvm optimization routine. >>:pr:`21408`<< by >>:user:`Juan Martín
  Loyola <jmloyola>`<<.

*   "svm.SVR()", "svm.SVC()", "svm.NuSVR()", "svm.OneClassSVM()",
  "svm.NuSVC()" now raise an error when the dual-gap estimation
  produce non-finite parameter weights. >>:pr:`22149`<< by

*   "svm.NuSVC", "svm.NuSVR", "svm.SVC", "svm.SVR", "svm.OneClassSVM"
  now validate input parameters in *fit* instead of *__init__*.
  >>:pr:`21436`<< by >>:user:`Haidar Almubarak <Haidar13 >`<<.

now validate input parameters in *fit* instead of *__init__*.

*   "tree.DecisionTreeClassifier" and "tree.ExtraTreeClassifier" have
  the new *criterion="log_loss"*, which is equivalent to
  *criterion="entropy"*. >>:pr:`23047`<< by >>:user:`Christian

*   Fix a bug in the Poisson splitting criterion for
  "tree.DecisionTreeRegressor". >>:pr:`22191`<< by >>:user:`Christian

*   Changed the default value of *max_features* to 1.0 for
  "tree.ExtraTreeRegressor" and to *"sqrt"* for
  "tree.ExtraTreeClassifier", which will not change the fit result.
  The original default value *"auto"* has been deprecated and will be
  removed in version 1.3. Setting *max_features* to *"auto"* is also
  deprecated for "tree.DecisionTreeClassifier" and
  "tree.DecisionTreeRegressor". >>:pr:`22476`<< by >>:user:`Zhehao Liu

*   "utils.check_array()" and "utils.multiclass.type_of_target()" now
  accept an *input_name* parameter to make the error message more
  informative when passed invalid input data (e.g. with NaN or
  infinite values). >>:pr:`21219`<< by >>:user:`Olivier Grisel

*   "utils.check_array()" returns a float ndarray with *np.nan* when
  passed a *Float32* or *Float64* pandas extension array with *pd.NA*.
  >>:pr:`21278`<< by Thomas Fan.

<ogrisel>`<<.

*   "utils.estimator_html_repr()" shows a more helpful error message
  when running in a jupyter notebook that is not trusted.
  >>:pr:`21316`<< by Thomas Fan.

*   "utils.estimator_html_repr()" displays an arrow on the top left
  corner of the HTML representation to show how the elements are
  clickable. >>:pr:`21298`<< by Thomas Fan.

*   "utils.check_array()" with *dtype=None* returns numeric arrays
  when passed in a pandas DataFrame with mixed dtypes.
  *dtype="numeric"* will also make better infer the dtype when the
  DataFrame has mixed dtypes. >>:pr:`22237`<< by Thomas Fan.

*   "utils.check_scalar()" now has better messages when displaying the
  type. >>:pr:`22218`<< by Thomas Fan.

*   Changes the error message of the *ValidationError* raised by
  "utils.check_X_y()" when y is None so that it is compatible with the
  *check_requires_y_none* estimator check. >>:pr:`22578`<< by
  >>:user:`Claudio Salvatore Arcidiacono
  <ClaudioSalvatoreArcidiacono>`<<.

*   "utils.class_weight.compute_class_weight()" now only requires that
  all classes in *y* have a weight in *class_weight*. An error is
  still raised when a class is present in *y* but not in
  *class_weight*. >>:pr:`22595`<< by Thomas Fan.

*   "utils.estimator_html_repr()" has an improved visualization for

still raised when a class is present in *y* but not in

  nested meta-estimators. >>:pr:`21310`<< by Thomas Fan.

*   "utils.check_scalar()" raises an error when
  *include_boundaries={"left", "right"}* and the boundaries are not
  set. >>:pr:`22027`<< by >>:user:`Marie Lanternier <mlant>`<<.

*   "utils.metaestimators.available_if()" correctly returns a bounded
  method that can be pickled. >>:pr:`23077`<< by Thomas Fan.

*   "utils.estimator_checks.check_estimator()"'s argument is now
  called *estimator* (previous name was *Estimator*). >>:pr:`22188`<<
  by >>:user:`Mathurin Massias <mathurinm>`<<.

*   "utils.metaestimators.if_delegate_has_method()" is deprecated and
  will be removed in version 1.3. Use
  "utils.metaestimators.available_if()" instead. >>:pr:`22830`<< by

improvement of the project since version 1.0, including:

2357juan, Abhishek Gupta, adamgonzo, Adam Li, adijohar, Aditya
Kumawat, Aditya Raghuwanshi, Aditya Singh, Adrian Trujillo Duron,
Adrin Jalali, ahmadjubair33, AJ Druck, aj-white, Alan Peixinho,
Alberto Mario Ceballos-Arroyo, Alek Lefebvre, Alex, Alexandr,
Alexandre Gramfort, alexanmv, almeidayoel, Amanda Dsouza, Aman Sharma,

Amar pratap singh, Amit, amrcode, András Simon, Andreas Grivas,
Andreas Mueller, Andrew Knyazev, Andriy, Angus L'Herrou, Ankit Sharma,
Anne Ducout, Arisa, Arth, arthurmello, Arturo Amor, ArturoAmor,
Atharva Patil, aufarkari, Aurélien Geron, avm19, Ayan Bag, baam,
Bardiya Ak, Behrouz B, Ben3940, Benjamin Bossan, Bharat Raghunathan,
Bijil Subhash, bmreiniger, Brandon Truth, Brenden Kadota, Brian Sun,
cdrig, Chalmer Lowe, Chiara Marmo, Chitteti Srinath Reddy, Chloe-
Agathe Azencott, Christian Lorentzen, Christian Ritter,
christopherlim98, Christoph T. Weidemann, Christos Aridas, Claudio
Salvatore Arcidiacono, combscCode, Daniela Fernandes, darioka, Darren
Nguyen, Dave Eargle, David Gilbertson, David Poznik, Dea María Léon,
Dennis Osei, DessyVV, Dev514, Dimitri Papadopoulos Orfanos, Diwakar
Gupta, Dr. Felix M. Riese, drskd, Emiko Sano, Emmanouil Gionanidis,
EricEllwanger, Erich Schubert, Eric Larson, Eric Ndirangu, ErmolaevPA,
Estefania Barreto-Ojeda, eyast, Fatima GASMI, Federico Luna, Felix
Glushchenkov, fkaren27, Fortune Uwha, FPGAwesome, francoisgoupil,
Frans Larsson, ftorres16, Gabor Berei, Gabor Kertesz, Gabriel
Stefanini Vicente, Gabriel S Vicente, Gael Varoquaux, GAURAV
CHOUDHARY, Gauthier I, genvalen, Geoffrey-Paris, Giancarlo Pablo,

glennfrutiz, gpapadok, Guillaume Lemaitre, Guillermo Tomás Fernández
Martín, Gustavo Oliveira, Haidar Almubarak, Hannah Bohle, Hansin
Ahuja, Haoyin Xu, Haya, Helder Geovane Gomes de Lima, henrymooresc,
Hideaki Imamura, Himanshu Kumar, Hind-M, hmasdev, hvassard, i-aki-y,
iasoon, Inclusive Coding Bot, Ingela, iofall, Ishan Kumar, Jack Liu,
Jake Cowton, jalexand3r, J Alexander, Jauhar, Jaya Surya Kommireddy,
Jay Stanley, Jeff Hale, je-kr, JElfner, Jenny Vo, Jérémie du
Boisberranger, Jihane, Jirka Borovec, Joel Nothman, Jon Haitz
Legarreta Gorroño, Jordan Silke, Jorge Ciprián, Jorge Loayza, Joseph
Chazalon, Joseph Schwartz-Messing, Jovan Stojanovic, JSchuerz, Juan
Carlos Alfaro Jiménez, Juan Martin Loyola, Julien Jerphanion,
katotten, Kaushik Roy Chowdhury, Ken4git, Kenneth Prabakaran, kernc,
Kevin Doucet, KimAYoung, Koushik Joshi, Kranthi Sedamaki, krishna
kumar, krumetoft, lesnee, Lisa Casino, Logan Thomas, Loic Esteve,
Louis Wagner, LucieClair, Lucy Liu, Luiz Eduardo Amaral, Magali,
MaggieChege, Mai, mandjevant, Mandy Gu, Manimaran, MarcoM, Marco
Wurps, Maren Westermann, Maria Boerner, MarieS-WiMLDS, Martel
Corentin, martin-kokos, mathurinm, Matías, matjansen, Matteo Francia,
Maxwell, Meekail Zain, Megabyte, Mehrdad Moradizadeh, melemo2, Michael
I Chen, michalkrawczyk, Micky774, milana2, millawell, Ming-Yang Ho,

Mitzi, miwojc, Mizuki, mlant, Mohamed Haseeb, Mohit Sharma,
Moonkyung94, mpoemsl, MrinalTyagi, Mr. Leu, msabatier, murata-yu, N,
Nadirhan Şahin, Naipawat Poolsawat, NartayXD, nastegiano, nathansquan,
nat-salt, Nicki Skafte Detlefsen, Nicolas Hug, Niket Jain, Nikhil
Suresh, Nikita Titov, Nikolay Kondratyev, Ohad Michel, Oleksandr
Husak, Olivier Grisel, partev, Patrick Ferreira, Paul, pelennor,
PierreAttard, Piet Brömmel, Pieter Gijsbers, Pinky, poloso, Pramod
Anantharam, puhuk, Purna Chandra Mansingh, QuadV, Rahil Parikh,
Randall Boyes, randomgeek78, Raz Hoshia, Reshama Shaikh, Ricardo
Ferreira, Richard Taylor, Rileran, Rishabh, Robin Thibaut, Rocco Meli,
Roman Feldbauer, Roman Yurchak, Ross Barnowski, rsnegrin, Sachin
Yadav, sakinaOuisrani, Sam Adam Day, Sanjay Marreddi, Sebastian
Pujalte, SEELE, SELEE, Seyedsaman (Sam) Emami, ShanDeng123, Shao Yang
Hong, sharmadharmpal, shaymerNaturalint, Shuangchi He, Shubhraneel
Pal, siavrez, slishak, Smile, spikebh, sply88, Srinath Kailasa,
Stéphane Collot, Sultan Orazbayev, Sumit Saha, Sven Eschlbeck, Sven
Stehle, Swapnil Jha, Sylvain Marié, Takeshi Oura, Tamires Santana,
Tenavi, teunpe, Theis Ferré Hjortkjær, Thiruvenkadam, Thomas J. Fan,
t-jakubek, toastedyeast, Tom Dupré la Tour, Tom McTiernan, TONY
GEORGE, Tyler Martin, Tyler Reddy, Udit Gupta, Ugo Marchand, Varun

Agrawal, Venkatachalam N, Vera Komeyer, victoirelouis, Vikas
Vishwakarma, Vikrant khedkar, Vladimir Chernyy, Vladimir Kim,
WeijiaDu, Xiao Yuan, Yar Khine Phyo, Ying Xiong, yiyangq, Yosshi999,
Yuki Koyama, Zach Deane-Mayer, Zeel B Patel, zempleni, zhenfisher, 赵
丰 (Zhao Feng)

Version 0.22.2.post1

**March 3 2020**

The 0.22.2.post1 release includes a packaging fix for the source
distribution but the content of the packages is otherwise identical to
the content of the wheels with the 0.22.2 version (without the .post1
suffix). Both contain the following changes.

Changelog
=========

*   Reduce "impute.KNNImputer()" asymptotic memory usage by chunking
  pairwise distance computation. >>:pr:`16397`<< by Joel Nothman.

*   Fixed a bug in "metrics.plot_roc_curve()" where the name of the
  estimator was passed in the "metrics.RocCurveDisplay" instead of the
  parameter *name*. It results in a different plot when calling
  "metrics.RocCurveDisplay.plot()" for the subsequent times.
  >>:pr:`16500`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   Fixed a bug in "metrics.plot_precision_recall_curve()" where the
  name of the estimator was passed in the
  "metrics.PrecisionRecallDisplay" instead of the parameter *name*. It
  results in a different plot when calling
  "metrics.PrecisionRecallDisplay.plot()" for the subsequent times.
  >>:pr:`16505`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   Fix a bug which converted a list of arrays into a 2-D object array
  instead of a 1-D array containing NumPy arrays. This bug was
  affecting "neighbors.NearestNeighbors.radius_neighbors()".
  >>:pr:`16076`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<< and

instead of a 1-D array containing NumPy arrays. This bug was
  >>:user:`Alex Shacked <alexshacked>`<<.

Version 0.22.1

**January 2 2020**

in version 0.22.0. It also includes minor documentation improvements

Changelog
=========

*   "cluster.KMeans" with "algorithm="elkan"" now uses the same
  stopping criterion as with the default "algorithm="full"".
  >>:pr:`15930`<< by >>:user:`inder128`<<.

*   "inspection.permutation_importance()" will return the same
  *importances* when a *random_state* is given for both *n_jobs=1* or
  *n_jobs>1* both with shared memory backends (thread-safety) and
  isolated memory, process-based backends. Also avoid casting the data
  as object dtype and avoid read-only error on large dataframes with
  *n_jobs>1* as reported in >>:issue:`15810`<<. Follow-up of
  >>:pr:`15898`<< by >>:user:`Shivam Gargsya <shivamgargsya>`<<.
  >>:pr:`15933`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<< and
  Olivier Grisel.

*   "inspection.plot_partial_dependence()" and
  "inspection.PartialDependenceDisplay.plot()" now consistently checks
  the number of axes passed in. >>:pr:`15760`<< by Thomas Fan.

"inspection.PartialDependenceDisplay.plot()" now consistently checks

*   "metrics.plot_confusion_matrix()" now raises error when
  *normalize* is invalid. Previously, it runs fine with no
  normalization. >>:pr:`15888`<< by Hanmin Qin.

*   "metrics.plot_confusion_matrix()" now colors the label color
  correctly to maximize contrast with its background. >>:pr:`15936`<<
  by Thomas Fan and >>:user:`DizietAsahi`<<.

*   "metrics.classification_report()" does no longer ignore the value
  of the "zero_division" keyword argument. >>:pr:`15879`<< by
  >>:user:`Bibhash Chandra Mitra <Bibyutatsu>`<<.

*   Fixed a bug in "metrics.plot_confusion_matrix()" to correctly pass
  the *values_format* parameter to the "ConfusionMatrixDisplay" plot()
  call. >>:pr:`15937`<< by >>:user:`Stephen Blystone <blynotes>`<<.

*   "model_selection.GridSearchCV" and
  "model_selection.RandomizedSearchCV" accept scalar values provided
  in *fit_params*. Change in 0.22 was breaking backward compatibility.
  >>:pr:`15863`<< by >>:user:`Adrin Jalali <adrinjalali>`<< and

*   Removed *abstractmethod* decorator for the method *_check_X* in
  "naive_bayes.BaseNB" that could break downstream projects inheriting
  from this deprecated public base class. >>:pr:`15996`<< by

  >>:user:`Brigitta Sipőcz <bsipocz>`<<.

*   "preprocessing.QuantileTransformer" now guarantees the
  *quantiles_* attribute to be completely sorted in non-decreasing
  manner. >>:pr:`15751`<< by >>:user:`Tirth Patel
  <tirthasheshpatel>`<<.

"sklearn.semi_supervised"

*   "semi_supervised.LabelPropagation" and
  "semi_supervised.LabelSpreading" now allow callable kernel function
  to return sparse weight matrix. >>:pr:`15868`<< by >>:user:`Niklas
  Smedemark-Margulies <nik-sm>`<<.

*   "utils.check_array()" now correctly converts pandas DataFrame with
  boolean columns to floats. >>:pr:`15797`<< by Thomas Fan.

*   "utils.check_is_fitted()" accepts back an explicit "attributes"
  argument to check for specific attributes as explicit markers of a
  fitted estimator. When no explicit "attributes" are provided, only
  the attributes that end with a underscore and do not start with
  double underscore are used as "fitted" markers. The "all_or_any"
  argument is also no longer deprecated. This change is made to
  restore some backward compatibility with the behavior of this
  utility in version 0.21. >>:pr:`15947`<< by Thomas Fan.

Version 0.22.0

**December 3 2019**

restore some backward compatibility with the behavior of this

lights_0_22_0.py*.

  memory.

Website update

Our website was revamped and given a fresh new look. >>:pr:`14849`<<
by Thomas Fan.

Clear definition of the public API

Scikit-learn has a public API, and a private API.

We do our best not to break the public API, and to only introduce
backward-compatible changes that do not require any user action.
However, in cases where that's not possible, any change to the public
API is subject to a deprecation cycle of two minor versions. The
private API isn't publicly documented and isn't subject to any
deprecation cycle, so users should not rely on its stability.

A function or object is public if it is documented in the API

Reference and if it can be imported with an import path without
leading underscores. For example "sklearn.pipeline.make_pipeline" is
public, while *sklearn.pipeline._name_estimators* is private.
"sklearn.ensemble._gb.BaseEnsemble" is private too because the whole
*_gb* module is private.

Up to 0.22, some tools were de-facto public (no leading underscore),
while they should have been private in the first place. In version
0.22, these tools have been made properly private, and the public API
space has been cleaned. In addition, importing from most sub-modules
is now deprecated: you should for example use "from sklearn.cluster
import Birch" instead of "from sklearn.cluster.birch import Birch" (in
practice, "birch.py" has been moved to "_birch.py").

Note:

  All the tools in the public API should be documented in the API
  Reference. If you find a public tool (without leading underscore)
  that isn't in the API reference, that means it should either be
  private or documented. Please let us know by opening an issue!

This work was tracked in issue 9250 and issue 12927.

Deprecations: using "FutureWarning" from now on
===============================================

When deprecating a feature, previous versions of scikit-learn used to
raise a "DeprecationWarning". Since the "DeprecationWarnings" aren't

shown by default by Python, scikit-learn needed to resort to a custom
warning filter to always show the warnings. That filter would
sometimes interfere with users custom warning filters.

Starting from version 0.22, scikit-learn will show "FutureWarnings"
for deprecations, as recommended by the Python documentation.
"FutureWarnings" are always shown by default by Python, so the custom
filter has been removed and scikit-learn no longer hinders with user
filters. >>:pr:`15080`<< by Nicolas Hug.

* "cluster.KMeans" when *n_jobs=1*.

* "decomposition.SparseCoder", "decomposition.DictionaryLearning", and
  "decomposition.MiniBatchDictionaryLearning"

* "decomposition.SparseCoder" with *algorithm='lasso_lars'*

* "decomposition.SparsePCA" where *normalize_components* has no effect
  due to deprecation.

* "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor"  ,  ,  .

* "impute.IterativeImputer" when *X* has features with no missing
  values.

* "linear_model.Ridge" when *X* is sparse.

"ensemble.HistGradientBoostingRegressor"  ,  ,  .

  values.

* "model_selection.StratifiedKFold" and any use of *cv=int* with a
  classifier.

* "cross_decomposition.CCA" when using scipy >= 1.3

Changelog
=========

*   From version 0.24 "base.BaseEstimator.get_params()" will raise an
  AttributeError rather than return None for parameters that are in
  the estimator's constructor but not stored as attributes on the
  instance. >>:pr:`14464`<< by Joel Nothman.

*   Fixed a bug that made "calibration.CalibratedClassifierCV" fail
  when given a *sample_weight* parameter of type *list* (in the case
  where *sample_weights* are not supported by the wrapped estimator).
  >>:pr:`13575`<< by >>:user:`William de Vazelhes <wdevazelhes>`<<.

*   "cluster.SpectralClustering" now accepts precomputed sparse
  neighbors graph as input. >>:issue:`10482`<< by Tom Dupre la Tour
  and >>:user:`Kumar Ashutosh <thechargedneutron>`<<.

*   "cluster.SpectralClustering" now accepts a "n_components"
  parameter. This parameter extends *SpectralClustering* class
  functionality to match "cluster.spectral_clustering()".

  >>:pr:`13726`<< by >>:user:`Shuzhe Xiao <fdas3213>`<<.

*   Fixed a bug where "cluster.KMeans" produced inconsistent results
  between *n_jobs=1* and *n_jobs>1* due to the handling of the random
  state. >>:pr:`9288`<< by >>:user:`Bryan Yang <bryanyang0528>`<<.

*   Fixed a bug where *elkan* algorithm in "cluster.KMeans" was
  producing Segmentation Fault on large arrays due to integer index
  overflow. >>:pr:`15057`<< by >>:user:`Vladimir Korolev <balodja>`<<.

*   "MeanShift" now accepts a *max_iter* with a default value of 300
  instead of always using the default 300. It also now exposes an
  "n_iter_" indicating the maximum number of iterations performed on
  each seed. >>:pr:`15120`<< by Adrin Jalali.

  "cluster.FeatureAgglomeration" now raise an error if
  *affinity='cosine'* and *X* has samples that are all-zeros.
  >>:pr:`7943`<< by >>:user:`mthorrell`<<.

*    Adds "compose.make_column_selector()" which is used with
  "compose.ColumnTransformer" to select DataFrame columns on the basis
  of name and dtype. >>:pr:`12303`<< by Thomas Fan.

*   Fixed a bug in "compose.ColumnTransformer" which failed to select
  the proper columns when using a boolean list, with NumPy older than
  1.12. >>:pr:`14510`<< by Guillaume Lemaitre.

*   Fixed a bug in "compose.TransformedTargetRegressor" which did not
  pass ***fit_params* to the underlying regressor. >>:pr:`14890`<< by
  >>:user:`Miguel Cabrera <mfcabrera>`<<.

*   The "compose.ColumnTransformer" now requires the number of
  features to be consistent between *fit* and *transform*. A
  *FutureWarning* is raised now, and this will raise an error in 0.24.
  If the number of features isn't consistent and negative indexing is
  used, an error is raised. >>:pr:`14544`<< by Adrin Jalali.

*   "cross_decomposition.PLSCanonical" and
  "cross_decomposition.PLSRegression" have a new function
  "inverse_transform" to transform data to the original space.
  >>:pr:`15304`<< by >>:user:`Jaime Ferrando Huertas <jiwidi>`<<.

*   "decomposition.KernelPCA" now properly checks the eigenvalues
  found by the solver for numerical or conditioning issues. This
  ensures consistency of results across solvers (different choices for
  "eigen_solver"), including approximate solvers such as
  "'randomized'" and "'lobpcg'" (see >>:issue:`12068`<<).
  >>:pr:`12145`<< by >>:user:`Sylvain Marié <smarie>`<<

*   Fixed a bug where "cross_decomposition.PLSCanonical" and
  "cross_decomposition.PLSRegression" were raising an error when

>>:pr:`12145`<< by >>:user:`Sylvain Marié <smarie>`<<

  fitted with a target matrix *Y* in which the first column was
  constant. >>:issue:`13609`<< by >>:user:`Camila Williamson
  <camilaagw>`<<.

*   "cross_decomposition.CCA" now produces the same results with scipy
  1.3 and previous scipy versions. >>:pr:`15661`<< by Thomas Fan.

*   "datasets.fetch_openml()" now supports heterogeneous data using
  pandas by setting *as_frame=True*. >>:pr:`13902`<< by Thomas Fan.

*   "datasets.fetch_openml()" now includes the *target_names* in the
  returned Bunch. >>:pr:`15160`<< by Thomas Fan.

*   The parameter *return_X_y* was added to
  "datasets.fetch_20newsgroups()" and
  "datasets.fetch_olivetti_faces()" . >>:pr:`14259`<< by
  >>:user:`Sourav Singh <souravsingh>`<<.

*   "datasets.make_classification()" now accepts array-like *weights*
  parameter, i.e. list or numpy.array, instead of list only.
  >>:pr:`14764`<< by >>:user:`Cat Chenal <CatChenal>`<<.

*   The parameter *normalize* was added to
     "datasets.fetch_20newsgroups_vectorized()". >>:pr:`14740`<< by
     >>:user:`Stéphan Tulkens <stephantul>`<<

*   Fixed a bug in "datasets.fetch_openml()", which failed to load an
  OpenML dataset that contains an ignored feature. >>:pr:`14623`<< by
  >>:user:`Sarra Habchi <HabchiSarra>`<<.

OpenML dataset that contains an ignored feature. >>:pr:`14623`<< by

*   "decomposition.NMF(solver='mu')" fitted on sparse input matrices
  now uses batching to avoid briefly allocating an array with size
  (#non-zero elements, n_components). >>:pr:`15257`<< by Mart Willocx.

*   "decomposition.dict_learning()" and
  "decomposition.dict_learning_online()" now accept *method_max_iter*
  and pass it to "decomposition.sparse_encode()". >>:issue:`12650`<<
  by Adrin Jalali.

*   "decomposition.SparseCoder", "decomposition.DictionaryLearning",
  and "decomposition.MiniBatchDictionaryLearning" now take a
  *transform_max_iter* parameter and pass it to either
  "decomposition.dict_learning()" or "decomposition.sparse_encode()".
  >>:issue:`12650`<< by Adrin Jalali.

*   "decomposition.IncrementalPCA" now accepts sparse matrices as
  input, converting them to dense in batches thereby avoiding the need
  to store the entire dense matrix at once. >>:pr:`13960`<< by
  >>:user:`Scott Gigante <scottgigante>`<<.

*   "decomposition.sparse_encode()" now passes the *max_iter* to the
  underlying "linear_model.LassoLars" when *algorithm='lasso_lars'*.

*   "dummy.DummyClassifier" now handles checking the existence of the
  provided constant in multiouput cases. >>:pr:`14908`<< by
  >>:user:`Martina G. Vilas <martinagvilas>`<<.

*   The default value of the *strategy* parameter in
  "dummy.DummyClassifier" will change from *'stratified'* in version
  0.22 to *'prior'* in 0.24. A FutureWarning is raised when the
  default value is used. >>:pr:`15382`<< by Thomas Fan.

*   The "outputs_2d_" attribute is deprecated in
  "dummy.DummyClassifier" and "dummy.DummyRegressor". It is equivalent
  to "n_outputs > 1". >>:pr:`14933`<< by Nicolas Hug

*   Added "ensemble.StackingClassifier" and
  "ensemble.StackingRegressor" to stack predictors using a final
  classifier or regressor.  >>:pr:`11047`<< by >>:user:`Guillaume
  Lemaitre <glemaitre>`<< and >>:user:`Caio Oliveira <caioaao>`<< and
  >>:pr:`15138`<< by >>:user:`Jon Cusick <jcusick13>`<<..

*   Many improvements were made to
  "ensemble.HistGradientBoostingRegressor":

  *   Estimators now natively support dense data with missing values
    both for training and predicting. They also support infinite
    values. >>:pr:`13911`<< and >>:pr:`14406`<< by Nicolas Hug, Adrin
    Jalali and Olivier Grisel.

  *   Estimators now have an additional *warm_start* parameter that
    enables warm starting. >>:pr:`14012`<< by >>:user:`Johann Faouzi
    <johannfaouzi>`<<.

  *   "inspection.partial_dependence()" and

*   Estimators now have an additional *warm_start* parameter that

    "inspection.plot_partial_dependence()" now support the fast
    'recursion' method for both estimators. >>:pr:`13769`<< by Nicolas
    Hug.

  *   for "ensemble.HistGradientBoostingClassifier" the training loss
    or score is now monitored on a class-wise stratified subsample to
    preserve the class balance of the original training set.
    >>:pr:`14194`<< by >>:user:`Johann Faouzi <johannfaouzi>`<<.

  *   "ensemble.HistGradientBoostingRegressor" now supports the
    'least_absolute_deviation' loss. >>:pr:`13896`<< by Nicolas Hug.

  *   Estimators now bin the training and validation data separately
    to avoid any data leak. >>:pr:`13933`<< by Nicolas Hug.

  *   Fixed a bug where early stopping would break with string
    targets. >>:pr:`14710`<< by Guillaume Lemaitre.

  *   "ensemble.HistGradientBoostingClassifier" now raises an error if
    "categorical_crossentropy" loss is given for a binary
    classification problem. >>:pr:`14869`<< by Adrin Jalali.

  Note that pickles from 0.21 will not work in 0.22.

*   Addition of "max_samples" argument allows limiting size of
  bootstrap samples to be less than size of dataset. Added to
  "ensemble.ExtraTreesClassifier", "ensemble.ExtraTreesRegressor".

bootstrap samples to be less than size of dataset. Added to
  >>:pr:`14682`<< by >>:user:`Matt Hancock <notmatthancock>`<< and
  >>:pr:`5963`<< by >>:user:`Pablo Duboue <DrDub>`<<.

*   "ensemble.VotingClassifier.predict_proba()" will no longer be
  present when *voting='hard'*. >>:pr:`14287`<< by Thomas Fan.

*   The *named_estimators_* attribute in "ensemble.VotingClassifier"
  and "ensemble.VotingRegressor" now correctly maps to dropped
  estimators. Previously, the *named_estimators_* mapping was
  incorrect whenever one of the estimators was dropped.
  >>:pr:`15375`<< by Thomas Fan.

*   Run by default "utils.estimator_checks.check_estimator()" on both
  "ensemble.VotingClassifier" and "ensemble.VotingRegressor". It leads
  to solve issues regarding shape consistency during *predict* which
  was failing when the underlying estimators were not outputting
  consistent array dimensions. Note that it should be replaced by
  refactoring the common tests in the future. >>:pr:`14305`<< by
  Guillaume Lemaitre.

*   "ensemble.AdaBoostClassifier" computes probabilities based on the
  decision function as in the literature. Thus, *predict* and
  *predict_proba* give consistent results. >>:pr:`14114`<< by

*   Stacking and Voting estimators now ensure that their underlying
  estimators are either all classifiers or all regressors.

Guillaume Lemaitre.

  "ensemble.StackingClassifier", "ensemble.StackingRegressor", and
  "ensemble.VotingClassifier" and "VotingRegressor" now raise
  consistent error messages. >>:pr:`15084`<< by Guillaume Lemaitre.

*   "ensemble.AdaBoostRegressor" where the loss should be normalized
  by the max of the samples with non-null weights only.
  >>:pr:`14294`<< by Guillaume Lemaitre.

*   "presort" is now deprecated in
  "ensemble.GradientBoostingRegressor", and the parameter has no
  effect. Users are recommended to use
  "ensemble.HistGradientBoostingRegressor" instead. >>:pr:`14907`<< by
  Adrin Jalali.

*   A warning  will  now be raised  if a parameter choice means that
  another parameter will be unused on calling the fit() method for
  "feature_extraction.text.HashingVectorizer",
  "feature_extraction.text.CountVectorizer" and
  "feature_extraction.text.TfidfVectorizer". >>:pr:`14602`<< by
  >>:user:`Gaurav Chawla <getgaurav2>`<<.

*   Functions created by "build_preprocessor" and "build_analyzer" of
  "feature_extraction.text.VectorizerMixin" can now be pickled.
  >>:pr:`14430`<< by >>:user:`Dillon Niederhut <deniederhut>`<<.

*   "feature_extraction.text.strip_accents_unicode()" now correctly

"feature_extraction.text.VectorizerMixin" can now be pickled.

  removes accents from strings that are in NFKD normalized form.
  >>:pr:`15100`<< by >>:user:`Daniel Grady <DGrady>`<<.

*   Fixed a bug that caused "feature_extraction.DictVectorizer" to
  raise an *OverflowError* during the *transform* operation when
  producing a *scipy.sparse* matrix on large input data.
  >>:pr:`15463`<< by >>:user:`Norvan Sahiner <norvan>`<<.

*   Deprecated unused *copy* param for
  "feature_extraction.text.TfidfVectorizer.transform()" it will be
  removed in v0.24. >>:pr:`14520`<< by >>:user:`Guillem G. Subies

*   Updated the following "feature_selection" estimators to allow
  NaN/Inf values in "transform" and "fit": "feature_selection.RFE",
  "feature_selection.RFECV", "feature_selection.SelectFromModel", and
  "feature_selection.VarianceThreshold". Note that if the underlying
  estimator of the feature selector does not allow NaN/Inf then it
  will still error, but the feature selectors themselves no longer
  enforce this restriction unnecessarily. >>:issue:`11635`<< by
  >>:user:`Alec Peters <adpeters>`<<.

*   Fixed a bug where "feature_selection.VarianceThreshold" with
  *threshold=0* did not remove constant features due to numerical
  instability, by using range rather than variance in this case.

  >>:pr:`13704`<< by >>:user:`Roddy MacSween <rlms>`<<.

*   Gaussian process models on structured data:
  "gaussian_process.GaussianProcessRegressor" and
  "gaussian_process.GaussianProcessClassifier" can now accept a list
  of generic objects (e.g. strings, trees, graphs, etc.) as the "X"
  argument to their training/prediction methods. A user-defined kernel
  should be provided for computing the kernel matrix among the generic
  objects, and should inherit from
  "gaussian_process.kernels.GenericKernelMixin" to notify the GPR/GPC
  model that it handles non-vectorial samples. >>:pr:`15557`<< by
  >>:user:`Yu-Hang Tang <yhtang>`<<.

*   "gaussian_process.GaussianProcessClassifier.log_marginal_likeliho
  od()" and "gaussian_process.GaussianProcessRegressor.log_marginal_l
  ikelihood()" now accept a "clone_kernel=True" keyword argument. When
  set to "False", the kernel attribute is modified, but may result in
  a performance improvement. >>:pr:`14378`<< by >>:user:`Masashi
  Shibata <c-bata>`<<.

*   From version 0.24 "gaussian_process.kernels.Kernel.get_params()"
  will raise an "AttributeError" rather than return "None" for
  parameters that are in the estimator's constructor but not stored as

  attributes on the instance. >>:pr:`14464`<< by Joel Nothman.

*   Added "impute.KNNImputer", to impute missing values using
  k-Nearest Neighbors. >>:issue:`12852`<< by >>:user:`Ashim Bhattarai
  <ashimb9>`<< and Thomas Fan and >>:pr:`15010`<< by Guillaume
  Lemaitre.

*   "impute.IterativeImputer" has new *skip_compute* flag that is
  False by default, which, when True, will skip computation on
  features that have no missing values during the fit phase.
  >>:issue:`13773`<< by >>:user:`Sergey Feldman <sergeyf>`<<.

*   "impute.MissingIndicator.fit_transform()" avoid repeated
  computation of the masked matrix. >>:pr:`14356`<< by >>:user:`Harsh
  Soni <harsh020>`<<.

*   "impute.IterativeImputer" now works when there is only one
  feature. By >>:user:`Sergey Feldman <sergeyf>`<<.

*   Fixed a bug in "impute.IterativeImputer" where features where
  imputed in the reverse desired order with "imputation_order" either
  ""ascending"" or ""descending"". >>:pr:`15393`<< by
  >>:user:`Venkatachalam N <venkyyuvy>`<<.

*   "inspection.permutation_importance()" has been added to measure
  the importance of each feature in an arbitrary trained model with
  respect to a given scoring function. >>:issue:`13146`<< by Thomas
  Fan.

the importance of each feature in an arbitrary trained model with
  Fan.

*   "inspection.partial_dependence()" and
  "inspection.plot_partial_dependence()" now support the fast
  'recursion' method for "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor". >>:pr:`13769`<< by Nicolas
  Hug.

*   "inspection.plot_partial_dependence()" has been extended to now
  support the new visualization API described in the User Guide.
  >>:pr:`14646`<< by Thomas Fan.

*   "inspection.partial_dependence()" accepts pandas DataFrame and
  "pipeline.Pipeline" containing "compose.ColumnTransformer". In
  addition "inspection.plot_partial_dependence()" will use the column
  names by default when a dataframe is passed. >>:pr:`14028`<< and
  >>:pr:`15429`<< by Guillaume Lemaitre.

*   Fixed a bug where "kernel_approximation.Nystroem" raised a
  *KeyError* when using *kernel="precomputed"*. >>:pr:`14706`<< by

*   The 'liblinear' logistic regression solver is now faster and
  requires less memory. >>:pr:`14108`<<, >>:pr:`14170`<<,
  >>:pr:`14296`<< by >>:user:`Alex Henrie <alexhenrie>`<<.

*   "linear_model.BayesianRidge" now accepts hyperparameters
  "alpha_init" and "lambda_init" which can be used to set the initial

>>:pr:`14296`<< by >>:user:`Alex Henrie <alexhenrie>`<<.

  value of the maximization procedure in *fit*. >>:pr:`13618`<< by
  >>:user:`Yoshihiro Uchida <c56pony>`<<.

*   "linear_model.Ridge" now correctly fits an intercept when *X* is
  sparse, *solver="auto"* and *fit_intercept=True*, because the
  default solver in this configuration has changed to *sparse_cg*,
  which can fit an intercept with sparse data. >>:pr:`13995`<< by
  >>:user:`Jérôme Dockès <jeromedockes>`<<.

*   "linear_model.Ridge" with *solver='sag'* now accepts F-ordered and
  non-contiguous arrays and makes a conversion instead of failing.
  >>:pr:`14458`<< by Guillaume Lemaitre.

*   "linear_model.LassoCV" no longer forces "precompute=False" when
  fitting the final model. >>:pr:`14591`<< by Andreas Müller.

  correctly scores when *cv=None*. >>:pr:`14864`<< by

  "scores_", "n_iter_" and "coefs_paths_" attribute would have a wrong
  ordering with "penalty='elastic-net'". >>:pr:`15044`<< by Nicolas
  Hug

*   "linear_model.MultiTaskLassoCV" and
  "linear_model.MultiTaskElasticNetCV" with X of dtype int and
  *fit_intercept=True*. >>:pr:`15086`<< by >>:user:`Alex Gramfort
  <agramfort>`<<.

Hug

*   The liblinear solver now supports "sample_weight". >>:pr:`15038`<<
  by Guillaume Lemaitre.

*   "manifold.Isomap", "manifold.TSNE", and
  "manifold.SpectralEmbedding" now accept precomputed sparse neighbors
  graph as input. >>:issue:`10482`<< by Tom Dupre la Tour and

*   Exposed the "n_jobs" parameter in "manifold.TSNE" for multi-core
  calculation of the neighbors graph. This parameter has no impact
  when "metric="precomputed"" or ("metric="euclidean"" and
  "method="exact""). >>:issue:`15082`<< by Roman Yurchak.

*   Improved efficiency of "manifold.TSNE" when "method="barnes-hut""
  by computing the gradient in parallel. >>:pr:`13213`<< by
  >>:user:`Thomas Moreau <tommoral>`<<

*   Fixed a bug where "manifold.spectral_embedding()" (and therefore
  "manifold.SpectralEmbedding" and "cluster.SpectralClustering")
  computed wrong eigenvalues with "eigen_solver='amg'" when "n_samples
  < 5 * n_components". >>:pr:`14647`<< by Andreas Müller.

*   Fixed a bug in "manifold.spectral_embedding()"  used in
  "manifold.SpectralEmbedding" and "cluster.SpectralClustering" where
  "eigen_solver="amg"" would sometimes result in a LinAlgError.
  >>:issue:`13393`<< by >>:user:`Andrew Knyazev <lobpcg>`<<

"manifold.SpectralEmbedding" and "cluster.SpectralClustering" where
  >>:pr:`13707`<< by >>:user:`Scott White <whitews>`<<

*   Deprecate "training_data_" unused attribute in "manifold.Isomap".
  >>:issue:`10482`<< by Tom Dupre la Tour.

*   "metrics.plot_roc_curve()" has been added to plot roc curves. This
  function introduces the visualization API described in the User
  Guide. >>:pr:`14357`<< by Thomas Fan.

*   Added a new parameter "zero_division" to multiple classification
  metrics: "precision_score()", "recall_score()", "f1_score()",
  "fbeta_score()", "precision_recall_fscore_support()",
  "classification_report()". This allows to set returned value for
  ill-defined metrics. >>:pr:`14900`<< by >>:user:`Marc Torrellas
  Socastro <marctorrellas>`<<.

*   Added the "metrics.pairwise.nan_euclidean_distances()" metric,
  which calculates euclidean distances in the presence of missing
  values. >>:issue:`12852`<< by >>:user:`Ashim Bhattarai <ashimb9>`<<
  and Thomas Fan.

*   New ranking metrics "metrics.ndcg_score()" and
  "metrics.dcg_score()" have been added to compute Discounted
  Cumulative Gain and Normalized Discounted Cumulative Gain.
  >>:pr:`9951`<< by >>:user:`Jérôme Dockès <jeromedockes>`<<.

*   "metrics.plot_precision_recall_curve()" has been added to plot
  precision recall curves. >>:pr:`14936`<< by Thomas Fan.

>>:pr:`9951`<< by >>:user:`Jérôme Dockès <jeromedockes>`<<.

*   "metrics.plot_confusion_matrix()" has been added to plot confusion
  matrices. >>:pr:`15083`<< by Thomas Fan.

*   Added multiclass support to "metrics.roc_auc_score()" with
  corresponding scorers *'roc_auc_ovr'*, *'roc_auc_ovo'*,
  *'roc_auc_ovr_weighted'*, and *'roc_auc_ovo_weighted'*.
  >>:pr:`12789`<< and >>:pr:`15274`<< by >>:user:`Kathy Chen
  <kathyxchen>`<<, >>:user:`Mohamed Maskani <maskani-moh>`<<, and

*   Add "metrics.mean_tweedie_deviance" measuring the Tweedie deviance
  for a given "power" parameter. Also add mean Poisson deviance
  "metrics.mean_poisson_deviance" and mean Gamma deviance
  "metrics.mean_gamma_deviance" that are special cases of the Tweedie
  deviance for "power=1" and "power=2" respectively. >>:pr:`13938`<<
  by >>:user:`Christian Lorentzen <lorentzenchr>`<< and Roman Yurchak.

*   Improved performance of "metrics.pairwise.manhattan_distances()"
  in the case of sparse matrices. >>:pr:`15049`<< by *Paolo Toccaceli
  <ptocca>*.

*   The parameter "beta" in "metrics.fbeta_score()" is updated to
  accept the zero and *float('+inf')* value. >>:pr:`13231`<< by
  >>:user:`Dong-hee Na <corona10>`<<.

*   Added parameter "squared" in "metrics.mean_squared_error()" to
  return root mean squared error. >>:pr:`13467`<< by >>:user:`Urvang
  Patel <urvang96>`<<.

>>:user:`Dong-hee Na <corona10>`<<.

*   Allow computing averaged metrics in the case of no true positives.
  >>:pr:`14595`<< by Andreas Müller.

*   Multilabel metrics now supports list of lists as input.
  >>:pr:`14865`<< >>:user:`Srivatsan Ramesh <srivatsan-ramesh>`<<,
  >>:user:`Herilalaina Rakotoarison <herilalaina>`<<, >>:user:`Léonard
  Binet <leonardbinet>`<<.

*   "metrics.median_absolute_error()" now supports "multioutput"
  parameter. >>:pr:`14732`<< by >>:user:`Agamemnon Krasoulis

*   'roc_auc_ovr_weighted' and 'roc_auc_ovo_weighted' can now be used
  as the *scoring* parameter of model-selection tools. >>:pr:`14417`<<

*   "metrics.confusion_matrix()" accepts a parameters *normalize*
  allowing to normalize the confusion matrix by column, rows, or
  overall. >>:pr:`15625`<< by *Guillaume Lemaitre <glemaitre>*.

*   Raise a ValueError in "metrics.silhouette_score()" when a
  precomputed distance matrix contains non-zero diagonal entries.
  >>:pr:`12258`<< by >>:user:`Stephen Tierney <sjtrny>`<<.

*   "scoring="neg_brier_score"" should be used instead of
  "scoring="brier_score_loss"" which is now deprecated.
  >>:pr:`14898`<< by >>:user:`Stefan Matcovici <stefan-matcovici>`<<.

*   Improved performance of multimetric scoring in

>>:pr:`14898`<< by >>:user:`Stefan Matcovici <stefan-matcovici>`<<.

  "model_selection.cross_validate()", "model_selection.GridSearchCV",
  and "model_selection.RandomizedSearchCV". >>:pr:`14593`<< by Thomas
  Fan.

*   "model_selection.learning_curve" now accepts parameter
  "return_times" which can be used to retrieve computation times in
  order to plot model scalability (see learning_curve example).
  >>:pr:`13938`<< by >>:user:`Hadrien Reboul <H4dr1en>`<<.

*   "model_selection.RandomizedSearchCV" now accepts lists of
  parameter distributions. >>:pr:`14549`<< by Andreas Müller.

*   Reimplemented "model_selection.StratifiedKFold" to fix an issue
  where one test set could be *n_classes* larger than another. Test
  sets should now be near-equally sized. >>:pr:`14704`<< by Joel
  Nothman.

*   The *cv_results_* attribute of "model_selection.GridSearchCV" and
  "model_selection.RandomizedSearchCV" now only contains unfitted
  estimators. This potentially saves a lot of memory since the state
  of the estimators isn't stored. >>:pr:`#15096`<< by Andreas Müller.

*   "model_selection.KFold" and "model_selection.StratifiedKFold" now
  raise a warning if *random_state* is set but *shuffle* is False.
  This will raise an error in 0.24.

*   "multioutput.MultiOutputClassifier" now has attribute "classes_".

This will raise an error in 0.24.

  >>:pr:`14629`<< by >>:user:`Agamemnon Krasoulis <agamemnonc>`<<.

*   "multioutput.MultiOutputClassifier" now has *predict_proba* as
  property and can be checked with *hasattr*. >>:issue:`15488`<<
  >>:pr:`15490`<< by >>:user:`Rebekah Kim <rebekahkim>`<<

*   Added "naive_bayes.CategoricalNB" that implements the Categorical
  Naive Bayes classifier. >>:pr:`12569`<< by >>:user:`Tim Bicker
  <timbicker>`<< and >>:user:`Florian Wilhelm <FlorianWilhelm>`<<.

*   Added "neighbors.KNeighborsTransformer" and
  "neighbors.RadiusNeighborsTransformer", which transform input
  dataset into a sparse neighbors graph. They give finer control on
  nearest neighbors computations and enable easy pipeline caching for
  multiple use. >>:issue:`10482`<< by Tom Dupre la Tour.

*   "neighbors.KNeighborsClassifier", "neighbors.KNeighborsRegressor",
  "neighbors.RadiusNeighborsClassifier",
  "neighbors.RadiusNeighborsRegressor", and
  "neighbors.LocalOutlierFactor" now accept precomputed sparse

*   "neighbors.RadiusNeighborsClassifier" now supports predicting
  probabilities by using *predict_proba* and supports more

and >>:user:`Kumar Ashutosh <thechargedneutron>`<<.

  outlier_label options: 'most_frequent', or different outlier_labels
  for multi-outputs. >>:pr:`9597`<< by >>:user:`Wenbo Zhao
  <webber26232>`<<.

*   Efficiency improvements for
  "neighbors.RadiusNeighborsClassifier.predict()". >>:pr:`9597`<< by
  >>:user:`Wenbo Zhao <webber26232>`<<.

*   "neighbors.KNeighborsRegressor" now throws error when
  *metric='precomputed'* and fit on non-square data.  >>:pr:`14336`<<
  by >>:user:`Gregory Dexter <gdex1>`<<.

*   Add *max_fun* parameter in
  give control over maximum number of function evaluation to not meet
  "tol" improvement. >>:issue:`9274`<< by >>:user:`Daniel Perry
  <daniel-perry>`<<.

*   "pipeline.Pipeline" now supports *score_samples* if the final
  estimator does. >>:pr:`13806`<< by >>:user:`Anaël Beaugnon <ab-
  anssi>`<<.

*   The *fit* in "FeatureUnion" now accepts *fit_params* to pass to
  the underlying transformers. >>:pr:`15119`<< by Adrin Jalali.

*   *None* as a transformer is now deprecated in
  "pipeline.FeatureUnion". Please use *'drop'* instead.
  >>:pr:`15053`<< by Thomas Fan.

*   "preprocessing.PolynomialFeatures" is now faster when the input
  data is dense. >>:pr:`13290`<< by >>:user:`Xavier Dupré
  <sdpython>`<<.

*   Avoid unnecessary data copy when fitting preprocessors
  "preprocessing.StandardScaler", "preprocessing.MinMaxScaler",
  "preprocessing.MaxAbsScaler", "preprocessing.RobustScaler" and
  "preprocessing.QuantileTransformer" which results in a slight
  performance improvement. >>:pr:`13987`<< by Roman Yurchak.

*   KernelCenterer now throws error when fit on non-square
  "preprocessing.KernelCenterer" >>:pr:`14336`<< by >>:user:`Gregory
  Dexter <gdex1>`<<.

  *model_selection.RandomizedSearchCV* now supports the *_pairwise*
  property, which prevents an error during cross-validation for
  estimators with pairwise inputs (such as
  "neighbors.KNeighborsClassifier" when *metric* is set to
  'precomputed'). >>:pr:`13925`<< by >>:user:`Isaac S. Robson
  <isrobson>`<< and >>:pr:`15524`<< by >>:user:`Xun Tang <xun-
  tang>`<<.

*   "svm.SVC" and "svm.NuSVC" now accept a "break_ties" parameter.
  This parameter results in *predict* breaking the ties according to
  the confidence values of *decision_function*, if

  "decision_function_shape='ovr'", and the number of target classes >
  2. >>:pr:`12557`<< by Adrin Jalali.

*   SVM estimators now throw a more specific error when
  *kernel='precomputed'* and fit on non-square data. >>:pr:`14336`<<

*   "svm.SVC", "svm.SVR", "svm.NuSVR" and "svm.OneClassSVM" when
  received values negative or zero for parameter "sample_weight" in
  method fit(), generated an invalid model. This behavior occurred
  only in some border scenarios. Now in these cases, fit() will fail
  with an Exception. >>:pr:`14286`<< by >>:user:`Alex Shacked
  <alexshacked>`<<.

*   The *n_support_* attribute of "svm.SVR" and "svm.OneClassSVM" was
  previously non-initialized, and had size 2. It has now size 1 with
  the correct value. >>:pr:`15099`<< by Nicolas Hug.

*   fixed a bug in "BaseLibSVM._sparse_fit" where n_SV=0 raised a
  ZeroDivisionError. >>:pr:`14894`<< by >>:user:`Danna Naser <danna-
  naser>`<<.

*   Adds minimal cost complexity pruning, controlled by "ccp_alpha",
  to "tree.DecisionTreeClassifier", "tree.DecisionTreeRegressor",
  "tree.ExtraTreeClassifier", "tree.ExtraTreeRegressor",

  "ensemble.ExtraTreesClassifier", "ensemble.ExtraTreesRegressor",
  "ensemble.GradientBoostingRegressor". >>:pr:`12887`<< by Thomas Fan.

*   "presort" is now deprecated in "tree.DecisionTreeClassifier" and
  "tree.DecisionTreeRegressor", and the parameter has no effect.
  >>:pr:`14907`<< by Adrin Jalali.

*   The "classes_" and "n_classes_" attributes of
  "tree.DecisionTreeRegressor" are now deprecated. >>:pr:`15028`<< by
  >>:user:`Mei Guan <meiguan>`<<, Nicolas Hug, and Adrin Jalali.

*   "check_estimator()" can now generate checks by setting
  *generate_only=True*. Previously, running "check_estimator()" will
  stop when the first check fails. With *generate_only=True*, all
  checks can run independently and report the ones that are failing.
  Read more in Rolling your own estimator. >>:pr:`14381`<< by Thomas
  Fan.

*   Added a pytest specific decorator, "parametrize_with_checks()", to
  parametrize estimator checks for a list of estimators.
  >>:pr:`14381`<< by Thomas Fan.

*   A new random variable, "utils.fixes.loguniform" implements a log-
  uniform random variable (e.g., for use in RandomizedSearchCV). For

>>:pr:`14381`<< by Thomas Fan.

  example, the outcomes "1", "10" and "100" are all equally likely for
  "loguniform(1, 100)". See >>:issue:`11232`<< by >>:user:`Scott
  Sievert <stsievert>`<< and >>:user:`Nathaniel Saul <sauln>`<<, and
  *SciPy PR 10815 <https://github.com/scipy/scipy/pull/10815>*.

*   "utils.safe_indexing()" (now deprecated) accepts an "axis"
  parameter to index array-like across rows and columns. The column
  indexing can be done on NumPy array, SciPy sparse matrix, and Pandas
  DataFrame. An additional refactoring was done. >>:pr:`14035`<< and
  >>:pr:`14475`<< by Guillaume Lemaitre.

*   "utils.extmath.safe_sparse_dot()" works between 3D+ ndarray and
  sparse matrix. >>:pr:`14538`<< by >>:user:`Jérémie du Boisberranger

*   "utils.check_array()" is now raising an error instead of casting
  NaN to integer. >>:pr:`14872`<< by Roman Yurchak.

*   "utils.check_array()" will now correctly detect numeric dtypes in
  pandas dataframes, fixing a bug where "float32" was upcast to
  "float64" unnecessarily. >>:pr:`15094`<< by Andreas Müller.

*   The following utils have been deprecated and are now private:

  * "choose_check_classifiers_labels"

  * "enforce_estimator_tags_y"

  * "mocking.MockDataFrame"

  * "mocking.CheckingClassifier"

  * "optimize.newton_cg"

  * "random.random_choice_csc"

  * "utils.choose_check_classifiers_labels"

* "enforce_estimator_tags_y"

  * "utils.enforce_estimator_tags_y"

  * "utils.optimize.newton_cg"

  * "utils.random.random_choice_csc"

  * "utils.safe_indexing"

  * "utils.mocking"

  * "utils.fast_dict"

  * "utils.seq_dataset"

  * "utils.weight_vector"

  * "utils.fixes.parallel_helper" (removed)

  * All of "utils.testing" except for "all_estimators" which is now in
    "utils".

*   Fixed a bug where "isotonic.IsotonicRegression.fit" raised error
  when *X.dtype == 'float32'* and *X.dtype != y.dtype*.
  >>:pr:`14902`<< by >>:user:`Lucas <lostcoaster>`<<.

*   Port *lobpcg* from SciPy which implement some bug fixes but only
  available in 1.3+. >>:pr:`13609`<< and >>:pr:`14971`<< by Guillaume
  Lemaitre.

*   Scikit-learn now converts any input data structure implementing a
  duck array to a numpy array (using "__array__") to ensure consistent
  behavior instead of relying on "__array_function__" (see NEP 18).
  >>:pr:`14702`<< by Andreas Müller.

*   Replace manual checks with "check_is_fitted". Errors thrown when
  using a non-fitted estimators are now more uniform. >>:pr:`13013`<<
  by >>:user:`Agamemnon Krasoulis <agamemnonc>`<<.

by >>:user:`Agamemnon Krasoulis <agamemnonc>`<<.

* Estimators are now expected to raise a "NotFittedError" if "predict"
  or "transform" is called before "fit"; previously an
  "AttributeError" or "ValueError" was acceptable. >>:pr:`13013`<< by

* Binary only classifiers are now supported in estimator checks. Such
  classifiers need to have the *binary_only=True* estimator tag.
  >>:pr:`13875`<< by Trevor Stephens.

* Estimators are expected to convert input data ("X", "y",
  "sample_weights") to "numpy.ndarray" and never call
  "__array_function__" on the original datatype that is passed (see
  NEP 18). >>:pr:`14702`<< by Andreas Müller.

* *requires_positive_X* estimator tag (for models that require X to be
  non-negative) is now used by
  "utils.estimator_checks.check_estimator()" to make sure a proper
  error message is raised if X contains some negative entries.
  >>:pr:`14680`<< by >>:user:`Alex Gramfort <agramfort>`<<.

* Added check that pairwise estimators raise error on non-square data
  >>:pr:`14336`<< by >>:user:`Gregory Dexter <gdex1>`<<.

* Added two common multioutput estimator tests
  "check_classifier_multioutput()" and
  "check_regressor_multioutput()". >>:pr:`13392`<< by >>:user:`Rok
  Mihevc <rok>`<<.

*   Added "check_transformer_data_not_an_array" to checks where
  missing

"check_classifier_multioutput()" and

  missing

*   The estimators tags resolution now follows the regular MRO. They
  used to be overridable only once. >>:pr:`14884`<< by Andreas Müller.

improvement of the project since version 0.21, including:

Aaron Alphonsus, Abbie Popa, Abdur-Rahmaan Janhangeer, abenbihi,
Abhinav Sagar, Abhishek Jana, Abraham K. Lagat, Adam J. Stewart,
Aditya Vyas, Adrin Jalali, Agamemnon Krasoulis, Alec Peters,
Alessandro Surace, Alexandre de Siqueira, Alexandre Gramfort,
alexgoryainov, Alex Henrie, Alex Itkes, alexshacked, Allen Akinkunle,
Anaël Beaugnon, Anders Kaseorg, Andrea Maldonado, Andrea Navarrete,
Andreas Mueller, Andreas Schuderer, Andrew Nystrom, Angela Ambroz,
Anisha Keshavan, Ankit Jha, Antonio Gutierrez, Anuja Kelkar, Archana
Alva, arnaudstiegler, arpanchowdhry, ashimb9, Ayomide Bamidele, Baran
Buluttekin, barrycg, Bharat Raghunathan, Bill Mill, Biswadip Mandal,
blackd0t, Brian G. Barkley, Brian Wignall, Bryan Yang, c56pony,
camilaagw, cartman_nabana, catajara, Cat Chenal, Cathy, cgsavard,
Charles Vesteghem, Chiara Marmo, Chris Gregory, Christian Lorentzen,
Christos Aridas, Dakota Grusak, Daniel Grady, Daniel Perry, Danna

Naser, DatenBergwerk, David Dormagen, deeplook, Dillon Niederhut,
Dong-hee Na, Dougal J. Sutherland, DrGFreeman, Dylan Cashman,
edvardlindelof, Eric Larson, Eric Ndirangu, Eunseop Jeong, Fanny,
federicopisanu, Felix Divo, flaviomorelli, FranciDona, Franco M.
Luque, Frank Hoang, Frederic Haase, g0g0gadget, Gabriel Altay, Gabriel
do Vale Rios, Gael Varoquaux, ganevgv, gdex1, getgaurav2, Gideon
Sonoiya, Gordon Chen, gpapadok, Greg Mogavero, Grzegorz Szpak,
Guillaume Lemaitre, Guillem García Subies, H4dr1en, hadshirt, Hailey
Nguyen, Hanmin Qin, Hannah Bruce Macdonald, Harsh Mahajan, Harsh Soni,
Honglu Zhang, Hossein Pourbozorg, Ian Sanders, Ingrid Spielman, J-A16,
jaehong park, Jaime Ferrando Huertas, James Hill, James Myatt, Jay,
jeremiedbb, Jérémie du Boisberranger, jeromedockes, Jesper Dramsch,
Joan Massich, Joanna Zhang, Joel Nothman, Johann Faouzi, Jonathan
Rahn, Jon Cusick, Jose Ortiz, Kanika Sabharwal, Katarina Slama,
kellycarmody, Kennedy Kang'ethe, Kensuke Arai, Kesshi Jordan, Kevad,
Kevin Loftis, Kevin Winata, Kevin Yu-Sheng Li, Kirill Dolmatov, Kirthi
Shankar Sivamani, krishna katyal, Lakshmi Krishnan, Lakshya KD,
LalliAcqua, lbfin, Leland McInnes, Léonard Binet, Loic Esteve,
loopyme, lostcoaster, Louis Huynh, lrjball, Luca Ionescu, Lutz Roeder,

MaggieChege, Maithreyi Venkatesh, Maltimore, Maocx, Marc Torrellas,
Marie Douriez, Markus, Markus Frey, Martina G. Vilas, Martin Oywa,
Martin Thoma, Masashi SHIBATA, Maxwell Aladago, mbillingr, m-clare,
Meghann Agarwal, m.fab, Micah Smith, miguelbarao, Miguel Cabrera, Mina
Naghshhnejad, Ming Li, motmoti, mschaffenroth, mthorrell, Natasha
Borders, nezar-a, Nicolas Hug, Nidhin Pattaniyil, Nikita Titov, Nishan
Singh Mann, Nitya Mandyam, norvan, notmatthancock, novaya, nxorable,
Oleg Stikhin, Oleksandr Pavlyk, Olivier Grisel, Omar Saleem, Owen
Flanagan, panpiort8, Paolo, Paolo Toccaceli, Paresh Mathur, Paula,
Peng Yu, Peter Marko, pierretallotte, poorna-kumar, pspachtholz,
qdeffense, Rajat Garg, Raphaël Bournhonesque, Ray, Ray Bell, Rebekah
Kim, Reza Gharibi, Richard Payne, Richard W, rlms, Robert Juergens,
Rok Mihevc, Roman Feldbauer, Roman Yurchak, R Sanjabi, RuchitaGarde,
Ruth Waithera, Sackey, Sam Dixon, Samesh Lakhotia, Samuel Taylor,
Sarra Habchi, Scott Gigante, Scott Sievert, Scott White, Sebastian
Pölsterl, Sergey Feldman, SeWook Oh, she-dares, Shreya V, Shubham
Mehta, Shuzhe Xiao, SimonCW, smarie, smujjiga, Sönke Behrends,
Soumirai, Sourav Singh, stefan-matcovici, steinfurt, Stéphane
Couvreur, Stephan Tulkens, Stephen Cowley, Stephen Tierney,

SylvainLan, th0rwas, theoptips, theotheo, Thierno Ibrahima DIOP,
Thomas Edwards, Thomas J Fan, Thomas Moreau, Thomas Schmitt, Tilen
Kusterle, Tim Bicker, Timsaur, Tim Staley, Tirth Patel, Tola A, Tom
Augspurger, Tom Dupré la Tour, topisan, Trevor Stephens, ttang131,
Urvang Patel, Vathsala Achar, veerlosar, Venkatachalam N, Victor
Luzgin, Vincent Jeanselme, Vincent Lostanlen, Vladimir Korolev,
vnherdeiro, Wenbo Zhao, Wendy Hu, willdarnell, William de Vazelhes,
wolframalpha, xavier dupré, xcjason, x-martian, xsat, xun-tang,
Yinglr, yokasre, Yu-Hang "Maxin" Tang, Yulia Zamriy, Zhao Feng

Version 0.23.2

*   "inertia_" attribute of "cluster.KMeans" and
  "cluster.MiniBatchKMeans".

Changelog
=========

*   Fixed a bug in "cluster.KMeans" where rounding errors could
  prevent convergence to be declared when *tol=0*. >>:pr:`17959`<< by

*   Fixed a bug in "cluster.KMeans" and "cluster.MiniBatchKMeans"
  where the reported inertia was incorrectly weighted by the sample
  weights. >>:pr:`17848`<< by >>:user:`Jérémie du Boisberranger

*   Fixed a bug in "cluster.MeanShift" with *bin_seeding=True*. When
  the estimated bandwidth is 0, the behavior is equivalent to
  *bin_seeding=False*. >>:pr:`17742`<< by >>:user:`Jeremie du

*   Fixed a bug in "cluster.AffinityPropagation", that gives incorrect
  clusters when the array dtype is float32. >>:pr:`17995`<< by
  >>:user:`Thomaz Santana  <Wikilicious>`<< and >>:user:`Amanda Dsouza
  <amy12xx>`<<.

clusters when the array dtype is float32. >>:pr:`17995`<< by

  "decomposition.MiniBatchDictionaryLearning.partial_fit()" which
  should update the dictionary by iterating only once over a mini-
  batch. >>:pr:`17433`<< by >>:user:`Chiara Marmo <cmarmo>`<<.

*   Avoid overflows on Windows in
  "decomposition.IncrementalPCA.partial_fit()" for large "batch_size"
  and "n_samples" values. >>:pr:`17985`<< by >>:user:`Alan Butler
  <aldee153>`<< and >>:user:`Amanda Dsouza <amy12xx>`<<.

*   Fixed bug in "ensemble.MultinomialDeviance" where the average of
  logloss was incorrectly calculated as sum of logloss.
  >>:pr:`17694`<< by >>:user:`Markus Rempfler <rempfler>`<< and
  >>:user:`Tsutomu Kusanagi <t-kusanagi2>`<<.

*   Fixes "ensemble.StackingClassifier" and
  "ensemble.StackingRegressor" compatibility with estimators that do
  not define *n_features_in_*. >>:pr:`17357`<< by Thomas Fan.

*   Fixes bug in "feature_extraction.text.CountVectorizer" where
  sample order invariance was broken when *max_features* was set and
  features had the same count. >>:pr:`18016`<< by Thomas Fan, Roman
  Yurchak, and Joel Nothman.

*   "linear_model.lars_path()" does not overwrite *X* when

features had the same count. >>:pr:`18016`<< by Thomas Fan, Roman

  *X_copy=True* and *Gram='auto'*. >>:pr:`17914`<< by Thomas Fan.

*   Fixed a bug where "metrics.pairwise_distances()" would raise an
  error if "metric='seuclidean'" and "X" is not type "np.float64".
  >>:pr:`15730`<< by >>:user:`Forrest Koch <ForrestCKoch>`<<.

*   Fixed a bug in "metrics.mean_squared_error()" where the average of
  multiple RMSE values was incorrectly calculated as the root of the
  average of multiple MSE values. >>:pr:`17309`<< by >>:user:`Swier
  Heeres <swierh>`<<.

*   "pipeline.FeatureUnion" raises a deprecation warning when *None*
  is included in *transformer_list*. >>:pr:`17360`<< by Thomas Fan.

*   Fix "utils.estimator_checks.check_estimator()" so that all test
  cases support the *binary_only* estimator tag. >>:pr:`17812`<< by
  >>:user:`Bruno Charron <brcharron>`<<.

Version 0.23.1

**May 18 2020**

Changelog
=========

*   "cluster.KMeans" efficiency has been improved for very small
  datasets. In particular it cannot spawn idle threads any more.
  >>:pr:`17210`<< and >>:pr:`17235`<< by >>:user:`Jeremie du

datasets. In particular it cannot spawn idle threads any more.

*   Fixed a bug in "cluster.KMeans" where the sample weights provided
  by the user were modified in place. >>:pr:`17204`<< by

*   Fixed a bug in the *repr* of third-party estimators that use a
  ***kwargs* parameter in their constructor, when *changed_only* is
  True which is now the default. >>:pr:`17205`<< by Nicolas Hug.

Version 0.23.0

**May 12 2020**

lights_0_23_0.py*.

  memory.

Enforcing keyword-only arguments

In an effort to promote clear and non-ambiguous use of the library,
most constructor and function parameters are now expected to be passed

as keyword arguments (i.e. using the *param=value* syntax) instead of
positional. To ease the transition, a *FutureWarning* is raised if a
keyword-only parameter is used as positional. In version 1.0 (renaming
of 0.25), these parameters will be strictly keyword-only, and a
*TypeError* will be raised. >>:issue:`15005`<< by Joel Nothman, Adrin
Jalali, Thomas Fan, and Nicolas Hug. See SLEP009 for more details.

*   "ensemble.BaggingClassifier", "ensemble.BaggingRegressor", and
  "ensemble.IsolationForest".

*   "cluster.KMeans" with "algorithm="elkan"" and "algorithm="full"".

*   "cluster.Birch"

*   *compose.ColumnTransformer.get_feature_names*

*   "compose.ColumnTransformer.fit()"

*   "datasets.make_multilabel_classification()"

*   "decomposition.PCA" with *n_components='mle'*

*   "decomposition.NMF" and
  "decomposition.non_negative_factorization()" with float32 dtype
  input.

*   "decomposition.KernelPCA.inverse_transform()"

  "ensemble.HistGradientBoostingRegressor"

input.

*   "estimator_samples_" in "ensemble.BaggingClassifier",
  "ensemble.BaggingRegressor" and "ensemble.IsolationForest"

*   "ensemble.StackingClassifier" and "ensemble.StackingRegressor"
  with *sample_weight*

*   "gaussian_process.GaussianProcessRegressor"

*   "linear_model.RANSACRegressor" with "sample_weight".

*   "linear_model.RidgeClassifierCV"

*   "metrics.mean_squared_error()" with *squared* and
  *multioutput='raw_values'*.

*   "metrics.mutual_info_score()" with negative scores.

*   "metrics.confusion_matrix()" with zero length *y_true* and
  *y_pred*

*   "neural_network.MLPClassifier"

*   "preprocessing.StandardScaler" with *partial_fit* and sparse
  input.

*   "preprocessing.Normalizer" with norm='max'

*   Any model using the "svm.libsvm()" or the "svm.liblinear()"
  solver, including "svm.LinearSVC", "svm.LinearSVR", "svm.NuSVC",
  "svm.NuSVR", "svm.OneClassSVM", "svm.SVC", "svm.SVR",
  "linear_model.LogisticRegression".

*   "tree.DecisionTreeClassifier", "tree.ExtraTreeClassifier" and
  "ensemble.GradientBoostingClassifier" as well as "predict" method of
  "tree.DecisionTreeRegressor", "tree.ExtraTreeRegressor", and
  "ensemble.GradientBoostingRegressor" and read-only float32 input in
  "predict", "decision_path" and "predict_proba".

"predict", "decision_path" and "predict_proba".

Changelog
=========

*   "cluster.Birch" implementation of the predict method avoids high
  memory footprint by calculating the distances matrix using a chunked
  scheme. >>:pr:`16149`<< by >>:user:`Jeremie du Boisberranger
  <jeremiedbb>`<< and >>:user:`Alex Shacked <alexshacked>`<<.

*     The critical parts of "cluster.KMeans" have a more optimized
  implementation. Parallelism is now over the data instead of over
  initializations allowing better scalability. >>:pr:`11950`<< by

*   "cluster.KMeans" now supports sparse data when *solver = "elkan"*.
  >>:pr:`11950`<< by >>:user:`Jeremie du Boisberranger

*   "cluster.AgglomerativeClustering" has a faster and more memory
  efficient implementation of single linkage clustering.
  >>:pr:`11514`<< by >>:user:`Leland McInnes <lmcinnes>`<<.

*   "cluster.KMeans" with "algorithm="elkan"" now converges with
  "tol=0" as with the default "algorithm="full"". >>:pr:`16075`<< by
  >>:user:`Erich Schubert <kno10>`<<.

*   Fixed a bug in "cluster.Birch" where the *n_clusters* parameter
  could not have a *np.int64* type. >>:pr:`16484`<< by

*   "cluster.AgglomerativeCluClustering" add specific error when
  distance matrix is not square and *affinity=precomputed*.
  >>:pr:`16257`<< by >>:user:`Simona Maggio <simonamaggio>`<<.

*   The "n_jobs" parameter of "cluster.KMeans",
  "cluster.SpectralCoclustering" and "cluster.SpectralBiclustering" is
  deprecated. They now use OpenMP based parallelism. For more details
  on how to control the number of threads, please refer to our
  Parallelism notes. >>:pr:`11950`<< by >>:user:`Jeremie du

*   The "precompute_distances" parameter of "cluster.KMeans" is
  deprecated. It has no effect. >>:pr:`11950`<< by >>:user:`Jeremie du

*   The "random_state" parameter has been added to
  "cluster.AffinityPropagation". >>:pr:`16801`<< by
  >>:user:`rcwoolston`<< and >>:user:`Chiara Marmo <cmarmo>`<<.

*   "compose.ColumnTransformer" is now faster when working with
  dataframes and strings are used to specific subsets of data for
  transformers. >>:pr:`16431`<< by Thomas Fan.

*   "compose.ColumnTransformer" method "get_feature_names" now
  supports *'passthrough'* columns, with the feature name being either
  the column name for a dataframe, or *'xi'* for column index *i*.
  >>:pr:`14048`<< by >>:user:`Lewis Ball <lrjball>`<<.

supports *'passthrough'* columns, with the feature name being either

*   "compose.ColumnTransformer" method "get_feature_names" now returns
  correct results when one of the transformer steps applies on an
  empty list of columns >>:pr:`15963`<< by Roman Yurchak.

*   "compose.ColumnTransformer.fit()" will error when selecting a
  column name that is not unique in the dataframe. >>:pr:`16431`<< by

*   "datasets.fetch_openml()" has reduced memory usage because it no
  longer stores the full dataset text stream in memory.
  >>:pr:`16084`<< by Joel Nothman.

*   "datasets.fetch_california_housing()" now supports heterogeneous
  data using pandas by setting *as_frame=True*. >>:pr:`15950`<< by
  >>:user:`Stephanie Andrews <gitsteph>`<< and >>:user:`Reshama Shaikh
  <reshamas>`<<.

*   embedded dataset loaders "load_breast_cancer()",
  "load_diabetes()", "load_digits()", "load_iris()", "load_linnerud()"
  and "load_wine()" now support loading as a pandas "DataFrame" by
  setting *as_frame=True*. >>:pr:`15980`<< by >>:user:`wconnell`<< and
  >>:user:`Reshama Shaikh <reshamas>`<<.

*   Added "return_centers" parameter  in "datasets.make_blobs()",
  which can be used to return centers for each cluster.
  >>:pr:`15709`<< by >>:user:`shivamgargsya`<< and

which can be used to return centers for each cluster.

*   Functions "datasets.make_circles()" and "datasets.make_moons()"
  now accept two-element tuple. >>:pr:`15707`<< by >>:user:`Maciej J
  Mikulski <mjmikulski>`<<.

*   "datasets.make_multilabel_classification()" now generates
  *ValueError* for arguments *n_classes < 1* OR *length < 1*.
  >>:pr:`16006`<< by >>:user:`Rushabh Vasani <rushabh-v>`<<.

*   The *StreamHandler* was removed from *sklearn.logger* to avoid
  double logging of messages in common cases where a handler is
  attached to the root logger, and to follow the Python logging
  documentation recommendation for libraries to leave the log message
  handling to users and application code. >>:pr:`16451`<< by
  >>:user:`Christoph Deil <cdeil>`<<.

  "decomposition.non_negative_factorization()" now preserves float32
  dtype. >>:pr:`16280`<< by >>:user:`Jeremie du Boisberranger

*   "TruncatedSVD.transform()" is now faster on given sparse "csc"
  matrices. >>:pr:`16837`<< by >>:user:`wornbb`<<.

*   "decomposition.PCA" with a float *n_components* parameter, will
  exclusively choose the components that explain the variance greater
  than *n_components*. >>:pr:`15669`<< by >>:user:`Krishna Chaitanya
  <krishnachaitanya9>`<<

*   "decomposition.PCA" with *n_components='mle'* now correctly

than *n_components*. >>:pr:`15669`<< by >>:user:`Krishna Chaitanya

  handles small eigenvalues, and does not infer 0 as the correct
  number of components. >>:pr:`16224`<< by >>:user:`Lisa Schwetlick
  <lschwetlick>`<<, and >>:user:`Gelavizh Ahmadi <gelavizh1>`<< and
  >>:user:`Marija Vlajic Wheeler <marijavlajic>`<< and >>:pr:`16841`<<
  by Nicolas Hug.

*   "decomposition.KernelPCA" method "inverse_transform" now applies
  the correct inverse transform to the transformed data.
  >>:pr:`16655`<< by >>:user:`Lewis Ball <lrjball>`<<.

*   Fixed bug that was causing "decomposition.KernelPCA" to sometimes
  raise *invalid value encountered in multiply* during *fit*.
  >>:pr:`16718`<< by >>:user:`Gui Miotto <gui-miotto>`<<.

*   Added *n_components_* attribute to "decomposition.SparsePCA" and
  "decomposition.MiniBatchSparsePCA". >>:pr:`16981`<< by
  >>:user:`Mateusz Górski <Reksbril>`<<.

*    "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor" now support
  *sample_weight*. >>:pr:`14696`<< by Adrin Jalali and Nicolas Hug.

*   Early stopping in "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor" is now determined with a
  new *early_stopping* parameter instead of *n_iter_no_change*.
  Default value is 'auto', which enables early stopping if there are

"ensemble.HistGradientBoostingRegressor" is now determined with a
  at least 10,000 samples in the training set. >>:pr:`14516`<< by
  >>:user:`Johann Faouzi <johannfaouzi>`<<.

  "ensemble.HistGradientBoostingRegressor" now support monotonic
  constraints, useful when features are supposed to have a
  positive/negative effect on the target. >>:pr:`15582`<< by Nicolas
  Hug.

  "ensemble.VotingClassifier" and "ensemble.VotingRegressor".
  >>:pr:`16069`<< by >>:user:`Sam Bail <spbail>`<<, >>:user:`Hanna
  Bruce MacDonald <hannahbrucemacdonald>`<<, >>:user:`Reshama Shaikh
  <reshamas>`<<, and >>:user:`Chiara Marmo <cmarmo>`<<.

*   Fixed a bug in "ensemble.HistGradientBoostingClassifier" and
  "ensemble.HistGradientBoostingRegressor" that would not respect the
  *max_leaf_nodes* parameter if the criteria was reached at the same
  time as the *max_depth* criteria. >>:pr:`16183`<< by Nicolas Hug.

*    Changed the convention for *max_depth* parameter of
  "ensemble.HistGradientBoostingRegressor". The depth now corresponds
  to the number of edges to go from the root to the deepest leaf.
  Stumps (trees with one split) are now allowed. >>:pr:`16182`<< by
  >>:user:`Santhosh B <santhoshbala18>`<<

to the number of edges to go from the root to the deepest leaf.

*   Fixed a bug in "ensemble.BaggingClassifier",
  "ensemble.BaggingRegressor" and "ensemble.IsolationForest" where the
  attribute *estimators_samples_* did not generate the proper indices
  used during *fit*. >>:pr:`16437`<< by >>:user:`Jin-Hwan CHO
  <chofchof>`<<.

*   Fixed a bug in "ensemble.StackingClassifier" and
  "ensemble.StackingRegressor" where the *sample_weight* argument was
  not being passed to *cross_val_predict* when evaluating the base
  estimators on cross-validation folds to obtain the input to the meta
  estimator. >>:pr:`16539`<< by >>:user:`Bill DeRose <wderose>`<<.

*   Added additional option *loss="poisson"* to
  "ensemble.HistGradientBoostingRegressor", which adds Poisson
  deviance with log-link useful for modeling count data.
  >>:pr:`16692`<< by >>:user:`Christian Lorentzen <lorentzenchr>`<<

*   Fixed a bug where "ensemble.HistGradientBoostingRegressor" and
  "ensemble.HistGradientBoostingClassifier" would fail with multiple
  calls to fit when *warm_start=True*, *early_stopping=True*, and
  there is no validation set. >>:pr:`16663`<< by Thomas Fan.

*   "feature_extraction.text.CountVectorizer" now sorts features after
  pruning them by document frequency. This improves performances for

  datasets with large vocabularies combined with "min_df" or "max_df".
  >>:pr:`15834`<< by >>:user:`Santiago M. Mola <smola>`<<.

*   Added support for multioutput data in "feature_selection.RFE" and
  "feature_selection.RFECV". >>:pr:`16103`<< by >>:user:`Divyaprabha M
  <divyaprabha123>`<<.

*   Adds "feature_selection.SelectorMixin" back to public API.
  >>:pr:`16132`<< by >>:user:`trimeta`<<.

*   "gaussian_process.kernels.Matern()" returns the RBF kernel when
  "nu=np.inf". >>:pr:`15503`<< by >>:user:`Sam Dixon <sam-dixon>`<<.

*   Fixed bug in "gaussian_process.GaussianProcessRegressor" that
  caused predicted standard deviations to only be between 0 and 1 when
  WhiteKernel is not used. >>:pr:`15782`<< by >>:user:`plgreenLIRU`<<.

*   "impute.IterativeImputer" accepts both scalar and array-like
  inputs for "max_value" and "min_value". Array-like inputs allow a
  different max and min to be specified for each feature.
  >>:pr:`16403`<< by >>:user:`Narendra Mukherjee
  <narendramukherjee>`<<.

*   "impute.SimpleImputer", "impute.KNNImputer", and
  "impute.IterativeImputer" accepts pandas' nullable integer dtype
  with missing values. >>:pr:`16508`<< by Thomas Fan.

<narendramukherjee>`<<.

  'recursion' method for "ensemble.RandomForestRegressor" and
  "tree.DecisionTreeRegressor". >>:pr:`15864`<< by Nicolas Hug.

*   Added generalized linear models (GLM) with non normal error
  distributions, including "linear_model.PoissonRegressor",
  "linear_model.GammaRegressor" and "linear_model.TweedieRegressor"
  which use Poisson, Gamma and Tweedie distributions respectively.
  >>:pr:`14300`<< by >>:user:`Christian Lorentzen <lorentzenchr>`<<,
  Roman Yurchak, and Olivier Grisel.

*   Support of *sample_weight* in "linear_model.ElasticNet" and
  "linear_model.Lasso" for dense feature matrix *X*. >>:pr:`15436`<<
  by >>:user:`Christian Lorentzen <lorentzenchr>`<<.

  does not allocate a potentially large array to store dual
  coefficients for all hyperparameters during its *fit*, nor an array
  to store all error or LOO predictions unless *store_cv_values* is
  *True*. >>:pr:`15652`<< by >>:user:`Jérôme Dockès <jeromedockes>`<<.

*   "linear_model.LassoLars" and "linear_model.Lars" now support a

*True*. >>:pr:`15652`<< by >>:user:`Jérôme Dockès <jeromedockes>`<<.

  *jitter* parameter that adds random noise to the target. This might
  help with stability in some edge cases. >>:pr:`15179`<< by
  >>:user:`angelaambroz`<<.

*   Fixed a bug where if a *sample_weight* parameter was passed to the
  fit method of "linear_model.RANSACRegressor", it would not be passed
  to the wrapped *base_estimator* during the fitting of the final
  model. >>:pr:`15773`<< by >>:user:`Jeremy Alexandre <J-A16>`<<.

*   Add *best_score_* attribute to "linear_model.RidgeCV" and
  "linear_model.RidgeClassifierCV". >>:pr:`15655`<< by >>:user:`Jérôme
  Dockès <jeromedockes>`<<.

*   Fixed a bug in "linear_model.RidgeClassifierCV" to pass a specific
  scoring strategy. Before the internal estimator outputs score
  instead of predictions. >>:pr:`14848`<< by >>:user:`Venkatachalam N
  <venkyyuvy>`<<.

*   "linear_model.LogisticRegression" will now avoid an unnecessary
  iteration when *solver='newton-cg'* by checking for inferior or
  equal instead of strictly inferior for maximum of *absgrad* and
  *tol* in *utils.optimize._newton_cg*. >>:pr:`16266`<< by
  >>:user:`Rushabh Vasani <rushabh-v>`<<.

*   Deprecated public attributes *standard_coef_*,
  *standard_intercept_*, *average_coef_*, and *average_intercept_* in
  "linear_model.SGDClassifier", "linear_model.SGDRegressor",

*standard_intercept_*, *average_coef_*, and *average_intercept_* in
  "linear_model.PassiveAggressiveRegressor". >>:pr:`16261`<< by
  >>:user:`Carlos Brandt <chbrandt>`<<.

*     "linear_model.ARDRegression" is more stable and much faster when
  *n_samples > n_features*. It can now scale to hundreds of thousands
  of samples. The stability fix might imply changes in the number of
  non-zero coefficients and in the predicted output. >>:pr:`16849`<<

*   Fixed a bug in "linear_model.ElasticNetCV",
  "linear_model.MultiTaskElasticNetCV", "linear_model.LassoCV" and
  "linear_model.MultiTaskLassoCV" where fitting would fail when using
  joblib loky backend. >>:pr:`14264`<< by >>:user:`Jérémie du

*   Speed up "linear_model.MultiTaskLasso",
  "linear_model.MultiTaskLassoCV", "linear_model.MultiTaskElasticNet",
  "linear_model.MultiTaskElasticNetCV" by avoiding slower BLAS Level 2
  calls on small arrays >>:pr:`17021`<< by >>:user:`Alex Gramfort
  <agramfort>`<< and >>:user:`Mathurin Massias <mathurinm>`<<.

*   "metrics.pairwise.pairwise_distances_chunked()" now allows its
  "reduce_func" to not have a return value, enabling in-place
  operations. >>:pr:`16397`<< by Joel Nothman.

*   Fixed a bug in "metrics.mean_squared_error()" to not ignore
  argument *squared* when argument *multioutput='raw_values'*.

operations. >>:pr:`16397`<< by Joel Nothman.

  >>:pr:`16323`<< by >>:user:`Rushabh Vasani <rushabh-v>`<<

*   Fixed a bug in "metrics.mutual_info_score()" where negative scores
  could be returned. >>:pr:`16362`<< by Thomas Fan.

*   Fixed a bug in "metrics.confusion_matrix()" that would raise an
  error when *y_true* and *y_pred* were length zero and *labels* was
  not *None*. In addition, we raise an error when an empty list is
  given to the *labels* parameter. >>:pr:`16442`<< by >>:user:`Kyle
  Parsons <parsons-kyle-89>`<<.

*   Changed the formatting of values in
  "metrics.ConfusionMatrixDisplay.plot()" and
  "metrics.plot_confusion_matrix()" to pick the shorter format (either
  '2g' or 'd'). >>:pr:`16159`<< by >>:user:`Rick Mackenbach <Rick-
  Mackenbach>`<< and Thomas Fan.

*   From version 0.25, "metrics.pairwise.pairwise_distances()" will no
  longer automatically compute the "VI" parameter for Mahalanobis
  distance and the "V" parameter for seuclidean distance if "Y" is
  passed. The user will be expected to compute this parameter on the
  training data of their choice and pass it to *pairwise_distances*.
  >>:pr:`16993`<< by Joel Nothman.

  "model_selection.RandomizedSearchCV" yields stack trace information
  in fit failed warning messages in addition to previously emitted

  type and details. >>:pr:`15622`<< by >>:user:`Gregory Morse
  <GregoryMorse>`<<.

*   "model_selection.cross_val_predict()" supports
  *method="predict_proba"* when *y=None*. >>:pr:`15918`<< by
  >>:user:`Luca Kubin <lkubin>`<<.

*   "model_selection.fit_grid_point()" is deprecated in 0.23 and will
  be removed in 0.25. >>:pr:`16401`<< by >>:user:`Arie Pratama Sutiono
  <ariepratama>`<<

*   "multioutput.MultiOutputRegressor.fit()" and
  "multioutput.MultiOutputClassifier.fit()" now can accept
  *fit_params* to pass to the *estimator.fit* method of each step.
  >>:issue:`15953`<< >>:pr:`15959`<< by >>:user:`Ke Huang
  <huangk10>`<<.

*   "multioutput.RegressorChain" now supports *fit_params* for
  *base_estimator* during *fit*. >>:pr:`16111`<< by

*   A correctly formatted error message is shown in
  "naive_bayes.CategoricalNB" when the number of features in the input
  differs between *predict* and *fit*. >>:pr:`16090`<< by
  >>:user:`Madhura Jayaratne <madhuracj>`<<.

  has reduced memory footprint when using stochastic solvers, *'sgd'*

  or *'adam'*, and *shuffle=True*. >>:pr:`14075`<< by
  >>:user:`meyer89`<<.

*   Increases the numerical stability of the logistic loss function in
  "neural_network.MLPClassifier" by clipping the probabilities.
  >>:pr:`16117`<< by Thomas Fan.

*   "inspection.PartialDependenceDisplay" now exposes the deciles
  lines as attributes so they can be hidden or customized.
  >>:pr:`15785`<< by Nicolas Hug

*   argument *drop* of "preprocessing.OneHotEncoder" will now accept
  value 'if_binary' and will drop the first category of each feature
  with two categories. >>:pr:`16245`<< by >>:user:`Rushabh Vasani
  <rushabh-v>`<<.

*   "preprocessing.OneHotEncoder"'s *drop_idx_* ndarray can now
  contain *None*, where *drop_idx_[i] = None* means that no category
  is dropped for index *i*. >>:pr:`16585`<< by >>:user:`Chiara Marmo

*   "preprocessing.MaxAbsScaler", "preprocessing.MinMaxScaler",
  "preprocessing.StandardScaler", "preprocessing.PowerTransformer",
  "preprocessing.QuantileTransformer", "preprocessing.RobustScaler"
  now supports pandas' nullable integer dtype with missing values.
  >>:pr:`16508`<< by Thomas Fan.

*   "preprocessing.OneHotEncoder" is now faster at transforming.
  >>:pr:`15762`<< by Thomas Fan.

now supports pandas' nullable integer dtype with missing values.

*   Fix a bug in "preprocessing.StandardScaler" which was incorrectly
  computing statistics when calling *partial_fit* on sparse inputs.
  >>:pr:`16466`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   Fix a bug in "preprocessing.Normalizer" with norm='max', which was
  not taking the absolute value of the maximum values before
  normalizing the vectors. >>:pr:`16632`<< by >>:user:`Maura Pintor
  <Maupin1991>`<< and >>:user:`Battista Biggio <bbiggio>`<<.

*   "semi_supervised.LabelSpreading" and
  "semi_supervised.LabelPropagation" avoids divide by zero warnings
  when normalizing *label_distributions_*. >>:pr:`15946`<< by
  >>:user:`ngshya`<<.

*     Improved "libsvm" and "liblinear" random number generators used
  to randomly select coordinates in the coordinate descent algorithms.
  Platform-dependent C "rand()" was used, which is only able to
  generate numbers up to "32767" on windows platform (see this blog
  post) and also has poor randomization power as suggested by this
  presentation. It was replaced with C++11 "mt19937", a Mersenne
  Twister that correctly generates 31bits/63bits random numbers on all
  platforms. In addition, the crude "modulo" postprocessor used to get

Twister that correctly generates 31bits/63bits random numbers on all
  a random number in a bounded interval was replaced by the tweaked
  Lemire method as suggested by this blog post. Any model using the
  "svm.libsvm()" or the "svm.liblinear()" solver, including
  "svm.LinearSVC", "svm.LinearSVR", "svm.NuSVC", "svm.NuSVR",
  "svm.OneClassSVM", "svm.SVC", "svm.SVR",
  "linear_model.LogisticRegression", is affected. In particular users
  can expect a better convergence when the number of samples (LibSVM)
  or the number of features (LibLinear) is large. >>:pr:`13511`<< by
  >>:user:`Sylvain Marié <smarie>`<<.

*   Fix use of custom kernel not taking float entries such as string
  kernels in "svm.SVC" and "svm.SVR". Note that custom kennels are now
  expected to validate their input where they previously received
  valid numeric arrays. >>:pr:`11296`<< by Alexandre Gramfort and
  >>:user:`Georgi Peev <georgipeev>`<<.

*   "svm.SVR" and "svm.OneClassSVM" attributes, *probA_* and *probB_*,
  are now deprecated as they were not useful. >>:pr:`15558`<< by

*   "tree.plot_tree()" *rotate* parameter was unused and has been
  deprecated. >>:pr:`15806`<< by >>:user:`Chiara Marmo <cmarmo>`<<.

*   Fix support of read-only float32 array input in "predict",
  "decision_path" and "predict_proba" methods of
  "tree.DecisionTreeClassifier", "tree.ExtraTreeClassifier" and

  "ensemble.GradientBoostingRegressor". >>:pr:`16331`<< by
  >>:user:`Alexandre Batisse <batalex>`<<.

*   Estimators can now be displayed with a rich html representation.
  This can be enabled in Jupyter notebooks by setting
  *display='diagram'* in "set_config()". The raw html can be returned
  by using "utils.estimator_html_repr()". >>:pr:`14180`<< by Thomas
  Fan.

*   improve error message in "utils.validation.column_or_1d()".
  >>:pr:`15926`<< by >>:user:`Loïc Estève <lesteve>`<<.

*   add warning in "utils.check_array()" for pandas sparse DataFrame.
  >>:pr:`16021`<< by >>:user:`Rushabh Vasani <rushabh-v>`<<.

*   "utils.check_array()" now constructs a sparse matrix from a pandas
  DataFrame that contains only *SparseArray* columns. >>:pr:`16728`<<

*   "utils.validation.check_array()" supports pandas' nullable integer
  dtype with missing values when *force_all_finite* is set to *False*
  or *'allow-nan'* in which case the data is converted to floating
  point values where *pd.NA* values are replaced by *np.nan*. As a
  consequence, all "sklearn.preprocessing" transformers that accept

or *'allow-nan'* in which case the data is converted to floating
  numeric inputs with missing values represented as *np.nan* now also
  accepts being directly fed pandas dataframes with *pd.Int* or
  `pd.Uint** typed columns that use *pd.NA* as a missing value marker.

*   Passing classes to "utils.estimator_checks.check_estimator()" and
  "utils.estimator_checks.parametrize_with_checks()" is now
  deprecated, and support for classes will be removed in 0.24. Pass
  instances instead. >>:pr:`17032`<< by Nicolas Hug.

*   The private utility *_safe_tags* in *utils.estimator_checks* was
  removed, hence all tags should be obtained through
  *estimator._get_tags()*. Note that Mixins like *RegressorMixin* must
  come *before* base classes in the MRO for *_get_tags()* to work
  properly. >>:pr:`16950`<< by Nicolas Hug.

*   "utils.all_estimators()" now only returns public estimators.
  >>:pr:`15380`<< by Thomas Fan.

*   Adds a HTML representation of estimators to be shown in a jupyter
  notebook or lab. This visualization is acitivated by setting the
  *display* option in "sklearn.set_config()". >>:pr:`14180`<< by

*   "scikit-learn" now works with "mypy" without errors.
  >>:pr:`16726`<< by Roman Yurchak.

*   Most estimators now expose a *n_features_in_* attribute. This

Thomas Fan.

  attribute is equal to the number of features passed to the *fit*
  method. See SLEP010 for details. >>:pr:`16112`<< by Nicolas Hug.

*   Estimators now have a *requires_y* tags which is False by default
  except for estimators that inherit from
  *~sklearn.base.RegressorMixin* or *~sklearn.base.ClassifierMixin*.
  This tag is used to ensure that a proper error message is raised
  when y was expected but None was passed. >>:pr:`16622`<< by Nicolas
  Hug.

*   The default setting *print_changed_only* has been changed from
  False to True. This means that the *repr* of estimators is now more
  concise and only shows the parameters whose default value has been
  changed when printing an estimator. You can restore the previous
  behaviour by using *sklearn.set_config(print_changed_only=False)*.
  Also, note that it is always possible to quickly inspect the
  parameters of any estimator using *est.get_params(deep=False)*.
  >>:pr:`17061`<< by Nicolas Hug.

improvement of the project since version 0.22, including:

Abbie Popa, Adrin Jalali, Aleksandra Kocot, Alexandre Batisse,
Alexandre Gramfort, Alex Henrie, Alex Itkes, Alex Liang, alexshacked,

Alonso Silva Allende, Ana Casado, Andreas Mueller, Angela Ambroz,
Ankit810, Arie Pratama Sutiono, Arunav Konwar, Baptiste Maingret,
Benjamin Beier Liu, bernie gray, Bharathi Srinivasan, Bharat
Raghunathan, Bibhash Chandra Mitra, Brian Wignall, brigi, Brigitta
Sipőcz, Carlos H Brandt, CastaChick, castor, cgsavard, Chiara Marmo,
Chris Gregory, Christian Kastner, Christian Lorentzen, Corrie
Bartelheimer, Daniël van Gelder, Daphne, David Breuer, david-cortes,
dbauer9, Divyaprabha M, Edward Qian, Ekaterina Borovikova, ELNS, Emily
Taylor, Erich Schubert, Eric Leung, Evgeni Chasnovski, Fabiana,
Facundo Ferrín, Fan, Franziska Boenisch, Gael Varoquaux, Gaurav
Sharma, Geoffrey Bolmier, Georgi Peev, gholdman1, Gonthier Nicolas,
Gregory Morse, Gregory R. Lee, Guillaume Lemaitre, Gui Miotto, Hailey
Nguyen, Hanmin Qin, Hao Chun Chang, HaoYin, Hélion du Mas des
Bourboux, Himanshu Garg, Hirofumi Suzuki, huangk10, Hugo van Kemenade,
Hye Sung Jung, indecisiveuser, inderjeet, J-A16, Jérémie du
Boisberranger, Jin-Hwan CHO, JJmistry, Joel Nothman, Johann Faouzi,
Jon Haitz Legarreta Gorroño, Juan Carlos Alfaro Jiménez, judithabk6,
jumon, Kathryn Poole, Katrina Ni, Kesshi Jordan, Kevin Loftis, Kevin
Markham, krishnachaitanya9, Lam Gia Thuan, Leland McInnes, Lisa
Schwetlick, lkubin, Loic Esteve, lopusz, lrjball, lucgiffon,

lucyleeow, Lucy Liu, Lukas Kemkes, Maciej J Mikulski, Madhura
Jayaratne, Magda Zielinska, maikia, Mandy Gu, Manimaran, Manish
Aradwad, Maren Westermann, Maria, Mariana Meireles, Marie Douriez,
Marielle, Mateusz Górski, mathurinm, Matt Hall, Maura Pintor, mc4229,
meyer89, m.fab, Michael Shoemaker, Michał Słapek, Mina Naghshhnejad,
mo, Mohamed Maskani, Mojca Bertoncelj, narendramukherjee, ngshya,
Nicholas Won, Nicolas Hug, nicolasservel, Niklas, @nkish, Noa Tamir,
Oleksandr Pavlyk, olicairns, Oliver Urs Lenz, Olivier Grisel, parsons-
kyle-89, Paula, Pete Green, Pierre Delanoue, pspachtholz, Pulkit
Mehta, Qizhi  Jiang, Quang Nguyen, rachelcjordan, raduspaimoc, Reshama
Shaikh, Riccardo Folloni, Rick Mackenbach, Ritchie Ng, Roman
Feldbauer, Roman Yurchak, Rory Hartong-Redden, Rüdiger Busche, Rushabh
Vasani, Sambhav Kothari, Samesh Lakhotia, Samuel Duan, SanthoshBala18,
Santiago M. Mola, Sarat Addepalli, scibol, Sebastian Kießling,
SergioDSR, Sergul Aydore, Shiki-H, shivamgargsya, SHUBH CHATTERJEE,
Siddharth Gupta, simonamaggio, smarie, Snowhite, stareh, Stephen
Blystone, Stephen Marsh, Sunmi Yoon, SylvainLan, talgatomarov,
tamirlan1, th0rwas, theoptips, Thomas J Fan, Thomas Li, Thomas
Schmitt, Tim Nonner, Tim Vink, Tiphaine Viard, Tirth Patel, Titus

Christian, Tom Dupré la Tour, trimeta, Vachan D A, Vandana Iyer,
Venkatachalam N, waelbenamara, wconnell, wderose, wenliwyan, Windber,
wornbb, Yu-Hang "Maxin" Tang

Version 1.0.2

*   "cluster.Birch", "feature_selection.RFECV",
  "ensemble.RandomForestRegressor", "ensemble.RandomForestClassifier",
  "ensemble.GradientBoostingRegressor", and
  "ensemble.GradientBoostingClassifier" do not raise warning when
  fitted on a pandas DataFrame anymore. >>:pr:`21578`<< by Thomas Fan.

Changelog
=========

*   Fixed an infinite loop in "cluster.SpectralClustering()" by moving
  an iteration counter from try to except. >>:pr:`21271`<< by
  >>:user:`Tyler Martin <martintb>`<<.

*   "datasets.fetch_openml()" is now thread safe. Data is first
  downloaded to a temporary subfolder and then renamed.
  >>:pr:`21833`<< by >>:user:`Siavash Rezazadeh <siavrez>`<<.

*   Fixed the constraint on the objective function of
  "decomposition.SparsePCA" and "decomposition.MiniBatchSparsePCA" to
  be convex and match the referenced article. >>:pr:`19210`<< by

  now raise a "ValueError" when "bootstrap=False" and "max_samples" is
  not "None". >>:pr:`21295`<< >>:user:`Haoyin Xu <PSSF23>`<<.

"ensemble.ExtraTreesRegressor", and "ensemble.RandomTreesEmbedding"

*   Solve a bug in "ensemble.GradientBoostingClassifier" where the
  exponential loss was computing the positive gradient instead of the
  negative one. >>:pr:`22050`<< by >>:user:`Guillaume Lemaitre

*   Fixed "feature_selection.SelectFromModel" by improving support for
  base estimators that do not set *feature_names_in_*. >>:pr:`21991`<<

*   Fix a bug in "linear_model.RidgeClassifierCV" where the method
  *predict* was performing an *argmax* on the scores obtained from
  *decision_function* instead of returning the multilabel indicator
  matrix. >>:pr:`19869`<< by >>:user:`Guillaume Lemaitre

*   Fixed an unnecessary error when fitting "manifold.Isomap" with a
  precomputed dense distance matrix where the neighbors graph has
  multiple disconnected components. >>:pr:`21915`<< by Tom Dupre la
  Tour.

precomputed dense distance matrix where the neighbors graph has
  Tour.

*   All "sklearn.metrics.DistanceMetric" subclasses now correctly
  support read-only buffer attributes. This fixes a regression
  introduced in 1.0.0 with respect to 0.24.2. >>:pr:`21694`<< by

*   All "sklearn.metrics.MinkowskiDistance" now accepts a weight
  parameter that makes it possible to write code that behaves
  consistently both with scipy 1.8 and earlier versions. In turns this
  means that all neighbors-based estimators (except those that use
  *algorithm="kd_tree"*) now accept a weight parameter with
  *metric="minknowski"* to yield results that are always consistent
  with *scipy.spatial.distance.cdist*. >>:pr:`21741`<< by

*   "multiclass.OneVsRestClassifier.predict_proba()" does not error
  when fitted on constant integer targets. >>:pr:`21871`<< by Thomas
  Fan.

*   "neighbors.KDTree" and "neighbors.BallTree" correctly supports
  read-only buffer attributes. >>:pr:`21845`<< by Thomas Fan.

*   Fixes compatibility bug with NumPy 1.22 in
  "preprocessing.OneHotEncoder". >>:pr:`21517`<< by Thomas Fan.

*   Prevents "tree.plot_tree()" from drawing out of the boundary of

"preprocessing.OneHotEncoder". >>:pr:`21517`<< by Thomas Fan.

  the figure. >>:pr:`21917`<< by Thomas Fan.

*   Support loading pickles of decision tree models when the pickle
  has been generated on a platform with a different bitness. A typical
  example is to train and pickle the model on 64 bit machine and load
  the model on a 32 bit machine for prediction. >>:pr:`21552`<< by

*   "utils.estimator_html_repr()" now escapes all the estimator
  descriptions in the generated HTML. >>:pr:`21493`<< by

Version 1.0.1

**October 2021**

Changelog
=========

Fixed models

*   Non-fit methods in the following classes do not raise a
  UserWarning when fitted on DataFrames with valid feature names:
  "covariance.EllipticEnvelope", "ensemble.IsolationForest",
  "ensemble.AdaBoostClassifier", "neighbors.KNeighborsClassifier",
  "neighbors.KNeighborsRegressor",
  "neighbors.RadiusNeighborsRegressor". >>:pr:`21199`<< by Thomas Fan.

*   Fixed "calibration.CalibratedClassifierCV" to take into account
  *sample_weight* when computing the base estimator prediction when
  *ensemble=False*. >>:pr:`20638`<< by >>:user:`Julien Bohné
  <JulienB-78>`<<.

*   Fixed a bug in "calibration.CalibratedClassifierCV" with

*ensemble=False*. >>:pr:`20638`<< by >>:user:`Julien Bohné

  *method="sigmoid"* that was ignoring the *sample_weight* when
  computing the the Bayesian priors. >>:pr:`21179`<< by

*   Fixed a bug in "cluster.KMeans", ensuring reproducibility and
  equivalence between sparse and dense input. >>:pr:`21195`<< by

*   Fixed a bug that could produce a segfault in rare cases for
  "ensemble.HistGradientBoostingRegressor". >>:pr:`21130`<<

*   Compute *y_std* properly with multi-target in
  "sklearn.gaussian_process.GaussianProcessRegressor" allowing proper
  normalization in multi-target scene. >>:pr:`20761`<< by
  >>:user:`Patrick de C. T. R. Ferreira <patrickctrf>`<<.

*   Fixed an efficiency regression introduced in version 1.0.0 in the
  *transform* method of "feature_extraction.text.CountVectorizer"
  which no longer checks for uppercase characters in the provided
  vocabulary. >>:pr:`21251`<< by >>:user:`Jérémie du Boisberranger

*   Fixed a bug in "feature_extraction.CountVectorizer" and

vocabulary. >>:pr:`21251`<< by >>:user:`Jérémie du Boisberranger

  "feature_extraction.TfidfVectorizer" by raising an error when
  'min_idf' or 'max_idf' are floating-point numbers greater than 1.
  >>:pr:`20752`<< by >>:user:`Alek Lefebvre <AlekLefebvre>`<<.

*   Improves stability of "linear_model.LassoLars" for different
  versions of openblas. >>:pr:`21340`<< by Thomas Fan.

*   "linear_model.LogisticRegression" now raises a better error
  message when the solver does not support sparse matrices with int64
  indices. >>:pr:`21093`<< by Tom Dupre la Tour.

  "neighbors.RadiusNeighborsRegressor" with *metric="precomputed"*
  raises an error for *bsr* and *dok* sparse matrices in methods:
  *fit*, *kneighbors* and *radius_neighbors*, due to handling of
  explicit zeros in *bsr* and *dok* *sparse graph* formats.
  >>:pr:`21199`<< by Thomas Fan.

*   "pipeline.Pipeline.get_feature_names_out()" correctly passes
  feature names out from one step of a pipeline to the next.
  >>:pr:`21351`<< by Thomas Fan.

*   "svm.SVC" and "svm.SVR" check for an inconsistency in its internal
  representation and raise an error instead of segfaulting. This fix

>>:pr:`21351`<< by Thomas Fan.

  also resolves CVE-2020-28975. >>:pr:`21336`<< by Thomas Fan.

*   "utils.validation._check_sample_weight()" can perform a non-
  negativity check on the sample weights. It can be turned on using
  the only_non_negative bool parameter. Estimators that check for non-
  negative weights are updated: "linear_model.LinearRegression()"
  (here the previous error message was misleading),
  "ensemble.AdaBoostClassifier()", "ensemble.AdaBoostRegressor()",
  "neighbors.KernelDensity()". >>:pr:`20880`<< by >>:user:`Guillaume
  Lemaitre <glemaitre>`<< and >>:user:`András Simon <simonandras>`<<.

*   Solve a bug in "if_delegate_has_method()" where the underlying
  check for an attribute did not work with NumPy arrays.
  >>:pr:`21145`<< by >>:user:`Zahlii <Zahlii>`<<.

*   Fitting an estimator on a dataset that has no feature names, that
  was previously fitted on a dataset with feature names no longer
  keeps the old feature names stored in the *feature_names_in_*
  attribute. >>:pr:`21389`<< by >>:user:`Jérémie du Boisberranger

Version 1.0.0

**September 2021**

lights_1_0_0.py*.

  memory.

Version 1.0.0 of scikit-learn requires python 3.7+, numpy 1.14.6+ and
scipy 1.1.0+. Optional minimal dependency is matplotlib 2.2.2+.

most constructor and function parameters must now be passed as keyword
arguments (i.e. using the *param=value* syntax) instead of positional.
If a keyword-only parameter is used as positional, a *TypeError* is
now raised. >>:issue:`15005`<< >>:pr:`20002`<< by Joel Nothman, Adrin
Jalali, Thomas Fan, Nicolas Hug, and Tom Dupre la Tour. See SLEP009
for more details.

*   "manifold.TSNE" now avoids numerical underflow issues during
  affinity matrix computation.

*   "manifold.Isomap" now connects disconnected components of the
  neighbors graph along some minimum distance pairs, instead of
  changing every infinite distances to zero.

*   The splitting criterion of "tree.DecisionTreeClassifier" and
  "tree.DecisionTreeRegressor" can be impacted by a fix in the
  handling of rounding errors. Previously some extra spurious splits
  could occur.

*   "model_selection.train_test_split()" with a *stratify* parameter
  and "model_selection.StratifiedShuffleSplit" may lead to slightly
  different results.

Changelog
=========

*   The option for using the squared error via "loss" and "criterion"
  parameters was made more consistent. The preferred way is by setting
  the value to *"squared_error"*. Old option names are still valid,
  produce the same models, but are deprecated and will be removed in

the value to *"squared_error"*. Old option names are still valid,
  version 1.2. >>:pr:`19310`<< by >>:user:`Christian Lorentzen

  * For "ensemble.ExtraTreesRegressor", *criterion="mse"* is
    deprecated, use *"squared_error"* instead which is now the
    default.

  * For "ensemble.GradientBoostingRegressor", *loss="ls"* is
    default.

  * For "ensemble.RandomForestRegressor", *criterion="mse"* is
    default.

  * For "ensemble.HistGradientBoostingRegressor",
    *loss="least_squares"* is deprecated, use *"squared_error"*
    instead which is now the default.

  * For "linear_model.RANSACRegressor", *loss="squared_loss"* is
    deprecated, use *"squared_error"* instead.

  * For "linear_model.SGDRegressor", *loss="squared_loss"* is
    default.

  * For "tree.DecisionTreeRegressor", *criterion="mse"* is deprecated,
    use *"squared_error"* instead which is now the default.

  * For "tree.ExtraTreeRegressor", *criterion="mse"* is deprecated,

*   The option for using the absolute error via "loss" and "criterion"
  the value to *"absolute_error"*. Old option names are still valid,

parameters was made more consistent. The preferred way is by setting
  version 1.2. >>:pr:`19733`<< by >>:user:`Christian Lorentzen

  * For "ensemble.ExtraTreesRegressor", *criterion="mae"* is
    deprecated, use *"absolute_error"* instead.

  * For "ensemble.GradientBoostingRegressor", *loss="lad"* is

  * For "ensemble.RandomForestRegressor", *criterion="mae"* is

    *loss="least_absolute_deviation"* is deprecated, use
    *"absolute_error"* instead.

  * For "linear_model.RANSACRegressor", *loss="absolute_loss"* is
    deprecated, use *"absolute_error"* instead which is now the
    default.

  * For "tree.DecisionTreeRegressor", *criterion="mae"* is deprecated,
    use *"absolute_error"* instead.

  * For "tree.ExtraTreeRegressor", *criterion="mae"* is deprecated,

*   *np.matrix* usage is deprecated in 1.0 and will raise a
  *TypeError* in 1.2. >>:pr:`20165`<< by Thomas Fan.

*   *get_feature_names_out* has been added to the transformer API to
  get the names of the output features. *get_feature_names* has in
  turn been deprecated. >>:pr:`18444`<< by Thomas Fan.

*   All estimators store *feature_names_in_* when fitted on pandas

get the names of the output features. *get_feature_names* has in

  Dataframes. These feature names are compared to names seen in
  non-*fit* methods, e.g. *transform* and will raise a *FutureWarning*
  if they are not consistent. These "FutureWarning" s will become
  "ValueError" s in 1.2. >>:pr:`18010`<< by Thomas Fan.

*   "config_context()" is now threadsafe. >>:pr:`18736`<< by Thomas
  Fan.

*   "calibration.CalibrationDisplay()" added to plot calibration
  curves. >>:pr:`17443`<< by >>:user:`Lucy Liu <lucyleeow>`<<.

*   The "predict" and "predict_proba" methods of
  "calibration.CalibratedClassifierCV" can now properly be used on
  prefitted pipelines. >>:pr:`19641`<< by >>:user:`Alek Lefebvre
  <AlekLefebvre>`<<.

*   Fixed an error when using a "ensemble.VotingClassifier" as
  *base_estimator* in "calibration.CalibratedClassifierCV".
  >>:pr:`20087`<< by >>:user:`Clément Fauchereau <clement-f>`<<.

*   The ""k-means++"" initialization of "cluster.KMeans" and
  "cluster.MiniBatchKMeans" is now faster, especially in multicore
  settings. >>:pr:`19002`<< by >>:user:`Jon Crall <Erotemic>`<< and

*   "cluster.KMeans" with *algorithm='elkan'* is now faster in
  multicore settings. >>:pr:`19052`<< by >>:user:`Yusuke Nagasaka

  <YusukeNagasaka>`<<.

*   "cluster.MiniBatchKMeans" is now faster in multicore settings.
  >>:pr:`17622`<< by >>:user:`Jérémie du Boisberranger

*   "cluster.OPTICS" can now cache the output of the computation of
  the tree, using the *memory* parameter.  >>:pr:`19024`<< by
  >>:user:`Frankie Robertson <frankier>`<<.

*   The *predict* and *fit_predict* methods of
  "cluster.AffinityPropagation" now accept sparse data type for input
  data. >>:pr:`20117`<< by >>:user:`Venkatachalam Natchiappan
  <venkyyuvy>`<<

*   Fixed a bug in "cluster.MiniBatchKMeans" where the sample weights
  were partially ignored when the input is sparse. >>:pr:`17622`<< by

*   Improved convergence detection based on center change in
  "cluster.MiniBatchKMeans" which was almost never achievable.

*   "cluster.AgglomerativeClustering" now supports readonly memory-
  mapped datasets. >>:pr:`19883`<< by >>:user:`Julien Jerphanion

*   "cluster.AgglomerativeClustering" correctly connects components
  when connectivity and affinity are both precomputed and the number
  of connected components is greater than 1. >>:pr:`20597`<< by Thomas
  Fan.

when connectivity and affinity are both precomputed and the number
  Fan.

*   "cluster.FeatureAgglomeration" does not accept a "**params" kwarg
  in the "fit" function anymore, resulting in a more concise error
  message. >>:pr:`20899`<< by >>:user:`Adam Li <adam2392>`<<.

  equivalence between sparse and dense input. >>:pr:`20200`<< by

*   "cluster.Birch" attributes, *fit_* and *partial_fit_*, are
  deprecated and will be removed in 1.2. >>:pr:`19297`<< by Thomas
  Fan.

*   the default value for the *batch_size* parameter of
  "cluster.MiniBatchKMeans" was changed from 100 to 1024 due to
  efficiency reasons. The *n_iter_* attribute of
  "cluster.MiniBatchKMeans" now reports the number of started epochs
  and the *n_steps_* attribute reports the number of mini batches
  processed. >>:pr:`17622`<< by >>:user:`Jérémie du Boisberranger

*   "cluster.spectral_clustering()" raises an improved error when
  passed a *np.matrix*. >>:pr:`20560`<< by Thomas Fan.

*   "compose.ColumnTransformer" now records the output of each
  transformer in *output_indices_*. >>:pr:`18393`<< by >>:user:`Luca
  Bittarello <lbittarello>`<<.

*   "compose.ColumnTransformer" now allows DataFrame input to have its
  columns appear in a changed order in *transform*. Further, columns

Bittarello <lbittarello>`<<.

  that are dropped will not be required in transform, and additional
  columns will be ignored if *remainder='drop'*. >>:pr:`19263`<< by

*   Adds ***predict_params* keyword argument to
  "compose.TransformedTargetRegressor.predict()" that passes keyword
  argument to the regressor. >>:pr:`19244`<< by >>:user:`Ricardo
  <ricardojnf>`<<.

*   *compose.ColumnTransformer.get_feature_names* supports non-string
  feature names returned by any of its transformers. However, note
  that "get_feature_names" is deprecated, use "get_feature_names_out"
  instead. >>:pr:`18459`<< by >>:user:`Albert Villanova del Moral
  <albertvillanova>`<< and >>:user:`Alonso Silva Allende
  <alonsosilvaallende>`<<.

*   "compose.TransformedTargetRegressor" now takes nD targets with an
  adequate transformer. >>:pr:`18898`<< by >>:user:`Oras Phongpanagnam
  <panangam>`<<.

*   Adds *verbose_feature_names_out* to "compose.ColumnTransformer".
  This flag controls the prefixing of feature names out in
  *get_feature_names_out*. >>:pr:`18444`<< and >>:pr:`21080`<< by

*   Adds arrays check to "covariance.ledoit_wolf()" and
  "covariance.ledoit_wolf_shrinkage()". >>:pr:`20416`<< by
  >>:user:`Hugo Defois <defoishugo>`<<.

*   Deprecates the following keys in *cv_results_*: *'mean_score'*,

"covariance.ledoit_wolf_shrinkage()". >>:pr:`20416`<< by

  *'std_score'*, and *'split(k)_score'* in favor of
  *'mean_test_score'* *'std_test_score'*, and *'split(k)_test_score'*.
  >>:pr:`20583`<< by Thomas Fan.

*   "datasets.fetch_openml()" now supports categories with missing
  values when returning a pandas dataframe. >>:pr:`19365`<< by Thomas
  Fan and >>:user:`Amanda Dsouza <amy12xx>`<< and >>:user:`EL-ATEIF
  Sara <elateifsara>`<<.

*   "datasets.fetch_kddcup99()" raises a better message when the
  cached file is invalid. >>:pr:`19669`<< Thomas Fan.

*   Replace usages of "__file__" related to resource file I/O with
  "importlib.resources" to avoid the assumption that these resource
  files (e.g. "iris.csv") already exist on a filesystem, and by
  extension to enable compatibility with tools such as "PyOxidizer".
  >>:pr:`20297`<< by >>:user:`Jack Liu <jackzyliu>`<<.

*   Shorten data file names in the openml tests to better support
  installing on Windows and its default 260 character limit on file
  names. >>:pr:`20209`<< by Thomas Fan.

*   "datasets.fetch_kddcup99()" returns dataframes when
  *return_X_y=True* and *as_frame=True*. >>:pr:`19011`<< by Thomas
  Fan.

*   Deprecates "datasets.load_boston()" in 1.0 and it will be removed
  in 1.2. Alternative code snippets to load similar datasets are

Fan.

  provided. Please report to the docstring of the function for
  details. >>:pr:`20729`<< by Guillaume Lemaitre.

*   added a new approximate solver (randomized SVD, available with
  *eigen_solver='randomized'*) to "decomposition.KernelPCA". This
  significantly accelerates computation when the number of samples is
  much larger than the desired number of components. >>:pr:`12069`<<
  by >>:user:`Sylvain Marié <smarie>`<<.

*   Fixes incorrect multiple data-conversion warnings when clustering
  boolean data. >>:pr:`19046`<< by >>:user:`Surya Prakash
  <jdsurya>`<<.

*   Fixed "dict_learning()", used by
  "decomposition.DictionaryLearning", to ensure determinism of the
  output. Achieved by flipping signs of the SVD output which is used
  to initialize the code. >>:pr:`18433`<< by >>:user:`Bruno Charron
  <brcharron>`<<.

*   Fixed a bug in "decomposition.MiniBatchDictionaryLearning",
  "decomposition.MiniBatchSparsePCA" and
  "decomposition.dict_learning_online()" where the update of the
  dictionary was incorrect. >>:pr:`19198`<< by >>:user:`Jérémie du

*   Fixed a bug in "decomposition.DictionaryLearning",
  "decomposition.SparsePCA",
  "decomposition.MiniBatchSparsePCA", "decomposition.dict_learning()"

  and "decomposition.dict_learning_online()" where the restart of
  unused atoms during the dictionary update was not working as
  expected. >>:pr:`19198`<< by >>:user:`Jérémie du Boisberranger

*   In "decomposition.DictionaryLearning",
  "decomposition.dict_learning()" and
  "decomposition.dict_learning_online()", *transform_alpha* will be
  equal to *alpha* instead of 1.0 by default starting from version 1.2
  >>:pr:`19159`<< by >>:user:`Benoît Malézieux <bmalezieux>`<<.

*   Rename variable names in "KernelPCA" to improve readability.
  *lambdas_* and *alphas_* are renamed to *eigenvalues_* and
  *eigenvectors_*, respectively. *lambdas_* and *alphas_* are
  deprecated and will be removed in 1.2. >>:pr:`19908`<< by
  >>:user:`Kei Ishikawa <kstoneriv3>`<<.

*   The *alpha* and *regularization* parameters of "decomposition.NMF"
  and "decomposition.non_negative_factorization()" are deprecated and
  will be removed in 1.2. Use the new parameters *alpha_W* and
  *alpha_H* instead. >>:pr:`20512`<< by >>:user:`Jérémie du

*   Attribute *n_features_in_* in "dummy.DummyRegressor" and
  "dummy.DummyRegressor" is deprecated and will be removed in 1.2.
  >>:pr:`20960`<< by Thomas Fan.

*   "HistGradientBoostingClassifier" and
  "HistGradientBoostingRegressor" take cgroups quotas into account
  when deciding the number of threads used by OpenMP. This avoids
  performance problems caused by over-subscription when using those
  classes in a docker container for instance. >>:pr:`20477`<< by

  "HistGradientBoostingRegressor" are no longer experimental. They are
  now considered stable and are subject to the same deprecation cycles
  as all other estimators. >>:pr:`19799`<< by Nicolas Hug.

*   Improve the HTML rendering of the "ensemble.StackingClassifier"
  and "ensemble.StackingRegressor". >>:pr:`19564`<< by Thomas Fan.

*   Added Poisson criterion to "ensemble.RandomForestRegressor".
  >>:pr:`19836`<< by >>:user:`Brian Sun <bsun94>`<<.

*   Do not allow to compute out-of-bag (OOB) score in
  "ensemble.RandomForestClassifier" and
  "ensemble.ExtraTreesClassifier" with multiclass-multioutput target
  since scikit-learn does not provide any metric supporting this type
  of target. Additional private refactoring was performed.
  >>:pr:`19162`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   Improve numerical precision for weights boosting in
  "ensemble.AdaBoostClassifier" and "ensemble.AdaBoostRegressor" to

>>:pr:`19162`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

  avoid underflows. >>:pr:`10096`<< by >>:user:`Fenil Suchak
  <fenilsuchak>`<<.

*   Fixed the range of the argument "max_samples" to be "(0.0, 1.0]"
  in "ensemble.RandomForestClassifier",
  "ensemble.RandomForestRegressor", where *max_samples=1.0* is
  interpreted as using all *n_samples* for bootstrapping.
  >>:pr:`20159`<< by >>:user:`murata-yu`<<.

*   Fixed a bug in "ensemble.AdaBoostClassifier" and
  "ensemble.AdaBoostRegressor" where the *sample_weight* parameter got
  overwritten during *fit*. >>:pr:`20534`<< by >>:user:`Guillaume

*   Removes *tol=None* option in
  "ensemble.HistGradientBoostingRegressor". Please use *tol=0* for the
  same behavior. >>:pr:`19296`<< by Thomas Fan.

*   Fixed a bug in "feature_extraction.text.HashingVectorizer" where
  some input strings would result in negative indices in the
  transformed data. >>:pr:`19035`<< by >>:user:`Liu Yu
  <ly648499246>`<<.

*   Fixed a bug in "feature_extraction.DictVectorizer" by raising an
  error with unsupported value type. >>:pr:`19520`<< by >>:user:`Jeff
  Zhao <kamiyaa>`<<.

*   Fixed a bug in "feature_extraction.image.img_to_graph()" and
  "feature_extraction.image.grid_to_graph()" where singleton connected

Zhao <kamiyaa>`<<.

  components were not handled properly, resulting in a wrong vertex
  indexing. >>:pr:`18964`<< by Bertrand Thirion.

*   Raise a warning in "feature_extraction.text.CountVectorizer" with
  *lowercase=True* when there are vocabulary entries with uppercase
  characters to avoid silent misses in the resulting feature vectors.
  >>:pr:`19401`<< by >>:user:`Zito Relova <zitorelova>`<<

*   "feature_selection.r_regression()" computes Pearson's R
  correlation coefficients between the features and the target.
  >>:pr:`17169`<< by >>:user:`Dmytro Lituiev <DSLituiev>`<< and

*   "feature_selection.RFE.fit()" accepts additional estimator
  parameters that are passed directly to the estimator's *fit* method.
  >>:pr:`20380`<< by >>:user:`Iván Pulido <ijpulidos>`<<,
  >>:user:`Felipe Bidu <fbidu>`<<, >>:user:`Gil Rutter <g-rutter>`<<,
  and >>:user:`Adrin Jalali <adrinjalali>`<<.

*   Fix a bug in "isotonic.isotonic_regression()" where the
  *sample_weight* passed by a user were overwritten during "fit".
  >>:pr:`20515`<< by >>:user:`Carsten Allefeld <allefeld>`<<.

*   Change "feature_selection.SequentialFeatureSelector()" to allow
  for unsupervised modelling so that the *fit* signature need not do
  any *y* validation and allow for *y=None*. >>:pr:`19568`<< by

  >>:user:`Shyam Desai <ShyamDesai>`<<.

*   Raises an error in "feature_selection.VarianceThreshold" when the
  variance threshold is negative. >>:pr:`20207`<< by >>:user:`Tomohiro
  Endo <europeanplaice>`<<

*   Deprecates *grid_scores_* in favor of split scores in
  *cv_results_* in "feature_selection.RFECV". *grid_scores_* will be
  removed in version 1.2. >>:pr:`20161`<< by >>:user:`Shuhei Kayawari
  <wowry>`<< and >>:user:`arka204`<<.

*   Add *max_samples* parameter in
  "inspection.permutation_importance()". It enables to draw a subset
  of the samples to compute the permutation importance. This is useful
  to keep the method tractable when evaluating feature importance on
  large datasets. >>:pr:`20431`<< by >>:user:`Oliver Pfaffel
  <o1iv3r>`<<.

*   Add kwargs to format ICE and PD lines separately in partial
  dependence plots "inspection.plot_partial_dependence()" and
  "inspection.PartialDependenceDisplay.plot()". >>:pr:`19428`<< by
  >>:user:`Mehdi Hamoumi <mhham>`<<.

*   Allow multiple scorers input to
  "inspection.permutation_importance()". >>:pr:`19411`<< by
  >>:user:`Simona Maggio <simonamaggio>`<<.

*   "inspection.PartialDependenceDisplay" exposes a class method:
  "from_estimator()". "inspection.plot_partial_dependence()" is

>>:user:`Simona Maggio <simonamaggio>`<<.

  deprecated in favor of the class method and will be removed in 1.2.
  >>:pr:`20959`<< by Thomas Fan.

*   Fix a bug in "kernel_approximation.Nystroem" where the attribute
  *component_indices_* did not correspond to the subset of sample
  indices used to generate the approximated kernel. >>:pr:`20554`<< by
  >>:user:`Xiangyin Kong <kxytim>`<<.

*   Added "linear_model.QuantileRegressor" which implements linear
  quantile regression with L1 penalty. >>:pr:`9978`<< by
  >>:user:`David Dale <avidale>`<< and >>:user:`Christian Lorentzen

*   The new "linear_model.SGDOneClassSVM" provides an SGD
  implementation of the linear One-Class SVM. Combined with kernel
  approximation techniques, this implementation approximates the
  solution of a kernelized One Class SVM while benefitting from a
  linear complexity in the number of samples. >>:pr:`10027`<< by

*   Added *sample_weight* parameter to "linear_model.LassoCV" and
  "linear_model.ElasticNetCV". >>:pr:`16449`<< by >>:user:`Christian

*   Added new solver *lbfgs* (available with *solver="lbfgs"*) and
  *positive* argument to "linear_model.Ridge". When *positive* is set

  to *True*, forces the coefficients to be positive (only supported by
  *lbfgs*). >>:pr:`20231`<< by >>:user:`Toshihiro Nakae <tnakae>`<<.

*   The implementation of "linear_model.LogisticRegression" has been
  optimised for dense matrices when using *solver='newton-cg'* and
  *multi_class!='multinomial'*. >>:pr:`19571`<< by >>:user:`Julien

*   *fit* method preserves dtype for numpy.float32 in
  "linear_model.Lars", "linear_model.LassoLars",
  "linear_model.LassoLars", "linear_model.LarsCV" and
  "linear_model.LassoLarsCV". >>:pr:`20155`<< by >>:user:`Takeshi Oura

*   Validate user-supplied gram matrix passed to linear models via the
  *precompute* argument. >>:pr:`19004`<< by >>:user:`Adam Midvidy
  <amidvidy>`<<.

*   "linear_model.ElasticNet.fit()" no longer modifies *sample_weight*
  in place. >>:pr:`19055`<< by Thomas Fan.

*   "linear_model.Lasso" and "linear_model.ElasticNet" no longer have
  a *dual_gap_* not corresponding to their objective. >>:pr:`19172`<<
  by >>:user:`Mathurin Massias <mathurinm>`<<

*   *sample_weight* are now fully taken into account in linear models
  when *normalize=True* for both feature centering and feature
  scaling. >>:pr:`19426`<< by >>:user:`Alexandre Gramfort
  <agramfort>`<< and >>:user:`Maria Telenczuk <maikia>`<<.

when *normalize=True* for both feature centering and feature

*   Points with residuals equal to  "residual_threshold" are now
  considered as inliers for "linear_model.RANSACRegressor". This
  allows fitting a model perfectly on some datasets when
  *residual_threshold=0*. >>:pr:`19499`<< by >>:user:`Gregory Strubel
  <gregorystrubel>`<<.

*   Sample weight invariance for "linear_model.Ridge" was fixed in
  >>:pr:`19616`<< by >>:user:`Oliver Grisel <ogrisel>`<< and

*   The dictionary *params* in "linear_model.enet_path()" and
  "linear_model.lasso_path()" should only contain parameter of the
  coordinate descent solver. Otherwise, an error will be raised.
  >>:pr:`19391`<< by >>:user:`Shao Yang Hong <hongshaoyang>`<<.

*   Raise a warning in "linear_model.RANSACRegressor" that from
  version 1.2, *min_samples* need to be set explicitly for models
  other than "linear_model.LinearRegression". >>:pr:`19390`<< by
  >>:user:`Shao Yang Hong <hongshaoyang>`<<.

*  : The parameter "normalize" of "linear_model.LinearRegression" is
  deprecated and will be removed in 1.2. Motivation for this
  deprecation: "normalize" parameter did not take any effect if
  "fit_intercept" was set to False and therefore was deemed confusing.
  The behavior of the deprecated "LinearModel(normalize=True)" can be

"fit_intercept" was set to False and therefore was deemed confusing.
  reproduced with a "Pipeline" with "LinearModel" (where "LinearModel"
  is "LinearRegression", "Ridge", "RidgeClassifier", "RidgeCV" or
  "RidgeClassifierCV") as follows:
  "make_pipeline(StandardScaler(with_mean=False), LinearModel())". The
  "normalize" parameter in "LinearRegression" was deprecated in
  >>:pr:`17743`<< by >>:user:`Maria Telenczuk <maikia>`<< and
  >>:user:`Alexandre Gramfort <agramfort>`<<. Same for "Ridge",
  "RidgeClassifier", "RidgeCV", and "RidgeClassifierCV", in:
  >>:pr:`17772`<< by >>:user:`Maria Telenczuk <maikia>`<< and
  >>:user:`Alexandre Gramfort <agramfort>`<<. Same for
  "BayesianRidge", "ARDRegression" in: >>:pr:`17746`<< by
  >>:user:`Maria Telenczuk <maikia>`<<. Same for "Lasso", "LassoCV",
  "ElasticNet", "ElasticNetCV", "MultiTaskLasso", "MultiTaskLassoCV",
  "MultiTaskElasticNet", "MultiTaskElasticNetCV", in: >>:pr:`17785`<<
  by >>:user:`Maria Telenczuk <maikia>`<< and >>:user:`Alexandre
  Gramfort <agramfort>`<<.

*   The "normalize" parameter of "OrthogonalMatchingPursuit" and
  "OrthogonalMatchingPursuitCV" will default to False in 1.2 and will
  be removed in 1.4. >>:pr:`17750`<< by >>:user:`Maria Telenczuk
  <maikia>`<< and >>:user:`Alexandre Gramfort <agramfort>`<<. Same for
  "Lars" "LarsCV" "LassoLars" "LassoLarsCV" "LassoLarsIC", in
  >>:pr:`17769`<< by >>:user:`Maria Telenczuk <maikia>`<< and

<maikia>`<< and >>:user:`Alexandre Gramfort <agramfort>`<<. Same for
  >>:user:`Alexandre Gramfort <agramfort>`<<.

*   Keyword validation has moved from *__init__* and *set_params* to
  *fit* for the following estimators conforming to scikit-learn's
  conventions: "SGDClassifier", "SGDRegressor", "SGDOneClassSVM",
  "PassiveAggressiveClassifier", and "PassiveAggressiveRegressor".
  >>:pr:`20683`<< by Guillaume Lemaitre.

*   Implement *'auto'* heuristic for the *learning_rate* in
  "manifold.TSNE". It will become default in 1.2. The default
  initialization will change to *pca* in 1.2. PCA initialization will
  be scaled to have standard deviation 1e-4 in 1.2. >>:pr:`19491`<< by
  >>:user:`Dmitry Kobak <dkobak>`<<.

*   Change numerical precision to prevent underflow issues during
  affinity matrix computation for "manifold.TSNE". >>:pr:`19472`<< by

*   "manifold.Isomap" now uses *scipy.sparse.csgraph.shortest_path* to
  compute the graph shortest path. It also connects disconnected
  components of the neighbors graph along some minimum distance pairs,
  instead of changing every infinite distances to zero.
  >>:pr:`20531`<< by Roman Yurchak and Tom Dupre la Tour.

*   Decrease the numerical default tolerance in the lobpcg call in
  "manifold.spectral_embedding()" to prevent numerical instability.

>>:pr:`20531`<< by Roman Yurchak and Tom Dupre la Tour.

  >>:pr:`21194`<< by >>:user:`Andrew Knyazev <lobpcg>`<<.

*   "metrics.mean_pinball_loss()" exposes the pinball loss for
  quantile regression. >>:pr:`19415`<< by >>:user:`Xavier Dupré
  <sdpython>`<< and >>:user:`Oliver Grisel <ogrisel>`<<.

*   "metrics.d2_tweedie_score()" calculates the D^2 regression score
  for Tweedie deviances with power parameter "power". This is a
  generalization of the *r2_score* and can be interpreted as
  percentage of Tweedie deviance explained. >>:pr:`17036`<< by

*    "metrics.mean_squared_log_error()" now supports *squared=False*.
  >>:pr:`20326`<< by >>:user:`Uttam kumar <helper-uttam>`<<.

*   Improved speed of "metrics.confusion_matrix()" when labels are
  integral. >>:pr:`9843`<< by >>:user:`Jon Crall <Erotemic>`<<.

*   A fix to raise an error in "metrics.hinge_loss()" when
  "pred_decision" is 1d whereas it is a multiclass classification or
  when "pred_decision" parameter is not consistent with the "labels"
  parameter. >>:pr:`19643`<< by >>:user:`Pierre Attard
  <PierreAttard>`<<.

*   "metrics.ConfusionMatrixDisplay.plot()" uses the correct max for
  colormap. >>:pr:`19784`<< by Thomas Fan.

*   Samples with zero *sample_weight* values do not affect the results

<PierreAttard>`<<.

  from "metrics.det_curve()", "metrics.precision_recall_curve()" and
  "metrics.roc_curve()". >>:pr:`18328`<< by >>:user:`Albert Villanova
  del Moral <albertvillanova>`<< and >>:user:`Alonso Silva Allende

*   avoid overflow in "metrics.cluster.adjusted_rand_score()" with
  large amount of data. >>:pr:`20312`<< by >>:user:`Divyanshu Deoli
  <divyanshudeoli>`<<.

*   "metrics.ConfusionMatrixDisplay" exposes two class methods
  "from_estimator()" and "from_predictions()" allowing to create a
  confusion matrix plot using an estimator or the predictions.
  "metrics.plot_confusion_matrix()" is deprecated in favor of these
  two class methods and will be removed in 1.2. >>:pr:`18543`<< by

*   "metrics.PrecisionRecallDisplay" exposes two class methods
  precision-recall curve using an estimator or the predictions.
  "metrics.plot_precision_recall_curve()" is deprecated in favor of
  these two class methods and will be removed in 1.2. >>:pr:`20552`<<

*   "metrics.DetCurveDisplay" exposes two class methods

  "metrics.plot_det_curve()" is deprecated in favor of these two class
  methods and will be removed in 1.2. >>:pr:`19278`<< by Guillaume
  Lemaitre.

*   Ensure that the best parameters are set appropriately in the case
  of divergency for "mixture.GaussianMixture" and
  "mixture.BayesianGaussianMixture". >>:pr:`20030`<< by
  >>:user:`Tingshan Liu <tliu68>`<< and >>:user:`Benjamin Pedigo
  <bdpedigo>`<<.

*   added "model_selection.StratifiedGroupKFold", that combines
  "model_selection.StratifiedKFold" and "model_selection.GroupKFold",
  providing an ability to split data preserving the distribution of
  classes in each split while keeping each group within a single
  split. >>:pr:`18649`<< by >>:user:`Leandro Hermida <hermidalc>`<<
  and >>:user:`Rodion Martynov <marrodion>`<<.

*   warn only once in the main process for per-split fit failures in
  cross-validation. >>:pr:`20619`<< by >>:user:`Loïc Estève
  <lesteve>`<<

*   The "model_selection.BaseShuffleSplit" base class is now public.
  >>:pr:`20056`<< by >>:user:`pabloduque0`<<.

*   Avoid premature overflow in "model_selection.train_test_split()".
  >>:pr:`20904`<< by >>:user:`Tomasz Jakubek <t-jakubek>`<<.

*   The *fit* and *partial_fit* methods of the discrete naive Bayes
  classifiers ("naive_bayes.BernoulliNB", "naive_bayes.CategoricalNB",
  "naive_bayes.ComplementNB", and "naive_bayes.MultinomialNB") now
  correctly handle the degenerate case of a single class in the
  training set. >>:pr:`18925`<< by >>:user:`David Poznik <dpoznik>`<<.

*   The attribute "sigma_" is now deprecated in
  "naive_bayes.GaussianNB" and will be removed in 1.2. Use "var_"
  instead. >>:pr:`18842`<< by >>:user:`Hong Shao Yang
  <hongshaoyang>`<<.

*   The creation of "neighbors.KDTree" and "neighbors.BallTree" has
  been improved for their worst-cases time complexity from
  \mathcal{O}(n^2) to \mathcal{O}(n). >>:pr:`19473`<< by
  >>:user:`jiefangxuanyan <jiefangxuanyan>`<< and >>:user:`Julien

*   "neighbors.DistanceMetric" subclasses now support readonly memory-

*   "neighbors.NearestNeighbors", "neighbors.KNeighborsClassifier",
  "neighbors.KNeighborsRegressor" and
  "neighbors.RadiusNeighborsRegressor" do not validate *weights* in
  *__init__* and validates *weights* in *fit* instead. >>:pr:`20072`<<
  by >>:user:`Juan Carlos Alfaro Jiménez <alfaro96>`<<.

"neighbors.RadiusNeighborsRegressor" do not validate *weights* in

*   The parameter *kwargs* of "neighbors.RadiusNeighborsClassifier" is
  deprecated and will be removed in 1.2. >>:pr:`20842`<< by
  >>:user:`Juan Martín Loyola <jmloyola>`<<.

  now correctly support continued training when loading from a pickled
  file. >>:pr:`19631`<< by Thomas Fan.

*   The *predict_proba* and *predict_log_proba* methods of the
  "pipeline.Pipeline" now support passing prediction kwargs to the
  final estimator. >>:pr:`19790`<< by >>:user:`Christopher Flynn
  <crflynn>`<<.

*   The new "preprocessing.SplineTransformer" is a feature
  preprocessing tool for the generation of B-splines, parametrized by
  the polynomial "degree" of the splines, number of knots "n_knots"
  and knot positioning strategy "knots". >>:pr:`18368`<< by
  "preprocessing.SplineTransformer" also supports periodic splines via
  the "extrapolation" argument. >>:pr:`19483`<< by >>:user:`Malte
  Londschien <mlondschien>`<<. "preprocessing.SplineTransformer"
  supports sample weights for knot position strategy ""quantile"".

the "extrapolation" argument. >>:pr:`19483`<< by >>:user:`Malte
  >>:pr:`20526`<< by >>:user:`Malte Londschien <mlondschien>`<<.

*   "preprocessing.OrdinalEncoder" supports passing through missing
  values by default. >>:pr:`19069`<< by Thomas Fan.

*   "preprocessing.OneHotEncoder" now supports
  *handle_unknown='ignore'* and dropping categories. >>:pr:`19041`<<

*   "preprocessing.PolynomialFeatures" now supports passing a tuple to
  *degree*, i.e. *degree=(min_degree, max_degree)*. >>:pr:`20250`<< by

*   "preprocessing.StandardScaler" is faster and more memory
  efficient. >>:pr:`20652`<< by Thomas Fan.

*   Changed "algorithm" argument for "cluster.KMeans" in
  "preprocessing.KBinsDiscretizer" from "auto" to "full".
  >>:pr:`19934`<< by >>:user:`Gleb Levitskiy <GLevV>`<<.

*   The implementation of *fit* for "preprocessing.PolynomialFeatures"
  transformer is now faster. This is especially noticeable on large
  sparse input. >>:pr:`19734`<< by >>:user:`Fred Robinson <frrad>`<<.

*   The "preprocessing.StandardScaler.inverse_transform()" method now
  raises error when the input data is 1D. >>:pr:`19752`<< by
  >>:user:`Zhehao Liu <Max1993Liu>`<<.

*   "preprocessing.scale()", "preprocessing.StandardScaler" and
  similar scalers detect near-constant features to avoid scaling them

>>:user:`Zhehao Liu <Max1993Liu>`<<.

  to very large values. This problem happens in particular when using
  a scaler on sparse data with a constant column with sample weights,
  in which case centering is typically disabled. >>:pr:`19527`<< by
  >>:user:`Oliver Grisel <ogrisel>`<< and >>:user:`Maria Telenczuk
  <maikia>`<< and >>:pr:`19788`<< by >>:user:`Jérémie du Boisberranger

*   "preprocessing.StandardScaler.inverse_transform()" now correctly
  handles integer dtypes. >>:pr:`19356`<< by >>:user:`makoeppel`<<.

*   "preprocessing.OrdinalEncoder.inverse_transform()" is not
  supporting sparse matrix and raises the appropriate error message.
  >>:pr:`19879`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   The *fit* method of "preprocessing.OrdinalEncoder" will not raise
  error when *handle_unknown='ignore'* and unknown categories are
  given to *fit*. >>:pr:`19906`<< by >>:user:`Zhehao Liu

*   Fix a regression in "preprocessing.OrdinalEncoder" where large
  Python numeric would raise an error due to overflow when casted to C
  type (*np.float64* or *np.int64*). >>:pr:`20727`<< by Guillaume
  Lemaitre.

*   "preprocessing.FunctionTransformer" does not set *n_features_in_*
  based on the input to *inverse_transform*. >>:pr:`20961`<< by Thomas
  Fan.

*   The *n_input_features_* attribute of

Lemaitre.

  Fan.

  "preprocessing.PolynomialFeatures" is deprecated in favor of
  *n_features_in_* and will be removed in 1.2. >>:pr:`20240`<< by

*   The parameter ***params* of "svm.OneClassSVM.fit()" is deprecated
  and will be removed in 1.2. >>:pr:`20843`<< by >>:user:`Juan Martín

*   Add *fontname* argument in "tree.export_graphviz()" for non-
  English characters. >>:pr:`18959`<< by >>:user:`Zero <Zeroto521>`<<
  and >>:user:`wstates <wstates>`<<.

*   Improves compatibility of "tree.plot_tree()" with high DPI
  screens. >>:pr:`20023`<< by Thomas Fan.

*   Fixed a bug in "tree.DecisionTreeClassifier",
  "tree.DecisionTreeRegressor" where a node could be split whereas it
  should not have been due to incorrect handling of rounding errors.
  >>:pr:`19336`<< by >>:user:`Jérémie du Boisberranger

*   The *n_features_* attribute of "tree.DecisionTreeClassifier",
  "tree.DecisionTreeRegressor", "tree.ExtraTreeClassifier" and
  "tree.ExtraTreeRegressor" is deprecated in favor of *n_features_in_*
  and will be removed in 1.2. >>:pr:`20272`<< by >>:user:`Jérémie du

"tree.ExtraTreeRegressor" is deprecated in favor of *n_features_in_*

*   Deprecated the default value of the *random_state=0* in
  "randomized_svd()". Starting in 1.2, the default value of
  *random_state* will be set to *None*. >>:pr:`19459`<< by
  >>:user:`Cindy Bezuidenhout <cinbez>`<< and >>:user:`Clifford Akai-
  Nettey<cliffordEmmanuel>`<<.

*   Added helper decorator "utils.metaestimators.available_if()" to
  provide flexiblity in metaestimators making methods available or
  unavailable on the basis of state, in a more readable way.
  >>:pr:`19948`<< by Joel Nothman.

*   "utils.validation.check_is_fitted()" now uses
  "__sklearn_is_fitted__" if available, instead of checking for
  attributes ending with an underscore. This also makes
  "pipeline.Pipeline" and "preprocessing.FunctionTransformer" pass
  "check_is_fitted(estimator)". >>:pr:`20657`<< by Adrin Jalali.

*   Fixed a bug in "utils.sparsefuncs.mean_variance_axis()" where the
  precision of the computed variance was very poor when the real
  variance is exactly zero. >>:pr:`19766`<< by >>:user:`Jérémie du

*   The docstrings of propreties that are decorated with
  "utils.deprecated()" are now properly wrapped. >>:pr:`20385`<< by

*   "utils.stats._weighted_percentile()" now correctly ignores zero-

"utils.deprecated()" are now properly wrapped. >>:pr:`20385`<< by

  weighted observations smaller than the smallest observation with
  positive weight for "percentile=0". Affected classes are
  "dummy.DummyRegressor" for "quantile=0" and
  "ensemble.HuberLossFunction" and "ensemble.HuberLossFunction" for
  "alpha=0". >>:pr:`20528`<< by >>:user:`Malte Londschien
  <mlondschien>`<<.

*   "utils._safe_indexing()" explicitly takes a dataframe copy when
  integer indices are provided avoiding to raise a warning from
  Pandas. This warning was previously raised in resampling utilities
  and functions using those utilities (e.g.
  "model_selection.train_test_split()",
  "model_selection.cross_val_predict()"). >>:pr:`20673`<< by
  >>:user:`Joris Van den Bossche  <jorisvandenbossche>`<<.

*   Fix a regression in "utils.is_scalar_nan()" where large Python
  numbers would raise an error due to overflow in C types
  (*np.float64* or *np.int64*). >>:pr:`20727`<< by Guillaume Lemaitre.

*   Support for *np.matrix* is deprecated in "check_array()" in 1.0
  and will raise a *TypeError* in 1.2. >>:pr:`20165`<< by Thomas Fan.

*   *utils._testing.assert_warns* and
  *utils._testing.assert_warns_message* are deprecated in 1.0 and will
  be removed in 1.2. Used *pytest.warns* context manager instead. Note

  that these functions were not documented and part from the public
  API. >>:pr:`20521`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

*   Fixed several bugs in "utils.graph.graph_shortest_path()", which
  is now deprecated. Use *scipy.sparse.csgraph.shortest_path* instead.
  >>:pr:`20531`<< by Tom Dupre la Tour.

improvement of the project since version 0.24, including:

Abdulelah S. Al Mesfer, Abhinav Gupta, Adam J. Stewart, Adam Li, Adam
Midvidy, Adrian Garcia Badaracco, Adrian Sadłocha, Adrin Jalali,
Agamemnon Krasoulis, Alberto Rubiales, Albert Thomas, Albert Villanova
del Moral, Alek Lefebvre, Alessia Marcolini, Alexandr Fonari, Alihan
Zihna, Aline Ribeiro de Almeida, Amanda, Amanda Dsouza, Amol Deshmukh,
Ana Pessoa, Anavelyz, Andreas Mueller, Andrew Delong, Ashish, Ashvith
Shetty, Atsushi Nukariya, Aurélien Geron, Avi Gupta, Ayush Singh,
baam, BaptBillard, Benjamin Pedigo, Bertrand Thirion, Bharat
Raghunathan, bmalezieux, Brian Rice, Brian Sun, Bruno Charron, Bryan
Chen, bumblebee, caherrera-meli, Carsten Allefeld, CeeThinwa, Chiara
Marmo, chrissobel, Christian Lorentzen, Christopher Yeh, Chuliang
Xiao, Clément Fauchereau, cliffordEmmanuel, Conner Shen, Connor Tann,

David Dale, David Katz, David Poznik, Dimitri Papadopoulos Orfanos,
Divyanshu Deoli, dmallia17, Dmitry Kobak, DS_anas, Eduardo Jardim,
EdwinWenink, EL-ATEIF Sara, Eleni Markou, EricEllwanger, Eric Fiegel,
Erich Schubert, Ezri-Mudde, Fatos Morina, Felipe Rodrigues, Felix
Hafner, Fenil Suchak, flyingdutchman23, Flynn, Fortune Uwha, Francois
Berenger, Frankie Robertson, Frans Larsson, Frederick Robinson,
frellwan, Gabriel S Vicente, Gael Varoquaux, genvalen, Geoffrey
Thomas, geroldcsendes, Gleb Levitskiy, Glen, Glòria Macià Muñoz,
gregorystrubel, groceryheist, Guillaume Lemaitre, guiweber, Haidar
Almubarak, Hans Moritz Günther, Haoyin Xu, Harris Mirza, Harry Wei,
Harutaka Kawamura, Hassan Alsawadi, Helder Geovane Gomes de Lima, Hugo
DEFOIS, Igor Ilic, Ikko Ashimine, Isaack Mungui, Ishaan Bhat, Ishan
Mishra, Iván Pulido, iwhalvic, J Alexander, Jack Liu, James Alan
Preiss, James Budarz, James Lamb, Jannik, Jeff Zhao, Jennifer
Maldonado, Jérémie du Boisberranger, Jesse Lima, Jianzhu Guo, jnboehm,
Joel Nothman, JohanWork, John Paton, Jonathan Schneider, Jon Crall,
Jon Haitz Legarreta Gorroño, Joris Van den Bossche, José Manuel
Nápoles Duarte, Juan Carlos Alfaro Jiménez, Juan Martin Loyola, Julien
Jerphanion, Julio Batista Silva, julyrashchenko, JVM, Kadatatlu
Kishore, Karen Palacio, Kei Ishikawa, kmatt10, kobaski, Kot271828,

Kunj, KurumeYuta, kxytim, lacrosse91, LalliAcqua, Laveen Bagai,
Leonardo Rocco, Leonardo Uieda, Leopoldo Corona, Loic Esteve,
LSturtew, Luca Bittarello, Luccas Quadros, Lucy Jiménez, Lucy Liu,
ly648499246, Mabu Manaileng, Manimaran, makoeppel, Marco Gorelli,
Maren Westermann, Mariangela, Maria Telenczuk, marielaraj, Martin
Hirzel, Mateo Noreña, Mathieu Blondel, Mathis Batoul, mathurinm,
Matthew Calcote, Maxime Prieur, Maxwell, Mehdi Hamoumi, Mehmet Ali
Özer, Miao Cai, Michal Karbownik, michalkrawczyk, Mitzi, mlondschien,
Mohamed Haseeb, Mohamed Khoualed, Muhammad Jarir Kanji, murata-yu,
Nadim Kawwa, Nanshan Li, naozin555, Nate Parsons, Neal Fultz, Nic
Annau, Nicolas Hug, Nicolas Miller, Nico Stefani, Nigel Bosch, Nikita
Titov, Nodar Okroshiashvili, Norbert Preining, novaya, Ogbonna
Chibuike Stephen, OGordon100, Oliver Pfaffel, Olivier Grisel, Oras
Phongpanangam, Pablo Duque, Pablo Ibieta-Jimenez, Patric Lacouth,
Paulo S. Costa, Paweł Olszewski, Peter Dye, PierreAttard, Pierre-Yves
Le Borgne, PranayAnchuri, Prince Canuma, putschblos, qdeffense,
RamyaNP, ranjanikrishnan, Ray Bell, Rene Jean Corneille, Reshama
Shaikh, ricardojnf, RichardScottOZ, Rodion Martynov, Rohan Paul, Roman
Lutz, Roman Yurchak, Samuel Brice, Sandy Khosasi, Sean Benhur J,

Sebastian Flores, Sebastian Pölsterl, Shao Yang Hong, shinehide,
shinnar, shivamgargsya, Shooter23, Shuhei Kayawari, Shyam Desai,
simonamaggio, Sina Tootoonian, solosilence, Steven Kolawole, Steve
Stagg, Surya Prakash, swpease, Sylvain Marié, Takeshi Oura, Terence
Honles, TFiFiE, Thomas A Caswell, Thomas J. Fan, Tim Gates,
TimotheeMathieu, Timothy Wolodzko, Tim Vink, t-jakubek, t-kusanagi,
tliu68, Tobias Uhmann, tom1092, Tomás Moreyra, Tomás Ronald Hughes,
Tom Dupré la Tour, Tommaso Di Noto, Tomohiro Endo, TONY GEORGE,
Toshihiro NAKAE, tsuga, Uttam kumar, vadim-ushtanit, Vangelis
Gkiastas, Venkatachalam N, Vilém Zouhar, Vinicius Rios Fuck,
Vlasovets, waijean, Whidou, xavier dupré, xiaoyuchai, Yasmeen Alsaedy,
yoch, Yosuke KOBAYASHI, Yu Feng, YusukeNagasaka, yzhenman, Zero,
ZeyuSun, ZhaoweiWang, Zito, Zito Relova

Version 0.24.2

**April 2021**

Changelog
=========

*   *compose.ColumnTransformer.get_feature_names* does not call
  *get_feature_names* on transformers with an empty column selection.
  >>:pr:`19579`<< by Thomas Fan.

*   Fixed a regression in "cross_decomposition.CCA". >>:pr:`19646`<<

*   "cross_decomposition.PLSRegression" raises warning for constant y
  residuals instead of a *StopIteration* error. >>:pr:`19922`<< by

*   Fixed a bug in "decomposition.KernelPCA"'s "inverse_transform".
  >>:pr:`19732`<< by >>:user:`Kei Ishikawa <kstoneriv3>`<<.

*   Fixed a bug in "ensemble.HistGradientBoostingRegressor" *fit* with
  *sample_weight* parameter and *least_absolute_deviation* loss
  function. >>:pr:`19407`<< by >>:user:`Vadim Ushtanit <vadim-
  ushtanit>`<<.

"feature_extraction"

*   Fixed a bug to support multiple strings for a category when
  *sparse=False* in "feature_extraction.DictVectorizer".
  >>:pr:`19982`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

*   Avoid explicitly forming inverse covariance matrix in
  "gaussian_process.GaussianProcessRegressor" when set to output
  standard deviation. With certain covariance matrices this inverse is
  unstable to compute explicitly. Calling Cholesky solver mitigates

standard deviation. With certain covariance matrices this inverse is
  this issue in computation. >>:pr:`19939`<< by >>:user:`Ian Halvic
  <iwhalvic>`<<.

*   Avoid division by zero when scaling constant target in
  "gaussian_process.GaussianProcessRegressor". It was due to a std.
  dev. equal to 0. Now, such case is detected and the std. dev. is
  affected to 1 avoiding a division by zero and thus the presence of
  NaN values in the normalized target. >>:pr:`19703`<< by
  >>:user:`sobkevich`<<, >>:user:`Boris Villazón-Terrazas
  <boricles>`<< and >>:user:`Alexandr Fonari <afonari>`<<.

*  : Fixed a bug in "linear_model.LogisticRegression": the
  sample_weight object is not modified anymore. >>:pr:`19182`<< by
  >>:user:`Yosuke KOBAYASHI <m7142yosuke>`<<.

*   "metrics.top_k_accuracy_score()" now supports multiclass problems
  where only two classes appear in *y_true* and all the classes are
  specified in *labels*. >>:pr:`19721`<< by >>:user:`Joris Clement
  <flyingdutchman23>`<<.

*   "model_selection.RandomizedSearchCV" and
  "model_selection.GridSearchCV" now correctly shows the score for
  single metrics and verbose > 2. >>:pr:`19659`<< by Thomas Fan.

*   Some values in the *cv_results_* attribute of
  "model_selection.HalvingRandomSearchCV" and

single metrics and verbose > 2. >>:pr:`19659`<< by Thomas Fan.

  "model_selection.HalvingGridSearchCV" were not properly converted to
  numpy arrays. >>:pr:`19211`<< by Nicolas Hug.

*   The *fit* method of the successive halving parameter search
  ("model_selection.HalvingGridSearchCV", and
  "model_selection.HalvingRandomSearchCV") now correctly handles the
  *groups* parameter. >>:pr:`19847`<< by >>:user:`Xiaoyu Chai
  <xiaoyuchai>`<<.

*   "multioutput.MultiOutputRegressor" now works with estimators that
  dynamically define *predict* during fitting, such as
  "ensemble.StackingRegressor". >>:pr:`19308`<< by Thomas Fan.

*   Validate the constructor parameter *handle_unknown* in
  "preprocessing.OrdinalEncoder" to only allow for *'error'* and
  *'use_encoded_value'* strategies. >>:pr:`19234`<< by *Guillaume
  Lemaitre <glemaitre>*.

*   Fix encoder categories having dtype='S'
  "preprocessing.OneHotEncoder" and "preprocessing.OrdinalEncoder".
  >>:pr:`19727`<< by >>:user:`Andrew Delong <andrewdelong>`<<.

*   "preprocessing.OrdinalEncoder.transfrom()" correctly handles
  unknown values for string dtypes. >>:pr:`19888`<< by Thomas Fan.

*   "preprocessing.OneHotEncoder.fit()" no longer alters the *drop*
  parameter. >>:pr:`19924`<< by Thomas Fan.

*   Avoid NaN during label propagation in "LabelPropagation".
  >>:pr:`19271`<< by >>:user:`Zhaowei Wang <ThuWangzw>`<<.

*   Fix a bug in *fit* of "tree.BaseDecisionTree" that caused
  segmentation faults under certain conditions. *fit* now deep copies
  the *Criterion* object to prevent shared concurrent accesses.
  >>:pr:`19580`<< by >>:user:`Samuel Brice <samdbrice>`<< and
  >>:user:`Alex Adamson <aadamson>`<< and >>:user:`Wil Yegelwel
  <wyegelwel>`<<.

*   Better contains the CSS provided by "utils.estimator_html_repr()"
  by giving CSS ids to the html representation. >>:pr:`19417`<< by

Version 0.24.1

**January 2021**

Packaging
=========

The 0.24.0 scikit-learn wheels were not working with MacOS <1.15 due
to *libomp*. The version of *libomp* used to build the wheels was too
recent for older macOS versions. This issue has been fixed for 0.24.1
scikit-learn wheels. Scikit-learn wheels published on PyPI.org now
officially support macOS 10.13 and later.

Changelog
=========

*   Fix numerical stability bug that could happen in
  "metrics.adjusted_mutual_info_score()" and
  "metrics.mutual_info_score()" with NumPy 1.20+. >>:pr:`19179`<< by

"metrics.adjusted_mutual_info_score()" and

*   "semi_supervised.SelfTrainingClassifier" is now accepting meta-
  estimator (e.g. "ensemble.StackingClassifier"). The validation of
  this estimator is done on the fitted estimator, once we know the
  existence of the method *predict_proba*. >>:pr:`19126`<< by

Version 0.24.0

**December 2020**

lights_0_24_0.py*.

  memory.

Put the changes in their relevant module.

*   "decomposition.KernelPCA" behaviour is now more consistent between
  32-bits and 64-bits data when the kernel has small positive
  eigenvalues.

*   "decomposition.TruncatedSVD" becomes deterministic by exposing a
  *random_state* parameter.

*   "linear_model.Perceptron" when *penalty='elasticnet'*.

*   Change in the random sampling procedures for the center
  initialization of "cluster.KMeans".

Changelog
=========

*   "base.BaseEstimator.get_params()" now will raise an
  *AttributeError* if a parameter cannot be retrieved as an instance
  attribute. Previously it would return *None*. >>:pr:`17448`<< by
  >>:user:`Juan Carlos Alfaro Jiménez <alfaro96>`<<.

*   "calibration.CalibratedClassifierCV.fit" now supports
  parallelization via *joblib.Parallel* using argument *n_jobs*.
  >>:pr:`17107`<< by >>:user:`Julien Jerphanion <jjerphan>`<<.

*   Allow "calibration.CalibratedClassifierCV" use with prefit
  "pipeline.Pipeline" where data is not *X* is not array-like, sparse
  matrix or dataframe at the start. >>:pr:`17546`<< by >>:user:`Lucy
  Liu <lucyleeow>`<<.

"pipeline.Pipeline" where data is not *X* is not array-like, sparse

*   Add *ensemble* parameter to "calibration.CalibratedClassifierCV",
  which enables implementation of calibration via an ensemble of
  calibrators (current method) or just one calibrator using all the
  data (similar to the built-in feature of "sklearn.svm" estimators
  with the *probabilities=True* parameter). >>:pr:`17856`<< by
  >>:user:`Lucy Liu <lucyleeow>`<< and >>:user:`Andrea Esuli
  <aesuli>`<<.

*   "cluster.AgglomerativeClustering" has a new parameter
  *compute_distances*. When set to *True*, distances between clusters
  are computed and stored in the *distances_* attribute even when the
  parameter *distance_threshold* is not used. This new parameter is
  useful to produce dendrogram visualizations, but introduces a
  computational and memory overhead. >>:pr:`17984`<< by
  >>:user:`Michael Riedmann <mriedmann>`<<, >>:user:`Emilie Delattre
  <EmilieDel>`<<, and >>:user:`Francesco Casalegno
  <FrancescoCasalegno>`<<.

  have a new keyword argument *verbose*. When set to *True*,
  additional messages will be displayed which can aid with debugging.
  >>:pr:`18052`<< by >>:user:`Sean O. Stalley <sstalley>`<<.

*   Added "cluster.kmeans_plusplus()" as public function.

additional messages will be displayed which can aid with debugging.

  Initialization by KMeans++ can now be called separately to generate
  initial cluster centroids. >>:pr:`17937`<< by >>:user:`g-walsh`<<

*   "cluster.MiniBatchKMeans" attributes, *counts_* and *init_size_*,
  are deprecated and will be removed in 1.1 (renaming of 0.26).
  >>:pr:`17864`<< by >>:user:`Jérémie du Boisberranger

*   "compose.ColumnTransformer" will skip transformers the column
  selector is a list of bools that are False. >>:pr:`17616`<< by

*   "compose.ColumnTransformer" now displays the remainder in the
  diagram display. >>:pr:`18167`<< by Thomas Fan.

*   "compose.ColumnTransformer" enforces strict count and order of
  column names between *fit* and *transform* by raising an error
  instead of a warning, following the deprecation cycle.
  >>:pr:`18256`<< by >>:user:`Madhura Jayratne <madhuracj>`<<.

*   Deprecates *cv_alphas_* in favor of *cv_results_['alphas']* and
  *grid_scores_* in favor of split scores in *cv_results_* in
  "covariance.GraphicalLassoCV". *cv_alphas_* and *grid_scores_* will
  be removed in version 1.1 (renaming of 0.26). >>:pr:`16392`<< by

be removed in version 1.1 (renaming of 0.26). >>:pr:`16392`<< by

*   Fixed a bug in "cross_decomposition.PLSSVD" which would sometimes
  return components in the reversed order of importance.
  >>:pr:`17095`<< by Nicolas Hug.

*   Fixed a bug in "cross_decomposition.PLSSVD",
  "cross_decomposition.CCA", and "cross_decomposition.PLSCanonical",
  which would lead to incorrect predictions for *est.transform(Y)*
  when the training data is single-target. >>:pr:`17095`<< by Nicolas
  Hug.

*   Increases the stability of "cross_decomposition.CCA"
  >>:pr:`18746`<< by Thomas Fan.

*   For "cross_decomposition.NMF", the *init* value, when 'init=None'
  and n_components <= min(n_samples, n_features) will be changed from
  *'nndsvd'* to *'nndsvda'* in 1.1 (renaming of 0.26). >>:pr:`18525`<<
  by >>:user:`Chiara Marmo <cmarmo>`<<.

*   The bounds of the *n_components* parameter is now restricted:

  * into *[1, min(n_samples, n_features, n_targets)]*, for
    "cross_decomposition.PLSSVD", "cross_decomposition.CCA", and
    "cross_decomposition.PLSCanonical".

  * into *[1, n_features]* or "cross_decomposition.PLSRegression".

  An error will be raised in 1.1 (renaming of 0.26). >>:pr:`17095`<<

*   For "cross_decomposition.PLSSVD", "cross_decomposition.CCA", and
  "cross_decomposition.PLSCanonical", the *x_scores_* and *y_scores_*
  attributes were deprecated and will be removed in 1.1 (renaming of

"cross_decomposition.PLSCanonical", the *x_scores_* and *y_scores_*
  0.26). They can be retrieved by calling *transform* on the training
  data. The *norm_y_weights* attribute will also be removed.

*   For "cross_decomposition.PLSRegression",
  "cross_decomposition.PLSCanonical", "cross_decomposition.CCA", and
  "cross_decomposition.PLSSVD", the *x_mean_*, *y_mean_*, *x_std_*,
  and *y_std_* attributes were deprecated and will be removed in 1.1
  (renaming of 0.26). >>:pr:`18768`<< by >>:user:`Maren Westermann

*   "decomposition.TruncatedSVD" becomes deterministic by using the
  *random_state*. It controls the weights' initialization of the
  underlying ARPACK solver. :pr:` #18302` by >>:user:`Gaurav Desai
  <gauravkdesai>`<< and >>:user:`Ivan Panico <FollowKenny>`<<.

*   "datasets.fetch_openml()" now validates md5 checksum of arff files
  downloaded or cached to ensure data integrity. >>:pr:`14800`<< by
  >>:user:`Shashank Singh <shashanksingh28>`<< and Joel Nothman.

*   "datasets.fetch_openml()" now allows argument *as_frame* to be
  'auto', which tries to convert returned data to pandas DataFrame
  unless data is sparse. >>:pr:`17396`<< by >>:user:`Jiaxiang
  <fujiaxiang>`<<.

*   "datasets.fetch_covtype()" now supports the optional argument
  *as_frame*; when it is set to True, the returned Bunch object's

<fujiaxiang>`<<.

  *data* and *frame* members are pandas DataFrames, and the *target*
  member is a pandas Series. >>:pr:`17491`<< by >>:user:`Alex Liang
  <tianchuliang>`<<.

*   "datasets.fetch_kddcup99()" now supports the optional argument
  member is a pandas Series. >>:pr:`18280`<< by >>:user:`Alex Liang
  <tianchuliang>`<< and Guillaume Lemaitre.

*   "datasets.fetch_20newsgroups_vectorized()" now supports loading as
  a pandas "DataFrame" by setting "as_frame=True". >>:pr:`17499`<< by
  >>:user:`Brigitta Sipőcz <bsipocz>`<< and Guillaume Lemaitre.

*   The default value of *as_frame* in "datasets.fetch_openml()" is
  changed from False to 'auto'. >>:pr:`17610`<< by >>:user:`Jiaxiang

*   "decomposition.FactorAnalysis()" now supports the optional
  argument *rotation*, which can take the value *None*, *'varimax'* or
  *'quartimax'*. >>:pr:`11064`<< by >>:user:`Jona Sassenhagen <jona-
  sassenhagen>`<<.

*   "decomposition.NMF" now supports the optional parameter
  *regularization*, which can take the values *None*, 'components',
  'transformation' or 'both', in accordance with
  "decomposition.NMF.non_negative_factorization()". >>:pr:`17414`<< by

*regularization*, which can take the values *None*, 'components',
  >>:user:`Bharat Raghunathan <Bharat123rox>`<<.

  32-bits and 64-bits data input when the kernel has small positive
  eigenvalues. Small positive eigenvalues were not correctly discarded
  for 32-bits data. >>:pr:`18149`<< by >>:user:`Sylvain Marié
  <smarie>`<<.

*   Fix "decomposition.SparseCoder" such that it follows scikit-learn
  API and support cloning. The attribute *components_* is deprecated
  in 0.24 and will be removed in 1.1 (renaming of 0.26). This
  attribute was redundant with the *dictionary* attribute and
  constructor parameter. >>:pr:`17679`<< by >>:user:`Xavier Dupré

*   "TruncatedSVD.fit_transform()" consistently returns the same as
  "TruncatedSVD.fit()" followed by "TruncatedSVD.transform()".
  >>:pr:`18528`<< by >>:user:`Albert Villanova del Moral
  <albertvillanova>`<< and >>:user:`Ruifeng Zheng <zhengruifeng>`<<.

*   "discriminant_analysis.LinearDiscriminantAnalysis" can now use
  custom covariance estimate by setting the *covariance_estimator*
  parameter. >>:pr:`14446`<< by >>:user:`Hugo Richard
  <hugorichard>`<<.

*   "ensemble.HistGradientBoostingRegressor" and

parameter. >>:pr:`14446`<< by >>:user:`Hugo Richard

  "ensemble.HistGradientBoostingClassifier" now have native support
  for categorical features with the *categorical_features* parameter.
  >>:pr:`18394`<< by Nicolas Hug and Thomas Fan.

  "ensemble.HistGradientBoostingClassifier" now support the method
  *staged_predict*, which allows monitoring of each stage.
  >>:pr:`16985`<< by >>:user:`Hao Chun Chang <haochunchang>`<<.

*   break cyclic references in the tree nodes used internally in
  "ensemble.HistGradientBoostingRegressor" and
  "ensemble.HistGradientBoostingClassifier" to allow for the timely
  garbage collection of large intermediate datastructures and to
  improve memory usage in *fit*. >>:pr:`18334`<< by Olivier Grisel
  Nicolas Hug, Thomas Fan and Andreas Müller.

*   Histogram initialization is now done in parallel in
  "ensemble.HistGradientBoostingClassifier" which results in speed
  improvement for problems that build a lot of nodes on multicore
  machines. >>:pr:`18341`<< by Olivier Grisel, Nicolas Hug, Thomas
  Fan, and >>:user:`Egor Smirnov <SmirnovEgorRu>`<<.

*   Fixed a bug in "ensemble.HistGradientBoostingRegressor" and
  "ensemble.HistGradientBoostingClassifier" which can now accept data
  with *uint8* dtype in *predict*. >>:pr:`18410`<< by Nicolas Hug.

*   The parameter "n_classes_" is now deprecated in
  "ensemble.GradientBoostingRegressor" and returns *1*.
  >>:pr:`17702`<< by >>:user:`Simona Maggio <simonamaggio>`<<.

*   Mean absolute error ('mae') is now deprecated for the parameter
  "criterion" in "ensemble.GradientBoostingRegressor" and
  "ensemble.GradientBoostingClassifier". >>:pr:`18326`<< by

"sklearn.exceptions"

*   "exceptions.ChangedBehaviorWarning" and
  "exceptions.NonBLASDotWarning" are deprecated and will be removed in
  1.1 (renaming of 0.26). >>:pr:`17804`<< by Adrin Jalali.

*   "feature_extraction.DictVectorizer" accepts multiple values for
  one categorical feature. >>:pr:`17367`<< by >>:user:`Peng Yu
  <yupbank>`<< and >>:user:`Chiara Marmo <cmarmo>`<<.

*   "feature_extraction.CountVectorizer" raises an issue if a custom
  token pattern which capture more than one group is provided.
  >>:pr:`15427`<< by >>:user:`Gangesh Gudmalwar <ggangesh>`<< and
  >>:user:`Erin R Hoffman <hoffm386>`<<.

*   Added "feature_selection.SequentialFeatureSelector" which
  implements forward and backward sequential feature selection.

  >>:pr:`6545`<< by Sebastian Raschka and >>:pr:`17159`<< by Nicolas
  Hug.

*   A new parameter *importance_getter* was added to
  "feature_selection.RFE", "feature_selection.RFECV" and
  "feature_selection.SelectFromModel", allowing the user to specify an
  attribute name/path or a *callable* for extracting feature
  importance from the estimator.  >>:pr:`15361`<< by

*   Reduce memory footprint in
  "feature_selection.mutual_info_classif()" and
  "feature_selection.mutual_info_regression()" by calling
  "neighbors.KDTree" for counting nearest neighbors. >>:pr:`17878`<<
  by >>:user:`Noel Rogers <noelano>`<<.

*   "feature_selection.RFE" supports the option for the number of
  *n_features_to_select* to be given as a float representing the
  percentage of features to select. >>:pr:`17090`<< by >>:user:`Lisa
  Schwetlick <lschwetlick>`<< and >>:user:`Marija Vlajic Wheeler
  <marijavlajic>`<<.

*   A new method "gaussian_process.Kernel._check_bounds_params()" is
  called after fitting a Gaussian Process and raises a
  "ConvergenceWarning" if the bounds of the hyperparameters are too
  tight. >>:issue:`12638`<< by >>:user:`Sylvain Lannuzel
  <SylvainLan>`<<.

"ConvergenceWarning" if the bounds of the hyperparameters are too

*   "impute.SimpleImputer" now supports a list of strings when
  "strategy='most_frequent'" or "strategy='constant'". >>:pr:`17526`<<
  by >>:user:`Ayako YAGI <yagi-3>`<< and >>:user:`Juan Carlos Alfaro
  Jiménez <alfaro96>`<<.

*   Added method "impute.SimpleImputer.inverse_transform()" to revert
  imputed data to original when instantiated with
  "add_indicator=True". >>:pr:`17612`<< by >>:user:`Srimukh Sripada
  <d3b0unce>`<<.

*   replace the default values in "impute.IterativeImputer" of
  *min_value* and *max_value* parameters to *-np.inf* and *np.inf*,
  respectively instead of *None*. However, the behaviour of the class
  does not change since *None* was defaulting to these values already.
  >>:pr:`16493`<< by >>:user:`Darshan N <DarshanGowda0>`<<.

*   "impute.IterativeImputer" will not attempt to set the estimator's
  *random_state* attribute, allowing to use it with more external
  classes. >>:pr:`15636`<< by >>:user:`David Cortes <david-cortes>`<<.

*   "impute.SimpleImputer" is now faster with *object* dtype array.
  when *strategy='most_frequent'* in "SimpleImputer". >>:pr:`18987`<<
  by >>:user:`David Katz <DavidKatz-il>`<<.

  "inspection.plot_partial_dependence()" now support calculating and

by >>:user:`David Katz <DavidKatz-il>`<<.

  plotting Individual Conditional Expectation (ICE) curves controlled
  by the "kind" parameter. >>:pr:`16619`<< by >>:user:`Madhura
  Jayratne <madhuracj>`<<.

*   Add *sample_weight* parameter to
  "inspection.permutation_importance()". >>:pr:`16906`<< by
  >>:user:`Roei Kahny <RoeiKa>`<<.

*   Positional arguments are deprecated in
  "inspection.PartialDependenceDisplay.plot()" and will error in 1.1
  (renaming of 0.26). >>:pr:`18293`<< by Thomas Fan.

*   Expose fitted attributes "X_thresholds_" and "y_thresholds_" that
  hold the de-duplicated interpolation thresholds of an
  "isotonic.IsotonicRegression" instance for model inspection purpose.
  >>:pr:`16289`<< by >>:user:`Masashi Kishimoto <kishimoto-banana>`<<
  and >>:user:`Olivier Grisel <ogrisel>`<<.

*   "isotonic.IsotonicRegression" now accepts 2d array with 1 feature
  as input array. >>:pr:`17379`<< by >>:user:`Jiaxiang

*   Add tolerance when determining duplicate X values to prevent inf
  values from being predicted by "isotonic.IsotonicRegression".
  >>:pr:`18639`<< by >>:user:`Lucy Liu <lucyleeow>`<<.

*   Added class "kernel_approximation.PolynomialCountSketch" which

>>:pr:`18639`<< by >>:user:`Lucy Liu <lucyleeow>`<<.

  implements the Tensor Sketch algorithm for polynomial kernel feature
  map approximation. >>:pr:`13003`<< by >>:user:`Daniel López Sánchez
  <lopeLH>`<<.

*   "kernel_approximation.Nystroem" now supports parallelization via
  *joblib.Parallel* using argument *n_jobs*. >>:pr:`18545`<< by
  >>:user:`Laurenz Reitsam <LaurenzReitsam>`<<.

*   "linear_model.LinearRegression" now forces coefficients to be all
  positive when "positive" is set to "True". >>:pr:`17578`<< by
  >>:user:`Joseph Knox <jknox13>`<<, >>:user:`Nelle Varoquaux
  <NelleV>`<< and >>:user:`Chiara Marmo <cmarmo>`<<.

*   "linear_model.RidgeCV" now supports finding an optimal
  regularization value *alpha* for each target separately by setting
  "alpha_per_target=True". This is only supported when using the
  default efficient leave-one-out cross-validation scheme "cv=None".
  >>:pr:`6624`<< by >>:user:`Marijn van Vliet <wmvanvliet>`<<.

*   Fixes bug in "linear_model.TheilSenRegressor" where *predict* and
  *score* would fail when *fit_intercept=False* and there was one
  feature during fitting. >>:pr:`18121`<< by Thomas Fan.

*   Fixes bug in "linear_model.ARDRegression" where *predict* was
  raising an error when *normalize=True* and *return_std=True* because

feature during fitting. >>:pr:`18121`<< by Thomas Fan.

  *X_offset_* and *X_scale_* were undefined. >>:pr:`18607`<< by
  >>:user:`fhaselbeck <fhaselbeck>`<<.

*   Added the missing *l1_ratio* parameter in
  "linear_model.Perceptron", to be used when *penalty='elasticnet'*.
  This changes the default from 0 to 0.15. >>:pr:`18622`<< by
  >>:user:`Haesun Park <rickiepark>`<<.

*   Fixed >>:issue:`10493`<<. Improve Local Linear Embedding (LLE)
  that raised *MemoryError* exception when used with large inputs.
  >>:pr:`17997`<< by >>:user:`Bertrand Maisonneuve <bmaisonn>`<<.

*   Add *square_distances* parameter to "manifold.TSNE", which
  provides backward compatibility during deprecation of legacy
  squaring behavior. Distances will be squared by default in 1.1
  (renaming of 0.26), and this parameter will be removed in 1.3.
  >>:pr:`17662`<< by >>:user:`Joshua Newton <joshuacwnewton>`<<.

*   "manifold.MDS" now correctly sets its *_pairwise* attribute.
  >>:pr:`18278`<< by Thomas Fan.

*   Added "metrics.cluster.pair_confusion_matrix()" implementing the
  confusion matrix arising from pairs of elements from two
  clusterings. >>:pr:`17412`<< by >>:user:`Uwe F Mayer <ufmayer>`<<.

*   new metric "metrics.top_k_accuracy_score()". It's a generalization

confusion matrix arising from pairs of elements from two

  of "metrics.top_k_accuracy_score()", the difference is that a
  prediction is considered correct as long as the true label is
  associated with one of the *k* highest predicted scores.
  "accuracy_score()" is the special case of *k = 1*. >>:pr:`16625`<<
  by >>:user:`Geoffrey Bolmier <gbolmier>`<<.

*   Added "metrics.det_curve()" to compute Detection Error Tradeoff
  curve classification metric. >>:pr:`10591`<< by >>:user:`Jeremy
  Karnowski <jkarnows>`<< and >>:user:`Daniel Mohns <dmohns>`<<.

*   Added "metrics.plot_det_curve()" and "metrics.DetCurveDisplay" to
  ease the plot of DET curves. >>:pr:`18176`<< by >>:user:`Guillaume

*   Added "metrics.mean_absolute_percentage_error()" metric and the
  associated scorer for regression problems. >>:issue:`10708`<< fixed
  with the PR >>:pr:`15007`<< by >>:user:`Ashutosh Hathidara
  <ashutosh1919>`<<. The scorer and some practical test cases were
  taken from PR >>:pr:`10711`<< by >>:user:`Mohamed Ali Jamaoui
  <mohamed-ali>`<<.

*   Added "metrics.rand_score()" implementing the (unadjusted) Rand
  index. >>:pr:`17412`<< by >>:user:`Uwe F Mayer <ufmayer>`<<.

*   "metrics.plot_confusion_matrix()" now supports making colorbar
  optional in the matplotlib plot by setting *colorbar=False*.

index. >>:pr:`17412`<< by >>:user:`Uwe F Mayer <ufmayer>`<<.

  >>:pr:`17192`<< by >>:user:`Avi Gupta <avigupta2612>`<<

  optional in the matplotlib plot by setting colorbar=False.
  >>:pr:`17192`<< by >>:user:`Avi Gupta <avigupta2612>`<<.

  "metrics.median_absolute_error()". >>:pr:`17225`<< by >>:user:`Lucy

*   Add *pos_label* parameter in
  "metrics.plot_precision_recall_curve()" in order to specify the
  positive class to be used when computing the precision and recall
  statistics. >>:pr:`17569`<< by >>:user:`Guillaume Lemaitre

*   Add *pos_label* parameter in "metrics.plot_roc_curve()" in order
  to specify the positive class to be used when computing the roc auc
  statistics. >>:pr:`17651`<< by >>:user:`Clara Matos <claramatos>`<<.

*   Fixed a bug in "metrics.classification_report()" which was raising
  AttributeError when called with *output_dict=True* for 0-length
  values. >>:pr:`17777`<< by >>:user:`Shubhanshu Mishra
  <napsternxg>`<<.

*   Fixed a bug in "metrics.jaccard_score()" which recommended the

values. >>:pr:`17777`<< by >>:user:`Shubhanshu Mishra

  *zero_division* parameter when called with no true or predicted
  samples. >>:pr:`17826`<< by >>:user:`Richard Decal <crypdick>`<< and
  >>:user:`Joseph Willard <josephwillard>`<<

*   bug in "metrics.hinge_loss()" where error occurs when "y_true" is
  missing some labels that are provided explicitly in the "labels"
  parameter. >>:pr:`17935`<< by >>:user:`Cary Goltermann
  <Ultramann>`<<.

*   Fix scorers that accept a pos_label parameter and compute their
  metrics from values returned by *decision_function* or
  *predict_proba*. Previously, they would return erroneous values when
  pos_label was not corresponding to *classifier.classes_[1]*. This is
  especially important when training classifiers directly with string
  labeled target classes. >>:pr:`18114`<< by >>:user:`Guillaume

*   Fixed bug in "metrics.plot_confusion_matrix()" where error occurs
  when *y_true* contains labels that were not previously seen by the
  classifier while the *labels* and *display_labels* parameters are
  set to *None*. >>:pr:`18405`<< by >>:user:`Thomas J. Fan
  <thomasjpfan>`<< and >>:user:`Yakov Pchelintsev <kyouma>`<<.

*   Added (experimental) parameter search estimators
  "model_selection.HalvingGridSearchCV" which implement Successive

  Halving, and can be used as a drop-in replacements for
  "model_selection.RandomizedSearchCV" and
  "model_selection.GridSearchCV". >>:pr:`13900`<< by Nicolas Hug, Joel
  Nothman and Andreas Müller.

  "model_selection.GridSearchCV" now have the method "score_samples"
  >>:pr:`17478`<< by >>:user:`Teon Brooks <teonbrooks>`<< and
  >>:user:`Mohamed Maskani <maskani-moh>`<<.

*   "model_selection.TimeSeriesSplit" has two new keyword arguments
  *test_size* and *gap*. *test_size* allows the out-of-sample time
  series length to be fixed for all folds. *gap* removes a fixed
  number of samples between the train and test set on each fold.
  >>:pr:`13204`<< by >>:user:`Kyle Kosic <kykosic>`<<.

*   "model_selection.permutation_test_score()" and
  "model_selection.validation_curve()" now accept fit_params to pass
  additional estimator parameters. >>:pr:`18527`<< by >>:user:`Gaurav
  Dhingra <gxyd>`<<, >>:user:`Julien Jerphanion <jjerphan>`<< and
  >>:user:`Amanda Dsouza <amy12xx>`<<.

*   "model_selection.cross_val_score()",
  and "model_selection.RandomizedSearchCV" allows estimator to fail
  scoring and replace the score with *error_score*. If

"model_selection.cross_validate()", "model_selection.GridSearchCV",
  *error_score="raise"*, the error will be raised. >>:pr:`18343`<< by
  Guillaume Lemaitre and >>:user:`Devi Sandeep <dsandeep0138>`<<.

*   "model_selection.learning_curve()" now accept fit_params to pass
  additional estimator parameters. >>:pr:`18595`<< by >>:user:`Amanda
  Dsouza <amy12xx>`<<.

*   Fixed the *len* of "model_selection.ParameterSampler" when all
  distributions are lists and *n_iter* is more than the number of
  unique parameter combinations. >>:pr:`18222`<< by Nicolas Hug.

*   A fix to raise warning when one or more CV splits of
  "model_selection.RandomizedSearchCV" results in non-finite scores.
  >>:pr:`18266`<< by >>:user:`Subrat Sahu <subrat93>`<<,
  >>:user:`Nirvan <Nirvan101>`<< and >>:user:`Arthur Book
  <ArthurBook>`<<.

  "model_selection.cross_validate()" support *scoring* being a
  callable returning a dictionary of of multiple metric names/values
  association. >>:pr:`15126`<< by Thomas Fan.

*   "multiclass.OneVsOneClassifier" now accepts the inputs with
  missing values. Hence, estimators which can handle missing values
  (may be a pipeline with imputation step) can be used as a estimator

  for multiclass wrappers. >>:pr:`17987`<< by >>:user:`Venkatachalam N

*   A fix to allow "multiclass.OutputCodeClassifier" to accept sparse
  input data in its *fit* and *predict* methods. The check for
  validity of the input is now delegated to the base estimator.
  >>:pr:`17233`<< by >>:user:`Zolisa Bleki <zoj613>`<<.

*   "multioutput.MultiOutputClassifier" and
  "multioutput.MultiOutputRegressor" now accepts the inputs with
  (may be a pipeline with imputation step, HistGradientBoosting
  estimators) can be used as a estimator for multiclass wrappers.
  >>:pr:`17987`<< by >>:user:`Venkatachalam N <venkyyuvy>`<<.

*   A fix to accept tuples for the "order" parameter in
  "multioutput.ClassifierChain". >>:pr:`18124`<< by >>:user:`Gus
  Brocchini <boldloop>`<< and >>:user:`Amanda Dsouza <amy12xx>`<<.

*   Adds a parameter *min_categories* to "naive_bayes.CategoricalNB"
  that allows a minimum number of categories per feature to be
  specified. This allows categories unseen during training to be
  accounted for. >>:pr:`16326`<< by >>:user:`George Armstrong
  <gwarmstrong>`<<.

specified. This allows categories unseen during training to be

*   The attributes "coef_" and "intercept_" are now deprecated in
  "naive_bayes.MultinomialNB", "naive_bayes.ComplementNB",
  "naive_bayes.BernoulliNB" and "naive_bayes.CategoricalNB", and will
  be removed in v1.1 (renaming of 0.26). >>:pr:`17427`<< by

*   Speed up "seuclidean", "wminkowski", "mahalanobis" and "haversine"
  metrics in "neighbors.DistanceMetric" by avoiding unexpected GIL
  acquiring in Cython when setting "n_jobs>1" in
  "neighbors.KNeighborsClassifier", "neighbors.KNeighborsRegressor",
  "neighbors.RadiusNeighborsRegressor", "metrics.pairwise_distances()"
  and by validating data out of loops. >>:pr:`17038`<< by

*   "neighbors.NeighborsBase" benefits of an improved *algorithm =
  'auto'* heuristic. In addition to the previous set of rules, now,
  when the number of features exceeds 15, *brute* is selected,
  assuming the data intrinsic dimensionality is too high for tree-
  based methods. >>:pr:`17148`<< by >>:user:`Geoffrey Bolmier
  <gbolmier>`<<.

*   "neighbors.BinaryTree" will raise a *ValueError* when fitting on
  data array having points with different dimensions. >>:pr:`18691`<<

<gbolmier>`<<.

*   "neighbors.NearestCentroid" with a numerical *shrink_threshold*
  will raise a *ValueError* when fitting on data with all constant
  features. >>:pr:`18370`<< by >>:user:`Trevor Waite <trewaite>`<<.

*   In  methods *radius_neighbors* and *radius_neighbors_graph* of
  "neighbors.NearestNeighbors", "neighbors.RadiusNeighborsClassifier",
  "neighbors.RadiusNeighborsTransformer", using *sort_results=True*
  now correctly sorts the results even when fitting with the "brute"
  algorithm. >>:pr:`18612`<< by Tom Dupre la Tour.

*   Neural net training and prediction are now a little faster.
  >>:pr:`17603`<<, >>:pr:`17604`<<, >>:pr:`17606`<<, >>:pr:`17608`<<,
  >>:pr:`17609`<<, >>:pr:`17633`<<, >>:pr:`17661`<<, >>:pr:`17932`<<
  by >>:user:`Alex Henrie <alexhenrie>`<<.

*   Avoid converting float32 input to float64 in
  "neural_network.BernoulliRBM". >>:pr:`16352`<< by >>:user:`Arthur
  Imbert <Henley13>`<<.

*   Support 32-bit computations in "neural_network.MLPClassifier" and
  "neural_network.MLPRegressor". >>:pr:`17759`<< by >>:user:`Srimukh
  Sripada <d3b0unce>`<<.

*   Fix method  "fit()" of "neural_network.MLPClassifier" not
  iterating to "max_iter" if warm started. >>:pr:`18269`<< by

Sripada <d3b0unce>`<<.

  >>:user:`Norbert Preining <norbusan>`<< and >>:user:`Guillaume

*   References to transformers passed through "transformer_weights" to
  "pipeline.FeatureUnion" that aren't present in "transformer_list"
  will raise a "ValueError". >>:pr:`17876`<< by >>:user:`Cary
  Goltermann <Ultramann>`<<.

*   A slice of a "pipeline.Pipeline" now inherits the parameters of
  the original pipeline (*memory* and *verbose*). >>:pr:`18429`<< by
  >>:user:`Albert Villanova del Moral <albertvillanova>`<< and
  >>:user:`Paweł Biernat <pwl>`<<.

*   "preprocessing.OneHotEncoder" now supports missing values by
  treating them as a category. >>:pr:`17317`<< by Thomas Fan.

*   Add a new "handle_unknown" parameter with a "use_encoded_value"
  option, along with a new "unknown_value" parameter, to
  "preprocessing.OrdinalEncoder" to allow unknown categories during
  transform and set the encoded value of the unknown categories.
  >>:pr:`17406`<< by >>:user:`Felix Wick <FelixWick>`<< and
  >>:pr:`18406`<< by Nicolas Hug.

*   Add "clip" parameter to "preprocessing.MinMaxScaler", which clips
  the transformed values of test data to "feature_range".
  >>:pr:`17833`<< by >>:user:`Yashika Sharma <yashika51>`<<.

*   Add "sample_weight" parameter to "preprocessing.StandardScaler".
  Allows setting individual weights for each sample. >>:pr:`18510`<<
  and >>:pr:`18447`<< and >>:pr:`16066`<< and >>:pr:`18682`<< by
  >>:user:`Maria Telenczuk <maikia>`<< and >>:user:`Albert Villanova
  <albertvillanova>`<< and >>:user:`panpiort8`<< and >>:user:`Alex

*   Verbose output of "model_selection.GridSearchCV" has been improved
  for readability. >>:pr:`16935`<< by >>:user:`Raghav Rajagopalan
  <raghavrv>`<< and >>:user:`Chiara Marmo <cmarmo>`<<.

*   Add "unit_variance" to "preprocessing.RobustScaler", which scales
  output data such that normally distributed features have a variance
  of 1. >>:pr:`17193`<< by >>:user:`Lucy Liu <lucyleeow>`<< and
  >>:user:`Mabel Villalba <mabelvj>`<<.

*   Add *dtype* parameter to "preprocessing.KBinsDiscretizer".
  >>:pr:`16335`<< by >>:user:`Arthur Imbert <Henley13>`<<.

*   Raise error on
  "sklearn.preprocessing.OneHotEncoder.inverse_transform()" when
  *handle_unknown='error'* and *drop=None* for samples encoded as all
  zeros. >>:pr:`14982`<< by >>:user:`Kevin Winata <kwinata>`<<.

*   Added "semi_supervised.SelfTrainingClassifier", a meta-classifier
  that allows any supervised classifier to function as a semi-

  supervised classifier that can learn from unlabeled data.
  >>:issue:`11682`<< by >>:user:`Oliver Rausch <orausch>`<< and
  >>:user:`Patrice Becker <pr0duktiv>`<<.

*   Fix incorrect encoding when using unicode string dtypes in
  >>:pr:`15763`<< by Thomas Fan.

*   invoke SciPy BLAS API for SVM kernel function in "fit", "predict"
  and related methods of "svm.SVC", "svm.NuSVC", "svm.SVR",
  "svm.NuSVR", "OneClassSVM". >>:pr:`16530`<< by >>:user:`Shuhua Fan
  <jim0421>`<<.

*   "tree.DecisionTreeRegressor" now supports the new splitting
  criterion "'poisson'" useful for modeling count data.
  >>:pr:`17386`<< by >>:user:`Christian Lorentzen <lorentzenchr>`<<.

*   "tree.plot_tree()" now uses colors from the matplotlib
  configuration settings. >>:pr:`17187`<< by Andreas Müller.

*   The parameter "X_idx_sorted" is now deprecated in
  "tree.DecisionTreeClassifier.fit()" and
  "tree.DecisionTreeRegressor.fit()", and has not effect.
  >>:pr:`17614`<< by >>:user:`Juan Carlos Alfaro Jiménez
  <alfaro96>`<<.

*   Add "check_methods_sample_order_invariance" to
  "check_estimator()", which checks that estimator methods are

<alfaro96>`<<.

  invariant if applied to the same dataset with different sample order
  >>:pr:`17598`<< by >>:user:`Jason Ngo <ngojason9>`<<.

*   Add support for weights in
  "utils.sparse_func.incr_mean_variance_axis()". By >>:user:`Maria
  Telenczuk <maikia>`<< and >>:user:`Alex Gramfort <agramfort>`<<.

*   Raise ValueError with clear error message in "check_array()" for
  sparse DataFrames with mixed types. >>:pr:`17992`<< by
  >>:user:`Thomas J. Fan <thomasjpfan>`<< and >>:user:`Alex Shacked

*   Allow serialized tree based models to be unpickled on a machine
  with different endianness. >>:pr:`17644`<< by >>:user:`Qi Zhang
  <qzhang90>`<<.

*   Check that we raise proper error when axis=1 and the dimensions do
  not match in "utils.sparse_func.incr_mean_variance_axis()". By
  >>:user:`Alex Gramfort <agramfort>`<<.

*   Calls to "repr" are now faster when *print_changed_only=True*,
  especially with meta-estimators. >>:pr:`18508`<< by >>:user:`Nathan
  C. <Xethan>`<<.

improvement of the project since version 0.23, including:

Abo7atm, Adam Spannbauer, Adrin Jalali, adrinjalali, Agamemnon
Krasoulis, Akshay Deodhar, Albert Villanova del Moral, Alessandro

Gentile, Alex Henrie, Alex Itkes, Alex Liang, Alexander Lenail,
alexandracraciun, Alexandre Gramfort, alexshacked, Allan D Butler,
Amanda Dsouza, amy12xx, Anand Tiwari, Anderson Nelson, Andreas
Mueller, Ankit Choraria, Archana Subramaniyan, Arthur Imbert, Ashutosh
Hathidara, Ashutosh Kushwaha, Atsushi Nukariya, Aura Munoz, AutoViz
and Auto_ViML, Avi Gupta, Avinash Anakal, Ayako YAGI, barankarakus,
barberogaston, beatrizsmg, Ben Mainye, Benjamin Bossan, Benjamin
Pedigo, Bharat Raghunathan, Bhavika Devnani, Biprateep Dey, bmaisonn,
Bo Chang, Boris Villazón-Terrazas, brigi, Brigitta Sipőcz, Bruno
Charron, Byron Smith, Cary Goltermann, Cat Chenal, CeeThinwa,
chaitanyamogal, Charles Patel, Chiara Marmo, Christian Kastner,
Christian Lorentzen, Christoph Deil, Christos Aridas, Clara Matos,
clmbst, Coelhudo, crispinlogan, Cristina Mulas, Daniel López, Daniel
Mohns, darioka, Darshan N, david-cortes, Declan O'Neill, Deeksha
Madan, Elizabeth DuPre, Eric Fiegel, Eric Larson, Erich Schubert, Erin
Khoo, Erin R Hoffman, eschibli, Felix Wick, fhaselbeck, Forrest Koch,
Francesco Casalegno, Frans Larsson, Gael Varoquaux, Gaurav Desai,
Gaurav Sheni, genvalen, Geoffrey Bolmier, George Armstrong, George
Kiragu, Gesa Stupperich, Ghislain Antony Vaillant, Gim Seng, Gordon

Walsh, Gregory R. Lee, Guillaume Chevalier, Guillaume Lemaitre, Haesun
Park, Hannah Bohle, Hao Chun Chang, Harry Scholes, Harsh Soni, Henry,
Hirofumi Suzuki, Hitesh Somani, Hoda1394, Hugo Le Moine, hugorichard,
indecisiveuser, Isuru Fernando, Ivan Wiryadi, j0rd1smit, Jaehyun Ahn,
Jake Tae, James Hoctor, Jan Vesely, Jeevan Anand Anne, JeroenPeterBos,
JHayes, Jiaxiang, Jie Zheng, Jigna Panchal, jim0421, Jin Li, Joaquin
Vanschoren, Joel Nothman, Jona Sassenhagen, Jonathan, Jorge Gorbe
Moya, Joseph Lucas, Joshua Newton, Juan Carlos Alfaro Jiménez, Julien
Jerphanion, Justin Huber, Jérémie du Boisberranger, Kartik Chugh,
Katarina Slama, kaylani2, Kendrick Cetina, Kenny Huynh, Kevin Markham,
Kevin Winata, Kiril Isakov, kishimoto, Koki Nishihara, Krum Arnaudov,
Kyle Kosic, Lauren Oldja, Laurenz Reitsam, Lisa Schwetlick, Louis
Douge, Louis Guitton, Lucy Liu, Madhura Jayaratne, maikia, Manimaran,
Manuel López-Ibáñez, Maren Westermann, Maria Telenczuk, Mariam-ke,
Marijn van Vliet, Markus Löning, Martin Scheubrein, Martina G. Vilas,
Martina Megasari, Mateusz Górski, mathschy, mathurinm, Matthias
Bussonnier, Max Del Giudice, Michael, Milan Straka, Muoki Caleb, N.
Haiat, Nadia Tahiri, Ph. D, Naoki Hamada, Neil Botelho, Nicolas Hug,
Nils Werner, noelano, Norbert Preining, oj_lappi, Oleh Kozynets,

Olivier Grisel, Pankaj Jindal, Pardeep Singh, Parthiv Chigurupati,
Patrice Becker, Pete Green, pgithubs, Poorna Kumar, Prabakaran
Kumaresshan, Probinette4, pspachtholz, pwalchessen, Qi Zhang, rachel
fischoff, Rachit Toshniwal, Rafey Iqbal Rahman, Rahul Jakhar, Ram
Rachum, RamyaNP, rauwuckl, Ravi Kiran Boggavarapu, Ray Bell, Reshama
Shaikh, Richard Decal, Rishi Advani, Rithvik Rao, Rob Romijnders,
roei, Romain Tavenard, Roman Yurchak, Ruby Werman, Ryotaro Tsukada,
sadak, Saket Khandelwal, Sam, Sam Ezebunandu, Sam Kimbinyi, Sarah
Brown, Saurabh Jain, Sean O. Stalley, Sergio, Shail Shah, Shane
Keller, Shao Yang Hong, Shashank Singh, Shooter23, Shubhanshu Mishra,
simonamaggio, Soledad Galli, Srimukh Sripada, Stephan Steinfurt,
subrat93, Sunitha Selvan, Swier, Sylvain Marié, SylvainLan,
t-kusanagi2, Teon L Brooks, Terence Honles, Thijs van den Berg, Thomas
J Fan, Thomas J. Fan, Thomas S Benjamin, Thomas9292, Thorben Jensen,
tijanajovanovic, Timo Kaufmann, tnwei, Tom Dupré la Tour, Trevor
Waite, ufmayer, Umberto Lupo, Venkatachalam N, Vikas Pandey, Vinicius
Rios Fuck, Violeta, watchtheblur, Wenbo Zhao, willpeppo, xavier dupré,
Xethan, Xue Qianming, xun-tang, yagi-3, Yakov Pchelintsev, Yashika
Sharma, Yi-Yan Ge, Yue Wu, Yutaro Ikeda, Zaccharie Ramzi, zoj613, Zhao
Feng.

Version 0.18.2

**June 20, 2017**

Last release with Python 2.6 support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Scikit-learn 0.18 is the last major release of scikit-learn to support
Python 2.6. Later versions of scikit-learn will require Python 2.7 or
above.

Changelog
=========

* Fixes for compatibility with NumPy 1.13.0: >>:issue:`7946`<<
  >>:issue:`8355`<< by Loic Esteve.

* Minor compatibility changes in the examples >>:issue:`9010`<<
  >>:issue:`8040`<< >>:issue:`9149`<<.

Code Contributors

Aman Dalmia, Loic Esteve, Nate Guerin, Sergei Lebedev

Version 0.18.1

**November 11, 2016**

Changelog
=========

Enhancements

* Improved "sample_without_replacement" speed by utilizing
  numpy.random.permutation for most cases. As a result, samples may
  differ in this release for a fixed random state. Affected

  * "ensemble.BaggingClassifier"

  * "ensemble.BaggingRegressor"

  * "linear_model.RANSACRegressor"

  * "model_selection.RandomizedSearchCV"

  This also affects the "datasets.make_classification()" method.

Bug fixes
---------

* Fix issue where "min_grad_norm" and "n_iter_without_progress"
  parameters were not being utilised by "manifold.TSNE".
  >>:issue:`6497`<< by >>:user:`Sebastian Säger <ssaeger>`<<

* Fix bug for svm's decision values when "decision_function_shape" is
  "ovr" in "svm.SVC". "svm.SVC"'s decision_function was incorrect from

>>:issue:`6497`<< by >>:user:`Sebastian Säger <ssaeger>`<<

  versions 0.17.0 through 0.18.0. >>:issue:`7724`<< by Bing Tian Dai

* Attribute "explained_variance_ratio" of
  "discriminant_analysis.LinearDiscriminantAnalysis" calculated with
  SVD and Eigen solver are now of the same length. >>:issue:`7632`<<
  by >>:user:`JPFrancoia <JPFrancoia>`<<

* Fixes issue in Univariate feature selection where score functions
  were not accepting multi-label targets. >>:issue:`7676`<< by
  >>:user:`Mohammed Affan <affanv14>`<<

* Fixed setting parameters when calling "fit" multiple times on
  "feature_selection.SelectFromModel". >>:issue:`7756`<< by Andreas
  Müller

* Fixes issue in "partial_fit" method of
  "multiclass.OneVsRestClassifier" when number of classes used in
  "partial_fit" was less than the total number of classes in the data.
  >>:issue:`7786`<< by Srivatsan Ramesh

* Fixes issue in "calibration.CalibratedClassifierCV" where the sum of
  probabilities of each class for a data was not 1, and
  "CalibratedClassifierCV" now handles the case where the training set
  has less number of classes than the total data. >>:issue:`7799`<< by
  Srivatsan Ramesh

* Fix a bug where "sklearn.feature_selection.SelectFdr" did not
  exactly implement Benjamini-Hochberg procedure. It formerly may have
  selected fewer features than it should. >>:issue:`7490`<< by

  >>:user:`Peng Meng <mpjlu>`<<.

* "sklearn.manifold.LocallyLinearEmbedding" now correctly handles
  integer inputs. >>:issue:`6282`<< by Jake Vanderplas.

* The "min_weight_fraction_leaf" parameter of tree-based classifiers
  and regressors now assumes uniform sample weights by default if the
  "sample_weight" argument is not passed to the "fit" function.
  Previously, the parameter was silently ignored. >>:issue:`7301`<< by
  >>:user:`Nelson Liu <nelson-liu>`<<.

* Numerical issue with "linear_model.RidgeCV" on centered data when
  *n_features > n_samples*. >>:issue:`6178`<< by Bertrand Thirion

* Tree splitting criterion classes' cloning/pickling is now memory
  safe >>:issue:`7680`<< by >>:user:`Ibraim Ganiev <olologin>`<<.

* Fixed a bug where "decomposition.NMF" sets its "n_iters_" attribute
  in *transform()*. >>:issue:`7553`<< by >>:user:`Ekaterina Krivich
  <kiote>`<<.

* "sklearn.linear_model.LogisticRegressionCV" now correctly handles
  string labels. >>:issue:`5874`<< by Raghav RV.

* Fixed a bug where "sklearn.model_selection.train_test_split()"
  raised an error when "stratify" is a list of string labels.
  >>:issue:`7593`<< by Raghav RV.

* Fixed a bug where "sklearn.model_selection.GridSearchCV" and
  "sklearn.model_selection.RandomizedSearchCV" were not pickleable

>>:issue:`7593`<< by Raghav RV.

  because of a pickling bug in "np.ma.MaskedArray". >>:issue:`7594`<<
  by Raghav RV.

* All cross-validation utilities in "sklearn.model_selection" now
  permit one time cross-validation splitters for the "cv" parameter.
  Also non-deterministic cross-validation splitters (where multiple
  calls to "split" produce dissimilar splits) can be used as "cv"
  parameter. The "sklearn.model_selection.GridSearchCV" will cross-
  validate each parameter setting on the split produced by the first
  "split" call to the cross-validation splitter.  >>:issue:`7660`<< by
  Raghav RV.

* Fix bug where "preprocessing.MultiLabelBinarizer.fit_transform()"
  returned an invalid CSR matrix. >>:issue:`7750`<< by >>:user:`CJ
  Carey <perimosocordiae>`<<.

* Fixed a bug where "metrics.pairwise.cosine_distances()" could return
  a small negative distance. >>:issue:`7732`<< by >>:user:`Artsion
  <asanakoy>`<<.

API changes summary

Trees and forests

Previously, the parameter was silently ignored. >>:issue:`7301`<< by

  safe. >>:issue:`7680`<< by >>:user:`Ibraim Ganiev <olologin>`<<.

Linear, kernelized and related models

* Length of "explained_variance_ratio" of
  "discriminant_analysis.LinearDiscriminantAnalysis" changed for both
  Eigen and SVD solvers. The attribute has now a length of
  min(n_components, n_classes - 1). >>:issue:`7632`<< by
  >>:user:`JPFrancoia <JPFrancoia>`<<

  "n_features > n_samples". >>:issue:`6178`<< by Bertrand Thirion

Version 0.18
************

**September 28, 2016**

Scikit-learn 0.18 will be the last version of scikit-learn to support
above.

Model Selection Enhancements and API Changes
============================================

* **The model_selection module**

  The new module "sklearn.model_selection", which groups together the
  functionalities of formerly "sklearn.cross_validation",
  "sklearn.grid_search" and "sklearn.learning_curve", introduces new
  possibilities such as nested cross-validation and better
  manipulation of parameter searches with Pandas.

  Many things will stay the same but there are some key differences.
  Read below to know more about the changes.

manipulation of parameter searches with Pandas.

* **Data-independent CV splitters enabling nested cross-validation**

  The new cross-validation splitters, defined in the
  "sklearn.model_selection", are no longer initialized with any data-
  dependent parameters such as "y". Instead they expose a "split()"
  method that takes in the data and yields a generator for the
  different splits.

  This change makes it possible to use the cross-validation splitters
  to perform nested cross-validation, facilitated by
  "model_selection.RandomizedSearchCV" utilities.

* **The enhanced cv_results_ attribute**

  The new "cv_results_" attribute (of "model_selection.GridSearchCV"
  and "model_selection.RandomizedSearchCV") introduced in lieu of the
  "grid_scores_" attribute is a dict of 1D arrays with elements in
  each array corresponding to the parameter settings (i.e. search
  candidates).

  The "cv_results_" dict can be easily imported into "pandas" as a
  "DataFrame" for exploring the search results.

  The "cv_results_" arrays include scores for each cross-validation
  split (with keys such as "'split0_test_score'"), as well as their
  mean ("'mean_test_score'") and standard deviation
  ("'std_test_score'").

  The ranks for the search candidates (based on their mean cross-
  validation score) is available at "cv_results_['rank_test_score']".

("'std_test_score'").

  The parameter values for each parameter is stored separately as
  numpy masked object arrays. The value, for that search candidate, is
  masked if the corresponding parameter is not applicable.
  Additionally a list of all the parameter dicts are stored at
  "cv_results_['params']".

* **Parameters n_folds and n_iter renamed to n_splits**

  Some parameter names have changed: The "n_folds" parameter in new
  "model_selection.KFold", "model_selection.GroupKFold" (see below for
  the name change), and "model_selection.StratifiedKFold" is now
  renamed to "n_splits". The "n_iter" parameter in
  "model_selection.ShuffleSplit", the new class
  "model_selection.GroupShuffleSplit" and
  "model_selection.StratifiedShuffleSplit" is now renamed to
  "n_splits".

* **Rename of splitter classes which accepts group labels along with
  data**

  The cross-validation splitters "LabelKFold", "LabelShuffleSplit",
  "LeaveOneLabelOut" and "LeavePLabelOut" have been renamed to
  "model_selection.GroupKFold", "model_selection.GroupShuffleSplit",
  "model_selection.LeaveOneGroupOut" and
  "model_selection.LeavePGroupsOut" respectively.

  Note the change from singular to plural form in
  "model_selection.LeavePGroupsOut".

* **Fit parameter labels renamed to groups**

  The "labels" parameter in the "split()" method of the newly renamed

Note the change from singular to plural form in

  splitters "model_selection.GroupKFold",
  "model_selection.LeaveOneGroupOut",
  "model_selection.LeavePGroupsOut",
  "model_selection.GroupShuffleSplit" is renamed to "groups" following
  the new nomenclature of their class names.

* **Parameter n_labels renamed to n_groups**

  The parameter "n_labels" in the newly renamed
  "model_selection.LeavePGroupsOut" is changed to "n_groups".

* Training scores and Timing information

  "cv_results_" also includes the training scores for each cross-
  validation split (with keys such as "'split0_train_score'"), as well
  as their mean ("'mean_train_score'") and standard deviation
  ("'std_train_score'"). To avoid the cost of evaluating training
  score, set "return_train_score=False".

  Additionally the mean and standard deviation of the times taken to
  split, train and score the model across all the cross-validation
  splits is available at the key "'mean_time'" and "'std_time'"
  respectively.

Changelog
=========

New features

Classifiers and Regressors

* The Gaussian Process module has been reimplemented and now offers
  classification and regression estimators through
  "gaussian_process.GaussianProcessClassifier" and
  "gaussian_process.GaussianProcessRegressor". Among other things, the

classification and regression estimators through
  new implementation supports kernel engineering, gradient-based
  hyperparameter optimization or sampling of functions from GP prior
  and GP posterior. Extensive documentation and examples are provided.
  By Jan Hendrik Metzen.

* Added new supervised learning algorithm: Multi-layer Perceptron
  >>:issue:`3204`<< by >>:user:`Issam H. Laradji <IssamLaradji>`<<

* Added "linear_model.HuberRegressor", a linear model robust to
  outliers. >>:issue:`5291`<< by Manoj Kumar.

* Added the "multioutput.MultiOutputRegressor" meta-estimator. It
  converts single output regressors to multi-output regressors by
  fitting one regressor per output. By >>:user:`Tim Head <betatim>`<<.

Other estimators

* New "mixture.GaussianMixture" and "mixture.BayesianGaussianMixture"
  replace former mixture models, employing faster inference for
  sounder results. >>:issue:`7295`<< by >>:user:`Wei Xue <xuewei4d>`<<
  and >>:user:`Thierry Guillemot <tguillemot>`<<.

* Class "decomposition.RandomizedPCA" is now factored into
  "decomposition.PCA" and it is available calling with parameter
  "svd_solver='randomized'". The default number of "n_iter" for
  "'randomized'" has changed to 4. The old behavior of PCA is
  recovered by "svd_solver='full'". An additional solver calls
  "arpack" and performs truncated (non-randomized) SVD. By default,

"'randomized'" has changed to 4. The old behavior of PCA is
  the best solver is selected depending on the size of the input and
  the number of components requested. >>:issue:`5299`<< by
  >>:user:`Giorgio Patrini <giorgiop>`<<.

* Added two functions for mutual information estimation:
  "feature_selection.mutual_info_regression()". These functions can be
  used in "feature_selection.SelectKBest" and
  "feature_selection.SelectPercentile" as score functions. By
  >>:user:`Andrea Bravi <AndreaBravi>`<< and >>:user:`Nikolay Mayorov
  <nmayorov>`<<.

* Added the "ensemble.IsolationForest" class for anomaly detection
  based on random forests. By Nicolas Goix.

* Added "algorithm="elkan"" to "cluster.KMeans" implementing Elkan's
  fast K-Means algorithm. By Andreas Müller.

Model selection and evaluation

* Added "metrics.cluster.fowlkes_mallows_score()", the Fowlkes Mallows
  Index which measures the similarity of two clusterings of a set of
  points By >>:user:`Arnaud Fouchet <afouchet>`<< and >>:user:`Thierry
  Guillemot <tguillemot>`<<.

* Added "metrics.calinski_harabaz_score()", which computes the
  Calinski and Harabaz score to evaluate the resulting clustering of a
  set of points. By >>:user:`Arnaud Fouchet <afouchet>`<< and
  >>:user:`Thierry Guillemot <tguillemot>`<<.

* Added new cross-validation splitter

set of points. By >>:user:`Arnaud Fouchet <afouchet>`<< and

  "model_selection.TimeSeriesSplit" to handle time series data.
  >>:issue:`6586`<< by >>:user:`YenChen Lin <yenchenlin>`<<

* The cross-validation iterators are replaced by cross-validation
  splitters available from "sklearn.model_selection", allowing for
  nested cross-validation. See Model Selection Enhancements and API
  Changes for more information. >>:issue:`4294`<< by Raghav RV.

Trees and ensembles

* Added a new splitting criterion for "tree.DecisionTreeRegressor",
  the mean absolute error. This criterion can also be used in
  "ensemble.ExtraTreesRegressor", "ensemble.RandomForestRegressor",
  and the gradient boosting estimators. >>:issue:`6667`<< by

* Added weighted impurity-based early stopping criterion for decision
  tree growth. >>:issue:`6954`<< by >>:user:`Nelson Liu <nelson-
  liu>`<<

* The random forest, extra tree and decision tree estimators now has a
  method "decision_path" which returns the decision path of samples in
  the tree. By Arnaud Joly.

* A new example has been added unveiling the decision tree structure.
  By Arnaud Joly.

* Random forest, extra trees, decision trees and gradient boosting
  estimator accept the parameter "min_samples_split" and
  "min_samples_leaf" provided as a percentage of the training samples.
  By >>:user:`yelite <yelite>`<< and Arnaud Joly.

estimator accept the parameter "min_samples_split" and

* Gradient boosting estimators accept the parameter "criterion" to
  specify to splitting criterion used in built decision trees.
  >>:issue:`6667`<< by >>:user:`Nelson Liu <nelson-liu>`<<.

* The memory footprint is reduced (sometimes greatly) for
  "ensemble.bagging.BaseBagging" and classes that inherit from it,
  i.e, "ensemble.BaggingClassifier", "ensemble.BaggingRegressor", and
  "ensemble.IsolationForest", by dynamically generating attribute
  "estimators_samples_" only when it is needed. By >>:user:`David
  Staub <staubda>`<<.

* Added "n_jobs" and "sample_weight" parameters for
  "ensemble.VotingClassifier" to fit underlying estimators in
  parallel. >>:issue:`5805`<< by >>:user:`Ibraim Ganiev <olologin>`<<.

* In "linear_model.LogisticRegression", the SAG solver is now
  available in the multinomial case. >>:issue:`5251`<< by Tom Dupre la
  Tour.

* "linear_model.RANSACRegressor", "svm.LinearSVC" and "svm.LinearSVR"
  now support "sample_weight". By >>:user:`Imaculate <Imaculate>`<<.

* Add parameter "loss" to "linear_model.RANSACRegressor" to measure
  the error on the samples for every trial. By Manoj Kumar.

* Prediction of out-of-sample events with Isotonic Regression
  ("isotonic.IsotonicRegression") is now much faster (over 1000x in

the error on the samples for every trial. By Manoj Kumar.

  tests with synthetic data). By >>:user:`Jonathan Arfa <jarfa>`<<.

* Isotonic regression ("isotonic.IsotonicRegression") now uses a
  better algorithm to avoid *O(n^2)* behavior in pathological cases,
  and is also generally faster (>>:issue:`#6691`<<). By Antony Lee.

* "naive_bayes.GaussianNB" now accepts data-independent class-priors
  through the parameter "priors". By >>:user:`Guillaume Lemaitre

* "linear_model.ElasticNet" and "linear_model.Lasso" now works with
  "np.float32" input data without converting it into "np.float64".
  This allows to reduce the memory consumption. >>:issue:`6913`<< by
  >>:user:`YenChen Lin <yenchenlin>`<<.

* "semi_supervised.LabelPropagation" and
  "semi_supervised.LabelSpreading" now accept arbitrary kernel
  functions in addition to strings "knn" and "rbf". >>:issue:`5762`<<
  by >>:user:`Utkarsh Upadhyay <musically-ut>`<<.

Decomposition, manifold learning and clustering

* Added "inverse_transform" function to "decomposition.NMF" to compute
  data matrix of original shape. By >>:user:`Anish Shah
  <AnishShah>`<<.

* "cluster.KMeans" and "cluster.MiniBatchKMeans" now works with
  "np.float32" and "np.float64" input data without converting it. This
  allows to reduce the memory consumption by using "np.float32".

  >>:issue:`6846`<< by >>:user:`Sebastian Säger <ssaeger>`<< and

Preprocessing and feature selection

* "preprocessing.RobustScaler" now accepts "quantile_range" parameter.
  >>:issue:`5929`<< by >>:user:`Konstantin Podshumok <podshumok>`<<.

* "feature_extraction.FeatureHasher" now accepts string values.
  >>:issue:`6173`<< by >>:user:`Ryad Zenine <ryadzenine>`<< and
  >>:user:`Devashish Deshpande <dsquareindia>`<<.

* Keyword arguments can now be supplied to "func" in
  "preprocessing.FunctionTransformer" by means of the "kw_args"
  parameter. By Brian McFee.

* "feature_selection.SelectKBest" and
  "feature_selection.SelectPercentile" now accept score functions that
  take X, y as input and return only the scores. By >>:user:`Nikolay
  Mayorov <nmayorov>`<<.

Model evaluation and meta-estimators

* "multiclass.OneVsOneClassifier" and "multiclass.OneVsRestClassifier"
  now support "partial_fit". By >>:user:`Asish Panda <kaichogami>`<<
  and >>:user:`Philipp Dowling <phdowling>`<<.

* Added support for substituting or disabling "pipeline.Pipeline" and
  "pipeline.FeatureUnion" components using the "set_params" interface
  that powers "sklearn.grid_search". See
  *sphx_glr_auto_examples_compose_plot_compare_reduction.py* By Joel

"pipeline.FeatureUnion" components using the "set_params" interface
  Nothman and >>:user:`Robert McGibbon <rmcgibbo>`<<.

* The new "cv_results_" attribute of "model_selection.GridSearchCV"
  (and "model_selection.RandomizedSearchCV") can be easily imported
  into pandas as a "DataFrame". Ref Model Selection Enhancements and
  API Changes for more information. >>:issue:`6697`<< by Raghav RV.

* Generalization of "model_selection.cross_val_predict()". One can
  pass method names such as *predict_proba* to be used in the cross
  validation framework instead of the default *predict*. By
  >>:user:`Ori Ziv <zivori>`<< and >>:user:`Sears Merritt
  <merritts>`<<.

* The training scores and time taken for training followed by scoring
  for each search candidate are now available at the "cv_results_"
  dict. See Model Selection Enhancements and API Changes for more
  information. >>:issue:`7325`<< by >>:user:`Eugene Chen <eyc88>`<<
  and Raghav RV.

Metrics

* Added "labels" flag to "metrics.log_loss" to explicitly provide the
  labels when the number of classes in "y_true" and "y_pred" differ.
  >>:issue:`7239`<< by >>:user:`Hong Guangguo <hongguangguo>`<< with
  help from >>:user:`Mads Jensen <indianajensen>`<< and

* Support sparse contingency matrices in cluster evaluation
  ("metrics.cluster.supervised") to scale to a large number of

>>:user:`Nelson Liu <nelson-liu>`<<.

  clusters. >>:issue:`7419`<< by >>:user:`Gregory Stupp <stuppie>`<<
  and Joel Nothman.

* Add "sample_weight" parameter to "metrics.matthews_corrcoef()". By
  >>:user:`Jatin Shah <jatinshah>`<< and Raghav RV.

* Speed up "metrics.silhouette_score()" by using vectorized
  operations. By Manoj Kumar.

* Add "sample_weight" parameter to "metrics.confusion_matrix()". By
  >>:user:`Bernardo Stein <DanielSidhion>`<<.

* Added "n_jobs" parameter to "feature_selection.RFECV" to compute the
  score on the test folds in parallel. By Manoj Kumar

* Codebase does not contain C/C++ cython generated files: they are
  generated during build. Distribution packages will still contain
  generated C/C++ files. By >>:user:`Arthur Mensch <arthurmensch>`<<.

* Reduce the memory usage for 32-bit float input arrays of
  "utils.sparse_func.mean_variance_axis()" and
  "utils.sparse_func.incr_mean_variance_axis()" by supporting cython
  fused types. By >>:user:`YenChen Lin <yenchenlin>`<<.

* The "ignore_warnings()" now accept a category argument to ignore
  only the warnings of a specified type. By >>:user:`Thierry Guillemot
  <tguillemot>`<<.

* Added parameter "return_X_y" and return type "(data, target) :
  tuple" option to "load_iris()" dataset >>:issue:`7049`<<,
  "load_breast_cancer()" dataset >>:issue:`7152`<<, "load_digits()"

  dataset, "load_diabetes()" dataset, "load_linnerud()" dataset,
  "load_boston()" dataset >>:issue:`7154`<< by >>:user:`Manvendra
  Singh<manu-chroma>`<<.

* Simplification of the "clone" function, deprecate support for
  estimators that modify parameters in "__init__". >>:issue:`5540`<<
  by Andreas Müller.

* When unpickling a scikit-learn estimator in a different version than
  the one the estimator was trained with, a "UserWarning" is raised,
  see the documentation on model persistence for more details.
  (>>:issue:`7248`<<) By Andreas Müller.

Bug fixes
---------

  won't accept anymore "min_samples_split=1" as at least 2 samples are
  required to split a decision tree node. By Arnaud Joly

* "ensemble.VotingClassifier" now raises "NotFittedError" if
  "predict", "transform" or "predict_proba" are called on the non-
  fitted estimator. by Sebastian Raschka.

* Fix bug where "ensemble.AdaBoostClassifier" and
  "ensemble.AdaBoostRegressor" would perform poorly if the
  "random_state" was fixed (>>:issue:`7411`<<). By Joel Nothman.

* Fix bug in ensembles with randomization where the ensemble would not
  set "random_state" on base estimators in a pipeline or similar
  nesting. (>>:issue:`7411`<<). Note, results for

  "ensemble.BaggingClassifier" "ensemble.BaggingRegressor",
  "ensemble.AdaBoostClassifier" and "ensemble.AdaBoostRegressor" will
  now differ from previous versions. By Joel Nothman.

* Fixed incorrect gradient computation for
  "loss='squared_epsilon_insensitive'" in "linear_model.SGDClassifier"
  and "linear_model.SGDRegressor" (>>:issue:`6764`<<). By
  >>:user:`Wenhua Yang <geekoala>`<<.

* Fix bug in "linear_model.LogisticRegressionCV" where
  "solver='liblinear'" did not accept "class_weights='balanced".
  (>>:issue:`6817`<<). By Tom Dupre la Tour.

* Fix bug in "neighbors.RadiusNeighborsClassifier" where an error
  occurred when there were outliers being labelled and a weight
  function specified (>>:issue:`6902`<<).  By LeonieBorne.

* Fix "linear_model.ElasticNet" sparse decision function to match
  output with dense in the multioutput case.

* "decomposition.RandomizedPCA" default number of *iterated_power* is
  4 instead of 3. >>:issue:`5141`<< by >>:user:`Giorgio Patrini
  <giorgiop>`<<.

* "utils.extmath.randomized_svd()" performs 4 power iterations by
  default, instead or 0. In practice this is enough for obtaining a
  good approximation of the true eigenvalues/vectors in the presence

  of noise. When *n_components* is small ("< .1 * min(X.shape)")
  *n_iter* is set to 7, unless the user specifies a higher number.
  This improves precision with few components. >>:issue:`5299`<< by
  >>:user:`Giorgio Patrini<giorgiop>`<<.

* Whiten/non-whiten inconsistency between components of
  "decomposition.PCA" and "decomposition.RandomizedPCA" (now factored
  into PCA, see the New features) is fixed. *components_* are stored
  with no whitening. >>:issue:`5299`<< by >>:user:`Giorgio Patrini

* Fixed bug in "manifold.spectral_embedding()" where diagonal of
  unnormalized Laplacian matrix was incorrectly set to 1.
  >>:issue:`4995`<< by >>:user:`Peter Fischer <yanlend>`<<.

* Fixed incorrect initialization of "utils.arpack.eigsh()" on all
  occurrences. Affects "cluster.bicluster.SpectralBiclustering",
  "decomposition.KernelPCA", "manifold.LocallyLinearEmbedding", and
  "manifold.SpectralEmbedding" (>>:issue:`5012`<<). By >>:user:`Peter
  Fischer <yanlend>`<<.

* Attribute "explained_variance_ratio_" calculated with the SVD solver
  of "discriminant_analysis.LinearDiscriminantAnalysis" now returns
  correct results. By >>:user:`JPFrancoia <JPFrancoia>`<<

* "preprocessing.data._transform_selected()" now always passes a copy

correct results. By >>:user:`JPFrancoia <JPFrancoia>`<<

  of "X" to transform function when "copy=True" (>>:issue:`7194`<<).
  By Caio Oliveira.

* "model_selection.StratifiedKFold" now raises error if all n_labels
  for individual classes is less than n_folds. >>:issue:`6182`<< by

* Fixed bug in "model_selection.StratifiedShuffleSplit" where train
  and test sample could overlap in some edge cases, see
  >>:issue:`6121`<< for more details. By Loic Esteve.

* Fix in "sklearn.model_selection.StratifiedShuffleSplit" to return
  splits of size "train_size" and "test_size" in all cases
  (>>:issue:`6472`<<). By Andreas Müller.

* Cross-validation of "OneVsOneClassifier" and "OneVsRestClassifier"
  now works with precomputed kernels. >>:issue:`7350`<< by
  >>:user:`Russell Smith <rsmith54>`<<.

* Fix incomplete "predict_proba" method delegation from
  "model_selection.GridSearchCV" to "linear_model.SGDClassifier"
  (>>:issue:`7159`<<) by Yichuan Liu.

Metrics

* Fix bug in "metrics.silhouette_score()" in which clusters of size 1
  were incorrectly scored. They should get a score of 0. By Joel
  Nothman.

* Fix bug in "metrics.silhouette_samples()" so that it now works with
  arbitrary labels, not just those ranging from 0 to n_clusters - 1.

* Fix bug where expected and adjusted mutual information were

  incorrect if cluster contingency cells exceeded "2**16". By Joel
  Nothman.

* "metrics.pairwise.pairwise_distances()" now converts arrays to
  boolean arrays when required in "scipy.spatial.distance".
  >>:issue:`5460`<< by Tom Dupre la Tour.

* Fix sparse input support in "metrics.silhouette_score()" as well as
  example examples/text/document_clustering.py. By >>:user:`YenChen
  Lin <yenchenlin>`<<.

* "metrics.roc_curve()" and "metrics.precision_recall_curve()" no
  longer round "y_score" values when creating ROC curves; this was
  causing problems for users with very small differences in scores
  (>>:issue:`7353`<<).

* "model_selection.tests._search._check_param_grid()" now works
  correctly with all types that extends/implements *Sequence* (except
  string), including range (Python 3.x) and xrange (Python 2.x).
  >>:issue:`7323`<< by Viacheslav Kovalevskyi.

* "utils.extmath.randomized_range_finder()" is more numerically stable
  when many power iterations are requested, since it applies LU
  normalization by default. If "n_iter<2" numerical issues are
  unlikely, thus no normalization is applied. Other normalization
  options are available: "'none', 'LU'" and "'QR'". >>:issue:`5141`<<
  by >>:user:`Giorgio Patrini <giorgiop>`<<.

unlikely, thus no normalization is applied. Other normalization

* Fix a bug where some formats of "scipy.sparse" matrix, and
  estimators with them as parameters, could not be passed to
  "base.clone()". By Loic Esteve.

* "datasets.load_svmlight_file()" now is able to read long int QID
  values. >>:issue:`7101`<< by >>:user:`Ibraim Ganiev <olologin>`<<.

* "residual_metric" has been deprecated in
  "linear_model.RANSACRegressor". Use "loss" instead. By Manoj Kumar.

* Access to public attributes ".X_" and ".y_" has been deprecated in
  "isotonic.IsotonicRegression". By >>:user:`Jonathan Arfa <jarfa>`<<.

* The old "mixture.DPGMM" is deprecated in favor of the new
  "mixture.BayesianGaussianMixture" (with the parameter
  "weight_concentration_prior_type='dirichlet_process'"). The new
  class solves the computational problems of the old class and
  computes the Gaussian mixture with a Dirichlet process prior faster
  than before. >>:issue:`7295`<< by >>:user:`Wei Xue <xuewei4d>`<< and

* The old "mixture.VBGMM" is deprecated in favor of the new
  "weight_concentration_prior_type='dirichlet_distribution'"). The new

  computes the Variational Bayesian Gaussian mixture faster than
  before. >>:issue:`6651`<< by >>:user:`Wei Xue <xuewei4d>`<< and

* The old "mixture.GMM" is deprecated in favor of the new
  "mixture.GaussianMixture". The new class computes the Gaussian
  mixture faster than before and some of computational problems have
  been solved. >>:issue:`6666`<< by >>:user:`Wei Xue <xuewei4d>`<< and

* The "sklearn.cross_validation", "sklearn.grid_search" and
  "sklearn.learning_curve" have been deprecated and the classes and
  functions have been reorganized into the "sklearn.model_selection"
  module. Ref Model Selection Enhancements and API Changes for more
  information. >>:issue:`4294`<< by Raghav RV.

* The "grid_scores_" attribute of "model_selection.GridSearchCV" and
  "model_selection.RandomizedSearchCV" is deprecated in favor of the
  attribute "cv_results_". Ref Model Selection Enhancements and API
  Changes for more information. >>:issue:`6697`<< by Raghav RV.

* The parameters "n_iter" or "n_folds" in old CV splitters are
  replaced by the new parameter "n_splits" since it can provide a

Changes for more information. >>:issue:`6697`<< by Raghav RV.

  consistent and unambiguous interface to represent the number of
  train-test splits. >>:issue:`7187`<< by >>:user:`YenChen Lin
  <yenchenlin>`<<.

* "classes" parameter was renamed to "labels" in
  "metrics.hamming_loss()". >>:issue:`7260`<< by >>:user:`Sebastián
  Vanrell <srvanrell>`<<.

* The splitter classes "LabelKFold", "LabelShuffleSplit",
  "LeaveOneLabelOut" and "LeavePLabelsOut" are renamed to
  "model_selection.LeavePGroupsOut" respectively. Also the parameter
  "labels" in the "split()" method of the newly renamed splitters
  "model_selection.LeavePGroupsOut" is renamed to "groups".
  Additionally in "model_selection.LeavePGroupsOut", the parameter
  "n_labels" is renamed to "n_groups". >>:issue:`6660`<< by Raghav RV.

* Error and loss names for "scoring" parameters are now prefixed by
  "'neg_'", such as "neg_mean_squared_error". The unprefixed versions
  are deprecated and will be removed in version 0.20.
  >>:issue:`7261`<< by >>:user:`Tim Head <betatim>`<<.

Aditya Joshi, Alejandro, Alexander Fabisch, Alexander Loginov,
Alexander Minyushkin, Alexander Rudy, Alexandre Abadie, Alexandre

Abraham, Alexandre Gramfort, Alexandre Saint, alexfields, Alvaro
Ulloa, alyssaq, Amlan Kar, Andreas Mueller, andrew giessel, Andrew
Jackson, Andrew McCulloh, Andrew Murray, Anish Shah, Arafat, Archit
Sharma, Ariel Rokem, Arnaud Joly, Arnaud Rachez, Arthur Mensch, Ash
Hoover, asnt, b0noI, Behzad Tabibian, Bernardo, Bernhard Kratzwald,
Bhargav Mangipudi, blakeflei, Boyuan Deng, Brandon Carter, Brett Naul,
Brian McFee, Caio Oliveira, Camilo Lamus, Carol Willing, Cass, CeShine
Lee, Charles Truong, Chyi-Kwei Yau, CJ Carey, codevig, Colin Ni, Dan
Shiebler, Daniel, Daniel Hnyk, David Ellis, David Nicholson, David
Staub, David Thaler, David Warshaw, Davide Lasagna, Deborah,
definitelyuncertain, Didi Bar-Zev, djipey, dsquareindia, edwinENSAE,
Elias Kuthe, Elvis DOHMATOB, Ethan White, Fabian Pedregosa, Fabio
Ticconi, fisache, Florian Wilhelm, Francis, Francis O'Donovan, Gael
Varoquaux, Ganiev Ibraim, ghg, Gilles Louppe, Giorgio Patrini,
Giovanni Cherubin, Giovanni Lanzani, Glenn Qian, Gordon Mohr, govin-
vatsan, Graham Clenaghan, Greg Reda, Greg Stupp, Guillaume Lemaitre,
Gustav Mörtberg, halwai, Harizo Rajaona, Harry Mavroforakis,
hashcode55, hdmetor, Henry Lin, Hobson Lane, Hugo Bowne-Anderson, Igor
Andriushchenko, Imaculate, Inki Hwang, Isaac Sijaranamual, Ishank
Gulati, Issam Laradji, Iver Jordal, jackmartin, Jacob Schreiber, Jake

Vanderplas, James Fiedler, James Routley, Jan Zikes, Janna Brettingen,
jarfa, Jason Laska, jblackburne, jeff levesque, Jeffrey Blackburne,
Jeffrey04, Jeremy Hintz, jeremynixon, Jeroen, Jessica Yung, Jill-Jênn
Vie, Jimmy Jia, Jiyuan Qian, Joel Nothman, johannah, John, John
Boersma, John Kirkham, John Moeller, jonathan.striebel, joncrall,
Jordi, Joseph Munoz, Joshua Cook, JPFrancoia, jrfiedler,
JulianKahnert, juliathebrave, kaichogami, KamalakerDadi, Kenneth
Lyons, Kevin Wang, kingjr, kjell, Konstantin Podshumok, Kornel
Kielczewski, Krishna Kalyan, krishnakalyan3, Kvle Putnam, Kyle
Jackson, Lars Buitinck, ldavid, LeiG, LeightonZhang, Leland McInnes,
Liang-Chi Hsieh, Lilian Besson, lizsz, Loic Esteve, Louis Tiao, Léonie
Borne, Mads Jensen, Maniteja Nandana, Manoj Kumar, Manvendra Singh,
Marco, Mario Krell, Mark Bao, Mark Szepieniec, Martin Madsen,
MartinBpr, MaryanMorel, Massil, Matheus, Mathieu Blondel, Mathieu
Dubois, Matteo, Matthias Ekman, Max Moroz, Michael Scherer, michiaki
ariga, Mikhail Korobov, Moussa Taifi, mrandrewandrade, Mridul Seth,
nadya-p, Naoya Kanai, Nate George, Nelle Varoquaux, Nelson Liu, Nick
James, NickleDave, Nico, Nicolas Goix, Nikolay Mayorov, ningchi,
nlathia, okbalefthanded, Okhlopkov, Olivier Grisel, Panos Louridas,
Paul Strickland, Perrine Letellier, pestrickland, Peter Fischer,

Pieter, Ping-Yao, Chang, practicalswift, Preston Parry, Qimu Zheng,
Rachit Kansal, Raghav RV, Ralf Gommers, Ramana.S, Rammig, Randy Olson,
Rob Alexander, Robert Lutz, Robin Schucker, Rohan Jain, Ruifeng Zheng,
Ryan Yu, Rémy Léone, saihttam, Saiwing Yeung, Sam Shleifer, Samuel St-
Jean, Sartaj Singh, Sasank Chilamkurthy, saurabh.bansod, Scott
Andrews, Scott Lowe, seales, Sebastian Raschka, Sebastian Saeger,
Sebastián Vanrell, Sergei Lebedev, shagun Sodhani, shanmuga cv,
Shashank Shekhar, shawpan, shengxiduan, Shota, shuckle16, Skipper
Seabold, sklearn-ci, SmedbergM, srvanrell, Sébastien Lerique,
Taranjeet, themrmax, Thierry, Thierry Guillemot, Thomas, Thomas
Hallock, Thomas Moreau, Tim Head, tKammy, toastedcornflakes, Tom,
TomDLT, Toshihiro Kamishima, tracer0tong, Trent Hauck, trevorstephens,
Tue Vo, Varun, Varun Jewalikar, Viacheslav, Vighnesh Birodkar, Vikram,
Villu Ruusmann, Vinayak Mehta, walter, waterponey, Wenhua Yang,
Wenjian Huang, Will Welch, wyseguy7, xyguo, yanlend, Yaroslav
Halchenko, yelite, Yen, YenChenLin, Yichuan Liu, Yoav Ram, Yoshiki,
Zheng RuiFeng, zivori, Óscar Nájera

Version 0.19.2

**July, 2018**

This release is exclusively in order to support Python 3.7.

Related changes

* "n_iter_" may vary from previous releases in
  "linear_model.HuberRegressor".  For Scipy <= 1.0.0, the optimizer

Version 0.19.1

**October 23, 2017**

and enhancements to features released in 0.19.0.

Note there may be minor differences in TSNE output in this release
(due to >>:issue:`9623`<<), in the case where multiple samples have
equal distance to some sample.

Changelog
=========

API changes

* Reverted the addition of "metrics.ndcg_score" and
  "metrics.dcg_score" which had been merged into version 0.19.0 by
  error.  The implementations were broken and undocumented.

* "return_train_score" which was added to
  "model_selection.GridSearchCV", "model_selection.RandomizedSearchCV"
  and "model_selection.cross_validate()" in version 0.19.0 will be
  changing its default value from True to False in version 0.21.  We
  found that calculating training score could have a great effect on
  cross validation runtime in some cases.  Users should explicitly set
  "return_train_score" to False if prediction or scoring functions are

cross validation runtime in some cases.  Users should explicitly set
  slow, resulting in a deleterious effect on CV runtime, or to True if
  they wish to use the calculated scores. >>:issue:`9677`<< by
  >>:user:`Kumar Ashutosh <thechargedneutron>`<< and Joel Nothman.

* "correlation_models" and "regression_models" from the legacy
  gaussian processes implementation have been belatedly deprecated.
  >>:issue:`9717`<< by >>:user:`Kumar Ashutosh <thechargedneutron>`<<.

Bug fixes
---------

* Avoid integer overflows in "metrics.matthews_corrcoef()".
  >>:issue:`9693`<< by >>:user:`Sam Steingold <sam-s>`<<.

* Fixed a bug in the objective function for "manifold.TSNE" (both
  exact and with the Barnes-Hut approximation) when "n_components >=
  3". >>:issue:`9711`<< by >>:user:`goncalo-rodrigues`<<.

* Fix regression in "model_selection.cross_val_predict()" where it
  raised an error with "method='predict_proba'" for some probabilistic
  classifiers. >>:issue:`9641`<< by >>:user:`James Bourbeau
  <jrbourbeau>`<<.

* Fixed a bug where "datasets.make_classification()" modified its
  input "weights". >>:issue:`9865`<< by >>:user:`Sachin Kelkar
  <s4chin>`<<.

* "model_selection.StratifiedShuffleSplit" now works with multioutput
  multiclass or multilabel data with more than 1000 columns.
  >>:issue:`9922`<< by >>:user:`Charlie Brummitt <crbrummitt>`<<.

* Fixed a bug with nested and conditional parameter setting, e.g.

multiclass or multilabel data with more than 1000 columns.

  setting a pipeline step and its parameter at the same time.
  >>:issue:`9945`<< by Andreas Müller and Joel Nothman.

Regressions in 0.19.0 fixed in 0.19.1:

* Fixed a bug where parallelised prediction in random forests was not
  thread-safe and could (rarely) result in arbitrary errors.
  >>:issue:`9830`<< by Joel Nothman.

* Fix regression in "model_selection.cross_val_predict()" where it no
  longer accepted "X" as a list. >>:issue:`9600`<< by >>:user:`Rasul
  Kerimov <CoderINusE>`<<.

* Fixed handling of "cross_val_predict()" for binary classification
  with "method='decision_function'". >>:issue:`9593`<< by
  >>:user:`Reiichiro Nakano <reiinakano>`<< and core devs.

* Fix regression in "pipeline.Pipeline" where it no longer accepted
  "steps" as a tuple. >>:issue:`9604`<< by >>:user:`Joris Van den
  Bossche <jorisvandenbossche>`<<.

* Fix bug where "n_iter" was not properly deprecated, leaving "n_iter"
  unavailable for interim use in "linear_model.SGDClassifier",
  "linear_model.Perceptron". >>:issue:`9558`<< by Andreas Müller.

* Dataset fetchers make sure temporary files are closed before
  removing them, which caused errors on Windows. >>:issue:`9847`<< by

"linear_model.Perceptron". >>:issue:`9558`<< by Andreas Müller.

  >>:user:`Joan Massich <massich>`<<.

* Fixed a regression in "manifold.TSNE" where it no longer supported
  metrics other than 'euclidean' and 'precomputed'. >>:issue:`9623`<<
  by >>:user:`Oli Blum <oliblum90>`<<.

* Our test suite and "utils.estimator_checks.check_estimators()" can
  now be run without Nose installed. >>:issue:`9697`<< by

* To improve usability of version 0.19's "pipeline.Pipeline" caching,
  "memory" now allows "joblib.Memory" instances. This make use of the
  new "utils.validation.check_memory()" helper. issue:*9584* by
  >>:user:`Kumar Ashutosh <thechargedneutron>`<<

* Some fixes to examples: >>:issue:`9750`<<, >>:issue:`9788`<<,
  >>:issue:`9815`<<

* Made a FutureWarning in SGD-based estimators less verbose.
  >>:issue:`9802`<< by >>:user:`Vrishank Bhardwaj <vrishank97>`<<.

Joel Nothman, Loic Esteve, Andreas Mueller, Kumar Ashutosh, Vrishank
Bhardwaj, Hanmin Qin, Rasul Kerimov, James Bourbeau, Nagarjuna Kumar,
Nathaniel Saul, Olivier Grisel, Roman Yurchak, Reiichiro Nakano,
Sachin Kelkar, Sam Steingold, Yaroslav Halchenko, diegodlh, felix,
goncalo-rodrigues, jkleint, oliblum90, pasbi, Anthony Gitter, Ben

Lawson, Charlie Brummitt, Didi Bar-Zev, Gael Varoquaux, Joan Massich,
Joris Van den Bossche, nielsenmarkus11

Version 0.19

**August 12, 2017**

We are excited to release a number of great new features including
"neighbors.LocalOutlierFactor" for anomaly detection,
"preprocessing.QuantileTransformer" for robust feature transformation,
and the "multioutput.ClassifierChain" meta-estimator to simply account
for dependencies between classes in multilabel problems. We have some
new algorithms in existing estimators, such as multiplicative update
in "decomposition.NMF" and multinomial
"linear_model.LogisticRegression" with L1 loss (use "solver='saga'").

Cross validation is now able to return the results from multiple
metric evaluations. The new "model_selection.cross_validate()" can
return many scores on the test data as well as training set
performance and timings, and we have extended the "scoring" and
"refit" parameters for grid/randomized search to handle multiple
metrics.

You can also learn faster.  For instance, the new option to cache
transformations in "pipeline.Pipeline" makes grid search over
pipelines including slow transformations much more efficient.  And you
can predict faster: if you're sure you know what you're doing, you can

turn off validating that the input is finite using "config_context()".

We've made some important fixes too.  We've fixed a longstanding
implementation error in "metrics.average_precision_score()", so please
be cautious with prior results reported from that function.  A number
of errors in the "manifold.TSNE" implementation have been fixed,
particularly in the default Barnes-Hut approximation.
"semi_supervised.LabelSpreading" and
"semi_supervised.LabelPropagation" have had substantial fixes.
LabelPropagation was previously broken. LabelSpreading should now
correctly respect its alpha parameter.

* "cluster.KMeans" with sparse X and initial centroids given (bug fix)

* "cross_decomposition.PLSRegression" with "scale=True" (bug fix)

* "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor" where "min_impurity_split" is
  used (bug fix)

* gradient boosting "loss='quantile'" (bug fix)

* "ensemble.IsolationForest" (bug fix)

* "feature_selection.SelectFdr" (bug fix)

* "linear_model.RANSACRegressor" (bug fix)

* "linear_model.LassoLars" (bug fix)

* "linear_model.LassoLarsIC" (bug fix)

* "manifold.TSNE" (bug fix)

* "neighbors.NearestCentroid" (bug fix)

* "semi_supervised.LabelSpreading" (bug fix)

* "semi_supervised.LabelPropagation" (bug fix)

* tree based models where "min_weight_fraction_leaf" is used
  (enhancement)

* "model_selection.StratifiedKFold" with "shuffle=True" (this change,
  due to >>:issue:`7823`<< was not mentioned in the release notes at
  the time)

Changelog
=========

Classifiers and regressors

* Added "multioutput.ClassifierChain" for multi-label classification.
  By >>:user:`Adam Kleczewski <adamklec>`<<.

* Added solver "'saga'" that implements the improved version of
  Stochastic Average Gradient, in "linear_model.LogisticRegression"
  and "linear_model.Ridge". It allows the use of L1 penalty with
  multinomial logistic loss, and behaves marginally better than 'sag'
  during the first epochs of ridge and logistic regression.
  >>:issue:`8446`<< by Arthur Mensch.

* Added the "neighbors.LocalOutlierFactor" class for anomaly detection
  based on nearest neighbors. >>:issue:`5279`<< by Nicolas Goix and

* Added "preprocessing.QuantileTransformer" class and
  "preprocessing.quantile_transform()" function for features
  normalization based on quantiles. >>:issue:`8363`<< by
  >>:user:`Denis Engemann <dengemann>`<<, >>:user:`Guillaume Lemaitre
  <glemaitre>`<<, Olivier Grisel, Raghav RV, >>:user:`Thierry
  Guillemot <tguillemot>`<<, and Gael Varoquaux.

* The new solver "'mu'" implements a Multiplicate Update in
  "decomposition.NMF", allowing the optimization of all beta-
  divergences, including the Frobenius norm, the generalized Kullback-
  Leibler divergence and the Itakura-Saito divergence.
  >>:issue:`5295`<< by Tom Dupre la Tour.

* "model_selection.GridSearchCV" and
  "model_selection.RandomizedSearchCV" now support simultaneous
  evaluation of multiple metrics. Refer to the Specifying multiple
  metrics for evaluation section of the user guide for more
  information. >>:issue:`7388`<< by Raghav RV

* Added the "model_selection.cross_validate()" which allows evaluation
  of multiple metrics. This function returns a dict with more useful
  information from cross-validation such as the train scores, fit
  times and score times. Refer to The cross_validate function and
  multiple metric evaluation section of the userguide for more

times and score times. Refer to The cross_validate function and

* Added "metrics.mean_squared_log_error()", which computes the mean
  square error of the logarithmic transformation of targets,
  particularly useful for targets with an exponential trend.
  >>:issue:`7655`<< by >>:user:`Karan Desai <karandesai-96>`<<.

* Added "metrics.dcg_score()" and "metrics.ndcg_score()", which
  compute Discounted cumulative gain (DCG) and Normalized discounted
  cumulative gain (NDCG). >>:issue:`7739`<< by >>:user:`David Gasquez
  <davidgasquez>`<<.

* Added the "model_selection.RepeatedKFold" and
  "model_selection.RepeatedStratifiedKFold". >>:issue:`8120`<< by
  Neeraj Gangwar.

* Validation that input data contains no NaN or inf can now be
  suppressed using "config_context()", at your own risk. This will
  save on runtime, and may be particularly useful for prediction time.
  >>:issue:`7548`<< by Joel Nothman.

* Added a test to ensure parameter listing in docstrings match the
  function/class signature. >>:issue:`9206`<< by Alexandre Gramfort

* The "min_weight_fraction_leaf" constraint in tree construction is
  now more efficient, taking a fast path to declare a node a leaf if
  its weight is less than 2 * the minimum. Note that the constructed
  tree will be different from previous versions where

now more efficient, taking a fast path to declare a node a leaf if
  "min_weight_fraction_leaf" is used. >>:issue:`7441`<< by

  "ensemble.GradientBoostingRegressor" now support sparse input for
  prediction. >>:issue:`6101`<< by >>:user:`Ibraim Ganiev
  <olologin>`<<.

* "ensemble.VotingClassifier" now allows changing estimators by using
  "ensemble.VotingClassifier.set_params()". An estimator can also be
  removed by setting it to "None". >>:issue:`7674`<< by
  >>:user:`Yichuan Liu <yl565>`<<.

* "tree.export_graphviz()" now shows configurable number of decimal
  places. >>:issue:`8698`<< by >>:user:`Guillaume Lemaitre

* Added "flatten_transform" parameter to "ensemble.VotingClassifier"
  to change output shape of *transform* method to 2 dimensional.
  >>:issue:`7794`<< by >>:user:`Ibraim Ganiev <olologin>`<< and

* "linear_model.SGDClassifier", "linear_model.SGDRegressor",
  "linear_model.Perceptron" now expose "max_iter" and "tol"
  parameters, to handle convergence more precisely. "n_iter" parameter
  is deprecated, and the fitted estimator exposes a "n_iter_"

"linear_model.Perceptron" now expose "max_iter" and "tol"
  attribute, with actual number of iterations before convergence.
  >>:issue:`5036`<< by Tom Dupre la Tour.

* Added "average" parameter to perform weight averaging in
  "linear_model.PassiveAggressiveClassifier". >>:issue:`4939`<< by
  >>:user:`Andrea Esuli <aesuli>`<<.

* "linear_model.RANSACRegressor" no longer throws an error when
  calling "fit" if no inliers are found in its first iteration.
  Furthermore, causes of skipped iterations are tracked in newly added
  attributes, "n_skips_*". >>:issue:`7914`<< by >>:user:`Michael
  Horrell <mthorrell>`<<.

* In "gaussian_process.GaussianProcessRegressor", method "predict" is
  a lot faster with "return_std=True". >>:issue:`8591`<< by
  >>:user:`Hadrien Bertrand <hbertrand>`<<.

* Added "return_std" to "predict" method of
  "linear_model.ARDRegression" and "linear_model.BayesianRidge".
  >>:issue:`7838`<< by >>:user:`Sergey Feldman <sergeyf>`<<.

* Memory usage enhancements: Prevent cast from float32 to float64 in:
  "linear_model.MultiTaskElasticNet";
  "linear_model.LogisticRegression" when using newton-cg solver; and
  "linear_model.Ridge" when using svd, sparse_cg, cholesky or lsqr
  solvers. >>:issue:`8835`<<, >>:issue:`8061`<< by >>:user:`Joan
  Massich <massich>`<< and >>:user:`Nicolas Cordier <ncordier>`<< and

solvers. >>:issue:`8835`<<, >>:issue:`8061`<< by >>:user:`Joan

Other predictors

* Custom metrics for the "neighbors" binary trees now have fewer
  constraints: they must take two 1d-arrays and return a float.
  >>:issue:`6288`<< by Jake Vanderplas.

* "algorithm='auto" in "neighbors" estimators now chooses the most
  appropriate algorithm for all input types and metrics.
  >>:issue:`9145`<< by >>:user:`Herilalaina Rakotoarison
  <herilalaina>`<< and >>:user:`Reddy Chinthala <preddy5>`<<.

* "cluster.MiniBatchKMeans" and "cluster.KMeans" now use significantly
  less memory when assigning data points to their nearest cluster
  center. >>:issue:`7721`<< by >>:user:`Jon Crall <Erotemic>`<<.

* "decomposition.PCA", "decomposition.IncrementalPCA" and
  "decomposition.TruncatedSVD" now expose the singular values from the
  underlying SVD. They are stored in the attribute "singular_values_",
  like in "decomposition.IncrementalPCA". >>:issue:`7685`<< by
  >>:user:`Tommy Löfstedt <tomlof>`<<

* "decomposition.NMF" now faster when "beta_loss=0". >>:issue:`9277`<<
  by >>:user:`hongkahjun`<<.

* Memory improvements for method "barnes_hut" in "manifold.TSNE"
  >>:issue:`7089`<< by >>:user:`Thomas Moreau <tomMoral>`<< and

* Optimization schedule improvements for Barnes-Hut "manifold.TSNE" so

>>:issue:`7089`<< by >>:user:`Thomas Moreau <tomMoral>`<< and

  the results are closer to the one from the reference implementation
  lvdmaaten/bhtsne by >>:user:`Thomas Moreau <tomMoral>`<< and Olivier
  Grisel.

* Memory usage enhancements: Prevent cast from float32 to float64 in
  "decomposition.PCA" and "decomposition.randomized_svd_low_rank()".
  >>:issue:`9067`<< by Raghav RV.

* Added "norm_order" parameter to "feature_selection.SelectFromModel"
  to enable selection of the norm order when "coef_" is more than 1D.
  >>:issue:`6181`<< by >>:user:`Antoine Wendlinger <antoinewdg>`<<.

* Added ability to use sparse matrices in
  "feature_selection.f_regression()" with "center=True".
  >>:issue:`8065`<< by >>:user:`Daniel LeJeune <acadiansith>`<<.

* Small performance improvement to n-gram creation in
  "feature_extraction.text" by binding methods for loops and special-
  casing unigrams. >>:issue:`7567`<< by >>:user:`Jaye Doepke
  <jtdoepke>`<<

* Relax assumption on the data for the
  "kernel_approximation.SkewedChi2Sampler". Since the Skewed-Chi2
  kernel is defined on the open interval (-skewedness; +\infty)^d, the
  transform function should not check whether "X < 0" but whether "X <
  -self.skewedness". >>:issue:`7573`<< by >>:user:`Romain Brault
  <RomainBrault>`<<.

* Made default kernel parameters kernel-dependent in
  "kernel_approximation.Nystroem". >>:issue:`5229`<< by

-self.skewedness". >>:issue:`7573`<< by >>:user:`Romain Brault

  >>:user:`Saurabh Bansod <mth4saurabh>`<< and Andreas Müller.

* "pipeline.Pipeline" is now able to cache transformers within a
  pipeline by using the "memory" constructor parameter.
  >>:issue:`7990`<< by >>:user:`Guillaume Lemaitre <glemaitre>`<<.

* "pipeline.Pipeline" steps can now be accessed as attributes of its
  "named_steps" attribute. >>:issue:`8586`<< by >>:user:`Herilalaina
  Rakotoarison <herilalaina>`<<.

* Added "sample_weight" parameter to "pipeline.Pipeline.score()".
  >>:issue:`7723`<< by >>:user:`Mikhail Korobov <kmike>`<<.

* Added ability to set "n_jobs" parameter to "pipeline.make_union()".
  A "TypeError" will be raised for any other kwargs. >>:issue:`8028`<<
  by >>:user:`Alexander Booth <alexandercbooth>`<<.

* "model_selection.GridSearchCV", "model_selection.RandomizedSearchCV"
  and "model_selection.cross_val_score()" now allow estimators with
  callable kernels which were previously prohibited. >>:issue:`8005`<<
  by Andreas Müller .

* "model_selection.cross_val_predict()" now returns output of the
  correct shape for all values of the argument "method".
  >>:issue:`7863`<< by >>:user:`Aman Dalmia <dalmia>`<<.

* Added "shuffle" and "random_state" parameters to shuffle training
  data before taking prefixes of it based on training sizes in

>>:issue:`7863`<< by >>:user:`Aman Dalmia <dalmia>`<<.

  "model_selection.learning_curve()". >>:issue:`7506`<< by
  >>:user:`Narine Kokhlikyan <NarineK>`<<.

  multiclass (or multilabel) data.  >>:issue:`9044`<< by Vlad Niculae.

* Speed improvements to "model_selection.StratifiedShuffleSplit".
  >>:issue:`5991`<< by >>:user:`Arthur Mensch <arthurmensch>`<< and

* Add "shuffle" parameter to "model_selection.train_test_split()".
  >>:issue:`8845`<< by  >>:user:`themrmax <themrmax>`<<

* "multioutput.MultiOutputRegressor" and
  "multioutput.MultiOutputClassifier" now support online learning
  using "partial_fit". :issue: *8053* by >>:user:`Peng Yu
  <yupbank>`<<.

* Add "max_train_size" parameter to "model_selection.TimeSeriesSplit"
  >>:issue:`8282`<< by >>:user:`Aman Dalmia <dalmia>`<<.

* More clustering metrics are now available through
  "metrics.get_scorer()" and "scoring" parameters. >>:issue:`8117`<<

* A scorer based on "metrics.explained_variance_score()" is also
  available. >>:issue:`9259`<< by >>:user:`Hanmin Qin

Metrics

* "metrics.matthews_corrcoef()" now support multiclass classification.
  >>:issue:`8094`<< by >>:user:`Jon Crall <Erotemic>`<<.

* Add "sample_weight" parameter to "metrics.cohen_kappa_score()".

Metrics

  >>:issue:`8335`<< by >>:user:`Victor Poughon <vpoughon>`<<.

* "utils.check_estimator()" now attempts to ensure that methods
  transform, predict, etc.  do not set attributes on the estimator.
  >>:issue:`7533`<< by >>:user:`Ekaterina Krivich <kiote>`<<.

* Added type checking to the "accept_sparse" parameter in
  "utils.validation" methods. This parameter now accepts only boolean,
  string, or list/tuple of strings. "accept_sparse=None" is deprecated
  and should be replaced by "accept_sparse=False". >>:issue:`7880`<<
  by >>:user:`Josh Karnofsky <jkarno>`<<.

* Make it possible to load a chunk of an svmlight formatted file by
  passing a range of bytes to "datasets.load_svmlight_file()".
  >>:issue:`935`<< by >>:user:`Olivier Grisel <ogrisel>`<<.

* "dummy.DummyClassifier" and "dummy.DummyRegressor" now accept non-
  finite features. >>:issue:`8931`<< by >>:user:`Attractadore`<<.

Bug fixes
---------

* Fixed a memory leak in trees when using trees with
  "criterion='mae'". >>:issue:`8002`<< by Raghav RV.

* Fixed a bug where "ensemble.IsolationForest" uses an an incorrect
  formula for the average path length >>:issue:`8549`<< by Peter Wang.

* Fixed a bug where "ensemble.AdaBoostClassifier" throws
  "ZeroDivisionError" while fitting data with single class labels.

formula for the average path length >>:issue:`8549`<< by Peter Wang.

  >>:issue:`7501`<< by >>:user:`Dominik Krzeminski <dokato>`<<.

* Fixed a bug in "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor" where a float being compared to
  "0.0" using "==" caused a divide by zero error. >>:issue:`7970`<< by
  >>:user:`He Chen <chenhe95>`<<.

* Fix a bug where "ensemble.GradientBoostingClassifier" and
  "ensemble.GradientBoostingRegressor" ignored the
  "min_impurity_split" parameter. >>:issue:`8006`<< by
  >>:user:`Sebastian Pölsterl <sebp>`<<.

* Fixed "oob_score" in "ensemble.BaggingClassifier". >>:issue:`8936`<<
  by >>:user:`Michael Lewis <mlewis1729>`<<

* Fixed excessive memory usage in prediction for random forests
  estimators. >>:issue:`8672`<< by >>:user:`Mike Benfield
  <mikebenfield>`<<.

* Fixed a bug where "sample_weight" as a list broke random forests in
  Python 2 >>:issue:`8068`<< by >>:user:`xor`<<.

* Fixed a bug where "ensemble.IsolationForest" fails when
  "max_features" is less than 1. >>:issue:`5732`<< by >>:user:`Ishank
  Gulati <IshankGulati>`<<.

* Fix a bug where gradient boosting with "loss='quantile'" computed
  negative errors for negative values of "ytrue - ypred" leading to
  wrong values when calling "__call__". >>:issue:`8087`<< by
  >>:user:`Alexis Mignon <AlexisMignon>`<<

negative errors for negative values of "ytrue - ypred" leading to

* Fix a bug where "ensemble.VotingClassifier" raises an error when a
  numpy array is passed in for weights. >>:issue:`7983`<< by
  >>:user:`Vincent Pham <vincentpham1991>`<<.

* Fixed a bug where "tree.export_graphviz()" raised an error when the
  length of features_names does not match n_features in the decision
  tree. >>:issue:`8512`<< by >>:user:`Li Li <aikinogard>`<<.

* Fixed a bug where "linear_model.RANSACRegressor.fit()" may run until
  "max_iter" if it finds a large inlier group early. >>:issue:`8251`<<
  by >>:user:`aivision2020`<<.

* Fixed a bug where "naive_bayes.MultinomialNB" and
  "naive_bayes.BernoulliNB" failed when "alpha=0". >>:issue:`5814`<<
  by >>:user:`Yichuan Liu <yl565>`<< and >>:user:`Herilalaina

* Fixed a bug where "linear_model.LassoLars" does not give the same
  result as the LassoLars implementation available in R (lars
  library). >>:issue:`7849`<< by >>:user:`Jair Montoya Martinez
  <jmontoyam>`<<.

* Fixed a bug in "linear_model.RandomizedLasso", "linear_model.Lars",
  "linear_model.LassoLarsCV", where the parameter "precompute" was not
  used consistently across classes, and some values proposed in the
  docstring could raise errors. >>:issue:`5359`<< by Tom Dupre la
  Tour.

used consistently across classes, and some values proposed in the
  Tour.

* Fix inconsistent results between "linear_model.RidgeCV" and
  "linear_model.Ridge" when using "normalize=True". >>:issue:`9302`<<
  by Alexandre Gramfort.

* Fix a bug where "linear_model.LassoLars.fit()" sometimes left
  "coef_" as a list, rather than an ndarray. >>:issue:`8160`<< by
  >>:user:`CJ Carey <perimosocordiae>`<<.

* Fix "linear_model.BayesianRidge.fit()" to return ridge parameter
  "alpha_" and "lambda_" consistent with calculated coefficients
  "coef_" and "intercept_". >>:issue:`8224`<< by >>:user:`Peter Gedeck
  <gedeck>`<<.

* Fixed a bug in "svm.OneClassSVM" where it returned floats instead of
  integer classes. >>:issue:`8676`<< by >>:user:`Vathsala Achar
  <VathsalaAchar>`<<.

* Fix AIC/BIC criterion computation in "linear_model.LassoLarsIC".
  >>:issue:`9022`<< by Alexandre Gramfort and >>:user:`Mehmet Basbug
  <mehmetbasbug>`<<.

* Fixed a memory leak in our LibLinear implementation.
  >>:issue:`9024`<< by >>:user:`Sergei Lebedev <superbobry>`<<

* Fix bug where stratified CV splitters did not work with
  "linear_model.LassoCV". >>:issue:`8973`<< by >>:user:`Paulo Haddad
  <paulochf>`<<.

* Fixed a bug in "gaussian_process.GaussianProcessRegressor" when the
  standard deviation and covariance predicted without fit would fail
  with a unmeaningful error by default. >>:issue:`6573`<< by
  >>:user:`Quazi Marufur Rahman <qmaruf>`<< and Manoj Kumar.

standard deviation and covariance predicted without fit would fail

* Fix "semi_supervised.BaseLabelPropagation" to correctly implement
  "LabelPropagation" and "LabelSpreading" as done in the referenced
  papers. >>:issue:`9239`<< by >>:user:`Andre Ambrosio Boechat
  <boechat107>`<<, >>:user:`Utkarsh Upadhyay <musically-ut>`<<, and

* Fixed the implementation of "manifold.TSNE":

* "early_exageration" parameter had no effect and is now used for the
  first 250 optimization iterations.

* Fixed the "AssertionError: Tree consistency failed" exception
  reported in >>:issue:`8992`<<.

* Improve the learning schedule to match the one from the reference
  implementation lvdmaaten/bhtsne. by >>:user:`Thomas Moreau
  <tomMoral>`<< and Olivier Grisel.

* Fix a bug in "decomposition.LatentDirichletAllocation" where the
  "perplexity" method was returning incorrect results because the
  "transform" method returns normalized document topic distributions
  as of version 0.18. >>:issue:`7954`<< by >>:user:`Gary Foreman
  <garyForeman>`<<.

* Fix output shape and bugs with n_jobs > 1 in
  "decomposition.SparseCoder" transform and
  "decomposition.sparse_encode()" for one-dimensional data and one
  component. This also impacts the output shape of
  "decomposition.DictionaryLearning". >>:issue:`8086`<< by Andreas
  Müller.

"decomposition.sparse_encode()" for one-dimensional data and one
  Müller.

* Fixed the implementation of "explained_variance_" in
  "decomposition.PCA", "decomposition.RandomizedPCA" and
  "decomposition.IncrementalPCA". >>:issue:`9105`<< by Hanmin Qin.

* Fixed the implementation of "noise_variance_" in
  "decomposition.PCA". >>:issue:`9108`<< by Hanmin Qin.

* Fixed a bug where "cluster.DBSCAN" gives incorrect result when input
  is a precomputed sparse matrix with initial rows all zero.
  >>:issue:`8306`<< by >>:user:`Akshay Gupta <Akshay0724>`<<

* Fix a bug regarding fitting "cluster.KMeans" with a sparse array X
  and initial centroids, where X's means were unnecessarily being
  subtracted from the centroids. >>:issue:`7872`<< by >>:user:`Josh
  Karnofsky <jkarno>`<<.

* Fixes to the input validation in "covariance.EllipticEnvelope".
  >>:issue:`8086`<< by Andreas Müller.

* Fixed a bug in "covariance.MinCovDet" where inputting data that
  produced a singular covariance matrix would cause the helper method
  "_c_step" to throw an exception. >>:issue:`3367`<< by
  >>:user:`Jeremy Steward <ThatGeoGuy>`<<

* Fixed a bug in "manifold.TSNE" affecting convergence of the gradient
  descent. >>:issue:`8768`<< by >>:user:`David DeTomaso <deto>`<<.

* Fixed a bug in "manifold.TSNE" where it stored the incorrect
  "kl_divergence_". >>:issue:`6507`<< by >>:user:`Sebastian Saeger
  <ssaeger>`<<.

* Fixed improper scaling in "cross_decomposition.PLSRegression" with
  "scale=True". >>:issue:`7819`<< by >>:user:`jayzed82 <jayzed82>`<<.

* "cluster.bicluster.SpectralCoclustering" and
  "cluster.bicluster.SpectralBiclustering" "fit" method conforms with
  API by accepting "y" and returning the object.  >>:issue:`6126`<<,
  >>:issue:`7814`<< by >>:user:`Laurent Direr <ldirer>`<< and
  >>:user:`Maniteja Nandana <maniteja123>`<<.

* Fix bug where "mixture" "sample" methods did not return as many
  samples as requested. >>:issue:`7702`<< by >>:user:`Levi John Wolf
  <ljwolf>`<<.

* Fixed the shrinkage implementation in "neighbors.NearestCentroid".
  >>:issue:`9219`<< by Hanmin Qin.

* For sparse matrices, "preprocessing.normalize()" with
  "return_norm=True" will now raise a "NotImplementedError" with 'l1'
  or 'l2' norm and with norm 'max' the norms returned will be the same
  as for dense matrices. >>:issue:`7771`<< by Ang Lu.

* Fix a bug where "feature_selection.SelectFdr" did not exactly
  implement Benjamini-Hochberg procedure. It formerly may have

* Fixed a bug where "linear_model.RandomizedLasso" and
  "linear_model.RandomizedLogisticRegression" breaks for sparse input.
  >>:issue:`8259`<< by >>:user:`Aman Dalmia <dalmia>`<<.

* Fix a bug where "feature_extraction.FeatureHasher" mandatorily
  applied a sparse random projection to the hashed features,
  preventing the use of "feature_extraction.text.HashingVectorizer" in
  a pipeline with  "feature_extraction.text.TfidfTransformer".
  >>:issue:`7565`<< by >>:user:`Roman Yurchak <rth>`<<.

* Fix a bug where "feature_selection.mutual_info_regression" did not
  correctly use "n_neighbors". >>:issue:`8181`<< by >>:user:`Guillaume

* Fixed a bug where "model_selection.BaseSearchCV.inverse_transform()"
  returns "self.best_estimator_.transform()" instead of
  "self.best_estimator_.inverse_transform()". >>:issue:`8344`<< by
  >>:user:`Akshay Gupta <Akshay0724>`<< and >>:user:`Rasmus Eriksson
  <MrMjauh>`<<.

* Added "classes_" attribute to "model_selection.GridSearchCV",
  "model_selection.RandomizedSearchCV",  "grid_search.GridSearchCV",
  and  "grid_search.RandomizedSearchCV" that matches the "classes_"
  attribute of "best_estimator_". >>:issue:`7661`<< and
  >>:issue:`8295`<< by >>:user:`Alyssa Batula <abatula>`<<,
  >>:user:`Dylan Werner-Meier <unautre>`<<, and >>:user:`Stephen

* Fixed a bug where "model_selection.validation_curve()" reused the

>>:user:`Dylan Werner-Meier <unautre>`<<, and >>:user:`Stephen

  same estimator for each parameter value. >>:issue:`7365`<< by
  >>:user:`Aleksandr Sandrovskii <Sundrique>`<<.

* "model_selection.permutation_test_score()" now works with Pandas
  types. >>:issue:`5697`<< by >>:user:`Stijn Tonk <equialgo>`<<.

* Several fixes to input validation in
  "multiclass.OutputCodeClassifier" >>:issue:`8086`<< by Andreas
  Müller.

* "multiclass.OneVsOneClassifier"'s "partial_fit" now ensures all
  classes are provided up-front. >>:issue:`6250`<< by >>:user:`Asish
  Panda <kaichogami>`<<.

* Fix "multioutput.MultiOutputClassifier.predict_proba()" to return a
  list of 2d arrays, rather than a 3d array. In the case where
  different target columns had different numbers of classes, a
  "ValueError" would be raised on trying to stack matrices with
  different dimensions. >>:issue:`8093`<< by >>:user:`Peter Bull
  <pjbull>`<<.

* Cross validation now works with Pandas datatypes that have a read-
  only index. >>:issue:`9507`<< by Loic Esteve.

Metrics

* "metrics.average_precision_score()" no longer linearly interpolates
  between operating points, and instead weighs precisions by the
  change in recall since the last operating point, as per the
  Wikipedia entry. (#7356). By >>:user:`Nick Dingwall <ndingwall>`<<
  and Gael Varoquaux.

* Fix a bug in "metrics.classification._check_targets()" which would

Wikipedia entry. (#7356). By >>:user:`Nick Dingwall <ndingwall>`<<

  return "'binary'" if "y_true" and "y_pred" were both "'binary'" but
  the union of "y_true" and "y_pred" was "'multiclass'".
  >>:issue:`8377`<< by Loic Esteve.

* Fixed an integer overflow bug in "metrics.confusion_matrix()" and
  hence "metrics.cohen_kappa_score()". >>:issue:`8354`<<,
  >>:issue:`7929`<< by Joel Nothman and >>:user:`Jon Crall
  <Erotemic>`<<.

* Fixed passing of "gamma" parameter to the "chi2" kernel in
  "metrics.pairwise.pairwise_kernels()" >>:issue:`5211`<< by
  >>:user:`Nick Rhinehart <nrhine1>`<<, >>:user:`Saurabh Bansod
  <mth4saurabh>`<< and Andreas Müller.

* Fixed a bug when "datasets.make_classification()" fails when
  generating more than 30 features. >>:issue:`8159`<< by

* Fixed a bug where "datasets.make_moons()" gives an incorrect result
  when "n_samples" is odd. >>:issue:`8198`<< by >>:user:`Josh Levy
  <levy5674>`<<.

* Some "fetch_" functions in "datasets" were ignoring the
  "download_if_missing" keyword. >>:issue:`7944`<< by >>:user:`Ralf
  Gommers <rgommers>`<<.

* Fix estimators to accept a "sample_weight" parameter of type
  "pandas.Series" in their "fit" function. >>:issue:`7825`<< by
  Kathleen Chen.

* Fix a bug in cases where "numpy.cumsum" may be numerically unstable,

"pandas.Series" in their "fit" function. >>:issue:`7825`<< by

  raising an exception if instability is identified. >>:issue:`7376`<<
  and >>:issue:`7331`<< by Joel Nothman and >>:user:`yangarbiter`<<.

* Fix a bug where "base.BaseEstimator.__getstate__()" obstructed
  pickling customizations of child-classes, when used in a multiple
  inheritance context. >>:issue:`8316`<< by >>:user:`Holger Peters
  <HolgerPeters>`<<.

* Update Sphinx-Gallery from 0.1.4 to 0.1.7 for resolving links in
  documentation build with Sphinx>1.5 >>:issue:`8010`<<,
  >>:issue:`7986`<< by >>:user:`Oscar Najera <Titan-C>`<<

* Add "data_home" parameter to "sklearn.datasets.fetch_kddcup99()".
  >>:issue:`9289`<< by Loic Esteve.

* Fix dataset loaders using Python 3 version of makedirs to also work
  in Python 2. >>:issue:`9284`<< by >>:user:`Sebastin Santy
  <SebastinSanty>`<<.

* Several minor issues were fixed with thanks to the alerts of
  lgtm.com. >>:issue:`9278`<< by >>:user:`Jean Helie <jhelie>`<<,
  among others.

* Gradient boosting base models are no longer estimators. By Andreas
  Müller.

* All tree based estimators now accept a "min_impurity_decrease"
  parameter in lieu of the "min_impurity_split", which is now
  deprecated. The "min_impurity_decrease" helps stop splitting the
  nodes in which the weighted impurity decrease from splitting is no

parameter in lieu of the "min_impurity_split", which is now
  longer at least "min_impurity_decrease". >>:issue:`8449`<< by Raghav
  RV.

* "n_iter" parameter is deprecated in "linear_model.SGDClassifier",
  "linear_model.Perceptron". By Tom Dupre la Tour.

* "neighbors.LSHForest" has been deprecated and will be removed in
  0.21 due to poor performance. >>:issue:`9078`<< by >>:user:`Laurent
  Direr <ldirer>`<<.

* "neighbors.NearestCentroid" no longer purports to support
  "metric='precomputed'" which now raises an error. >>:issue:`8515`<<
  by >>:user:`Sergul Aydore <sergulaydore>`<<.

* The "alpha" parameter of "semi_supervised.LabelPropagation" now has
  no effect and is deprecated to be removed in 0.21. >>:issue:`9239`<<
  by >>:user:`Andre Ambrosio Boechat <boechat107>`<<, >>:user:`Utkarsh
  Upadhyay <musically-ut>`<<, and Joel Nothman.

* Deprecate the "doc_topic_distr" argument of the "perplexity" method
  in "decomposition.LatentDirichletAllocation" because the user no
  longer has access to the unnormalized document topic distribution
  needed for the perplexity calculation. >>:issue:`7954`<< by
  >>:user:`Gary Foreman <garyForeman>`<<.

longer has access to the unnormalized document topic distribution

* The "n_topics" parameter of
  "decomposition.LatentDirichletAllocation" has been renamed to
  "n_components" and will be removed in version 0.21.
  >>:issue:`8922`<< by >>:user:`Attractadore`<<.

* "decomposition.SparsePCA.transform()"'s "ridge_alpha" parameter is
  deprecated in preference for class parameter. >>:issue:`8137`<< by
  >>:user:`Naoya Kanai <naoyak>`<<.

* "cluster.DBSCAN" now has a "metric_params" parameter.
  >>:issue:`8139`<< by >>:user:`Naoya Kanai <naoyak>`<<.

* "feature_selection.SelectFromModel" now has a "partial_fit" method
  only if the underlying estimator does. By Andreas Müller.

* "feature_selection.SelectFromModel" now validates the "threshold"
  parameter and sets the "threshold_" attribute during the call to
  "fit", and no longer during the call to "transform`". By Andreas
  Müller.

* The "non_negative" parameter in "feature_extraction.FeatureHasher"
  has been deprecated, and replaced with a more principled
  alternative, "alternate_sign". >>:issue:`7565`<< by >>:user:`Roman
  Yurchak <rth>`<<.

* "linear_model.RandomizedLogisticRegression", and
  "linear_model.RandomizedLasso" have been deprecated and will be
  removed in version 0.21. >>:issue:`8995`<< by >>:user:`Ramana.S
  <sentient07>`<<.

"linear_model.RandomizedLasso" have been deprecated and will be

* Deprecate the "fit_params" constructor input to the
  "model_selection.RandomizedSearchCV" in favor of passing keyword
  parameters to the "fit" methods of those classes. Data-dependent
  parameters needed for model training should be passed as keyword
  arguments to "fit", and conforming to this convention will allow the
  hyperparameter selection classes to be used with tools such as
  "model_selection.cross_val_predict()". >>:issue:`2879`<< by
  >>:user:`Stephen Hoover <stephen-hoover>`<<.

* In version 0.21, the default behavior of splitters that use the
  "test_size" and "train_size" parameter will change, such that
  specifying "train_size" alone will cause "test_size" to be the
  remainder. >>:issue:`7459`<< by >>:user:`Nelson Liu <nelson-liu>`<<.

* "multiclass.OneVsRestClassifier" now has "partial_fit",
  "decision_function" and "predict_proba" methods only when the
  underlying estimator does.  >>:issue:`7812`<< by Andreas Müller and
  >>:user:`Mikhail Korobov <kmike>`<<.

* "multiclass.OneVsRestClassifier" now has a "partial_fit" method only
  if the underlying estimator does.  By Andreas Müller.

* The "decision_function" output shape for binary classification in
  "multiclass.OneVsRestClassifier" and "multiclass.OneVsOneClassifier"

if the underlying estimator does.  By Andreas Müller.

  is now "(n_samples,)" to conform to scikit-learn conventions.
  >>:issue:`9100`<< by Andreas Müller.

* The "multioutput.MultiOutputClassifier.predict_proba()" function
  used to return a 3d array ("n_samples", "n_classes", "n_outputs").
  In the case where different target columns had different numbers of
  classes, a "ValueError" would be raised on trying to stack matrices
  with different dimensions. This function now returns a list of
  arrays where the length of the list is "n_outputs", and each array
  is ("n_samples", "n_classes") for that particular output.
  >>:issue:`8093`<< by >>:user:`Peter Bull <pjbull>`<<.

* Replace attribute "named_steps" "dict" to "utils.Bunch" in
  "pipeline.Pipeline" to enable tab completion in interactive
  environment. In the case conflict value on "named_steps" and "dict"
  attribute, "dict" behavior will be prioritized. >>:issue:`8481`<< by

* Deprecate the "y" parameter in "transform" and "inverse_transform".
  The method  should not accept "y" parameter, as it's used at the
  prediction time. >>:issue:`8174`<< by >>:user:`Tahar Zanouda
  <tzano>`<<, Alexandre Gramfort and Raghav RV.

* SciPy >= 0.13.3 and NumPy >= 1.8.2 are now the minimum supported

prediction time. >>:issue:`8174`<< by >>:user:`Tahar Zanouda

  versions for scikit-learn. The following backported functions in
  "utils" have been removed or deprecated accordingly.
  >>:issue:`8854`<< and >>:issue:`8874`<< by >>:user:`Naoya Kanai
  <naoyak>`<<

* The "store_covariances" and "covariances_" parameters of
  "discriminant_analysis.QuadraticDiscriminantAnalysis" has been
  renamed to "store_covariance" and "covariance_" to be consistent
  with the corresponding parameter names of the
  "discriminant_analysis.LinearDiscriminantAnalysis". They will be
  removed in version 0.21. >>:issue:`7998`<< by >>:user:`Jiacheng
  <mrbeann>`<<

  Removed in 0.19:

  * "utils.fixes.argpartition"

  * "utils.fixes.array_equal"

  * "utils.fixes.astype"

  * "utils.fixes.bincount"

  * "utils.fixes.expit"

  * "utils.fixes.frombuffer_empty"

  * "utils.fixes.in1d"

  * "utils.fixes.norm"

  * "utils.fixes.rankdata"

  * "utils.fixes.safe_copy"

  Deprecated in 0.19, to be removed in 0.21:

  * "utils.arpack.eigs"

  * "utils.arpack.eigsh"

  * "utils.arpack.svds"

  * "utils.extmath.fast_dot"

  * "utils.extmath.logsumexp"

  * "utils.extmath.norm"

  * "utils.extmath.pinvh"

  * "utils.graph.graph_laplacian"

  * "utils.random.choice"

  * "utils.sparsetools.connected_components"

  * "utils.stats.rankdata"

* Estimators with both methods "decision_function" and "predict_proba"

* "utils.graph.graph_laplacian"

  are now required to have a monotonic relation between them. The
  method "check_decision_proba_consistency" has been added in
  **utils.estimator_checks** to check their consistency.
  >>:issue:`7578`<< by >>:user:`Shubham Bhardwaj <shubham0704>`<<

* All checks in "utils.estimator_checks", in particular
  "utils.estimator_checks.check_estimator()" now accept estimator
  instances. Most other checks do not accept estimator classes any
  more. >>:issue:`9019`<< by Andreas Müller.

* Ensure that estimators' attributes ending with "_" are not set in
  the constructor but only in the "fit" method. Most notably, ensemble
  estimators (deriving from "ensemble.BaseEnsemble") now only have
  "self.estimators_" available after "fit". >>:issue:`7464`<< by Lars
  Buitinck and Loic Esteve.

improvement of the project since version 0.18, including:

Joel Nothman, Loic Esteve, Andreas Mueller, Guillaume Lemaitre,
Olivier Grisel, Hanmin Qin, Raghav RV, Alexandre Gramfort, themrmax,
Aman Dalmia, Gael Varoquaux, Naoya Kanai, Tom Dupré la Tour,
Rishikesh, Nelson Liu, Taehoon Lee, Nelle Varoquaux, Aashil, Mikhail
Korobov, Sebastin Santy, Joan Massich, Roman Yurchak, RAKOTOARISON

Herilalaina, Thierry Guillemot, Alexandre Abadie, Carol Willing,
Balakumaran Manoharan, Josh Karnofsky, Vlad Niculae, Utkarsh Upadhyay,
Dmitry Petrov, Minghui Liu, Srivatsan, Vincent Pham, Albert Thomas,
Jake VanderPlas, Attractadore, JC Liu, alexandercbooth, chkoar, Óscar
Nájera, Aarshay Jain, Kyle Gilliam, Ramana Subramanyam, CJ Carey,
Clement Joudet, David Robles, He Chen, Joris Van den Bossche, Karan
Desai, Katie Luangkote, Leland McInnes, Maniteja Nandana, Michele
Lacchia, Sergei Lebedev, Shubham Bhardwaj, akshay0724, omtcyfz,
rickiepark, waterponey, Vathsala Achar, jbDelafosse, Ralf Gommers,
Ekaterina Krivich, Vivek Kumar, Ishank Gulati, Dave Elliott, ldirer,
Reiichiro Nakano, Levi John Wolf, Mathieu Blondel, Sid Kapur, Dougal
J. Sutherland, midinas, mikebenfield, Sourav Singh, Aseem Bansal,
Ibraim Ganiev, Stephen Hoover, AishwaryaRK, Steven C. Howell, Gary
Foreman, Neeraj Gangwar, Tahar, Jon Crall, dokato, Kathy Chen, ferria,
Thomas Moreau, Charlie Brummitt, Nicolas Goix, Adam Kleczewski, Sam
Shleifer, Nikita Singh, Basil Beirouti, Giorgio Patrini, Manoj Kumar,
Rafael Possas, James Bourbeau, James A. Bednar, Janine Harper, Jaye,
Jean Helie, Jeremy Steward, Artsiom, John Wei, Jonathan LIgo, Jonathan
Rahn, seanpwilliams, Arthur Mensch, Josh Levy, Julian Kuhlmann, Julien

Aubert, Jörn Hees, Kai, shivamgargsya, Kat Hempstalk, Kaushik
Lakshmikanth, Kennedy, Kenneth Lyons, Kenneth Myers, Kevin Yap, Kirill
Bobyrev, Konstantin Podshumok, Arthur Imbert, Lee Murray,
toastedcornflakes, Lera, Li Li, Arthur Douillard, Mainak Jas,
tobycheese, Manraj Singh, Manvendra Singh, Marc Meketon, MarcoFalke,
Matthew Brett, Matthias Gilch, Mehul Ahuja, Melanie Goetz, Meng, Peng,
Michael Dezube, Michal Baumgartner, vibrantabhi19, Artem Golubin,
Milen Paskov, Antonin Carette, Morikko, MrMjauh, NALEPA Emmanuel,
Namiya, Antoine Wendlinger, Narine Kokhlikyan, NarineK, Nate Guerin,
Angus Williams, Ang Lu, Nicole Vavrova, Nitish Pandey, Okhlopkov
Daniil Olegovich, Andy Craze, Om Prakash, Parminder Singh, Patrick
Carlson, Patrick Pei, Paul Ganssle, Paulo Haddad, Paweł Lorek, Peng
Yu, Pete Bachant, Peter Bull, Peter Csizsek, Peter Wang, Pieter Arthur
de Jong, Ping-Yao, Chang, Preston Parry, Puneet Mathur, Quentin Hibon,
Andrew Smith, Andrew Jackson, 1kastner, Rameshwar Bhaskaran, Rebecca
Bilbro, Remi Rampin, Andrea Esuli, Rob Hall, Robert Bradshaw, Romain
Brault, Aman Pratik, Ruifeng Zheng, Russell Smith, Sachin Agarwal,
Sailesh Choyal, Samson Tan, Samuël Weber, Sarah Brown, Sebastian
Pölsterl, Sebastian Raschka, Sebastian Saeger, Alyssa Batula, Abhyuday
Pratap Singh, Sergey Feldman, Sergul Aydore, Sharan Yalburgi,

willduan, Siddharth Gupta, Sri Krishna, Almer, Stijn Tonk, Allen
Riddell, Theofilos Papapanagiotou, Alison, Alexis Mignon, Tommy
Boucher, Tommy Löfstedt, Toshihiro Kamishima, Tyler Folkman, Tyler
Lanigan, Alexander Junge, Varun Shenoy, Victor Poughon, Vilhelm von
Ehrenheim, Aleksandr Sandrovskii, Alan Yee, Vlasios Vasileiou, Warut
Vijitbenjaronk, Yang Zhang, Yaroslav Halchenko, Yichuan Liu, Yuichi
Fujikawa, affanv14, aivision2020, xor, andreh7, brady salz,
campustrampus, Agamemnon Krasoulis, ditenberg, elena-sharova, filipj8,
fukatani, gedeck, guiniol, guoci, hakaa1, hongkahjun, i-am-xhy,
jakirkham, jaroslaw-weber, jayzed82, jeroko, jmontoyam,
jonathan.striebel, josephsalmon, jschendel, leereeves, martin-hahn,
mathurinm, mehak-sachdeva, mlewis1729, mlliou112, mthorrell,
ndingwall, nuffe, yangarbiter, plagree, pldtc325, Breno Freitas, Brett
Olsen, Brian A. Alfano, Brian Burns, polmauri, Brandon Carter,
Charlton Austin, Chayant T15h, Chinmaya Pancholi, Christian Danielsen,
Chung Yen, Chyi-Kwei Yau, pravarmahajan, DOHMATOB Elvis, Daniel
LeJeune, Daniel Hnyk, Darius Morawiec, David DeTomaso, David Gasquez,
David Haberthür, David Heryanto, David Kirkby, David Nicholson,
rashchedrin, Deborah Gertrude Digges, Denis Engemann, Devansh D,
Dickson, Bob Baxley, Don86, E. Lynch-Klarup, Ed Rogers, Elizabeth

Ferriss, Ellen-Co2, Fabian Egli, Fang-Chieh Chou, Bing Tian Dai, Greg
Stupp, Grzegorz Szpak, Bertrand Thirion, Hadrien Bertrand, Harizo
Rajaona, zxcvbnius, Henry Lin, Holger Peters, Icyblade Dai, Igor
Andriushchenko, Ilya, Isaac Laughlin, Iván Vallés, Aurélien Bellet,
JPFrancoia, Jacob Schreiber, Asish Mahapatra

Version 0.14

**August 7, 2013**

Changelog
=========

* Missing values with sparse and dense matrices can be imputed with
  the transformer "preprocessing.Imputer" by Nicolas Trésegnie.

* The core implementation of decisions trees has been rewritten from
  scratch, allowing for faster tree induction and lower memory
  consumption in all tree-based estimators. By Gilles Louppe.

* Added "ensemble.AdaBoostClassifier" and
  "ensemble.AdaBoostRegressor", by Noel Dawe  and Gilles Louppe. See
  the AdaBoost section of the user guide for details and examples.

* Added "grid_search.RandomizedSearchCV" and
  "grid_search.ParameterSampler" for randomized hyperparameter
  optimization. By Andreas Müller.

* Added biclustering algorithms
  ("sklearn.cluster.bicluster.SpectralCoclustering" and
  "sklearn.cluster.bicluster.SpectralBiclustering"), data generation
  methods ("sklearn.datasets.make_biclusters()" and
  "sklearn.datasets.make_checkerboard()"), and scoring metrics
  ("sklearn.metrics.consensus_score()"). By Kemal Eren.

* Added Restricted Boltzmann Machines ("neural_network.BernoulliRBM").
  By Yann Dauphin.

* Python 3 support by >>:user:`Justin Vincent <justinvf>`<<, Lars
  Buitinck, >>:user:`Subhodeep Moitra <smoitra87>`<< and Olivier
  Grisel. All tests now pass under Python 3.3.

* Ability to pass one penalty (alpha value) per target in
  "linear_model.Ridge", by @eickenberg and Mathieu Blondel.

* Fixed "sklearn.linear_model.stochastic_gradient.py" L2

  regularization issue (minor practical significance). By
  >>:user:`Norbert Crombach <norbert>`<< and Mathieu Blondel .

* Added an interactive version of Andreas Müller's Machine Learning
  Cheat Sheet (for scikit-learn) to the documentation. See Choosing
  the right estimator. By Jaques Grobler.

* "grid_search.GridSearchCV" and "cross_validation.cross_val_score()"
  now support the use of advanced scoring function such as area under
  the ROC curve and f-beta scores. See The scoring parameter: defining
  model evaluation rules for details. By Andreas Müller and Lars
  Buitinck. Passing a function from "sklearn.metrics" as "score_func"
  is deprecated.

* Multi-label classification output is now supported by
  "metrics.accuracy_score()", "metrics.zero_one_loss()",
  "metrics.f1_score()", "metrics.fbeta_score()",
  "metrics.classification_report()", "metrics.precision_score()" and
  "metrics.recall_score()" by Arnaud Joly.

* Two new metrics "metrics.hamming_loss()" and
  "metrics.jaccard_similarity_score()" are added with multi-label
  support by Arnaud Joly.

* Speed and memory usage improvements in
  "feature_extraction.text.TfidfVectorizer", by Jochen Wersdörfer and
  Roman Sinayev.

* The "min_df" parameter in "feature_extraction.text.CountVectorizer"

"feature_extraction.text.TfidfVectorizer", by Jochen Wersdörfer and

  and "feature_extraction.text.TfidfVectorizer", which used to be 2,
  has been reset to 1 to avoid unpleasant surprises (empty
  vocabularies) for novice users who try it out on tiny document
  collections. A value of at least 2 is still recommended for
  practical use.

* "svm.LinearSVC", "linear_model.SGDClassifier" and
  "linear_model.SGDRegressor" now have a "sparsify" method that
  converts their "coef_" into a sparse matrix, meaning stored models
  trained using these estimators can be made much more compact.

* "linear_model.SGDClassifier" now produces multiclass probability
  estimates when trained under log loss or modified Huber loss.

* Hyperlinks to documentation in example code on the website by
  >>:user:`Martin Luessi <mluessi>`<<.

* Fixed bug in "preprocessing.MinMaxScaler" causing incorrect scaling
  of the features for non-default "feature_range" settings. By Andreas
  Müller.

* "max_features" in "tree.DecisionTreeClassifier",
  "tree.DecisionTreeRegressor" and all derived ensemble estimators now
  supports percentage values. By Gilles Louppe.

* Performance improvements in "isotonic.IsotonicRegression" by Nelle

* "metrics.accuracy_score()" has an option normalize to return the
  fraction or the number of correctly classified sample by Arnaud
  Joly.

* Added "metrics.log_loss()" that computes log loss, aka cross-entropy

fraction or the number of correctly classified sample by Arnaud
  Joly.

  loss. By Jochen Wersdörfer and Lars Buitinck.

* A bug that caused "ensemble.AdaBoostClassifier"'s to output
  incorrect probabilities has been fixed.

* Feature selectors now share a mixin providing consistent
  "transform", "inverse_transform" and "get_support" methods. By Joel
  Nothman.

* A fitted "grid_search.GridSearchCV" or
  "grid_search.RandomizedSearchCV" can now generally be pickled. By

* Refactored and vectorized implementation of "metrics.roc_curve()"
  and "metrics.precision_recall_curve()". By Joel Nothman.

* The new estimator "sklearn.decomposition.TruncatedSVD" performs
  dimensionality reduction using SVD on sparse matrices, and can be
  used for latent semantic analysis (LSA). By Lars Buitinck.

* Added self-contained example of out-of-core learning on text data *
  sphx_glr_auto_examples_applications_plot_out_of_core_classification
  .py*. By >>:user:`Eustache Diemert <oddskool>`<<.

* The default number of components for
  "sklearn.decomposition.RandomizedPCA" is now correctly documented to
  be "n_features". This was the default behavior, so programs using it
  will continue to work as they did.

* "sklearn.cluster.KMeans" now fits several orders of magnitude faster
  on sparse data (the speedup depends on the sparsity). By Lars
  Buitinck.

* Reduce memory footprint of FastICA by Denis Engemann and Alexandre
  Gramfort.

on sparse data (the speedup depends on the sparsity). By Lars
  Buitinck.

  Gramfort.

* Verbose output in "sklearn.ensemble.gradient_boosting" now uses a
  column format and prints progress in decreasing frequency. It also
  shows the remaining time. By Peter Prettenhofer.

* "sklearn.ensemble.gradient_boosting" provides out-of-bag improvement
  "oob_improvement_" rather than the OOB score for model selection. An
  example that shows how to use OOB estimates to select the number of
  trees was added. By Peter Prettenhofer.

* Most metrics now support string labels for multiclass classification
  by Arnaud Joly and Lars Buitinck.

* New OrthogonalMatchingPursuitCV class by Alexandre Gramfort and Vlad
  Niculae.

* Fixed a bug in "sklearn.covariance.GraphLassoCV": the 'alphas'
  parameter now works as expected when given a list of values. By
  Philippe Gervais.

* Fixed an important bug in "sklearn.covariance.GraphLassoCV" that
  prevented all folds provided by a CV object to be used (only the
  first 3 were used). When providing a CV object, execution time may
  thus increase significantly compared to the previous version (bug
  results are correct now). By Philippe Gervais.

* "cross_validation.cross_val_score" and the "grid_search" module is
  now tested with multi-output data by Arnaud Joly.

* "datasets.make_multilabel_classification()" can now return the
  output in label indicator multilabel format  by Arnaud Joly.

now tested with multi-output data by Arnaud Joly.

* K-nearest neighbors, "neighbors.KNeighborsRegressor" and
  "neighbors.RadiusNeighborsRegressor", and radius neighbors,
  "neighbors.RadiusNeighborsRegressor" and
  "neighbors.RadiusNeighborsClassifier" support multioutput data by
  Arnaud Joly.

* Random state in LibSVM-based estimators ("svm.SVC", "NuSVC",
  "OneClassSVM", "svm.SVR", "svm.NuSVR") can now be controlled.  This
  is useful to ensure consistency in the probability estimates for the
  classifiers trained with "probability=True". By Vlad Niculae.

* Out-of-core learning support for discrete naive Bayes classifiers
  "sklearn.naive_bayes.MultinomialNB" and
  "sklearn.naive_bayes.BernoulliNB" by adding the "partial_fit" method
  by Olivier Grisel.

* New website design and navigation by Gilles Louppe, Nelle Varoquaux,
  Vincent Michel and Andreas Müller.

* Improved documentation on multi-class, multi-label and multi-output
  classification by Yannick Schwartz and Arnaud Joly.

* Better input and error handling in the "metrics" module by Arnaud
  Joly and Joel Nothman.

* Speed optimization of the "hmm" module by >>:user:`Mikhail Korobov
  <kmike>`<<

* Significant speed improvements for "sklearn.cluster.DBSCAN" by
  cleverless

* The "auc_score()" was renamed "roc_auc_score()".

<kmike>`<<

* Testing scikit-learn with "sklearn.test()" is deprecated. Use
  "nosetests sklearn" from the command line.

* Feature importances in "tree.DecisionTreeClassifier",
  "tree.DecisionTreeRegressor" and all derived ensemble estimators are
  now computed on the fly when accessing  the "feature_importances_"
  attribute. Setting "compute_importances=True" is no longer required.
  By Gilles Louppe.

* "linear_model.lasso_path" and "linear_model.enet_path" can return
  its results in the same format as that of "linear_model.lars_path".
  This is done by setting the "return_models" parameter to "False". By
  Jaques Grobler and Alexandre Gramfort

* "grid_search.IterGrid" was renamed to "grid_search.ParameterGrid".

* Fixed bug in "KFold" causing imperfect class balance in some cases.
  By Alexandre Gramfort and Tadej Janež.

* "sklearn.neighbors.BallTree" has been refactored, and a
  "sklearn.neighbors.KDTree" has been added which shares the same
  interface.  The Ball Tree now works with a wide variety of distance
  metrics.  Both classes have many new methods, including single-tree
  and dual-tree queries, breadth-first and depth-first searching, and
  more advanced queries such as kernel density estimation and 2-point
  correlation functions. By Jake Vanderplas

* Support for scipy.spatial.cKDTree within neighbors queries has been

more advanced queries such as kernel density estimation and 2-point

  removed, and the functionality replaced with the new "KDTree" class.

* "sklearn.neighbors.KernelDensity" has been added, which performs
  efficient kernel density estimation with a variety of kernels.

* "sklearn.decomposition.KernelPCA" now always returns output with
  "n_components" components, unless the new parameter
  "remove_zero_eig" is set to "True". This new behavior is consistent
  with the way kernel PCA was always documented; previously, the
  removal of components with zero eigenvalues was tacitly performed on
  all data.

* "gcv_mode="auto"" no longer tries to perform SVD on a densified
  sparse matrix in "sklearn.linear_model.RidgeCV".

* Sparse matrix support in "sklearn.decomposition.RandomizedPCA" is
  now deprecated in favor of the new "TruncatedSVD".

* "cross_validation.KFold" and "cross_validation.StratifiedKFold" now
  enforce *n_folds >= 2* otherwise a "ValueError" is raised. By

* "datasets.load_files()"'s "charset" and "charset_errors" parameters
  were renamed "encoding" and "decode_errors".

* Attribute "oob_score_" in
  "sklearn.ensemble.GradientBoostingRegressor" and
  "sklearn.ensemble.GradientBoostingClassifier" is deprecated and has
  been replaced by "oob_improvement_" .

* Attributes in OrthogonalMatchingPursuit have been deprecated

"sklearn.ensemble.GradientBoostingClassifier" is deprecated and has

  (copy_X, Gram, ...) and precompute_gram renamed precompute for
  consistency. See #2224.

* "sklearn.preprocessing.StandardScaler" now converts integer input to
  float, and raises a warning. Previously it rounded for dense integer
  input.

* "sklearn.multiclass.OneVsRestClassifier" now has a
  "decision_function" method. This will return the distance of each
  sample from the decision boundary for each class, as long as the
  underlying estimators implement the "decision_function" method. By
  Kyle Kastner.

* Better input validation, warning on unexpected shapes for y.

People
======

List of contributors for release 0.14 by number of commits.

   * 277  Gilles Louppe

   * 245  Lars Buitinck

   * 187  Andreas Mueller

   * 124  Arnaud Joly

   * 112  Jaques Grobler

   * 109  Gael Varoquaux

   * 107  Olivier Grisel

   * 102  Noel Dawe

   * 99  Kemal Eren

   * 79  Joel Nothman

   * 75  Jake VanderPlas

   * 73  Nelle Varoquaux

   * 71  Vlad Niculae

   * 65  Peter Prettenhofer

   * 64  Alexandre Gramfort

   * 54  Mathieu Blondel

   * 38  Nicolas Trésegnie

   * 35  eustache

   * 27  Denis Engemann

   * 25  Yann N. Dauphin

   * 19  Justin Vincent

   * 17  Robert Layton

   * 15  Doug Coleman

   * 14  Michael Eickenberg

   * 13  Robert Marchman

   * 11  Fabian Pedregosa

   * 11  Philippe Gervais

   * 10  Jim Holmström

   * 10  Tadej Janež

   * 10  syhw

* 17  Robert Layton

   * 10  syhw

   * 9  Mikhail Korobov

   * 9  Steven De Gryze

   * 8  sergeyf

   * 7  Ben Root

   * 7  Hrishikesh Huilgolkar

   * 6  Kyle Kastner

   * 6  Martin Luessi

   * 6  Rob Speer

   * 5  Federico Vaggi

   * 5  Raul Garreta

   * 5  Rob Zinkov

   * 4  Ken Geis

   * 3  A. Flaxman

   * 3  Denton Cockburn

   * 3  Dougal Sutherland

   * 3  Ian Ozsvald

   * 3  Johannes Schönberger

   * 3  Robert McGibbon

   * 3  Roman Sinayev

   * 3  Szabo Roland

   * 2  Diego Molla

   * 2  Imran Haque

   * 2  Jochen Wersdörfer

   * 2  Sergey Karayev

   * 2  Yannick Schwartz

   * 2  jamestwebber

   * 1  Abhijeet Kolhe

   * 1  Alexander Fabisch

   * 1  Bastiaan van den Berg

   * 1  Benjamin Peterson

   * 1  Daniel Velkov

   * 1  Fazlul Shahriar

   * 1  Felix Brockherde

   * 1  Félix-Antoine Fortin

   * 1  Harikrishnan S

   * 1  Jack Hale

   * 1  JakeMick

   * 1  James McDermott

   * 1  John Benediktsson

   * 1  John Zwinck

   * 1  Joshua Vredevoogd

   * 1  Justin Pati

   * 1  Kevin Hughes

   * 1  Kyle Kelley

   * 1  Matthias Ekman

   * 1  Miroslav Shubernetskiy

   * 1  Naoki Orii

   * 1  Norbert Crombach

   * 1  Rafael Cunha de Almeida

   * 1  Rolando Espinoza La fuente

   * 1  Seamus Abshere

   * 1  Sergey Feldman

   * 1  Sergio Medina

   * 1  Stefano Lattarini

   * 1  Steve Koch

   * 1  Sturla Molden

   * 1  Thomas Jarosch

   * 1  Yaroslav Halchenko

Version 0.15.2

**September 4, 2014**

Bug fixes
=========

* Fixed handling of the "p" parameter of the Minkowski distance that
  was previously ignored in nearest neighbors models. By
  >>:user:`Nikolay Mayorov <nmayorov>`<<.

* Fixed duplicated alphas in "linear_model.LassoLars" with early
  stopping on 32 bit Python. By Olivier Grisel and Fabian Pedregosa.

* Fixed the build under Windows when scikit-learn is built with MSVC
  while NumPy is built with MinGW. By Olivier Grisel and
  >>:user:`Federico Vaggi <FedericoV>`<<.

* Fixed an array index overflow bug in the coordinate descent solver.
  By Gael Varoquaux.

* Better handling of numpy 1.9 deprecation warnings. By Gael

* Removed unnecessary data copy in "cluster.KMeans". By Gael

* Explicitly close open files to avoid "ResourceWarnings" under Python
  3. By Calvin Giles.

* The "transform" of
  "discriminant_analysis.LinearDiscriminantAnalysis" now projects the
  input on the most discriminant directions. By Martin Billinger.

* Fixed potential overflow in "_tree.safe_realloc" by Lars Buitinck.

* Performance optimization in "isotonic.IsotonicRegression". By Robert
  Bradshaw.

* "nose" is non-longer a runtime dependency to import "sklearn", only
  for running the tests. By Joel Nothman.

* Many documentation and website fixes by Joel Nothman, Lars Buitinck
  >>:user:`Matt Pico <MattpSoftware>`<<, and others.

Version 0.15.1

**August 1, 2014**

Bug fixes
=========

* Made "cross_validation.cross_val_score()" use

>>:user:`Matt Pico <MattpSoftware>`<<, and others.

Bug fixes
=========

  "cross_validation.KFold" instead of
  "cross_validation.StratifiedKFold" on multi-output classification
  problems. By >>:user:`Nikolay Mayorov <nmayorov>`<<.

* Support unseen labels "preprocessing.LabelBinarizer" to restore the
  default behavior of 0.14.1 for backward compatibility. By
  >>:user:`Hamzeh Alsalhi <hamsal>`<<.

* Fixed the "cluster.KMeans" stopping criterion that prevented early
  convergence detection. By Edward Raff and Gael Varoquaux.

* Fixed the behavior of "multiclass.OneVsOneClassifier". in case of
  ties at the per-class vote level by computing the correct per-class
  sum of prediction scores. By Andreas Müller.

* Made "cross_validation.cross_val_score()" and
  "grid_search.GridSearchCV" accept Python lists as input data. This
  is especially useful for cross-validation and model selection of
  text processing pipelines. By Andreas Müller.

* Fixed data input checks of most estimators to accept input data that
  implements the NumPy "__array__" protocol. This is the case for for
  "pandas.Series" and "pandas.DataFrame" in recent versions of pandas.

* Fixed a regression for "linear_model.SGDClassifier" with
  "class_weight="auto"" on data with non-contiguous labels. By Olivier
  Grisel.

Version 0.15

**July 15, 2014**

  Grisel.

* Many speed and memory improvements all across the code

* Huge speed and memory improvements to random forests (and extra
  trees) that also benefit better from parallel computing.

* Incremental fit to "BernoulliRBM"

* Added "cluster.AgglomerativeClustering" for hierarchical
  agglomerative clustering with average linkage, complete linkage and
  ward strategies.

* Added "linear_model.RANSACRegressor" for robust regression models.

* Added dimensionality reduction with "manifold.TSNE" which can be
  used to visualize high-dimensional data.

Changelog
=========

* Added "ensemble.BaggingClassifier" and "ensemble.BaggingRegressor"
  meta-estimators for ensembling any kind of base estimator. See the
  Bagging section of the user guide for details and examples. By
  Gilles Louppe.

* New unsupervised feature selection algorithm
  "feature_selection.VarianceThreshold", by Lars Buitinck.

* Added "linear_model.RANSACRegressor" meta-estimator for the robust
  fitting of regression models. By >>:user:`Johannes Schönberger
  <ahojnnes>`<<.

  ward strategies, by  Nelle Varoquaux and Gael Varoquaux.

* Shorthand constructors "pipeline.make_pipeline()" and

agglomerative clustering with average linkage, complete linkage and

  "pipeline.make_union()" were added by Lars Buitinck.

* Shuffle option for "cross_validation.StratifiedKFold". By
  >>:user:`Jeffrey Blackburne <jblackburne>`<<.

* Incremental learning ("partial_fit") for Gaussian Naive Bayes by
  Imran Haque.

* Added "partial_fit" to "BernoulliRBM" By >>:user:`Danny Sullivan
  <dsullivan7>`<<.

* Added "learning_curve" utility to chart performance with respect to
  training size. See
  *sphx_glr_auto_examples_model_selection_plot_learning_curve.py*. By
  Alexander Fabisch.

* Add positive option in "LassoCV" and "ElasticNetCV". By Brian
  Wignall and Alexandre Gramfort.

* Added "linear_model.MultiTaskElasticNetCV" and
  "linear_model.MultiTaskLassoCV". By Manoj Kumar.

* Added "manifold.TSNE". By Alexander Fabisch.

* Add sparse input support to "ensemble.AdaBoostClassifier" and
  "ensemble.AdaBoostRegressor" meta-estimators. By >>:user:`Hamzeh
  Alsalhi <hamsal>`<<.

* Memory improvements of decision trees, by Arnaud Joly.

* Decision trees can now be built in best-first manner by using
  "max_leaf_nodes" as the stopping criteria. Refactored the tree code
  to use either a stack or a priority queue for tree building. By
  Peter Prettenhofer and Gilles Louppe.

* Decision trees can now be fitted on fortran- and c-style arrays, and

to use either a stack or a priority queue for tree building. By

  non-continuous arrays without the need to make a copy. If the input
  array has a different dtype than "np.float32", a fortran- style copy
  will be made since fortran-style memory layout has speed advantages.
  By Peter Prettenhofer and Gilles Louppe.

* Speed improvement of regression trees by optimizing the the
  computation of the mean square error criterion. This lead to speed
  improvement of the tree, forest and gradient boosting tree modules.
  By Arnaud Joly

* The "img_to_graph" and "grid_tograph" functions in
  "sklearn.feature_extraction.image" now return "np.ndarray" instead
  of "np.matrix" when "return_as=np.ndarray".  See the Notes section
  for more information on compatibility.

* Changed the internal storage of decision trees to use a struct
  array. This fixed some small bugs, while improving code and
  providing a small speed gain. By Joel Nothman.

* Reduce memory usage and overhead when fitting and predicting with
  forests of randomized trees in parallel with "n_jobs != 1" by
  leveraging new threading backend of joblib 0.8 and releasing the GIL
  in the tree fitting Cython code.  By Olivier Grisel and Gilles
  Louppe.

* Speed improvement of the "sklearn.ensemble.gradient_boosting"
  module. By Gilles Louppe and Peter Prettenhofer.

* Various enhancements to the  "sklearn.ensemble.gradient_boosting"

Louppe.

  module: a "warm_start" argument to fit additional trees, a
  "max_leaf_nodes" argument to fit GBM style trees, a "monitor" fit
  argument to inspect the estimator during training, and refactoring
  of the verbose code. By Peter Prettenhofer.

* Faster "sklearn.ensemble.ExtraTrees" by caching feature values. By

* Faster depth-based tree building algorithm such as decision tree,
  random forest, extra trees or gradient tree boosting (with depth
  based growing strategy) by avoiding trying to split on found
  constant features in the sample subset. By Arnaud Joly.

* Add "min_weight_fraction_leaf" pre-pruning parameter to tree-based
  methods: the minimum weighted fraction of the input samples required
  to be at a leaf node. By Noel Dawe.

* Added "metrics.pairwise_distances_argmin_min()", by Philippe
  Gervais.

* Added predict method to "cluster.AffinityPropagation" and
  "cluster.MeanShift", by Mathieu Blondel.

* Vector and matrix multiplications have been optimised throughout the
  library by Denis Engemann, and Alexandre Gramfort. In particular,
  they should take less memory with older NumPy versions (prior to
  1.7.2).

* Precision-recall and ROC examples now use train_test_split, and have
  more explanation of why these metrics are useful. By Kyle Kastner

1.7.2).

* The training algorithm for "decomposition.NMF" is faster for sparse
  matrices and has much lower memory complexity, meaning it will scale
  up gracefully to large datasets. By Lars Buitinck.

* Added svd_method option with default value to "randomized" to
  "decomposition.FactorAnalysis" to save memory and significantly
  speedup computation by Denis Engemann, and Alexandre Gramfort.

* Changed "cross_validation.StratifiedKFold" to try and preserve as
  much of the original ordering of samples as possible so as not to
  hide overfitting on datasets with a non-negligible level of samples
  dependency. By Daniel Nouri and Olivier Grisel.

* Add multi-output support to "gaussian_process.GaussianProcess" by
  John Novak.

* Support for precomputed distance matrices in nearest neighbor
  estimators by Robert Layton and Joel Nothman.

* Norm computations optimized for NumPy 1.6 and later versions by Lars
  Buitinck. In particular, the k-means algorithm no longer needs a
  temporary data structure the size of its input.

* "dummy.DummyClassifier" can now be used to predict a constant output
  value. By Manoj Kumar.

* "dummy.DummyRegressor" has now a strategy parameter which allows to
  predict the mean, the median of the training set or a constant
  output value. By >>:user:`Maheshakya Wijewardena <maheshakya>`<<.

* Multi-label classification output in multilabel indicator format is
  now supported by "metrics.roc_auc_score()" and
  "metrics.average_precision_score()" by Arnaud Joly.

* Significant performance improvements (more than 100x speedup for
  large problems) in "isotonic.IsotonicRegression" by Andrew Tulloch.

* Speed and memory usage improvements to the SGD algorithm for linear
  models: it now uses threads, not separate processes, when
  "n_jobs>1". By Lars Buitinck.

* Grid search and cross validation allow NaNs in the input arrays so
  that preprocessors such as "preprocessing.Imputer" can be trained
  within the cross validation loop, avoiding potentially skewed
  results.

* Ridge regression can now deal with sample weights in feature space
  (only sample space until then). By >>:user:`Michael Eickenberg
  <eickenberg>`<<. Both solutions are provided by the Cholesky solver.

* Several classification and regression metrics now support weighted
  samples with the new "sample_weight" argument:
  "metrics.precision_score()", "metrics.average_precision_score()",
  "metrics.recall_score()", "metrics.roc_auc_score()",
  "metrics.explained_variance_score()",

"metrics.f1_score()", "metrics.fbeta_score()",
  "metrics.mean_squared_error()", "metrics.mean_absolute_error()",
  "metrics.r2_score()". By Noel Dawe.

* Speed up of the sample generator
  "datasets.make_multilabel_classification()". By Joel Nothman.

Documentation improvements

* The Working With Text Data tutorial has now been worked in to the
  main documentation's tutorial section. Includes exercises and
  skeletons for tutorial presentation. Original tutorial created by
  several authors including Olivier Grisel, Lars Buitinck and many
  others. Tutorial integration into the scikit-learn documentation by
  Jaques Grobler

* Added Computational Performance documentation. Discussion and
  examples of prediction latency / throughput and different factors
  that have influence over speed. Additional tips for building faster
  models and choosing a relevant compromise between speed and
  predictive power. By >>:user:`Eustache Diemert <oddskool>`<<.

Bug fixes
---------

* Fixed bug in "decomposition.MiniBatchDictionaryLearning" :
  "partial_fit" was not working properly.

* Fixed bug in "linear_model.stochastic_gradient" : "l1_ratio" was
  used as "(1.0 - l1_ratio)" .

* Fixed bug in "multiclass.OneVsOneClassifier" with string labels

* Fixed a bug in "LassoCV" and "ElasticNetCV": they would not pre-
  compute the Gram matrix with "precompute=True" or

  "precompute="auto"" and "n_samples > n_features". By Manoj Kumar.

* Fixed incorrect estimation of the degrees of freedom in
  "feature_selection.f_regression()" when variates are not centered.
  By >>:user:`Virgile Fritsch <VirgileFritsch>`<<.

* Fixed a race condition in parallel processing with "pre_dispatch !=
  "all"" (for instance, in "cross_val_score"). By Olivier Grisel.

* Raise error in "cluster.FeatureAgglomeration" and
  "cluster.WardAgglomeration" when no samples are given, rather than
  returning meaningless clustering.

* Fixed bug in "gradient_boosting.GradientBoostingRegressor" with
  "loss='huber'": "gamma" might have not been initialized.

* Fixed feature importances as computed with a forest of randomized
  trees when fit with "sample_weight != None" and/or with
  "bootstrap=True". By Gilles Louppe.

* "sklearn.hmm" is deprecated. Its removal is planned for the 0.17
  release.

* Use of "covariance.EllipticEnvelop" has now been removed after
  deprecation. Please use "covariance.EllipticEnvelope" instead.

* "cluster.Ward" is deprecated. Use "cluster.AgglomerativeClustering"
  instead.

* "cluster.WardClustering" is deprecated. Use

* "cluster.AgglomerativeClustering" instead.

* "cross_validation.Bootstrap" is deprecated. "cross_validation.KFold"

instead.

  or "cross_validation.ShuffleSplit" are recommended instead.

* Direct support for the sequence of sequences (or list of lists)
  multilabel format is deprecated. To convert to and from the
  supported binary indicator matrix format, use "MultiLabelBinarizer".
  By Joel Nothman.

* Add score method to "PCA" following the model of probabilistic PCA
  and deprecate "ProbabilisticPCA" model whose score implementation is
  not correct. The computation now also exploits the matrix inversion
  lemma for faster computation. By Alexandre Gramfort.

* The score method of "FactorAnalysis" now returns the average log-
  likelihood of the samples. Use score_samples to get log-likelihood
  of each sample. By Alexandre Gramfort.

* Generating boolean masks (the setting "indices=False") from cross-
  validation generators is deprecated. Support for masks will be
  removed in 0.17. The generators have produced arrays of indices by
  default since 0.10. By Joel Nothman.

* 1-d arrays containing strings with "dtype=object" (as used in
  Pandas) are now considered valid classification targets. This fixes
  a regression from version 0.13 in some classifiers. By Joel Nothman.

* Fix wrong "explained_variance_ratio_" attribute in "RandomizedPCA".
  By Alexandre Gramfort.

* Fit alphas for each "l1_ratio" instead of "mean_l1_ratio" in

  "linear_model.ElasticNetCV" and "linear_model.LassoCV". This changes
  the shape of "alphas_" from "(n_alphas,)" to "(n_l1_ratio,
  n_alphas)" if the "l1_ratio" provided is a 1-D array like object of
  length greater than one. By Manoj Kumar.

* Fix "linear_model.ElasticNetCV" and "linear_model.LassoCV" when
  fitting intercept and input data is sparse. The automatic grid of
  alphas was not computed correctly and the scaling with normalize was
  wrong. By Manoj Kumar.

* Fix wrong maximal number of features drawn ("max_features") at each
  split for decision trees, random forests and gradient tree boosting.
  Previously, the count for the number of drawn features started only
  after one non constant features in the split. This bug fix will
  affect computational and generalization performance of those
  algorithms in the presence of constant features. To get back
  previous generalization performance, you should modify the value of
  "max_features". By Arnaud Joly.

  split for "ensemble.ExtraTreesClassifier" and
  "ensemble.ExtraTreesRegressor". Previously, only non constant
  features in the split was counted as drawn. Now constant features
  are counted as drawn. Furthermore at least one feature must be non

"ensemble.ExtraTreesRegressor". Previously, only non constant
  constant in order to make a valid split. This bug fix will affect
  computational and generalization performance of extra trees in the
  presence of constant features. To get back previous generalization
  performance, you should modify the value of "max_features". By

* Fix "utils.compute_class_weight()" when "class_weight=="auto"".
  Previously it was broken for input of non-integer "dtype" and the
  weighted array that was returned was wrong. By Manoj Kumar.

* Fix "cross_validation.Bootstrap" to return "ValueError" when
  "n_train + n_test > n". By >>:user:`Ronald Phlypo <rphlypo>`<<.

People
======

List of contributors for release 0.15 by number of commits.

* 312   Olivier Grisel

* 275   Lars Buitinck

* 221   Gael Varoquaux

* 148   Arnaud Joly

* 134   Johannes Schönberger

* 119   Gilles Louppe

* 113   Joel Nothman

* 111   Alexandre Gramfort

* 95   Jaques Grobler

* 89   Denis Engemann

* 83   Peter Prettenhofer

* 83   Alexander Fabisch

* 62   Mathieu Blondel

* 60   Eustache Diemert

* 60   Nelle Varoquaux

* 49   Michael Bommarito

* 45   Manoj-Kumar-S

* 28   Kyle Kastner

* 26   Andreas Mueller

* 22   Noel Dawe

* 21   Maheshakya Wijewardena

* 21   Brooke Osborn

* 21   Hamzeh Alsalhi

* 21   Jake VanderPlas

* 21   Philippe Gervais

* 19   Bala Subrahmanyam Varanasi

* 12   Ronald Phlypo

* 10   Mikhail Korobov

* 8   Thomas Unterthiner

* 8   Jeffrey Blackburne

* 8   eltermann

* 8   bwignall

* 7   Ankit Agrawal

* 7   CJ Carey

* 6   Daniel Nouri

* 6   Chen Liu

* 6   Michael Eickenberg

* 6   ugurthemaster

* 5   Aaron Schumacher

* 5   Baptiste Lagarde

* 5   Rajat Khanduja

* 5   Robert McGibbon

* 5   Sergio Pascual

* 4   Alexis Metaireau

* 4   Ignacio Rossi

* 4   Virgile Fritsch

* 4   Sebastian Säger

* 4   Ilambharathi Kanniah

* 4   sdenton4

* 4   Robert Layton

* 4   Alyssa

* 4   Amos Waterland

* 3   Andrew Tulloch

* 3   murad

* 3   Steven Maude

* 3   Karol Pysniak

* 3   Jacques Kvam

* 3   cgohlke

* 3   cjlin

* 3   Michael Becker

* 3   hamzeh

* 3   Eric Jacobsen

* 3   john collins

* 3   kaushik94

* 3   Erwin Marsi

* 2   csytracy

* 2   LK

* 2   Vlad Niculae

* 2   Laurent Direr

* 2   Erik Shilts

* 2   Raul Garreta

* 2   Yoshiki Vázquez Baeza

* 2   Yung Siang Liau

* 2   abhishek thakur

* 2   James Yu

* 2   Rohit Sivaprasad

* 2   Roland Szabo

* 2   amormachine

* 2   Alexis Mignon

* 2   Oscar Carlsson

* 2   Nantas Nardelli

* 2   jess010

* 2   kowalski87

* 2   Andrew Clegg

* 2   Federico Vaggi

* 2   Simon Frid

* 2   Félix-Antoine Fortin

* 1   Ralf Gommers

* 1   t-aft

* 1   Ronan Amicel

* 1   Rupesh Kumar Srivastava

* 1   Ryan Wang

* 1   Samuel Charron

* 1   Samuel St-Jean

* 1   Fabian Pedregosa

* 1   Skipper Seabold

* 1   Stefan Walk

* 1   Stefan van der Walt

* 1   t-aft

* 1   Stephan Hoyer

* 1   Allen Riddell

* 1   Valentin Haenel

* 1   Vijay Ramesh

* 1   Will Myers

* 1   Yaroslav Halchenko

* 1   Yoni Ben-Meshulam

* 1   Yury V. Zaytsev

* 1   adrinjalali

* 1   ai8rahim

* 1   alemagnani

* 1   alex

* 1   benjamin wilson

* 1   chalmerlowe

* 1   dzikie drożdże

* 1   jamestwebber

* 1   matrixorz

* 1   popo

* 1   samuela

* 1   François Boulogne

* 1   Alexander Measure

* 1   Ethan White

* 1   Guilherme Trein

* 1   Hendrik Heuer

* 1   IvicaJovic

* 1   Jan Hendrik Metzen

* 1   Jean Michel Rouly

* 1   Eduardo Ariño de la Rubia

* 1   Jelle Zijlstra

* 1   Eddy L O Jansson

* 1   Denis

* 1   John

* 1   John Schmidt

* 1   Jorge Cañardo Alastuey

* 1   Joseph Perla

* 1   Joshua Vredevoogd

* 1   José Ricardo

* 1   Julien Miotte

* 1   Kemal Eren

* 1   Kenta Sato

* 1   David Cournapeau

* 1   Kyle Kelley

* 1   Daniele Medri

* 1   Laurent Luce

* 1   Laurent Pierron

* 1   Luis Pedro Coelho

* 1   DanielWeitzenfeld

* 1   Craig Thompson

* 1   Chyi-Kwei Yau

* 1   Matthew Brett

* 1   Matthias Feurer

* 1   Max Linke

* 1   Chris Filo Gorgolewski

* 1   Charles Earl

* 1   Michael Hanke

* 1   Michele Orrù

* 1   Bryan Lunt

* 1   Brian Kearns

* 1   Paul Butler

* 1   Paweł Mandera

* 1   Peter

* 1   Andrew Ash

* 1   Pietro Zambelli

* 1   staubda

Version 0.17.1

**February 18, 2016**

Changelog
=========

Bug fixes
---------

* Upgrade vendored joblib to version 0.9.4 that fixes an important bug
  in "joblib.Parallel" that can silently yield to wrong results when
  working on datasets larger than 1MB:
  https://github.com/joblib/joblib/blob/0.9.4/CHANGES.rst

* Fixed reading of Bunch pickles generated with scikit-learn version
  <= 0.16. This can affect users who have already downloaded a dataset
  with scikit-learn 0.16 and are loading it with scikit-learn 0.17.
  See >>:issue:`6196`<< for how this affected
  "datasets.fetch_20newsgroups()". By Loic Esteve.

* Fixed a bug that prevented using ROC AUC score to perform grid
  search on several CPU / cores on large arrays. See >>:issue:`6147`<<
  By Olivier Grisel.

* Fixed a bug that prevented to properly set the "presort" parameter
  in "ensemble.GradientBoostingRegressor". See >>:issue:`5857`<< By
  Andrew McCulloh.

* Fixed a joblib error when evaluating the perplexity of a
  "decomposition.LatentDirichletAllocation" model. See
  >>:issue:`6258`<< By Chyi-Kwei Yau.

Version 0.17

**November 5, 2015**

Changelog
=========

* All the Scaler classes but "preprocessing.RobustScaler" can be
  fitted online by calling *partial_fit*. By >>:user:`Giorgio Patrini

* The new class "ensemble.VotingClassifier" implements a "majority
  rule" / "soft voting" ensemble classifier to combine estimators for
  classification. By Sebastian Raschka.

<giorgiop>`<<.

* The new class "preprocessing.RobustScaler" provides an alternative
  to "preprocessing.StandardScaler" for feature-wise centering and
  range normalization that is robust to outliers. By >>:user:`Thomas
  Unterthiner <untom>`<<.

* The new class "preprocessing.MaxAbsScaler" provides an alternative
  to "preprocessing.MinMaxScaler" for feature-wise range normalization
  when the data is already centered or sparse. By >>:user:`Thomas

* The new class "preprocessing.FunctionTransformer" turns a Python
  function into a "Pipeline"-compatible transformer object. By Joe
  Jevnik.

* The new classes "cross_validation.LabelKFold" and
  "cross_validation.LabelShuffleSplit" generate train-test folds,
  respectively similar to "cross_validation.KFold" and
  "cross_validation.ShuffleSplit", except that the folds are
  conditioned on a label array. By Brian McFee, >>:user:`Jean Kossaifi
  <JeanKossaifi>`<< and Gilles Louppe.

* "decomposition.LatentDirichletAllocation" implements the Latent
  Dirichlet Allocation topic model with online  variational inference.
  By >>:user:`Chyi-Kwei Yau <chyikwei>`<<, with code based on an
  implementation by Matt Hoffman. (>>:issue:`3659`<<)

* The new solver "sag" implements a Stochastic Average Gradient

By >>:user:`Chyi-Kwei Yau <chyikwei>`<<, with code based on an

  descent and is available in both "linear_model.LogisticRegression"
  and "linear_model.Ridge". This solver is very efficient for large
  datasets. By >>:user:`Danny Sullivan <dsullivan7>`<< and Tom Dupre
  la Tour. (>>:issue:`4738`<<)

* The new solver "cd" implements a Coordinate Descent in
  "decomposition.NMF". Previous solver based on Projected Gradient is
  still available setting new parameter "solver" to "pg", but is
  deprecated and will be removed in 0.19, along with
  "decomposition.ProjectedGradientNMF" and parameters "sparseness",
  "eta", "beta" and "nls_max_iter". New parameters "alpha" and
  "l1_ratio" control L1 and L2 regularization, and "shuffle" adds a
  shuffling step in the "cd" solver. By Tom Dupre la Tour and Mathieu
  Blondel.

* "manifold.TSNE" now supports approximate optimization via the
  Barnes-Hut method, leading to much faster fitting. By Christopher
  Erick Moody. (>>:issue:`4025`<<)

* "cluster.mean_shift_.MeanShift" now supports parallel execution, as
  implemented in the "mean_shift" function. By >>:user:`Martino
  Sorbaro <martinosorb>`<<.

* "naive_bayes.GaussianNB" now supports fitting with "sample_weight".

* "dummy.DummyClassifier" now supports a prior fitting strategy. By

* Added a "fit_predict" method for "mixture.GMM" and subclasses. By
  >>:user:`Cory Lorenz <clorenz7>`<<.

* Added the "metrics.label_ranking_loss()" metric. By Arnaud Joly.

* Added the "metrics.cohen_kappa_score()" metric.

* Added a "warm_start" constructor parameter to the bagging ensemble
  models to increase the size of the ensemble. By >>:user:`Tim Head
  <betatim>`<<.

* Added option to use multi-output regression metrics without
  averaging. By Konstantin Shmelkov and >>:user:`Michael
  Eickenberg<eickenberg>`<<.

* Added "stratify" option to "cross_validation.train_test_split()" for
  stratified splitting. By Miroslav Batchkarov.

* The "tree.export_graphviz()" function now supports aesthetic
  improvements for "tree.DecisionTreeClassifier" and
  "tree.DecisionTreeRegressor", including options for coloring nodes
  by their majority class or impurity, showing variable names, and
  using node proportions instead of raw sample counts. By Trevor
  Stephens.

* Improved speed of "newton-cg" solver in
  "linear_model.LogisticRegression", by avoiding loss computation. By
  Mathieu Blondel and Tom Dupre la Tour.

* The "class_weight="auto"" heuristic in classifiers supporting
  "class_weight" was deprecated and replaced by the
  "class_weight="balanced"" option, which has a simpler formula and

  interpretation. By Hanna Wallach and Andreas Müller.

* Add "class_weight" parameter to automatically weight samples by
  class frequency for "linear_model.PassiveAggressiveClassifier". By
  Trevor Stephens.

* Added backlinks from the API reference pages to the user guide. By

* The "labels" parameter to "sklearn.metrics.f1_score()",
  "sklearn.metrics.fbeta_score()", "sklearn.metrics.recall_score()"
  and "sklearn.metrics.precision_score()" has been extended. It is now
  possible to ignore one or more labels, such as where a multiclass
  problem has a majority class to ignore. By Joel Nothman.

* Add "sample_weight" support to "linear_model.RidgeClassifier". By

* Provide an option for sparse output from
  "sklearn.metrics.pairwise.cosine_similarity()". By >>:user:`Jaidev
  Deshpande <jaidevd>`<<.

* Add "minmax_scale()" to provide a function interface for
  "MinMaxScaler". By >>:user:`Thomas Unterthiner <untom>`<<.

* "dump_svmlight_file" now handles multi-label datasets. By Chih-Wei
  Chang.

* RCV1 dataset loader ("sklearn.datasets.fetch_rcv1()"). By Tom Dupre
  la Tour.

* The "Wisconsin Breast Cancer" classical two-class classification
  dataset is now included in scikit-learn, available with
  "sklearn.dataset.load_breast_cancer()".

la Tour.

* Upgraded to joblib 0.9.3 to benefit from the new automatic batching
  of short tasks. This makes it possible for scikit-learn to benefit
  from parallelism when many very short tasks are executed in
  parallel, for instance by the "grid_search.GridSearchCV" meta-
  estimator with "n_jobs > 1" used with a large grid of parameters on
  a small dataset. By Vlad Niculae, Olivier Grisel and Loic Esteve.

* For more details about changes in joblib 0.9.3 see the release
  notes:
  https://github.com/joblib/joblib/blob/master/CHANGES.rst#release-093

* Improved speed (3 times per iteration) of
  "decomposition.DictLearning" with coordinate descent method from
  "linear_model.Lasso". By >>:user:`Arthur Mensch <arthurmensch>`<<.

* Parallel processing (threaded) for queries of nearest neighbors
  (using the ball-tree) by Nikolay Mayorov.

* Allow "datasets.make_multilabel_classification()" to output a sparse
  "y". By Kashif Rasul.

* "cluster.DBSCAN" now accepts a sparse matrix of precomputed
  distances, allowing memory-efficient distance precomputation. By

* "tree.DecisionTreeClassifier" now exposes an "apply" method for
  retrieving the leaf indices samples are predicted as. By
  >>:user:`Daniel Galvez <galv>`<< and Gilles Louppe.

* Speed up decision tree regressors, random forest regressors, extra

retrieving the leaf indices samples are predicted as. By

  trees regressors and gradient boosting estimators by computing a
  proxy of the impurity improvement during the tree growth. The proxy
  quantity is such that the split that maximizes this value also
  maximizes the impurity improvement. By Arnaud Joly, >>:user:`Jacob
  Schreiber <jmschrei>`<< and Gilles Louppe.

* Speed up tree based methods by reducing the number of computations
  needed when computing the impurity measure taking into account
  linear relationship of the computed statistics. The effect is
  particularly visible with extra trees and on datasets with
  categorical or sparse features. By Arnaud Joly.

* "ensemble.GradientBoostingRegressor" and
  "ensemble.GradientBoostingClassifier" now expose an "apply" method
  for retrieving the leaf indices each sample ends up in under each
  try. By >>:user:`Jacob Schreiber <jmschrei>`<<.

* Add "sample_weight" support to "linear_model.LinearRegression". By
  Sonny Hu. (>>:issue:`#4881`<<)

* Add "n_iter_without_progress" to "manifold.TSNE" to control the
  stopping criterion. By Santi Villalba. (>>:issue:`5186`<<)

* Added optional parameter "random_state" in "linear_model.Ridge" , to
  set the seed of the pseudo random generator used in "sag" solver. By
  Tom Dupre la Tour.

* Added optional parameter "warm_start" in

set the seed of the pseudo random generator used in "sag" solver. By

  "linear_model.LogisticRegression". If set to True, the solvers
  "lbfgs", "newton-cg" and "sag" will be initialized with the
  coefficients computed in the previous fit. By Tom Dupre la Tour.

* Added "sample_weight" support to "linear_model.LogisticRegression"
  for the "lbfgs", "newton-cg", and "sag" solvers. By Valentin
  Stolbunov. Support added to the "liblinear" solver. By Manoj Kumar.

* Added optional parameter "presort" to
  "ensemble.GradientBoostingRegressor" and
  "ensemble.GradientBoostingClassifier", keeping default behavior the
  same. This allows gradient boosters to turn off presorting when
  building deep trees or using sparse data. By >>:user:`Jacob
  Schreiber <jmschrei>`<<.

* Altered "metrics.roc_curve()" to drop unnecessary thresholds by
  default. By >>:user:`Graham Clenaghan <gclenaghan>`<<.

* Added "feature_selection.SelectFromModel" meta-transformer which can
  be used along with estimators that have *coef_* or
  *feature_importances_* attribute to select important features of the
  input data. By >>:user:`Maheshakya Wijewardena <maheshakya>`<<, Joel
  Nothman and Manoj Kumar.

* Added "metrics.pairwise.laplacian_kernel()".  By Clyde Fare.

* "covariance.GraphLasso" allows separate control of the convergence
  criterion for the Elastic-Net subproblem via  the "enet_tol"
  parameter.

* Improved verbosity in "decomposition.DictionaryLearning".

criterion for the Elastic-Net subproblem via  the "enet_tol"

* "ensemble.RandomForestClassifier" and
  "ensemble.RandomForestRegressor" no longer explicitly store the
  samples used in bagging, resulting in a much reduced memory
  footprint for storing random forest models.

* Added "positive" option to "linear_model.Lars" and
  "linear_model.lars_path()" to force coefficients to be positive.
  (>>:issue:`5131`<<)

* Added the "X_norm_squared" parameter to
  "metrics.pairwise.euclidean_distances()" to provide precomputed
  squared norms for "X".

* Added the "fit_predict" method to "pipeline.Pipeline".

* Added the "preprocessing.min_max_scale()" function.

Bug fixes
---------

* Fixed non-determinism in "dummy.DummyClassifier" with sparse multi-
  label output. By Andreas Müller.

* Fixed the output shape of "linear_model.RANSACRegressor" to
  "(n_samples, )". By Andreas Müller.

* Fixed bug in "decomposition.DictLearning" when "n_jobs < 0". By

* Fixed bug where "grid_search.RandomizedSearchCV" could consume a lot
  of memory for large discrete grids. By Joel Nothman.

* Fixed bug in "linear_model.LogisticRegressionCV" where *penalty* was
  ignored in the final fit. By Manoj Kumar.

* Fixed bug in "ensemble.forest.ForestClassifier" while computing
  oob_score and X is a sparse.csc_matrix. By >>:user:`Ankur Ankan
  <ankurankan>`<<.

* All regressors now consistently handle and warn when given "y" that

oob_score and X is a sparse.csc_matrix. By >>:user:`Ankur Ankan

  is of shape "(n_samples, 1)". By Andreas Müller and Henry Lin.
  (>>:issue:`5431`<<)

* Fix in "cluster.KMeans" cluster reassignment for sparse input by
  Lars Buitinck.

* Fixed a bug in "lda.LDA" that could cause asymmetric covariance
  matrices when using shrinkage. By Martin Billinger.

* Fixed "cross_validation.cross_val_predict()" for estimators with
  sparse predictions. By Buddha Prakash.

* Fixed the "predict_proba" method of
  "linear_model.LogisticRegression" to use soft-max instead of one-vs-
  rest normalization. By Manoj Kumar. (>>:issue:`5182`<<)

* Fixed the "partial_fit()" method of "linear_model.SGDClassifier"
  when called with "average=True". By >>:user:`Andrew Lamb
  <andylamb>`<<. (>>:issue:`5282`<<)

* Dataset fetchers use different filenames under Python 2 and Python 3
  to avoid pickling compatibility issues. By Olivier Grisel.
  (>>:issue:`5355`<<)

* Fixed a bug in "naive_bayes.GaussianNB" which caused classification
  results to depend on scale. By Jake Vanderplas.

* Fixed temporarily "linear_model.Ridge", which was incorrect when
  fitting the intercept in the case of sparse data. The fix
  automatically changes the solver to 'sag' in this case.
  >>:issue:`5360`<< by Tom Dupre la Tour.

* Fixed a performance bug in "decomposition.RandomizedPCA" on data
  with a large number of features and fewer samples.

>>:issue:`5360`<< by Tom Dupre la Tour.

  (>>:issue:`4478`<<) By Andreas Müller, Loic Esteve and

* Fixed bug in "cross_decomposition.PLS" that yielded unstable and
  platform dependent output, and failed on *fit_transform*. By
  >>:user:`Arthur Mensch <arthurmensch>`<<.

* Fixes to the "Bunch" class used to store datasets.

* Fixed "ensemble.plot_partial_dependence()" ignoring the
  "percentiles" parameter.

* Providing a "set" as vocabulary in "CountVectorizer" no longer leads
  to inconsistent results when pickling.

* Fixed the conditions on when a precomputed Gram matrix needs to be
  recomputed in "linear_model.LinearRegression",
  "linear_model.OrthogonalMatchingPursuit", "linear_model.Lasso" and
  "linear_model.ElasticNet".

* Fixed inconsistent memory layout in the coordinate descent solver
  that affected "linear_model.DictionaryLearning" and
  "covariance.GraphLasso". (>>:issue:`5337`<<) By Olivier Grisel.

* "manifold.LocallyLinearEmbedding" no longer ignores the "reg"

* Nearest Neighbor estimators with custom distance metrics can now be
  pickled. (>>:issue:`4362`<<)

* Fixed a bug in "pipeline.FeatureUnion" where "transformer_weights"
  were not properly handled when performing grid-searches.

* Fixed a bug in "linear_model.LogisticRegression" and
  "linear_model.LogisticRegressionCV" when using

were not properly handled when performing grid-searches.

  "class_weight='balanced'" or "class_weight='auto'". By Tom Dupre la
  Tour.

* Fixed bug >>:issue:`5495`<< when doing
  OVR(SVC(decision_function_shape="ovr")). Fixed by >>:user:`Elvis
  Dohmatob <dohmatob>`<<.

* Attribute *data_min*, *data_max* and *data_range* in
  "preprocessing.MinMaxScaler" are deprecated and won't be available
  from 0.19. Instead, the class now exposes *data_min_*, *data_max_*
  and *data_range_*. By >>:user:`Giorgio Patrini <giorgiop>`<<.

* All Scaler classes now have an *scale_* attribute, the feature-wise
  rescaling applied by their *transform* methods. The old attribute
  *std_* in "preprocessing.StandardScaler" is deprecated and
  superseded by *scale_*; it won't be available in 0.19. By

* "svm.SVC`" and "svm.NuSVC" now have an "decision_function_shape"
  parameter to make their decision function of shape "(n_samples,
  n_classes)" by setting "decision_function_shape='ovr'". This will be
  the default behavior starting in 0.19. By Andreas Müller.

* Passing 1D data arrays as input to estimators is now deprecated as
  it caused confusion in how the array elements should be interpreted
  as features or as samples. All data arrays are now expected to be
  explicitly shaped "(n_samples, n_features)". By >>:user:`Vighnesh

as features or as samples. All data arrays are now expected to be
  Birodkar <vighneshbirodkar>`<<.

* "lda.LDA" and "qda.QDA" have been moved to
  "discriminant_analysis.LinearDiscriminantAnalysis" and

* The "store_covariance" and "tol" parameters have been moved from the
  fit method to the constructor in
  "discriminant_analysis.LinearDiscriminantAnalysis" and the
  "store_covariances" and "tol" parameters have been moved from the

* Models inheriting from "_LearntSelectorMixin" will no longer support
  the transform methods. (i.e,  RandomForests, GradientBoosting,
  LogisticRegression, DecisionTrees, SVMs and SGD related models).
  Wrap these models around the metatransfomer
  "feature_selection.SelectFromModel" to remove features (according to
  *coefs_* or *feature_importances_*) which are below a certain
  threshold value instead.

* "cluster.KMeans" re-runs cluster-assignments in case of non-
  convergence, to ensure consistency of "predict(X)" and "labels_". By
  >>:user:`Vighnesh Birodkar <vighneshbirodkar>`<<.

* Classifier and Regressor models are now tagged as such using the
  "_estimator_type" attribute.

* Cross-validation iterators always provide indices into training and
  test set, not boolean masks.

* The "decision_function" on all regressors was deprecated and will be

"_estimator_type" attribute.

  removed in 0.19.  Use "predict" instead.

* "datasets.load_lfw_pairs()" is deprecated and will be removed in
  0.19. Use "datasets.fetch_lfw_pairs()" instead.

* The deprecated "hmm" module was removed.

* The deprecated "Bootstrap" cross-validation iterator was removed.

* The deprecated "Ward" and "WardAgglomerative" classes have been
  removed. Use "clustering.AgglomerativeClustering" instead.

* "cross_validation.check_cv()" is now a public function.

* The property "residues_" of "linear_model.LinearRegression" is
  deprecated and will be removed in 0.19.

* The deprecated "n_jobs" parameter of "linear_model.LinearRegression"
  has been moved to the constructor.

* Removed deprecated "class_weight" parameter from
  "linear_model.SGDClassifier"'s "fit" method. Use the construction
  parameter instead.

* The deprecated support for the sequence of sequences (or list of
  lists) multilabel format was removed. To convert to and from the

* The behavior of calling the "inverse_transform" method of
  "Pipeline.pipeline" will change in 0.19. It will no longer reshape
  one-dimensional input to two-dimensional input.

* The deprecated attributes "indicator_matrix_", "multilabel_" and
  "classes_" of "preprocessing.LabelBinarizer" were removed.

one-dimensional input to two-dimensional input.

* Using "gamma=0" in "svm.SVC" and "svm.SVR" to automatically set the
  gamma to "1. / n_features" is deprecated and will be removed in
  0.19. Use "gamma="auto"" instead.

Aaron Schumacher, Adithya Ganesh, akitty, Alexandre Gramfort, Alexey
Grigorev, Ali Baharev, Allen Riddell, Ando Saabas, Andreas Mueller,
Andrew Lamb, Anish Shah, Ankur Ankan, Anthony Erlinger, Ari Rouvinen,
Arnaud Joly, Arnaud Rachez, Arthur Mensch, banilo, Barmaley.exe,
benjaminirving, Boyuan Deng, Brett Naul, Brian McFee, Buddha Prakash,
Chi Zhang, Chih-Wei Chang, Christof Angermueller, Christoph Gohlke,
Christophe Bourguignat, Christopher Erick Moody, Chyi-Kwei Yau, Cindy
Sridharan, CJ Carey, Clyde-fare, Cory Lorenz, Dan Blanchard, Daniel
Galvez, Daniel Kronovet, Danny Sullivan, Data1010, David, David D
Lowe, David Dotson, djipey, Dmitry Spikhalskiy, Donne Martin, Dougal
J. Sutherland, Dougal Sutherland, edson duarte, Eduardo Caro, Eric
Larson, Eric Martin, Erich Schubert, Fernando Carrillo, Frank C.
Eckert, Frank Zalkow, Gael Varoquaux, Ganiev Ibraim, Gilles Louppe,
Giorgio Patrini, giorgiop, Graham Clenaghan, Gryllos Prokopis, gwulfs,
Henry Lin, Hsuan-Tien Lin, Immanuel Bayer, Ishank Gulati, Jack Martin,
Jacob Schreiber, Jaidev Deshpande, Jake Vanderplas, Jan Hendrik

Metzen, Jean Kossaifi, Jeffrey04, Jeremy, jfraj, Jiali Mei, Joe
Jevnik, Joel Nothman, John Kirkham, John Wittenauer, Joseph, Joshua
Loyal, Jungkook Park, KamalakerDadi, Kashif Rasul, Keith Goodman, Kian
Ho, Konstantin Shmelkov, Kyler Brown, Lars Buitinck, Lilian Besson,
Loic Esteve, Louis Tiao, maheshakya, Maheshakya Wijewardena, Manoj
Kumar, MarkTab marktab.net, Martin Ku, Martin Spacek, MartinBpr,
martinosorb, MaryanMorel, Masafumi Oyamada, Mathieu Blondel, Matt
Krump, Matti Lyra, Maxim Kolganov, mbillinger, mhg, Michael Heilman,
Michael Patterson, Miroslav Batchkarov, Nelle Varoquaux, Nicolas,
Nikolay Mayorov, Olivier Grisel, Omer Katz, Óscar Nájera, Pauli
Virtanen, Peter Fischer, Peter Prettenhofer, Phil Roth, pianomania,
Preston Parry, Raghav RV, Rob Zinkov, Robert Layton, Rohan Ramanath,
Saket Choudhary, Sam Zhang, santi, saurabh.bansod, scls19fr, Sebastian
Raschka, Sebastian Saeger, Shivan Sornarajah, SimonPL, sinhrks,
Skipper Seabold, Sonny Hu, sseg, Stephen Hoover, Steven De Gryze,
Steven Seguin, Theodore Vasiloudis, Thomas Unterthiner, Tiago Freitas
Pereira, Tian Wang, Tim Head, Timothy Hopper, tokoroten, Tom Dupré la
Tour, Trevor Stephens, Valentin Stolbunov, Vighnesh Birodkar, Vinayak
Mehta, Vincent, Vincent Michel, vstolbunov, wangz10, Wei Xue, Yucheng
Low, Yury Zhauniarovich, Zac Stewart, zhai_pro, Zichen Wang

Version 0.12.1

**October 8, 2012**

The 0.12.1 release is a bug-fix release with no additional features,
but is instead a set of bug fixes

Changelog
=========

* Improved numerical stability in spectral embedding by Gael Varoquaux

* Doctest under windows 64bit by Gael Varoquaux

* Documentation fixes for elastic net by Andreas Müller and Alexandre
  Gramfort

* Proper behavior with fortran-ordered NumPy arrays by Gael Varoquaux

* Make GridSearchCV work with non-CSR sparse matrix by Lars Buitinck

* Fix parallel computing in MDS by Gael Varoquaux

* Fix Unicode support in count vectorizer by Andreas Müller

* Fix MinCovDet breaking with X.shape = (3, 1) by >>:user:`Virgile
  Fritsch <VirgileFritsch>`<<

* Fix clone of SGD objects by Peter Prettenhofer

* Stabilize GMM by >>:user:`Virgile Fritsch <VirgileFritsch>`<<

People
======

   * 14  Peter Prettenhofer

   * 12  Gael Varoquaux

   * 10  Andreas Müller

   * 5  Lars Buitinck

   * 3  >>:user:`Virgile Fritsch <VirgileFritsch>`<<

   * 1  Alexandre Gramfort

   * 1  Gilles Louppe

   * 1  Mathieu Blondel

Version 0.12

**September 4, 2012**

Changelog
=========

* Various speed improvements of the decision trees module, by Gilles
  Louppe.

* "GradientBoostingRegressor" and "GradientBoostingClassifier" now
  support feature subsampling via the "max_features" argument, by
  Peter Prettenhofer.

* Added Huber and Quantile loss functions to
  "GradientBoostingRegressor", by Peter Prettenhofer.

* Decision trees and forests of randomized trees now support multi-

Peter Prettenhofer.

  output classification and regression problems, by Gilles Louppe.

* Added "LabelEncoder", a simple utility class to normalize labels or
  transform non-numerical labels, by Mathieu Blondel.

* Added the epsilon-insensitive loss and the ability to make
  probabilistic predictions with the modified huber loss in Stochastic
  Gradient Descent, by Mathieu Blondel.

* Added Multi-dimensional Scaling (MDS), by Nelle Varoquaux.

* SVMlight file format loader now detects compressed (gzip/bzip2)
  files and decompresses them on the fly, by Lars Buitinck.

* SVMlight file format serializer now preserves double precision
  floating point values, by Olivier Grisel.

* A common testing framework for all estimators was added, by Andreas
  Müller.

* Understandable error messages for estimators that do not accept
  sparse input by Gael Varoquaux

* Speedups in hierarchical clustering by Gael Varoquaux. In particular
  building the tree now supports early stopping. This is useful when
  the number of clusters is not small compared to the number of
  samples.

* Add MultiTaskLasso and MultiTaskElasticNet for joint feature
  selection, by Alexandre Gramfort.

* Added "metrics.auc_score()" and "metrics.average_precision_score()"
  convenience functions by Andreas Müller.

* Improved sparse matrix support in the Feature selection module by

* New word boundaries-aware character n-gram analyzer for the Text
  feature extraction module by >>:user:`@kernc <kernc>`<<.

* Fixed bug in spectral clustering that led to single point clusters

* In "CountVectorizer", added an option to ignore infrequent words,
  "min_df" by  Andreas Müller.

* Add support for multiple targets in some linear models (ElasticNet,
  Lasso and OrthogonalMatchingPursuit) by Vlad Niculae and Alexandre
  Gramfort.

* Fixes in "ProbabilisticPCA" score function by Wei Li.

* Fixed feature importance computation in Gradient Tree Boosting.

* The old "scikits.learn" package has disappeared; all code should
  import from "sklearn" instead, which was introduced in 0.9.

* In "metrics.roc_curve()", the "thresholds" array is now returned
  with it's order reversed, in order to keep it consistent with the
  order of the returned "fpr" and "tpr".

* In "hmm" objects, like "GaussianHMM", "MultinomialHMM", etc., all
  parameters must be passed to the object when initialising it and not
  through "fit". Now "fit" will only accept the data as an input

* For all SVM classes, a faulty behavior of "gamma" was fixed.
  Previously, the default gamma value was only computed the first time

parameter.

  "fit" was called and then stored. It is now recalculated on every
  call to "fit".

* All "Base" classes are now abstract meta classes so that they can
  not be instantiated.

* "cluster.ward_tree()" now also returns the parent array. This is
  necessary for early-stopping in which case the tree is not
  completely built.

* In "CountVectorizer" the parameters "min_n" and "max_n" were joined
  to the parameter "n_gram_range" to enable grid-searching both at
  once.

* In "CountVectorizer", words that appear only in one document are now
  ignored by default. To reproduce the previous behavior, set
  "min_df=1".

* Fixed API inconsistency:
  "linear_model.SGDClassifier.predict_proba()" now returns 2d array
  when fit on two classes.

* Fixed API inconsistency: "discriminant_analysis.QuadraticDiscrimina
  ntAnalysis.decision_function()" and "discriminant_analysis.LinearDi
  scriminantAnalysis.decision_function()" now return 1d arrays when
  fit on two classes.

* Grid of alphas used for fitting "LassoCV" and "ElasticNetCV" is now
  stored in the attribute "alphas_" rather than overriding the init
  parameter "alphas".

* Linear models when alpha is estimated by cross-validation store the
  estimated value in the "alpha_" attribute rather than just "alpha"
  or "best_alpha".

* "GradientBoostingClassifier" now supports "staged_predict_proba()",
  and "staged_predict()".

estimated value in the "alpha_" attribute rather than just "alpha"

* "SVC" and other sparse SVM classes are now deprecated. The all
  classes in the Support Vector Machines module now automatically
  select the sparse or dense representation base on the input.

* All clustering algorithms now interpret the array "X" given to "fit"
  as input data, in particular "SpectralClustering" and
  "AffinityPropagation" which previously expected affinity matrices.

* For clustering algorithms that take the desired number of clusters
  as a parameter, this parameter is now called "n_clusters".

People
======

   * 267  Andreas Müller

   * 94  Gilles Louppe

   * 89  Gael Varoquaux

   * 79  Peter Prettenhofer

   * 60  Mathieu Blondel

   * 57  Alexandre Gramfort

   * 52  Vlad Niculae

   * 45  Lars Buitinck

   * 44  Nelle Varoquaux

   * 37  Jaques Grobler

   * 30  Alexis Mignon

   * 30  Immanuel Bayer

   * 27  Olivier Grisel

   * 16  Subhodeep Moitra

   * 13  Yannick Schwartz

   * 12  >>:user:`@kernc <kernc>`<<

   * 11  >>:user:`Virgile Fritsch <VirgileFritsch>`<<

   * 9  Daniel Duckworth

   * 9  Fabian Pedregosa

   * 9  Robert Layton

   * 8  John Benediktsson

   * 7  Marko Burjek

   * 5  Nicolas Pinto

   * 4  Alexandre Abraham

   * 4  Jake Vanderplas

   * 3  Brian Holt

   * 3  Edouard Duchesnay

   * 3  Florian Hoenig

   * 3  flyingimmidev

   * 2  Francois Savard

   * 2  Hannes Schulz

   * 2  Peter Welinder

* 4  Alexandre Abraham

   * 2  Yaroslav Halchenko

   * 2  Wei Li

   * 1  Alex Companioni

   * 1  Brandyn A. White

   * 1  Bussonnier Matthias

   * 1  Charles-Pierre Astolfi

   * 1  Dan O'Huiginn

   * 1  David Cournapeau

   * 1  Keith Goodman

   * 1  Ludwig Schwardt

   * 1  Olivier Hervieu

   * 1  Shiqiao Du

   * 1  Tim Sheerman-Chase

   * 1  buguen

Version 0.11

**May 7, 2012**

Changelog
=========

* Gradient boosted regression trees (Gradient Tree Boosting) for
  classification and regression by Peter Prettenhofer and Scott White
  .

* Simple dict-based feature loader with support for categorical
  variables ("DictVectorizer") by Lars Buitinck.

* Added Matthews correlation coefficient
  ("metrics.matthews_corrcoef()") and added macro and micro average
  options to "precision_score()", "metrics.recall_score()" and
  "f1_score()" by Satrajit Ghosh.

* Out of Bag Estimates of generalization error for Ensemble methods by

* Randomized sparse linear models for feature selection, by Alexandre
  Gramfort and Gael Varoquaux

* Label Propagation for semi-supervised learning, by Clay Woolam.
  **Note** the semi-supervised API is still work in progress, and may
  change.

* Added BIC/AIC model selection to classical Gaussian mixture models

**Note** the semi-supervised API is still work in progress, and may
  change.

  and unified the API with the remainder of scikit-learn, by Bertrand
  Thirion

* Added "StratifiedShuffleSplit", which is a "ShuffleSplit" with
  balanced splits, by Yannick Schwartz.

* "NearestCentroid" classifier added, along with a "shrink_threshold"
  parameter, which implements **shrunken centroid classification**, by
  Robert Layton.

Other changes

* Merged dense and sparse implementations of Stochastic Gradient
  Descent module and exposed utility extension types for sequential
  datasets "seq_dataset" and weight vectors "weight_vector" by Peter
  Prettenhofer.

* Added "partial_fit" (support for online/minibatch learning) and
  warm_start to the Stochastic Gradient Descent module by Mathieu
  Blondel.

* Dense and sparse implementations of Support Vector Machines classes
  and "LogisticRegression" merged by Lars Buitinck.

* Regressors can now be used as base estimator in the Multiclass and
  multioutput algorithms module by Mathieu Blondel.

* Added n_jobs option to "metrics.pairwise.pairwise_distances()" and
  "metrics.pairwise.pairwise_kernels()" for parallel computation, by
  Mathieu Blondel.

* K-means can now be run in parallel, using the "n_jobs" argument to
  either K-means or "KMeans", by Robert Layton.

* Improved Cross-validation: evaluating estimator performance and
  Tuning the hyper-parameters of an estimator documentation and

either K-means or "KMeans", by Robert Layton.

  introduced the new "cross_validation.train_test_split()" helper
  function by Olivier Grisel

* "SVC" members "coef_" and "intercept_" changed sign for consistency
  with "decision_function"; for "kernel==linear", "coef_" was fixed in
  the one-vs-one case, by Andreas Müller.

* Performance improvements to efficient leave-one-out cross-validated
  Ridge regression, esp. for the "n_samples > n_features" case, in
  "RidgeCV", by Reuben Fletcher-Costin.

* Refactoring and simplification of the Text feature extraction API
  and fixed a bug that caused possible negative IDF, by Olivier
  Grisel.

* Beam pruning option in "_BaseHMM" module has been removed since it
  is difficult to Cythonize. If you are interested in contributing a
  Cython version, you can use the python version in the git history as
  a reference.

* Classes in Nearest Neighbors now support arbitrary Minkowski metric
  for nearest neighbors searches. The metric can be specified by
  argument "p".

* "EllipticEnvelop" is now deprecated - Please use "EllipticEnvelope"
  instead.

* "NeighborsClassifier" and "NeighborsRegressor" are gone in the
  module Nearest Neighbors. Use the classes "KNeighborsClassifier",
  "RadiusNeighborsClassifier", "KNeighborsRegressor" and/or
  "RadiusNeighborsRegressor" instead.

module Nearest Neighbors. Use the classes "KNeighborsClassifier",

* Sparse classes in the Stochastic Gradient Descent module are now
  deprecated.

* In "GMM", "DPGMM" and "VBGMM", parameters must be passed to an
  object when initialising it and not through "fit". Now "fit" will
  only accept the data as an input parameter.

* methods "rvs" and "decode" in "GMM" module are now deprecated.
  "sample" and "score" or "predict" should be used instead.

* attribute "_scores" and "_pvalues" in univariate feature selection
  objects are now deprecated. "scores_" or "pvalues_" should be used
  instead.

* In "LogisticRegression", "LinearSVC", "SVC" and "NuSVC", the
  "class_weight" parameter is now an initialization parameter, not a
  parameter to fit. This makes grid searches over this parameter
  possible.

* LFW "data" is now always shape "(n_samples, n_features)" to be
  consistent with the Olivetti faces dataset. Use "images" and "pairs"
  attribute to access the natural images shapes instead.

* In "LinearSVC", the meaning of the "multi_class" parameter changed.
  Options now are "'ovr'" and "'crammer_singer'", with "'ovr'" being
  the default.  This does not change the default behavior but
  hopefully is less confusing.

* Class "Vectorizer" is deprecated and replaced by "TfidfVectorizer".

* The preprocessor / analyzer nested structure for text feature

hopefully is less confusing.

  extraction has been removed. All those features are now directly
  passed as flat constructor arguments to "TfidfVectorizer" and
  "CountVectorizer", in particular the following parameters are now
  used:

* "analyzer" can be "'word'" or "'char'" to switch the default
  analysis scheme, or use a specific python callable (as previously).

* "tokenizer" and "preprocessor" have been introduced to make it still
  possible to customize those steps with the new API.

* "input" explicitly control how to interpret the sequence passed to
  "fit" and "predict": filenames, file objects or direct (byte or
  Unicode) strings.

* charset decoding is explicit and strict by default.

* the "vocabulary", fitted or not is now stored in the "vocabulary_"
  attribute to be consistent with the project conventions.

* Class "TfidfVectorizer" now derives directly from "CountVectorizer"
  to make grid search trivial.

* methods "rvs" in "_BaseHMM" module are now deprecated. "sample"
  should be used instead.

* Beam pruning option in "_BaseHMM" module is removed since it is
  difficult to be Cythonized. If you are interested, you can look in
  the history codes by git.

* The SVMlight format loader now supports files with both zero-based
  and one-based column indices, since both occur "in the wild".

* Arguments in class "ShuffleSplit" are now consistent with

  "StratifiedShuffleSplit". Arguments "test_fraction" and
  "train_fraction" are deprecated and renamed to "test_size" and
  "train_size" and can accept both "float" and "int".

* Arguments in class "Bootstrap" are now consistent with
  "StratifiedShuffleSplit". Arguments "n_test" and "n_train" are
  deprecated and renamed to "test_size" and "train_size" and can
  accept both "float" and "int".

* Argument "p" added to classes in Nearest Neighbors to specify an
  arbitrary Minkowski metric for nearest neighbors searches.

People
======

   * 282  Andreas Müller

   * 239  Peter Prettenhofer

   * 198  Gael Varoquaux

   * 129  Olivier Grisel

   * 114  Mathieu Blondel

   * 103  Clay Woolam

   * 96  Lars Buitinck

   * 88  Jaques Grobler

   * 82  Alexandre Gramfort

   * 50  Bertrand Thirion

   * 42  Robert Layton

   * 28  flyingimmidev

   * 26  Jake Vanderplas

   * 26  Shiqiao Du

   * 21  Satrajit Ghosh

   * 17  David Marek

   * 17  Gilles Louppe

   * 14  Vlad Niculae

   * 11  Yannick Schwartz

   * 10  Fabian Pedregosa

   * 9  fcostin

   * 7  Nick Wilson

   * 5  Adrien Gaidon

   * 4  David Warde-Farley

   * 5  Nelle Varoquaux

   * 5  Emmanuelle Gouillart

   * 3  Joonas Sillanpää

   * 3  Paolo Losi

   * 2  Charles McCarthy

   * 2  Roy Hyunjin Han

   * 2  Scott White

   * 2  ibayer

   * 1  Brandyn White

* 5  Nelle Varoquaux

   * 2  ibayer

   * 1  Carlos Scheidegger

   * 1  Claire Revillet

   * 1  Conrad Lee

   * 1  Edouard Duchesnay

   * 1  Jan Hendrik Metzen

   * 1  Meng Xinfan

   * 1  Rob Zinkov

   * 1  Shiqiao

   * 1  Udi Weinsberg

   * 1  Virgile Fritsch

   * 1  Xinfan Meng

   * 1  jansoe

   * 1  Leon Palafox

Version 0.10

**January 11, 2012**

Changelog
=========

* Python 2.5 compatibility was dropped; the minimum Python version
  needed to use scikit-learn is now 2.6.

* Sparse inverse covariance estimation using the graph Lasso, with
  associated cross-validated estimator, by Gael Varoquaux

* New Tree module by Brian Holt, Peter Prettenhofer, Satrajit Ghosh
  and Gilles Louppe. The module comes with complete documentation and
  examples.

* Fixed a bug in the RFE module by Gilles Louppe (issue #378).

* Fixed a memory leak in Support Vector Machines module by Brian Holt
  (issue #367).

* Faster tests by Fabian Pedregosa and others.

* Silhouette Coefficient cluster analysis evaluation metric added as
  "silhouette_score()" by Robert Layton.

* Fixed a bug in K-means in the handling of the "n_init" parameter:
  the clustering algorithm used to be run "n_init" times but the last
  solution was retained instead of the best solution by Olivier
  Grisel.

* Minor refactoring in Stochastic Gradient Descent module;

the clustering algorithm used to be run "n_init" times but the last
  Grisel.

  consolidated dense and sparse predict methods; Enhanced test time
  performance by converting model parameters to fortran-style arrays
  after fitting (only multi-class).

* Adjusted Mutual Information metric added as
  "adjusted_mutual_info_score()" by Robert Layton.

* Models like SVC/SVR/LinearSVC/LogisticRegression from
  libsvm/liblinear now support scaling of C regularization parameter
  by the number of samples by Alexandre Gramfort.

* New Ensemble Methods module by Gilles Louppe and Brian Holt. The
  module comes with the random forest algorithm and the extra-trees
  method, along with documentation and examples.

* Novelty and Outlier Detection: outlier and novelty detection, by
  >>:user:`Virgile Fritsch <VirgileFritsch>`<<.

* Kernel Approximation: a transform implementing kernel approximation
  for fast SGD on non-linear kernels by Andreas Müller.

* Fixed a bug due to atom swapping in Orthogonal Matching Pursuit
  (OMP) by Vlad Niculae.

* Sparse coding with a precomputed dictionary by Vlad Niculae.

* Mini Batch K-Means performance improvements by Olivier Grisel.

* K-means support for sparse matrices by Mathieu Blondel.

* Improved documentation for developers and for the "sklearn.utils"
  module, by Jake Vanderplas.

* Vectorized 20newsgroups dataset loader

  ("fetch_20newsgroups_vectorized()") by Mathieu Blondel.

* Multiclass and multioutput algorithms by Lars Buitinck.

* Utilities for fast computation of mean and variance for sparse
  matrices by Mathieu Blondel.

* Make "scale()" and "Scaler" work on sparse matrices by Olivier
  Grisel

* Feature importances using decision trees and/or forest of trees, by

* Parallel implementation of forests of randomized trees by Gilles
  Louppe.

* "ShuffleSplit" can subsample the train sets as well as the test sets

* Errors in the build of the documentation fixed by Andreas Müller.

Here are the code migration instructions when upgrading from scikit-
learn version 0.9:

* Some estimators that may overwrite their inputs to save memory
  previously had "overwrite_" parameters; these have been replaced
  with "copy_" parameters with exactly the opposite meaning.

  This particularly affects some of the estimators in "linear_model".
  The default behavior is still to copy everything passed in.

* The SVMlight dataset loader "load_svmlight_file()" no longer
  supports loading two files at once; use "load_svmlight_files"
  instead. Also, the (unused) "buffer_mb" parameter is gone.

* Sparse estimators in the Stochastic Gradient Descent module use

supports loading two files at once; use "load_svmlight_files"

  dense parameter vector "coef_" instead of "sparse_coef_". This
  significantly improves test time performance.

* The Covariance estimation module now has a robust estimator of
  covariance, the Minimum Covariance Determinant estimator.

* Cluster evaluation metrics in "metrics.cluster" have been refactored
  but the changes are backwards compatible. They have been moved to
  the "metrics.cluster.supervised", along with
  "metrics.cluster.unsupervised" which contains the Silhouette
  Coefficient.

* The "permutation_test_score" function now behaves the same way as
  "cross_val_score" (i.e. uses the mean score across the folds.)

* Cross Validation generators now use integer indices ("indices=True")
  by default instead of boolean masks. This make it more intuitive to
  use with sparse matrix data.

* The functions used for sparse coding, "sparse_encode" and
  "sparse_encode_parallel" have been combined into "sparse_encode()",
  and the shapes of the arrays have been transposed for consistency
  with the matrix factorization setting, as opposed to the regression
  setting.

* Fixed an off-by-one error in the SVMlight/LibSVM file format
  handling; files generated using "dump_svmlight_file()" should be re-
  generated. (They should continue to work, but accidentally had one
  extra column of zeros prepended.)

handling; files generated using "dump_svmlight_file()" should be re-

* "BaseDictionaryLearning" class replaced by "SparseCodingMixin".

* "fast_svd()" has been renamed "randomized_svd()" and the default
  oversampling is now fixed to 10 additional random vectors instead of
  doubling the number of components to extract. The new behavior
  follows the reference paper.

People
======

The following people contributed to scikit-learn since last release:

   * 246  Andreas Müller

   * 242  Olivier Grisel

   * 220  Gilles Louppe

   * 183  Brian Holt

   * 166  Gael Varoquaux

   * 144  Lars Buitinck

   * 73  Vlad Niculae

   * 64  Fabian Pedregosa

   * 60  Robert Layton

   * 55  Mathieu Blondel

   * 52  Jake Vanderplas

   * 44  Noel Dawe

   * 38  Alexandre Gramfort

   * 24  >>:user:`Virgile Fritsch <VirgileFritsch>`<<

   * 23  Satrajit Ghosh

   * 3  Jan Hendrik Metzen

   * 3  Kenneth C. Arnold

   * 3  Shiqiao Du

   * 3  Tim Sheerman-Chase

   * 3  Yaroslav Halchenko

   * 2  Bala Subrahmanyam Varanasi

   * 2  DraXus

   * 2  Michael Eickenberg

   * 1  Bogdan Trach

   * 1  Juan Manuel Caicedo Carvajal

   * 1  Nelle Varoquaux

   * 1  Nicolas Pinto

   * 1  Tiziano Zito

Version 0.9
***********

**September 21, 2011**

scikit-learn 0.9 was released on September 2011, three months after
the 0.8 release and includes the new modules Manifold learning, The

* 1  Xinfan Meng

Dirichlet Process as well as several new algorithms and documentation
improvements.

This release also includes the dictionary-learning work developed by
Vlad Niculae as part of the Google Summer of Code program.

 [image: banner2][image] [image: banner1][image] [image:
banner3][image]

Changelog
=========

* New Manifold learning module by Jake Vanderplas and Fabian
  Pedregosa.

* New Dirichlet Process Gaussian Mixture Model by Alexandre Passos

* Nearest Neighbors module refactoring by Jake Vanderplas : general
  refactoring, support for sparse matrices in input, speed and
  documentation improvements. See the next section for a full list of
  API changes.

* Improvements on the Feature selection module by Gilles Louppe :
  refactoring of the RFE classes, documentation rewrite, increased
  efficiency and minor API changes.

* Sparse principal components analysis (SparsePCA and
  MiniBatchSparsePCA) by Vlad Niculae, Gael Varoquaux and Alexandre
  Gramfort

* Printing an estimator now behaves independently of architectures and
  Python version thanks to >>:user:`Jean Kossaifi <JeanKossaifi>`<<.

* Loader for libsvm/svmlight format by Mathieu Blondel and Lars
  Buitinck

* Documentation improvements: thumbnails in example gallery by Fabian

  Buitinck

* Important bugfixes in Support Vector Machines module (segfaults, bad
  performance) by Fabian Pedregosa.

* Added Multinomial Naive Bayes and Bernoulli Naive Bayes by Lars
  Buitinck

* Text feature extraction optimizations by Lars Buitinck

* Chi-Square feature selection
  ("feature_selection.univariate_selection.chi2()") by Lars Buitinck.

* Generated datasets module refactoring by Gilles Louppe

* Multiclass and multioutput algorithms by Mathieu Blondel

* Ball tree rewrite by Jake Vanderplas

* Implementation of DBSCAN algorithm by Robert Layton

* Kmeans predict and transform by Robert Layton

* Preprocessing module refactoring by Olivier Grisel

* Faster mean shift by Conrad Lee

* New "Bootstrap", Random permutations cross-validation a.k.a. Shuffle
  & Split and various other improvements in cross validation schemes
  by Olivier Grisel and Gael Varoquaux

* Adjusted Rand index and V-Measure clustering evaluation metrics by
  Olivier Grisel

* Added "Orthogonal Matching Pursuit" by Vlad Niculae

* Added 2D-patch extractor utilities in the Feature extraction module
  by Vlad Niculae

* Implementation of "LassoLarsCV" (cross-validated Lasso solver using
  the Lars algorithm) and "LassoLarsIC" (BIC/AIC model selection in
  Lars) by Gael Varoquaux and Alexandre Gramfort

* Scalability improvements to "metrics.roc_curve()" by Olivier Hervieu

the Lars algorithm) and "LassoLarsIC" (BIC/AIC model selection in

* Distance helper functions "metrics.pairwise.pairwise_distances()"
  and "metrics.pairwise.pairwise_kernels()" by Robert Layton

* "Mini-Batch K-Means" by Nelle Varoquaux and Peter Prettenhofer.

* mldata utilities by Pietro Berkes.

* *olivetti_faces_dataset* by David Warde-Farley.

learn version 0.8:

* The "scikits.learn" package was renamed "sklearn". There is still a
  "scikits.learn" package alias for backward compatibility.

  Third-party projects with a dependency on scikit-learn 0.9+ should
  upgrade their codebase. For instance, under Linux / MacOSX just run
  (make a backup first!):

     find -name "*.py" | xargs sed -i 's/\bscikits.learn\b/sklearn/g'

* Estimators no longer accept model parameters as "fit" arguments:
  instead all parameters must be only be passed as constructor
  arguments or using the now public "set_params" method inherited from
  "BaseEstimator".

  Some estimators can still accept keyword arguments on the "fit" but
  this is restricted to data-dependent values (e.g. a Gram matrix or
  an affinity matrix that are precomputed from the "X" data matrix.

* The "cross_val" package has been renamed to "cross_validation"
  although there is also a "cross_val" package alias in place for

an affinity matrix that are precomputed from the "X" data matrix.

  backward compatibility.

     find -name "*.py" | xargs sed -i 's/\bcross_val\b/cross_validation/g'

* The "score_func" argument of the
  "sklearn.cross_validation.cross_val_score" function is now expected
  to accept "y_test" and "y_predicted" as only arguments for
  classification and regression tasks or "X_test" for unsupervised
  estimators.

* "gamma" parameter for support vector machine algorithms is set to "1
  / n_features" by default, instead of "1 / n_samples".

* The "sklearn.hmm" has been marked as orphaned: it will be removed
  from scikit-learn in version 0.11 unless someone steps up to
  contribute documentation, examples and fix lurking numerical
  stability issues.

* "sklearn.neighbors" has been made into a submodule.  The two
  previously available estimators, "NeighborsClassifier" and
  "NeighborsRegressor" have been marked as deprecated.  Their
  functionality has been divided among five new classes:
  "NearestNeighbors" for unsupervised neighbors searches,
  "KNeighborsClassifier" & "RadiusNeighborsClassifier" for supervised
  classification problems, and "KNeighborsRegressor" &

"NearestNeighbors" for unsupervised neighbors searches,
  "RadiusNeighborsRegressor" for supervised regression problems.

* "sklearn.ball_tree.BallTree" has been moved to
  "sklearn.neighbors.BallTree".  Using the former will generate a
  warning.

* "sklearn.linear_model.LARS()" and related classes (LassoLARS,
  LassoLARSCV, etc.) have been renamed to
  "sklearn.linear_model.Lars()".

* All distance metrics and kernels in "sklearn.metrics.pairwise" now
  have a Y parameter, which by default is None. If not given, the
  result is the distance (or kernel similarity) between each sample in
  Y. If given, the result is the pairwise distance (or kernel
  similarity) between samples in X to Y.

* "sklearn.metrics.pairwise.l1_distance" is now called
  "manhattan_distance", and by default returns the pairwise distance.
  For the component wise distance, set the parameter
  "sum_over_features" to "False".

Backward compatibility package aliases and other deprecated classes
and functions will be removed in version 0.11.

People
======

38 people contributed to this release.

* 387  Vlad Niculae

* 320  Olivier Grisel

* 192  Lars Buitinck

* 179  Gael Varoquaux

* 168  Fabian Pedregosa (INRIA, Parietal Team)

* 127  Jake Vanderplas

* 120  Mathieu Blondel

* 85  Alexandre Passos

* 67  Alexandre Gramfort

* 57  Peter Prettenhofer

* 56  Gilles Louppe

* 42  Robert Layton

* 38  Nelle Varoquaux

* 32  >>:user:`Jean Kossaifi <JeanKossaifi>`<<

* 30  Conrad Lee

* 22  Pietro Berkes

* 18  andy

* 17  David Warde-Farley

* 12  Brian Holt

* 11  Robert

* 8  Amit Aides

* 8  >>:user:`Virgile Fritsch <VirgileFritsch>`<<

* 7  Yaroslav Halchenko

* 6  Salvatore Masecchia

* 5  Paolo Losi

* 4  Vincent Schut

* 3  Alexis Metaireau

* 3  Bryan Silverthorn

* 3  Andreas Müller

* 2  Minwoo Jake Lee

* 1  Emmanuelle Gouillart

* 1  Keith Goodman

* 1  Lucas Wiman

* 1  Nicolas Pinto

* 1  Thouis (Ray) Jones

* 1  Tim Sheerman-Chase

Version 0.8

**May 11, 2011**

scikit-learn 0.8 was released on May 2011, one month after the first
"international" scikit-learn coding sprint and is marked by the
inclusion of important modules: Hierarchical clustering, Cross
decomposition, Non-negative matrix factorization (NMF or NNMF),
initial support for Python 3 and by important enhancements and bug
fixes.

Changelog
=========

Several new modules where introduced during this release:

* New Hierarchical clustering module by Vincent Michel, Bertrand
  Thirion, Alexandre Gramfort and Gael Varoquaux.

* Kernel Principal Component Analysis (kPCA) implementation by Mathieu
  Blondel

* *labeled_faces_in_the_wild_dataset* by Olivier Grisel.

* New Cross decomposition module by Edouard Duchesnay.

* Non-negative matrix factorization (NMF or NNMF) module Vlad Niculae

Blondel

* Implementation of the Oracle Approximating Shrinkage algorithm by
  >>:user:`Virgile Fritsch <VirgileFritsch>`<< in the Covariance
  estimation module.

Some other modules benefited from significant improvements or
cleanups.

* Initial support for Python 3: builds and imports cleanly, some
  modules are usable while others have failing tests by Fabian

* "PCA" is now usable from the Pipeline object by Olivier Grisel.

* Guide How to optimize for speed by Olivier Grisel.

* Fixes for memory leaks in libsvm bindings, 64-bit safer BallTree by

* bug and style fixing in K-means algorithm by Jan Schlüter.

* Add attribute converged to Gaussian Mixture Models by Vincent Schut.

* Implemented "transform", "predict_log_proba" in
  "LinearDiscriminantAnalysis" By Mathieu Blondel.

* Refactoring in the Support Vector Machines module and bug fixes by
  Fabian Pedregosa, Gael Varoquaux and Amit Aides.

* Refactored SGD module (removed code duplication, better variable
  naming), added interface for sample weight by Peter Prettenhofer.

* Wrapped BallTree with Cython by Thouis (Ray) Jones.

* Added function "svm.l1_min_c()" by Paolo Losi.

* Typos, doc style, etc. by Yaroslav Halchenko, Gael Varoquaux,
  Olivier Grisel, Yann Malet, Nicolas Pinto, Lars Buitinck and Fabian

People
======

People
======

People that made this release possible preceded by number of commits:

* 159  Olivier Grisel

* 96  Gael Varoquaux

* 96  Vlad Niculae

* 94  Fabian Pedregosa

* 36  Alexandre Gramfort

* 32  Paolo Losi

* 31  Edouard Duchesnay

* 30  Mathieu Blondel

* 25  Peter Prettenhofer

* 22  Nicolas Pinto

* 11  >>:user:`Virgile Fritsch <VirgileFritsch>`<<
     * 7  Lars Buitinck

     * 6  Vincent Michel

     * 5  Bertrand Thirion

     * 4  Thouis (Ray) Jones

     * 4  Vincent Schut

     * 3  Jan Schlüter

     * 2  Julien Miotte

     * 2  Matthieu Perrot

     * 2  Yann Malet

     * 2  Yaroslav Halchenko

     * 1  Amit Aides

     * 1  Andreas Müller

     * 1  Feth Arezki

     * 1  Meng Xinfan

Version 0.7

**March 2, 2011**

scikit-learn 0.7 was released in March 2011, roughly three months
after the 0.6 release. This release is marked by the speed
improvements in existing algorithms like k-Nearest Neighbors and
K-Means algorithm and by the inclusion of an efficient algorithm for
computing the Ridge Generalized Cross Validation solution. Unlike the
preceding release, no new modules where added to this release.

Changelog
=========

* Performance improvements for Gaussian Mixture Model sampling [Jan
  Schlüter].

* Implementation of efficient leave-one-out cross-validated Ridge in
  "RidgeCV" [Mathieu Blondel]

* Better handling of collinearity and early stopping in

Schlüter].

  "linear_model.lars_path()" [Alexandre Gramfort and Fabian
  Pedregosa].

* Fixes for liblinear ordering of labels and sign of coefficients [Dan
  Yamins, Paolo Losi, Mathieu Blondel and Fabian Pedregosa].

* Performance improvements for Nearest Neighbors algorithm in high-
  dimensional spaces [Fabian Pedregosa].

* Performance improvements for "KMeans" [Gael Varoquaux and James
  Bergstra].

* Sanity checks for SVM-based classes [Mathieu Blondel].

* Refactoring of "NeighborsClassifier" and
  "neighbors.kneighbors_graph()": added different algorithms for the
  k-Nearest Neighbor Search and implemented a more stable algorithm
  for finding barycenter weights. Also added some developer
  documentation for this module, see notes_neighbors for more
  information [Fabian Pedregosa].

* Documentation improvements: Added "RandomizedPCA" and
  "LogisticRegression" to the class reference. Also added references
  of matrices used for clustering and other fixes [Gael Varoquaux,
  Fabian Pedregosa, Mathieu Blondel, Olivier Grisel, Virgile Fritsch ,
  Emmanuelle Gouillart]

* Binded decision_function in classes that make use of liblinear,
  dense and sparse variants, like "LinearSVC" or "LogisticRegression"
  [Fabian Pedregosa].

* Performance and API improvements to "metrics.euclidean_distances()"
  and to "RandomizedPCA" [James Bergstra].

[Fabian Pedregosa].

* Fix compilation issues under NetBSD [Kamel Ibn Hassen Derouiche]

* Allow input sequences of different lengths in "GaussianHMM" [Ron
  Weiss].

* Fix bug in affinity propagation caused by incorrect indexing [Xinfan
  Meng]

People
======

* 85  Fabian Pedregosa

* 67  Mathieu Blondel

* 20  Alexandre Gramfort

* 19  James Bergstra

* 14  Dan Yamins

* 13  Olivier Grisel

* 12  Gael Varoquaux

* 4  Edouard Duchesnay

* 4  Ron Weiss

* 2  Satrajit Ghosh

* 2  Vincent Dubourg

* 1  Kamel Ibn Hassen Derouiche

* 1  Paolo Losi

* 1  VirgileFritsch

* 1  Yaroslav Halchenko

Version 0.6

**December 21, 2010**

scikit-learn 0.6 was released on December 2010. It is marked by the
inclusion of several new modules and a general renaming of old ones.
It is also marked by the inclusion of new example, including
applications to real-world datasets.

Changelog
=========

* New stochastic gradient descent module by Peter Prettenhofer. The
  module comes with complete documentation and examples.

* Improved svm module: memory consumption has been reduced by 50%,
  heuristic to automatically set class weights, possibility to assign
  weights to samples (see
  *sphx_glr_auto_examples_svm_plot_weighted_samples.py* for an
  example).

* New Gaussian Processes module by Vincent Dubourg. This module also

weights to samples (see
  example).

  has great documentation and some very neat examples. See
  example_gaussian_process_plot_gp_regression.py or example_gaussian_
  process_plot_gp_probabilistic_classification_after_regression.py for
  a taste of what can be done.

* It is now possible to use liblinear's Multi-class SVC (option
  multi_class in "LinearSVC")

* New features and performance improvements of text feature
  extraction.

* Improved sparse matrix support, both in main classes
  ("GridSearchCV") as in modules sklearn.svm.sparse and
  sklearn.linear_model.sparse.

* Lots of cool new examples and a new section that uses real-world
  datasets was created. These include:
  *sphx_glr_auto_examples_applications_plot_face_recognition.py*, *sp
  hx_glr_auto_examples_applications_plot_species_distribution_modelin
  g.py*, *sphx_glr_auto_examples_applications_svm_gui.py*, *sphx_glr_
  auto_examples_applications_wikipedia_principal_eigenvector.py* and
  others.

* Faster Least Angle Regression algorithm. It is now 2x faster than
  the R version on worst case and up to 10x times faster on some
  cases.

* Faster coordinate descent algorithm. In particular, the full path
  version of lasso ("linear_model.lasso_path()") is more than 200x
  times faster than before.

* It is now possible to get probability estimates from a
  "LogisticRegression" model.

version of lasso ("linear_model.lasso_path()") is more than 200x

* module renaming: the glm module has been renamed to linear_model,
  the gmm module has been included into the more general mixture model
  and the sgd module has been included in linear_model.

* Lots of bug fixes and documentation improvements.

People
======

   * 207  Olivier Grisel

   * 167 Fabian Pedregosa

   * 97 Peter Prettenhofer

   * 68 Alexandre Gramfort

   * 59  Mathieu Blondel

   * 55  Gael Varoquaux

   * 33  Vincent Dubourg

   * 21  Ron Weiss

   * 9  Bertrand Thirion

   * 3  Alexandre Passos

   * 3  Anne-Laure Fouque

   * 2  Ronan Amicel

   * 1 Christian Osendorfer

Version 0.5

**October 11, 2010**

Changelog
=========

New classes

* Support for sparse matrices in some classifiers of modules "svm" and
  "linear_model" (see "SVC", "SVR", "LinearSVC", "Lasso",
  "ElasticNet")

* New "Pipeline" object to compose different estimators.

* Recursive Feature Elimination routines in module Feature selection.

* Addition of various classes capable of cross validation in the
  linear_model module ("LassoCV", "ElasticNetCV", etc.).

* New, more efficient LARS algorithm implementation. The Lasso variant
  of the algorithm is also implemented. See "lars_path", "Lars" and
  "LassoLars".

* New Hidden Markov Models module (see classes "GaussianHMM",

of the algorithm is also implemented. See "lars_path", "Lars" and

  "MultinomialHMM", "GMMHMM")

* New module feature_extraction (see class reference)

* New FastICA algorithm in module sklearn.fastica

Documentation
=============

* Improved documentation for many modules, now separating narrative
  documentation from the class reference. As an example, see
  documentation for the SVM module and the complete class reference.

Fixes
=====

* API changes: adhere variable names to PEP-8, give more meaningful
  names.

* Fixes for svm module to run on a shared memory context
  (multiprocessing).

* It is again possible to generate latex (and thus PDF) from the
  sphinx docs.

Examples
========

* new examples using some of the mlcomp datasets:
  "sphx_glr_auto_examples_mlcomp_sparse_document_classification.py"
  (since removed) and *sphx_glr_auto_examples_text_plot_document_clas
  sification_20newsgroups.py*

* Many more examples. See here the full list of examples.

External dependencies

* Joblib is now a dependency of this package, although it is shipped
  with (sklearn.externals.joblib).

Removed modules

* Module ann (Artificial Neural Networks) has been removed from the
  distribution. Users wanting this sort of algorithms should take a
  look into pybrain.

Misc
====

* New sphinx theme for the web page.

Authors
=======

The following is a list of authors for this release, preceded by
number of commits:

look into pybrain.

Misc
====

Authors
=======

   * 262  Fabian Pedregosa

   * 240  Gael Varoquaux

   * 149  Alexandre Gramfort

   * 116  Olivier Grisel

   * 40  Vincent Michel

   * 38  Ron Weiss

   * 23  Matthieu Perrot

   * 10  Bertrand Thirion

   * 7  Yaroslav Halchenko

   * 9  VirgileFritsch

   * 6  Edouard Duchesnay

   * 4  Mathieu Blondel

   * 1  Ariel Rokem

   * 1  Matthieu Brucher

Version 0.4

**August 26, 2010**

Changelog
=========

Major changes in this release include:

* Coordinate Descent algorithm (Lasso, ElasticNet) refactoring & speed
  improvements (roughly 100x times faster).

* Coordinate Descent Refactoring (and bug fixing) for consistency with
  R's package GLMNET.

* New metrics module.

* New GMM module contributed by Ron Weiss.

* Implementation of the LARS algorithm (without Lasso variant for
  now).

* feature_selection module redesign.

* Migration to GIT as version control system.

* Removal of obsolete attrselect module.

* Rename of private compiled extensions (added underscore).

* Removal of legacy unmaintained code.

* Documentation improvements (both docstring and rst).

* Improvement of the build system to (optionally) link with MKL. Also,
  provide a lite BLAS implementation in case no system-wide BLAS is
  found.

* Lots of new examples.

* Many, many bug fixes ...

Authors
=======

The committer list for this release is the following (preceded by
number of commits):

found.

Authors
=======

   * 143  Fabian Pedregosa

   * 35  Alexandre Gramfort

   * 34  Olivier Grisel

   * 11  Gael Varoquaux

   * 5  Yaroslav Halchenko

   * 2  Vincent Michel

   * 1  Chris Filo Gorgolewski

Earlier versions

Earlier versions included contributions by Fred Mailhot, David Cooke,
David Huard, Dave Morrill, Ed Schofield, Travis Oliphant, Pearu
Peterson.

Version 0.16.1

**April 14, 2015**

Changelog
=========

Bug fixes
---------

* Allow input data larger than "block_size" in "covariance.LedoitWolf"

* Fix a bug in "isotonic.IsotonicRegression" deduplication that caused
  unstable result in "calibration.CalibratedClassifierCV" by Jan
  Hendrik Metzen.

* Fix sorting of labels in func:*preprocessing.label_binarize* by
  Michael Heilman.

* Fix several stability and convergence issues in
  "cross_decomposition.CCA" and "cross_decomposition.PLSCanonical" by
  Andreas Müller

* Fix a bug in "cluster.KMeans" when "precompute_distances=False" on
  fortran-ordered data.

* Fix a speed regression in "ensemble.RandomForestClassifier"'s
  "predict" and "predict_proba" by Andreas Müller.

* Fix a regression where "utils.shuffle" converted lists and
  dataframes to arrays, by Olivier Grisel

Version 0.16

**March 26, 2015**

* Speed improvements (notably in "cluster.DBSCAN"), reduced memory
  requirements, bug-fixes and better default settings.

* Multinomial Logistic regression and a path algorithm in
  "linear_model.LogisticRegressionCV".

* Out-of core learning of PCA via "decomposition.IncrementalPCA".

* Probability calibration of classifiers using
  "calibration.CalibratedClassifierCV".

* "cluster.Birch" clustering method for large-scale datasets.

* Scalable approximate nearest neighbors search with Locality-
  sensitive hashing forests in "neighbors.LSHForest".

* Improved error messages and better validation when using malformed
  input data.

* More robust integration with pandas dataframes.

Changelog
=========

* The new "neighbors.LSHForest" implements locality-sensitive hashing
  for approximate nearest neighbors search. By >>:user:`Maheshakya
  Wijewardena<maheshakya>`<<.

* Added "svm.LinearSVR". This class uses the liblinear implementation
  of Support Vector Regression which is much faster for large sample
  sizes than "svm.SVR" with linear kernel. By Fabian Pedregosa and
  Qiang Luo.

* Incremental fit for "GaussianNB".

* Added "sample_weight" support to "dummy.DummyClassifier" and
  "dummy.DummyRegressor". By Arnaud Joly.

* Added the "metrics.label_ranking_average_precision_score()" metrics.

* Add the "metrics.coverage_error()" metrics. By Arnaud Joly.

* Added "linear_model.LogisticRegressionCV". By Manoj Kumar, Fabian
  Pedregosa, Gael Varoquaux and Alexandre Gramfort.

* Added "warm_start" constructor parameter to make it possible for any
  trained forest model to grow additional trees incrementally. By
  >>:user:`Laurent Direr<ldirer>`<<.

* Added "sample_weight" support to
  "ensemble.GradientBoostingRegressor". By Peter Prettenhofer.

>>:user:`Laurent Direr<ldirer>`<<.

* Added "decomposition.IncrementalPCA", an implementation of the PCA
  algorithm that supports out-of-core learning with a "partial_fit"
  method. By Kyle Kastner.

* Averaged SGD for "SGDClassifier" and "SGDRegressor" By
  >>:user:`Danny Sullivan <dsullivan7>`<<.

* Added "cross_val_predict" function which computes cross-validated
  estimates. By Luis Pedro Coelho

* Added "linear_model.TheilSenRegressor", a robust generalized-median-
  based estimator. By >>:user:`Florian Wilhelm <FlorianWilhelm>`<<.

* Added "metrics.median_absolute_error()", a robust metric. By Gael
  Varoquaux and >>:user:`Florian Wilhelm <FlorianWilhelm>`<<.

* Add "cluster.Birch", an online clustering algorithm. By Manoj Kumar,
  Alexandre Gramfort and Joel Nothman.

* Added shrinkage support to
  "discriminant_analysis.LinearDiscriminantAnalysis" using two new
  solvers. By >>:user:`Clemens Brunner <cle1109>`<< and Martin
  Billinger.

* Added "kernel_ridge.KernelRidge", an implementation of kernelized
  ridge regression. By Mathieu Blondel and Jan Hendrik Metzen.

* All solvers in "linear_model.Ridge" now support *sample_weight*. By

* Added "cross_validation.PredefinedSplit" cross-validation for fixed
  user-provided cross-validation folds. By >>:user:`Thomas Unterthiner
  <untom>`<<.

Mathieu Blondel.

* Added "calibration.CalibratedClassifierCV", an approach for
  calibrating the predicted probabilities of a classifier. By
  Alexandre Gramfort, Jan Hendrik Metzen, Mathieu Blondel and
  >>:user:`Balazs Kegl <kegl>`<<.

* Add option "return_distance" in "hierarchical.ward_tree()" to return
  distances between nodes for both structured and unstructured
  versions of the algorithm. By Matteo Visconti di Oleggio Castello.
  The same option was added in "hierarchical.linkage_tree()". By Manoj
  Kumar

* Add support for sample weights in scorer objects.  Metrics with
  sample weight support will automatically benefit from it. By Noel
  Dawe and Vlad Niculae.

* Added "newton-cg" and *lbfgs* solver support in
  "linear_model.LogisticRegression". By Manoj Kumar.

* Add "selection="random"" parameter to implement stochastic
  coordinate descent for "linear_model.Lasso",
  "linear_model.ElasticNet" and related. By Manoj Kumar.

* Add "sample_weight" parameter to
  "metrics.jaccard_similarity_score()" and "metrics.log_loss()". By
  >>:user:`Jatin Shah <jatinshah>`<<.

* Support sparse multilabel indicator representation in
  "preprocessing.LabelBinarizer" and "multiclass.OneVsRestClassifier"
  (by >>:user:`Hamzeh Alsalhi <hamsal>`<< with thanks to Rohit
  Sivaprasad), as well as evaluation metrics (by Joel Nothman).

"preprocessing.LabelBinarizer" and "multiclass.OneVsRestClassifier"

* Add "sample_weight" parameter to *metrics.jaccard_similarity_score*.
  By *Jatin Shah*.

* Add support for multiclass in *metrics.hinge_loss*. Added
  "labels=None" as optional parameter. By *Saurabh Jha*.

* Add "sample_weight" parameter to *metrics.hinge_loss*. By *Saurabh
  Jha*.

* Add "multi_class="multinomial"" option in
  "linear_model.LogisticRegression" to implement a Logistic Regression
  solver that minimizes the cross-entropy or multinomial loss instead
  of the default One-vs-Rest setting. Supports *lbfgs* and *newton-cg*
  solvers. By Lars Buitinck and Manoj Kumar. Solver option *newton-cg*
  by Simon Wu.

* "DictVectorizer" can now perform "fit_transform" on an iterable in a
  single pass, when giving the option "sort=False". By >>:user:`Dan
  Blanchard <dan-blanchard>`<<.

* "GridSearchCV" and "RandomizedSearchCV" can now be configured to
  work with estimators that may fail and raise errors on individual
  folds. This option is controlled by the *error_score* parameter.
  This does not affect errors raised on re-fit. By >>:user:`Michal
  Romaniuk <romaniukm>`<<.

* Add "digits" parameter to *metrics.classification_report* to allow
  report to show different precision of floating point numbers. By
  >>:user:`Ian Gilmore <agileminor>`<<.

Romaniuk <romaniukm>`<<.

* Add a quantile prediction strategy to the "dummy.DummyRegressor". By
  >>:user:`Aaron Staple <staple>`<<.

* Add "handle_unknown" option to "preprocessing.OneHotEncoder" to
  handle unknown categorical features more gracefully during
  transform. By Manoj Kumar.

* Added support for sparse input data to decision trees and their
  ensembles. By Fares Hedyati and Arnaud Joly.

* Optimized "cluster.AffinityPropagation" by reducing the number of
  memory allocations of large temporary data-structures. By Antony
  Lee.

* Parellization of the computation of feature importances in random
  forest. By Olivier Grisel and Arnaud Joly.

* Add "n_iter_" attribute to estimators that accept a "max_iter"
  attribute in their constructor. By Manoj Kumar.

* Added decision function for "multiclass.OneVsOneClassifier" By
  Raghav RV and >>:user:`Kyle Beauchamp <kyleabeauchamp>`<<.

* "neighbors.kneighbors_graph()" and "radius_neighbors_graph()"
  support non-Euclidean metrics. By Manoj Kumar

* Parameter "connectivity" in "cluster.AgglomerativeClustering" and
  family now accept callables that return a connectivity matrix. By
  Manoj Kumar.

* Sparse support for "paired_distances()". By Joel Nothman.

* "cluster.DBSCAN" now supports sparse input and sample weights and

Manoj Kumar.

  has been optimized: the inner loop has been rewritten in Cython and
  radius neighbors queries are now computed in batch. By Joel Nothman
  and Lars Buitinck.

  class frequency for "ensemble.RandomForestClassifier",
  "tree.DecisionTreeClassifier", "ensemble.ExtraTreesClassifier" and
  "tree.ExtraTreeClassifier". By Trevor Stephens.

* "grid_search.RandomizedSearchCV" now does sampling without
  replacement if all parameters are given as lists. By Andreas Müller.

* Parallelized calculation of "pairwise_distances()" is now supported
  for scipy metrics and custom callables. By Joel Nothman.

* Allow the fitting and scoring of all clustering algorithms in
  "pipeline.Pipeline". By Andreas Müller.

* More robust seeding and improved error messages in
  "cluster.MeanShift" by Andreas Müller.

* Make the stopping criterion for "mixture.GMM", "mixture.DPGMM" and
  "mixture.VBGMM" less dependent on the number of samples by
  thresholding the average log-likelihood change instead of its sum
  over all samples. By Hervé Bredin.

* The outcome of "manifold.spectral_embedding()" was made
  deterministic by flipping the sign of eigenvectors. By
  >>:user:`Hasil Sharma <Hasil-Sharma>`<<.

* Significant performance and memory usage improvements in
  "preprocessing.PolynomialFeatures". By Eric Martin.

>>:user:`Hasil Sharma <Hasil-Sharma>`<<.

* Numerical stability improvements for "preprocessing.StandardScaler"
  and "preprocessing.scale()". By Nicolas Goix

* "svm.SVC" fitted on sparse input now implements "decision_function".
  By Rob Zinkov and Andreas Müller.

* "cross_validation.train_test_split()" now preserves the input type,
  instead of converting to numpy arrays.

* Added example of using "FeatureUnion" for heterogeneous input. By
  >>:user:`Matt Terry <mrterry>`<<

* Documentation on scorers was improved, to highlight the handling of
  loss functions. By >>:user:`Matt Pico <MattpSoftware>`<<.

* A discrepancy between liblinear output and scikit-learn's wrappers
  is now noted. By Manoj Kumar.

* Improved documentation generation: examples referring to a class or
  function are now shown in a gallery on the class/function's API
  reference page. By Joel Nothman.

* More explicit documentation of sample generators and of data
  transformation. By Joel Nothman.

* "sklearn.neighbors.BallTree" and "sklearn.neighbors.KDTree" used to
  point to empty pages stating that they are aliases of BinaryTree.
  This has been fixed to show the correct class docs. By Manoj Kumar.

* Added silhouette plots for analysis of KMeans clustering using
  "metrics.silhouette_samples()" and "metrics.silhouette_score()". See

  *sphx_glr_auto_examples_cluster_plot_kmeans_silhouette_analysis.py*

Bug fixes
---------

* Metaestimators now support ducktyping for the presence of
  "decision_function", "predict_proba" and other methods. This fixes
  behavior of "grid_search.GridSearchCV",
  "grid_search.RandomizedSearchCV", "pipeline.Pipeline",
  "feature_selection.RFE", "feature_selection.RFECV" when nested. By
  Joel Nothman

* The "scoring" attribute of grid-search and cross-validation methods
  is no longer ignored when a "grid_search.GridSearchCV" is given as a
  base estimator or the base estimator doesn't have predict.

* The function "hierarchical.ward_tree()" now returns the children in
  the same order for both the structured and unstructured versions. By
  Matteo Visconti di Oleggio Castello.

* "feature_selection.RFECV" now correctly handles cases when "step" is
  not equal to 1. By >>:user:`Nikolay Mayorov <nmayorov>`<<

* The "decomposition.PCA" now undoes whitening in its
  "inverse_transform". Also, its "components_" now always have unit
  length. By >>:user:`Michael Eickenberg <eickenberg>`<<.

* Fix incomplete download of the dataset when
  "datasets.download_20newsgroups()" is called. By Manoj Kumar.

* Various fixes to the Gaussian processes subpackage by Vincent
  Dubourg and Jan Hendrik Metzen.

* Calling "partial_fit" with "class_weight=='auto'" throws an

  appropriate error message and suggests a work around. By

* "RBFSampler" with "gamma=g" formerly approximated "rbf_kernel" with
  "gamma=g/2."; the definition of "gamma" is now consistent, which may
  substantially change your results if you use a fixed value. (If you
  cross-validated over "gamma", it probably doesn't matter too much.)
  By >>:user:`Dougal Sutherland <dougalsutherland>`<<.

* Pipeline object delegate the "classes_" attribute to the underlying
  estimator. It allows, for instance, to make bagging of a pipeline
  object. By Arnaud Joly

* "neighbors.NearestCentroid" now uses the median as the centroid when
  metric is set to "manhattan". It was using the mean before. By Manoj
  Kumar

* Fix numerical stability issues in "linear_model.SGDClassifier" and
  "linear_model.SGDRegressor" by clipping large gradients and ensuring
  that weight decay rescaling is always positive (for large l2
  regularization and large learning rate values). By Olivier Grisel

* When *compute_full_tree* is set to "auto", the full tree is built
  when n_clusters is high and is early stopped when n_clusters is low,
  while the behavior should be vice-versa in
  "cluster.AgglomerativeClustering" (and friends). This has been fixed
  By Manoj Kumar

* Fix lazy centering of data in "linear_model.enet_path()" and

while the behavior should be vice-versa in

  "linear_model.lasso_path()". It was centered around one. It has been
  changed to be centered around the origin. By Manoj Kumar

* Fix handling of precomputed affinity matrices in
  "cluster.AgglomerativeClustering" when using connectivity
  constraints. By >>:user:`Cathy Deng <cathydeng>`<<

* Correct "partial_fit" handling of "class_prior" for
  "sklearn.naive_bayes.BernoulliNB". By Trevor Stephens.

* Fixed a crash in "metrics.precision_recall_fscore_support()" when
  using unsorted "labels" in the multi-label setting. By Andreas
  Müller.

* Avoid skipping the first nearest neighbor in the methods
  "radius_neighbors", "kneighbors", "kneighbors_graph" and
  "radius_neighbors_graph" in "sklearn.neighbors.NearestNeighbors" and
  family, when the query data is not the same as fit data. By Manoj
  Kumar.

* Fix log-density calculation in the "mixture.GMM" with tied
  covariance. By Will Dawson

* Fixed a scaling error in "feature_selection.SelectFdr" where a
  factor "n_features" was missing. By Andrew Tulloch

* Fix zero division in "neighbors.KNeighborsRegressor" and related
  classes when using distance weighting and having identical data
  points. By Garret-R.

* Fixed round off errors with non positive-definite covariance

classes when using distance weighting and having identical data

  matrices in GMM. By >>:user:`Alexis Mignon <AlexisMignon>`<<.

* Fixed a error in the computation of conditional probabilities in
  "naive_bayes.BernoulliNB". By Hanna Wallach.

* Make the method "radius_neighbors" of "neighbors.NearestNeighbors"
  return the samples lying on the boundary for "algorithm='brute'". By
  Yan Yi.

* Flip sign of "dual_coef_" of "svm.SVC" to make it consistent with
  the documentation and "decision_function". By Artem Sobolev.

* Fixed handling of ties in "isotonic.IsotonicRegression". We now use
  the weighted average of targets (secondary method). By Andreas
  Müller and Michael Bommarito.

* "GridSearchCV" and "cross_val_score" and other meta-estimators don't
  convert pandas DataFrames into arrays any more, allowing DataFrame
  specific operations in custom estimators.

* "multiclass.fit_ovr()", "multiclass.predict_ovr()",
  "predict_proba_ovr()", "multiclass.fit_ovo()",
  "multiclass.predict_ovo()", "multiclass.fit_ecoc()" and
  "multiclass.predict_ecoc()" are deprecated. Use the underlying
  estimators instead.

* Nearest neighbors estimators used to take arbitrary keyword
  arguments and pass these to their distance metric. This will no
  longer be supported in scikit-learn 0.18; use the "metric_params"
  argument instead.

arguments and pass these to their distance metric. This will no

* *n_jobs* parameter of the fit method shifted to the constructor of
  the
     LinearRegression class.

* The "predict_proba" method of "multiclass.OneVsRestClassifier" now
  returns two probabilities per sample in the multiclass case; this is
  consistent with other estimators and with the method's
  documentation, but previous versions accidentally returned only the
  positive probability. Fixed by Will Lamond and Lars Buitinck.

* Change default value of precompute in "ElasticNet" and "Lasso" to
  False. Setting precompute to "auto" was found to be slower when
  n_samples > n_features since the computation of the Gram matrix is
  computationally expensive and outweighs the benefit of fitting the
  Gram for just one alpha. "precompute="auto"" is now deprecated and
  will be removed in 0.18 By Manoj Kumar.

* Expose "positive" option in "linear_model.enet_path()" and
  "linear_model.enet_path()" which constrains coefficients to be
  positive. By Manoj Kumar.

* Users should now supply an explicit "average" parameter to
  "sklearn.metrics.f1_score()", "sklearn.metrics.fbeta_score()",
  "sklearn.metrics.recall_score()" and
  "sklearn.metrics.precision_score()" when performing multiclass or
  multilabel (i.e. not binary) classification. By Joel Nothman.

* *scoring* parameter for cross validation now accepts *'f1_micro'*,

"sklearn.metrics.precision_score()" when performing multiclass or

  *'f1_macro'* or *'f1_weighted'*. *'f1'* is now for binary
  classification only. Similar changes apply to *'precision'* and
  *'recall'*. By Joel Nothman.

* The "fit_intercept", "normalize" and "return_models" parameters in
  "linear_model.enet_path()" and "linear_model.lasso_path()" have been
  removed. They were deprecated since 0.14

* From now onwards, all estimators will uniformly raise
  "NotFittedError" ("utils.validation.NotFittedError"), when any of
  the "predict" like methods are called before the model is fit. By
  Raghav RV.

* Input data validation was refactored for more consistent input
  validation. The "check_arrays" function was replaced by
  "check_array" and "check_X_y". By Andreas Müller.

* Allow "X=None" in the methods "radius_neighbors", "kneighbors",
  "kneighbors_graph" and "radius_neighbors_graph" in
  "sklearn.neighbors.NearestNeighbors" and family. If set to None,
  then for every sample this avoids setting the sample itself as the
  first nearest neighbor. By Manoj Kumar.

* Add parameter "include_self" in "neighbors.kneighbors_graph()" and
  "neighbors.radius_neighbors_graph()" which has to be explicitly set
  by the user. If set to True, then the sample itself is considered as
  the first nearest neighbor.

"neighbors.radius_neighbors_graph()" which has to be explicitly set

* *thresh* parameter is deprecated in favor of new *tol* parameter in
  "GMM", "DPGMM" and "VBGMM". See *Enhancements* section for details.
  By Hervé Bredin.

* Estimators will treat input with dtype object as numeric when
  possible. By Andreas Müller

* Estimators now raise *ValueError* consistently when fitted on empty
  data (less than 1 sample or less than 1 feature for 2D input). By

* The "shuffle" option of "linear_model.SGDClassifier",
  "linear_model.SGDRegressor", "linear_model.Perceptron",
  "linear_model.PassiveAggressiveClassifier" and
  "linear_model.PassiveAggressiveRegressor" now defaults to "True".

* "cluster.DBSCAN" now uses a deterministic initialization. The
  *random_state* parameter is deprecated. By >>:user:`Erich Schubert
  <kno10>`<<.

A. Flaxman, Aaron Schumacher, Aaron Staple, abhishek thakur, Akshay,
akshayah3, Aldrian Obaja, Alexander Fabisch, Alexandre Gramfort,
Alexis Mignon, Anders Aagaard, Andreas Mueller, Andreas van
Cranenburgh, Andrew Tulloch, Andrew Walker, Antony Lee, Arnaud Joly,
banilo, Barmaley.exe, Ben Davies, Benedikt Koehler, bhsu, Boris Feld,
Borja Ayerdi, Boyuan Deng, Brent Pedersen, Brian Wignall, Brooke
Osborn, Calvin Giles, Cathy Deng, Celeo, cgohlke, chebee7i, Christian

Stade-Schuldt, Christof Angermueller, Chyi-Kwei Yau, CJ Carey, Clemens
Brunner, Daiki Aminaka, Dan Blanchard, danfrankj, Danny Sullivan,
David Fletcher, Dmitrijs Milajevs, Dougal J. Sutherland, Erich
Schubert, Fabian Pedregosa, Florian Wilhelm, floydsoft, Félix-Antoine
Fortin, Gael Varoquaux, Garrett-R, Gilles Louppe, gpassino, gwulfs,
Hampus Bengtsson, Hamzeh Alsalhi, Hanna Wallach, Harry Mavroforakis,
Hasil Sharma, Helder, Herve Bredin, Hsiang-Fu Yu, Hugues SALAMIN, Ian
Gilmore, Ilambharathi Kanniah, Imran Haque, isms, Jake VanderPlas, Jan
Dlabal, Jan Hendrik Metzen, Jatin Shah, Javier López Peña,
jdcaballero, Jean Kossaifi, Jeff Hammerbacher, Joel Nothman, Jonathan
Helmus, Joseph, Kaicheng Zhang, Kevin Markham, Kyle Beauchamp, Kyle
Kastner, Lagacherie Matthieu, Lars Buitinck, Laurent Direr, leepei,
Loic Esteve, Luis Pedro Coelho, Lukas Michelbacher, maheshakya, Manoj
Kumar, Manuel, Mario Michael Krell, Martin, Martin Billinger, Martin
Ku, Mateusz Susik, Mathieu Blondel, Matt Pico, Matt Terry, Matteo
Visconti dOC, Matti Lyra, Max Linke, Mehdi Cherti, Michael Bommarito,
Michael Eickenberg, Michal Romaniuk, MLG, mr.Shu, Nelle Varoquaux,
Nicola Montecchio, Nicolas, Nikolay Mayorov, Noel Dawe, Okal Billy,
Olivier Grisel, Óscar Nájera, Paolo Puggioni, Peter Prettenhofer,
Pratap Vardhan, pvnguyen, queqichao, Rafael Carrascosa, Raghav R V,

Rahiel Kasim, Randall Mason, Rob Zinkov, Robert Bradshaw, Saket
Choudhary, Sam Nicholls, Samuel Charron, Saurabh Jha, sethdandridge,
sinhrks, snuderl, Stefan Otte, Stefan van der Walt, Steve Tjoa, swu,
Sylvain Zimmer, tejesh95, terrycojones, Thomas Delteil, Thomas
Unterthiner, Tomas Kazmar, trevorstephens, tttthomasssss, Tzu-Ming
Kuo, ugurcaliskan, ugurthemaster, Vinayak Mehta, Vincent Dubourg,
Vjacheslav Murashkin, Vlad Niculae, wadawson, Wei Xue, Will Lamond, Wu
Jiang, x0l, Xinfan Meng, Yan Yi, Yu-Chin

Version 0.13.1

**February 23, 2013**

The 0.13.1 release only fixes some bugs and does not add any new
functionality.

Changelog
=========

* Fixed a testing error caused by the function
  "cross_validation.train_test_split()" being interpreted as a test by
  Yaroslav Halchenko.

* Fixed a bug in the reassignment of small clusters in the
  "cluster.MiniBatchKMeans" by Gael Varoquaux.

* Fixed default value of "gamma" in "decomposition.KernelPCA" by Lars
  Buitinck.

* Updated joblib to "0.7.0d" by Gael Varoquaux.

* Fixed scaling of the deviance in
  "ensemble.GradientBoostingClassifier" by Peter Prettenhofer.

* Better tie-breaking in "multiclass.OneVsOneClassifier" by Andreas
  Müller.

* Other small improvements to tests and documentation.

People
======

List of contributors for release 0.13.1 by number of commits.
   * 16  Lars Buitinck

   * 12  Andreas Müller

   * 8  Gael Varoquaux

   * 5  Robert Marchman

   * 3  Peter Prettenhofer

   * 2  Hrishikesh Huilgolkar

   * 1  Diego Molla

   * 1  Vlad Niculae

Version 0.13

**January 21, 2013**

New Estimator Classes

* "dummy.DummyClassifier" and "dummy.DummyRegressor", two data-
  independent predictors by Mathieu Blondel. Useful to sanity-check
  your estimators. See Dummy estimators in the user guide. Multioutput

  support added by Arnaud Joly.

* "decomposition.FactorAnalysis", a transformer implementing the
  classical factor analysis, by Christian Osendorfer and Alexandre
  Gramfort. See Factor Analysis in the user guide.

* "feature_extraction.FeatureHasher", a transformer implementing the
  "hashing trick" for fast, low-memory feature extraction from string
  fields by Lars Buitinck and
  "feature_extraction.text.HashingVectorizer" for text documents by
  Olivier Grisel  See Feature hashing and Vectorizing a large text
  corpus with the hashing trick for the documentation and sample
  usage.

* "pipeline.FeatureUnion", a transformer that concatenates results of
  several other transformers by Andreas Müller. See FeatureUnion:
  composite feature spaces in the user guide.

* "random_projection.GaussianRandomProjection",
  "random_projection.SparseRandomProjection" and the function
  "random_projection.johnson_lindenstrauss_min_dim()". The first two
  are transformers implementing Gaussian and sparse random projection
  matrix by Olivier Grisel and Arnaud Joly. See Random Projection in
  the user guide.

* "kernel_approximation.Nystroem", a transformer for approximating
  arbitrary kernels by Andreas Müller. See Nystroem Method for Kernel
  Approximation in the user guide.

the user guide.

* "preprocessing.OneHotEncoder", a transformer that computes binary
  encodings of categorical features by Andreas Müller. See Encoding
  categorical features in the user guide.

* "linear_model.PassiveAggressiveClassifier" and
  "linear_model.PassiveAggressiveRegressor", predictors implementing
  an efficient stochastic optimization for linear models by Rob Zinkov
  and Mathieu Blondel. See Passive Aggressive Algorithms in the user
  guide.

* "ensemble.RandomTreesEmbedding", a transformer for creating high-
  dimensional sparse representations using ensembles of totally random
  trees by  Andreas Müller. See Totally Random Trees Embedding in the
  user guide.

* "manifold.SpectralEmbedding" and function
  "manifold.spectral_embedding()", implementing the "laplacian
  eigenmaps" transformation for non-linear dimensionality reduction by
  Wei Li. See Spectral Embedding in the user guide.

* "isotonic.IsotonicRegression" by Fabian Pedregosa, Alexandre
  Gramfort and Nelle Varoquaux,

Changelog
=========

* "metrics.zero_one_loss()" (formerly "metrics.zero_one") now has
  option for normalized output that reports the fraction of
  misclassifications, rather than the raw number of
  misclassifications. By Kyle Beauchamp.

* "tree.DecisionTreeClassifier" and all derived ensemble models now

misclassifications, rather than the raw number of

  support sample weighting, by Noel Dawe  and Gilles Louppe.

* Speedup improvement when using bootstrap samples in forests of
  randomized trees, by Peter Prettenhofer  and Gilles Louppe.

* Partial dependence plots for Gradient Tree Boosting in
  "ensemble.partial_dependence.partial_dependence()" by Peter
  Prettenhofer. See
  *sphx_glr_auto_examples_inspection_plot_partial_dependence.py* for
  an example.

* The table of contents on the website has now been made expandable by
  Jaques Grobler.

* "feature_selection.SelectPercentile" now breaks ties
  deterministically instead of returning all equally ranked features.

  "feature_selection.SelectPercentile" are more numerically stable
  since they use scores, rather than p-values, to rank results. This
  means that they might sometimes select different features than they
  did previously.

* Ridge regression and ridge classification fitting with "sparse_cg"
  solver no longer has quadratic memory complexity, by Lars Buitinck
  and Fabian Pedregosa.

* Ridge regression and ridge classification now support a new fast
  solver called "lsqr", by Mathieu Blondel.

* Speed up of "metrics.precision_recall_curve()" by Conrad Lee.

* Added support for reading/writing svmlight files with pairwise
  preference attribute (qid in svmlight file format) in

  "datasets.dump_svmlight_file()" and "datasets.load_svmlight_file()"
  by Fabian Pedregosa.

* Faster and more robust "metrics.confusion_matrix()" and Clustering
  performance evaluation by Wei Li.

* "cross_validation.cross_val_score()" now works with precomputed
  kernels and affinity matrices, by Andreas Müller.

* LARS algorithm made more numerically stable with heuristics to drop
  regressors too correlated as well as to stop the path when numerical
  noise becomes predominant, by Gael Varoquaux.

* Faster implementation of "metrics.precision_recall_curve()" by
  Conrad Lee.

* New kernel "metrics.chi2_kernel" by Andreas Müller, often used in
  computer vision applications.

* Fix of longstanding bug in "naive_bayes.BernoulliNB" fixed by Shaun
  Jackman.

* Implemented "predict_proba" in "multiclass.OneVsRestClassifier", by
  Andrew Winterman.

* Improve consistency in gradient boosting: estimators
  "ensemble.GradientBoostingClassifier" use the estimator
  "tree.DecisionTreeRegressor" instead of the "tree._tree.Tree" data
  structure by Arnaud Joly.

* Fixed a floating point exception in the decision trees module, by
  Seberg.

* Fix "metrics.roc_curve()" fails when y_true has only one class by
  Wei Li.

* Add the "metrics.mean_absolute_error()" function which computes the

Seberg.

  Wei Li.

  mean absolute error. The "metrics.mean_squared_error()",
  "metrics.mean_absolute_error()" and "metrics.r2_score()" metrics
  support multioutput by Arnaud Joly.

* Fixed "class_weight" support in "svm.LinearSVC" and
  "linear_model.LogisticRegression" by Andreas Müller. The meaning of
  "class_weight" was reversed as erroneously higher weight meant less
  positives of a given class in earlier releases.

* Improve narrative documentation and consistency in "sklearn.metrics"
  for regression and classification metrics by Arnaud Joly.

* Fixed a bug in "sklearn.svm.SVC" when using csr-matrices with
  unsorted indices by Xinfan Meng and Andreas Müller.

* "MiniBatchKMeans": Add random reassignment of cluster centers with
  little observations attached to them, by Gael Varoquaux.

* Renamed all occurrences of "n_atoms" to "n_components" for
  consistency. This applies to "decomposition.DictionaryLearning",
  "decomposition.dict_learning()",
  "decomposition.dict_learning_online()".

* Renamed all occurrences of "max_iters" to "max_iter" for
  consistency. This applies to "semi_supervised.LabelPropagation" and
  "semi_supervised.label_propagation.LabelSpreading".

* Renamed all occurrences of "learn_rate" to "learning_rate" for

consistency. This applies to "semi_supervised.LabelPropagation" and

  consistency in "ensemble.BaseGradientBoosting" and
  "ensemble.GradientBoostingRegressor".

* The module "sklearn.linear_model.sparse" is gone. Sparse matrix
  support was already integrated into the "regular" linear models.

* "sklearn.metrics.mean_square_error()", which incorrectly returned
  the accumulated error, was removed. Use "mean_squared_error"
  instead.

* Passing "class_weight" parameters to "fit" methods is no longer
  supported. Pass them to estimator constructors instead.

* GMMs no longer have "decode" and "rvs" methods. Use the "score",
  "predict" or "sample" methods instead.

* The "solver" fit option in Ridge regression and classification is
  now deprecated and will be removed in v0.14. Use the constructor
  option instead.

* "feature_extraction.text.DictVectorizer" now returns sparse matrices
  in the CSR format, instead of COO.

* Renamed "k" in "cross_validation.KFold" and
  "cross_validation.StratifiedKFold" to "n_folds", renamed
  "n_bootstraps" to "n_iter" in "cross_validation.Bootstrap".

* Renamed all occurrences of "n_iterations" to "n_iter" for
  consistency. This applies to "cross_validation.ShuffleSplit",
  "cross_validation.StratifiedShuffleSplit",
  "utils.randomized_range_finder()" and "utils.randomized_svd()".

* Replaced "rho" in "linear_model.ElasticNet" and

"cross_validation.StratifiedShuffleSplit",

  "linear_model.SGDClassifier" by "l1_ratio". The "rho" parameter had
  different meanings; "l1_ratio" was introduced to avoid confusion. It
  has the same meaning as previously "rho" in
  "linear_model.ElasticNet" and "(1-rho)" in
  "linear_model.SGDClassifier".

* "linear_model.LassoLars" and "linear_model.Lars" now store a list of
  paths in the case of multiple targets, rather than an array of
  paths.

* The attribute "gmm" of "hmm.GMMHMM" was renamed to "gmm_" to adhere
  more strictly with the API.

* "cluster.spectral_embedding()" was moved to
  "manifold.spectral_embedding()".

* Renamed "eig_tol" in "manifold.spectral_embedding()",
  "cluster.SpectralClustering" to "eigen_tol", renamed "mode" to
  "eigen_solver".

* Renamed "mode" in "manifold.spectral_embedding()" and
  "cluster.SpectralClustering" to "eigen_solver".

* "classes_" and "n_classes_" attributes of
  "tree.DecisionTreeClassifier" and all derived ensemble models are
  now flat in case of single output problems and nested in case of
  multi-output problems.

* The "estimators_" attribute of
  "ensemble.gradient_boosting.GradientBoostingRegressor" and
  "ensemble.gradient_boosting.GradientBoostingClassifier" is now an
  array of :class:'tree.DecisionTreeRegressor'.

* Renamed "chunk_size" to "batch_size" in

array of :class:'tree.DecisionTreeRegressor'.

  "decomposition.MiniBatchSparsePCA" for consistency.

* "svm.SVC" and "svm.NuSVC" now provide a "classes_" attribute and
  support arbitrary dtypes for labels "y". Also, the dtype returned by
  "predict" now reflects the dtype of "y" during "fit" (used to be
  "np.float").

* Changed default test_size in "cross_validation.train_test_split()"
  to None, added possibility to infer "test_size" from "train_size" in
  "cross_validation.ShuffleSplit" and
  "cross_validation.StratifiedShuffleSplit".

* Renamed function "sklearn.metrics.zero_one()" to
  "sklearn.metrics.zero_one_loss()". Be aware that the default
  behavior in "sklearn.metrics.zero_one_loss()" is different from
  "sklearn.metrics.zero_one()": "normalize=False" is changed to
  "normalize=True".

* Renamed function "metrics.zero_one_score()" to
  "metrics.accuracy_score()".

* "datasets.make_circles()" now has the same number of inner and outer
  points.

* In the Naive Bayes classifiers, the "class_prior" parameter was
  moved from "fit" to "__init__".

People
======

List of contributors for release 0.13 by number of commits.

   * 364  Andreas Müller

   * 143  Arnaud Joly

   * 137  Peter Prettenhofer

   * 131  Gael Varoquaux

   * 117  Mathieu Blondel

   * 108  Lars Buitinck

   * 106  Wei Li

   * 101  Olivier Grisel

   * 65  Vlad Niculae

   * 54  Gilles Louppe

   * 40  Jaques Grobler

* 117  Mathieu Blondel

   * 106  Wei Li

   * 30  Rob Zinkov

   * 19  Aymeric Masurelle

   * 18  Andrew Winterman

   * 17  Fabian Pedregosa

   * 17  Nelle Varoquaux

   * 16  Christian Osendorfer

   * 14  Daniel Nouri

   * 13  >>:user:`Virgile Fritsch <VirgileFritsch>`<<

   * 13  syhw

   * 12  Satrajit Ghosh

   * 10  Corey Lynch

   * 10  Kyle Beauchamp

   * 9  Brian Cheung

   * 9  Immanuel Bayer

   * 9  mr.Shu

   * 8  Conrad Lee

   * 8  James Bergstra

   * 7  Tadej Janež

   * 6  Brian Cajes

   * 6  Jake Vanderplas

   * 6  Michael

   * 6  Noel Dawe

   * 6  Tiago Nunes

   * 6  cow

   * 5  Anze

   * 5  Shiqiao Du

   * 4  Christian Jauvin

   * 4  Jacques Kvam

   * 4  Richard T. Guy

   * 4  Robert Layton

   * 3  Alexandre Abraham

   * 3  Doug Coleman

   * 3  Scott Dickerson

   * 2  ApproximateIdentity

   * 2  John Benediktsson

   * 2  Mark Veronda

   * 2  Matti Lyra

   * 2  Mikhail Korobov

   * 2  Xinfan Meng

   * 1  Alejandro Weinstein

   * 1  Alexandre Passos

   * 1  Christoph Deil

   * 1  Eugene Nizhibitsky

   * 1  Kenneth C. Arnold

   * 1  Luis Pedro Coelho

   * 1  Miroslav Batchkarov

   * 1  Pavel

   * 1  Sebastian Berg

   * 1  Shaun Jackman

   * 1  Subhodeep Moitra

   * 1  bob

   * 1  dengemann

   * 1  emanuele

   * 1  x006

scikit-learn Tutorials

* An introduction to machine learning with scikit-learn

  * Machine learning: the problem setting

  * Loading an example dataset

  * Learning and predicting

  * Conventions

* A tutorial on statistical-learning for scientific data processing

  * Statistical learning: the setting and the estimator object in
    scikit-learn

  * Supervised learning: predicting an output variable from high-
    dimensional observations

  * Model selection: choosing estimators and their parameters

  * Unsupervised learning: seeking representations of the data

  * Putting it all together

* Working With Text Data

  * Tutorial setup

  * Loading the 20 newsgroups dataset

  * Extracting features from text files

  * Training a classifier

  * Building a pipeline

  * Evaluation of the performance on the test set

  * Parameter tuning using grid search

  * Exercise 1: Language identification

  * Exercise 2: Sentiment Analysis on movie reviews

  * Exercise 3: CLI text classification utility

  * Where to from here

* Choosing the right estimator

* External Resources, Videos and Talks

  * New to Scientific Python?

  * External Tutorials

  * Videos

Note:

  **Doctest Mode**The code-examples in the above tutorials are written
  in a *python-console* format. If you wish to easily execute these
  examples in **IPython**, use:

     %doctest_mode

  in the IPython-console. You can then simply copy and paste the
  examples directly into IPython without having to worry about
  removing the **>>>** manually.

An introduction to machine learning with scikit-learn
*****************************************************

Section contents
^^^^^^^^^^^^^^^^

In this section, we introduce the machine learning vocabulary that we
use throughout scikit-learn and give a simple learning example.

Machine learning: the problem setting
=====================================

In general, a learning problem considers a set of n samples of data
and then tries to predict properties of unknown data. If each sample
is more than a single number and, for instance, a multi-dimensional
entry (aka multivariate data), it is said to have several attributes
or **features**.

Learning problems fall into a few categories:

   * supervised learning, in which the data comes with additional
     attributes that we want to predict (Click here to go to the
     scikit-learn supervised learning page).This problem can be
     either:

        * classification: samples belong to two or more classes and we
          want to learn from already labeled data how to predict the
          class of unlabeled data. An example of a classification
          problem would be handwritten digit recognition, in which the
          aim is to assign each input vector to one of a finite number
          of discrete categories.  Another way to think of
          classification is as a discrete (as opposed to continuous)
          form of supervised learning where one has a limited number
          of categories and for each of the n samples provided, one is

form of supervised learning where one has a limited number
          to try to label them with the correct category or class.

        * regression: if the desired output consists of one or more
          continuous variables, then the task is called *regression*.
          An example of a regression problem would be the prediction
          of the length of a salmon as a function of its age and
          weight.

   * unsupervised learning, in which the training data consists of a
     set of input vectors x without any corresponding target values.
     The goal in such problems may be to discover groups of similar
     examples within the data, where it is called clustering, or to
     determine the distribution of data within the input space, known
     as density estimation, or to project the data from a high-
     dimensional space down to two or three dimensions for the purpose
     of *visualization* (Click here to go to the Scikit-Learn
     unsupervised learning page).

Training set and testing set
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Machine learning is about learning some properties of a data set and
then testing those properties against another data set. A common
practice in machine learning is to evaluate an algorithm by splitting
a data set into two. We call one of those sets the **training set**,
on which we learn some properties; we call the other set the **testing
set**, on which we test the learned properties.

Loading an example dataset

*scikit-learn* comes with a few standard datasets, for instance the
iris and digits datasets for classification and the diabetes dataset
for regression.

In the following, we start a Python interpreter from our shell and
then load the "iris" and "digits" datasets.  Our notational convention
is that "$" denotes the shell prompt while ">>>" denotes the Python
interpreter prompt:

   $ python
   >>> iris = datasets.load_iris()
   >>> digits = datasets.load_digits()

A dataset is a dictionary-like object that holds all the data and some
metadata about the data. This data is stored in the ".data" member,
which is a "n_samples, n_features" array. In the case of supervised
problems, one or more response variables are stored in the ".target"
member. More details on the different datasets can be found in the
dedicated section.

For instance, in the case of the digits dataset, "digits.data" gives
access to the features that can be used to classify the digits
samples:

   >>> print(digits.data)
   [[ 0.   0.   5. ...   0.   0.   0.]
    [ 0.   0.   0. ...  10.   0.   0.]
    [ 0.   0.   0. ...  16.   9.   0.]
    ...
    [ 0.   0.   1. ...   6.   0.   0.]
    [ 0.   0.   2. ...  12.   0.   0.]
    [ 0.   0.  10. ...  12.   1.   0.]]

[ 0.   0.   0. ...  10.   0.   0.]
    ...

and "digits.target" gives the ground truth for the digit dataset, that
is the number corresponding to each digit image that we are trying to
learn:

   >>> digits.target
   array([0, 1, 2, ..., 8, 9, 8])

Shape of the data arrays
^^^^^^^^^^^^^^^^^^^^^^^^

The data is always a 2D array, shape "(n_samples, n_features)",
although the original data may have had a different shape. In the case
of the digits, each original sample is an image of shape "(8, 8)" and
can be accessed using:

   >>> digits.images[0]
   array([[  0.,   0.,   5.,  13.,   9.,   1.,   0.,   0.],
          [  0.,   0.,  13.,  15.,  10.,  15.,   5.,   0.],
          [  0.,   3.,  15.,   2.,   0.,  11.,   8.,   0.],
          [  0.,   4.,  12.,   0.,   0.,   8.,   8.,   0.],
          [  0.,   5.,   8.,   0.,   0.,   9.,   8.,   0.],
          [  0.,   4.,  11.,   0.,   1.,  12.,   7.,   0.],
          [  0.,   2.,  14.,   5.,  10.,  12.,   0.,   0.],
          [  0.,   0.,   6.,  13.,  10.,   0.,   0.,   0.]])

The *simple example on this dataset* illustrates how starting from the
original problem one can shape the data for consumption in scikit-
learn.

Loading from external datasets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To load from an external dataset, please refer to loading external
datasets.

Learning and predicting

learn.

datasets.

In the case of the digits dataset, the task is to predict, given an
image, which digit it represents. We are given samples of each of the
10 possible classes (the digits zero through nine) on which we *fit*
an estimator to be able to *predict* the classes to which unseen
samples belong.

In scikit-learn, an estimator for classification is a Python object
that implements the methods "fit(X, y)" and "predict(T)".

An example of an estimator is the class "sklearn.svm.SVC", which
implements support vector classification. The estimator's constructor
takes as arguments the model's parameters.

For now, we will consider the estimator as a black box:

   >>> clf = svm.SVC(gamma=0.001, C=100.)

Choosing the parameters of the model

In this example, we set the value of "gamma" manually. To find good
values for these parameters, we can use tools such as grid search and
cross validation.

The "clf" (for classifier) estimator instance is first fitted to the
model; that is, it must *learn* from the model. This is done by
passing our training set to the "fit" method. For the training set,
we'll use all the images from our dataset, except for the last image,
which we'll reserve for our predicting. We select the training set

with the "[:-1]" Python syntax, which produces a new array that
contains all but the last item from "digits.data":

   >>> clf.fit(digits.data[:-1], digits.target[:-1])
   SVC(C=100.0, gamma=0.001)

Now you can *predict* new values. In this case, you'll predict using
the last image from "digits.data". By predicting, you'll determine the
image from the training set that best matches the last image.

>>> clf.predict(digits.data[-1:])
array([8])

The corresponding image is:

[image]

As you can see, it is a challenging task: after all, the images are of
poor resolution. Do you agree with the classifier?

A complete example of this classification problem is available as an
example that you can run and study:
*sphx_glr_auto_examples_classification_plot_digits_classification.py*.

Conventions

scikit-learn estimators follow certain rules to make their behavior
more predictive.  These are described in more detail in the Glossary
of Common Terms and API Elements.

Type casting

Where possible, input of type "float32" will maintain its data type.
Otherwise input will be cast to "float64":

   >>> from sklearn import kernel_approximation

   >>> X = rng.rand(10, 2000)
   >>> X = np.array(X, dtype='float32')
   >>> X.dtype
   dtype('float32')

   >>> transformer = kernel_approximation.RBFSampler()

>>> rng = np.random.RandomState(0)

   >>> X_new = transformer.fit_transform(X)
   >>> X_new.dtype

In this example, "X" is "float32", and is unchanged by
"fit_transform(X)".

Using *float32*-typed training (or testing) data is often more
efficient than using the usual "float64" "dtype": it allows to reduce
the memory usage and sometimes also reduces processing time by
leveraging the vector instructions of the CPU. However it can
sometimes lead to numerical stability problems causing the algorithm
to be more sensitive to the scale of the values and require adequate
preprocessing.

Keep in mind however that not all scikit-learn estimators attempt to
work in *float32* mode. For instance, some transformers will always
cast there input to *float64* and return *float64* transformed values
as a result.

Regression targets are cast to "float64" and classification targets
are maintained:

   >>> from sklearn.svm import SVC
   >>> clf = SVC()
   >>> clf.fit(iris.data, iris.target)
   SVC()

   >>> list(clf.predict(iris.data[:3]))
   [0, 0, 0]

   >>> clf.fit(iris.data, iris.target_names[iris.target])
   SVC()

   ['setosa', 'setosa', 'setosa']

Here, the first "predict()" returns an integer array, since

>>> clf.fit(iris.data, iris.target_names[iris.target])
   SVC()

"iris.target" (an integer array) was used in "fit". The second
"predict()" returns a string array, since "iris.target_names" was for
fitting.

Refitting and updating parameters

Hyper-parameters of an estimator can be updated after it has been
constructed via the *set_params()* method. Calling "fit()" more than
once will overwrite what was learned by any previous "fit()":

   >>> clf.set_params(kernel='linear').fit(X, y)
   SVC(kernel='linear')
   >>> clf.predict(X[:5])
   array([0, 0, 0, 0, 0])

   >>> clf.set_params(kernel='rbf').fit(X, y)
   SVC()

Here, the default kernel "rbf" is first changed to "linear" via
"SVC.set_params()" after the estimator has been constructed, and
changed back to "rbf" to refit the estimator and to make a second
prediction.

Multiclass vs. multilabel fitting

When using "multiclass classifiers", the learning and prediction task
that is performed is dependent on the format of the target data fit
upon:

   >>> from sklearn.multiclass import OneVsRestClassifier

upon:

   >>> from sklearn.preprocessing import LabelBinarizer

   >>> X = [[1, 2], [2, 4], [4, 5], [3, 2], [3, 1]]
   >>> y = [0, 0, 1, 1, 2]

   >>> classif = OneVsRestClassifier(estimator=SVC(random_state=0))
   >>> classif.fit(X, y).predict(X)
   array([0, 0, 1, 1, 2])

In the above case, the classifier is fit on a 1d array of multiclass
labels and the "predict()" method therefore provides corresponding
multiclass predictions. It is also possible to fit upon a 2d array of
binary label indicators:

   >>> y = LabelBinarizer().fit_transform(y)
   array([[1, 0, 0],
          [1, 0, 0],
          [0, 1, 0],
          [0, 0, 0],
          [0, 0, 0]])

Here, the classifier is "fit()"  on a 2d binary label representation
of "y", using the "LabelBinarizer". In this case "predict()" returns a
2d array representing the corresponding multilabel predictions.

Note that the fourth and fifth instances returned all zeroes,
indicating that they matched none of the three labels "fit" upon. With
multilabel outputs, it is similarly possible for an instance to be
assigned multiple labels:

   >>> from sklearn.preprocessing import MultiLabelBinarizer
   >>> y = [[0, 1], [0, 2], [1, 3], [0, 2, 3], [2, 4]]
   >>> y = MultiLabelBinarizer().fit_transform(y)
   array([[1, 1, 0, 0, 0],

>>> y = [[0, 1], [0, 2], [1, 3], [0, 2, 3], [2, 4]]
          [1, 0, 1, 0, 0],
          [0, 1, 0, 1, 0],
          [1, 0, 1, 0, 0]])

In this case, the classifier is fit upon instances each assigned
multiple labels. The "MultiLabelBinarizer" is used to binarize the 2d
array of multilabels to "fit" upon. As a result, "predict()" returns a
2d array with multiple predicted labels for each instance.

Model selection: choosing estimators and their parameters
*********************************************************

Score, and cross-validated scores

As we have seen, every estimator exposes a "score" method that can
judge the quality of the fit (or the prediction) on new data. **Bigger
is better**.

   >>> from sklearn import datasets, svm
   >>> X_digits, y_digits = datasets.load_digits(return_X_y=True)
   >>> svc = svm.SVC(C=1, kernel='linear')
   >>> svc.fit(X_digits[:-100], y_digits[:-100]).score(X_digits[-100:], y_digits[-100:])
   0.98

To get a better measure of prediction accuracy (which we can use as a
proxy for goodness of fit of the model), we can successively split the
data in *folds* that we use for training and testing:

   >>> X_folds = np.array_split(X_digits, 3)
   >>> y_folds = np.array_split(y_digits, 3)
   >>> scores = list()
   >>> for k in range(3):
   ...     # We use 'list' to copy, in order to 'pop' later on
   ...     X_train = list(X_folds)
   ...     X_test = X_train.pop(k)
   ...     X_train = np.concatenate(X_train)
   ...     y_train = list(y_folds)
   ...     y_test = y_train.pop(k)
   ...     y_train = np.concatenate(y_train)
   ...     scores.append(svc.fit(X_train, y_train).score(X_test, y_test))
   >>> print(scores)
   [0.934..., 0.956..., 0.939...]

This is called a "KFold" cross-validation.

Cross-validation generators

>>> print(scores)

Scikit-learn has a collection of classes which can be used to generate
lists of train/test indices for popular cross-validation strategies.

They expose a "split" method which accepts the input dataset to be
split and yields the train/test set indices for each iteration of the
chosen cross-validation strategy.

This example shows an example usage of the "split" method.

>>> from sklearn.model_selection import KFold, cross_val_score
>>> X = ["a", "a", "a", "b", "b", "c", "c", "c", "c", "c"]
>>> k_fold = KFold(n_splits=5)
>>> for train_indices, test_indices in k_fold.split(X):
...      print('Train: %s | test: %s' % (train_indices, test_indices))
Train: [2 3 4 5 6 7 8 9] | test: [0 1]
Train: [0 1 4 5 6 7 8 9] | test: [2 3]
Train: [0 1 2 3 6 7 8 9] | test: [4 5]
Train: [0 1 2 3 4 5 8 9] | test: [6 7]
Train: [0 1 2 3 4 5 6 7] | test: [8 9]

The cross-validation can then be performed easily:

   >>> [svc.fit(X_digits[train], y_digits[train]).score(X_digits[test], y_digits[test])
   ...  for train, test in k_fold.split(X_digits)]
   [0.963..., 0.922..., 0.963..., 0.963..., 0.930...]

The cross-validation score can be directly calculated using the
"cross_val_score()" helper. Given an estimator, the cross-validation
object and the input dataset, the "cross_val_score()" splits the data
repeatedly into a training and a testing set, trains the estimator

using the training set and computes the scores based on the testing
set for each iteration of cross-validation.

By default the estimator's "score" method is used to compute the
individual scores.

Refer the metrics module to learn more on the available scoring
methods.

>>> cross_val_score(svc, X_digits, y_digits, cv=k_fold, n_jobs=-1)
array([0.96388889, 0.92222222, 0.9637883 , 0.9637883 , 0.93036212])

*n_jobs=-1* means that the computation will be dispatched on all the
CPUs of the computer.

Alternatively, the "scoring" argument can be provided to specify an
alternative scoring method.

   >>> cross_val_score(svc, X_digits, y_digits, cv=k_fold,
   ...                 scoring='precision_macro')
   array([0.96578289, 0.92708922, 0.96681476, 0.96362897, 0.93192644])

   **Cross-validation generators**

+-----------------------------------+-----------------------------------+-----------------------------------+
| "KFold" **(n_splits, shuffle,     | "StratifiedKFold" **(n_splits,    | "GroupKFold" **(n_splits)**       |
| random_state)**                   | shuffle, random_state)**          |                                   |
| Splits it into K folds, trains on | Same as K-Fold but preserves the  | Ensures that the same group is    |

| K-1 and then tests on the left-   | class distribution within each    | not in both testing and training  |
| out.                              | fold.                             | sets.                             |

| "ShuffleSplit" **(n_splits,       | "StratifiedShuffleSplit"          | "GroupShuffleSplit"               |
| test_size, train_size,            |                                   |                                   |
| random_state)**                   |                                   |                                   |
| Generates train/test indices      | Same as shuffle split but         | Ensures that the same group is    |
| based on random permutation.      | preserves the class distribution  | not in both testing and training  |
|                                   | within each iteration.            | sets.                             |

| "LeaveOneGroupOut" **()**         | "LeavePGroupsOut"  **(n_groups)** | "LeaveOneOut" **()**              |
| Takes a group array to group      | Leave P groups out.               | Leave one observation out.        |
| observations.                     |                                   |                                   |

+----------------------------------------------------+----------------------------------------------------+
| "LeavePOut" **(p)**                                | "PredefinedSplit"                                  |
| Leave P observations out.                          | Generates train/test indices based on predefined   |
|                                                    | splits.                                            |

**Exercise**
^^^^^^^^^^^^

On the digits dataset, plot the cross-validation score of a "SVC"

estimator with a linear kernel as a function of parameter "C" (use a
logarithmic grid of points, from 1 to 10).

[image]

**Solution:** *sphx_glr_auto_examples_exercises_plot_cv_digits.py*

Grid-search and cross-validated estimators

Grid-search

scikit-learn provides an object that, given data, computes the score
during the fit of an estimator on a parameter grid and chooses the
parameters to maximize the cross-validation score. This object takes
an estimator during the construction and exposes an estimator API:

   >>> from sklearn.model_selection import GridSearchCV, cross_val_score
   >>> Cs = np.logspace(-6, -1, 10)
   >>> clf = GridSearchCV(estimator=svc, param_grid=dict(C=Cs),
   ...                    n_jobs=-1)
   >>> clf.fit(X_digits[:1000], y_digits[:1000])        
   GridSearchCV(cv=None,...
   >>> clf.best_score_                                  
   0.925...
   >>> clf.best_estimator_.C                            
   0.0077...

   >>> # Prediction performance on test set is not as good as on train set
   >>> clf.score(X_digits[1000:], y_digits[1000:])      
   0.943...

By default, the "GridSearchCV" uses a 5-fold cross-validation.
However, if it detects that a classifier is passed, rather than a
regressor, it uses a stratified 5-fold.

0.943...

Nested cross-validation
^^^^^^^^^^^^^^^^^^^^^^^

   >>> cross_val_score(clf, X_digits, y_digits) 
   array([0.938..., 0.963..., 0.944...])

Two cross-validation loops are performed in parallel: one by the
"GridSearchCV" estimator to set "gamma" and the other one by
"cross_val_score" to measure the prediction performance of the
estimator. The resulting scores are unbiased estimates of the
prediction score on new data.

Warning:

  You cannot nest objects with parallel computing ("n_jobs" different
  than 1).

Cross-validated estimators

Cross-validation to set a parameter can be done more efficiently on an
algorithm-by-algorithm basis. This is why, for certain estimators,
scikit-learn exposes Cross-validation: evaluating estimator
performance estimators that set their parameter automatically by
cross-validation:

   >>> from sklearn import linear_model, datasets
   >>> lasso = linear_model.LassoCV()
   >>> X_diabetes, y_diabetes = datasets.load_diabetes(return_X_y=True)
   >>> lasso.fit(X_diabetes, y_diabetes)
   LassoCV()
   >>> # The estimator chose automatically its lambda:
   >>> lasso.alpha_
   0.00375...

These estimators are called similarly to their counterparts, with 'CV'
appended to their name.

On the diabetes dataset, find the optimal regularization parameter
alpha.

alpha.

**Bonus**: How much can you trust the selection of alpha?

**Solution:** *sphx_glr_auto_examples_exercises_plot_cv_diabetes.py*

A tutorial on statistical-learning for scientific data processing
*****************************************************************

Statistical learning
^^^^^^^^^^^^^^^^^^^^

Machine learning is a technique with a growing importance, as the size
of the datasets experimental sciences are facing is rapidly growing.
Problems it tackles range from building a prediction function linking
different observations, to classifying observations, or learning the
structure in an unlabeled dataset.

This tutorial will explore *statistical learning*, the use of machine
learning techniques with the goal of statistical inference: drawing
conclusions on the data at hand.

Scikit-learn is a Python module integrating classic machine learning
algorithms in the tightly-knit world of scientific Python packages
(NumPy, SciPy, matplotlib).

* Statistical learning: the setting and the estimator object in

  * Datasets

  * Estimators objects

* Supervised learning: predicting an output variable from high-
  dimensional observations

  * Nearest neighbor and the curse of dimensionality

  * Linear model: from regression to sparsity

  * Support vector machines (SVMs)

* Model selection: choosing estimators and their parameters

  * Score, and cross-validated scores

  * Cross-validation generators

  * Grid-search and cross-validated estimators

* Unsupervised learning: seeking representations of the data

  * Clustering: grouping observations together

  * Decompositions: from a signal to components and loadings

* Putting it all together

  * Pipelining

  * Face recognition with eigenfaces

  * Open problem: Stock Market Structure

Putting it all together

Pipelining

We have seen that some estimators can transform data and that some
estimators can predict variables. We can also create combined
estimators:

[image]

Face recognition with eigenfaces

The dataset used in this example is a preprocessed excerpt of the
"Labeled Faces in the Wild", also known as LFW:

   http://vis-www.cs.umass.edu/lfw/lfw-funneled.tgz (233MB)

   [image]**Prediction**

   [image]**Eigenfaces**

Expected results for the top 5 most represented people in the dataset:

                      precision    recall  f1-score   support

   Gerhard_Schroeder       0.91      0.75      0.82        28
     Donald_Rumsfeld       0.84      0.82      0.83        33
          Tony_Blair       0.65      0.82      0.73        34
        Colin_Powell       0.78      0.88      0.83        58
       George_W_Bush       0.93      0.86      0.90       129

         avg / total       0.86      0.84      0.85       282

Open problem: Stock Market Structure
====================================

Can we predict the variation in stock prices for Google over a given
time frame?

*stock_market*

Statistical learning: the setting and the estimator object in scikit-learn
**************************************************************************

Datasets
========

Scikit-learn deals with learning information from one or more datasets
that are represented as 2D arrays. They can be understood as a list of
multi-dimensional observations. We say that the first axis of these
arrays is the **samples** axis, while the second is the **features**
axis.

A simple example shipped with scikit-learn: iris dataset
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

   >>> data = iris.data
   >>> data.shape
   (150, 4)

It is made of 150 observations of irises, each described by 4
features: their sepal and petal length and width, as detailed in
"iris.DESCR".

When the data is not initially in the "(n_samples, n_features)" shape,
it needs to be preprocessed in order to be used by scikit-learn.

An example of reshaping data would be the digits dataset

The digits dataset is made of 1797 8x8 images of hand-written digits

   >>> digits.images.shape
   (1797, 8, 8)
   >>> import matplotlib.pyplot as plt
   >>> plt.imshow(digits.images[-1],
   ...            cmap=plt.cm.gray_r)
   <...>

[image]

To use this dataset with scikit-learn, we transform each 8x8 image
into a feature vector of length 64

   >>> data = digits.images.reshape(

...            cmap=plt.cm.gray_r)
   <...>

[image]

   ...     (digits.images.shape[0], -1)
   ... )

Estimators objects

**Fitting data**: the main API implemented by scikit-learn is that of
the *estimator*. An estimator is any object that learns from data; it
may be a classification, regression or clustering algorithm or a
*transformer* that extracts/filters useful features from raw data.

All estimator objects expose a "fit" method that takes a dataset
(usually a 2-d array):

>>> estimator.fit(data)

**Estimator parameters**: All the parameters of an estimator can be
set when it is instantiated or by modifying the corresponding
attribute:

   >>> estimator = Estimator(param1=1, param2=2)
   >>> estimator.param1
   1

**Estimated parameters**: When data is fitted with an estimator,
parameters are estimated from the data at hand. All the estimated
parameters are attributes of the estimator object ending by an
underscore:

   >>> estimator.estimated_param_

Unsupervised learning: seeking representations of the data
**********************************************************

Clustering: grouping observations together

The problem solved in clustering
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Given the iris dataset, if we knew that there were 3 types of iris,
but did not have access to a taxonomist to label them: we could try a
**clustering task**: split the observations into well-separated group
called *clusters*.

K-means clustering

Note that there exist a lot of different clustering criteria and
associated algorithms. The simplest clustering algorithm is K-means.

   >>> from sklearn import cluster, datasets
   >>> X_iris, y_iris = datasets.load_iris(return_X_y=True)

   >>> k_means = cluster.KMeans(n_clusters=3)
   >>> k_means.fit(X_iris)
   KMeans(n_clusters=3)
   >>> print(k_means.labels_[::10])
   [1 1 1 1 1 0 0 0 0 0 2 2 2 2 2]
   >>> print(y_iris[::10])
   [0 0 0 0 0 1 1 1 1 1 2 2 2 2 2]

   [image]

Warning:

  There is absolutely no guarantee of recovering a ground truth.
  First, choosing the right number of clusters is hard. Second, the
  algorithm is sensitive to initialization, and can fall into local
  minima, although scikit-learn employs several tricks to mitigate
  this issue.For instance, on the image above, we can observe the
  difference between the ground-truth (bottom right figure) and
  different clustering. We do not recover the expected labels, either

this issue.For instance, on the image above, we can observe the
  because the number of cluster was chosen to be to large (top left
  figure) or suffer from a bad initialization (bottom left
  figure).**It is therefore important to not over-interpret clustering
  results.**

**Application example: vector quantization**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Clustering in general and KMeans, in particular, can be seen as a way
of choosing a small number of exemplars to compress the information.
The problem is sometimes known as vector quantization. For instance,
this can be used to posterize an image:

   >>> import scipy as sp
   >>> try:
   ...    face = sp.face(gray=True)
   ... except AttributeError:
   ...    from scipy import misc
   ...    face = misc.face(gray=True)
   >>> X = face.reshape((-1, 1)) # We need an (n_sample, n_feature) array
   >>> k_means = cluster.KMeans(n_clusters=5, n_init=1)
   >>> k_means.fit(X)
   KMeans(n_clusters=5, n_init=1)
   >>> values = k_means.cluster_centers_.squeeze()
   >>> labels = k_means.labels_
   >>> face_compressed = np.choose(labels, values)
   >>> face_compressed.shape = face.shape

**Raw image**

   [image]

**K-means quantization**

   [image]

**Equal bins**

   [image]

Hierarchical agglomerative clustering: Ward
-------------------------------------------

A Hierarchical clustering method is a type of cluster analysis that

[image]

   [image]

aims to build a hierarchy of clusters. In general, the various
approaches of this technique are either:

   * **Agglomerative** - bottom-up approaches: each observation starts
     in its own cluster, and clusters are iteratively merged in such a
     way to minimize a *linkage* criterion. This approach is
     particularly interesting when the clusters of interest are made
     of only a few observations. When the number of clusters is large,
     it is much more computationally efficient than k-means.

   * **Divisive** - top-down approaches: all observations start in one
     cluster, which is iteratively split as one moves down the
     hierarchy. For estimating large numbers of clusters, this
     approach is both slow (due to all observations starting as one
     cluster, which it splits recursively) and statistically ill-
     posed.

Connectivity-constrained clustering
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With agglomerative clustering, it is possible to specify which samples
can be clustered together by giving a connectivity graph. Graphs in
scikit-learn are represented by their adjacency matrix. Often, a
sparse matrix is used. This can be useful, for instance, to retrieve
connected regions (sometimes also referred to as connected components)
when clustering an image.

[image]

[image]

   >>> from skimage.data import coins
   >>> from scipy.ndimage import gaussian_filter
   >>> from skimage.transform import rescale
   >>> rescaled_coins = rescale(
   ...     gaussian_filter(coins(), sigma=2),
   ...     0.2, mode='reflect', anti_aliasing=False
   ... )
   >>> X = np.reshape(rescaled_coins, (-1, 1))

We need a vectorized version of the image. *'rescaled_coins'* is a
down-scaled version of the coins image to speed up the process:

   >>> from sklearn.feature_extraction import grid_to_graph
   >>> connectivity = grid_to_graph(*rescaled_coins.shape)

Define the graph structure of the data. Pixels connected to their
neighbors:

   >>> n_clusters = 27  # number of regions

   >>> from sklearn.cluster import AgglomerativeClustering
   >>> ward = AgglomerativeClustering(n_clusters=n_clusters, linkage='ward',
   ...                                connectivity=connectivity)
   >>> ward.fit(X)
   AgglomerativeClustering(connectivity=..., n_clusters=27)
   >>> label = np.reshape(ward.labels_, rescaled_coins.shape)

Feature agglomeration
~~~~~~~~~~~~~~~~~~~~~

We have seen that sparsity could be used to mitigate the curse of
dimensionality, *i.e* an insufficient amount of observations compared
to the number of features. Another approach is to merge together
similar features: **feature agglomeration**. This approach can be

implemented by clustering in the feature direction, in other words
clustering the transposed data.

[image]

   >>> images = digits.images
   >>> X = np.reshape(images, (len(images), -1))
   >>> connectivity = grid_to_graph(*images[0].shape)

   >>> agglo = cluster.FeatureAgglomeration(connectivity=connectivity,
   ...                                      n_clusters=32)
   >>> agglo.fit(X)
   FeatureAgglomeration(connectivity=..., n_clusters=32)
   >>> X_reduced = agglo.transform(X)

   >>> X_approx = agglo.inverse_transform(X_reduced)
   >>> images_approx = np.reshape(X_approx, images.shape)

"transform" and "inverse_transform" methods
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some estimators expose a "transform" method, for instance to reduce
the dimensionality of the dataset.

Decompositions: from a signal to components and loadings
========================================================

**Components and loadings**
^^^^^^^^^^^^^^^^^^^^^^^^^^^

If X is our multivariate data, then the problem that we are trying to
solve is to rewrite it on a different observational basis: we want to
learn loadings L and a set of components C such that *X = L C*.
Different criteria exist to choose the components

Principal component analysis: PCA

Principal component analysis (PCA) selects the successive components
that explain the maximum variance in the signal.

[image: pca_3d_axis][image] [image: pca_3d_aligned][image]

The point cloud spanned by the observations above is very flat in one
direction: one of the three univariate features can almost be exactly
computed using the other two. PCA finds the directions in which the
data is not *flat*

When used to *transform* data, PCA can reduce the dimensionality of
the data by projecting on a principal subspace.

   >>> # Create a signal with only 2 useful dimensions
   >>> x1 = np.random.normal(size=100)
   >>> x2 = np.random.normal(size=100)
   >>> x3 = x1 + x2
   >>> X = np.c_[x1, x2, x3]

   >>> from sklearn import decomposition
   >>> pca = decomposition.PCA()
   >>> pca.fit(X)
   PCA()
   >>> print(pca.explained_variance_)  
   [  2.18565811e+00   1.19346747e+00   8.43026679e-32]

   >>> # As we can see, only the 2 first components are useful
   >>> pca.n_components = 2
   >>> X_reduced = pca.fit_transform(X)
   >>> X_reduced.shape
   (100, 2)

Independent Component Analysis: ICA
-----------------------------------

Independent component analysis (ICA) selects components so that the
distribution of their loadings carries a maximum amount of independent
information. It is able to recover **non-Gaussian** independent
signals:

signals:

[image]

   >>> # Generate sample data
   >>> from scipy import signal
   >>> time = np.linspace(0, 10, 2000)
   >>> s1 = np.sin(2 * time)  # Signal 1 : sinusoidal signal
   >>> s2 = np.sign(np.sin(3 * time))  # Signal 2 : square signal
   >>> s3 = signal.sawtooth(2 * np.pi * time)  # Signal 3: saw tooth signal
   >>> S = np.c_[s1, s2, s3]
   >>> S += 0.2 * np.random.normal(size=S.shape)  # Add noise
   >>> S /= S.std(axis=0)  # Standardize data
   >>> # Mix data
   >>> A = np.array([[1, 1, 1], [0.5, 2, 1], [1.5, 1, 2]])  # Mixing matrix
   >>> X = np.dot(S, A.T)  # Generate observations

   >>> # Compute ICA
   >>> ica = decomposition.FastICA()
   >>> S_ = ica.fit_transform(X)  # Get the estimated sources
   >>> A_ = ica.mixing_.T
   >>> np.allclose(X,  np.dot(S_, A_) + ica.mean_)
   True

Supervised learning: predicting an output variable from high-dimensional observations
*************************************************************************************

The problem solved in supervised learning
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Supervised learning consists in learning the link between two
datasets: the observed data "X" and an external variable "y" that we
are trying to predict, usually called "target" or "labels". Most
often, "y" is a 1D array of length "n_samples".

All supervised estimators in scikit-learn implement a "fit(X, y)"
method to fit the model and a "predict(X)" method that, given
unlabeled observations "X", returns the predicted labels "y".

Vocabulary: classification and regression

If the prediction task is to classify the observations in a set of
finite labels, in other words to "name" the objects observed, the task
is said to be a **classification** task. On the other hand, if the
goal is to predict a continuous target variable, it is said to be a
**regression** task.

When doing classification in scikit-learn, "y" is a vector of integers
or strings.

Note: See the Introduction to machine learning with scikit-learn
Tutorial for a quick run-through on the basic machine learning
vocabulary used within scikit-learn.

Nearest neighbor and the curse of dimensionality
================================================

Classifying irises:
^^^^^^^^^^^^^^^^^^^

The iris dataset is a classification task consisting in identifying 3

different types of irises (Setosa, Versicolour, and Virginica) from
their petal and sepal length and width:

   >>> iris_X, iris_y = datasets.load_iris(return_X_y=True)
   >>> np.unique(iris_y)
   array([0, 1, 2])

[image]

k-Nearest neighbors classifier

The simplest possible classifier is the nearest neighbor: given a new
observation "X_test", find in the training set (i.e. the data used to
train the estimator) the observation with the closest feature vector.
(Please see the Nearest Neighbors section of the online Scikit-learn
documentation for more information about this type of classifier.)

While experimenting with any learning algorithm, it is important not
to test the prediction of an estimator on the data used to fit the
estimator as this would not be evaluating the performance of the
estimator on **new data**. This is why datasets are often split into
*train* and *test* data.

**KNN (k nearest neighbors) classification example**:

[image]

   >>> # Split iris data in train and test data
   >>> # A random permutation, to split the data randomly
   >>> np.random.seed(0)
   >>> indices = np.random.permutation(len(iris_X))
   >>> iris_X_train = iris_X[indices[:-10]]

>>> # A random permutation, to split the data randomly
   >>> iris_y_train = iris_y[indices[:-10]]
   >>> iris_X_test = iris_X[indices[-10:]]
   >>> iris_y_test = iris_y[indices[-10:]]
   >>> # Create and fit a nearest-neighbor classifier
   >>> from sklearn.neighbors import KNeighborsClassifier
   >>> knn = KNeighborsClassifier()
   >>> knn.fit(iris_X_train, iris_y_train)
   KNeighborsClassifier()
   >>> knn.predict(iris_X_test)
   array([1, 2, 1, 0, 0, 0, 2, 1, 2, 0])
   >>> iris_y_test
   array([1, 1, 1, 0, 0, 0, 2, 1, 2, 0])

The curse of dimensionality

For an estimator to be effective, you need the distance between
neighboring points to be less than some value d, which depends on the
problem. In one dimension, this requires on average n \sim 1/d points.
In the context of the above k-NN example, if the data is described by
just one feature with values ranging from 0 to 1 and with n training
observations, then new data will be no further away than 1/n.
Therefore, the nearest neighbor decision rule will be efficient as
soon as 1/n is small compared to the scale of between-class feature
variations.

If the number of features is p, you now require n \sim 1/d^p points.
Let's say that we require 10 points in one dimension: now 10^p points
are required in p dimensions to pave the [0, 1] space. As p becomes

large, the number of training points required for a good estimator
grows exponentially.

For example, if each point is just a single number (8 bytes), then an
effective k-NN estimator in a paltry p \sim 20 dimensions would
require more training data than the current estimated size of the
entire internet (±1000 Exabytes or so).

This is called the curse of dimensionality and is a core problem that
machine learning addresses.

Linear model: from regression to sparsity

Diabetes dataset

The diabetes dataset consists of 10 physiological variables (age, sex,
weight, blood pressure) measured on 442 patients, and an indication of
disease progression after one year:

   >>> diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True)
   >>> diabetes_X_train = diabetes_X[:-20]
   >>> diabetes_X_test  = diabetes_X[-20:]
   >>> diabetes_y_train = diabetes_y[:-20]
   >>> diabetes_y_test  = diabetes_y[-20:]

The task at hand is to predict disease progression from physiological
variables.

Linear regression

"LinearRegression", in its simplest form, fits a linear model to the
data set by adjusting a set of parameters in order to make the sum of
the squared residuals of the model as small as possible.

Linear models: y = X\beta + \epsilon

   * X: data

   * y: target variable

   * \beta: Coefficients

   * X: data

   * \epsilon: Observation noise

[image]

   >>> from sklearn import linear_model
   >>> regr = linear_model.LinearRegression()
   >>> regr.fit(diabetes_X_train, diabetes_y_train)
   LinearRegression()
   >>> print(regr.coef_) 
   [   0.30349955 -237.63931533  510.53060544  327.73698041 -814.13170937
     492.81458798  102.84845219  184.60648906  743.51961675   76.09517222]

   >>> # The mean square error
   >>> np.mean((regr.predict(diabetes_X_test) - diabetes_y_test)**2)
   2004.5...

   >>> # Explained variance score: 1 is perfect prediction
   >>> # and 0 means that there is no linear relationship
   >>> # between X and y.
   >>> regr.score(diabetes_X_test, diabetes_y_test)
   0.585...

Shrinkage
---------

If there are few data points per dimension, noise in the observations
induces high variance:

   >>> X = np.c_[ .5, 1].T
   >>> y = [.5, 1]
   >>> test = np.c_[ 0, 2].T

   >>> plt.figure()
   <...>
   >>> for _ in range(6):
   ...     this_X = .1 * np.random.normal(size=(2, 1)) + X
   ...     regr.fit(this_X, y)
   ...     plt.plot(test, regr.predict(test))
   ...     plt.scatter(this_X, y, s=3)
   LinearRegression...

[image]

A solution in high-dimensional statistical learning is to *shrink* the

...     plt.plot(test, regr.predict(test))

[image]

regression coefficients to zero: any two randomly chosen set of
observations are likely to be uncorrelated. This is called "Ridge"
regression:

   >>> regr = linear_model.Ridge(alpha=.1)

   <...>
   Ridge...

[image]

This is an example of **bias/variance tradeoff**: the larger the ridge
"alpha" parameter, the higher the bias and the lower the variance.

We can choose "alpha" to minimize left out error, this time using the
diabetes dataset rather than our synthetic data:

   >>> alphas = np.logspace(-4, -1, 6)
   >>> print([regr.set_params(alpha=alpha)
   ...            .fit(diabetes_X_train, diabetes_y_train)
   ...            .score(diabetes_X_test, diabetes_y_test)
   ...        for alpha in alphas])
   [0.585..., 0.585..., 0.5854..., 0.5855..., 0.583..., 0.570...]

Note:

  Capturing in the fitted parameters noise that prevents the model to
  generalize to new data is called overfitting. The bias introduced by
  the ridge regression is called a regularization.

Sparsity
--------

**Fitting only features 1 and 2**

[image: diabetes_ols_1][image] [image: diabetes_ols_3][image] [image:

the ridge regression is called a regularization.

Sparsity
--------

diabetes_ols_2][image]

Note:

  A representation of the full diabetes dataset would involve 11
  dimensions (10 feature dimensions and one of the target variable).
  It is hard to develop an intuition on such representation, but it
  may be useful to keep in mind that it would be a fairly *empty*
  space.

We can see that, although feature 2 has a strong coefficient on the
full model, it conveys little information on "y" when considered with
feature 1.

To improve the conditioning of the problem (i.e. mitigating the The
curse of dimensionality), it would be interesting to select only the
informative features and set non-informative ones, like feature 2 to
0. Ridge regression will decrease their contribution, but not set them
to zero. Another penalization approach, called Lasso (least absolute
shrinkage and selection operator), can set some coefficients to zero.
Such methods are called **sparse methods** and sparsity can be seen as
an application of Occam's razor: *prefer simpler models*.

   >>> regr = linear_model.Lasso()
   >>> scores = [regr.set_params(alpha=alpha)
   ...               .fit(diabetes_X_train, diabetes_y_train)
   ...               .score(diabetes_X_test, diabetes_y_test)
   ...           for alpha in alphas]
   >>> best_alpha = alphas[scores.index(max(scores))]
   >>> regr.alpha = best_alpha

...               .score(diabetes_X_test, diabetes_y_test)
   Lasso(alpha=0.025118864315095794)
   >>> print(regr.coef_)
   [   0.         -212.4...   517.2...  313.7... -160.8...
      -0.         -187.1...   69.3...  508.6...   71.8... ]

**Different algorithms for the same problem**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Different algorithms can be used to solve the same mathematical
problem. For instance the "Lasso" object in scikit-learn solves the
lasso regression problem using a coordinate descent method, that is
efficient on large datasets. However, scikit-learn also provides the
"LassoLars" object using the *LARS* algorithm, which is very efficient
for problems in which the weight vector estimated is very sparse (i.e.
problems with very few observations).

Classification

For classification, as in the labeling iris task, linear regression is
not the right approach as it will give too much weight to data far
from the decision frontier. A linear approach is to fit a sigmoid
function or **logistic** function:

[image]

   y = \textrm{sigmoid}(X\beta - \textrm{offset}) + \epsilon =
   \frac{1}{1 + \textrm{exp}(- X\beta + \textrm{offset})} + \epsilon

   >>> log = linear_model.LogisticRegression(C=1e5)
   >>> log.fit(iris_X_train, iris_y_train)
   LogisticRegression(C=100000.0)

\frac{1}{1 + \textrm{exp}(- X\beta + \textrm{offset})} + \epsilon

This is known as "LogisticRegression".

[image]

Multiclass classification
^^^^^^^^^^^^^^^^^^^^^^^^^

If you have several classes to predict, an option often used is to fit
one-versus-all classifiers and then use a voting heuristic for the
final decision.

Shrinkage and sparsity with logistic regression
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The "C" parameter controls the amount of regularization in the
"LogisticRegression" object: a large value for "C" results in less
regularization. "penalty="l2"" gives Shrinkage (i.e. non-sparse
coefficients), while "penalty="l1"" gives Sparsity.

Try classifying the digits dataset with nearest neighbors and a linear
model. Leave out the last 10% and test prediction performance on these
observations.

A solution can be downloaded "here".

Support vector machines (SVMs)

Linear SVMs

Support Vector Machines belong to the discriminant model family: they
try to find a combination of samples to build a plane maximizing the
margin between the two classes. Regularization is set by the "C"
parameter: a small value for "C" means the margin is calculated using
many or all of the observations around the separating line (more
regularization); a large value for "C" means the margin is calculated

on observations close to the separating line (less regularization).

   [image]**Unregularized SVM**

   [image]**Regularized SVM (default)**

Example:
^^^^^^^^

* *sphx_glr_auto_examples_svm_plot_iris_svc.py*

SVMs can be used in regression --"SVR" (Support Vector Regression)--,
or in classification --"SVC" (Support Vector Classification).

   >>> svc = svm.SVC(kernel='linear')
   >>> svc.fit(iris_X_train, iris_y_train)

Warning:

  **Normalizing data**For many estimators, including the SVMs, having
  datasets with unit standard deviation for each feature is important
  to get good prediction.

Using kernels

Classes are not always linearly separable in feature space. The
solution is to build a decision function that is not linear but may be
polynomial instead. This is done using the *kernel trick* that can be
seen as creating a decision energy by positioning *kernels* on
observations:

Linear kernel
~~~~~~~~~~~~~

[image]

Polynomial kernel
~~~~~~~~~~~~~~~~~

   >>> svc = svm.SVC(kernel='poly',
   ...               degree=3)
   >>> # degree: polynomial degree

[image]

RBF kernel (Radial Basis Function)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   >>> svc = svm.SVC(kernel='rbf')
   >>> # gamma: inverse of size of
   >>> # radial kernel

[image]

**Interactive example**

[image]

See the *SVM GUI* to download "svm_gui.py"; add data points of both
classes with right and left button, fit the model and change
parameters and data.

Try classifying classes 1 and 2 from the iris dataset with SVMs, with
the 2 first features. Leave out 10% of each class and test prediction
performance on these observations.

**Warning**: the classes are ordered, do not leave out the last 10%,
you would be testing on only one class.

**Hint**: You can use the "decision_function" method on a grid to get
intuitions.

[image]

A solution can be downloaded "here"

Working With Text Data

The goal of this guide is to explore some of the main "scikit-learn"
tools on a single practical task: analyzing a collection of text
documents (newsgroups posts) on twenty different topics.

In this section we will see how to:

   * load the file contents and the categories

   * extract feature vectors suitable for machine learning

   * train a linear model to perform categorization

   * use a grid search strategy to find a good configuration of both
     the feature extraction components and the classifier

Tutorial setup

To get started with this tutorial, you must first install *scikit-
learn* and all of its required dependencies.

Please refer to the installation instructions page for more
information and for system-specific instructions.

The source of this tutorial can be found within your scikit-learn
folder:

   scikit-learn/doc/tutorial/text_analytics/

The source can also be found on Github.

The tutorial folder should contain the following sub-folders:

   * "*.rst files" - the source of the tutorial document written with
     sphinx

   * "data" - folder to put the datasets used during the tutorial

   * "skeletons" - sample incomplete scripts for the exercises

   * "solutions" - solutions of the exercises

You can already copy the skeletons into a new folder somewhere on your
hard-drive named "sklearn_tut_workspace", where you will edit your own
files for the exercises while keeping the original skeletons intact:

Machine learning algorithms need data. Go to each
"$TUTORIAL_HOME/data" sub-folder and run the "fetch_data.py" script
from there (after having read them first).

For instance:

Loading the 20 newsgroups dataset

The dataset is called "Twenty Newsgroups". Here is the official
description, quoted from the website:

   The 20 Newsgroups data set is a collection of approximately 20,000
   newsgroup documents, partitioned (nearly) evenly across 20
   different newsgroups. To the best of our knowledge, it was
   originally collected by Ken Lang, probably for his paper
   "Newsweeder: Learning to filter netnews," though he does not
   explicitly mention this collection. The 20 newsgroups collection
   has become a popular data set for experiments in text applications
   of machine learning techniques, such as text classification and
   text clustering.

In the following we will use the built-in dataset loader for 20
newsgroups from scikit-learn. Alternatively, it is possible to
download the dataset manually from the website and use the
"sklearn.datasets.load_files()" function by pointing it to the
"20news-bydate-train" sub-folder of the uncompressed archive folder.

In order to get faster execution times for this first example, we will
work on a partial dataset with only 4 categories out of the 20
available in the dataset:

   >>> categories = ['alt.atheism', 'soc.religion.christian',
   ...               'comp.graphics', 'sci.med']

We can now load the list of files matching those categories as
follows:

   >>> from sklearn.datasets import fetch_20newsgroups
   >>> twenty_train = fetch_20newsgroups(subset='train',
   ...     categories=categories, shuffle=True, random_state=42)

The returned dataset is a "scikit-learn" "bunch": a simple holder
object with fields that can be both accessed as python "dict" keys or
"object" attributes for convenience, for instance the "target_names"
holds the list of the requested category names:

   >>> twenty_train.target_names
   ['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']

The files themselves are loaded in memory in the "data" attribute. For
reference the filenames are also available:

   >>> len(twenty_train.data)
   2257
   >>> len(twenty_train.filenames)
   2257

Let's print the first lines of the first loaded file:

   >>> print("\n".join(twenty_train.data[0].split("\n")[:3]))
   From: sd345@city.ac.uk (Michael Collier)
   Subject: Converting images to HP LaserJet III?
   Nntp-Posting-Host: hampton

   >>> print(twenty_train.target_names[twenty_train.target[0]])
   comp.graphics

Supervised learning algorithms will require a category label for each

Nntp-Posting-Host: hampton

document in the training set. In this case the category is the name of
the newsgroup which also happens to be the name of the folder holding
the individual documents.

For speed and space efficiency reasons, "scikit-learn" loads the
target attribute as an array of integers that corresponds to the index
of the category name in the "target_names" list. The category integer
id of each sample is stored in the "target" attribute:

   >>> twenty_train.target[:10]
   array([1, 1, 3, 3, 3, 3, 3, 2, 2, 2])

It is possible to get back the category names as follows:

   >>> for t in twenty_train.target[:10]:
   ...     print(twenty_train.target_names[t])
   ...
   soc.religion.christian
   sci.med
   sci.med
   sci.med

You might have noticed that the samples were shuffled randomly when we
called "fetch_20newsgroups(..., shuffle=True, random_state=42)": this
is useful if you wish to select only a subset of samples to quickly
train a model and get a first idea of the results before re-training
on the complete dataset later.

Extracting features from text files

In order to perform machine learning on text documents, we first need

to turn the text content into numerical feature vectors.

Bags of words

The most intuitive way to do so is to use a bags of words
representation:

   1. Assign a fixed integer id to each word occurring in any document
      of the training set (for instance by building a dictionary from
      words to integer indices).

   2. For each document "#i", count the number of occurrences of each
      word "w" and store it in "X[i, j]" as the value of feature "#j"
      where "j" is the index of word "w" in the dictionary.

The bags of words representation implies that "n_features" is the
number of distinct words in the corpus: this number is typically
larger than 100,000.

If "n_samples == 10000", storing "X" as a NumPy array of type float32
would require 10000 x 100000 x 4 bytes = **4GB in RAM** which is
barely manageable on today's computers.

Fortunately, **most values in X will be zeros** since for a given
document less than a few thousand distinct words will be used. For
this reason we say that bags of words are typically **high-dimensional
sparse datasets**. We can save a lot of memory by only storing the
non-zero parts of the feature vectors in memory.

"scipy.sparse" matrices are data structures that do exactly this, and
"scikit-learn" has built-in support for these structures.

Tokenizing text with "scikit-learn"

Text preprocessing, tokenizing and filtering of stopwords are all
included in "CountVectorizer", which builds a dictionary of features
and transforms documents to feature vectors:

   >>> from sklearn.feature_extraction.text import CountVectorizer
   >>> count_vect = CountVectorizer()
   >>> X_train_counts = count_vect.fit_transform(twenty_train.data)
   >>> X_train_counts.shape
   (2257, 35788)

"CountVectorizer" supports counts of N-grams of words or consecutive
characters. Once fitted, the vectorizer has built a dictionary of
feature indices:

   >>> count_vect.vocabulary_.get(u'algorithm')
   4690

The index value of a word in the vocabulary is linked to its frequency
in the whole training corpus.

From occurrences to frequencies

Occurrence count is a good start but there is an issue: longer
documents will have higher average count values than shorter
documents, even though they might talk about the same topics.

To avoid these potential discrepancies it suffices to divide the
number of occurrences of each word in a document by the total number
of words in the document: these new features are called "tf" for Term
Frequencies.

Another refinement on top of tf is to downscale weights for words that
occur in many documents in the corpus and are therefore less

informative than those that occur only in a smaller portion of the
corpus.

This downscaling is called tf–idf for "Term Frequency times Inverse
Document Frequency".

Both **tf** and **tf–idf** can be computed as follows using
"TfidfTransformer":

   >>> from sklearn.feature_extraction.text import TfidfTransformer
   >>> tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
   >>> X_train_tf = tf_transformer.transform(X_train_counts)
   >>> X_train_tf.shape

In the above example-code, we firstly use the "fit(..)" method to fit
our estimator to the data and secondly the "transform(..)" method to
transform our count-matrix to a tf-idf representation. These two steps
can be combined to achieve the same end result faster by skipping
redundant processing. This is done through using the
"fit_transform(..)" method as shown below, and as mentioned in the
note in the previous section:

   >>> tfidf_transformer = TfidfTransformer()
   >>> X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
   >>> X_train_tfidf.shape

Training a classifier

Now that we have our features, we can train a classifier to try to
predict the category of a post. Let's start with a naïve Bayes
classifier, which provides a nice baseline for this task. "scikit-

learn" includes several variants of this classifier, and the one most
suitable for word counts is the multinomial variant:

   >>> from sklearn.naive_bayes import MultinomialNB
   >>> clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

To try to predict the outcome on a new document we need to extract the
features using almost the same feature extracting chain as before. The
difference is that we call "transform" instead of "fit_transform" on
the transformers, since they have already been fit to the training
set:

   >>> docs_new = ['God is love', 'OpenGL on the GPU is fast']
   >>> X_new_counts = count_vect.transform(docs_new)
   >>> X_new_tfidf = tfidf_transformer.transform(X_new_counts)

   >>> predicted = clf.predict(X_new_tfidf)

   >>> for doc, category in zip(docs_new, predicted):
   ...     print('%r => %s' % (doc, twenty_train.target_names[category]))
   ...
   'God is love' => soc.religion.christian
   'OpenGL on the GPU is fast' => comp.graphics

Building a pipeline

In order to make the vectorizer => transformer => classifier easier to
work with, "scikit-learn" provides a "Pipeline" class that behaves
like a compound classifier:

   >>> from sklearn.pipeline import Pipeline
   >>> text_clf = Pipeline([
   ...     ('vect', CountVectorizer()),

   ...     ('tfidf', TfidfTransformer()),
   ...     ('clf', MultinomialNB()),
   ... ])

The names "vect", "tfidf" and "clf" (classifier) are arbitrary. We
will use them to perform grid search for suitable hyperparameters
below. We can now train the model with a single command:

   >>> text_clf.fit(twenty_train.data, twenty_train.target)
   Pipeline(...)

Evaluation of the performance on the test set
=============================================

Evaluating the predictive accuracy of the model is equally easy:

   >>> twenty_test = fetch_20newsgroups(subset='test',
   >>> docs_test = twenty_test.data
   >>> predicted = text_clf.predict(docs_test)
   >>> np.mean(predicted == twenty_test.target)
   0.8348...

We achieved 83.5% accuracy. Let's see if we can do better with a
linear support vector machine (SVM), which is widely regarded as one
of the best text classification algorithms (although it's also a bit
slower than naïve Bayes). We can change the learner by simply plugging
a different classifier object into our pipeline:

   ...     ('clf', SGDClassifier(loss='hinge', penalty='l2',

>>> text_clf = Pipeline([
   ...                           alpha=1e-3, random_state=42,
   ...                           max_iter=5, tol=None)),
   ... ])

   0.9101...

We achieved 91.3% accuracy using the SVM. "scikit-learn" provides
further utilities for more detailed performance analysis of the
results:

   >>> from sklearn import metrics
   >>> print(metrics.classification_report(twenty_test.target, predicted,
   ...     target_names=twenty_test.target_names))
                           precision    recall  f1-score   support

              alt.atheism       0.95      0.80      0.87       319
            comp.graphics       0.87      0.98      0.92       389
                  sci.med       0.94      0.89      0.91       396
   soc.religion.christian       0.90      0.95      0.93       398

                 accuracy                           0.91      1502
                macro avg       0.91      0.91      0.91      1502
             weighted avg       0.91      0.91      0.91      1502

   >>> metrics.confusion_matrix(twenty_test.target, predicted)
   array([[256,  11,  16,  36],
          [  4, 380,   3,   2],
          [  5,  35, 353,   3],
          [  5,  11,   4, 378]])

>>> metrics.confusion_matrix(twenty_test.target, predicted)

As expected the confusion matrix shows that posts from the newsgroups
on atheism and Christianity are more often confused for one another
than with computer graphics.

Parameter tuning using grid search

We've already encountered some parameters such as "use_idf" in the
"TfidfTransformer". Classifiers tend to have many parameters as well;
e.g., "MultinomialNB" includes a smoothing parameter "alpha" and
"SGDClassifier" has a penalty parameter "alpha" and configurable loss
and penalty terms in the objective function (see the module
documentation, or use the Python "help" function to get a description
of these).

Instead of tweaking the parameters of the various components of the
chain, it is possible to run an exhaustive search of the best
parameters on a grid of possible values. We try out all classifiers on
either words or bigrams, with or without idf, and with a penalty
parameter of either 0.01 or 0.001 for the linear SVM:

   >>> from sklearn.model_selection import GridSearchCV
   >>> parameters = {
   ...     'vect__ngram_range': [(1, 1), (1, 2)],
   ...     'tfidf__use_idf': (True, False),
   ...     'clf__alpha': (1e-2, 1e-3),
   ... }

Obviously, such an exhaustive search can be expensive. If we have
multiple CPU cores at our disposal, we can tell the grid searcher to

...     'clf__alpha': (1e-2, 1e-3),
   ... }

try these eight parameter combinations in parallel with the "n_jobs"
parameter. If we give this parameter a value of "-1", grid search will
detect how many cores are installed and use them all:

   >>> gs_clf = GridSearchCV(text_clf, parameters, cv=5, n_jobs=-1)

The grid search instance behaves like a normal "scikit-learn" model.
Let's perform the search on a smaller subset of the training data to
speed up the computation:

   >>> gs_clf = gs_clf.fit(twenty_train.data[:400], twenty_train.target[:400])

The result of calling "fit" on a "GridSearchCV" object is a classifier
that we can use to "predict":

   >>> twenty_train.target_names[gs_clf.predict(['God is love'])[0]]
   'soc.religion.christian'

The object's "best_score_" and "best_params_" attributes store the
best mean score and the parameters setting corresponding to that
score:

   >>> gs_clf.best_score_
   0.9...
   >>> for param_name in sorted(parameters.keys()):
   ...     print("%s: %r" % (param_name, gs_clf.best_params_[param_name]))
   ...
   clf__alpha: 0.001
   tfidf__use_idf: True
   vect__ngram_range: (1, 1)

A more detailed summary of the search is available at
"gs_clf.cv_results_".

The "cv_results_" parameter can be easily imported into pandas as a
"DataFrame" for further inspection.

Exercises
---------

Exercises
---------

To do the exercises, copy the content of the 'skeletons' folder as a
new folder named 'workspace':

You can then edit the content of the workspace without fear of losing
the original exercise instructions.

Then fire an ipython shell and run the work-in-progress script with:

   [1] %run workspace/exercise_XX_script.py arg1 arg2 arg3

If an exception is triggered, use "%debug" to fire-up a post mortem
ipdb session.

Refine the implementation and iterate until the exercise is solved.

**For each exercise, the skeleton file provides all the necessary
import statements, boilerplate code to load the data and sample code
to evaluate the predictive accuracy of the model.**

Exercise 1: Language identification

* Write a text classification pipeline using a custom preprocessor and
  "CharNGramAnalyzer" using data from Wikipedia articles as training
  set.

* Evaluate the performance on some held out test set.

ipython command line:

   %run workspace/exercise_01_language_train_model.py data/languages/paragraphs/

Exercise 2: Sentiment Analysis on movie reviews

* Write a text classification pipeline to classify movie reviews as
  either positive or negative.

* Find a good set of parameters using grid search.

* Evaluate the performance on a held out test set.

   %run workspace/exercise_02_sentiment.py data/movie_reviews/txt_sentoken/

Exercise 3: CLI text classification utility

Using the results of the previous exercises and the "cPickle" module
of the standard library, write a command line utility that detects the
language of some text provided on "stdin" and estimate the polarity
(positive or negative) if the text is written in English.

Bonus point if the utility is able to give a confidence level for its
predictions.

Where to from here

Here are a few suggestions to help further your scikit-learn intuition
upon the completion of this tutorial:

* Try playing around with the "analyzer" and "token normalisation"
  under "CountVectorizer".

* If you don't have labels, try using *Clustering* on your problem.

* If you have multiple labels per document, e.g categories, have a
  look at the Multiclass and multilabel section.

* Try using Truncated SVD for latent semantic analysis.

* Have a look at using *Out-of-core Classification* to learn from data
  that would not fit into the computer main memory.

* Have a look at the Hashing Vectorizer as a memory efficient
  alternative to "CountVectorizer".

Choosing the right estimator

Often the hardest part of solving a machine learning problem can be
finding the right estimator for the job.

Different estimators are better suited for different types of data and
different problems.

The flowchart below is designed to give users a bit of a rough guide
on how to approach problems with regard to which estimators to try on
your data.

Click on any estimator in the chart below to see its documentation.

Who is using scikit-learn?

J.P.Morgan

Scikit-learn is an indispensable part of the Python machine learning
toolkit at JPMorgan. It is very widely used across all parts of the
bank for classification, predictive analytics, and very many other
machine learning tasks. Its straightforward API, its breadth of
algorithms, and the quality of its documentation combine to make
scikit-learn simultaneously very approachable and very powerful.

Stephen Simmons, VP, Athena Research, JPMorgan

[image]

Spotify
=======

Scikit-learn provides a toolbox with solid implementations of a bunch
of state-of-the-art models and makes it easy to plug them into
existing applications. We've been using it quite a lot for music
recommendations at Spotify and I think it's the most well-designed ML
package I've seen so far.

Erik Bernhardsson, Engineering Manager Music Discovery & Machine
Learning, Spotify

[image]

Inria
=====

At INRIA, we use scikit-learn to support leading-edge basic research
in many teams: Parietal for neuroimaging, Lear for computer vision,
Visages for medical image analysis, Privatics for security. The
project is a fantastic tool to address difficult applications of
machine learning in an academic environment as it is performant and
versatile, but all easy-to-use and well documented, which makes it
well suited to grad students.

Gaël Varoquaux, research at Parietal

[image]

betaworks
=========

Betaworks is a NYC-based startup studio that builds new products,

[image]

betaworks
=========

grows companies, and invests in others. Over the past 8 years we've
launched a handful of social data analytics-driven services, such as
Bitly, Chartbeat, digg and Scale Model. Consistently the betaworks
data science team uses Scikit-learn for a variety of tasks. From
exploratory analysis, to product development, it is an essential part
of our toolkit. Recent uses are included in digg's new video
recommender system, and Poncho's dynamic heuristic subspace
clustering.

Gilad Lotan, Chief Data Scientist

[image]

Hugging Face

At Hugging Face we're using NLP and probabilistic models to generate
conversational Artificial intelligences that are fun to chat with.
Despite using deep neural nets for a few of our NLP tasks, scikit-
learn is still the bread-and-butter of our daily machine learning
routine. The ease of use and predictability of the interface, as well
as the straightforward mathematical explanations that are here when
you need them, is the killer feature. We use a variety of scikit-learn
models in production and they are also operationally very pleasant to
work with.

Julien Chaumond, Chief Technology Officer

[image]

Evernote
========

Building a classifier is typically an iterative process of exploring
the data, selecting the features (the attributes of the data believed
to be predictive in some way), training the models, and finally

evaluating them. For many of these tasks, we relied on the excellent
scikit-learn package for Python.

Read more

Mark Ayzenshtat, VP, Augmented Intelligence

[image]

Télécom ParisTech

At Telecom ParisTech, scikit-learn is used for hands-on sessions and
home assignments in introductory and advanced machine learning
courses. The classes are for undergrads and masters students. The
great benefit of scikit-learn is its fast learning curve that allows
students to quickly start working on interesting and motivating
problems.

Alexandre Gramfort, Assistant Professor

[image]

Booking.com

At Booking.com, we use machine learning algorithms for many different
applications, such as recommending hotels and destinations to our
customers, detecting fraudulent reservations, or scheduling our
customer service agents. Scikit-learn is one of the tools we use when
implementing standard algorithms for prediction tasks. Its API and
documentations are excellent and make it easy to use. The scikit-learn
developers do a great job of incorporating state of the art
implementations and new algorithms into the package. Thus, scikit-
learn provides convenient access to a wide spectrum of algorithms, and
allows us to readily find the right tool for the right job.

Melanie Mueller, Data Scientist

[image]

AWeber

[image]

AWeber
======

The scikit-learn toolkit is indispensable for the Data Analysis and
Management team at AWeber.  It allows us to do AWesome stuff we would
not otherwise have the time or resources to accomplish. The
documentation is excellent, allowing new engineers to quickly evaluate
and apply many different algorithms to our data. The text feature
extraction utilities are useful when working with the large volume of
email content we have at AWeber. The RandomizedPCA implementation,
along with Pipelining and FeatureUnions, allows us to develop complex
machine learning algorithms efficiently and reliably.

Anyone interested in learning more about how AWeber deploys scikit-
learn in a production environment should check out talks from PyData
Boston by AWeber's Michael Becker available at
https://github.com/mdbecker/pydata_2013

Michael Becker, Software Engineer, Data Analysis and Management Ninjas

[image]

Yhat
====

The combination of consistent APIs, thorough documentation, and top
notch implementation make scikit-learn our favorite machine learning
package in Python. scikit-learn makes doing advanced analysis in
Python accessible to anyone. At Yhat, we make it easy to integrate
these models into your production applications. Thus eliminating the
unnecessary dev time encountered productionizing analytical work.

Greg Lamp, Co-founder Yhat

[image]

Rangespan
=========

The Python scikit-learn toolkit is a core tool in the data science
group at Rangespan. Its large collection of well documented models and
algorithms allow our team of data scientists to prototype fast and
quickly iterate to find the right solution to our learning problems.
We find that scikit-learn is not only the right tool for prototyping,
but its careful and well tested implementation give us the confidence
to run scikit-learn models in production.

Jurgen Van Gael, Data Science Director at Rangespan Ltd

[image]

Birchbox
========

At Birchbox, we face a range of machine learning problems typical to
E-commerce: product recommendation, user clustering, inventory
prediction, trends detection, etc. Scikit-learn lets us experiment
with many models, especially in the exploration phase of a new
project: the data can be passed around in a consistent way; models are
easy to save and reuse; updates keep us informed of new developments
from the pattern discovery research community. Scikit-learn is an
important tool for our team, built the right way in the right
language.

Thierry Bertin-Mahieux, Birchbox, Data Scientist

[image]

Bestofmedia Group

Scikit-learn is our #1 toolkit for all things machine learning at
Bestofmedia. We use it for a variety of tasks (e.g. spam fighting, ad

[image]

click prediction, various ranking models) thanks to the varied, state-
of-the-art algorithm implementations packaged into it. In the lab it
accelerates prototyping of complex pipelines. In production I can say
it has proven to be robust and efficient enough to be deployed for
business critical components.

Eustache Diemert, Lead Scientist Bestofmedia Group

[image]

Change.org

At change.org we automate the use of scikit-learn's
RandomForestClassifier in our production systems to drive email
targeting that reaches millions of users across the world each week.
In the lab, scikit-learn's ease-of-use, performance, and overall
variety of algorithms implemented has proved invaluable in giving us a
single reliable source to turn to for our machine-learning needs.

Vijay Ramesh, Software Engineer in Data/science at Change.org

[image]

PHIMECA Engineering

At PHIMECA Engineering, we use scikit-learn estimators as surrogates
for expensive-to-evaluate numerical models (mostly but not exclusively
finite-element mechanical models) for speeding up the intensive post-
processing operations involved in our simulation-based decision making
framework. Scikit-learn's fit/predict API together with its efficient
cross-validation tools considerably eases the task of selecting the

best-fit estimator. We are also using scikit-learn for illustrating
concepts in our training sessions. Trainees are always impressed by
the ease-of-use of scikit-learn despite the apparent theoretical
complexity of machine learning.

Vincent Dubourg, PHIMECA Engineering, PhD Engineer

[image]

HowAboutWe

At HowAboutWe, scikit-learn lets us implement a wide array of machine
learning techniques in analysis and in production, despite having a
small team.  We use scikit-learn's classification algorithms to
predict user behavior, enabling us to (for example) estimate the value
of leads from a given traffic source early in the lead's tenure on our
site. Also, our users' profiles consist of primarily unstructured data
(answers to open-ended questions), so we use scikit-learn's feature
extraction and dimensionality reduction tools to translate these
unstructured data into inputs for our matchmaking system.

Daniel Weitzenfeld, Senior Data Scientist at HowAboutWe

[image]

PeerIndex
=========

At PeerIndex we use scientific methodology to build the Influence
Graph - a unique dataset that allows us to identify who's really
influential and in which context. To do this, we have to tackle a
range of machine learning and predictive modeling problems. Scikit-
learn has emerged as our primary tool for developing prototypes and

making quick progress. From predicting missing data and classifying
tweets to clustering communities of social media users, scikit- learn
proved useful in a variety of applications. Its very intuitive
interface and excellent compatibility with other python tools makes it
and indispensable tool in our daily research efforts.

Ferenc Huszar - Senior Data Scientist at Peerindex

[image]

DataRobot
=========

DataRobot is building next generation predictive analytics software to
make data scientists more productive, and scikit-learn is an integral
part of our system. The variety of machine learning techniques in
combination with the solid implementations that scikit-learn offers
makes it a one-stop-shopping library for machine learning in Python.
Moreover, its consistent API, well-tested code and permissive
licensing allow us to use it in a production environment. Scikit-learn
has literally saved us years of work we would have had to do ourselves
to bring our product to market.

Jeremy Achin, CEO & Co-founder DataRobot Inc.

[image]

OkCupid
=======

We're using scikit-learn at OkCupid to evaluate and improve our
matchmaking system. The range of features it has, especially
preprocessing utilities, means we can use it for a wide variety of
projects, and it's performant enough to handle the volume of data that

we need to sort through. The documentation is really thorough, as
well, which makes the library quite easy to use.

David Koh - Senior Data Scientist at OkCupid

[image]

Lovely
======

At Lovely, we strive to deliver the best apartment marketplace, with
respect to our users and our listings. From understanding user
behavior, improving data quality, and detecting fraud, scikit-learn is
a regular tool for gathering insights, predictive modeling and
improving our product. The easy-to-read documentation and intuitive
architecture of the API makes machine learning both explorable and
accessible to a wide range of python developers. I'm constantly
recommending that more developers and scientists try scikit-learn.

Simon Frid - Data Scientist, Lead at Lovely

[image]

Data Publica

Data Publica builds a new predictive sales tool for commercial and
marketing teams called C-Radar. We extensively use scikit-learn to
build segmentations of customers through clustering, and to predict
future customers based on past partnerships success or failure. We
also categorize companies using their website communication thanks to
scikit-learn and its machine learning algorithm implementations.
Eventually, machine learning makes it possible to detect weak signals
that traditional tools cannot see. All these complex tasks are

performed in an easy and straightforward way thanks to the great
quality of the scikit-learn framework.

Guillaume Lebourgeois & Samuel Charron - Data Scientists at Data
Publica

[image]

Machinalis

Scikit-learn is the cornerstone of all the machine learning projects
carried at Machinalis. It has a consistent API, a wide selection of
algorithms and lots of auxiliary tools to deal with the boilerplate.
We have used it in production environments on a variety of projects
including click-through rate prediction, information extraction, and
even counting sheep!

In fact, we use it so much that we've started to freeze our common use
cases into Python packages, some of them open-sourced, like
FeatureForge . Scikit-learn in one word: Awesome.

Rafael Carrascosa, Lead developer

[image]

solido
======

Scikit-learn is helping to drive Moore's Law, via Solido. Solido
creates computer-aided design tools used by the majority of top-20
semiconductor companies and fabs, to design the bleeding-edge chips
inside smartphones, automobiles, and more. Scikit-learn helps to power
Solido's algorithms for rare-event estimation, worst-case
verification, optimization, and more. At Solido, we are particularly
fond of scikit-learn's libraries for Gaussian Process models, large-

scale regularized linear regression, and classification. Scikit-learn
has increased our productivity, because for many ML problems we no
longer need to “roll our own” code. This PyData 2014 talk has details.

Trent McConaghy, founder, Solido Design Automation Inc.

[image]

INFONEA
=======

We employ scikit-learn for rapid prototyping and custom-made Data
Science solutions within our in-memory based Business Intelligence
Software INFONEA®. As a well-documented and comprehensive collection
of state-of-the-art algorithms and pipelining methods, scikit-learn
enables us to provide flexible and scalable scientific analysis
solutions. Thus, scikit-learn is immensely valuable in realizing a
powerful integration of Data Science technology within self-service
business analytics.

Thorsten Kranz, Data Scientist, Coma Soft AG.

[image]

Dataiku
=======

Our software, Data Science Studio (DSS), enables users to create data
services that combine ETL with Machine Learning. Our Machine Learning
module integrates many scikit-learn algorithms. The scikit-learn
library is a perfect integration with DSS because it offers algorithms
for virtually all business cases. Our goal is to offer a transparent
and flexible tool that makes it easier to optimize time consuming
aspects of building a data service, preparing data, and training

machine learning algorithms on all types of data.

Florian Douetteau, CEO, Dataiku

[image]

Otto Group

Here at Otto Group, one of global Big Five B2C online retailers, we
are using scikit-learn in all aspects of our daily work from data
exploration to development of machine learning application to the
productive deployment of those services. It helps us to tackle machine
learning problems ranging from e-commerce to logistics. It consistent
APIs enabled us to build the Palladium REST-API framework around it
and continuously deliver scikit-learn based services.

Christian Rammig, Head of Data Science, Otto Group

[image]

Zopa
====

At Zopa, the first ever Peer-to-Peer lending platform, we extensively
use scikit-learn to run the business and optimize our users'
experience. It powers our Machine Learning models involved in credit
risk, fraud risk, marketing, and pricing, and has been used for
originating at least 1 billion GBP worth of Zopa loans. It is very
well documented, powerful, and simple to use. We are grateful for the
capabilities it has provided, and for allowing us to deliver on our
mission of making money simple and fair.

Vlasios Vasileiou, Head of Data Science, Zopa

[image]

MARS
====

Scikit-Learn is integral to the Machine Learning Ecosystem at Mars.

[image]

MARS
====

Whether we're designing better recipes for petfood or closely
analysing our cocoa supply chain, Scikit-Learn is used as a tool for
rapidly prototyping ideas and taking them to production. This allows
us to better understand and meet the needs of our consumers worldwide.
Scikit-Learn's feature-rich toolset is easy to use and equips our
associates with the capabilities they need to solve the business
challenges they face every day.

Michael Fitzke Next Generation Technologies Sr Leader, Mars Inc.

[image]

BNP Paribas Cardif

BNP Paribas Cardif uses scikit-learn for several of its machine
learning models in production. Our internal community of developers
and data scientists has been using scikit-learn since 2015, for
several reasons: the quality of the developments, documentation and
contribution governance, and the sheer size of the contributing
community. We even explicitly mention the use of scikit-learn's
pipelines in our internal model risk governance as one of our good
practices to decrease operational risks and overfitting risk. As a way
to support open source software development and in particular scikit-
learn project, we decided to participate to scikit-learn's consortium
at La Fondation Inria since its creation in 2018.

Sébastien Conort, Chief Data Scientist, BNP Paribas Cardif

[image]

7.2. Real world datasets

scikit-learn provides tools to load larger datasets, downloading them
if necessary.

They can be loaded using the following functions:

7.4. Loading other datasets
***************************

7.4.1. Sample images

Scikit-learn also embeds a couple of sample JPEG images published
under Creative Commons license by their authors. Those images can be
useful to test algorithms and pipelines on 2D data.

[image]

Warning:

  The default coding of images is based on the "uint8" dtype to spare
  memory. Often machine learning algorithms work best if the input is
  converted to a floating point representation first. Also, if you
  plan to use "matplotlib.pyplpt.imshow", don't forget to scale to the
  range 0 - 1 as done in the following example.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_color_quantization.py*

7.4.2. Datasets in svmlight / libsvm format

scikit-learn includes utility functions for loading datasets in the
svmlight / libsvm format. In this format, each line takes the form
"<label> <feature-id>:<feature-value> <feature-id>:<feature-value>
...". This format is especially suitable for sparse datasets. In this
module, scipy sparse CSR matrices are used for "X" and numpy arrays
are used for "y".

You may load a dataset like as follows:

   >>> from sklearn.datasets import load_svmlight_file
   >>> X_train, y_train = load_svmlight_file("/path/to/train_dataset.txt")
   ...                                                         

You may also load two (or more) datasets at once:

   >>> X_train, y_train, X_test, y_test = load_svmlight_files(

...                                                         

   ...     ("/path/to/train_dataset.txt", "/path/to/test_dataset.txt"))
   ...                                                         

In this case, "X_train" and "X_test" are guaranteed to have the same
number of features. Another way to achieve the same result is to fix
the number of features:

   >>> X_test, y_test = load_svmlight_file(
   ...     "/path/to/test_dataset.txt", n_features=X_train.shape[1])
   ...                                                         

Related links:
^^^^^^^^^^^^^^

: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets

: https://github.com/mblondel/svmlight-loader

7.4.3. Downloading datasets from the openml.org repository
==========================================================

openml.org is a public repository for machine learning data and
experiments, that allows everybody to upload open datasets.

The "sklearn.datasets" package is able to download datasets from the
repository using the function "sklearn.datasets.fetch_openml()".

For example, to download a dataset of gene expressions in mice brains:

   >>> from sklearn.datasets import fetch_openml
   >>> mice = fetch_openml(name='miceprotein', version=4, parser="auto")

To fully specify a dataset, you need to provide a name and a version,
though the version is optional, see Dataset Versions below. The

dataset contains a total of 1080 examples belonging to 8 different
classes:

   >>> mice.data.shape
   (1080, 77)
   >>> mice.target.shape
   (1080,)
   >>> np.unique(mice.target)
   array(['c-CS-m', 'c-CS-s', 'c-SC-m', 'c-SC-s', 't-CS-m', 't-CS-s', 't-SC-m', 't-SC-s'], dtype=object)

You can get more information on the dataset by looking at the "DESCR"
and "details" attributes:

   >>> print(mice.DESCR) 
   **Author**: Clara Higuera, Katheleen J. Gardiner, Krzysztof J. Cios
   **Source**: [UCI](https://archive.ics.uci.edu/ml/datasets/Mice+Protein+Expression) - 2015
   **Please cite**: Higuera C, Gardiner KJ, Cios KJ (2015) Self-Organizing
   Feature Maps Identify Proteins Critical to Learning in a Mouse Model of Down
   Syndrome. PLoS ONE 10(6): e0129126...

   >>> mice.details 
   {'id': '40966', 'name': 'MiceProtein', 'version': '4', 'format': 'ARFF',
   'upload_date': '2017-11-08T16:00:15', 'licence': 'Public',
   'url': 'https://www.openml.org/data/v1/download/17928620/MiceProtein.arff',
   'file_id': '17928620', 'default_target_attribute': 'class',
   'row_id_attribute': 'MouseID',
   'ignore_attribute': ['Genotype', 'Treatment', 'Behavior'],
   'tag': ['OpenML-CC18', 'study_135', 'study_98', 'study_99'],
   'visibility': 'public', 'status': 'active',
   'md5_checksum': '3c479a6885bfa0438971388283a1ce32'}

The "DESCR" contains a free-text description of the data, while

'visibility': 'public', 'status': 'active',

"details" contains a dictionary of meta-data stored by openml, like
the dataset id. For more details, see the OpenML documentation The
"data_id" of the mice protein dataset is 40966, and you can use this
(or the name) to get more information on the dataset on the openml
website:

   >>> mice.url
   'https://www.openml.org/d/40966'

The "data_id" also uniquely identifies a dataset from OpenML:

   >>> mice = fetch_openml(data_id=40966, parser="auto")
   {'id': '4550', 'name': 'MiceProtein', 'version': '1', 'format': 'ARFF',
   'creator': ...,
   'upload_date': '2016-02-17T14:32:49', 'licence': 'Public', 'url':
   'https://www.openml.org/data/v1/download/1804243/MiceProtein.ARFF', 'file_id':
   '1804243', 'default_target_attribute': 'class', 'citation': 'Higuera C,
   Gardiner KJ, Cios KJ (2015) Self-Organizing Feature Maps Identify Proteins
   Critical to Learning in a Mouse Model of Down Syndrome. PLoS ONE 10(6):
   e0129126. [Web Link] journal.pone.0129126', 'tag': ['OpenML100', 'study_14',
   'study_34'], 'visibility': 'public', 'status': 'active', 'md5_checksum':
   '3c479a6885bfa0438971388283a1ce32'}

7.4.3.1. Dataset Versions

A dataset is uniquely specified by its "data_id", but not necessarily
by its name. Several different "versions" of a dataset with the same

name can exist which can contain entirely different datasets. If a
particular version of a dataset has been found to contain significant
issues, it might be deactivated. Using a name to specify a dataset
will yield the earliest version of a dataset that is still active.
That means that "fetch_openml(name="miceprotein", parser="auto")" can
yield different results at different times if earlier versions become
inactive. You can see that the dataset with "data_id" 40966 that we
fetched above is the first version of the "miceprotein" dataset:

   >>> mice.details['version']  
   '1'

In fact, this dataset only has one version. The iris dataset on the
other hand has multiple versions:

   >>> iris = fetch_openml(name="iris", parser="auto")
   >>> iris.details['version']  
   '1'
   >>> iris.details['id']  
   '61'

   >>> iris_61 = fetch_openml(data_id=61, parser="auto")
   >>> iris_61.details['version']
   '1'
   >>> iris_61.details['id']
   '61'

   >>> iris_969 = fetch_openml(data_id=969, parser="auto")
   >>> iris_969.details['version']
   '3'
   >>> iris_969.details['id']
   '969'

Specifying the dataset by the name "iris" yields the lowest version,
version 1, with the "data_id" 61. To make sure you always get this
exact dataset, it is safest to specify it by the dataset "data_id".

The other dataset, with "data_id" 969, is version 3 (version 2 has
become inactive), and contains a binarized version of the data:

   >>> np.unique(iris_969.target)
   array(['N', 'P'], dtype=object)

You can also specify both the name and the version, which also
uniquely identifies the dataset:

   >>> iris_version_3 = fetch_openml(name="iris", version=3, parser="auto")
   >>> iris_version_3.details['version']
   '3'
   >>> iris_version_3.details['id']
   '969'

References:
^^^^^^^^^^^

* >>:arxiv:`Vanschoren, van Rijn, Bischl and Torgo. "OpenML: networked
  science in machine learning" ACM SIGKDD Explorations Newsletter,
  15(2), 49-60, 2014. <1407.7722>`<<

7.4.3.2. ARFF parser

From version 1.2, scikit-learn provides a new keyword argument
*parser* that provides several options to parse the ARFF files
provided by OpenML. The legacy parser (i.e. *parser="liac-arff"*) is
based on the project LIAC-ARFF. This parser is however slow and
consume more memory than required. A new parser based on pandas (i.e.
*parser="pandas"*) is both faster and more memory efficient. However,
this parser does not support sparse data. Therefore, we recommend
using *parser="auto"* which will use the best parser available for the
requested dataset.

The *"pandas"* and *"liac-arff"* parsers can lead to different data
types in the output. The notable differences are the following:

* The *"liac-arff"* parser always encodes categorical features as
  *str* objects. To the contrary, the *"pandas"* parser instead infers
  the type while reading and numerical categories will be casted into
  integers whenever possible.

* The *"liac-arff"* parser uses float64 to encode numerical features
  tagged as 'REAL' and 'NUMERICAL' in the metadata. The *"pandas"*
  parser instead infers if these numerical features corresponds to
  integers and uses panda's Integer extension dtype.

* In particular, classification datasets with integer categories are
  typically loaded as such *(0, 1, ...)* with the *"pandas"* parser
  while *"liac-arff"* will force the use of string encoded class
  labels such as *"0"*, *"1"* and so on.

* The *"pandas"* parser will not strip single quotes - i.e. *'* - from
  string columns. For instance, a string *'my string'* will be kept as
  is while the *"liac-arff"* parser will strip the single quotes. For
  categorical columns, the single quotes are stripped from the values.

In addition, when *as_frame=False* is used, the *"liac-arff"* parser
returns ordinally encoded data where the categories are provided in
the attribute *categories* of the *Bunch* instance. Instead,
*"pandas"* returns a NumPy array were the categories. Then it's up to
the user to design a feature engineering pipeline with an instance of

*OneHotEncoder* or *OrdinalEncoder* typically wrapped in a
*ColumnTransformer* to preprocess the categorical columns explicitly.
See for instance: *sphx_glr_auto_examples_compose_plot_column_transfo
rmer_mixed_types.py*.

7.4.4. Loading from external datasets

scikit-learn works on any numeric data stored as numpy arrays or scipy
sparse matrices. Other types that are convertible to numeric arrays
such as pandas DataFrame are also acceptable.

Here are some recommended ways to load standard columnar data into a
format usable by scikit-learn:

* pandas.io provides tools to read data from common formats including
  CSV, Excel, JSON and SQL. DataFrames may also be constructed from
  lists of tuples or dicts. Pandas handles heterogeneous data smoothly
  and provides tools for manipulation and conversion into a numeric
  array suitable for scikit-learn.

* scipy.io specializes in binary formats often used in scientific
  computing context such as .mat and .arff

* numpy/routines.io for standard loading of columnar data into numpy
  arrays

* scikit-learn's "datasets.load_svmlight_file()" for the svmlight or
  libSVM sparse format

* scikit-learn's "datasets.load_files()" for directories of text files
  where the name of each directory is the name of each category and

libSVM sparse format

  each file inside of each directory corresponds to one sample from
  that category

For some miscellaneous data such as images, videos, and audio, you may
wish to refer to:

* skimage.io or Imageio for loading images and videos into numpy
  arrays

* scipy.io.wavfile.read for reading WAV files into a numpy array

Categorical (or nominal) features stored as strings (common in pandas
DataFrames) will need converting to numerical features using
"OneHotEncoder" or "OrdinalEncoder" or similar. See Preprocessing
data.

Note: if you manage your own numerical data it is recommended to use
an optimized file format such as HDF5 to reduce data load times.
Various libraries such as H5Py, PyTables and pandas provides a Python
interface for reading and writing data in that format.

7.1. Toy datasets

scikit-learn comes with a few small standard datasets that do not
require to download any file from some external website.

These datasets are useful to quickly illustrate the behavior of the
various algorithms implemented in scikit-learn. They are however often
too small to be representative of real world machine learning tasks.

7.3. Generated datasets

In addition, scikit-learn includes various random sample generators
that can be used to build artificial datasets of controlled size and
complexity.

7.3.1. Generators for classification and clustering
===================================================

These generators produce a matrix of features and corresponding
discrete targets.

7.3.1.1. Single label

Both "make_blobs()" and "make_classification()" create multiclass
datasets by allocating each class one or more normally-distributed
clusters of points.  "make_blobs()" provides greater control regarding
the centers and standard deviations of each cluster, and is used to
demonstrate clustering. "make_classification()" specializes in
introducing noise by way of: correlated, redundant and uninformative
features; multiple Gaussian clusters per class; and linear
transformations of the feature space.

"make_gaussian_quantiles()" divides a single Gaussian cluster into
near-equal-size classes separated by concentric hyperspheres.
"make_hastie_10_2()" generates a similar binary, 10-dimensional
problem.

[image]

"make_circles()" and "make_moons()" generate 2d binary classification
datasets that are challenging to certain algorithms (e.g. centroid-
based clustering or linear classification), including optional
Gaussian noise. They are useful for visualization. "make_circles()"
produces Gaussian data with a spherical decision boundary for binary

classification, while "make_moons()" produces two interleaving half
circles.

7.3.1.2. Multilabel

"make_multilabel_classification()" generates random samples with
multiple labels, reflecting a bag of words drawn from a mixture of
topics. The number of topics for each document is drawn from a Poisson
distribution, and the topics themselves are drawn from a fixed random
distribution. Similarly, the number of words is drawn from Poisson,
with words drawn from a multinomial, where each topic defines a
probability distribution over words. Simplifications with respect to
true bag-of-words mixtures include:

* Per-topic word distributions are independently drawn, where in
  reality all would be affected by a sparse base distribution, and
  would be correlated.

* For a document generated from multiple topics, all topics are
  weighted equally in generating its bag of words.

* Documents without labels words at random, rather than from a base
  distribution.

[image]

7.3.1.3. Biclustering

7.3.2. Generators for regression

"make_regression()" produces regression targets as an optionally-
sparse random linear combination of random features, with noise. Its
informative features may be uncorrelated, or low rank (few features

account for most of the variance).

Other regression generators generate functions deterministically from
randomized features.  "make_sparse_uncorrelated()" produces a target
as a linear combination of four features with fixed coefficients.
Others encode explicitly non-linear relations: "make_friedman1()" is
related by polynomial and sine transforms; "make_friedman2()" includes
feature multiplication and reciprocation; and "make_friedman3()" is
similar with an arctan transformation on the target.

7.3.3. Generators for manifold learning

7.3.4. Generators for decomposition

"{{module}}".{{objname}} {{ underline }}==============

Warning:

  **DEPRECATED**

"{{module}}".{{objname}} {{ underline }}====================

Warning:

{{index}} {{summary}} {{extended_summary}} {{parameters}} {{returns}}
{{yields}} {{other_parameters}} {{attributes}} {{raises}} {{warns}}
{{warnings}} {{see_also}} {{notes}} {{references}} {{examples}}
{{methods}}

"{{module}}".{{objname}} {{ underline }}===============

Warning:

8.3. Parallelism, resource management, and configuration
********************************************************

8.3.1. Parallelism

Some scikit-learn estimators and utilities parallelize costly
operations using multiple CPU cores.

Depending on the type of estimator and sometimes the values of the
constructor parameters, this is either done:

* with higher-level parallelism via joblib.

* with lower-level parallelism via OpenMP, used in C or Cython code.

* with lower-level parallelism via BLAS, used by NumPy and SciPy for
  generic operations on arrays.

The *n_jobs* parameters of estimators always controls the amount of
parallelism managed by joblib (processes or threads depending on the
joblib backend). The thread-level parallelism managed by OpenMP in
scikit-learn's own Cython code or by BLAS & LAPACK libraries used by
NumPy and SciPy operations used in scikit-learn is always controlled
by environment variables or *threadpoolctl* as explained below. Note
that some estimators can leverage all three kinds of parallelism at
different points of their training and prediction methods.

We describe these 3 types of parallelism in the following subsections
in more details.

8.3.1.1. Higher-level parallelism with joblib

When the underlying implementation uses joblib, the number of workers
(threads or processes) that are spawned in parallel can be controlled
via the "n_jobs" parameter.

Note:

Note:

  Where (and how) parallelization happens in the estimators using
  joblib by specifying *n_jobs* is currently poorly documented. Please
  help us by improving our docs and tackle issue 14228!

Joblib is able to support both multi-processing and multi-threading.
Whether joblib chooses to spawn a thread or a process depends on the
**backend** that it's using.

scikit-learn generally relies on the "loky" backend, which is joblib's
default backend. Loky is a multi-processing backend. When doing multi-
processing, in order to avoid duplicating the memory in each process
(which isn't reasonable with big datasets), joblib will create a
memmap that all processes can share, when the data is bigger than 1MB.

In some specific cases (when the code that is run in parallel releases
the GIL), scikit-learn will indicate to "joblib" that a multi-
threading backend is preferable.

As a user, you may control the backend that joblib will use
(regardless of what scikit-learn recommends) by using a context
manager:

   from joblib import parallel_backend

   with parallel_backend('threading', n_jobs=2):
       # Your scikit-learn code here

Please refer to the joblib's docs for more details.

In practice, whether parallelism is helpful at improving runtime
depends on many factors. It is usually a good idea to experiment

rather than assuming that increasing the number of workers is always a
good thing. In some cases it can be highly detrimental to performance
to run multiple copies of some estimators or functions in parallel
(see oversubscription below).

8.3.1.2. Lower-level parallelism with OpenMP
--------------------------------------------

OpenMP is used to parallelize code written in Cython or C, relying on
multi-threading exclusively. By default, the implementations using
OpenMP will use as many threads as possible, i.e. as many threads as
logical cores.

You can control the exact number of threads that are used either:

   * via the "OMP_NUM_THREADS" environment variable, for instance
     when: running a python script:

   * or via *threadpoolctl* as explained by this piece of
     documentation.

8.3.1.3. Parallel NumPy and SciPy routines from numerical libraries
-------------------------------------------------------------------

scikit-learn relies heavily on NumPy and SciPy, which internally call
multi-threaded linear algebra routines (BLAS & LAPACK) implemented in
libraries such as MKL, OpenBLAS or BLIS.

You can control the exact number of threads used by BLAS for each
library using environment variables, namely:

   * "MKL_NUM_THREADS" sets the number of thread MKL uses,

   * "OPENBLAS_NUM_THREADS" sets the number of threads OpenBLAS uses

   * "BLIS_NUM_THREADS" sets the number of threads BLIS uses

Note that BLAS & LAPACK implementations can also be impacted by
*OMP_NUM_THREADS*. To check whether this is the case in your
environment, you can inspect how the number of threads effectively
used by those libraries is affected when running the the following
command in a bash or zsh terminal for different values of
*OMP_NUM_THREADS*:

   .. prompt:: bash $

   OMP_NUM_THREADS=2 python -m threadpoolctl -i numpy scipy

Note:

  At the time of writing (2022), NumPy and SciPy packages which are
  distributed on pypi.org (i.e. the ones installed via "pip install")
  and on the conda-forge channel (i.e. the ones installed via "conda
  install --channel conda-forge") are linked with OpenBLAS, while
  NumPy and SciPy packages packages shipped on the "defaults" conda
  channel from Anaconda.org (i.e. the ones installed via "conda
  install") are linked by default with MKL.

8.3.1.4. Oversubscription: spawning too many threads

It is generally recommended to avoid using significantly more
processes or threads than the number of CPUs on a machine. Over-
subscription happens when a program is running too many threads at the
same time.

same time.

Suppose you have a machine with 8 CPUs. Consider a case where you're
running a "GridSearchCV" (parallelized with joblib) with "n_jobs=8"
over a "HistGradientBoostingClassifier" (parallelized with OpenMP).
Each instance of "HistGradientBoostingClassifier" will spawn 8 threads
(since you have 8 CPUs). That's a total of "8 * 8 = 64" threads, which
leads to oversubscription of threads for physical CPU resources and
thus to scheduling overhead.

Oversubscription can arise in the exact same fashion with parallelized
routines from MKL, OpenBLAS or BLIS that are nested in joblib calls.

Starting from "joblib >= 0.14", when the "loky" backend is used (which
is the default), joblib will tell its child **processes** to limit the
number of threads they can use, so as to avoid oversubscription. In
practice the heuristic that joblib uses is to tell the processes to
use "max_threads = n_cpus // n_jobs", via their corresponding
environment variable. Back to our example from above, since the joblib
backend of "GridSearchCV" is "loky", each process will only be able to
use 1 thread instead of 8, thus mitigating the oversubscription issue.

Note that:

* Manually setting one of the environment variables
  ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", or
  "BLIS_NUM_THREADS") will take precedence over what joblib tries to

Note that:

  do. The total number of threads will be "n_jobs *
  <LIB>_NUM_THREADS". Note that setting this limit will also impact
  your computations in the main process, which will only use
  "<LIB>_NUM_THREADS". Joblib exposes a context manager for finer
  control over the number of threads in its workers (see joblib docs
  linked below).

* When joblib is configured to use the "threading" backend, there is
  no mechanism to avoid oversubscriptions when calling into parallel
  native libraries in the joblib-managed threads.

* All scikit-learn estimators that explicitly rely on OpenMP in their
  Cython code always use *threadpoolctl* internally to automatically
  adapt the numbers of threads used by OpenMP and potentially nested
  BLAS calls so as to avoid oversubscription.

You will find additional details about joblib mitigation of
oversubscription in joblib documentation.

You will find additional details about parallelism in numerical python
libraries in this document from Thomas J. Fan.

8.3.2. Configuration switches

8.3.2.1. Python API

"sklearn.set_config()" and "sklearn.config_context()" can be used to
change parameters of the configuration which control aspect of
parallelism.

8.3.2.2. Environment variables

These environment variables should be set before importing scikit-
learn.

8.3.2.2.1. *SKLEARN_ASSUME_FINITE*

Sets the default value for the *assume_finite* argument of
"sklearn.set_config()".

8.3.2.2.2. *SKLEARN_WORKING_MEMORY*

Sets the default value for the *working_memory* argument of

8.3.2.2.3. *SKLEARN_SEED*
~~~~~~~~~~~~~~~~~~~~~~~~~

Sets the seed of the global random generator when running the tests,
for reproducibility.

Note that scikit-learn tests are expected to run deterministically
with explicit seeding of their own independent RNG instances instead
of relying on the numpy or Python standard library RNG singletons to
make sure that test results are independent of the test execution
order. However some tests might forget to use explicit seeding and
this variable is a way to control the initial state of the
aforementioned singletons.

8.3.2.2.4. *SKLEARN_TESTS_GLOBAL_RANDOM_SEED*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Controls the seeding of the random number generator used in tests that
rely on the *global_random_seed`* fixture.

All tests that use this fixture accept the contract that they should
deterministically pass for any seed value from 0 to 99 included.

If the *SKLEARN_TESTS_GLOBAL_RANDOM_SEED* environment variable is set

to *"any"* (which should be the case on nightly builds on the CI), the
fixture will choose an arbitrary seed in the above range (based on the
BUILD_NUMBER or the current day) and all fixtured tests will run for
that specific seed. The goal is to ensure that, over time, our CI will
run all tests with different seeds while keeping the test duration of
a single run of the full test suite limited. This will check that the
assertions of tests written to use this fixture are not dependent on a
specific seed value.

The range of admissible seed values is limited to [0, 99] because it
is often not possible to write a test that can work for any possible
seed and we want to avoid having tests that randomly fail on the CI.

Valid values for *SKLEARN_TESTS_GLOBAL_RANDOM_SEED*:

* *SKLEARN_TESTS_GLOBAL_RANDOM_SEED="42"*: run tests with a fixed seed
  of 42

* *SKLEARN_TESTS_GLOBAL_RANDOM_SEED="40-42"*: run the tests with all
  seeds between 40 and 42 included

* *SKLEARN_TESTS_GLOBAL_RANDOM_SEED="any"*: run the tests with an
  arbitrary seed selected between 0 and 99 included

* *SKLEARN_TESTS_GLOBAL_RANDOM_SEED="all"*: run the tests with all
  seeds between 0 and 99 included. This can take a long time: only use
  for individual tests, not the full test suite!

If the variable is not set, then 42 is used as the global seed in a

seeds between 0 and 99 included. This can take a long time: only use

deterministic manner. This ensures that, by default, the scikit-learn
test suite is as deterministic as possible to avoid disrupting our
friendly third-party package maintainers. Similarly, this variable
should not be set in the CI config of pull-requests to make sure that
our friendly contributors are not the first people to encounter a
seed-sensitivity regression in a test unrelated to the changes of
their own PR. Only the scikit-learn maintainers who watch the results
of the nightly builds are expected to be annoyed by this.

When writing a new test function that uses this fixture, please use
the following command to make sure that it passes deterministically
for all admissible seeds on your local machine:

8.3.2.2.5. *SKLEARN_SKIP_NETWORK_TESTS*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When this environment variable is set to a non zero value, the tests
that need network access are skipped. When this environment variable
is not set then network tests are skipped.

8.3.2.2.6. *SKLEARN_RUN_FLOAT32_TESTS*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When this environment variable is set to '1', the tests using the
*global_dtype* fixture are also run on float32 data. When this
environment variable is not set, the tests are only run on float64
data.

8.3.2.2.7. *SKLEARN_ENABLE_DEBUG_CYTHON_DIRECTIVES*

data.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When this environment variable is set to a non zero value, the
*Cython* derivative, *boundscheck* is set to *True*. This is useful
for finding segfaults.

8.3.2.2.8. *SKLEARN_BUILD_ENABLE_DEBUG_SYMBOLS*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When this environment variable is set to a non zero value, the debug
symbols will be included in the compiled C extensions. Only debug
symbols for POSIX systems is configured.

8.3.2.2.9. *SKLEARN_PAIRWISE_DIST_CHUNK_SIZE*

This sets the size of chunk to be used by the underlying
*PairwiseDistancesReductions* implementations. The default value is
*256* which has been showed to be adequate on most machines.

Users looking for the best performance might want to tune this
variable using powers of 2 so as to get the best parallelism behavior
for their hardware, especially with respect to their caches' sizes.

8.1. Strategies to scale computationally: bigger data

For some applications the amount of examples, features (or both)
and/or the speed at which they need to be processed are challenging
for traditional approaches. In these cases scikit-learn has a number
of options you can consider to make your system scale.

8.1.1. Scaling with instances using out-of-core learning

Out-of-core (or "external memory") learning is a technique used to
learn from data that cannot fit in a computer's main memory (RAM).

Here is a sketch of a system designed to achieve this goal:

   1. a way to stream instances

   2. a way to extract features from instances

   3. an incremental algorithm

8.1.1.1. Streaming instances

Basically, 1. may be a reader that yields instances from files on a
hard drive, a database, from a network stream etc. However, details on
how to achieve this are beyond the scope of this documentation.

8.1.1.2. Extracting features

2. could be any relevant way to extract features among the different
feature extraction methods supported by scikit-learn. However, when
working with data that needs vectorization and where the set of
features or values is not known in advance one should take explicit
care. A good example is text classification where unknown terms are
likely to be found during training. It is possible to use a stateful

vectorizer if making multiple passes over the data is reasonable from
an application point of view. Otherwise, one can turn up the
difficulty by using a stateless feature extractor. Currently the
preferred way to do this is to use the so-called hashing trick as
implemented by "sklearn.feature_extraction.FeatureHasher" for datasets
with categorical variables represented as list of Python dicts or
"sklearn.feature_extraction.text.HashingVectorizer" for text
documents.

8.1.1.3. Incremental learning

Finally, for 3. we have a number of options inside scikit-learn.
Although not all algorithms can learn incrementally (i.e. without
seeing all the instances at once), all estimators implementing the
"partial_fit" API are candidates. Actually, the ability to learn
incrementally from a mini-batch of instances (sometimes called "online
learning") is key to out-of-core learning as it guarantees that at any
given time there will be only a small amount of instances in the main
memory. Choosing a good size for the mini-batch that balances
relevancy and memory footprint could involve some tuning [1].

Here is a list of incremental estimators for different tasks:

   * Classification
        * "sklearn.naive_bayes.MultinomialNB"

        * "sklearn.naive_bayes.BernoulliNB"

        * "sklearn.linear_model.Perceptron"

* Classification

        * "sklearn.linear_model.SGDClassifier"

        * "sklearn.linear_model.PassiveAggressiveClassifier"

        * "sklearn.neural_network.MLPClassifier"

   * Regression
        * "sklearn.linear_model.SGDRegressor"

        * "sklearn.linear_model.PassiveAggressiveRegressor"

        * "sklearn.neural_network.MLPRegressor"

   * Clustering
        * "sklearn.cluster.MiniBatchKMeans"

        * "sklearn.cluster.Birch"

   * Decomposition / feature Extraction
        * "sklearn.decomposition.MiniBatchDictionaryLearning"

        * "sklearn.decomposition.IncrementalPCA"

        * "sklearn.decomposition.LatentDirichletAllocation"

        * "sklearn.decomposition.MiniBatchNMF"

   * Preprocessing
        * "sklearn.preprocessing.StandardScaler"

        * "sklearn.preprocessing.MinMaxScaler"

        * "sklearn.preprocessing.MaxAbsScaler"

For classification, a somewhat important thing to note is that
although a stateless feature extraction routine may be able to cope
with new/unseen attributes, the incremental learner itself may be
unable to cope with new/unseen targets classes. In this case you have
to pass all the possible classes to the first "partial_fit" call using
the "classes=" parameter.

Another aspect to consider when choosing a proper algorithm is that
not all of them put the same importance on each example over time.

Namely, the "Perceptron" is still sensitive to badly labeled examples
even after many examples whereas the "SGD*" and "PassiveAggressive*"
families are more robust to this kind of artifacts. Conversely, the
latter also tend to give less importance to remarkably different, yet
properly labeled examples when they come late in the stream as their
learning rate decreases over time.

8.1.1.4. Examples

Finally, we have a full-fledged example of *sphx_glr_auto_examples_ap
plications_plot_out_of_core_classification.py*. It is aimed at
providing a starting point for people wanting to build out-of-core
learning systems and demonstrates most of the notions discussed above.

Furthermore, it also shows the evolution of the performance of
different algorithms with the number of processed examples.

[image: accuracy_over_time][image]

Now looking at the computation time of the different parts, we see
that the vectorization is much more expensive than learning itself.
From the different algorithms, "MultinomialNB" is the most expensive,
but its overhead can be mitigated by increasing the size of the mini-
batches (exercise: change "minibatch_size" to 100 and 10000 in the
program and compare).

[image: computation_time][image]

8.1.1.5. Notes

[1] Depending on the algorithm the mini-batch size can influence

    results or not. SGD*, PassiveAggressive*, and discrete NaiveBayes
    are truly online and are not affected by batch size. Conversely,
    MiniBatchKMeans convergence rate is affected by the batch size.
    Also, its memory footprint can vary dramatically with batch size.

8.2. Computational Performance

For some applications the performance (mainly latency and throughput
at prediction time) of estimators is crucial. It may also be of
interest to consider the training throughput but this is often less
important in a production setup (where it often takes place offline).

We will review here the orders of magnitude you can expect from a
number of scikit-learn estimators in different contexts and provide
some tips and tricks for overcoming performance bottlenecks.

Prediction latency is measured as the elapsed time necessary to make a
prediction (e.g. in micro-seconds). Latency is often viewed as a
distribution and operations engineers often focus on the latency at a
given percentile of this distribution (e.g. the 90 percentile).

Prediction throughput is defined as the number of predictions the
software can deliver in a given amount of time (e.g. in predictions
per second).

An important aspect of performance optimization is also that it can
hurt prediction accuracy. Indeed, simpler models (e.g. linear instead
of non-linear, or with fewer parameters) often run faster but are not
always able to take into account the same exact properties of the data
as more complex ones.

8.2.1. Prediction Latency

One of the most straight-forward concerns one may have when
using/choosing a machine learning toolkit is the latency at which
predictions can be made in a production environment.

The main factors that influence the prediction latency are
   1. Number of features

   2. Input data representation and sparsity

   3. Model complexity

   4. Feature extraction

A last major parameter is also the possibility to do predictions in
bulk or one-at-a-time mode.

8.2.1.1. Bulk versus Atomic mode

In general doing predictions in bulk (many instances at the same time)
is more efficient for a number of reasons (branching predictability,
CPU cache, linear algebra libraries optimizations etc.). Here we see
on a setting with few features that independently of estimator choice
the bulk mode is always faster, and for some of them by 1 to 2 orders
of magnitude:

[image: atomic_prediction_latency][image][image:
bulk_prediction_latency][image]

To benchmark different estimators for your case you can simply change
the "n_features" parameter in this example:
*sphx_glr_auto_examples_applications_plot_prediction_latency.py*. This
should give you an estimate of the order of magnitude of the
prediction latency.

8.2.1.2. Configuring Scikit-learn for reduced validation overhead
-----------------------------------------------------------------

Scikit-learn does some validation on data that increases the overhead
per call to "predict" and similar functions. In particular, checking

that features are finite (not NaN or infinite) involves a full pass
over the data. If you ensure that your data is acceptable, you may
suppress checking for finiteness by setting the environment variable
"SKLEARN_ASSUME_FINITE" to a non-empty string before importing scikit-
learn, or configure it in Python with "set_config()". For more control
than these global settings, a "config_context()" allows you to set
this configuration within a specified context:

   >>> import sklearn
   >>> with sklearn.config_context(assume_finite=True):
   ...     pass  # do learning/prediction here with reduced validation

Note that this will affect all uses of "assert_all_finite()" within
the context.

8.2.1.3. Influence of the Number of Features

Obviously when the number of features increases so does the memory
consumption of each example. Indeed, for a matrix of M instances with
N features, the space complexity is in O(NM). From a computing
perspective it also means that the number of basic operations (e.g.,
multiplications for vector-matrix products in linear models) increases
too. Here is a graph of the evolution of the prediction latency with

[image: influence_of_n_features_on_latency][image]

Overall you can expect the prediction time to increase at least

linearly with the number of features (non-linear cases can happen
depending on the global memory footprint and estimator).

8.2.1.4. Influence of the Input Data Representation
---------------------------------------------------

Scipy provides sparse matrix data structures which are optimized for
storing sparse data. The main feature of sparse formats is that you
don't store zeros so if your data is sparse then you use much less
memory. A non-zero value in a sparse (CSR or CSC) representation will
only take on average one 32bit integer position + the 64 bit floating
point value + an additional 32bit per row or column in the matrix.
Using sparse input on a dense (or sparse) linear model can speedup
prediction by quite a bit as only the non zero valued features impact
the dot product and thus the model predictions. Hence if you have 100
non zeros in 1e6 dimensional space, you only need 100 multiply and add
operation instead of 1e6.

Calculation over a dense representation, however, may leverage highly
optimized vector operations and multithreading in BLAS, and tends to
result in fewer CPU cache misses. So the sparsity should typically be
quite high (10% non-zeros max, to be checked depending on the
hardware) for the sparse input representation to be faster than the
dense input representation on a machine with many CPUs and an
optimized BLAS implementation.

Here is sample code to test the sparsity of your input:

   def sparsity_ratio(X):
       return 1.0 - np.count_nonzero(X) / float(X.shape[0] * X.shape[1])
   print("input sparsity ratio:", sparsity_ratio(X))

As a rule of thumb you can consider that if the sparsity ratio is
greater than 90% you can probably benefit from sparse formats. Check
Scipy's sparse matrix formats documentation for more information on
how to build (or convert your data to) sparse matrix formats. Most of
the time the "CSR" and "CSC" formats work best.

8.2.1.5. Influence of the Model Complexity

Generally speaking, when model complexity increases, predictive power
and latency are supposed to increase. Increasing predictive power is
usually interesting, but for many applications we would better not
increase prediction latency too much. We will now review this idea for
different families of supervised models.

For "sklearn.linear_model" (e.g. Lasso, ElasticNet,
SGDClassifier/Regressor, Ridge & RidgeClassifier,
PassiveAggressiveClassifier/Regressor, LinearSVC,
LogisticRegression...) the decision function that is applied at
prediction time is the same (a dot product) , so latency should be
equivalent.

Here is an example using "SGDClassifier" with the "elasticnet"
penalty. The regularization strength is globally controlled by the

"alpha" parameter. With a sufficiently high "alpha", one can then
increase the "l1_ratio" parameter of "elasticnet" to enforce various
levels of sparsity in the model coefficients. Higher sparsity here is
interpreted as less model complexity as we need fewer coefficients to
describe it fully. Of course sparsity influences in turn the
prediction time as the sparse dot-product takes time roughly
proportional to the number of non-zero coefficients.

[image: en_model_complexity][image]

For the "sklearn.svm" family of algorithms with a non-linear kernel,
the latency is tied to the number of support vectors (the fewer the
faster). Latency and throughput should (asymptotically) grow linearly
with the number of support vectors in a SVC or SVR model. The kernel
will also influence the latency as it is used to compute the
projection of the input vector once per support vector. In the
following graph the "nu" parameter of "NuSVR" was used to influence
the number of support vectors.

[image: nusvr_model_complexity][image]

For "sklearn.ensemble" of trees (e.g. RandomForest, GBT, ExtraTrees
etc) the number of trees and their depth play the most important role.
Latency and throughput should scale linearly with the number of trees.
In this case we used directly the "n_estimators" parameter of
"GradientBoostingRegressor".

[image: gbt_model_complexity][image]

In any case be warned that decreasing model complexity can hurt
accuracy as mentioned above. For instance a non-linearly separable
problem can be handled with a speedy linear model but prediction power
will very likely suffer in the process.

8.2.1.6. Feature Extraction Latency

Most scikit-learn models are usually pretty fast as they are
implemented either with compiled Cython extensions or optimized
computing libraries. On the other hand, in many real world
applications the feature extraction process (i.e. turning raw data
like database rows or network packets into numpy arrays) governs the
overall prediction time. For example on the Reuters text
classification task the whole preparation (reading and parsing SGML
files, tokenizing the text and hashing it into a common vector space)
is taking 100 to 500 times more time than the actual prediction code,
depending on the chosen model.

[image: prediction_time][image]

In many cases it is thus recommended to carefully time and profile
your feature extraction code as it may be a good place to start
optimizing when your overall latency is too slow for your application.

8.2.2. Prediction Throughput

Another important metric to care about when sizing production systems

is the throughput i.e. the number of predictions you can make in a
given amount of time. Here is a benchmark from the
*sphx_glr_auto_examples_applications_plot_prediction_latency.py*
example that measures this quantity for a number of estimators on
synthetic data:

[image: throughput_benchmark][image]

These throughputs are achieved on a single process. An obvious way to
increase the throughput of your application is to spawn additional
instances (usually processes in Python because of the GIL) that share
the same model. One might also add machines to spread the load. A
detailed explanation on how to achieve this is beyond the scope of
this documentation though.

8.2.3. Tips and Tricks

8.2.3.1. Linear algebra libraries

As scikit-learn relies heavily on Numpy/Scipy and linear algebra in
general it makes sense to take explicit care of the versions of these
libraries. Basically, you ought to make sure that Numpy is built using
an optimized BLAS / LAPACK library.

Not all models benefit from optimized BLAS and Lapack implementations.
For instance models based on (randomized) decision trees typically do
not rely on BLAS calls in their inner loops, nor do kernel SVMs
("SVC", "SVR", "NuSVC", "NuSVR").  On the other hand a linear model

implemented with a BLAS DGEMM call (via "numpy.dot") will typically
benefit hugely from a tuned BLAS implementation and lead to orders of
magnitude speedup over a non-optimized BLAS.

You can display the BLAS / LAPACK implementation used by your NumPy /
SciPy / scikit-learn install with the following command:

   python -c "import sklearn; sklearn.show_versions()"

Optimized BLAS / LAPACK implementations include:
   * Atlas (need hardware specific tuning by rebuilding on the target
     machine)

   * OpenBLAS

   * MKL

   * Apple Accelerate and vecLib frameworks (OSX only)

More information can be found on the NumPy install page and in this
blog post from Daniel Nouri which has some nice step by step install
instructions for Debian / Ubuntu.

8.2.3.2. Limiting Working Memory

Some calculations when implemented using standard numpy vectorized
operations involve using a large amount of temporary memory.  This may
potentially exhaust system memory.  Where computations can be
performed in fixed-memory chunks, we attempt to do so, and allow the
user to hint at the maximum size of this working memory (defaulting to
1GB) using "set_config()" or "config_context()".  The following
suggests to limit temporary working memory to 128 MiB:

   >>> with sklearn.config_context(working_memory=128):
   ...     pass  # do chunked work here

An example of a chunked operation adhering to this setting is
"pairwise_distances_chunked()", which facilitates computing row-wise
reductions of a pairwise distance matrix.

8.2.3.3. Model Compression

Model compression in scikit-learn only concerns linear models for the
moment. In this context it means that we want to control the model
sparsity (i.e. the number of non-zero coordinates in the model
vectors). It is generally a good idea to combine model sparsity with
sparse input data representation.

Here is sample code that illustrates the use of the "sparsify()"
method:

   clf = SGDRegressor(penalty='elasticnet', l1_ratio=0.25)
   clf.fit(X_train, y_train).sparsify()
   clf.predict(X_test)

In this example we prefer the "elasticnet" penalty as it is often a
good compromise between model compactness and prediction power. One
can also further tune the "l1_ratio" parameter (in combination with
the regularization strength "alpha") to control this tradeoff.

A typical benchmark on synthetic data yields a >30% decrease in
latency when both the model and input are sparse (with 0.000024 and
0.027400 non-zero coefficients ratio respectively). Your mileage may
vary depending on the sparsity and size of your data and model.

Furthermore, sparsifying can be very useful to reduce the memory usage
of predictive models deployed on production servers.

8.2.3.4. Model Reshaping

Model reshaping consists in selecting only a portion of the available
features to fit a model. In other words, if a model discards features
during the learning phase we can then strip those from the input. This
has several benefits. Firstly it reduces memory (and therefore time)
overhead of the model itself. It also allows to discard explicit
feature selection components in a pipeline once we know which features
to keep from a previous run. Finally, it can help reduce processing
time and I/O usage upstream in the data access and feature extraction
layers by not collecting and building features that are discarded by
the model. For instance if the raw data come from a database, it can
make it possible to write simpler and faster queries or reduce I/O
usage by making the queries return lighter records. At the moment,
reshaping needs to be performed manually in scikit-learn. In the case
of sparse input (particularly in "CSR" format), it is generally
sufficient to not generate the relevant features, leaving their
columns empty.

8.2.3.5. Links

   * scikit-learn developer performance documentation

   * Scipy sparse matrix formats documentation

2.3. Clustering

Clustering of unlabeled data can be performed with the module
"sklearn.cluster".

Each clustering algorithm comes in two variants: a class, that
implements the "fit" method to learn the clusters on train data, and a
function, that, given train data, returns an array of integer labels
corresponding to the different clusters. For the class, the labels
over the training data can be found in the "labels_" attribute.

Input data
^^^^^^^^^^

One important thing to note is that the algorithms implemented in this
module can take different kinds of matrix as input. All the methods
accept standard data matrices of shape "(n_samples, n_features)".
These can be obtained from the classes in the
"sklearn.feature_extraction" module. For "AffinityPropagation",
"SpectralClustering" and "DBSCAN" one can also input similarity
matrices of shape "(n_samples, n_samples)". These can be obtained from
the functions in the "sklearn.metrics.pairwise" module.

2.3.1. Overview of clustering methods

   [image]A comparison of the clustering algorithms in scikit-learn

+----------------+-----------------+---------------------+---------------------------+----------------------+
| Method name    | Parameters      | Scalability         | Usecase                   | Geometry (metric     |
|                |                 |                     |                           | used)                |

|================|=================|=====================|===========================|======================|
| K-Means        | number of       | Very large          | General-purpose, even     | Distances between    |
|                | clusters        | "n_samples", medium | cluster size, flat        | points               |
|                |                 | "n_clusters" with   | geometry, not too many    |                      |
|                |                 | MiniBatch code      | clusters, inductive       |                      |
| Affinity       | damping, sample | Not scalable with   | Many clusters, uneven     | Graph distance (e.g. |
| propagation    | preference      | n_samples           | cluster size, non-flat    | nearest-neighbor     |
|                |                 |                     | geometry, inductive       | graph)               |
| Mean-shift     | bandwidth       | Not scalable with   | Many clusters, uneven     | Distances between    |
|                |                 | "n_samples"         | cluster size, non-flat    | points               |

|                |                 |                     | geometry, inductive       |                      |
| Spectral       | number of       | Medium "n_samples", | Few clusters, even        | Graph distance (e.g. |
| clustering     | clusters        | small "n_clusters"  | cluster size, non-flat    | nearest-neighbor     |
|                |                 |                     | geometry, transductive    | graph)               |
| Ward           | number of       | Large "n_samples"   | Many clusters, possibly   | Distances between    |
| hierarchical   | clusters or     | and "n_clusters"    | connectivity constraints, | points               |
| clustering     | distance        |                     | transductive              |                      |
|                | threshold       |                     |                           |                      |
| Agglomerative  | number of       | Large "n_samples"   | Many clusters, possibly   | Any pairwise         |

| clustering     | clusters or     | and "n_clusters"    | connectivity constraints, | distance             |
|                | distance        |                     | non Euclidean distances,  |                      |
|                | threshold,      |                     | transductive              |                      |
|                | linkage type,   |                     |                           |                      |
|                | distance        |                     |                           |                      |
| DBSCAN         | neighborhood    | Very large          | Non-flat geometry, uneven | Distances between    |
|                | size            | "n_samples", medium | cluster sizes, outlier    | nearest points       |
|                |                 | "n_clusters"        | removal, transductive     |                      |
| OPTICS         | minimum cluster | Very large          | Non-flat geometry, uneven | Distances between    |
|                | membership      | "n_samples", large  | cluster sizes, variable   | points               |

|                |                 | "n_clusters"        | cluster density, outlier  |                      |
|                |                 |                     | removal, transductive     |                      |
| Gaussian       | many            | Not scalable        | Flat geometry, good for   | Mahalanobis          |
| mixtures       |                 |                     | density estimation,       | distances to         |
|                |                 |                     | inductive                 | centers              |
| BIRCH          | branching       | Large "n_clusters"  | Large dataset, outlier    | Euclidean distance   |
|                | factor,         | and "n_samples"     | removal, data reduction,  | between points       |
|                | threshold,      |                     | inductive                 |                      |
|                | optional global |                     |                           |                      |
|                | clusterer.      |                     |                           |                      |

| Bisecting      | number of       | Very large          | General-purpose, even     | Distances between    |
| K-Means        | clusters        | "n_samples", medium | cluster size, flat        | points               |
|                |                 | "n_clusters"        | geometry, no empty        |                      |
|                |                 |                     | clusters, inductive,      |                      |
|                |                 |                     | hierarchical              |                      |

Non-flat geometry clustering is useful when the clusters have a
specific shape, i.e. a non-flat manifold, and the standard euclidean
distance is not the right metric. This case arises in the two top rows
of the figure above.

Gaussian mixture models, useful for clustering, are described in
another chapter of the documentation dedicated to mixture models.
KMeans can be seen as a special case of Gaussian mixture model with
equal covariance per component.

*Transductive* clustering methods (in contrast to *inductive*
clustering methods) are not designed to be applied to new, unseen
data.

2.3.2. K-means

data.

The "KMeans" algorithm clusters data by trying to separate samples in
n groups of equal variance, minimizing a criterion known as the
*inertia* or within-cluster sum-of-squares (see below). This algorithm
requires the number of clusters to be specified. It scales well to
large numbers of samples and has been used across a large range of
application areas in many different fields.

The k-means algorithm divides a set of N samples X into K disjoint
clusters C, each described by the mean \mu_j of the samples in the
cluster. The means are commonly called the cluster "centroids"; note
that they are not, in general, points from X, although they live in
the same space.

The K-means algorithm aims to choose centroids that minimise the
**inertia**, or **within-cluster sum-of-squares criterion**:

   \sum_{i=0}^{n}\min_{\mu_j \in C}(||x_i - \mu_j||^2)

Inertia can be recognized as a measure of how internally coherent
clusters are. It suffers from various drawbacks:

* Inertia makes the assumption that clusters are convex and isotropic,
  which is not always the case. It responds poorly to elongated
  clusters, or manifolds with irregular shapes.

* Inertia is not a normalized metric: we just know that lower values
  are better and zero is optimal. But in very high-dimensional spaces,

clusters, or manifolds with irregular shapes.

  Euclidean distances tend to become inflated (this is an instance of
  the so-called "curse of dimensionality"). Running a dimensionality
  reduction algorithm such as Principal component analysis (PCA) prior
  to k-means clustering can alleviate this problem and speed up the
  computations.

[image]

K-means is often referred to as Lloyd's algorithm. In basic terms, the
algorithm has three steps. The first step chooses the initial
centroids, with the most basic method being to choose k samples from
the dataset X. After initialization, K-means consists of looping
between the two other steps. The first step assigns each sample to its
nearest centroid. The second step creates new centroids by taking the
mean value of all of the samples assigned to each previous centroid.
The difference between the old and the new centroids are computed and
the algorithm repeats these last two steps until this value is less
than a threshold. In other words, it repeats until the centroids do
not move significantly.

[image]

K-means is equivalent to the expectation-maximization algorithm with a
small, all-equal, diagonal covariance matrix.

The algorithm can also be understood through the concept of Voronoi
diagrams. First the Voronoi diagram of the points is calculated using

the current centroids. Each segment in the Voronoi diagram becomes a
separate cluster. Secondly, the centroids are updated to the mean of
each segment. The algorithm then repeats this until a stopping
criterion is fulfilled. Usually, the algorithm stops when the relative
decrease in the objective function between iterations is less than the
given tolerance value. This is not the case in this implementation:
iteration stops when centroids move less than the tolerance.

Given enough time, K-means will always converge, however this may be
to a local minimum. This is highly dependent on the initialization of
the centroids. As a result, the computation is often done several
times, with different initializations of the centroids. One method to
help address this issue is the k-means++ initialization scheme, which
has been implemented in scikit-learn (use the "init='k-means++'"
parameter). This initializes the centroids to be (generally) distant
from each other, leading to probably better results than random
initialization, as shown in the reference.

K-means++ can also be called independently to select seeds for other
clustering algorithms, see "sklearn.cluster.kmeans_plusplus()" for
details and example usage.

The algorithm supports sample weights, which can be given by a

parameter "sample_weight". This allows to assign more weight to some
samples when computing cluster centers and values of inertia. For
example, assigning a weight of 2 to a sample is equivalent to adding a
duplicate of that sample to the dataset X.

K-means can be used for vector quantization. This is achieved using
the transform method of a trained model of "KMeans".

2.3.2.1. Low-level parallelism

"KMeans" benefits from OpenMP based parallelism through Cython. Small
chunks of data (256 samples) are processed in parallel, which in
addition yields a low memory footprint. For more details on how to
control the number of threads, please refer to our Parallelism notes.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_kmeans_assumptions.py*:
  Demonstrating when k-means performs intuitively and when it does not

* *sphx_glr_auto_examples_cluster_plot_kmeans_digits.py*: Clustering
  handwritten digits

* "k-means++: The advantages of careful seeding" Arthur, David, and
  Sergei Vassilvitskii, *Proceedings of the eighteenth annual ACM-SIAM
  symposium on Discrete algorithms*, Society for Industrial and
  Applied Mathematics (2007)

2.3.2.2. Mini Batch K-Means

The "MiniBatchKMeans" is a variant of the "KMeans" algorithm which

Applied Mathematics (2007)

uses mini-batches to reduce the computation time, while still
attempting to optimise the same objective function. Mini-batches are
subsets of the input data, randomly sampled in each training
iteration. These mini-batches drastically reduce the amount of
computation required to converge to a local solution. In contrast to
other algorithms that reduce the convergence time of k-means, mini-
batch k-means produces results that are generally only slightly worse
than the standard algorithm.

The algorithm iterates between two major steps, similar to vanilla
k-means. In the first step, b samples are drawn randomly from the
dataset, to form a mini-batch. These are then assigned to the nearest
centroid. In the second step, the centroids are updated. In contrast
to k-means, this is done on a per-sample basis. For each sample in the
mini-batch, the assigned centroid is updated by taking the streaming
average of the sample and all previous samples assigned to that
centroid. This has the effect of decreasing the rate of change for a
centroid over time. These steps are performed until convergence or a
predetermined number of iterations is reached.

"MiniBatchKMeans" converges faster than "KMeans", but the quality of
the results is reduced. In practice this difference in quality can be
quite small, as shown in the example and cited reference.

   [image]

Examples:

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_mini_batch_kmeans.py*:
  Comparison of KMeans and MiniBatchKMeans

* *sphx_glr_auto_examples_text_plot_document_clustering.py*: Document
  clustering using sparse MiniBatchKMeans

* *sphx_glr_auto_examples_cluster_plot_dict_face_patches.py*

* "Web Scale K-Means clustering" D. Sculley, *Proceedings of the 19th
  international conference on World wide web* (2010)

2.3.3. Affinity Propagation

"AffinityPropagation" creates clusters by sending messages between
pairs of samples until convergence. A dataset is then described using
a small number of exemplars, which are identified as those most
representative of other samples. The messages sent between pairs
represent the suitability for one sample to be the exemplar of the
other, which is updated in response to the values from other pairs.
This updating happens iteratively until convergence, at which point
the final exemplars are chosen, and hence the final clustering is
given.

   [image]

Affinity Propagation can be interesting as it chooses the number of
clusters based on the data provided. For this purpose, the two
important parameters are the *preference*, which controls how many
exemplars are used, and the *damping factor* which damps the
responsibility and availability messages to avoid numerical

oscillations when updating these messages.

The main drawback of Affinity Propagation is its complexity. The
algorithm has a time complexity of the order O(N^2 T), where N is the
number of samples and T is the number of iterations until convergence.
Further, the memory complexity is of the order O(N^2) if a dense
similarity matrix is used, but reducible if a sparse similarity matrix
is used. This makes Affinity Propagation most appropriate for small to
medium sized datasets.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_affinity_propagation.py*:
  Affinity Propagation on a synthetic 2D datasets with 3 classes.

* *sphx_glr_auto_examples_applications_plot_stock_market.py* Affinity
  Propagation on Financial time series to find groups of companies

**Algorithm description:** The messages sent between points belong to
one of two categories. The first is the responsibility r(i, k), which
is the accumulated evidence that sample k should be the exemplar for
sample i. The second is the availability a(i, k) which is the
accumulated evidence that sample i should choose sample k to be its
exemplar, and considers the values for all other samples that k should
be an exemplar. In this way, exemplars are chosen by samples if they
are (1) similar enough to many samples and (2) chosen by many samples

to be representative of themselves.

More formally, the responsibility of a sample k to be the exemplar of
sample i is given by:

   r(i, k) \leftarrow s(i, k) - max [ a(i, k') + s(i, k') \forall k'
   \neq k ]

Where s(i, k) is the similarity between samples i and k. The
availability of sample k to be the exemplar of sample i is given by:

   a(i, k) \leftarrow min [0, r(k, k) + \sum_{i'~s.t.~i' \notin \{i,
   k\}}{r(i', k)}]

To begin with, all values for r and a are set to zero, and the
calculation of each iterates until convergence. As discussed above, in
order to avoid numerical oscillations when updating the messages, the
damping factor \lambda is introduced to iteration process:

   r_{t+1}(i, k) = \lambda\cdot r_{t}(i, k) + (1-\lambda)\cdot
   r_{t+1}(i, k)

   a_{t+1}(i, k) = \lambda\cdot a_{t}(i, k) + (1-\lambda)\cdot
   a_{t+1}(i, k)

where t indicates the iteration times.

2.3.4. Mean Shift

"MeanShift" clustering aims to discover *blobs* in a smooth density of
samples. It is a centroid based algorithm, which works by updating
candidates for centroids to be the mean of the points within a given
region. These candidates are then filtered in a post-processing stage
to eliminate near-duplicates to form the final set of centroids.

The position of centroid candidates is iteratively adjusted using a

technique called hill climbing, which finds local maxima of the
estimated probability density. Given a candidate centroid x for
iteration t, the candidate is updated according to the following
equation:

   x^{t+1} = x^t + m(x^t)

Where m is the *mean shift* vector that is computed for each centroid
that points towards a region of the maximum increase in the density of
points. To compute m we define N(x) as the neighborhood of samples
within a given distance around x. Then m is computed using the
following equation, effectively updating a centroid to be the mean of
the samples within its neighborhood:

   m(x) = \frac{1}{|N(x)|} \sum_{x_j \in N(x)}x_j - x

In general, the equation for m depends on a kernel used for density
estimation. The generic formula is:

   m(x) = \frac{\sum_{x_j \in N(x)}K(x_j - x)x_j}{\sum_{x_j \in
   N(x)}K(x_j - x)} - x

In our implementation, K(x) is equal to 1 if x is small enough and is
equal to 0 otherwise. Effectively K(y - x) indicates whether y is in
the neighborhood of x.

The algorithm automatically sets the number of clusters, instead of
relying on a parameter "bandwidth", which dictates the size of the
region to search through. This parameter can be set manually, but can
be estimated using the provided "estimate_bandwidth" function, which

is called if the bandwidth is not set.

The algorithm is not highly scalable, as it requires multiple nearest
neighbor searches during the execution of the algorithm. The algorithm
is guaranteed to converge, however the algorithm will stop iterating
when the change in centroids is small.

Labelling a new sample is performed by finding the nearest centroid
for a given sample.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_mean_shift.py*: Mean Shift
  clustering on a synthetic 2D datasets with 3 classes.

* >>:doi:`"Mean shift: A robust approach toward feature space
  analysis" <10.1109/34.1000236>`<< D. Comaniciu and P. Meer, *IEEE
  Transactions on Pattern Analysis and Machine Intelligence* (2002)

2.3.5. Spectral clustering

"SpectralClustering" performs a low-dimension embedding of the
affinity matrix between samples, followed by clustering, e.g., by
KMeans, of the components of the eigenvectors in the low dimensional
space. It is especially computationally efficient if the affinity
matrix is sparse and the *amg* solver is used for the eigenvalue
problem (Note, the *amg* solver requires that the pyamg module is
installed.)

The present version of SpectralClustering requires the number of
clusters to be specified in advance. It works well for a small number

of clusters, but is not advised for many clusters.

For two clusters, SpectralClustering solves a convex relaxation of the
normalized cuts problem on the similarity graph: cutting the graph in
two so that the weight of the edges cut is small compared to the
weights of the edges inside each cluster. This criteria is especially
interesting when working on images, where graph vertices are pixels,
and weights of the edges of the similarity graph are computed using a
function of a gradient of the image.

[image: noisy_img][image] [image: segmented_img][image]

Warning:

  Transforming distance to well-behaved similaritiesNote that if the
  values of your similarity matrix are not well distributed, e.g. with
  negative values or with a distance matrix rather than a similarity,
  the spectral problem will be singular and the problem not solvable.
  In which case it is advised to apply a transformation to the entries
  of the matrix. For instance, in the case of a signed distance
  matrix, is common to apply a heat kernel:

     similarity = np.exp(-beta * distance / distance.std())

  See the examples for such an application.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_segmentation_toy.py*:
  Segmenting objects from a noisy background using spectral
  clustering.

* *sphx_glr_auto_examples_cluster_plot_coin_segmentation.py*: Spectral

Segmenting objects from a noisy background using spectral

  clustering to split the image of coins in regions.

2.3.5.1. Different label assignment strategies
----------------------------------------------

Different label assignment strategies can be used, corresponding to
the "assign_labels" parameter of "SpectralClustering". ""kmeans""
strategy can match finer details, but can be unstable. In particular,
unless you control the "random_state", it may not be reproducible from
run-to-run, as it depends on random initialization. The alternative
""discretize"" strategy is 100% reproducible, but tends to create
parcels of fairly even and geometrical shape. The recently added
""cluster_qr"" option is a deterministic alternative that tends to
create the visually best partitioning on the example application
below.

+----------------------------------+----------------------------------+----------------------------------+
| "assign_labels="kmeans""         | "assign_labels="discretize""     | "assign_labels="cluster_qr""     |
|==================================|==================================|==================================|
| [image: coin_kmeans][image]      | [image: coin_discretize][image]  | [image: coin_cluster_qr][image]  |

* "Multiclass spectral clustering" Stella X. Yu, Jianbo Shi, 2003

* >>:doi:`"Simple, direct, and efficient multi-way spectral
  clustering"<10.1093/imaiai/iay008>`<< Anil Damle, Victor Minden,
  Lexing Ying, 2019

2.3.5.2. Spectral Clustering Graphs

Spectral Clustering can also be used to partition graphs via their
spectral embeddings.  In this case, the affinity matrix is the
adjacency matrix of the graph, and SpectralClustering is initialized
with *affinity='precomputed'*:

   >>> from sklearn.cluster import SpectralClustering
   >>> sc = SpectralClustering(3, affinity='precomputed', n_init=100,
   ...                         assign_labels='discretize')
   >>> sc.fit_predict(adjacency_matrix)  

* >>:doi:`"A Tutorial on Spectral Clustering"
  <10.1007/s11222-007-9033-z>`<< Ulrike von Luxburg, 2007

* >>:doi:`"Normalized cuts and image segmentation"
  <10.1109/34.868688>`<< Jianbo Shi, Jitendra Malik, 2000

* "A Random Walks View of Spectral Segmentation" Marina Meila, Jianbo
  Shi, 2001

* "On Spectral Clustering: Analysis and an algorithm" Andrew Y. Ng,
  Michael I. Jordan, Yair Weiss, 2001

* >>:arxiv:`"Preconditioned Spectral Clustering for Stochastic Block
  Partition Streaming Graph Challenge" <1708.07481>`<< David
  Zhuzhunashvili, Andrew Knyazev

2.3.6. Hierarchical clustering

Partition Streaming Graph Challenge" <1708.07481>`<< David

Hierarchical clustering is a general family of clustering algorithms
that build nested clusters by merging or splitting them successively.
This hierarchy of clusters is represented as a tree (or dendrogram).
The root of the tree is the unique cluster that gathers all the
samples, the leaves being the clusters with only one sample. See the
Wikipedia page for more details.

The "AgglomerativeClustering" object performs a hierarchical
clustering using a bottom up approach: each observation starts in its
own cluster, and clusters are successively merged together. The
linkage criteria determines the metric used for the merge strategy:

* **Ward** minimizes the sum of squared differences within all
  clusters. It is a variance-minimizing approach and in this sense is
  similar to the k-means objective function but tackled with an
  agglomerative hierarchical approach.

* **Maximum** or **complete linkage** minimizes the maximum distance
  between observations of pairs of clusters.

* **Average linkage** minimizes the average of the distances between
  all observations of pairs of clusters.

* **Single linkage** minimizes the distance between the closest
  observations of pairs of clusters.

"AgglomerativeClustering" can also scale to large number of samples
when it is used jointly with a connectivity matrix, but is

observations of pairs of clusters.

computationally expensive when no connectivity constraints are added
between samples: it considers at each step all the possible merges.

"FeatureAgglomeration"
^^^^^^^^^^^^^^^^^^^^^^

The "FeatureAgglomeration" uses agglomerative clustering to group
together features that look very similar, thus decreasing the number
of features. It is a dimensionality reduction tool, see Unsupervised
dimensionality reduction.

2.3.6.1. Different linkage type: Ward, complete, average, and single linkage

"AgglomerativeClustering" supports Ward, single, average, and complete
linkage strategies.

[image]

Agglomerative cluster has a "rich get richer" behavior that leads to
uneven cluster sizes. In this regard, single linkage is the worst
strategy, and Ward gives the most regular sizes. However, the affinity
(or distance used in clustering) cannot be varied with Ward, thus for
non Euclidean metrics, average linkage is a good alternative. Single
linkage, while not robust to noisy data, can be computed very
efficiently and can therefore be useful to provide hierarchical
clustering of larger datasets. Single linkage can also perform well on
non-globular data.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_digits_linkage.py*: exploration

Examples:
^^^^^^^^^

  of the different linkage strategies in a real dataset.

2.3.6.2. Visualization of cluster hierarchy

It's possible to visualize the tree representing the hierarchical
merging of clusters as a dendrogram. Visual inspection can often be
useful for understanding the structure of the data, though more so in
the case of small sample sizes.

[image]

2.3.6.3. Adding connectivity constraints
----------------------------------------

An interesting aspect of "AgglomerativeClustering" is that
connectivity constraints can be added to this algorithm (only adjacent
clusters can be merged together), through a connectivity matrix that
defines for each sample the neighboring samples following a given
structure of the data. For instance, in the swiss-roll example below,
the connectivity constraints forbid the merging of points that are not
adjacent on the swiss roll, and thus avoid forming clusters that
extend across overlapping folds of the roll.

[image: unstructured][image] [image: structured][image]

These constraint are useful to impose a certain local structure, but
they also make the algorithm faster, especially when the number of the
samples is high.

The connectivity constraints are imposed via an connectivity matrix: a
scipy sparse matrix that has elements only at the intersection of a

row and a column with indices of the dataset that should be connected.
This matrix can be constructed from a-priori information: for
instance, you may wish to cluster web pages by only merging pages with
a link pointing from one to another. It can also be learned from the
data, for instance using "sklearn.neighbors.kneighbors_graph()" to
restrict merging to nearest neighbors as in *this example*, or using
"sklearn.feature_extraction.image.grid_to_graph()" to enable only
merging of neighboring pixels on an image, as in the *coin* example.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_coin_ward_segmentation.py*:
  Ward clustering to split the image of coins in regions.

* *sphx_glr_auto_examples_cluster_plot_ward_structured_vs_unstructure
  d.py*: Example of Ward algorithm on a swiss-roll, comparison of
  structured approaches versus unstructured approaches.

* *sphx_glr_auto_examples_cluster_plot_feature_agglomeration_vs_univa
  riate_selection.py*: Example of dimensionality reduction with
  feature agglomeration based on Ward hierarchical clustering.

* *sphx_glr_auto_examples_cluster_plot_agglomerative_clustering.py*

Warning:

  **Connectivity constraints with single, average and complete
  linkage**Connectivity constraints and single, complete or average
  linkage can enhance the 'rich getting richer' aspect of

Warning:

  agglomerative clustering, particularly so if they are built with
  "sklearn.neighbors.kneighbors_graph()". In the limit of a small
  number of clusters, they tend to give a few macroscopically occupied
  clusters and almost empty ones. (see the discussion in
  *sphx_glr_auto_examples_cluster_plot_agglomerative_clustering.py*).
  Single linkage is the most brittle linkage option with regard to
  this issue.

[image][image][image][image]

2.3.6.4. Varying the metric

Single, average and complete linkage can be used with a variety of
distances (or affinities), in particular Euclidean distance (*l2*),
Manhattan distance (or Cityblock, or *l1*), cosine distance, or any
precomputed affinity matrix.

* *l1* distance is often good for sparse features, or sparse noise:
  i.e. many of the features are zero, as in text mining using
  occurrences of rare words.

* *cosine* distance is interesting because it is invariant to global
  scalings of the signal.

The guidelines for choosing a metric is to use one that maximizes the
distance between samples in different classes, and minimizes that
within each class.

[image][image][image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_agglomerative_clustering_metri
  cs.py*

2.3.6.5. Bisecting K-Means

Examples:
^^^^^^^^^

  cs.py*

The "BisectingKMeans" is an iterative variant of "KMeans", using
divisive hierarchical clustering. Instead of creating all centroids at
once, centroids are picked progressively based on a previous
clustering: a cluster is split into two new clusters repeatedly until
the target number of clusters is reached.

"BisectingKMeans" is more efficient than "KMeans" when the number of
clusters is large since it only works on a subset of the data at each
bisection while "KMeans" always works on the entire dataset.

Although "BisectingKMeans" can't benefit from the advantages of the
*"k-means++"* initialization by design, it will still produce
comparable results than *KMeans(init="k-means++")* in terms of inertia
at cheaper computational costs, and will likely produce better results
than *KMeans* with a random initialization.

This variant is more efficient to agglomerative clustering if the
number of clusters is small compared to the number of data points.

This variant also does not produce empty clusters.

There exist two strategies for selecting the cluster to split:
   * "bisecting_strategy="largest_cluster"" selects the cluster having
     the most points

   * "bisecting_strategy="biggest_inertia"" selects the cluster with
     biggest inertia (cluster with biggest Sum of Squared Errors
     within)

the most points

     within)

Picking by largest amount of data points in most cases produces result
as accurate as picking by inertia and is faster (especially for larger
amount of data points, where calculating error may be costly).

Picking by largest amount of data points will also likely produce
clusters of similar sizes while *KMeans* is known to produce clusters
of different sizes.

Difference between Bisecting K-Means and regular K-Means can be seen
on example *sphx_glr_auto_examples_cluster_plot_bisect_kmeans.py*.
While the regular K-Means algorithm tends to create non-related
clusters, clusters from Bisecting K-Means are well ordered and create
quite a visible hierarchy.

* "A Comparison of Document Clustering Techniques" Michael Steinbach,
  George Karypis and Vipin Kumar, Department of Computer Science and
  Egineering, University of Minnesota (June 2000)

* "Performance Analysis of K-Means and Bisecting K-Means Algorithms in
  Weblog Data" K.Abirami and Dr.P.Mayilvahanan, International Journal
  of Emerging Technologies in Engineering Research (IJETER) Volume 4,
  Issue 8, (August 2016)

* "Bisecting K-means Algorithm Based on K-valued Self-determining and
  Clustering Center Optimization" Jian Di, Xinyue Gou School of
  Control and Computer Engineering,North China Electric Power
  University, Baoding, Hebei, China (August 2017)

Clustering Center Optimization" Jian Di, Xinyue Gou School of

2.3.7. DBSCAN

The "DBSCAN" algorithm views clusters as areas of high density
separated by areas of low density. Due to this rather generic view,
clusters found by DBSCAN can be any shape, as opposed to k-means which
assumes that clusters are convex shaped. The central component to the
DBSCAN is the concept of *core samples*, which are samples that are in
areas of high density. A cluster is therefore a set of core samples,
each close to each other (measured by some distance measure) and a set
of non-core samples that are close to a core sample (but are not
themselves core samples). There are two parameters to the algorithm,
"min_samples" and "eps", which define formally what we mean when we
say *dense*. Higher "min_samples" or lower "eps" indicate higher
density necessary to form a cluster.

More formally, we define a core sample as being a sample in the
dataset such that there exist "min_samples" other samples within a
distance of "eps", which are defined as *neighbors* of the core
sample. This tells us that the core sample is in a dense area of the
vector space. A cluster is a set of core samples that can be built by
recursively taking a core sample, finding all of its neighbors that
are core samples, finding all of *their* neighbors that are core
samples, and so on. A cluster also has a set of non-core samples,

which are samples that are neighbors of a core sample in the cluster
but are not themselves core samples. Intuitively, these samples are on
the fringes of a cluster.

Any core sample is part of a cluster, by definition. Any sample that
is not a core sample, and is at least "eps" in distance from any core
sample, is considered an outlier by the algorithm.

While the parameter "min_samples" primarily controls how tolerant the
algorithm is towards noise (on noisy and large data sets it may be
desirable to increase this parameter), the parameter "eps" is *crucial
to choose appropriately* for the data set and distance function and
usually cannot be left at the default value. It controls the local
neighborhood of the points. When chosen too small, most data will not
be clustered at all (and labeled as "-1" for "noise"). When chosen too
large, it causes close clusters to be merged into one cluster, and
eventually the entire data set to be returned as a single cluster.
Some heuristics for choosing this parameter have been discussed in the
literature, for example based on a knee in the nearest neighbor
distances plot (as discussed in the references below).

In the figure below, the color indicates cluster membership, with
large circles indicating core samples found by the algorithm. Smaller

circles are non-core samples that are still part of a cluster.
Moreover, the outliers are indicated by black points below.

[image: dbscan_results][image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_dbscan.py*

Implementation

The DBSCAN algorithm is deterministic, always generating the same
clusters when given the same data in the same order.  However, the
results can differ when data is provided in a different order. First,
even though the core samples will always be assigned to the same
clusters, the labels of those clusters will depend on the order in
which those samples are encountered in the data. Second and more
importantly, the clusters to which non-core samples are assigned can
differ depending on the data order.  This would happen when a non-core
sample has a distance lower than "eps" to two core samples in
different clusters. By the triangular inequality, those two core
samples must be more distant than "eps" from each other, or they would
be in the same cluster. The non-core sample is assigned to whichever
cluster is generated first in a pass through the data, and so the
results will depend on the data ordering.

The current implementation uses ball trees and kd-trees to determine
the neighborhood of points, which avoids calculating the full distance

matrix (as was done in scikit-learn versions before 0.14). The
possibility to use custom metrics is retained; for details, see
"NearestNeighbors".

Memory consumption for large sample sizes

This implementation is by default not memory efficient because it
constructs a full pairwise similarity matrix in the case where kd-
trees or ball-trees cannot be used (e.g., with sparse matrices). This
matrix will consume n^2 floats. A couple of mechanisms for getting
around this are:

* Use OPTICS clustering in conjunction with the *extract_dbscan*
  method. OPTICS clustering also calculates the full pairwise matrix,
  but only keeps one row in memory at a time (memory complexity n).

* A sparse radius neighborhood graph (where missing entries are
  presumed to be out of eps) can be precomputed in a memory-efficient
  way and dbscan can be run over this with "metric='precomputed'".
  See "sklearn.neighbors.NearestNeighbors.radius_neighbors_graph()".

* The dataset can be compressed, either by removing exact duplicates
  if these occur in your data, or by using BIRCH. Then you only have a
  relatively small number of representatives for a large number of
  points. You can then provide a "sample_weight" when fitting DBSCAN.

relatively small number of representatives for a large number of

* "A Density-Based Algorithm for Discovering Clusters in Large Spatial
  Databases with Noise" Ester, M., H. P. Kriegel, J. Sander, and X.
  Xu, In Proceedings of the 2nd International Conference on Knowledge
  Discovery and Data Mining, Portland, OR, AAAI Press, pp. 226–231.
  1996

* >>:doi:`"DBSCAN revisited, revisited: why and how you should (still)
  use DBSCAN." <10.1145/3068335>`<< Schubert, E., Sander, J., Ester,
  M., Kriegel, H. P., & Xu, X. (2017). In ACM Transactions on Database
  Systems (TODS), 42(3), 19.

2.3.8. OPTICS

The "OPTICS" algorithm shares many similarities with the "DBSCAN"
algorithm, and can be considered a generalization of DBSCAN that
relaxes the "eps" requirement from a single value to a value range.
The key difference between DBSCAN and OPTICS is that the OPTICS
algorithm builds a *reachability* graph, which assigns each sample
both a "reachability_" distance, and a spot within the cluster
"ordering_" attribute; these two attributes are assigned when the
model is fitted, and are used to determine cluster membership. If
OPTICS is run with the default value of *inf* set for "max_eps", then
DBSCAN style cluster extraction can be performed repeatedly in linear
time for any given "eps" value using the "cluster_optics_dbscan"
method. Setting "max_eps" to a lower value will result in shorter run

times, and can be thought of as the maximum neighborhood radius from
each point to find other potential reachable points.

[image: optics_results][image]

The *reachability* distances generated by OPTICS allow for variable
density extraction of clusters within a single data set. As shown in
the above plot, combining *reachability* distances and data set
"ordering_" produces a *reachability plot*, where point density is
represented on the Y-axis, and points are ordered such that nearby
points are adjacent. 'Cutting' the reachability plot at a single value
produces DBSCAN like results; all points above the 'cut' are
classified as noise, and each time that there is a break when reading
from left to right signifies a new cluster. The default cluster
extraction with OPTICS looks at the steep slopes within the graph to
find clusters, and the user can define what counts as a steep slope
using the parameter "xi". There are also other possibilities for
analysis on the graph itself, such as generating hierarchical
representations of the data through reachability-plot dendrograms, and
the hierarchy of clusters detected by the algorithm can be accessed
through the "cluster_hierarchy_" parameter. The plot above has been
color-coded so that cluster colors in planar space match the linear
segment clusters of the reachability plot. Note that the blue and red

clusters are adjacent in the reachability plot, and can be
hierarchically represented as children of a larger parent cluster.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_optics.py*

Comparison with DBSCAN

The results from OPTICS "cluster_optics_dbscan" method and DBSCAN are
very similar, but not always identical; specifically, labeling of
periphery and noise points. This is in part because the first samples
of each dense area processed by OPTICS have a large reachability value
while being close to other points in their area, and will thus
sometimes be marked as noise rather than periphery. This affects
adjacent points when they are considered as candidates for being
marked as either periphery or noise.

Note that for any single value of "eps", DBSCAN will tend to have a
shorter run time than OPTICS; however, for repeated runs at varying
"eps" values, a single run of OPTICS may require less cumulative
runtime than DBSCAN. It is also important to note that OPTICS' output
is close to DBSCAN's only if "eps" and "max_eps" are close.

Computational Complexity

Spatial indexing trees are used to avoid calculating the full distance
matrix, and allow for efficient memory usage on large sets of samples.
Different distance metrics can be supplied via the "metric" keyword.

For large datasets, similar (but not identical) results can be
obtained via HDBSCAN. The HDBSCAN implementation is multithreaded, and
has better algorithmic runtime complexity than OPTICS, at the cost of
worse memory scaling. For extremely large datasets that exhaust system
memory using HDBSCAN, OPTICS will maintain n (as opposed to n^2)
memory scaling; however, tuning of the "max_eps" parameter will likely
need to be used to give a solution in a reasonable amount of wall
time.

* "OPTICS: ordering points to identify the clustering structure."
  Ankerst, Mihael, Markus M. Breunig, Hans-Peter Kriegel, and Jörg
  Sander. In ACM Sigmod Record, vol. 28, no. 2, pp. 49-60. ACM, 1999.

2.3.9. BIRCH

The "Birch" builds a tree called the Clustering Feature Tree (CFT) for
the given data. The data is essentially lossy compressed to a set of
Clustering Feature nodes (CF Nodes). The CF Nodes have a number of
subclusters called Clustering Feature subclusters (CF Subclusters) and
these CF Subclusters located in the non-terminal CF Nodes can have CF
Nodes as children.

The CF Subclusters hold the necessary information for clustering which
prevents the need to hold the entire input data in memory. This
information includes:

* Number of samples in a subcluster.

* Linear Sum - An n-dimensional vector holding the sum of all samples

* Squared Sum - Sum of the squared L2 norm of all samples.

* Centroids - To avoid recalculation linear sum / n_samples.

* Squared norm of the centroids.

The BIRCH algorithm has two parameters, the threshold and the
branching factor. The branching factor limits the number of
subclusters in a node and the threshold limits the distance between
the entering sample and the existing subclusters.

This algorithm can be viewed as an instance or data reduction method,
since it reduces the input data to a set of subclusters which are
obtained directly from the leaves of the CFT. This reduced data can be
further processed by feeding it into a global clusterer. This global
clusterer can be set by "n_clusters". If "n_clusters" is set to None,
the subclusters from the leaves are directly read off, otherwise a
global clustering step labels these subclusters into global clusters
(labels) and the samples are mapped to the global label of the nearest
subcluster.

**Algorithm description:**

* A new sample is inserted into the root of the CF Tree which is a CF
  Node. It is then merged with the subcluster of the root, that has
  the smallest radius after merging, constrained by the threshold and
  branching factor conditions. If the subcluster has any child node,
  then this is done repeatedly till it reaches a leaf. After finding

branching factor conditions. If the subcluster has any child node,
  the nearest subcluster in the leaf, the properties of this
  subcluster and the parent subclusters are recursively updated.

* If the radius of the subcluster obtained by merging the new sample
  and the nearest subcluster is greater than the square of the
  threshold and if the number of subclusters is greater than the
  branching factor, then a space is temporarily allocated to this new
  sample. The two farthest subclusters are taken and the subclusters
  are divided into two groups on the basis of the distance between
  these subclusters.

* If this split node has a parent subcluster and there is room for a
  new subcluster, then the parent is split into two. If there is no
  room, then this node is again split into two and the process is
  continued recursively, till it reaches the root.

**BIRCH or MiniBatchKMeans?**

   * BIRCH does not scale very well to high dimensional data. As a
     rule of thumb if "n_features" is greater than twenty, it is
     generally better to use MiniBatchKMeans.

   * If the number of instances of data needs to be reduced, or if one
     wants a large number of subclusters either as a preprocessing
     step or otherwise, BIRCH is more useful than MiniBatchKMeans.

**How to use partial_fit?**

To avoid the computation of global clustering, for every call of
"partial_fit" the user is advised

   1. To set "n_clusters=None" initially

   2. Train all data by multiple calls to partial_fit.

   3. Set "n_clusters" to a required value using
      "brc.set_params(n_clusters=n_clusters)".

   4. Call "partial_fit" finally with no arguments, i.e.
      "brc.partial_fit()" which performs the global clustering.

[image]

* Tian Zhang, Raghu Ramakrishnan, Maron Livny BIRCH: An efficient data
  clustering method for large databases.
  https://www.cs.sfu.ca/CourseCentral/459/han/papers/zhang96.pdf

* Roberto Perdisci JBirch - Java implementation of BIRCH clustering
  algorithm https://code.google.com/archive/p/jbirch

2.3.10. Clustering performance evaluation

Evaluating the performance of a clustering algorithm is not as trivial
as counting the number of errors or the precision and recall of a
supervised classification algorithm. In particular any evaluation
metric should not take the absolute values of the cluster labels into
account but rather if this clustering define separations of the data
similar to some ground truth set of classes or satisfying some
assumption such that members belong to the same class are more similar
than members of different classes according to some similarity metric.

2.3.10.1. Rand index

Given the knowledge of the ground truth class assignments

"labels_true" and our clustering algorithm assignments of the same
samples "labels_pred", the **(adjusted or unadjusted) Rand index** is
a function that measures the **similarity** of the two assignments,
ignoring permutations:

   >>> labels_true = [0, 0, 0, 1, 1, 1]
   >>> labels_pred = [0, 0, 1, 1, 2, 2]
   >>> metrics.rand_score(labels_true, labels_pred)
   0.66...

The Rand index does not ensure to obtain a value close to 0.0 for a
random labelling. The adjusted Rand index **corrects for chance** and
will give such a baseline.

>>> metrics.adjusted_rand_score(labels_true, labels_pred)
0.24...

As with all clustering metrics, one can permute 0 and 1 in the
predicted labels, rename 2 to 3, and get the same score:

   >>> labels_pred = [1, 1, 0, 0, 3, 3]
   0.66...
   >>> metrics.adjusted_rand_score(labels_true, labels_pred)
   0.24...

Furthermore, both "rand_score()" "adjusted_rand_score()" are
**symmetric**: swapping the argument does not change the scores. They
can thus be used as **consensus measures**:

   >>> metrics.rand_score(labels_pred, labels_true)
   0.66...
   >>> metrics.adjusted_rand_score(labels_pred, labels_true)
   0.24...

Perfect labeling is scored 1.0:

   >>> labels_pred = labels_true[:]

0.66...
   0.24...

   1.0
   1.0

Poorly agreeing labels (e.g. independent labelings) have lower scores,
and for the adjusted Rand index the score will be negative or close to
zero. However, for the unadjusted Rand index the score, while lower,
will not necessarily be close to zero.:

   >>> labels_true = [0, 0, 0, 0, 0, 0, 1, 1]
   >>> labels_pred = [0, 1, 2, 3, 4, 5, 5, 6]
   0.39...
   -0.07...

2.3.10.1.1. Advantages

* **Interpretability**: The unadjusted Rand index is proportional to
  the number of sample pairs whose labels are the same in both
  *labels_pred* and *labels_true*, or are different in both.

* **Random (uniform) label assignments have an adjusted Rand index
  score close to 0.0** for any value of "n_clusters" and "n_samples"
  (which is not the case for the unadjusted Rand index or the
  V-measure for instance).

* **Bounded range**: Lower values indicate different labelings,
  similar clusterings have a high (adjusted or unadjusted) Rand index,
  1.0 is the perfect match score. The score range is [0, 1] for the
  unadjusted Rand index and [-1, 1] for the adjusted Rand index.

1.0 is the perfect match score. The score range is [0, 1] for the

* **No assumption is made on the cluster structure**: The (adjusted or
  unadjusted) Rand index can be used to compare all kinds of
  clustering algorithms, and can be used to compare clustering
  algorithms such as k-means which assumes isotropic blob shapes with
  results of spectral clustering algorithms which can find cluster
  with "folded" shapes.

2.3.10.1.2. Drawbacks

* Contrary to inertia, the **(adjusted or unadjusted) Rand index
  requires knowledge of the ground truth classes** which is almost
  never available in practice or requires manual assignment by human
  annotators (as in the supervised learning setting).

  However (adjusted or unadjusted) Rand index can also be useful in a
  purely unsupervised setting as a building block for a Consensus
  Index that can be used for clustering model selection (TODO).

* The **unadjusted Rand index is often close to 1.0** even if the
  clusterings themselves differ significantly. This can be understood
  when interpreting the Rand index as the accuracy of element pair
  labeling resulting from the clusterings: In practice there often is
  a majority of element pairs that are assigned the "different" pair
  label under both the predicted and the ground truth clustering
  resulting in a high proportion of pair labels that agree, which
  leads subsequently to a high score.

Examples:
^^^^^^^^^

label under both the predicted and the ground truth clustering

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_adjusted_for_chance_measures.p
  y*: Analysis of the impact of the dataset size on the value of
  clustering measures for random assignments.

2.3.10.1.3. Mathematical formulation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If C is a ground truth class assignment and K the clustering, let us
define a and b as:

* a, the number of pairs of elements that are in the same set in C and
  in the same set in K

* b, the number of pairs of elements that are in different sets in C
  and in different sets in K

The unadjusted Rand index is then given by:

   \text{RI} = \frac{a + b}{C_2^{n_{samples}}}

where C_2^{n_{samples}} is the total number of possible pairs in the
dataset. It does not matter if the calculation is performed on ordered
pairs or unordered pairs as long as the calculation is performed
consistently.

However, the Rand index does not guarantee that random label
assignments will get a value close to zero (esp. if the number of
clusters is in the same order of magnitude as the number of samples).

To counter this effect we can discount the expected RI E[\text{RI}] of
random labelings by defining the adjusted Rand index as follows:

   \text{ARI} = \frac{\text{RI} - E[\text{RI}]}{\max(\text{RI}) -
   E[\text{RI}]}

References

* Comparing Partitions L. Hubert and P. Arabie, Journal of

\text{ARI} = \frac{\text{RI} - E[\text{RI}]}{\max(\text{RI}) -

  Classification 1985

* Properties of the Hubert-Arabie adjusted Rand index D. Steinley,
  Psychological Methods 2004

* Wikipedia entry for the Rand index

* Wikipedia entry for the adjusted Rand index

2.3.10.2. Mutual Information based scores
-----------------------------------------

samples "labels_pred", the **Mutual Information** is a function that
measures the **agreement** of the two assignments, ignoring
permutations.  Two different normalized versions of this measure are
available, **Normalized Mutual Information (NMI)** and **Adjusted
Mutual Information (AMI)**. NMI is often used in the literature, while
AMI was proposed more recently and is **normalized against chance**:

   >>> metrics.adjusted_mutual_info_score(labels_true, labels_pred)  
   0.22504...

One can permute 0 and 1 in the predicted labels, rename 2 to 3 and get
the same score:

All, "mutual_info_score()", "adjusted_mutual_info_score()" and
"normalized_mutual_info_score()" are symmetric: swapping the argument

0.22504...

does not change the score. Thus they can be used as a **consensus
measure**:

   >>> metrics.adjusted_mutual_info_score(labels_pred, labels_true)  

   1.0

   >>> metrics.normalized_mutual_info_score(labels_true, labels_pred)  
   1.0

This is not true for "mutual_info_score", which is therefore harder to
judge:

   >>> metrics.mutual_info_score(labels_true, labels_pred)  
   0.69...

Bad (e.g. independent labelings) have non-positive scores:

   >>> labels_true = [0, 1, 2, 0, 3, 4, 5, 1]
   >>> labels_pred = [1, 1, 0, 0, 2, 2, 2, 2]
   -0.10526...

2.3.10.2.1. Advantages

* **Random (uniform) label assignments have a AMI score close to 0.0**
  for any value of "n_clusters" and "n_samples" (which is not the case
  for raw Mutual Information or the V-measure for instance).

* **Upper bound  of 1**:  Values close to zero indicate two label
  assignments that are largely independent, while values close to one
  indicate significant agreement. Further, an AMI of exactly 1
  indicates that the two label assignments are equal (with or without
  permutation).

2.3.10.2.2. Drawbacks

indicate significant agreement. Further, an AMI of exactly 1

* Contrary to inertia, **MI-based measures require the knowledge of
  the ground truth classes** while almost never available in practice
  or requires manual assignment by human annotators (as in the
  supervised learning setting).

  However MI-based measures can also be useful in purely unsupervised
  setting as a building block for a Consensus Index that can be used
  for clustering model selection.

* NMI and MI are not adjusted against chance.

Examples:
^^^^^^^^^

  clustering measures for random assignments. This example also
  includes the Adjusted Rand Index.

2.3.10.2.3. Mathematical formulation

Assume two label assignments (of the same N objects), U and V. Their
entropy is the amount of uncertainty for a partition set, defined by:

   H(U) = - \sum_{i=1}^{|U|}P(i)\log(P(i))

where P(i) = |U_i| / N is the probability that an object picked at
random from U falls into class U_i. Likewise for V:

   H(V) = - \sum_{j=1}^{|V|}P'(j)\log(P'(j))

With P'(j) = |V_j| / N. The mutual information (MI) between U and V is
calculated by:

   \text{MI}(U, V) = \sum_{i=1}^{|U|}\sum_{j=1}^{|V|}P(i,
   j)\log\left(\frac{P(i,j)}{P(i)P'(j)}\right)

where P(i, j) = |U_i \cap V_j| / N is the probability that an object
picked at random falls into both classes U_i and V_j.

It also can be expressed in set cardinality formulation:

   \text{MI}(U, V) = \sum_{i=1}^{|U|} \sum_{j=1}^{|V|} \frac{|U_i \cap
   V_j|}{N}\log\left(\frac{N|U_i \cap V_j|}{|U_i||V_j|}\right)

The normalized mutual information is defined as

   \text{NMI}(U, V) = \frac{\text{MI}(U, V)}{\text{mean}(H(U), H(V))}

This value of the mutual information and also the normalized variant
is not adjusted for chance and will tend to increase as the number of
different labels (clusters) increases, regardless of the actual amount
of "mutual information" between the label assignments.

The expected value for the mutual information can be calculated using
the following equation [VEB2009]. In this equation, a_i = |U_i| (the
number of elements in U_i) and b_j = |V_j| (the number of elements in
V_j).

   E[\text{MI}(U,V)]=\sum_{i=1}^{|U|} \sum_{j=1}^{|V|}
   \sum_{n_{ij}=(a_i+b_j-N)^+ }^{\min(a_i, b_j)} \frac{n_{ij}}{N}\log
   \left( \frac{ N.n_{ij}}{a_i b_j}\right)
   \frac{a_i!b_j!(N-a_i)!(N-b_j)!}{N!n_{ij}!(a_i-n_{ij})!(b_j-n_{ij})!
   (N-a_i-b_j+n_{ij})!}

Using the expected value, the adjusted mutual information can then be
calculated using a similar form to that of the adjusted Rand index:

(N-a_i-b_j+n_{ij})!}

   \text{AMI} = \frac{\text{MI} - E[\text{MI}]}{\text{mean}(H(U),
   H(V)) - E[\text{MI}]}

For normalized mutual information and adjusted mutual information, the
normalizing value is typically some *generalized* mean of the
entropies of each clustering. Various generalized means exist, and no
firm rules exist for preferring one over the others.  The decision is
largely a field-by-field basis; for instance, in community detection,
the arithmetic mean is most common. Each normalizing method provides
"qualitatively similar behaviours" [YAT2016]. In our implementation,
this is controlled by the "average_method" parameter.

Vinh et al. (2010) named variants of NMI and AMI by their averaging
method [VEB2010]. Their 'sqrt' and 'sum' averages are the geometric
and arithmetic means; we use these more broadly common names.

* Strehl, Alexander, and Joydeep Ghosh (2002). "Cluster ensembles – a
  knowledge reuse framework for combining multiple partitions".
  Journal of Machine Learning Research 3: 583–617.
  doi:10.1162/153244303321897735.

* Wikipedia entry for the (normalized) Mutual Information

* Wikipedia entry for the Adjusted Mutual Information

[VEB2009] Vinh, Epps, and Bailey, (2009). "Information theoretic
          measures for clusterings comparison". Proceedings of the

          26th Annual International Conference on Machine Learning -
          ICML '09. doi:10.1145/1553374.1553511. ISBN 9781605585161.

[VEB2010] Vinh, Epps, and Bailey, (2010). "Information Theoretic
          Measures for Clusterings Comparison: Variants, Properties,
          Normalization and Correction for Chance". JMLR <https://jml
          r.csail.mit.edu/papers/volume11/vinh10a/vinh10a.pdf>

[YAT2016] Yang, Algesheimer, and Tessone, (2016). "A comparative
          analysis of community detection algorithms on artificial
          networks". Scientific Reports 6: 30750.
          doi:10.1038/srep30750.

2.3.10.3. Homogeneity, completeness and V-measure

Given the knowledge of the ground truth class assignments of the
samples, it is possible to define some intuitive metric using
conditional entropy analysis.

In particular Rosenberg and Hirschberg (2007) define the following two
desirable objectives for any cluster assignment:

* **homogeneity**: each cluster contains only members of a single
  class.

* **completeness**: all members of a given class are assigned to the
  same cluster.

We can turn those concept as scores "homogeneity_score()" and
"completeness_score()". Both are bounded below by 0.0 and above by 1.0
(higher is better):

same cluster.

   >>> metrics.homogeneity_score(labels_true, labels_pred)
   0.66...

   >>> metrics.completeness_score(labels_true, labels_pred)
   0.42...

Their harmonic mean called **V-measure** is computed by
"v_measure_score()":

   >>> metrics.v_measure_score(labels_true, labels_pred)
   0.51...

This function's formula is as follows:

   v = \frac{(1 + \beta) \times \text{homogeneity} \times
   \text{completeness}}{(\beta \times \text{homogeneity} +
   \text{completeness})}

*beta* defaults to a value of 1.0, but for using a value less than 1
for beta:

   >>> metrics.v_measure_score(labels_true, labels_pred, beta=0.6)
   0.54...

more weight will be attributed to homogeneity, and using a value
greater than 1:

   >>> metrics.v_measure_score(labels_true, labels_pred, beta=1.8)
   0.48...

more weight will be attributed to completeness.

The V-measure is actually equivalent to the mutual information (NMI)
discussed above, with the aggregation function being the arithmetic
mean [B2011].

Homogeneity, completeness and V-measure can be computed at once using
"homogeneity_completeness_v_measure()" as follows:

   >>> metrics.homogeneity_completeness_v_measure(labels_true, labels_pred)
   (0.66..., 0.42..., 0.51...)

The following clustering assignment is slightly better, since it is
homogeneous but not complete:

   >>> labels_pred = [0, 0, 0, 1, 2, 2]
   (1.0, 0.68..., 0.81...)

Note:

  "v_measure_score()" is **symmetric**: it can be used to evaluate the
  **agreement** of two independent assignments on the same
  dataset.This is not the case for "completeness_score()" and
  "homogeneity_score()": both are bound by the relationship:

     homogeneity_score(a, b) == completeness_score(b, a)

2.3.10.3.1. Advantages

* **Bounded scores**: 0.0 is as bad as it can be, 1.0 is a perfect
  score.

* Intuitive interpretation: clustering with bad V-measure can be
  **qualitatively analyzed in terms of homogeneity and completeness**
  to better feel what 'kind' of mistakes is done by the assignment.

* **No assumption is made on the cluster structure**: can be used to
  compare clustering algorithms such as k-means which assumes
  isotropic blob shapes with results of spectral clustering algorithms
  which can find cluster with "folded" shapes.

2.3.10.3.2. Drawbacks

* The previously introduced metrics are **not normalized with regards
  to random labeling**: this means that depending on the number of

  samples, clusters and ground truth classes, a completely random
  labeling will not always yield the same values for homogeneity,
  completeness and hence v-measure. In particular **random labeling
  won't yield zero scores especially when the number of clusters is
  large**.

  This problem can safely be ignored when the number of samples is
  more than a thousand and the number of clusters is less than 10.
  **For smaller sample sizes or larger number of clusters it is safer
  to use an adjusted index such as the Adjusted Rand Index (ARI)**.

   [image]

* These metrics **require the knowledge of the ground truth classes**
  while almost never available in practice or requires manual
  assignment by human annotators (as in the supervised learning
  setting).

Examples:
^^^^^^^^^

2.3.10.3.3. Mathematical formulation

Homogeneity and completeness scores are formally given by:

   h = 1 - \frac{H(C|K)}{H(C)}

   c = 1 - \frac{H(K|C)}{H(K)}

where H(C|K) is the **conditional entropy of the classes given the
cluster assignments** and is given by:

   H(C|K) = - \sum_{c=1}^{|C|} \sum_{k=1}^{|K|} \frac{n_{c,k}}{n}

c = 1 - \frac{H(K|C)}{H(K)}

   \cdot \log\left(\frac{n_{c,k}}{n_k}\right)

and H(C) is the **entropy of the classes** and is given by:

   H(C) = - \sum_{c=1}^{|C|} \frac{n_c}{n} \cdot
   \log\left(\frac{n_c}{n}\right)

with n the total number of samples, n_c and n_k the number of samples
respectively belonging to class c and cluster k, and finally n_{c,k}
the number of samples from class c assigned to cluster k.

The **conditional entropy of clusters given class** H(K|C) and the
**entropy of clusters** H(K) are defined in a symmetric manner.

Rosenberg and Hirschberg further define **V-measure** as the
**harmonic mean of homogeneity and completeness**:

   v = 2 \cdot \frac{h \cdot c}{h + c}

* V-Measure: A conditional entropy-based external cluster evaluation
  measure Andrew Rosenberg and Julia Hirschberg, 2007

[B2011] Identication and Characterization of Events in Social Media,
        Hila Becker, PhD Thesis.

2.3.10.4. Fowlkes-Mallows scores

The Fowlkes-Mallows index ("sklearn.metrics.fowlkes_mallows_score()")
can be used when the ground truth class assignments of the samples is
known. The Fowlkes-Mallows score FMI is defined as the geometric mean
of the pairwise precision and recall:

   \text{FMI} = \frac{\text{TP}}{\sqrt{(\text{TP} + \text{FP})
   (\text{TP} + \text{FN})}}

Where "TP" is the number of **True Positive** (i.e. the number of pair
of points that belong to the same clusters in both the true labels and
the predicted labels), "FP" is the number of **False Positive** (i.e.
the number of pair of points that belong to the same clusters in the
true labels and not in the predicted labels) and "FN" is the number of
**False Negative** (i.e the number of pair of points that belongs in
the same clusters in the predicted labels and not in the true labels).

The score ranges from 0 to 1. A high value indicates a good similarity
between two clusters.

>>> from sklearn import metrics
>>> labels_true = [0, 0, 0, 1, 1, 1]
>>> labels_pred = [0, 0, 1, 1, 2, 2]

>>> metrics.fowlkes_mallows_score(labels_true, labels_pred)
0.47140...

   >>> metrics.fowlkes_mallows_score(labels_true, labels_pred)
   0.47140...

   1.0

Bad (e.g. independent labelings) have zero scores:

   0.0

>>> labels_true = [0, 1, 2, 0, 3, 4, 5, 1]
   0.0

2.3.10.4.1. Advantages

* **Random (uniform) label assignments have a FMI score close to 0.0**

* **Upper-bounded at 1**:  Values close to zero indicate two label
  indicate significant agreement. Further, values of exactly 0
  indicate **purely** independent label assignments and a FMI of
  exactly 1 indicates that the two label assignments are equal (with
  or without permutation).

2.3.10.4.2. Drawbacks

* Contrary to inertia, **FMI-based measures require the knowledge of

* E. B. Fowkles and C. L. Mallows, 1983. "A method for comparing two
  hierarchical clusterings". Journal of the American Statistical
  Association.

supervised learning setting).

  https://www.tandfonline.com/doi/abs/10.1080/01621459.1983.10478008

* Wikipedia entry for the Fowlkes-Mallows Index

2.3.10.5. Silhouette Coefficient

If the ground truth labels are not known, evaluation must be performed
using the model itself. The Silhouette Coefficient
("sklearn.metrics.silhouette_score()") is an example of such an
evaluation, where a higher Silhouette Coefficient score relates to a
model with better defined clusters. The Silhouette Coefficient is
defined for each sample and is composed of two scores:

* **a**: The mean distance between a sample and all other points in
  the same class.

* **b**: The mean distance between a sample and all other points in
  the *next nearest cluster*.

The Silhouette Coefficient *s* for a single sample is then given as:

   s = \frac{b - a}{max(a, b)}

The Silhouette Coefficient for a set of samples is given as the mean
of the Silhouette Coefficient for each sample.

>>> from sklearn.metrics import pairwise_distances
>>> from sklearn import datasets
>>> X, y = datasets.load_iris(return_X_y=True)

In normal usage, the Silhouette Coefficient is applied to the results
of a cluster analysis.

>>> import numpy as np
>>> from sklearn.cluster import KMeans

>>> kmeans_model = KMeans(n_clusters=3, random_state=1).fit(X)
>>> labels = kmeans_model.labels_
>>> metrics.silhouette_score(X, labels, metric='euclidean')
0.55...

* Peter J. Rousseeuw (1987). >>:doi:`"Silhouettes: a Graphical Aid to
  the Interpretation and Validation of Cluster
  Analysis"<10.1016/0377-0427(87)90125-7>`<< . Computational and
  Applied Mathematics 20: 53–65.

2.3.10.5.1. Advantages

* The score is bounded between -1 for incorrect clustering and +1 for
  highly dense clustering. Scores around zero indicate overlapping
  clusters.

* The score is higher when clusters are dense and well separated,
  which relates to a standard concept of a cluster.

2.3.10.5.2. Drawbacks

* The Silhouette Coefficient is generally higher for convex clusters
  than other concepts of clusters, such as density based clusters like
  those obtained through DBSCAN.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cluster_plot_kmeans_silhouette_analysis.py*
  : In this example the silhouette analysis is used to choose an
  optimal value for n_clusters.

2.3.10.6. Calinski-Harabasz Index

If the ground truth labels are not known, the Calinski-Harabasz index
("sklearn.metrics.calinski_harabasz_score()") - also known as the

Variance Ratio Criterion - can be used to evaluate the model, where a
higher Calinski-Harabasz score relates to a model with better defined
clusters.

The index is the ratio of the sum of between-clusters dispersion and
of within-cluster dispersion for all clusters (where dispersion is
defined as the sum of distances squared):

In normal usage, the Calinski-Harabasz index is applied to the results
of a cluster analysis:

>>> metrics.calinski_harabasz_score(X, labels)
561.62...

2.3.10.6.1. Advantages

* The score is fast to compute.

2.3.10.6.2. Drawbacks

* The Calinski-Harabasz index is generally higher for convex clusters

2.3.10.6.3. Mathematical formulation

than other concepts of clusters, such as density based clusters like

For a set of data E of size n_E which has been clustered into k
clusters, the Calinski-Harabasz score s is defined as the ratio of the
between-clusters dispersion mean and the within-cluster dispersion:

   s = \frac{\mathrm{tr}(B_k)}{\mathrm{tr}(W_k)} \times \frac{n_E -
   k}{k - 1}

where \mathrm{tr}(B_k) is trace of the between group dispersion matrix
and \mathrm{tr}(W_k) is the trace of the within-cluster dispersion
matrix defined by:

   W_k = \sum_{q=1}^k \sum_{x \in C_q} (x - c_q) (x - c_q)^T

   B_k = \sum_{q=1}^k n_q (c_q - c_E) (c_q - c_E)^T

with C_q the set of points in cluster q, c_q the center of cluster q,
c_E the center of E, and n_q the number of points in cluster q.

* Caliński, T., & Harabasz, J. (1974). "A Dendrite Method for Cluster
  Analysis". >>:doi:`Communications in Statistics-theory and Methods
  3: 1-27 <10.1080/03610927408827101>`<<.

2.3.10.7. Davies-Bouldin Index

If the ground truth labels are not known, the Davies-Bouldin index
("sklearn.metrics.davies_bouldin_score()") can be used to evaluate the
model, where a lower Davies-Bouldin index relates to a model with
better separation between the clusters.

This index signifies the average 'similarity' between clusters, where
the similarity is a measure that compares the distance between

clusters with the size of the clusters themselves.

Zero is the lowest possible score. Values closer to zero indicate a
better partition.

In normal usage, the Davies-Bouldin index is applied to the results of
a cluster analysis as follows:

>>> iris = datasets.load_iris()
>>> X = iris.data
>>> from sklearn.metrics import davies_bouldin_score
>>> kmeans = KMeans(n_clusters=3, random_state=1).fit(X)
>>> labels = kmeans.labels_
>>> davies_bouldin_score(X, labels)
0.6619...

2.3.10.7.1. Advantages

* The computation of Davies-Bouldin is simpler than that of Silhouette
  scores.

* The index is solely based on quantities and features inherent to the
  dataset as its computation only uses point-wise distances.

2.3.10.7.2. Drawbacks

* The Davies-Boulding index is generally higher for convex clusters
  those obtained from DBSCAN.

* The usage of centroid distance limits the distance metric to
  Euclidean space.

2.3.10.7.3. Mathematical formulation

The index is defined as the average similarity between each cluster
C_i for i=1, ..., k and its most similar one C_j. In the context of

this index, similarity is defined as a measure R_{ij} that trades off:

* s_i, the average distance between each point of cluster i and the
  centroid of that cluster -- also know as cluster diameter.

* d_{ij}, the distance between cluster centroids i and j.

A simple choice to construct R_{ij} so that it is nonnegative and
symmetric is:

   R_{ij} = \frac{s_i + s_j}{d_{ij}}

Then the Davies-Bouldin index is defined as:

   DB = \frac{1}{k} \sum_{i=1}^k \max_{i \neq j} R_{ij}

* Davies, David L.; Bouldin, Donald W. (1979). >>:doi:`"A Cluster
  Separation Measure" <10.1109/TPAMI.1979.4766909>`<< IEEE
  Transactions on Pattern Analysis and Machine Intelligence. PAMI-1
  (2): 224-227.

* Halkidi, Maria; Batistakis, Yannis; Vazirgiannis, Michalis (2001).
  >>:doi:`"On Clustering Validation Techniques"
  <10.1023/A:1012801612483>`<< Journal of Intelligent Information
  Systems, 17(2-3), 107-145.

* Wikipedia entry for Davies-Bouldin index.

2.3.10.8. Contingency Matrix

Contingency matrix ("sklearn.metrics.cluster.contingency_matrix()")
reports the intersection cardinality for every true/predicted cluster
pair. The contingency matrix provides sufficient statistics for all
clustering metrics where the samples are independent and identically

distributed and one doesn't need to account for some instances not
being clustered.

Here is an example:

   >>> from sklearn.metrics.cluster import contingency_matrix
   >>> x = ["a", "a", "a", "b", "b", "b"]
   >>> y = [0, 0, 1, 1, 2, 2]
   >>> contingency_matrix(x, y)
   array([[2, 1, 0],
          [0, 1, 2]])

The first row of output array indicates that there are three samples
whose true cluster is "a". Of them, two are in predicted cluster 0,
one is in 1, and none is in 2. And the second row indicates that there
are three samples whose true cluster is "b". Of them, none is in
predicted cluster 0, one is in 1 and two are in 2.

A confusion matrix for classification is a square contingency matrix
where the order of rows and columns correspond to a list of classes.

2.3.10.8.1. Advantages

* Allows to examine the spread of each true cluster across predicted
  clusters and vice versa.

* The contingency table calculated is typically utilized in the
  calculation of a similarity statistic (like the others listed in
  this document) between the two clusterings.

2.3.10.8.2. Drawbacks

* Contingency matrix is easy to interpret for a small number of
  clusters, but becomes very hard to interpret for a large number of
  clusters.

* It doesn't give a single metric to use as an objective for
  clustering optimisation.

clusters, but becomes very hard to interpret for a large number of
  clusters.

* Wikipedia entry for contingency matrix

2.3.10.9. Pair Confusion Matrix

The pair confusion matrix
("sklearn.metrics.cluster.pair_confusion_matrix()") is a 2x2
similarity matrix

   C = \left[\begin{matrix} C_{00} & C_{01} \\ C_{10} & C_{11}
   \end{matrix}\right]

between two clusterings computed by considering all pairs of samples
and counting pairs that are assigned into the same or into different
clusters under the true and predicted clusterings.

It has the following entries:

   C_{00} : number of pairs with both clusterings having the samples
   not clustered together

   C_{10} : number of pairs with the true label clustering having the
   samples clustered together but the other clustering not having the
   samples clustered together

   C_{01} : number of pairs with the true label clustering not having
   the samples clustered together but the other clustering having the

   C_{11} : number of pairs with both clusterings having the samples
   clustered together

Considering a pair of samples that is clustered together a positive
pair, then as in binary classification the count of true negatives is
C_{00}, false negatives is C_{10}, true positives is C_{11} and false
positives is C_{01}.

Perfectly matching labelings have all non-zero entries on the diagonal

regardless of actual label values:

   >>> from sklearn.metrics.cluster import pair_confusion_matrix
   >>> pair_confusion_matrix([0, 0, 1, 1], [0, 0, 1, 1])
   array([[8, 0],
          [0, 4]])

   >>> pair_confusion_matrix([0, 0, 1, 1], [1, 1, 0, 0])
          [0, 4]])

Labelings that assign all classes members to the same clusters are
complete but may not always be pure, hence penalized, and have some
off-diagonal non-zero entries:

   >>> pair_confusion_matrix([0, 0, 1, 2], [0, 0, 1, 1])
   array([[8, 2],
          [0, 2]])

The matrix is not symmetric:

   >>> pair_confusion_matrix([0, 0, 1, 1], [0, 0, 1, 2])
          [2, 2]])

If classes members are completely split across different clusters, the
assignment is totally incomplete, hence the matrix has all zero
diagonal entries:

   >>> pair_confusion_matrix([0, 0, 0, 0], [0, 1, 2, 3])
   array([[ 0,  0],
          [12,  0]])

* >>:doi:`"Comparing Partitions" <10.1007/BF01908075>`<< L. Hubert and
  P. Arabie, Journal of Classification 1985

1.3. Kernel ridge regression

Kernel ridge regression (KRR) [M2012] combines Ridge regression and
classification (linear least squares with l2-norm regularization) with
the kernel trick. It thus learns a linear function in the space
induced by the respective kernel and the data. For non-linear kernels,
this corresponds to a non-linear function in the original space.

The form of the model learned by "KernelRidge" is identical to support
vector regression ("SVR"). However, different loss functions are used:
KRR uses squared error loss while support vector regression uses
\epsilon-insensitive loss, both combined with l2 regularization. In
contrast to "SVR", fitting "KernelRidge" can be done in closed-form
and is typically faster for medium-sized datasets. On the other hand,
the learned model is non-sparse and thus slower than "SVR", which
learns a sparse model for \epsilon > 0, at prediction-time.

The following figure compares "KernelRidge" and "SVR" on an artificial
dataset, which consists of a sinusoidal target function and strong
noise added to every fifth datapoint. The learned model of
"KernelRidge" and "SVR" is plotted, where both
complexity/regularization and bandwidth of the RBF kernel have been
optimized using grid-search. The learned functions are very similar;
however, fitting "KernelRidge" is approximately seven times faster
than fitting "SVR" (both with grid-search). However, prediction of
100000 target values is more than three times faster with "SVR" since

it has learned a sparse model using only approximately 1/3 of the 100
training datapoints as support vectors.

   [image]

The next figure compares the time for fitting and prediction of
"KernelRidge" and "SVR" for different sizes of the training set.
Fitting "KernelRidge" is faster than "SVR" for medium-sized training
sets (less than 1000 samples); however, for larger training sets "SVR"
scales better. With regard to prediction time, "SVR" is faster than
"KernelRidge" for all sizes of the training set because of the learned
sparse solution. Note that the degree of sparsity and thus the
prediction time depends on the parameters \epsilon and C of the "SVR";
\epsilon = 0 would correspond to a dense model.

   [image]

[M2012] "Machine Learning: A Probabilistic Perspective" Murphy, K. P.
        - chapter 14.4.3, pp. 492-493, The MIT Press, 2012

1.1. Linear Models
******************

The following are a set of methods intended for regression in which
the target value is expected to be a linear combination of the
features. In mathematical notation, if \hat{y} is the predicted value.

   \hat{y}(w, x) = w_0 + w_1 x_1 + ... + w_p x_p

Across the module, we designate the vector w = (w_1, ..., w_p) as
"coef_" and w_0 as "intercept_".

To perform classification with generalized linear models, see Logistic
regression.

1.1.1. Ordinary Least Squares

"LinearRegression" fits a linear model with coefficients w = (w_1,
..., w_p) to minimize the residual sum of squares between the observed
targets in the dataset, and the targets predicted by the linear
approximation. Mathematically it solves a problem of the form:

   \min_{w} || X w - y||_2^2

   [image]

"LinearRegression" will take in its "fit" method arrays X, y and will
store the coefficients w of the linear model in its "coef_" member:

   >>> reg = linear_model.LinearRegression()
   >>> reg.fit([[0, 0], [1, 1], [2, 2]], [0, 1, 2])
   >>> reg.coef_
   array([0.5, 0.5])

The coefficient estimates for Ordinary Least Squares rely on the
independence of the features. When features are correlated and the
columns of the design matrix X have an approximately linear
dependence, the design matrix becomes close to singular and as a
result, the least-squares estimate becomes highly sensitive to random

errors in the observed target, producing a large variance. This
situation of *multicollinearity* can arise, for example, when data are
collected without an experimental design.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_ols.py*

1.1.1.1. Non-Negative Least Squares

It is possible to constrain all the coefficients to be non-negative,
which may be useful when they represent some physical or naturally
non-negative quantities (e.g., frequency counts or prices of goods).
"LinearRegression" accepts a boolean "positive" parameter: when set to
*True* Non-Negative Least Squares are then applied.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_nnls.py*

1.1.1.2. Ordinary Least Squares Complexity

The least squares solution is computed using the singular value
decomposition of X. If X is a matrix of shape *(n_samples,
n_features)* this method has a cost of O(n_{\text{samples}}
n_{\text{features}}^2), assuming that n_{\text{samples}} \geq
n_{\text{features}}.

1.1.2. Ridge regression and classification

1.1.2.1. Regression

"Ridge" regression addresses some of the problems of Ordinary Least
Squares by imposing a penalty on the size of the coefficients. The

ridge coefficients minimize a penalized residual sum of squares:

   \min_{w} || X w - y||_2^2 + \alpha ||w||_2^2

The complexity parameter \alpha \geq 0 controls the amount of
shrinkage: the larger the value of \alpha, the greater the amount of
shrinkage and thus the coefficients become more robust to
collinearity.

   [image]

As with other linear models, "Ridge" will take in its "fit" method
arrays X, y and will store the coefficients w of the linear model in
its "coef_" member:

   >>> reg = linear_model.Ridge(alpha=.5)
   >>> reg.fit([[0, 0], [0, 0], [1, 1]], [0, .1, 1])
   Ridge(alpha=0.5)
   array([0.34545455, 0.34545455])
   >>> reg.intercept_
   0.13636...

Note that the class "Ridge" allows for the user to specify that the
solver be automatically chosen by setting *solver="auto"*. When this
option is specified, "Ridge" will choose between the *"lbfgs"*,
*"cholesky"*, and *"sparse_cg"* solvers. "Ridge" will begin checking
the conditions shown in the following table from top to bottom. If the
condition is true, the corresponding solver is chosen.

+---------------+------------------------------------------------------+
| **Solver**    | **Condition**                                        |

| 'lbfgs'       | The "positive=True" option is specified.             |
| 'cholesky'    | The input array X is not sparse.                     |
| 'sparse_cg'   | None of the above conditions are fulfilled.          |

1.1.2.2. Classification

The "Ridge" regressor has a classifier variant: "RidgeClassifier".
This classifier first converts binary targets to "{-1, 1}" and then
treats the problem as a regression task, optimizing the same objective
as above. The predicted class corresponds to the sign of the
regressor's prediction. For multiclass classification, the problem is
treated as multi-output regression, and the predicted class
corresponds to the output with the highest value.

It might seem questionable to use a (penalized) Least Squares loss to
fit a classification model instead of the more traditional logistic or
hinge losses. However, in practice, all those models can lead to
similar cross-validation scores in terms of accuracy or
precision/recall, while the penalized least squares loss used by the
"RidgeClassifier" allows for a very different choice of the numerical

solvers with distinct computational performance profiles.

The "RidgeClassifier" can be significantly faster than e.g.
"LogisticRegression" with a high number of classes because it can
compute the projection matrix (X^T X)^{-1} X^T only once.

This classifier is sometimes referred to as a Least Squares Support
Vector Machines with a linear kernel.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_ridge_path.py*

* *sphx_glr_auto_examples_text_plot_document_classification_20newsgro
  ups.py*

1.1.2.3. Ridge Complexity

This method has the same order of complexity as Ordinary Least
Squares.

1.1.2.4. Setting the regularization parameter: leave-one-out Cross-Validation
-----------------------------------------------------------------------------

"RidgeCV" implements ridge regression with built-in cross-validation
of the alpha parameter. The object works in the same way as
GridSearchCV except that it defaults to Leave-One-Out Cross-
Validation:

   >>> reg = linear_model.RidgeCV(alphas=np.logspace(-6, 6, 13))

   RidgeCV(alphas=array([1.e-06, 1.e-05, 1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01,
         1.e+02, 1.e+03, 1.e+04, 1.e+05, 1.e+06]))
   >>> reg.alpha_
   0.01

Specifying the value of the *cv* attribute will trigger the use of
cross-validation with "GridSearchCV", for example *cv=10* for 10-fold
cross-validation, rather than Leave-One-Out Cross-Validation.

* "Notes on Regularized Least Squares", Rifkin & Lippert (technical
  report, course slides).

1.1.3. Lasso

The "Lasso" is a linear model that estimates sparse coefficients. It
is useful in some contexts due to its tendency to prefer solutions
with fewer non-zero coefficients, effectively reducing the number of
features upon which the given solution is dependent. For this reason,
Lasso and its variants are fundamental to the field of compressed
sensing. Under certain conditions, it can recover the exact set of
non-zero coefficients (see *sphx_glr_auto_examples_applications_plot_
tomography_l1_reconstruction.py*).

Mathematically, it consists of a linear model with an added
regularization term. The objective function to minimize is:

   \min_{w} { \frac{1}{2n_{\text{samples}}} ||X w - y||_2 ^ 2 + \alpha
   ||w||_1}

The lasso estimate thus solves the minimization of the least-squares

\min_{w} { \frac{1}{2n_{\text{samples}}} ||X w - y||_2 ^ 2 + \alpha
   ||w||_1}

penalty with \alpha ||w||_1 added, where \alpha is a constant and
||w||_1 is the \ell_1-norm of the coefficient vector.

The implementation in the class "Lasso" uses coordinate descent as the
algorithm to fit the coefficients. See Least Angle Regression for
another implementation:

   >>> reg = linear_model.Lasso(alpha=0.1)
   >>> reg.fit([[0, 0], [1, 1]], [0, 1])
   Lasso(alpha=0.1)
   >>> reg.predict([[1, 1]])
   array([0.8])

The function "lasso_path()" is useful for lower-level tasks, as it
computes the coefficients along the full path of possible values.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_lasso_and_elasticnet.py*

* *sphx_glr_auto_examples_applications_plot_tomography_l1_reconstruct
  ion.py*

Note:

  **Feature selection with Lasso**As the Lasso regression yields
  sparse models, it can thus be used to perform feature selection, as
  detailed in L1-based feature selection.

The following two references explain the iterations used in the
coordinate descent solver of scikit-learn, as well as the duality gap
computation used for convergence control.

* "Regularization Path For Generalized linear Models by Coordinate
  Descent", Friedman, Hastie & Tibshirani, J Stat Softw, 2010 (Paper).

* "An Interior-Point Method for Large-Scale L1-Regularized Least
  Squares," S. J. Kim, K. Koh, M. Lustig, S. Boyd and D. Gorinevsky,
  in IEEE Journal of Selected Topics in Signal Processing, 2007
  (Paper)

1.1.3.1. Setting regularization parameter

The "alpha" parameter controls the degree of sparsity of the estimated
coefficients.

1.1.3.1.1. Using cross-validation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

scikit-learn exposes objects that set the Lasso "alpha" parameter by
cross-validation: "LassoCV" and "LassoLarsCV". "LassoLarsCV" is based
on the Least Angle Regression algorithm explained below.

For high-dimensional datasets with many collinear features, "LassoCV"
is most often preferable. However, "LassoLarsCV" has the advantage of
exploring more relevant values of *alpha* parameter, and if the number
of samples is very small compared to the number of features, it is
often faster than "LassoCV".

[image: lasso_cv_1][image] [image: lasso_cv_2][image]

1.1.3.1.2. Information-criteria based model selection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Alternatively, the estimator "LassoLarsIC" proposes to use the Akaike
information criterion (AIC) and the Bayes Information criterion (BIC).
It is a computationally cheaper alternative to find the optimal value

of alpha as the regularization path is computed only once instead of
k+1 times when using k-fold cross-validation.

Indeed, these criteria are computed on the in-sample training set. In
short, they penalize the over-optimistic scores of the different Lasso
models by their flexibility (cf. to "Mathematical details" section
below).

However, such criteria need a proper estimation of the degrees of
freedom of the solution, are derived for large samples (asymptotic
results) and assume the correct model is candidates under
investigation. They also tend to break when the problem is badly
conditioned (e.g. more features than samples).

   [image]

**Mathematical details**

The definition of AIC (and thus BIC) might differ in the literature.
In this section, we give more information regarding the criterion
computed in scikit-learn. The AIC criterion is defined as:

   AIC = -2 \log(\hat{L}) + 2 d

where \hat{L} is the maximum likelihood of the model and d is the
number of parameters (as well referred to as degrees of freedom in the
previous section).

The definition of BIC replace the constant 2 by \log(N):

   BIC = -2 \log(\hat{L}) + \log(N) d

where N is the number of samples.

For a linear Gaussian model, the maximum log-likelihood is defined as:

   \log(\hat{L}) = - \frac{n}{2} \log(2 \pi) - \frac{n}{2}
   \ln(\sigma^2) - \frac{\sum_{i=1}^{n} (y_i -
   \hat{y}_i)^2}{2\sigma^2}

\log(\hat{L}) = - \frac{n}{2} \log(2 \pi) - \frac{n}{2}

where \sigma^2 is an estimate of the noise variance, y_i and \hat{y}_i
are respectively the true and predicted targets, and n is the number
of samples.

Plugging the maximum log-likelihood in the AIC formula yields:

   AIC = n \log(2 \pi \sigma^2) + \frac{\sum_{i=1}^{n} (y_i -
   \hat{y}_i)^2}{\sigma^2} + 2 d

The first term of the above expression is sometimes discarded since it
is a constant when \sigma^2 is provided. In addition, it is sometimes
stated that the AIC is equivalent to the C_p statistic [12]. In a
strict sense, however, it is equivalent only up to some constant and a
multiplicative factor.

At last, we mentioned above that \sigma^2 is an estimate of the noise
variance. In "LassoLarsIC" when the parameter *noise_variance* is not
provided (default), the noise variance is estimated via the unbiased
estimator [13] defined as:

   \sigma^2 = \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{n - p}

where p is the number of features and \hat{y}_i is the predicted
target using an ordinary least squares regression. Note, that this
formula is valid only when *n_samples > n_features*.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_lasso_model_selection.py*

* *sphx_glr_auto_examples_linear_model_plot_lasso_lars_ic.py*

[12] >>:arxiv:`Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the

     degrees of freedom of the lasso." The Annals of Statistics 35.5
     (2007): 2173-2192. <0712.0881.pdf>`<<

[13] >>:doi:`Cherkassky, Vladimir, and Yunqian Ma. "Comparison of
     model selection for regression." Neural computation 15.7 (2003):
     1691-1714. <10.1162/089976603321891864>`<<

1.1.3.1.3. Comparison with the regularization parameter of SVM
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The equivalence between "alpha" and the regularization parameter of
SVM, "C" is given by "alpha = 1 / C" or "alpha = 1 / (n_samples * C)",
depending on the estimator and the exact objective function optimized
by the model.

1.1.4. Multi-task Lasso

The "MultiTaskLasso" is a linear model that estimates sparse
coefficients for multiple regression problems jointly: "y" is a 2D
array, of shape "(n_samples, n_tasks)". The constraint is that the
selected features are the same for all the regression problems, also
called tasks.

The following figure compares the location of the non-zero entries in
the coefficient matrix W obtained with a simple Lasso or a
MultiTaskLasso. The Lasso estimates yield scattered non-zeros while
the non-zeros of the MultiTaskLasso are full columns.

[image: multi_task_lasso_1][image] [image:
multi_task_lasso_2][image]Fitting a time-series model, imposing that

any active feature be active at all times.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_multi_task_lasso_support.
  py*

Mathematically, it consists of a linear model trained with a mixed
\ell_1 \ell_2-norm for regularization. The objective function to
minimize is:

   \min_{W} { \frac{1}{2n_{\text{samples}}} ||X W - Y||_{\text{Fro}} ^
   2 + \alpha ||W||_{21}}

where \text{Fro} indicates the Frobenius norm

   ||A||_{\text{Fro}} = \sqrt{\sum_{ij} a_{ij}^2}

and \ell_1 \ell_2 reads

   ||A||_{2 1} = \sum_i \sqrt{\sum_j a_{ij}^2}.

The implementation in the class "MultiTaskLasso" uses coordinate
descent as the algorithm to fit the coefficients.

1.1.5. Elastic-Net

"ElasticNet" is a linear regression model trained with both \ell_1 and
\ell_2-norm regularization of the coefficients. This combination
allows for learning a sparse model where few of the weights are non-
zero like "Lasso", while still maintaining the regularization
properties of "Ridge". We control the convex combination of \ell_1 and
\ell_2 using the "l1_ratio" parameter.

Elastic-net is useful when there are multiple features that are
correlated with one another. Lasso is likely to pick one of these at
random, while elastic-net is likely to pick both.

A practical advantage of trading-off between Lasso and Ridge is that

it allows Elastic-Net to inherit some of Ridge's stability under
rotation.

The objective function to minimize is in this case

   \rho ||w||_1 + \frac{\alpha(1-\rho)}{2} ||w||_2 ^ 2}

   [image]

The class "ElasticNetCV" can be used to set the parameters "alpha"
(\alpha) and "l1_ratio" (\rho) by cross-validation.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_lasso_coordinate_descent_
  path.py*

  (Paper)

1.1.6. Multi-task Elastic-Net

The "MultiTaskElasticNet" is an elastic-net model that estimates
sparse coefficients for multiple regression problems jointly: "Y" is a

2D array of shape "(n_samples, n_tasks)". The constraint is that the

\ell_1 \ell_2-norm and \ell_2-norm for regularization. The objective
function to minimize is:

   \min_{W} { \frac{1}{2n_{\text{samples}}} ||X W - Y||_{\text{Fro}}^2
   + \alpha \rho ||W||_{2 1} + \frac{\alpha(1-\rho)}{2}
   ||W||_{\text{Fro}}^2}

The implementation in the class "MultiTaskElasticNet" uses coordinate

The class "MultiTaskElasticNetCV" can be used to set the parameters
"alpha" (\alpha) and "l1_ratio" (\rho) by cross-validation.

1.1.7. Least Angle Regression

Least-angle regression (LARS) is a regression algorithm for high-
dimensional data, developed by Bradley Efron, Trevor Hastie, Iain
Johnstone and Robert Tibshirani. LARS is similar to forward stepwise
regression. At each step, it finds the feature most correlated with
the target. When there are multiple features having equal correlation,
instead of continuing along the same feature, it proceeds in a
direction equiangular between the features.

The advantages of LARS are:

   * It is numerically efficient in contexts where the number of

     features is significantly greater than the number of samples.

   * It is computationally just as fast as forward selection and has
     the same order of complexity as ordinary least squares.

   * It produces a full piecewise linear solution path, which is
     useful in cross-validation or similar attempts to tune the model.

   * If two features are almost equally correlated with the target,
     then their coefficients should increase at approximately the same
     rate. The algorithm thus behaves as intuition would expect, and
     also is more stable.

   * It is easily modified to produce solutions for other estimators,
     like the Lasso.

The disadvantages of the LARS method include:

   * Because LARS is based upon an iterative refitting of the
     residuals, it would appear to be especially sensitive to the
     effects of noise. This problem is discussed in detail by Weisberg
     in the discussion section of the Efron et al. (2004) Annals of
     Statistics article.

The LARS model can be used via the estimator "Lars", or its low-level
implementation "lars_path()" or "lars_path_gram()".

1.1.8. LARS Lasso

"LassoLars" is a lasso model implemented using the LARS algorithm, and
unlike the implementation based on coordinate descent, this yields the

exact solution, which is piecewise linear as a function of the norm of
its coefficients.

   [image]

   >>> reg = linear_model.LassoLars(alpha=.1)
   LassoLars(alpha=0.1)
   array([0.6..., 0.        ])

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_lasso_lars.py*

The Lars algorithm provides the full path of the coefficients along
the regularization parameter almost for free, thus a common operation
is to retrieve the path with one of the functions "lars_path()" or
"lars_path_gram()".

1.1.8.1. Mathematical formulation

The algorithm is similar to forward stepwise regression, but instead
of including features at each step, the estimated coefficients are
increased in a direction equiangular to each one's correlations with
the residual.

Instead of giving a vector result, the LARS solution consists of a
curve denoting the solution for each value of the \ell_1 norm of the
parameter vector. The full coefficients path is stored in the array
"coef_path_" of shape *(n_features, max_features + 1)*. The first
column is always zero.

* Original Algorithm is detailed in the paper Least Angle Regression
  by Hastie et al.

1.1.9. Orthogonal Matching Pursuit (OMP)

"OrthogonalMatchingPursuit" and "orthogonal_mp()" implement the OMP
algorithm for approximating the fit of a linear model with constraints
imposed on the number of non-zero coefficients (ie. the \ell_0 pseudo-
norm).

Being a forward feature selection method like Least Angle Regression,
orthogonal matching pursuit can approximate the optimum solution
vector with a fixed number of non-zero elements:

   \underset{w}{\operatorname{arg\,min\,}}  ||y - Xw||_2^2 \text{
   subject to } ||w||_0 \leq n_{\text{nonzero\_coefs}}

Alternatively, orthogonal matching pursuit can target a specific error
instead of a specific number of non-zero coefficients. This can be
expressed as:

   \underset{w}{\operatorname{arg\,min\,}} ||w||_0 \text{ subject to }
   ||y-Xw||_2^2 \leq \text{tol}

OMP is based on a greedy algorithm that includes at each step the atom
most highly correlated with the current residual. It is similar to the
simpler matching pursuit (MP) method, but better in that at each
iteration, the residual is recomputed using an orthogonal projection
on the space of the previously chosen dictionary elements.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_omp.py*

* https://www.cs.technion.ac.il/~ronrubin/Publications/KSVD-OMP-v2.pdf

Examples:
^^^^^^^^^

* Matching pursuits with time-frequency dictionaries, S. G. Mallat, Z.
  Zhang,

1.1.10. Bayesian Regression

Bayesian regression techniques can be used to include regularization
parameters in the estimation procedure: the regularization parameter
is not set in a hard sense but tuned to the data at hand.

This can be done by introducing uninformative priors over the hyper
parameters of the model. The \ell_{2} regularization used in Ridge
regression and classification is equivalent to finding a maximum a
posteriori estimation under a Gaussian prior over the coefficients w
with precision \lambda^{-1}. Instead of setting *lambda* manually, it
is possible to treat it as a random variable to be estimated from the
data.

To obtain a fully probabilistic model, the output y is assumed to be
Gaussian distributed around X w:

   p(y|X,w,\alpha) = \mathcal{N}(y|X w,\alpha)

where \alpha is again treated as a random variable that is to be
estimated from the data.

The advantages of Bayesian Regression are:

   * It adapts to the data at hand.

   * It can be used to include regularization parameters in the
     estimation procedure.

The disadvantages of Bayesian regression include:

   * Inference of the model can be time consuming.

* A good introduction to Bayesian methods is given in C. Bishop:

  Pattern Recognition and Machine learning

* Original Algorithm is detailed in the  book *Bayesian learning for
  neural networks* by Radford M. Neal

1.1.10.1. Bayesian Ridge Regression

"BayesianRidge" estimates a probabilistic model of the regression
problem as described above. The prior for the coefficient w is given
by a spherical Gaussian:

   p(w|\lambda) = \mathcal{N}(w|0,\lambda^{-1}\mathbf{I}_{p})

The priors over \alpha and \lambda are chosen to be gamma
distributions, the conjugate prior for the precision of the Gaussian.
The resulting model is called *Bayesian Ridge Regression*, and is
similar to the classical "Ridge".

The parameters w, \alpha and \lambda are estimated jointly during the
fit of the model, the regularization parameters \alpha and \lambda
being estimated by maximizing the *log marginal likelihood*. The
scikit-learn implementation is based on the algorithm described in
Appendix A of (Tipping, 2001) where the update of the parameters
\alpha and \lambda is done as suggested in (MacKay, 1992). The initial
value of the maximization procedure can be set with the
hyperparameters "alpha_init" and "lambda_init".

There are four more hyperparameters, \alpha_1, \alpha_2, \lambda_1 and
\lambda_2 of the gamma prior distributions over \alpha and \lambda.

These are usually chosen to be *non-informative*. By default \alpha_1
= \alpha_2 =  \lambda_1 = \lambda_2 = 10^{-6}.

Bayesian Ridge Regression is used for regression:

   >>> X = [[0., 0.], [1., 1.], [2., 2.], [3., 3.]]
   >>> Y = [0., 1., 2., 3.]
   >>> reg = linear_model.BayesianRidge()
   >>> reg.fit(X, Y)
   BayesianRidge()

After being fitted, the model can then be used to predict new values:

   >>> reg.predict([[1, 0.]])
   array([0.50000013])

The coefficients w of the model can be accessed:

   array([0.49999993, 0.49999993])

Due to the Bayesian framework, the weights found are slightly
different to the ones found by Ordinary Least Squares. However,
Bayesian Ridge Regression is more robust to ill-posed problems.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_bayesian_ridge_curvefit.p
  y*

* Section 3.3 in Christopher M. Bishop: Pattern Recognition and
  Machine Learning, 2006

* David J. C. MacKay, Bayesian Interpolation, 1992.

* Michael E. Tipping, Sparse Bayesian Learning and the Relevance
  Vector Machine, 2001.

1.1.10.2. Automatic Relevance Determination - ARD

The Automatic Relevance Determination (as being implemented in

Vector Machine, 2001.

"ARDRegression") is a kind of linear model which is very similar to
the Bayesian Ridge Regression, but that leads to sparser coefficients
w [1] [2].

"ARDRegression" poses a different prior over w: it drops the spherical
Gaussian distribution for a centered elliptic Gaussian distribution.
This means each coefficient w_{i} can itself be drawn from a Gaussian
distribution, centered on zero and with a precision \lambda_{i}:

   p(w|\lambda) = \mathcal{N}(w|0,A^{-1})

with A being a positive definite diagonal matrix and \text{diag}(A) =
\lambda = \{\lambda_{1},...,\lambda_{p}\}.

In contrast to the Bayesian Ridge Regression, each coordinate of w_{i}
has its own standard deviation \frac{1}{\lambda_i}. The prior over all
\lambda_i is chosen to be the same gamma distribution given by the
hyperparameters \lambda_1 and \lambda_2.

ARD is also known in the literature as *Sparse Bayesian Learning* and
*Relevance Vector Machine* [3] [4]. For a worked-out comparison
between ARD and Bayesian Ridge Regression, see the example below.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_ard.py*

[1] Christopher M. Bishop: Pattern Recognition and Machine Learning,
    Chapter 7.2.1

[2] David Wipf and Srikantan Nagarajan: A New View of Automatic
    Relevance Determination

[3] Michael E. Tipping: Sparse Bayesian Learning and the Relevance
    Vector Machine

[4] Tristan Fletcher: Relevance Vector Machines Explained

1.1.11. Logistic regression

The logistic regression is implemented in "LogisticRegression".
Despite its name, it is implemented as a linear model for
classification rather than regression in terms of the scikit-learn/ML
nomenclature. The logistic regression is also known in the literature
as logit regression, maximum-entropy classification (MaxEnt) or the
log-linear classifier. In this model, the probabilities describing the
possible outcomes of a single trial are modeled using a logistic
function.

This implementation can fit binary, One-vs-Rest, or multinomial
logistic regression with optional \ell_1, \ell_2 or Elastic-Net
regularization.

Note:

  **Regularization**Regularization is applied by default, which is
  common in machine learning but not in statistics. Another advantage
  of regularization is that it improves numerical stability. No
  regularization amounts to setting C to a very high value.

Note:

  **Logistic Regression as a special case of the Generalized Linear
  Models (GLM)**Logistic regression is a special case of Generalized
  Linear Models with a Binomial / Bernoulli conditional distribution

Models (GLM)**Logistic regression is a special case of Generalized
  and a Logit link. The numerical output of the logistic regression,
  which is the predicted probability, can be used as a classifier by
  applying a threshold (by default 0.5) to it. This is how it is
  implemented in scikit-learn, so it expects a categorical target,
  making the Logistic Regression a classifier.

1.1.11.1. Binary Case

For notational ease, we assume that the target y_i takes values in the
set \{0, 1\} for data point i. Once fitted, the "predict_proba()"
method of "LogisticRegression" predicts the probability of the
positive class P(y_i=1|X_i) as

   \hat{p}(X_i) = \operatorname{expit}(X_i w + w_0) = \frac{1}{1 +
   \exp(-X_i w - w_0)}.

As an optimization problem, binary class logistic regression with
regularization term r(w) minimizes the following cost function:

   \min_{w} C \sum_{i=1}^n \left(-y_i \log(\hat{p}(X_i)) - (1 - y_i)
   \log(1 - \hat{p}(X_i))\right) + r(w).

We currently provide four choices for the regularization term  r(w)
via the *penalty* argument:

+------------------+---------------------------------------------------+
| penalty          | r(w)                                              |
|==================|===================================================|
| *None*           | 0                                                 |

| \ell_1           | \|w\|_1                                           |
| \ell_2           | \frac{1}{2}\|w\|_2^2 = \frac{1}{2}w^T w           |
| *ElasticNet*     | \frac{1 - \rho}{2}w^T w + \rho \|w\|_1            |

For ElasticNet, \rho (which corresponds to the *l1_ratio* parameter)
controls the strength of \ell_1 regularization vs. \ell_2
regularization. Elastic-Net is equivalent to \ell_1 when \rho = 1 and
equivalent to \ell_2 when \rho=0.

1.1.11.2. Multinomial Case

The binary case can be extended to K classes leading to the
multinomial logistic regression, see also log-linear model.

Note:

  It is possible to parameterize a K-class classification model using
  only K-1 weight vectors, leaving one class probability fully
  determined by the other class probabilities by leveraging the fact
  that all class probabilities must sum to one. We deliberately choose
  to overparameterize the model using K weight vectors for ease of
  implementation and to preserve the symmetrical inductive bias
  regarding ordering of classes, see [16]. This effect becomes

to overparameterize the model using K weight vectors for ease of
  especially important when using regularization. The choice of
  overparameterization can be detrimental for unpenalized models since
  then the solution may not be unique, as shown in [16].

Let y_i \in {1, \ldots, K} be the label (ordinal) encoded target
variable for observation i. Instead of a single coefficient vector, we
now have a matrix of coefficients W where each row vector W_k
corresponds to class k. We aim at predicting the class probabilities
P(y_i=k|X_i) via "predict_proba()" as:

   \hat{p}_k(X_i) = \frac{\exp(X_i W_k + W_{0, k})}{\sum_{l=0}^{K-1}
   \exp(X_i W_l + W_{0, l})}.

The objective for the optimization becomes

   \min_W -C \sum_{i=1}^n \sum_{k=0}^{K-1} [y_i = k]
   \log(\hat{p}_k(X_i)) + r(W).

Where [P] represents the Iverson bracket which evaluates to 0 if P is
false, otherwise it evaluates to 1. We currently provide four choices
for the regularization term r(W) via the *penalty* argument:

+------------------+------------------------------------------------------------------------------------+
| penalty          | r(W)                                                                               |
|==================|====================================================================================|

| *None*           | 0                                                                                  |
| \ell_1           | \|W\|_{1,1} = \sum_{i=1}^n\sum_{j=1}^{K}|W_{i,j}|                                  |
| \ell_2           | \frac{1}{2}\|W\|_F^2 = \frac{1}{2}\sum_{i=1}^n\sum_{j=1}^{K} W_{i,j}^2             |
| *ElasticNet*     | \frac{1 - \rho}{2}\|W\|_F^2 + \rho \|W\|_{1,1}                                     |

1.1.11.3. Solvers

The solvers implemented in the class "LogisticRegression" are "lbfgs",
"liblinear", "newton-cg", "newton-cholesky", "sag" and "saga":

The solver "liblinear" uses a coordinate descent (CD) algorithm, and
relies on the excellent C++ LIBLINEAR library, which is shipped with
scikit-learn. However, the CD algorithm implemented in liblinear
cannot learn a true multinomial (multiclass) model; instead, the
optimization problem is decomposed in a "one-vs-rest" fashion so

separate binary classifiers are trained for all classes. This happens
under the hood, so "LogisticRegression" instances using this solver
behave as multiclass classifiers. For \ell_1 regularization
"sklearn.svm.l1_min_c()" allows to calculate the lower bound for C in
order to get a non "null" (all feature weights to zero) model.

The "lbfgs", "newton-cg" and "sag" solvers only support \ell_2
regularization or no regularization, and are found to converge faster
for some high-dimensional data. Setting *multi_class* to "multinomial"
with these solvers learns a true multinomial logistic regression model
[5], which means that its probability estimates should be better
calibrated than the default "one-vs-rest" setting.

The "sag" solver uses Stochastic Average Gradient descent [6]. It is
faster than other solvers for large datasets, when both the number of
samples and the number of features are large.

The "saga" solver [7] is a variant of "sag" that also supports the
non-smooth *penalty="l1"*. This is therefore the solver of choice for
sparse multinomial logistic regression. It is also the only solver
that supports *penalty="elasticnet"*.

The "lbfgs" is an optimization algorithm that approximates the
Broyden–Fletcher–Goldfarb–Shanno algorithm [8], which belongs to

quasi-Newton methods. As such, it can deal with a wide range of
different training data and is therefore the default solver. Its
performance, however, suffers on poorly scaled datasets and on
datasets with one-hot encoded categorical features with rare
categories.

The "newton-cholesky" solver is an exact Newton solver that calculates
the hessian matrix and solves the resulting linear system. It is a
very good choice for *n_samples* >> *n_features*, but has a few
shortcomings: Only \ell_2 regularization is supported. Furthermore,
because the hessian matrix is explicitly computed, the memory usage
has a quadratic dependency on *n_features* as well as on *n_classes*.
As a consequence, only the one-vs-rest scheme is implemented for the
multiclass case.

For a comparison of some of these solvers, see [9].

The following table summarizes the penalties supported by each solver:

+--------------------------------+---------------+----------+---------------+-------------------+-------------------------+-------------+--------------+
|                                | **Solvers**                                                                                                         |

| **Penalties**                  | **'lbfgs'**   | **'liblinear'**          | **'newton-cg'**   | **'newton-cholesky'**   | **'sag'**   | **'saga'**   |
| Multinomial + L2 penalty       | yes           | no                       | yes               | no                      | yes         | yes          |
| OVR + L2 penalty               | yes           | yes                      | yes               | yes                     | yes         | yes          |
| Multinomial + L1 penalty       | no            | no                       | no                | no                      | no          | yes          |

| OVR + L1 penalty               | no            | yes                      | no                | no                      | no          | yes          |
| Elastic-Net                    | no            | no                       | no                | no                      | no          | yes          |
| No penalty ('none')            | yes           | no                       | yes               | yes                     | yes         | yes          |
| **Behaviors**                  |                                                                                                                     |

| Penalize the intercept (bad)   | no            | yes                      | no                | no                      | no          | no           |
| Faster for large datasets      | no            | no                       | no                | no                      | yes         | yes          |
| Robust to unscaled datasets    | yes           | yes                      | yes               | yes                     | no          | no           |

The "lbfgs" solver is used by default for its robustness. For large
datasets the "saga" solver is usually faster. For large dataset, you
may also consider using "SGDClassifier" with *loss="log_loss"*, which
might be even faster but requires more tuning.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_logistic_l1_l2_sparsity.p
  y*

* *sphx_glr_auto_examples_linear_model_plot_logistic_path.py*

Examples:
^^^^^^^^^

  y*

* *sphx_glr_auto_examples_linear_model_plot_logistic_multinomial.py*

* *sphx_glr_auto_examples_linear_model_plot_sparse_logistic_regressio
  n_20newsgroups.py*

  n_mnist.py*

Differences from liblinear:

There might be a difference in the scores obtained between
"LogisticRegression" with "solver=liblinear" or "LinearSVC" and the
external liblinear library directly, when "fit_intercept=False" and
the fit "coef_" (or) the data to be predicted are zeroes. This is
because for the sample(s) with "decision_function" zero,
"LogisticRegression" and "LinearSVC" predict the negative class, while
liblinear predicts the positive class. Note that a model with
"fit_intercept=False" and having many samples with "decision_function"
zero, is likely to be a underfit, bad model and you are advised to set
"fit_intercept=True" and increase the intercept_scaling.

Note:

  **Feature selection with sparse logistic regression**A logistic
  regression with \ell_1 penalty yields sparse models, and can thus be
  used to perform feature selection, as detailed in L1-based feature
  selection.

Note:

  **P-value estimation**It is possible to obtain the p-values and
  confidence intervals for coefficients in cases of regression without

selection.

Note:

  penalization. The statsmodels package natively supports this. Within
  sklearn, one could use bootstrapping instead as well.

"LogisticRegressionCV" implements Logistic Regression with built-in
cross-validation support, to find the optimal *C* and *l1_ratio*
parameters according to the "scoring" attribute. The "newton-cg",
"sag", "saga" and "lbfgs" solvers are found to be faster for high-
dimensional dense data, due to warm-starting (see *Glossary*).

[5] Christopher M. Bishop: Pattern Recognition and Machine Learning,
    Chapter 4.3.4

[6] Mark Schmidt, Nicolas Le Roux, and Francis Bach: Minimizing Finite
    Sums with the Stochastic Average Gradient.

[7] Aaron Defazio, Francis Bach, Simon Lacoste-Julien: >>:arxiv:`SAGA:
    A Fast Incremental Gradient Method With Support for Non-Strongly
    Convex Composite Objectives. <1407.0202>`<<

[8] https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Go
    ldfarb%E2%80%93Shanno_algorithm

[9] Thomas P. Minka "A comparison of numerical optimizers for logistic
    regression"

[16] >>:arxiv:`Simon, Noah, J. Friedman and T. Hastie. "A Blockwise
     Descent Algorithm for Group-penalized Multiresponse and
     Multinomial Regression." <1311.6529>`<<

1.1.12. Generalized Linear Models

Descent Algorithm for Group-penalized Multiresponse and

Generalized Linear Models (GLM) extend linear models in two ways [10].
First, the predicted values \hat{y} are linked to a linear combination
of the input variables X via an inverse link function h as

   \hat{y}(w, X) = h(Xw).

Secondly, the squared loss function is replaced by the unit deviance d
of a distribution in the exponential family (or more precisely, a
reproductive exponential dispersion model (EDM) [11]).

The minimization problem becomes:

   \min_{w} \frac{1}{2 n_{\text{samples}}} \sum_i d(y_i, \hat{y}_i) +
   \frac{\alpha}{2} ||w||_2^2,

where \alpha is the L2 regularization penalty. When sample weights are
provided, the average becomes a weighted average.

The following table lists some specific EDMs and their unit deviance :

+-------------------+----------------------------------+-----------------------------------------------------------------------------------------------+
| Distribution      | Target Domain                    | Unit Deviance d(y, \hat{y})                                                                   |
|===================|==================================|===============================================================================================|

| Normal            | y \in (-\infty, \infty)          | (y-\hat{y})^2                                                                                 |
| Bernoulli         | y \in \{0, 1\}                   | 2({y}\log\frac{y}{\hat{y}}+({1}-{y})\log\frac{{1}-{y}}{{1}-\hat{y}})                          |
| Categorical       | y \in \{0, 1, ..., k\}           | 2\sum_{i \in \{0, 1, ..., k\}} I(y = i) y_\text{i}\log\frac{I(y = i)}{\hat{I(y = i)}}         |
| Poisson           | y \in [0, \infty)                | 2(y\log\frac{y}{\hat{y}}-y+\hat{y})                                                           |

| Gamma             | y \in (0, \infty)                | 2(\log\frac{y}{\hat{y}}+\frac{y}{\hat{y}}-1)                                                  |
| Inverse Gaussian  | y \in (0, \infty)                | \frac{(y-\hat{y})^2}{y\hat{y}^2}                                                              |

The Probability Density Functions (PDF) of these distributions are
illustrated in the following figure,

   [image]PDF of a random variable Y following Poisson, Tweedie
   (power=1.5) and Gamma distributions with different mean values
   (\mu). Observe the point mass at Y=0 for the Poisson distribution
   and the Tweedie (power=1.5) distribution, but not for the Gamma
   distribution which has a strictly positive target domain.

The Bernoulli distribution is a discrete probability distribution
modelling a Bernoulli trial - an event that has only two mutually
exclusive outcomes. The Categorical distribution is a generalization
of the Bernoulli distribution for a categorical random variable. While

a random variable in a Bernoulli distribution has two possible
outcomes, a Categorical random variable can take on one of K possible
categories, with the probability of each category specified
separately.

The choice of the distribution depends on the problem at hand:

* If the target values y are counts (non-negative integer valued) or
  relative frequencies (non-negative), you might use a Poisson
  distribution with a log-link.

* If the target values are positive valued and skewed, you might try a
  Gamma distribution with a log-link.

* If the target values seem to be heavier tailed than a Gamma
  distribution, you might try an Inverse Gaussian distribution (or
  even higher variance powers of the Tweedie family).

* If the target values y are probabilities, you can use the Bernoulli
  distribution. The Bernoulli distribution with a logit link can be
  used for binary classification. The Categorical distribution with a
  softmax link can be used for multiclass classification.

Examples of use cases include:

* Agriculture / weather modeling:  number of rain events per year
  (Poisson), amount of rainfall per event (Gamma), total rainfall per
  year (Tweedie / Compound Poisson Gamma).

* Risk modeling / insurance policy pricing:  number of claim events /
  policyholder per year (Poisson), cost per event (Gamma), total cost

year (Tweedie / Compound Poisson Gamma).

  per policyholder per year (Tweedie / Compound Poisson Gamma).

* Credit Default: probability that a loan can't be payed back
  (Bernouli).

* Fraud Detection: probability that a financial transaction like a
  cash transfer is a fraudulent transaction (Bernoulli).

* Predictive maintenance: number of production interruption events per
  year (Poisson), duration of interruption (Gamma), total interruption
  time per year (Tweedie / Compound Poisson Gamma).

* Medical Drug Testing: probability of curing a patient in a set of
  trials or probability that a patient will experience side effects
  (Bernoulli).

* News Classification: classification of news articles into three
  categories namely Business News, Politics and Entertainment news
  (Categorical).

[10] McCullagh, Peter; Nelder, John (1989). Generalized Linear Models,
     Second Edition. Boca Raton: Chapman and Hall/CRC. ISBN
     0-412-31760-5.

[11] Jørgensen, B. (1992). The theory of exponential dispersion models
     and analysis of deviance. Monografias de matemática, no. 51.  See
     also Exponential dispersion model.

1.1.12.1. Usage

"TweedieRegressor" implements a generalized linear model for the
Tweedie distribution, that allows to model any of the above mentioned

distributions using the appropriate "power" parameter. In particular:

* "power = 0": Normal distribution. Specific estimators such as
  "Ridge", "ElasticNet" are generally more appropriate in this case.

* "power = 1": Poisson distribution. "PoissonRegressor" is exposed for
  convenience. However, it is strictly equivalent to
  *TweedieRegressor(power=1, link='log')*.

* "power = 2": Gamma distribution. "GammaRegressor" is exposed for
  *TweedieRegressor(power=2, link='log')*.

* "power = 3": Inverse Gaussian distribution.

The link function is determined by the *link* parameter.

Usage example:

   >>> from sklearn.linear_model import TweedieRegressor
   >>> reg = TweedieRegressor(power=1, alpha=0.5, link='log')
   >>> reg.fit([[0, 0], [0, 1], [2, 2]], [0, 1, 2])
   TweedieRegressor(alpha=0.5, link='log', power=1)
   array([0.2463..., 0.4337...])
   -0.7638...

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_poisson_regression_non_no
  rmal_loss.py*

* *sphx_glr_auto_examples_linear_model_plot_tweedie_regression_insura
  nce_claims.py*

1.1.12.2. Practical considerations
----------------------------------

The feature matrix *X* should be standardized before fitting. This
ensures that the penalty treats features equally.

nce_claims.py*

Since the linear predictor Xw can be negative and Poisson, Gamma and
Inverse Gaussian distributions don't support negative values, it is
necessary to apply an inverse link function that guarantees the non-
negativeness. For example with *link='log'*, the inverse link function
becomes h(Xw)=\exp(Xw).

If you want to model a relative frequency, i.e. counts per exposure
(time, volume, ...) you can do so by using a Poisson distribution and
passing y=\frac{\mathrm{counts}}{\mathrm{exposure}} as target values
together with \mathrm{exposure} as sample weights. For a concrete
example see e.g. *sphx_glr_auto_examples_linear_model_plot_tweedie_re
gression_insurance_claims.py*.

When performing cross-validation for the *power* parameter of
*TweedieRegressor*, it is advisable to specify an explicit *scoring*
function, because the default scorer "TweedieRegressor.score()" is a
function of *power* itself.

1.1.13. Stochastic Gradient Descent - SGD

Stochastic gradient descent is a simple yet very efficient approach to
fit linear models. It is particularly useful when the number of
samples (and the number of features) is very large. The "partial_fit"
method allows online/out-of-core learning.

The classes "SGDClassifier" and "SGDRegressor" provide functionality

to fit linear models for classification and regression using different
(convex) loss functions and different penalties. E.g., with
"loss="log"", "SGDClassifier" fits a logistic regression model, while
with "loss="hinge"" it fits a linear support vector machine (SVM).

* Stochastic Gradient Descent

1.1.14. Perceptron

The "Perceptron" is another simple classification algorithm suitable
for large scale learning. By default:

   * It does not require a learning rate.

   * It is not regularized (penalized).

   * It updates its model only on mistakes.

The last characteristic implies that the Perceptron is slightly faster
to train than SGD with the hinge loss and that the resulting models
are sparser.

1.1.15. Passive Aggressive Algorithms

The passive-aggressive algorithms are a family of algorithms for
large-scale learning. They are similar to the Perceptron in that they
do not require a learning rate. However, contrary to the Perceptron,
they include a regularization parameter "C".

For classification, "PassiveAggressiveClassifier" can be used with
"loss='hinge'" (PA-I) or "loss='squared_hinge'" (PA-II).  For
regression, "PassiveAggressiveRegressor" can be used with
"loss='epsilon_insensitive'" (PA-I) or
"loss='squared_epsilon_insensitive'" (PA-II).

* "Online Passive-Aggressive Algorithms" K. Crammer, O. Dekel, J.
  Keshat, S. Shalev-Shwartz, Y. Singer - JMLR 7 (2006)

1.1.16. Robustness regression: outliers and modeling errors
===========================================================

Robust regression aims to fit a regression model in the presence of
corrupt data: either outliers, or error in the model.

   [image]

1.1.16.1. Different scenario and useful concepts
------------------------------------------------

There are different things to keep in mind when dealing with data
corrupted by outliers:

* **Outliers in X or in y**?

  +--------------------------------------+--------------------------------------+
  | Outliers in the y direction          | Outliers in the X direction          |
  |======================================|======================================|
  | [image: y_outliers][image]           | [image: X_outliers][image]           |

* **Fraction of outliers versus amplitude of error**

  The number of outlying points matters, but also how much they are
  outliers.

  | Small outliers                       | Large outliers                       |

outliers.

  | [image: y_outliers][image]           | [image: large_y_outliers][image]     |

An important notion of robust fitting is that of breakdown point: the
fraction of data that can be outlying for the fit to start missing the
inlying data.

Note that in general, robust fitting in high-dimensional setting
(large *n_features*) is very hard. The robust models here will
probably not work in these settings.

**Trade-offs: which estimator?**

   Scikit-learn provides 3 robust regression estimators: RANSAC, Theil
   Sen and HuberRegressor.

   * HuberRegressor should be faster than RANSAC and Theil Sen unless
     the number of samples are very large, i.e "n_samples" >>
     "n_features". This is because RANSAC and Theil Sen fit on smaller
     subsets of the data. However, both Theil Sen and RANSAC are
     unlikely to be as robust as HuberRegressor for the default
     parameters.

   * RANSAC is faster than Theil Sen and scales much better with the
     number of samples.

   * RANSAC will deal better with large outliers in the y direction
     (most common situation).

   * Theil Sen will cope better with medium-size outliers in the X

number of samples.

     direction, but this property will disappear in high-dimensional
     settings.

When in doubt, use RANSAC.

1.1.16.2. RANSAC: RANdom SAmple Consensus

RANSAC (RANdom SAmple Consensus) fits a model from random subsets of
inliers from the complete data set.

RANSAC is a non-deterministic algorithm producing only a reasonable
result with a certain probability, which is dependent on the number of
iterations (see *max_trials* parameter). It is typically used for
linear and non-linear regression problems and is especially popular in
the field of photogrammetric computer vision.

The algorithm splits the complete input sample data into a set of
inliers, which may be subject to noise, and outliers, which are e.g.
caused by erroneous measurements or invalid hypotheses about the data.
The resulting model is then estimated only from the determined
inliers.

   [image]

1.1.16.2.1. Details of the algorithm

Each iteration performs the following steps:

1. Select "min_samples" random samples from the original data and
   check whether the set of data is valid (see "is_data_valid").

2. Fit a model to the random subset ("base_estimator.fit") and check
   whether the estimated model is valid (see "is_model_valid").

check whether the set of data is valid (see "is_data_valid").

3. Classify all data as inliers or outliers by calculating the
   residuals to the estimated model ("base_estimator.predict(X) - y")
   - all data samples with absolute residuals smaller than or equal to
   the "residual_threshold" are considered as inliers.

4. Save fitted model as best model if number of inlier samples is
   maximal. In case the current estimated model has the same number of
   inliers, it is only considered as the best model if it has better
   score.

These steps are performed either a maximum number of times
("max_trials") or until one of the special stop criteria are met (see
"stop_n_inliers" and "stop_score"). The final model is estimated using
all inlier samples (consensus set) of the previously determined best
model.

The "is_data_valid" and "is_model_valid" functions allow to identify
and reject degenerate combinations of random sub-samples. If the
estimated model is not needed for identifying degenerate cases,
"is_data_valid" should be used as it is called prior to fitting the
model and thus leading to better computational performance.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_ransac.py*

* *sphx_glr_auto_examples_linear_model_plot_robust_fit.py*

* https://en.wikipedia.org/wiki/RANSAC

Examples:
^^^^^^^^^

* "Random Sample Consensus: A Paradigm for Model Fitting with
  Applications to Image Analysis and Automated Cartography" Martin A.
  Fischler and Robert C. Bolles - SRI International (1981)

* "Performance Evaluation of RANSAC Family" Sunglok Choi, Taemin Kim
  and Wonpil Yu - BMVC (2009)

1.1.16.3. Theil-Sen estimator: generalized-median-based estimator

The "TheilSenRegressor" estimator uses a generalization of the median
in multiple dimensions. It is thus robust to multivariate outliers.
Note however that the robustness of the estimator decreases quickly
with the dimensionality of the problem. It loses its robustness
properties and becomes no better than an ordinary least squares in
high dimension.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_theilsen.py*

* https://en.wikipedia.org/wiki/Theil%E2%80%93Sen_estimator

1.1.16.3.1. Theoretical considerations

"TheilSenRegressor" is comparable to the Ordinary Least Squares (OLS)
in terms of asymptotic efficiency and as an unbiased estimator. In
contrast to OLS, Theil-Sen is a non-parametric method which means it

makes no assumption about the underlying distribution of the data.
Since Theil-Sen is a median-based estimator, it is more robust against
corrupted data aka outliers. In univariate setting, Theil-Sen has a
breakdown point of about 29.3% in case of a simple linear regression
which means that it can tolerate arbitrary corrupted data of up to
29.3%.

   [image]

The implementation of "TheilSenRegressor" in scikit-learn follows a
generalization to a multivariate linear regression model [14] using
the spatial median which is a generalization of the median to multiple
dimensions [15].

In terms of time and space complexity, Theil-Sen scales according to

   \binom{n_{\text{samples}}}{n_{\text{subsamples}}}

which makes it infeasible to be applied exhaustively to problems with
a large number of samples and features. Therefore, the magnitude of a
subpopulation can be chosen to limit the time and space complexity by
considering only a random subset of all possible combinations.

Examples:
^^^^^^^^^

[14] Xin Dang, Hanxiang Peng, Xueqin Wang and Heping Zhang: Theil-Sen
     Estimators in a Multiple Linear Regression Model.

[15] 20. Kärkkäinen and S. Äyrämö: On Computation of Spatial Median
         for Robust Data Mining.

1.1.16.4. Huber Regression

The "HuberRegressor" is different to "Ridge" because it applies a
linear loss to samples that are classified as outliers. A sample is
classified as an inlier if the absolute error of that sample is lesser
than a certain threshold. It differs from "TheilSenRegressor" and
"RANSACRegressor" because it does not ignore the effect of the
outliers but gives a lesser weight to them.

   [image]

The loss function that "HuberRegressor" minimizes is given by

   \min_{w, \sigma} {\sum_{i=1}^n\left(\sigma +
   H_{\epsilon}\left(\frac{X_{i}w - y_{i}}{\sigma}\right)\sigma\right)
   + \alpha {||w||_2}^2}

where

   H_{\epsilon}(z) = \begin{cases}        z^2, & \text {if } |z| <
   \epsilon, \\        2\epsilon|z| - \epsilon^2, & \text{otherwise}
   \end{cases}

It is advised to set the parameter "epsilon" to 1.35 to achieve 95%
statistical efficiency.

1.1.16.5. Notes

The "HuberRegressor" differs from using "SGDRegressor" with loss set
to *huber* in the following ways.

* "HuberRegressor" is scaling invariant. Once "epsilon" is set,
  scaling "X" and "y" down or up by different values would produce the
  same robustness to outliers as before. as compared to "SGDRegressor"
  where "epsilon" has to be set again when "X" and "y" are scaled.

* "HuberRegressor" should be more efficient to use on data with small

where "epsilon" has to be set again when "X" and "y" are scaled.

  number of samples while "SGDRegressor" needs a number of passes on
  the training data to produce the same robustness.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_huber_vs_ridge.py*

* Peter J. Huber, Elvezio M. Ronchetti: Robust Statistics, Concomitant
  scale estimates, pg 172

Note that this estimator is different from the R implementation of
Robust Regression (https://stats.oarc.ucla.edu/r/dae/robust-
regression/) because the R implementation does a weighted least
squares implementation with weights given to each sample on the basis
of how much the residual is greater than a certain threshold.

1.1.17. Quantile Regression

Quantile regression estimates the median or other quantiles of y
conditional on X, while ordinary least squares (OLS) estimates the
conditional mean.

As a linear model, the "QuantileRegressor" gives linear predictions
\hat{y}(w, X) = Xw for the q-th quantile, q \in (0, 1). The weights or
coefficients w are then found by the following minimization problem:

   \min_{w} {\frac{1}{n_{\text{samples}}} \sum_i PB_q(y_i - X_i w) +
   \alpha ||w||_1}.

This consists of the pinball loss (also known as linear loss), see
also "mean_pinball_loss",

   PB_q(t) = q \max(t, 0) + (1 - q) \max(-t, 0) = \begin{cases}     q
   t, & t > 0, \\     0,    & t = 0, \\     (q-1) t, & t < 0

and the L1 penalty controlled by parameter "alpha", similar to
"Lasso".

As the pinball loss is only linear in the residuals, quantile
regression is much more robust to outliers than squared error based
estimation of the mean. Somewhat in between is the "HuberRegressor".

Quantile regression may be useful if one is interested in predicting
an interval instead of point prediction. Sometimes, prediction
intervals are calculated based on the assumption that prediction error
is distributed normally with zero mean and constant variance. Quantile
regression provides sensible prediction intervals even for errors with
non-constant (but predictable) variance or non-normal distribution.

   [image]

Based on minimizing the pinball loss, conditional quantiles can also
be estimated by models other than linear models. For example,
"GradientBoostingRegressor" can predict conditional quantiles if its
parameter "loss" is set to ""quantile"" and parameter "alpha" is set
to the quantile that should be predicted. See the example in
*sphx_glr_auto_examples_ensemble_plot_gradient_boosting_quantile.py*.

Most implementations of quantile regression are based on linear
programming problem. The current implementation is based on
"scipy.optimize.linprog()".

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_quantile_regression.py*

Examples:
^^^^^^^^^

* Koenker, R., & Bassett Jr, G. (1978). Regression quantiles.
  Econometrica: journal of the Econometric Society, 33-50.

* Portnoy, S., & Koenker, R. (1997). >>:doi:`The Gaussian hare and the
  Laplacian tortoise: computability of squared-error versus absolute-
  error estimators. Statistical Science, 12, 279-300
  <10.1214/ss/1030037960>`<<.

* Koenker, R. (2005). >>:doi:`Quantile Regression
  <10.1017/CBO9780511754098>`<<. Cambridge University Press.

1.1.18. Polynomial regression: extending linear models with basis functions
===========================================================================

One common pattern within machine learning is to use linear models
trained on nonlinear functions of the data.  This approach maintains
the generally fast performance of linear methods, while allowing them
to fit a much wider range of data.

For example, a simple linear regression can be extended by
constructing **polynomial features** from the coefficients.  In the
standard linear regression case, you might have a model that looks
like this for two-dimensional data:

   \hat{y}(w, x) = w_0 + w_1 x_1 + w_2 x_2

If we want to fit a paraboloid to the data instead of a plane, we can
combine the features in second-order polynomials, so that the model
looks like this:

\hat{y}(w, x) = w_0 + w_1 x_1 + w_2 x_2

   \hat{y}(w, x) = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1 x_2 + w_4 x_1^2 +
   w_5 x_2^2

The (sometimes surprising) observation is that this is *still a linear
model*: to see this, imagine creating a new set of features

   z = [x_1, x_2, x_1 x_2, x_1^2, x_2^2]

With this re-labeling of the data, our problem can be written

   \hat{y}(w, z) = w_0 + w_1 z_1 + w_2 z_2 + w_3 z_3 + w_4 z_4 + w_5
   z_5

We see that the resulting *polynomial regression* is in the same class
of linear models we considered above (i.e. the model is linear in w)
and can be solved by the same techniques.  By considering linear fits
within a higher-dimensional space built with these basis functions,
the model has the flexibility to fit a much broader range of data.

Here is an example of applying this idea to one-dimensional data,
using polynomial features of varying degrees:

   [image]

This figure is created using the "PolynomialFeatures" transformer,
which transforms an input data matrix into a new data matrix of a
given degree. It can be used as follows:

   >>> from sklearn.preprocessing import PolynomialFeatures
   >>> X = np.arange(6).reshape(3, 2)
   >>> X
   array([[0, 1],
          [2, 3],
          [4, 5]])
   >>> poly = PolynomialFeatures(degree=2)
   >>> poly.fit_transform(X)

   >>> X
          [2, 3],
          [4, 5]])
   array([[ 1.,  0.,  1.,  0.,  0.,  1.],
          [ 1.,  2.,  3.,  4.,  6.,  9.],
          [ 1.,  4.,  5., 16., 20., 25.]])

The features of "X" have been transformed from [x_1, x_2] to [1, x_1,
x_2, x_1^2, x_1 x_2, x_2^2], and can now be used within any linear
model.

This sort of preprocessing can be streamlined with the Pipeline tools.
A single object representing a simple polynomial regression can be
created and used as follows:

   >>> model = Pipeline([('poly', PolynomialFeatures(degree=3)),
   ...                   ('linear', LinearRegression(fit_intercept=False))])
   >>> # fit to an order-3 polynomial data
   >>> x = np.arange(5)
   >>> y = 3 - 2 * x + x ** 2 - x ** 3
   >>> model = model.fit(x[:, np.newaxis], y)
   >>> model.named_steps['linear'].coef_
   array([ 3., -2.,  1., -1.])

The linear model trained on polynomial features is able to exactly
recover the input polynomial coefficients.

In some cases it's not necessary to include higher powers of any
single feature, but only the so-called *interaction features* that
multiply together at most d distinct features. These can be gotten

from "PolynomialFeatures" with the setting "interaction_only=True".

For example, when dealing with boolean features, x_i^n = x_i for all n
and is therefore useless; but x_i x_j represents the conjunction of
two booleans. This way, we can solve the XOR problem with a linear
classifier:

   >>> from sklearn.linear_model import Perceptron
   >>> X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
   >>> y = X[:, 0] ^ X[:, 1]
   >>> y
   array([0, 1, 1, 0])
   >>> X = PolynomialFeatures(interaction_only=True).fit_transform(X).astype(int)
   >>> X
   array([[1, 0, 0, 0],
          [1, 0, 1, 0],
          [1, 1, 0, 0],
          [1, 1, 1, 1]])
   >>> clf = Perceptron(fit_intercept=False, max_iter=10, tol=None,
   ...                  shuffle=False).fit(X, y)

And the classifier "predictions" are perfect:

   >>> clf.predict(X)
   >>> clf.score(X, y)
   1.0

1.17. Neural network models (supervised)
****************************************

Warning:

  This implementation is not intended for large-scale applications. In
  particular, scikit-learn offers no GPU support. For much faster,
  GPU-based implementations, as well as frameworks offering much more
  flexibility to build deep learning architectures, see  Related
  Projects.

1.17.1. Multi-layer Perceptron

**Multi-layer Perceptron (MLP)** is a supervised learning algorithm
that learns a function f(\cdot): R^m \rightarrow R^o by training on a
dataset, where m is the number of dimensions for input and o is the
number of dimensions for output. Given a set of features X = {x_1,
x_2, ..., x_m} and a target y, it can learn a non-linear function
approximator for either classification or regression. It is different
from logistic regression, in that between the input and the output
layer, there can be one or more non-linear layers, called hidden
layers. Figure 1 shows a one hidden layer MLP with scalar output.

   [image]**Figure 1 : One hidden layer MLP.**

The leftmost layer, known as the input layer, consists of a set of
neurons \{x_i | x_1, x_2, ..., x_m\} representing the input features.
Each neuron in the hidden layer transforms the values from the
previous layer with a weighted linear summation w_1x_1 + w_2x_2 + ...
+ w_mx_m, followed by a non-linear activation function g(\cdot):R
\rightarrow R - like the hyperbolic tan function. The output layer

receives the values from the last hidden layer and transforms them
into output values.

The module contains the public attributes "coefs_" and "intercepts_".
"coefs_" is a list of weight matrices, where weight matrix at index i
represents the weights between layer i and layer i+1. "intercepts_" is
a list of bias vectors, where the vector at index i represents the
bias values added to layer i+1.

The advantages of Multi-layer Perceptron are:

   * Capability to learn non-linear models.

   * Capability to learn models in real-time (on-line learning) using
     "partial_fit".

The disadvantages of Multi-layer Perceptron (MLP) include:

   * MLP with hidden layers have a non-convex loss function where
     there exists more than one local minimum. Therefore different
     random weight initializations can lead to different validation
     accuracy.

   * MLP requires tuning a number of hyperparameters such as the
     number of hidden neurons, layers, and iterations.

   * MLP is sensitive to feature scaling.

Please see Tips on Practical Use section that addresses some of these
disadvantages.

1.17.2. Classification

Class "MLPClassifier" implements a multi-layer perceptron (MLP)
algorithm that trains using Backpropagation.

MLP trains on two arrays: array X of size (n_samples, n_features),

which holds the training samples represented as floating point feature
vectors; and array y of size (n_samples,), which holds the target
values (class labels) for the training samples:

   >>> from sklearn.neural_network import MLPClassifier
   >>> X = [[0., 0.], [1., 1.]]
   >>> y = [0, 1]
   >>> clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
   ...                     hidden_layer_sizes=(5, 2), random_state=1)
   ...
   MLPClassifier(alpha=1e-05, hidden_layer_sizes=(5, 2), random_state=1,
                 solver='lbfgs')

After fitting (training), the model can predict labels for new
samples:

   >>> clf.predict([[2., 2.], [-1., -2.]])
   array([1, 0])

MLP can fit a non-linear model to the training data. "clf.coefs_"
contains the weight matrices that constitute the model parameters:

   >>> [coef.shape for coef in clf.coefs_]
   [(2, 5), (5, 2), (2, 1)]

Currently, "MLPClassifier" supports only the Cross-Entropy loss
function, which allows probability estimates by running the
"predict_proba" method.

MLP trains using Backpropagation. More precisely, it trains using some
form of gradient descent and the gradients are calculated using
Backpropagation. For classification, it minimizes the Cross-Entropy
loss function, giving a vector of probability estimates P(y|x) per
sample x:

sample x:

   >>> clf.predict_proba([[2., 2.], [1., 2.]])
   array([[1.967...e-04, 9.998...-01],
          [1.967...e-04, 9.998...-01]])

"MLPClassifier" supports multi-class classification by applying
Softmax as the output function.

Further, the model supports multi-label classification in which a
sample can belong to more than one class. For each class, the raw
output passes through the logistic function. Values larger or equal to
*0.5* are rounded to *1*, otherwise to *0*. For a predicted output of
a sample, the indices where the value is *1* represents the assigned
classes of that sample:

   >>> y = [[0, 1], [1, 1]]
   ...                     hidden_layer_sizes=(15,), random_state=1)
   ...
   MLPClassifier(alpha=1e-05, hidden_layer_sizes=(15,), random_state=1,
   >>> clf.predict([[1., 2.]])
   array([[1, 1]])
   >>> clf.predict([[0., 0.]])
   array([[0, 1]])

See the examples below and the docstring of "MLPClassifier.fit()" for
further information.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neural_networks_plot_mlp_training_curves.py*

* *sphx_glr_auto_examples_neural_networks_plot_mnist_filters.py*

1.17.3. Regression

Class "MLPRegressor" implements a multi-layer perceptron (MLP) that

trains using backpropagation with no activation function in the output
layer, which can also be seen as using the identity function as
activation function. Therefore, it uses the square error as the loss
function, and the output is a set of continuous values.

"MLPRegressor" also supports multi-output regression, in which a
sample can have more than one target.

1.17.4. Regularization

Both "MLPRegressor" and "MLPClassifier" use parameter "alpha" for
regularization (L2 regularization) term which helps in avoiding
overfitting by penalizing weights with large magnitudes. Following
plot displays varying decision function with value of alpha.

   [image]

See the examples below for further information.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neural_networks_plot_mlp_alpha.py*

1.17.5. Algorithms

MLP trains using Stochastic Gradient Descent, >>:arxiv:`Adam
<1412.6980>`<<, or L-BFGS. Stochastic Gradient Descent (SGD) updates
parameters using the gradient of the loss function with respect to a
parameter that needs adaptation, i.e.

   w \leftarrow w - \eta (\alpha \frac{\partial R(w)}{\partial w} +
   \frac{\partial Loss}{\partial w})

where \eta is the learning rate which controls the step-size in the
parameter space search.  Loss is the loss function used for the
network.

\frac{\partial Loss}{\partial w})

network.

More details can be found in the documentation of SGD

Adam is similar to SGD in a sense that it is a stochastic optimizer,
but it can automatically adjust the amount to update parameters based
on adaptive estimates of lower-order moments.

With SGD or Adam, training supports online and mini-batch learning.

L-BFGS is a solver that approximates the Hessian matrix which
represents the second-order partial derivative of a function. Further
it approximates the inverse of the Hessian matrix to perform parameter
updates. The implementation uses the Scipy version of L-BFGS.

If the selected solver is 'L-BFGS', training does not support online
nor mini-batch learning.

1.17.6. Complexity

Suppose there are n training samples, m features, k hidden layers,
each containing h neurons - for simplicity, and o output neurons.  The
time complexity of backpropagation is O(n\cdot m \cdot h^k \cdot o
\cdot i), where i is the number of iterations. Since backpropagation
has a high time complexity, it is advisable to start with smaller
number of hidden neurons and few hidden layers for training.

1.17.7. Mathematical formulation

Given a set of training examples (x_1, y_1), (x_2, y_2), \ldots, (x_n,
y_n) where x_i \in \mathbf{R}^n and y_i \in \{0, 1\}, a one hidden

layer one hidden neuron MLP learns the function f(x) = W_2 g(W_1^T x +
b_1) + b_2 where W_1 \in \mathbf{R}^m and W_2, b_1, b_2 \in \mathbf{R}
are model parameters. W_1, W_2 represent the weights of the input
layer and hidden layer, respectively; and b_1, b_2 represent the bias
added to the hidden layer and the output layer, respectively. g(\cdot)
: R \rightarrow R is the activation function, set by default as the
hyperbolic tan. It is given as,

   g(z)= \frac{e^z-e^{-z}}{e^z+e^{-z}}

For binary classification, f(x) passes through the logistic function
g(z)=1/(1+e^{-z}) to obtain output values between zero and one. A
threshold, set to 0.5, would assign samples of outputs larger or equal
0.5 to the positive class, and the rest to the negative class.

If there are more than two classes, f(x) itself would be a vector of
size (n_classes,). Instead of passing through logistic function, it
passes through the softmax function, which is written as,

   \text{softmax}(z)_i = \frac{\exp(z_i)}{\sum_{l=1}^k\exp(z_l)}

where z_i represents the i th element of the input to softmax, which
corresponds to class i, and K is the number of classes. The result is
a vector containing the probabilities that sample x belong to each
class. The output is the class with the highest probability.

In regression, the output remains as f(x); therefore, output
activation function is just the identity function.

MLP uses different loss functions depending on the problem type. The
loss function for classification is Average Cross-Entropy, which in
binary case is given as,

   Loss(\hat{y},y,W) = -\dfrac{1}{n}\sum_{i=0}^n(y_i \ln {\hat{y_i}} +
   (1-y_i) \ln{(1-\hat{y_i})}) + \dfrac{\alpha}{2n} ||W||_2^2

where \alpha ||W||_2^2 is an L2-regularization term (aka penalty) that
penalizes complex models; and \alpha > 0 is a non-negative
hyperparameter that controls the magnitude of the penalty.

For regression, MLP uses the Mean Square Error loss function; written
as,

   Loss(\hat{y},y,W) = \frac{1}{2n}\sum_{i=0}^n||\hat{y}_i - y_i
   ||_2^2 + \frac{\alpha}{2n} ||W||_2^2

Starting from initial random weights, multi-layer perceptron (MLP)
minimizes the loss function by repeatedly updating these weights.
After computing the loss, a backward pass propagates it from the
output layer to the previous layers, providing each weight parameter
with an update value meant to decrease the loss.

In gradient descent, the gradient \nabla Loss_{W} of the loss with
respect to the weights is computed and deducted from W. More formally,
this is expressed as,

   W^{i+1} = W^i - \epsilon \nabla {Loss}_{W}^{i}

where i is the iteration step, and \epsilon is the learning rate with
a value larger than 0.

The algorithm stops when it reaches a preset maximum number of
iterations; or when the improvement in loss is below a certain, small
number.

1.17.8. Tips on Practical Use

   * Multi-layer Perceptron is sensitive to feature scaling, so it is
     highly recommended to scale your data. For example, scale each
     attribute on the input vector X to [0, 1] or [-1, +1], or
     standardize it to have mean 0 and variance 1. Note that you must
     apply the *same* scaling to the test set for meaningful results.
     You can use "StandardScaler" for standardization.

     >>> from sklearn.preprocessing import StandardScaler  
     >>> scaler = StandardScaler()  
     >>> # Don't cheat - fit only on training data
     >>> scaler.fit(X_train)  
     >>> X_train = scaler.transform(X_train)  
     >>> # apply same transformation to test data
     >>> X_test = scaler.transform(X_test)  

     An alternative and recommended approach is to use
     "StandardScaler" in a "Pipeline"

   * Finding a reasonable regularization parameter \alpha is best done
     using "GridSearchCV", usually in the range "10.0 ** -np.arange(1,
     7)".

   * Empirically, we observed that *L-BFGS* converges faster and with

using "GridSearchCV", usually in the range "10.0 ** -np.arange(1,
     7)".

     better solutions on small datasets. For relatively large
     datasets, however, *Adam* is very robust. It usually converges
     quickly and gives pretty good performance. *SGD* with momentum or
     nesterov's momentum, on the other hand, can perform better than
     those two algorithms if learning rate is correctly tuned.

1.17.9. More control with warm_start

If you want more control over stopping criteria or learning rate in
SGD, or want to do additional monitoring, using "warm_start=True" and
"max_iter=1" and iterating yourself can be helpful:

   >>> clf = MLPClassifier(hidden_layer_sizes=(15,), random_state=1, max_iter=1, warm_start=True)
   >>> for i in range(10):
   ...     clf.fit(X, y)
   ...     # additional monitoring / inspection
   MLPClassifier(...

* "Learning representations by back-propagating errors." Rumelhart,
  David E., Geoffrey E. Hinton, and Ronald J. Williams.

* "Stochastic Gradient Descent" L. Bottou - Website, 2010.

* "Backpropagation" Andrew Ng, Jiquan Ngiam, Chuan Yu Foo, Yifan Mai,
  Caroline Suen - Website, 2011.

* "Efficient BackProp" Y. LeCun, L. Bottou, G. Orr, K. Müller - In
  Neural Networks: Tricks of the Trade 1998.

* >>:arxiv:`"Adam: A method for stochastic optimization."

Caroline Suen - Website, 2011.

  <1412.6980>`<< Kingma, Diederik, and Jimmy Ba (2014)

API Reference

This is the class and function reference of scikit-learn. Please refer
to the full user guide for further details, as the class and function
raw specifications may not be enough to give full guidelines on their
uses. For reference on concepts repeated across the API, see Glossary

"sklearn.base": Base classes and utility functions
==================================================

Base classes

Functions
---------

"sklearn.calibration": Probability Calibration
==============================================

**User guide:** See the Probability calibration section for further
details.

"sklearn.cluster": Clustering

**User guide:** See the Clustering and Biclustering sections for
further details.

Classes
-------

Functions
---------

"sklearn.compose": Composite Estimators

**User guide:** See the Pipelines and composite estimators section for

"sklearn.covariance": Covariance Estimators

**User guide:** See the Covariance estimation section for further
details.

"sklearn.cross_decomposition": Cross decomposition

**User guide:** See the Cross decomposition section for further
details.

"sklearn.datasets": Datasets

**User guide:** See the Dataset loading utilities section for further
details.

Loaders
-------

Samples generator

details.

details.

Loaders
-------

"sklearn.decomposition": Matrix Decomposition

**User guide:** See the Decomposing signals in components (matrix
factorization problems) section for further details.

"sklearn.discriminant_analysis": Discriminant Analysis
======================================================

**User guide:** See the Linear and Quadratic Discriminant Analysis
section for further details.

"sklearn.dummy": Dummy estimators

**User guide:** See the Metrics and scoring: quantifying the quality
of predictions section for further details.

"sklearn.ensemble": Ensemble Methods

**User guide:** See the Ensemble methods section for further details.

"sklearn.exceptions": Exceptions and warnings

"sklearn.experimental": Experimental

"sklearn.feature_extraction": Feature Extraction

**User guide:** See the Feature extraction section for further
details.

From images

From text
---------

"sklearn.feature_selection": Feature Selection

**User guide:** See the Feature selection section for further details.

"sklearn.gaussian_process": Gaussian Processes

**User guide:** See the Gaussian Processes section for further
details.

Kernels:

"sklearn.impute": Impute

**User guide:** See the Imputation of missing values section for

"sklearn.inspection": Inspection

Plotting
--------

"sklearn.isotonic": Isotonic regression

**User guide:** See the Isotonic regression section for further
details.

"sklearn.kernel_approximation": Kernel Approximation
====================================================

**User guide:** See the Kernel Approximation section for further
details.

"sklearn.kernel_ridge": Kernel Ridge Regression

**User guide:** See the Kernel ridge regression section for further
details.

"sklearn.linear_model": Linear Models

**User guide:** See the Linear Models section for further details.

The following subsections are only rough guidelines: the same
estimator can fall into multiple categories, depending on its
parameters.

Linear classifiers

Classical linear regressors

Regressors with variable selection

The following estimators have built-in variable selection fitting

procedures, but any estimator using a L1 or elastic-net penalty also
performs variable selection: typically "SGDRegressor" or
"SGDClassifier" with an appropriate penalty.

Bayesian regressors

Multi-task linear regressors with variable selection

These estimators fit multiple regression problems (or tasks) jointly,
while inducing sparse coefficients. While the inferred coefficients
may differ between the tasks, they are constrained to agree on the
features that are selected (non-zero coefficients).

Outlier-robust regressors

Any estimator using the Huber loss would also be robust to outliers,
e.g. "SGDRegressor" with "loss='huber'".

Generalized linear models (GLM) for regression

These models allow for response variables to have error distributions
other than a normal distribution:

"sklearn.manifold": Manifold Learning

**User guide:** See the Manifold learning section for further details.

"sklearn.metrics": Metrics

See the Metrics and scoring: quantifying the quality of predictions
section and the Pairwise metrics, Affinities and Kernels section of
the user guide for further details.

Model Selection Interface

See the The scoring parameter: defining model evaluation rules section
of the user guide for further details.

Classification metrics

See the Classification metrics section of the user guide for further
details.

Regression metrics

See the Regression metrics section of the user guide for further
details.

Multilabel ranking metrics

See the Multilabel ranking metrics section of the user guide for

Clustering metrics

See the Clustering performance evaluation section of the user guide
for further details.

Biclustering metrics

See the Biclustering evaluation section of the user guide for further
details.

Distance metrics

Pairwise metrics

See the Pairwise metrics, Affinities and Kernels section of the user
guide for further details.

Plotting
--------

See the Visualizations section of the user guide for further details.

"sklearn.mixture": Gaussian Mixture Models

**User guide:** See the Gaussian mixture models section for further
details.

"sklearn.model_selection": Model Selection

details.

**User guide:** See the Cross-validation: evaluating estimator
performance, Tuning the hyper-parameters of an estimator and Learning
curve sections for further details.

Splitter Classes

Splitter Functions

Hyper-parameter optimizers

Model validation

Visualization

"sklearn.multiclass": Multiclass classification

**User guide:** See the Multiclass classification section for further
details.

"sklearn.multioutput": Multioutput regression and classification
================================================================

**User guide:** See the Multilabel classification, Multiclass-
multioutput classification, and Multioutput regression sections for

"sklearn.naive_bayes": Naive Bayes

**User guide:** See the Naive Bayes section for further details.

"sklearn.neighbors": Nearest Neighbors
======================================

**User guide:** See the Nearest Neighbors section for further details.

"sklearn.neural_network": Neural network models

**User guide:** See the Neural network models (supervised) and Neural
network models (unsupervised) sections for further details.

"sklearn.pipeline": Pipeline

"sklearn.preprocessing": Preprocessing and Normalization

**User guide:** See the Preprocessing data section for further
details.

"sklearn.random_projection": Random projection

**User guide:** See the Random Projection section for further details.

"sklearn.semi_supervised": Semi-Supervised Learning

**User guide:** See the Semi-supervised learning section for further
details.

"sklearn.svm": Support Vector Machines

**User guide:** See the Support Vector Machines section for further
details.

Estimators

"sklearn.tree": Decision Trees

**User guide:** See the Decision Trees section for further details.

Plotting
--------

"sklearn.utils": Utilities

**Developer guide:** See the Utilities for Developers page for further
details.

Specific utilities to list scikit-learn components:

Utilities from joblib:

Recently deprecated

To be removed in 1.3

1.5. Stochastic Gradient Descent
********************************

**Stochastic Gradient Descent (SGD)** is a simple yet very efficient
approach to fitting linear classifiers and regressors under convex
loss functions such as (linear) Support Vector Machines and Logistic
Regression. Even though SGD has been around in the machine learning
community for a long time, it has received a considerable amount of
attention just recently in the context of large-scale learning.

SGD has been successfully applied to large-scale and sparse machine
learning problems often encountered in text classification and natural
language processing.  Given that the data is sparse, the classifiers
in this module easily scale to problems with more than 10^5 training
examples and more than 10^5 features.

Strictly speaking, SGD is merely an optimization technique and does
not correspond to a specific family of machine learning models. It is
only a *way* to train a model. Often, an instance of "SGDClassifier"
or "SGDRegressor" will have an equivalent estimator in the scikit-
learn API, potentially using a different optimization technique. For
example, using *SGDClassifier(loss='log_loss')* results in logistic
regression, i.e. a model equivalent to "LogisticRegression" which is
fitted via SGD instead of being fitted by one of the other solvers in
"LogisticRegression". Similarly, *SGDRegressor(loss='squared_error',
penalty='l2')* and "Ridge" solve the same optimization problem, via
different means.

The advantages of Stochastic Gradient Descent are:

   * Efficiency.

   * Ease of implementation (lots of opportunities for code tuning).

The disadvantages of Stochastic Gradient Descent include:

   * SGD requires a number of hyperparameters such as the
     regularization parameter and the number of iterations.

   * SGD is sensitive to feature scaling.

Warning:

  Make sure you permute (shuffle) your training data before fitting
  the model or use "shuffle=True" to shuffle after each iteration
  (used by default). Also, ideally, features should be standardized
  using e.g. *make_pipeline(StandardScaler(), SGDClassifier())* (see
  Pipelines).

1.5.1. Classification

The class "SGDClassifier" implements a plain stochastic gradient
descent learning routine which supports different loss functions and
penalties for classification. Below is the decision boundary of a
"SGDClassifier" trained with the hinge loss, equivalent to a linear
SVM.

   [image]

As other classifiers, SGD has to be fitted with two arrays: an array
*X* of shape (n_samples, n_features) holding the training samples, and
an array y of shape (n_samples,) holding the target values (class
labels) for the training samples:

   >>> clf = SGDClassifier(loss="hinge", penalty="l2", max_iter=5)
   SGDClassifier(max_iter=5)

   >>> clf.predict([[2., 2.]])
   array([1])

SGD fits a linear model to the training data. The "coef_" attribute
holds the model parameters:

   >>> clf.coef_
   array([[9.9..., 9.9...]])

The "intercept_" attribute holds the intercept (aka offset or bias):

   >>> clf.intercept_
   array([-9.9...])

Whether or not the model should use an intercept, i.e. a biased
hyperplane, is controlled by the parameter "fit_intercept".

The signed distance to the hyperplane (computed as the dot product
between the coefficients and the input sample, plus the intercept) is
given by "SGDClassifier.decision_function()":

   >>> clf.decision_function([[2., 2.]])
   array([29.6...])

The concrete loss function can be set via the "loss" parameter.
"SGDClassifier" supports the following loss functions:

   * "loss="hinge"": (soft-margin) linear Support Vector Machine,

   * "loss="modified_huber"": smoothed hinge loss,

   * "loss="log_loss"": logistic regression,

   * and all regression losses below. In this case the target is
     encoded as -1 or 1, and the problem is treated as a regression

* "loss="log_loss"": logistic regression,

     problem. The predicted class then correspond to the sign of the
     predicted target.

Please refer to the mathematical section below for formulas. The first
two loss functions are lazy, they only update the model parameters if
an example violates the margin constraint, which makes training very
efficient and may result in sparser models (i.e. with more zero
coefficients), even when L2 penalty is used.

Using "loss="log_loss"" or "loss="modified_huber"" enables the
"predict_proba" method, which gives a vector of probability estimates
P(y|x) per sample x:

   >>> clf = SGDClassifier(loss="log_loss", max_iter=5).fit(X, y)
   >>> clf.predict_proba([[1., 1.]]) 
   array([[0.00..., 0.99...]])

The concrete penalty can be set via the "penalty" parameter. SGD
supports the following penalties:

   * "penalty="l2"": L2 norm penalty on "coef_".

   * "penalty="l1"": L1 norm penalty on "coef_".

   * "penalty="elasticnet"": Convex combination of L2 and L1; "(1 -
     l1_ratio) * L2 + l1_ratio * L1".

The default setting is "penalty="l2"". The L1 penalty leads to sparse
solutions, driving most coefficients to zero. The Elastic Net [11]
solves some deficiencies of the L1 penalty in the presence of highly
correlated attributes. The parameter "l1_ratio" controls the convex
combination of L1 and L2 penalty.

"SGDClassifier" supports multi-class classification by combining
multiple binary classifiers in a "one versus all" (OVA) scheme. For
each of the K classes, a binary classifier is learned that
discriminates between that and all other K-1 classes. At testing time,
we compute the confidence score (i.e. the signed distances to the
hyperplane) for each classifier and choose the class with the highest
confidence. The Figure below illustrates the OVA approach on the iris
dataset.  The dashed lines represent the three OVA classifiers; the
background colors show the decision surface induced by the three
classifiers.

   [image]

In the case of multi-class classification "coef_" is a two-dimensional
array of shape (n_classes, n_features) and "intercept_" is a one-
dimensional array of shape (n_classes,). The i-th row of "coef_" holds
the weight vector of the OVA classifier for the i-th class; classes
are indexed in ascending order (see attribute "classes_"). Note that,
in principle, since they allow to create a probability model,
"loss="log_loss"" and "loss="modified_huber"" are more suitable for
one-vs-all classification.

"SGDClassifier" supports both weighted classes and weighted instances
via the fit parameters "class_weight" and "sample_weight". See the
examples below and the docstring of "SGDClassifier.fit()" for further
information.

"SGDClassifier" supports averaged SGD (ASGD) [10]. Averaging can be
enabled by setting *average=True*. ASGD performs the same updates as
the regular SGD (see Mathematical formulation), but instead of using
the last value of the coefficients as the *coef_* attribute (i.e. the
values of the last update), *coef_* is set instead to the **average**
value of the coefficients across all updates. The same is done for the
*intercept_* attribute. When using ASGD the learning rate can be
larger and even constant, leading on some datasets to a speed up in
training time.

For classification with a logistic loss, another variant of SGD with
an averaging strategy is available with Stochastic Average Gradient
(SAG) algorithm, available as a solver in "LogisticRegression".

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_sgd_separating_hyperplane
  .py*,

* *sphx_glr_auto_examples_linear_model_plot_sgd_iris.py*

* *sphx_glr_auto_examples_linear_model_plot_sgd_weighted_samples.py*

* *sphx_glr_auto_examples_linear_model_plot_sgd_comparison.py*

* *sphx_glr_auto_examples_svm_plot_separating_hyperplane_unbalanced.p
  y* (See the Note in the example)

1.5.2. Regression

The class "SGDRegressor" implements a plain stochastic gradient

y* (See the Note in the example)

penalties to fit linear regression models. "SGDRegressor" is well
suited for regression problems with a large number of training samples
(> 10.000), for other problems we recommend "Ridge", "Lasso", or
"ElasticNet".

"SGDRegressor" supports the following loss functions:

   * "loss="squared_error"": Ordinary least squares,

   * "loss="huber"": Huber loss for robust regression,

   * "loss="epsilon_insensitive"": linear Support Vector Regression.

Please refer to the mathematical section below for formulas. The Huber
and epsilon-insensitive loss functions can be used for robust
regression. The width of the insensitive region has to be specified
via the parameter "epsilon". This parameter depends on the scale of
the target variables.

The *penalty* parameter determines the regularization to be used (see
description above in the classification section).

"SGDRegressor" also supports averaged SGD [10] (here again, see

For regression with a squared loss and a l2 penalty, another variant
of SGD with an averaging strategy is available with Stochastic Average
Gradient (SAG) algorithm, available as a solver in "Ridge".

1.5.3. Online One-Class SVM

The class "sklearn.linear_model.SGDOneClassSVM" implements an online
linear version of the One-Class SVM using a stochastic gradient
descent. Combined with kernel approximation techniques,
"sklearn.linear_model.SGDOneClassSVM" can be used to approximate the
solution of a kernelized One-Class SVM, implemented in
"sklearn.svm.OneClassSVM", with a linear complexity in the number of
samples. Note that the complexity of a kernelized One-Class SVM is at
best quadratic in the number of samples.
"sklearn.linear_model.SGDOneClassSVM" is thus well suited for datasets
with a large number of training samples (> 10,000) for which the SGD
variant can be several orders of magnitude faster.

Its implementation is based on the implementation of the stochastic
gradient descent. Indeed, the original optimization problem of the
One-Class SVM is given by

   \begin{aligned} \min_{w, \rho, \xi} & \quad \frac{1}{2}\Vert w
   \Vert^2 - \rho + \frac{1}{\nu n} \sum_{i=1}^n \xi_i \\ \text{s.t.}
   & \quad \langle w, x_i \rangle \geq \rho - \xi_i \quad 1 \leq i
   \leq n \\ & \quad \xi_i \geq 0 \quad 1 \leq i \leq n \end{aligned}

where \nu \in (0, 1] is the user-specified parameter controlling the
proportion of outliers and the proportion of support vectors. Getting
rid of the slack variables \xi_i this problem is equivalent to

   \min_{w, \rho} \frac{1}{2}\Vert w \Vert^2 - \rho + \frac{1}{\nu n}
   \sum_{i=1}^n \max(0, \rho - \langle w, x_i \rangle) \, .

Multiplying by the constant \nu and introducing the intercept b = 1 -
\rho we obtain the following equivalent optimization problem

   \min_{w, b} \frac{\nu}{2}\Vert w \Vert^2 + b\nu + \frac{1}{n}
   \sum_{i=1}^n \max(0, 1 - (\langle w, x_i \rangle + b)) \, .

This is similar to the optimization problems studied in section
Mathematical formulation with y_i = 1, 1 \leq i \leq n and \alpha =
\nu/2, L being the hinge loss function and R being the L2 norm. We
just need to add the term b\nu in the optimization loop.

As "SGDClassifier" and "SGDRegressor", "SGDOneClassSVM" supports
averaged SGD. Averaging can be enabled by setting "average=True".

1.5.4. Stochastic Gradient Descent for sparse data

Note:

  The sparse implementation produces slightly different results from
  the dense implementation, due to a shrunk learning rate for the
  intercept. See Implementation details.

There is built-in support for sparse data given in any matrix in a
format supported by scipy.sparse. For maximum efficiency, however, use
the CSR matrix format as defined in scipy.sparse.csr_matrix.

Examples:
^^^^^^^^^

Examples:
^^^^^^^^^

  ups.py*

1.5.5. Complexity

The major advantage of SGD is its efficiency, which is basically
linear in the number of training examples. If X is a matrix of size
(n, p) training has a cost of O(k n \bar p), where k is the number of
iterations (epochs) and \bar p is the average number of non-zero
attributes per sample.

Recent theoretical results, however, show that the runtime to get some
desired optimization accuracy does not increase as the training set
size increases.

1.5.6. Stopping criterion

The classes "SGDClassifier" and "SGDRegressor" provide two criteria to
stop the algorithm when a given level of convergence is reached:

   * With "early_stopping=True", the input data is split into a
     training set and a validation set. The model is then fitted on
     the training set, and the stopping criterion is based on the
     prediction score (using the *score* method) computed on the
     validation set. The size of the validation set can be changed
     with the parameter "validation_fraction".

   * With "early_stopping=False", the model is fitted on the entire
     input data and the stopping criterion is based on the objective
     function computed on the training data.

In both cases, the criterion is evaluated once by epoch, and the

input data and the stopping criterion is based on the objective

algorithm stops when the criterion does not improve "n_iter_no_change"
times in a row. The improvement is evaluated with absolute tolerance
"tol", and the algorithm stops in any case after a maximum number of
iteration "max_iter".

1.5.7. Tips on Practical Use

   * Stochastic Gradient Descent is sensitive to feature scaling, so
     it is highly recommended to scale your data. For example, scale
     each attribute on the input vector X to [0,1] or [-1,+1], or
     standardize it to have mean 0 and variance 1. Note that the
     *same* scaling must be applied to the test vector to obtain
     meaningful results. This can be easily done using
     "StandardScaler":

        from sklearn.preprocessing import StandardScaler
        scaler = StandardScaler()
        scaler.fit(X_train)  # Don't cheat - fit only on training data
        X_train = scaler.transform(X_train)
        X_test = scaler.transform(X_test)  # apply same transformation to test data

        # Or better yet: use a pipeline!
        from sklearn.pipeline import make_pipeline
        est = make_pipeline(StandardScaler(), SGDClassifier())
        est.fit(X_train)
        est.predict(X_test)

     If your attributes have an intrinsic scale (e.g. word frequencies
     or indicator features) scaling is not needed.

est.fit(X_train)

   * Finding a reasonable regularization term \alpha is best done
     using automatic hyper-parameter search, e.g. "GridSearchCV" or
     "RandomizedSearchCV", usually in the range
     "10.0**-np.arange(1,7)".

   * Empirically, we found that SGD converges after observing
     approximately 10^6 training samples. Thus, a reasonable first
     guess for the number of iterations is "max_iter = np.ceil(10**6 /
     n)", where "n" is the size of the training set.

   * If you apply SGD to features extracted using PCA we found that it
     is often wise to scale the feature values by some constant *c*
     such that the average L2 norm of the training data equals one.

   * We found that Averaged SGD works best with a larger number of
     features and a higher eta0

1.5.8. Mathematical formulation

We describe here the mathematical details of the SGD procedure. A good
overview with convergence rates can be found in [12].

Given a set of training examples (x_1, y_1), \ldots, (x_n, y_n) where
x_i \in \mathbf{R}^m and y_i \in \mathcal{R} (y_i \in {-1, 1} for
classification), our goal is to learn a linear scoring function f(x) =

w^T x + b with model parameters w \in \mathbf{R}^m and intercept b \in
\mathbf{R}. In order to make predictions for binary classification, we
simply look at the sign of f(x). To find the model parameters, we
minimize the regularized training error given by

   E(w,b) = \frac{1}{n}\sum_{i=1}^{n} L(y_i, f(x_i)) + \alpha R(w)

where L is a loss function that measures model (mis)fit and R is a
regularization term (aka penalty) that penalizes model complexity;
\alpha > 0 is a non-negative hyperparameter that controls the
regularization strength.

Different choices for L entail different classifiers or regressors:

* Hinge (soft-margin): equivalent to Support Vector Classification.
  L(y_i, f(x_i)) = \max(0, 1 - y_i f(x_i)).

* Perceptron: L(y_i, f(x_i)) = \max(0, - y_i f(x_i)).

* Modified Huber: L(y_i, f(x_i)) = \max(0, 1 - y_i f(x_i))^2 if y_i
  f(x_i) > 1, and L(y_i, f(x_i)) = -4 y_i f(x_i) otherwise.

* Log Loss: equivalent to Logistic Regression. L(y_i, f(x_i)) = \log(1
  + \exp (-y_i f(x_i))).

* Squared Error: Linear regression (Ridge or Lasso depending on R).
  L(y_i, f(x_i)) = \frac{1}{2}(y_i - f(x_i))^2.

* Huber: less sensitive to outliers than least-squares. It is
  equivalent to least squares when |y_i - f(x_i)| \leq \varepsilon,
  and L(y_i, f(x_i)) = \varepsilon |y_i - f(x_i)| - \frac{1}{2}
  \varepsilon^2 otherwise.

equivalent to least squares when |y_i - f(x_i)| \leq \varepsilon,

* Epsilon-Insensitive: (soft-margin) equivalent to Support Vector
  Regression. L(y_i, f(x_i)) = \max(0, |y_i - f(x_i)| - \varepsilon).

All of the above loss functions can be regarded as an upper bound on
the misclassification error (Zero-one loss) as shown in the Figure
below.

   [image]

Popular choices for the regularization term R (the *penalty*
parameter) include:

   * L2 norm: R(w) := \frac{1}{2} \sum_{j=1}^{m} w_j^2 = ||w||_2^2,

   * L1 norm: R(w) := \sum_{j=1}^{m} |w_j|, which leads to sparse
     solutions.

   * Elastic Net: R(w) := \frac{\rho}{2} \sum_{j=1}^{n} w_j^2 +
     (1-\rho) \sum_{j=1}^{m} |w_j|, a convex combination of L2 and L1,
     where \rho is given by "1 - l1_ratio".

The Figure below shows the contours of the different regularization
terms in a 2-dimensional parameter space (m=2) when R(w) = 1.

   [image]

1.5.8.1. SGD

Stochastic gradient descent is an optimization method for
unconstrained optimization problems. In contrast to (batch) gradient
descent, SGD approximates the true gradient of E(w,b) by considering a
single training example at a time.

The class "SGDClassifier" implements a first-order SGD learning
routine.  The algorithm iterates over the training examples and for
each example updates the model parameters according to the update rule
given by

given by

   w \leftarrow w - \eta \left[\alpha \frac{\partial R(w)}{\partial w}
   + \frac{\partial L(w^T x_i + b, y_i)}{\partial w}\right]

parameter space.  The intercept b is updated similarly but without
regularization (and with additional decay for sparse matrices, as
detailed in Implementation details).

The learning rate \eta can be either constant or gradually decaying.
For classification, the default learning rate schedule
("learning_rate='optimal'") is given by

   \eta^{(t)} = \frac {1}{\alpha  (t_0 + t)}

where t is the time step (there are a total of *n_samples * n_iter*
time steps), t_0 is determined based on a heuristic proposed by Léon
Bottou such that the expected initial updates are comparable with the
expected size of the weights (this assuming that the norm of the
training samples is approx. 1). The exact definition can be found in
"_init_t" in "BaseSGD".

For regression the default learning rate schedule is inverse scaling
("learning_rate='invscaling'"), given by

   \eta^{(t)} = \frac{eta_0}{t^{power\_t}}

where eta_0 and power\_t are hyperparameters chosen by the user via
"eta0" and "power_t", resp.

For a constant learning rate use "learning_rate='constant'" and use
"eta0" to specify the learning rate.

For an adaptively decreasing learning rate, use

"learning_rate='adaptive'" and use "eta0" to specify the starting
learning rate. When the stopping criterion is reached, the learning
rate is divided by 5, and the algorithm does not stop. The algorithm
stops when the learning rate goes below 1e-6.

The model parameters can be accessed through the "coef_" and
"intercept_" attributes: "coef_" holds the weights w and "intercept_"
holds b.

When using Averaged SGD (with the *average* parameter), *coef_* is set
to the average weight across all updates: *coef_* = \frac{1}{T}
\sum_{t=0}^{T-1} w^{(t)}, where T is the total number of updates,
found in the *t_* attribute.

1.5.9. Implementation details

The implementation of SGD is influenced by the *Stochastic Gradient
SVM* of [7]. Similar to SvmSGD, the weight vector is represented as
the product of a scalar and a vector which allows an efficient weight
update in the case of L2 regularization. In the case of sparse input
*X*, the intercept is updated with a smaller learning rate (multiplied
by 0.01) to account for the fact that it is updated more frequently.
Training examples are picked up sequentially and the learning rate is
lowered after each observed example. We adopted the learning rate
schedule from [8]. For multi-class classification, a "one versus all"

approach is used. We use the truncated gradient algorithm proposed in
[9] for L1 regularization (and the Elastic Net). The code is written
in Cython.

[7] "Stochastic Gradient Descent" L. Bottou - Website, 2010.

[8] >>:doi:`"Pegasos: Primal estimated sub-gradient solver for svm"
    <10.1145/1273496.1273598>`<< S. Shalev-Shwartz, Y. Singer, N.
    Srebro - In Proceedings of ICML '07.

[9] "Stochastic gradient descent training for l1-regularized log-
    linear models with cumulative penalty" Y. Tsuruoka, J. Tsujii, S.
    Ananiadou - In Proceedings of the AFNLP/ACL '09.

[10] >>:arxiv:`"Towards Optimal One Pass Large Scale Learning with
     Averaged Stochastic Gradient Descent" <1107.2490v2>`<< Xu, Wei
     (2011)

[11] >>:doi:`"Regularization and variable selection via the elastic
     net" <10.1111/j.1467-9868.2005.00503.x>`<< H. Zou, T. Hastie -
     Journal of the Royal Statistical Society Series B, 67 (2),
     301-320.

[12] >>:doi:`"Solving large scale linear prediction problems using
     stochastic gradient descent algorithms"
     <10.1145/1015330.1015332>`<< T. Zhang - In Proceedings of ICML
     '04.

11.1. Array API support (experimental)
**************************************

The Array API specification defines a standard API for all array
manipulation libraries with a NumPy-like API.

Some scikit-learn estimators that primarily rely on NumPy (as opposed
to using Cython) to implement the algorithmic logic of their *fit*,
*predict* or *transform* methods can be configured to accept any Array
API compatible input datastructures and automatically dispatch
operations to the underlying namespace instead of relying on NumPy.

At this stage, this support is **considered experimental** and must be
enabled explicitly as explained in the following.

Note:

  Currently, only *cupy.array_api* and *numpy.array_api* are known to
  work with scikit-learn's estimators.

11.1.1. Example usage

Here is an example code snippet to demonstrate how to use CuPy to run
"LinearDiscriminantAnalysis" on a GPU:

   >>> from sklearn import config_context
   >>> import cupy.array_api as xp

   >>> X_np, y_np = make_classification(random_state=0)
   >>> X_cu = xp.asarray(X_np)
   >>> y_cu = xp.asarray(y_np)
   >>> X_cu.device
   <CUDA Device 0>

   >>> with config_context(array_api_dispatch=True):
   ...     lda = LinearDiscriminantAnalysis()
   ...     X_trans = lda.fit_transform(X_cu, y_cu)
   >>> X_trans.device

>>> with config_context(array_api_dispatch=True):

After the model is trained, fitted attributes that are arrays will
also be from the same Array API namespace as the training data. For
example, if CuPy's Array API namespace was used for training, then
fitted attributes will be on the GPU. We provide a experimental
*_estimator_with_converted_arrays* utility that transfers an estimator
attributes from Array API to a ndarray:

   >>> from sklearn.utils._array_api import _estimator_with_converted_arrays
   >>> cupy_to_ndarray = lambda array : array._array.get()
   >>> lda_np = _estimator_with_converted_arrays(lda, cupy_to_ndarray)
   >>> X_trans = lda_np.transform(X_np)
   >>> type(X_trans)
   <class 'numpy.ndarray'>

11.1.2. Estimators with support for *Array API*-compatible inputs
=================================================================

* "discriminant_analysis.LinearDiscriminantAnalysis" (with
  *solver="svd"*)

Coverage for more estimators is expected to grow over time. Please
follow the dedicated meta-issue on GitHub to track progress.

2.5. Decomposing signals in components (matrix factorization problems)
**********************************************************************

2.5.1. Principal component analysis (PCA)

2.5.1.1. Exact PCA and probabilistic interpretation

PCA is used to decompose a multivariate dataset in a set of successive
orthogonal components that explain a maximum amount of the variance.
In scikit-learn, "PCA" is implemented as a *transformer* object that
learns n components in its "fit" method, and can be used on new data
to project it on these components.

PCA centers but does not scale the input data for each feature before
applying the SVD. The optional parameter "whiten=True" makes it
possible to project the data onto the singular space while scaling
each component to unit variance. This is often useful if the models
down-stream make strong assumptions on the isotropy of the signal:
this is for example the case for Support Vector Machines with the RBF
kernel and the K-Means clustering algorithm.

Below is an example of the iris dataset, which is comprised of 4
features, projected on the 2 dimensions that explain most variance:

   [image]

The "PCA" object also provides a probabilistic interpretation of the
PCA that can give a likelihood of data based on the amount of variance
it explains. As such it implements a *score* method that can be used
in cross-validation:

   [image]

Examples:
^^^^^^^^^

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_pca_vs_lda.py*

* *sphx_glr_auto_examples_decomposition_plot_pca_vs_fa_model_selectio
  n.py*

2.5.1.2. Incremental PCA

The "PCA" object is very useful, but has certain limitations for large
datasets. The biggest limitation is that "PCA" only supports batch
processing, which means all of the data to be processed must fit in
main memory. The "IncrementalPCA" object uses a different form of
processing and allows for partial computations which almost exactly
match the results of "PCA" while processing the data in a minibatch
fashion. "IncrementalPCA" makes it possible to implement out-of-core
Principal Component Analysis either by:

   * Using its "partial_fit" method on chunks of data fetched
     sequentially from the local hard drive or a network database.

   * Calling its fit method on a sparse matrix or a memory mapped file
     using "numpy.memmap".

"IncrementalPCA" only stores estimates of component and noise
variances, in order update "explained_variance_ratio_" incrementally.
This is why memory usage depends on the number of samples per batch,
rather than the number of samples to be processed in the dataset.

As in "PCA", "IncrementalPCA" centers but does not scale the input
data for each feature before applying the SVD.

   [image]

   [image]

   [image]

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_incremental_pca.py*

2.5.1.3. PCA using randomized SVD

It is often interesting to project data to a lower-dimensional space
that preserves most of the variance, by dropping the singular vector
of components associated with lower singular values.

For instance, if we work with 64x64 pixel gray-level pictures for face
recognition, the dimensionality of the data is 4096 and it is slow to
train an RBF support vector machine on such wide data. Furthermore we
know that the intrinsic dimensionality of the data is much lower than
4096 since all pictures of human faces look somewhat alike. The
samples lie on a manifold of much lower dimension (say around 200 for
instance). The PCA algorithm can be used to linearly transform the
data while both reducing the dimensionality and preserve most of the
explained variance at the same time.

The class "PCA" used with the optional parameter
"svd_solver='randomized'" is very useful in that case: since we are
going to drop most of the singular vectors it is much more efficient
to limit the computation to an approximated estimate of the singular
vectors we will keep to actually perform the transform.

For instance, the following shows 16 sample portraits (centered around

0.0) from the Olivetti dataset. On the right hand side are the first
16 singular vectors reshaped as portraits. Since we only require the
top 16 singular vectors of a dataset with size n_{samples} = 400 and
n_{features} = 64 \times 64 = 4096, the computation time is less than
1s:

[image: orig_img][image] [image: pca_img][image]

If we note n_{\max} = \max(n_{\mathrm{samples}},
n_{\mathrm{features}}) and n_{\min} = \min(n_{\mathrm{samples}},
n_{\mathrm{features}}), the time complexity of the randomized "PCA" is
O(n_{\max}^2 \cdot n_{\mathrm{components}}) instead of O(n_{\max}^2
\cdot n_{\min}) for the exact method implemented in "PCA".

The memory footprint of randomized "PCA" is also proportional to 2
\cdot n_{\max} \cdot n_{\mathrm{components}} instead of n_{\max} \cdot
n_{\min} for the exact method.

Note: the implementation of "inverse_transform" in "PCA" with
"svd_solver='randomized'" is not the exact inverse transform of
"transform" even when "whiten=False" (default).

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_applications_plot_face_recognition.py*

* *sphx_glr_auto_examples_decomposition_plot_faces_decomposition.py*

* Algorithm 4.3 in >>:arxiv:`"Finding structure with randomness:
  Stochastic algorithms for constructing approximate matrix

  decompositions" <0909.4061>`<< Halko, et al., 2009

* >>:arxiv:`"An implementation of a randomized algorithm for principal
  component analysis" <1412.3510>`<< A. Szlam et al. 2014

2.5.1.4. Sparse principal components analysis (SparsePCA and MiniBatchSparsePCA)
--------------------------------------------------------------------------------

"SparsePCA" is a variant of PCA, with the goal of extracting the set
of sparse components that best reconstruct the data.

Mini-batch sparse PCA ("MiniBatchSparsePCA") is a variant of
"SparsePCA" that is faster but less accurate. The increased speed is
reached by iterating over small chunks of the set of features, for a
given number of iterations.

Principal component analysis ("PCA") has the disadvantage that the
components extracted by this method have exclusively dense
expressions, i.e. they have non-zero coefficients when expressed as
linear combinations of the original variables. This can make
interpretation difficult. In many cases, the real underlying
components can be more naturally imagined as sparse vectors; for
example in face recognition, components might naturally map to parts
of faces.

Sparse principal components yields a more parsimonious, interpretable
representation, clearly emphasizing which of the original features
contribute to the differences between samples.

of faces.

The following example illustrates 16 components extracted using sparse
PCA from the Olivetti faces dataset.  It can be seen how the
regularization term induces many zeros. Furthermore, the natural
structure of the data causes the non-zero coefficients to be
vertically adjacent. The model does not enforce this mathematically:
each component is a vector h \in \mathbf{R}^{4096}, and there is no
notion of vertical adjacency except during the human-friendly
visualization as 64x64 pixel images. The fact that the components
shown below appear local is the effect of the inherent structure of
the data, which makes such local patterns minimize reconstruction
error. There exist sparsity-inducing norms that take into account
adjacency and different kinds of structure; see [Jen09] for a review
of such methods. For more details on how to use Sparse PCA, see the
Examples section, below.

[image: pca_img][image] [image: spca_img][image]

Note that there are many different formulations for the Sparse PCA
problem. The one implemented here is based on [Mrl09] . The
optimization problem solved is a PCA problem (dictionary learning)
with an \ell_1 penalty on the components:

   (U^*, V^*) = \underset{U, V}{\operatorname{arg\,min\,}} &
   \frac{1}{2}              ||X-UV||_{\text{Fro}}^2+\alpha||V||_{1,1}

   \\              \text{subject to } & ||U_k||_2 <= 1 \text{ for all
   }              0 \leq k < n_{components}

||.||_{\text{Fro}} stands for the Frobenius norm and ||.||_{1,1}
stands for the entry-wise matrix norm which is the sum of the absolute
values of all the entries in the matrix. The sparsity-inducing
||.||_{1,1} matrix norm also prevents learning components from noise
when few training samples are available. The degree of penalization
(and thus sparsity) can be adjusted through the hyperparameter
"alpha". Small values lead to a gently regularized factorization,
while larger values shrink many coefficients to zero.

Note:

  While in the spirit of an online algorithm, the class
  "MiniBatchSparsePCA" does not implement "partial_fit" because the
  algorithm is online along the features direction, not the samples
  direction.

Examples:
^^^^^^^^^

[Mrl09] "Online Dictionary Learning for Sparse Coding" J. Mairal, F.
        Bach, J. Ponce, G. Sapiro, 2009

[Jen09] "Structured Sparse Principal Component Analysis" R. Jenatton,
        G. Obozinski, F. Bach, 2009

2.5.2. Kernel Principal Component Analysis (kPCA)

2.5.2.1. Exact Kernel PCA

G. Obozinski, F. Bach, 2009

"KernelPCA" is an extension of PCA which achieves non-linear
dimensionality reduction through the use of kernels (see Pairwise
metrics, Affinities and Kernels) [Scholkopf1997]. It has many
applications including denoising, compression and structured
prediction (kernel dependency estimation). "KernelPCA" supports both
"transform" and "inverse_transform".

   [image]

Note:

  "KernelPCA.inverse_transform()" relies on a kernel ridge to learn
  the function mapping samples from the PCA basis into the original
  feature space [Bakir2003]. Thus, the reconstruction obtained with
  "KernelPCA.inverse_transform()" is an approximation. See the example
  linked below for more details.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_kernel_pca.py*

[Scholkopf1997] Schölkopf, Bernhard, Alexander Smola, and Klaus-Robert
                Müller. "Kernel principal component analysis."
                International conference on artificial neural
                networks. Springer, Berlin, Heidelberg, 1997.

[Bakir2003] Bakır, Gökhan H., Jason Weston, and Bernhard Schölkopf.
            "Learning to find pre-images." Advances in neural
            information processing systems 16 (2003): 449-456.

2.5.2.2. Choice of solver for Kernel PCA

information processing systems 16 (2003): 449-456.

While in "PCA" the number of components is bounded by the number of
features, in "KernelPCA" the number of components is bounded by the
number of samples. Many real-world datasets have large number of
samples! In these cases finding *all* the components with a full kPCA
is a waste of computation time, as data is mostly described by the
first few components (e.g. "n_components<=100"). In other words, the
centered Gram matrix that is eigendecomposed in the Kernel PCA fitting
process has an effective rank that is much smaller than its size. This
is a situation where approximate eigensolvers can provide speedup with
very low precision loss.

The optional parameter "eigen_solver='randomized'" can be used to
*significantly* reduce the computation time when the number of
requested "n_components" is small compared with the number of samples.
It relies on randomized decomposition methods to find an approximate
solution in a shorter time.

The time complexity of the randomized "KernelPCA" is
O(n_{\mathrm{samples}}^2 \cdot n_{\mathrm{components}}) instead of
O(n_{\mathrm{samples}}^3) for the exact method implemented with
"eigen_solver='dense'".

The memory footprint of randomized "KernelPCA" is also proportional to
2 \cdot n_{\mathrm{samples}} \cdot n_{\mathrm{components}} instead of
n_{\mathrm{samples}}^2 for the exact method.

Note: this technique is the same as in PCA using randomized SVD.

In addition to the above two solvers, "eigen_solver='arpack'" can be
used as an alternate way to get an approximate decomposition. In
practice, this method only provides reasonable execution times when
the number of components to find is extremely small. It is enabled by
default when the desired number of components is less than 10 (strict)
and the number of samples is more than 200 (strict). See "KernelPCA"
for details.

* *dense* solver: scipy.linalg.eigh documentation

* *randomized* solver:

     * Algorithm 4.3 in >>:arxiv:`"Finding structure with randomness:
       Stochastic algorithms for constructing approximate matrix
       decompositions" <0909.4061>`<< Halko, et al. (2009)

     * >>:arxiv:`"An implementation of a randomized algorithm for
       principal component analysis" <1412.3510>`<< A. Szlam et al.
       (2014)

* *arpack* solver: scipy.sparse.linalg.eigsh documentation R. B.
  Lehoucq, D. C. Sorensen, and C. Yang, (1998)

2.5.3. Truncated singular value decomposition and latent semantic analysis
==========================================================================

"TruncatedSVD" implements a variant of singular value decomposition
(SVD) that only computes the k largest singular values, where k is a

user-specified parameter.

When truncated SVD is applied to term-document matrices (as returned
by "CountVectorizer" or "TfidfVectorizer"), this transformation is
known as latent semantic analysis (LSA), because it transforms such
matrices to a "semantic" space of low dimensionality. In particular,
LSA is known to combat the effects of synonymy and polysemy (both of
which roughly mean there are multiple meanings per word), which cause
term-document matrices to be overly sparse and exhibit poor similarity
under measures such as cosine similarity.

Note:

  LSA is also known as latent semantic indexing, LSI, though strictly
  that refers to its use in persistent indexes for information
  retrieval purposes.

Mathematically, truncated SVD applied to training samples X produces a
low-rank approximation X:

   X \approx X_k = U_k \Sigma_k V_k^\top

After this operation, U_k \Sigma_k is the transformed training set
with k features (called "n_components" in the API).

To also transform a test set X, we multiply it with V_k:

   X' = X V_k

Note:

  Most treatments of LSA in the natural language processing (NLP) and
  information retrieval (IR) literature swap the axes of the matrix X
  so that it has shape "n_features" × "n_samples". We present LSA in a
  different way that matches the scikit-learn API better, but the
  singular values found are the same.

so that it has shape "n_features" × "n_samples". We present LSA in a

"TruncatedSVD" is very similar to "PCA", but differs in that the
matrix X does not need to be centered. When the columnwise (per-
feature) means of X are subtracted from the feature values, truncated
SVD on the resulting matrix is equivalent to PCA. In practical terms,
this means that the "TruncatedSVD" transformer accepts "scipy.sparse"
matrices without the need to densify them, as densifying may fill up
memory even for medium-sized document collections.

While the "TruncatedSVD" transformer works with any feature matrix,
using it on tf–idf matrices is recommended over raw frequency counts
in an LSA/document processing setting. In particular, sublinear
scaling and inverse document frequency should be turned on
("sublinear_tf=True, use_idf=True") to bring the feature values closer
to a Gaussian distribution, compensating for LSA's erroneous
assumptions about textual data.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_text_plot_document_clustering.py*

* Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze
  (2008), *Introduction to Information Retrieval*, Cambridge
  University Press, chapter 18: Matrix decompositions & latent
  semantic indexing

2.5.4. Dictionary Learning

2.5.4.1. Sparse coding with a precomputed dictionary

University Press, chapter 18: Matrix decompositions & latent

The "SparseCoder" object is an estimator that can be used to transform
signals into sparse linear combination of atoms from a fixed,
precomputed dictionary such as a discrete wavelet basis. This object
therefore does not implement a "fit" method. The transformation
amounts to a sparse coding problem: finding a representation of the
data as a linear combination of as few dictionary atoms as possible.
All variations of dictionary learning implement the following
transform methods, controllable via the "transform_method"
initialization parameter:

* Orthogonal matching pursuit (Orthogonal Matching Pursuit (OMP))

* Least-angle regression (Least Angle Regression)

* Lasso computed by least-angle regression

* Lasso using coordinate descent (Lasso)

* Thresholding

Thresholding is very fast but it does not yield accurate
reconstructions. They have been shown useful in literature for
classification tasks. For image reconstruction tasks, orthogonal
matching pursuit yields the most accurate, unbiased reconstruction.

The dictionary learning objects offer, via the "split_code" parameter,
the possibility to separate the positive and negative values in the
results of sparse coding. This is useful when dictionary learning is

used for extracting features that will be used for supervised
learning, because it allows the learning algorithm to assign different
weights to negative loadings of a particular atom, from to the
corresponding positive loading.

The split code for a single sample has length "2 * n_components" and
is constructed using the following rule: First, the regular code of
length "n_components" is computed. Then, the first "n_components"
entries of the "split_code" are filled with the positive part of the
regular code vector. The second half of the split code is filled with
the negative part of the code vector, only with a positive sign.
Therefore, the split_code is non-negative.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_sparse_coding.py*

2.5.4.2. Generic dictionary learning

Dictionary learning ("DictionaryLearning") is a matrix factorization
problem that amounts to finding a (usually overcomplete) dictionary
that will perform well at sparsely encoding the fitted data.

Representing data as sparse combinations of atoms from an overcomplete
dictionary is suggested to be the way the mammalian primary visual
cortex works. Consequently, dictionary learning applied on image
patches has been shown to give good results in image processing tasks
such as image completion, inpainting and denoising, as well as for

supervised recognition tasks.

Dictionary learning is an optimization problem solved by alternatively
updating the sparse code, as a solution to multiple Lasso problems,
considering the dictionary fixed, and then updating the dictionary to
best fit the sparse code.

   \frac{1}{2}              ||X-UV||_{\text{Fro}}^2+\alpha||U||_{1,1}
   \\              \text{subject to } & ||V_k||_2 <= 1 \text{ for all
   }              0 \leq k < n_{\mathrm{atoms}}

[image: pca_img2][image] [image: dict_img2][image]

values of all the entries in the matrix. After using such a procedure
to fit the dictionary, the transform is simply a sparse coding step
that shares the same implementation with all dictionary learning
objects (see Sparse coding with a precomputed dictionary).

It is also possible to constrain the dictionary and/or code to be
positive to match constraints that may be present in the data. Below
are the faces with different positivity constraints applied. Red
indicates negative values, blue indicates positive values, and white
represents zeros.

[image: dict_img_pos1][image] [image: dict_img_pos2][image][image:

dict_img_pos3][image] [image: dict_img_pos4][image]

The following image shows how a dictionary learned from 4x4 pixel
image patches extracted from part of the image of a raccoon face looks
like.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_image_denoising.py*

* "Online dictionary learning for sparse coding" J. Mairal, F. Bach,
  J. Ponce, G. Sapiro, 2009

2.5.4.3. Mini-batch dictionary learning
---------------------------------------

"MiniBatchDictionaryLearning" implements a faster, but less accurate
version of the dictionary learning algorithm that is better suited for
large datasets.

By default, "MiniBatchDictionaryLearning" divides the data into mini-
batches and optimizes in an online manner by cycling over the mini-
batches for the specified number of iterations. However, at the moment
it does not implement a stopping condition.

The estimator also implements "partial_fit", which updates the
dictionary by iterating only once over a mini-batch. This can be used
for online learning when the data is not readily available from the
start, or for when the data does not fit into the memory.

[image]

**Clustering for dictionary learning**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Note that when using dictionary learning to extract a representation

[image]

(e.g. for sparse coding) clustering can be a good proxy to learn the
dictionary. For instance the "MiniBatchKMeans" estimator is
computationally efficient and implements on-line learning with a
"partial_fit" method.

   Example: *sphx_glr_auto_examples_cluster_plot_dict_face_patches.py*

2.5.5. Factor Analysis

In unsupervised learning we only have a dataset X = \{x_1, x_2, \dots,
x_n \}. How can this dataset be described mathematically? A very
simple *continuous latent variable* model for X is

   x_i = W h_i + \mu + \epsilon

The vector h_i is called "latent" because it is unobserved. \epsilon
is considered a noise term distributed according to a Gaussian with
mean 0 and covariance \Psi (i.e. \epsilon \sim \mathcal{N}(0, \Psi)),
\mu is some arbitrary offset vector. Such a model is called
"generative" as it describes how x_i is generated from h_i. If we use
all the x_i's as columns to form a matrix \mathbf{X} and all the h_i's
as columns of a matrix \mathbf{H} then we can write (with suitably
defined \mathbf{M} and \mathbf{E}):

   \mathbf{X} = W \mathbf{H} + \mathbf{M} + \mathbf{E}

In other words, we *decomposed* matrix \mathbf{X}.

If h_i is given, the above equation automatically implies the
following probabilistic interpretation:

   p(x_i|h_i) = \mathcal{N}(Wh_i + \mu, \Psi)

For a complete probabilistic model we also need a prior distribution
for the latent variable h. The most straightforward assumption (based
on the nice properties of the Gaussian distribution) is h \sim
\mathcal{N}(0, \mathbf{I}).  This yields a Gaussian as the marginal
distribution of x:

   p(x) = \mathcal{N}(\mu, WW^T + \Psi)

Now, without any further assumptions the idea of having a latent
variable h would be superfluous -- x can be completely modelled with a
mean and a covariance. We need to impose some more specific structure
on one of these two parameters. A simple additional assumption regards
the structure of the error covariance \Psi:

* \Psi = \sigma^2 \mathbf{I}: This assumption leads to the
  probabilistic model of "PCA".

* \Psi = \mathrm{diag}(\psi_1, \psi_2, \dots, \psi_n): This model is
  called "FactorAnalysis", a classical statistical model. The matrix W
  is sometimes called the "factor loading matrix".

Both models essentially estimate a Gaussian with a low-rank covariance
matrix. Because both models are probabilistic they can be integrated
in more complex models, e.g. Mixture of Factor Analysers. One gets
very different models (e.g. "FastICA") if non-Gaussian priors on the
latent variables are assumed.

Factor analysis *can* produce similar components (the columns of its

loading matrix) to "PCA". However, one can not make any general
statements about these components (e.g. whether they are orthogonal):

[image: pca_img3][image] [image: fa_img3][image]

The main advantage for Factor Analysis over "PCA" is that it can model
the variance in every direction of the input space independently
(heteroscedastic noise):

   [image]

This allows better model selection than probabilistic PCA in the
presence of heteroscedastic noise:

   [image]

Factor Analysis is often followed by a rotation of the factors (with
the parameter *rotation*), usually to improve interpretability. For
example, Varimax rotation maximizes the sum of the variances of the
squared loadings, i.e., it tends to produce sparser factors, which are
influenced by only a few features each (the "simple structure"). See
e.g., the first example below.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_varimax_fa.py*

  n.py*

2.5.6. Independent component analysis (ICA)

Independent component analysis separates a multivariate signal into
additive subcomponents that are maximally independent. It is
implemented in scikit-learn using the "Fast ICA" algorithm. Typically,
ICA is not used for reducing dimensionality but for separating

superimposed signals. Since the ICA model does not include a noise
term, for the model to be correct, whitening must be applied. This can
be done internally using the whiten argument or manually using one of
the PCA variants.

It is classically used to separate mixed signals (a problem known as
*blind source separation*), as in the example below:

   [image]

ICA can also be used as yet another non linear decomposition that
finds components with some sparsity:

[image: pca_img4][image] [image: ica_img4][image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_decomposition_plot_ica_blind_source_separat
  ion.py*

* *sphx_glr_auto_examples_decomposition_plot_ica_vs_pca.py*

2.5.7. Non-negative matrix factorization (NMF or NNMF)

2.5.7.1. NMF with the Frobenius norm

"NMF" [1] is an alternative approach to decomposition that assumes
that the data and the components are non-negative. "NMF" can be
plugged in instead of "PCA" or its variants, in the cases where the
data matrix does not contain negative values. It finds a decomposition
of samples X into two matrices W and H of non-negative elements, by
optimizing the distance d between X and the matrix product WH. The

most widely used distance function is the squared Frobenius norm,
which is an obvious extension of the Euclidean norm to matrices:

   d_{\mathrm{Fro}}(X, Y) = \frac{1}{2} ||X - Y||_{\mathrm{Fro}}^2 =
   \frac{1}{2} \sum_{i,j} (X_{ij} - {Y}_{ij})^2

Unlike "PCA", the representation of a vector is obtained in an
additive fashion, by superimposing the components, without
subtracting. Such additive models are efficient for representing
images and text.

It has been observed in [Hoyer, 2004] [2] that, when carefully
constrained, "NMF" can produce a parts-based representation of the
dataset, resulting in interpretable models. The following example
displays 16 sparse components found by "NMF" from the images in the
Olivetti faces dataset, in comparison with the PCA eigenfaces.

[image: pca_img5][image] [image: nmf_img5][image]

The "init" attribute determines the initialization method applied,
which has a great impact on the performance of the method. "NMF"
implements the method Nonnegative Double Singular Value Decomposition.
NNDSVD [4] is based on two SVD processes, one approximating the data
matrix, the other approximating positive sections of the resulting
partial SVD factors utilizing an algebraic property of unit rank
matrices. The basic NNDSVD algorithm is better fit for sparse
factorization. Its variants NNDSVDa (in which all zeros are set equal

to the mean of all elements of the data), and NNDSVDar (in which the
zeros are set to random perturbations less than the mean of the data
divided by 100) are recommended in the dense case.

Note that the Multiplicative Update ('mu') solver cannot update zeros
present in the initialization, so it leads to poorer results when used
jointly with the basic NNDSVD algorithm which introduces a lot of
zeros; in this case, NNDSVDa or NNDSVDar should be preferred.

"NMF" can also be initialized with correctly scaled random non-
negative matrices by setting "init="random"". An integer seed or a
"RandomState" can also be passed to "random_state" to control
reproducibility.

In "NMF", L1 and L2 priors can be added to the loss function in order
to regularize the model. The L2 prior uses the Frobenius norm, while
the L1 prior uses an elementwise L1 norm. As in "ElasticNet", we
control the combination of L1 and L2 with the "l1_ratio" (\rho)
parameter, and the intensity of the regularization with the "alpha_W"
and "alpha_H" (\alpha_W and \alpha_H) parameters. The priors are
scaled by the number of samples (n\_samples) for *H* and the number of
features (n\_features) for *W* to keep their impact balanced with
respect to one another and to the data fit term as independent as

possible of the size of the training set. Then the priors terms are:

   (\alpha_W \rho ||W||_1 + \frac{\alpha_W(1-\rho)}{2}
   ||W||_{\mathrm{Fro}} ^ 2) * n\_features + (\alpha_H \rho ||H||_1 +
   \frac{\alpha_H(1-\rho)}{2} ||H||_{\mathrm{Fro}} ^ 2) * n\_samples

and the regularized objective function is:

   d_{\mathrm{Fro}}(X, WH) + (\alpha_W \rho ||W||_1 +
   \frac{\alpha_W(1-\rho)}{2} ||W||_{\mathrm{Fro}} ^ 2) * n\_features
   + (\alpha_H \rho ||H||_1 + \frac{\alpha_H(1-\rho)}{2}
   ||H||_{\mathrm{Fro}} ^ 2) * n\_samples

2.5.7.2. NMF with a beta-divergence

As described previously, the most widely used distance function is the
squared Frobenius norm, which is an obvious extension of the Euclidean
norm to matrices:

   d_{\mathrm{Fro}}(X, Y) = \frac{1}{2} ||X - Y||_{Fro}^2 =

Other distance functions can be used in NMF as, for example, the
(generalized) Kullback-Leibler (KL) divergence, also referred as
I-divergence:

   d_{KL}(X, Y) = \sum_{i,j} (X_{ij} \log(\frac{X_{ij}}{Y_{ij}}) -
   X_{ij} + Y_{ij})

Or, the Itakura-Saito (IS) divergence:

   d_{IS}(X, Y) = \sum_{i,j} (\frac{X_{ij}}{Y_{ij}} -
   \log(\frac{X_{ij}}{Y_{ij}}) - 1)

These three distances are special cases of the beta-divergence family,

with \beta = 2, 1, 0 respectively [6]. The beta-divergence are defined
by :

   d_{\beta}(X, Y) = \sum_{i,j} \frac{1}{\beta(\beta -
   1)}(X_{ij}^\beta + (\beta-1)Y_{ij}^\beta - \beta X_{ij}
   Y_{ij}^{\beta - 1})

   [image]

Note that this definition is not valid if \beta \in (0; 1), yet it can
be continuously extended to the definitions of d_{KL} and d_{IS}
respectively.

"NMF" implements two solvers, using Coordinate Descent ('cd') [5], and
Multiplicative Update ('mu') [6]. The 'mu' solver can optimize every
beta-divergence, including of course the Frobenius norm (\beta=2), the
(generalized) Kullback-Leibler divergence (\beta=1) and the Itakura-
Saito divergence (\beta=0). Note that for \beta \in (1; 2), the 'mu'
solver is significantly faster than for other values of \beta. Note
also that with a negative (or 0, i.e. 'itakura-saito') \beta, the
input matrix cannot contain zero values.

The 'cd' solver can only optimize the Frobenius norm. Due to the
underlying non-convexity of NMF, the different solvers may converge to
different minima, even when optimizing the same distance function.

NMF is best used with the "fit_transform" method, which returns the
matrix W. The matrix H is stored into the fitted model in the
"components_" attribute; the method "transform" will decompose a new

matrix X_new based on these stored components:

   >>> X = np.array([[1, 1], [2, 1], [3, 1.2], [4, 1], [5, 0.8], [6, 1]])
   >>> from sklearn.decomposition import NMF
   >>> model = NMF(n_components=2, init='random', random_state=0)
   >>> W = model.fit_transform(X)
   >>> H = model.components_
   >>> X_new = np.array([[1, 0], [1, 6.1], [1, 0], [1, 4], [3.2, 1], [0, 4]])
   >>> W_new = model.transform(X_new)

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_applications_plot_topics_extraction_with_nm
  f_lda.py*

* *sphx_glr_auto_examples_decomposition_plot_beta_divergence.py*

2.5.7.3. Mini-batch Non Negative Matrix Factorization

"MiniBatchNMF" [7] implements a faster, but less accurate version of
the non negative matrix factorization (i.e. "NMF"), better suited for

By default, "MiniBatchNMF" divides the data into mini-batches and
optimizes the NMF model in an online manner by cycling over the mini-
batches for the specified number of iterations. The "batch_size"
parameter controls the size of the batches.

In order to speed up the mini-batch algorithm it is also possible to
scale past batches, giving them less importance than newer batches.

This is done introducing a so-called forgetting factor controlled by
the "forget_factor" parameter.

The estimator also implements "partial_fit", which updates "H" by
iterating only once over a mini-batch. This can be used for online
learning when the data is not readily available from the start, or
when the data does not fit into memory.

[1] "Learning the parts of objects by non-negative matrix
    factorization" D. Lee, S. Seung, 1999

[2] "Non-negative Matrix Factorization with Sparseness Constraints" P.
    Hoyer, 2004

[4] "SVD based initialization: A head start for nonnegative matrix
    factorization" C. Boutsidis, E. Gallopoulos, 2008

[5] "Fast local algorithms for large scale nonnegative matrix and
    tensor factorizations." A. Cichocki, A. Phan, 2009

[6] >>:arxiv:`"Algorithms for nonnegative matrix factorization with
    the beta-divergence" <1010.1763>`<< C. Fevotte, J. Idier, 2011

[7] >>:arxiv:`"Online algorithms for nonnegative matrix factorization
    with the Itakura-Saito divergence" <1106.4198>`<< A. Lefevre, F.
    Bach, C. Fevotte, 2011

2.5.8. Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation is a generative probabilistic model for
collections of discrete dataset such as text corpora. It is also a

topic model that is used for discovering abstract topics from a
collection of documents.

The graphical model of LDA is a three-level generative model:

[image]

Note on notations presented in the graphical model above, which can be
found in Hoffman et al. (2013):

   * The corpus is a collection of D documents.

   * A document is a sequence of N words.

   * There are K topics in the corpus.

   * The boxes represent repeated sampling.

In the graphical model, each node is a random variable and has a role
in the generative process. A shaded node indicates an observed
variable and an unshaded node indicates a hidden (latent) variable. In
this case, words in the corpus are the only data that we observe. The
latent variables determine the random mixture of topics in the corpus
and the distribution of words in the documents. The goal of LDA is to
use the observed words to infer the hidden topic structure.

When modeling text corpora, the model assumes the following generative
process for a corpus with D documents and K topics, with K
corresponding to "n_components" in the API:

   1. For each topic k \in K, draw \beta_k \sim
      \mathrm{Dirichlet}(\eta). This provides a distribution over the
      words, i.e. the probability of a word appearing in topic k. \eta
      corresponds to "topic_word_prior".

\mathrm{Dirichlet}(\eta). This provides a distribution over the

   2. For each document d \in D, draw the topic proportions \theta_d
      \sim \mathrm{Dirichlet}(\alpha). \alpha corresponds to
      "doc_topic_prior".

   3. For each word i in document d:

      1. Draw the topic assignment z_{di} \sim \mathrm{Multinomial}
         (\theta_d)

      2. Draw the observed word w_{ij} \sim \mathrm{Multinomial}
         (\beta_{z_{di}})

For parameter estimation, the posterior distribution is:

   p(z, \theta, \beta |w, \alpha, \eta) =   \frac{p(z, \theta,
   \beta|\alpha, \eta)}{p(w|\alpha, \eta)}

Since the posterior is intractable, variational Bayesian method uses a
simpler distribution q(z,\theta,\beta | \lambda, \phi, \gamma) to
approximate it, and those variational parameters \lambda, \phi, \gamma
are optimized to maximize the Evidence Lower Bound (ELBO):

   \log\: P(w | \alpha, \eta) \geq L(w,\phi,\gamma,\lambda)
   \overset{\triangle}{=}
   E_{q}[\log\:p(w,z,\theta,\beta|\alpha,\eta)] - E_{q}[\log\:q(z,
   \theta, \beta)]

Maximizing ELBO is equivalent to minimizing the Kullback-Leibler(KL)
divergence between q(z,\theta,\beta) and the true posterior p(z,
\theta, \beta |w, \alpha, \eta).

"LatentDirichletAllocation" implements the online variational Bayes
algorithm and supports both online and batch update methods. While the
batch method updates variational variables after each full pass

through the data, the online method updates variational variables from
mini-batch data points.

Note:

  Although the online method is guaranteed to converge to a local
  optimum point, the quality of the optimum point and the speed of
  convergence may depend on mini-batch size and attributes related to
  learning rate setting.

When "LatentDirichletAllocation" is applied on a "document-term"
matrix, the matrix will be decomposed into a "topic-term" matrix and a
"document-topic" matrix. While "topic-term" matrix is stored as
"components_" in the model, "document-topic" matrix can be calculated
from "transform" method.

"LatentDirichletAllocation" also implements "partial_fit" method. This
is used when data can be fetched sequentially.

Examples:
^^^^^^^^^

  f_lda.py*

* "Latent Dirichlet Allocation" D. Blei, A. Ng, M. Jordan, 2003

* "Online Learning for Latent Dirichlet Allocation” M. Hoffman, D.
  Blei, F. Bach, 2010

* "Stochastic Variational Inference" M. Hoffman, D. Blei, C. Wang, J.
  Paisley, 2013

* "The varimax criterion for analytic rotation in factor analysis" H.
  F. Kaiser, 1958

See also Dimensionality reduction for dimensionality reduction with
Neighborhood Components Analysis.

1.14. Semi-supervised learning

Semi-supervised learning is a situation in which in your training data
some of the samples are not labeled. The semi-supervised estimators in
"sklearn.semi_supervised" are able to make use of this additional
unlabeled data to better capture the shape of the underlying data
distribution and generalize better to new samples. These algorithms
can perform well when we have a very small amount of labeled points
and a large amount of unlabeled points.

Unlabeled entries in *y*

It is important to assign an identifier to unlabeled points along with
the labeled data when training the model with the "fit" method. The
identifier that this implementation uses is the integer value -1. Note
that for string labels, the dtype of *y* should be object so that it
can contain both strings and integers.

Note:

  Semi-supervised algorithms need to make assumptions about the
  distribution of the dataset in order to achieve performance gains.
  See here for more details.

1.14.1. Self Training

This self-training implementation is based on Yarowsky's [1]
algorithm. Using this algorithm, a given supervised classifier can
function as a semi-supervised classifier, allowing it to learn from
unlabeled data.

"SelfTrainingClassifier" can be called with any classifier that
implements *predict_proba*, passed as the parameter *base_classifier*.
In each iteration, the *base_classifier* predicts labels for the

unlabeled samples and adds a subset of these labels to the labeled
dataset.

The choice of this subset is determined by the selection criterion.
This selection can be done using a *threshold* on the prediction
probabilities, or by choosing the *k_best* samples according to the
prediction probabilities.

The labels used for the final fit as well as the iteration in which
each sample was labeled are available as attributes. The optional
*max_iter* parameter specifies how many times the loop is executed at
most.

The *max_iter* parameter may be set to *None*, causing the algorithm
to iterate until all samples have labels or no new samples are
selected in that iteration.

Note:

  When using the self-training classifier, the calibration of the
  classifier is important.

Examples
^^^^^^^^

* *sphx_glr_auto_examples_semi_supervised_plot_self_training_varying_
  threshold.py*

* *sphx_glr_auto_examples_semi_supervised_plot_semi_supervised_versus
  _svm_iris.py*

[1] >>:doi:`"Unsupervised word sense disambiguation rivaling
    supervised methods" <10.3115/981658.981684>`<< David Yarowsky,
    Proceedings of the 33rd annual meeting on Association for
    Computational Linguistics (ACL '95). Association for Computational
    Linguistics, Stroudsburg, PA, USA, 189-196.

1.14.2. Label Propagation

Computational Linguistics (ACL '95). Association for Computational

Label propagation denotes a few variations of semi-supervised graph
inference algorithms.

A few features available in this model:
   * Used for classification tasks

   * Kernel methods to project data into alternate dimensional spaces

*scikit-learn* provides two label propagation models:
"LabelPropagation" and "LabelSpreading". Both work by constructing a
similarity graph over all items in the input dataset.

   [image]**An illustration of label-propagation:** *the structure of
   unlabeled observations is consistent with the class structure, and
   thus the class label can be propagated to the unlabeled
   observations of the training set.*

"LabelPropagation" and "LabelSpreading" differ in modifications to the
similarity matrix that graph and the clamping effect on the label
distributions. Clamping allows the algorithm to change the weight of
the true ground labeled data to some degree. The "LabelPropagation"
algorithm performs hard clamping of input labels, which means
\alpha=0. This clamping factor can be relaxed, to say \alpha=0.2,
which means that we will always retain 80 percent of our original
label distribution, but the algorithm gets to change its confidence of
the distribution within 20 percent.

"LabelPropagation" uses the raw similarity matrix constructed from the

data with no modifications. In contrast, "LabelSpreading" minimizes a
loss function that has regularization properties, as such it is often
more robust to noise. The algorithm iterates on a modified version of
the original graph and normalizes the edge weights by computing the
normalized graph Laplacian matrix. This procedure is also used in
Spectral clustering.

Label propagation models have two built-in kernel methods. Choice of
kernel effects both scalability and performance of the algorithms. The
following are available:

   * rbf (\exp(-\gamma |x-y|^2), \gamma > 0). \gamma is specified by
     keyword gamma.

   * knn (1[x' \in kNN(x)]). k is specified by keyword n_neighbors.

The RBF kernel will produce a fully connected graph which is
represented in memory by a dense matrix. This matrix may be very large
and combined with the cost of performing a full matrix multiplication
calculation for each iteration of the algorithm can lead to
prohibitively long running times. On the other hand, the KNN kernel
will produce a much more memory-friendly sparse matrix which can
drastically reduce running times.

Examples
^^^^^^^^

* *sphx_glr_auto_examples_semi_supervised_plot_label_propagation_stru
  cture.py*

Examples
^^^^^^^^

  cture.py*

* *sphx_glr_auto_examples_semi_supervised_plot_label_propagation_digi
  ts.py*

  ts_active_learning.py*

[2] Yoshua Bengio, Olivier Delalleau, Nicolas Le Roux. In Semi-
Supervised Learning (2006), pp. 193-216

[3] Olivier Delalleau, Yoshua Bengio, Nicolas Le Roux. Efficient Non-
Parametric Function Induction in Semi-Supervised Learning. AISTAT 2005
https://www.gatsby.ucl.ac.uk/aistats/fullpapers/204.pdf

1.11. Ensemble methods

The goal of **ensemble methods** is to combine the predictions of
several base estimators built with a given learning algorithm in order
to improve generalizability / robustness over a single estimator.

Two families of ensemble methods are usually distinguished:

* In **averaging methods**, the driving principle is to build several
  estimators independently and then to average their predictions. On
  average, the combined estimator is usually better than any of the
  single base estimator because its variance is reduced.

  **Examples:** Bagging methods, Forests of randomized trees, ...

* By contrast, in **boosting methods**, base estimators are built
  sequentially and one tries to reduce the bias of the combined
  estimator. The motivation is to combine several weak models to
  produce a powerful ensemble.

  **Examples:** AdaBoost, Gradient Tree Boosting, ...

1.11.1. Bagging meta-estimator

In ensemble algorithms, bagging methods form a class of algorithms
which build several instances of a black-box estimator on random
subsets of the original training set and then aggregate their
individual predictions to form a final prediction. These methods are
used as a way to reduce the variance of a base estimator (e.g., a
decision tree), by introducing randomization into its construction
procedure and then making an ensemble out of it. In many cases,
bagging methods constitute a very simple way to improve with respect

to a single model, without making it necessary to adapt the underlying
base algorithm. As they provide a way to reduce overfitting, bagging
methods work best with strong and complex models (e.g., fully
developed decision trees), in contrast with boosting methods which
usually work best with weak models (e.g., shallow decision trees).

Bagging methods come in many flavours but mostly differ from each
other by the way they draw random subsets of the training set:

   * When random subsets of the dataset are drawn as random subsets of
     the samples, then this algorithm is known as Pasting [B1999].

   * When samples are drawn with replacement, then the method is known
     as Bagging [B1996].

     the features, then the method is known as Random Subspaces
     [H1998].

   * Finally, when base estimators are built on subsets of both
     samples and features, then the method is known as Random Patches
     [LG2012].

In scikit-learn, bagging methods are offered as a unified
"BaggingClassifier" meta-estimator  (resp. "BaggingRegressor"), taking
as input a user-specified estimator along with parameters specifying
the strategy to draw random subsets. In particular, "max_samples" and

"max_features" control the size of the subsets (in terms of samples
and features), while "bootstrap" and "bootstrap_features" control
whether samples and features are drawn with or without replacement.
When using a subset of the available samples the generalization
accuracy can be estimated with the out-of-bag samples by setting
"oob_score=True". As an example, the snippet below illustrates how to
instantiate a bagging ensemble of "KNeighborsClassifier" estimators,
each built on random subsets of 50% of the samples and 50% of the
features.

>>> from sklearn.ensemble import BaggingClassifier
>>> from sklearn.neighbors import KNeighborsClassifier
>>> bagging = BaggingClassifier(KNeighborsClassifier(),
...                             max_samples=0.5, max_features=0.5)

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_bias_variance.py*

[B1999] L. Breiman, "Pasting small votes for classification in large
        databases and on-line", Machine Learning, 36(1), 85-103, 1999.

[B1996] L. Breiman, "Bagging predictors", Machine Learning, 24(2),
        123-140, 1996.

[H1998] T. Ho, "The random subspace method for constructing decision
        forests", Pattern Analysis and Machine Intelligence, 20(8),
        832-844, 1998.

[LG2012] G. Louppe and P. Geurts, "Ensembles on Random Patches",

forests", Pattern Analysis and Machine Intelligence, 20(8),

         Machine Learning and Knowledge Discovery in Databases,
         346-361, 2012.

1.11.2. Forests of randomized trees

The "sklearn.ensemble" module includes two averaging algorithms based
on randomized decision trees: the RandomForest algorithm and the
Extra-Trees method. Both algorithms are perturb-and-combine techniques
[B1998] specifically designed for trees. This means a diverse set of
classifiers is created by introducing randomness in the classifier
construction.  The prediction of the ensemble is given as the averaged
prediction of the individual classifiers.

As other classifiers, forest classifiers have to be fitted with two
arrays: a sparse or dense array X of shape "(n_samples, n_features)"
holding the training samples, and an array Y of shape "(n_samples,)"
holding the target values (class labels) for the training samples:

   >>> X = [[0, 0], [1, 1]]
   >>> Y = [0, 1]
   >>> clf = RandomForestClassifier(n_estimators=10)
   >>> clf = clf.fit(X, Y)

Like decision trees, forests of trees also extend to multi-output
problems  (if Y is an array of shape "(n_samples, n_outputs)").

1.11.2.1. Random Forests

In random forests (see "RandomForestClassifier" and

"RandomForestRegressor" classes), each tree in the ensemble is built
from a sample drawn with replacement (i.e., a bootstrap sample) from
the training set.

Furthermore, when splitting each node during the construction of a
tree, the best split is found either from all input features or a
random subset of size "max_features". (See the parameter tuning
guidelines for more details).

The purpose of these two sources of randomness is to decrease the
variance of the forest estimator. Indeed, individual decision trees
typically exhibit high variance and tend to overfit. The injected
randomness in forests yield decision trees with somewhat decoupled
prediction errors. By taking an average of those predictions, some
errors can cancel out. Random forests achieve a reduced variance by
combining diverse trees, sometimes at the cost of a slight increase in
bias. In practice the variance reduction is often significant hence
yielding an overall better model.

In contrast to the original publication [B2001], the scikit-learn
implementation combines classifiers by averaging their probabilistic
prediction, instead of letting each classifier vote for a single
class.

1.11.2.2. Extremely Randomized Trees

In extremely randomized trees (see "ExtraTreesClassifier" and

class.

"ExtraTreesRegressor" classes), randomness goes one step further in
the way splits are computed. As in random forests, a random subset of
candidate features is used, but instead of looking for the most
discriminative thresholds, thresholds are drawn at random for each
candidate feature and the best of these randomly-generated thresholds
is picked as the splitting rule. This usually allows to reduce the
variance of the model a bit more, at the expense of a slightly greater
increase in bias:

   >>> from sklearn.datasets import make_blobs
   >>> from sklearn.ensemble import ExtraTreesClassifier
   >>> from sklearn.tree import DecisionTreeClassifier

   >>> X, y = make_blobs(n_samples=10000, n_features=10, centers=100,
   ...     random_state=0)

   >>> clf = DecisionTreeClassifier(max_depth=None, min_samples_split=2,
   >>> scores = cross_val_score(clf, X, y, cv=5)
   >>> scores.mean()
   0.98...

   >>> clf = RandomForestClassifier(n_estimators=10, max_depth=None,
   ...     min_samples_split=2, random_state=0)
   0.999...

   >>> clf = ExtraTreesClassifier(n_estimators=10, max_depth=None,

>>> scores = cross_val_score(clf, X, y, cv=5)
   0.999...

   >>> scores.mean() > 0.999
   True

   [image]

1.11.2.3. Parameters

The main parameters to adjust when using these methods is
"n_estimators" and "max_features". The former is the number of trees
in the forest. The larger the better, but also the longer it will take
to compute. In addition, note that results will stop getting
significantly better beyond a critical number of trees. The latter is
the size of the random subsets of features to consider when splitting
a node. The lower the greater the reduction of variance, but also the
greater the increase in bias. Empirical good default values are
"max_features=1.0" or equivalently "max_features=None" (always
considering all features instead of a random subset) for regression
problems, and "max_features="sqrt"" (using a random subset of size
"sqrt(n_features)") for classification tasks (where "n_features" is
the number of features in the data). The default value of
"max_features=1.0" is equivalent to bagged trees and more randomness
can be achieved by setting smaller values (e.g. 0.3 is a typical
default in the literature). Good results are often achieved when
setting "max_depth=None" in combination with "min_samples_split=2"
(i.e., when fully developing the trees). Bear in mind though that

these values are usually not optimal, and might result in models that
consume a lot of RAM. The best parameter values should always be
cross-validated. In addition, note that in random forests, bootstrap
samples are used by default ("bootstrap=True") while the default
strategy for extra-trees is to use the whole dataset
("bootstrap=False"). When using bootstrap sampling the generalization
error can be estimated on the left out or out-of-bag samples. This can
be enabled by setting "oob_score=True".

Note:

  The size of the model with the default parameters is O( M * N * log
  (N) ), where M is the number of trees and N is the number of
  samples. In order to reduce the size of the model, you can change
  these parameters: "min_samples_split", "max_leaf_nodes", "max_depth"
  and "min_samples_leaf".

1.11.2.4. Parallelization

Finally, this module also features the parallel construction of the
trees and the parallel computation of the predictions through the
"n_jobs" parameter. If "n_jobs=k" then computations are partitioned
into "k" jobs, and run on "k" cores of the machine. If "n_jobs=-1"
then all cores available on the machine are used. Note that because of
inter-process communication overhead, the speedup might not be linear

(i.e., using "k" jobs will unfortunately not be "k" times as fast).
Significant speedup can still be achieved though when building a large
number of trees, or when building a single tree requires a fair amount
of time (e.g., on large datasets).

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_forest_iris.py*

* *sphx_glr_auto_examples_ensemble_plot_forest_importances_faces.py*

* *sphx_glr_auto_examples_miscellaneous_plot_multioutput_face_complet
  ion.py*

[B2001] 12. Breiman, "Random Forests", Machine Learning, 45(1), 5-32,
            2001.

[B1998] 12. Breiman, "Arcing Classifiers", Annals of Statistics 1998.

* P. Geurts, D. Ernst., and L. Wehenkel, "Extremely randomized trees",
  Machine Learning, 63(1), 3-42, 2006.

1.11.2.5. Feature importance evaluation

The relative rank (i.e. depth) of a feature used as a decision node in
a tree can be used to assess the relative importance of that feature
with respect to the predictability of the target variable. Features
used at the top of the tree contribute to the final prediction
decision of a larger fraction of the input samples. The **expected
fraction of the samples** they contribute to can thus be used as an
estimate of the **relative importance of the features**. In scikit-
learn, the fraction of samples a feature contributes to is combined

with the decrease in impurity from splitting them to create a
normalized estimate of the predictive power of that feature.

By **averaging** the estimates of predictive ability over several
randomized trees one can **reduce the variance** of such an estimate
and use it for feature selection. This is known as the mean decrease
in impurity, or MDI. Refer to [L2014] for more information on MDI and
feature importance evaluation with Random Forests.

Warning:

  The impurity-based feature importances computed on tree-based models
  suffer from two flaws that can lead to misleading conclusions. First
  they are computed on statistics derived from the training dataset
  and therefore **do not necessarily inform us on which features are
  most important to make good predictions on held-out dataset**.
  Secondly, **they favor high cardinality features**, that is features
  with many unique values. Permutation feature importance is an
  alternative to impurity-based feature importance that does not
  suffer from these flaws. These two methods of obtaining feature
  importance are explored in:
  *sphx_glr_auto_examples_inspection_plot_permutation_importance.py*.

The following example shows a color-coded representation of the
relative importances of each individual pixel for a face recognition
task using a "ExtraTreesClassifier" model.

   [image]

   [image]

In practice those estimates are stored as an attribute named
"feature_importances_" on the fitted model. This is an array with
shape "(n_features,)" whose values are positive and sum to 1.0. The
higher the value, the more important is the contribution of the
matching feature to the prediction function.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_forest_importances.py*

[L2014] G. Louppe, >>:arxiv:`"Understanding Random Forests: From
        Theory to Practice" <1407.7502>`<<, PhD Thesis, U. of Liege,
        2014.

1.11.2.6. Totally Random Trees Embedding

"RandomTreesEmbedding" implements an unsupervised transformation of
the data.  Using a forest of completely random trees,
"RandomTreesEmbedding" encodes the data by the indices of the leaves a
data point ends up in.  This index is then encoded in a one-of-K
manner, leading to a high dimensional, sparse binary coding. This
coding can be computed very efficiently and can then be used as a
basis for other learning tasks. The size and sparsity of the code can
be influenced by choosing the number of trees and the maximum depth
per tree. For each tree in the ensemble, the coding contains one entry

of one. The size of the coding is at most "n_estimators * 2 **
max_depth", the maximum number of leaves in the forest.

As neighboring data points are more likely to lie within the same leaf
of a tree, the transformation performs an implicit, non-parametric
density estimation.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_random_forest_embedding.py*

* *sphx_glr_auto_examples_manifold_plot_lle_digits.py* compares non-
  linear dimensionality reduction techniques on handwritten digits.

* *sphx_glr_auto_examples_ensemble_plot_feature_transformation.py*
  compares supervised and unsupervised tree based feature
  transformations.

See also:

  Manifold learning techniques can also be useful to derive non-linear
  representations of feature space, also these approaches focus also
  on dimensionality reduction.

1.11.3. AdaBoost

The module "sklearn.ensemble" includes the popular boosting algorithm
AdaBoost, introduced in 1995 by Freund and Schapire [FS1995].

The core principle of AdaBoost is to fit a sequence of weak learners
(i.e., models that are only slightly better than random guessing, such
as small decision trees) on repeatedly modified versions of the data.
The predictions from all of them are then combined through a weighted
majority vote (or sum) to produce the final prediction. The data

modifications at each so-called boosting iteration consist of applying
weights w_1, w_2, ..., w_N to each of the training samples. Initially,
those weights are all set to w_i = 1/N, so that the first step simply
trains a weak learner on the original data. For each successive
iteration, the sample weights are individually modified and the
learning algorithm is reapplied to the reweighted data. At a given
step, those training examples that were incorrectly predicted by the
boosted model induced at the previous step have their weights
increased, whereas the weights are decreased for those that were
predicted correctly. As iterations proceed, examples that are
difficult to predict receive ever-increasing influence. Each
subsequent weak learner is thereby forced to concentrate on the
examples that are missed by the previous ones in the sequence [HTF].

   [image]

AdaBoost can be used both for classification and regression problems:

   * For multi-class classification, "AdaBoostClassifier" implements
     AdaBoost-SAMME and AdaBoost-SAMME.R [ZZRH2009].

   * For regression, "AdaBoostRegressor" implements AdaBoost.R2
     [D1997].

1.11.3.1. Usage

The following example shows how to fit an AdaBoost classifier with 100
weak learners:

   >>> from sklearn.ensemble import AdaBoostClassifier

   >>> clf = AdaBoostClassifier(n_estimators=100)
   0.9...

The number of weak learners is controlled by the parameter
"n_estimators". The "learning_rate" parameter controls the
contribution of the weak learners in the final combination. By
default, weak learners are decision stumps. Different weak learners
can be specified through the "estimator" parameter. The main
parameters to tune to obtain good results are "n_estimators" and the
complexity of the base estimators (e.g., its depth "max_depth" or
minimum required number of samples to consider a split
"min_samples_split").

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_adaboost_hastie_10_2.py*
  compares the classification error of a decision stump, decision
  tree, and a boosted decision stump using AdaBoost-SAMME and
  AdaBoost-SAMME.R.

* *sphx_glr_auto_examples_ensemble_plot_adaboost_multiclass.py* shows
  the performance of AdaBoost-SAMME and AdaBoost-SAMME.R on a multi-
  class problem.

* *sphx_glr_auto_examples_ensemble_plot_adaboost_twoclass.py* shows
  the decision boundary and decision function values for a non-
  linearly separable two-class problem using AdaBoost-SAMME.

* *sphx_glr_auto_examples_ensemble_plot_adaboost_regression.py*
  demonstrates regression with the AdaBoost.R2 algorithm.

[FS1995] Y. Freund, and R. Schapire, "A Decision-Theoretic
         Generalization of On-Line Learning and an Application to
         Boosting", 1997.

[ZZRH2009] J. Zhu, H. Zou, S. Rosset, T. Hastie. "Multi-class
           AdaBoost", 2009.

[D1997] 8. Drucker. "Improving Regressors using Boosting Techniques",
           1997.

[HTF] T. Hastie, R. Tibshirani and J. Friedman, "Elements of
      Statistical Learning Ed. 2", Springer, 2009.

1.11.4. Gradient Tree Boosting

Gradient Tree Boosting or Gradient Boosted Decision Trees (GBDT) is a
generalization of boosting to arbitrary differentiable loss functions,
see the seminal work of [Friedman2001]. GBDT is an accurate and
effective off-the-shelf procedure that can be used for both regression
and classification problems in a variety of areas including Web search
ranking and ecology.

The module "sklearn.ensemble" provides methods for both classification
and regression via gradient boosted decision trees.

Note:

  Scikit-learn 0.21 introduces two new implementations of gradient
  boosting trees, namely "HistGradientBoostingClassifier" and
  "HistGradientBoostingRegressor", inspired by LightGBM (See

Note:

  [LightGBM]).These histogram-based estimators can be **orders of
  magnitude faster** than "GradientBoostingClassifier" and
  "GradientBoostingRegressor" when the number of samples is larger
  than tens of thousands of samples.They also have built-in support
  for missing values, which avoids the need for an imputer.These
  estimators are described in more detail below in Histogram-Based
  Gradient Boosting.The following guide focuses on
  "GradientBoostingClassifier" and "GradientBoostingRegressor", which
  might be preferred for small sample sizes since binning may lead to
  split points that are too approximate in this setting.

The usage and the parameters of "GradientBoostingClassifier" and
"GradientBoostingRegressor" are described below. The 2 most important
parameters of these estimators are *n_estimators* and *learning_rate*.

1.11.4.1. Classification

"GradientBoostingClassifier" supports both binary and multi-class
classification. The following example shows how to fit a gradient
boosting classifier with 100 decision stumps as weak learners:

   >>> from sklearn.datasets import make_hastie_10_2

   >>> X, y = make_hastie_10_2(random_state=0)
   >>> X_train, X_test = X[:2000], X[2000:]

>>> from sklearn.ensemble import GradientBoostingClassifier

   >>> y_train, y_test = y[:2000], y[2000:]

   >>> clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,
   ...     max_depth=1, random_state=0).fit(X_train, y_train)
   >>> clf.score(X_test, y_test)
   0.913...

The number of weak learners (i.e. regression trees) is controlled by
the parameter "n_estimators"; The size of each tree can be controlled
either by setting the tree depth via "max_depth" or by setting the
number of leaf nodes via "max_leaf_nodes". The "learning_rate" is a
hyper-parameter in the range (0.0, 1.0] that controls overfitting via
shrinkage .

Note:

  Classification with more than 2 classes requires the induction of
  "n_classes" regression trees at each iteration, thus, the total
  number of induced trees equals "n_classes * n_estimators". For
  datasets with a large number of classes we strongly recommend to use
  "HistGradientBoostingClassifier" as an alternative to
  "GradientBoostingClassifier" .

1.11.4.2. Regression

"GradientBoostingRegressor" supports a number of different loss
functions for regression which can be specified via the argument
"loss"; the default loss function for regression is squared error
("'squared_error'").

   >>> from sklearn.datasets import make_friedman1

   >>> from sklearn.ensemble import GradientBoostingRegressor

   >>> X, y = make_friedman1(n_samples=1200, random_state=0, noise=1.0)
   >>> X_train, X_test = X[:200], X[200:]
   >>> y_train, y_test = y[:200], y[200:]
   >>> est = GradientBoostingRegressor(
   ...     n_estimators=100, learning_rate=0.1, max_depth=1, random_state=0,
   ...     loss='squared_error'
   ... ).fit(X_train, y_train)
   >>> mean_squared_error(y_test, est.predict(X_test))
   5.00...

The figure below shows the results of applying
"GradientBoostingRegressor" with least squares loss and 500 base
learners to the diabetes dataset ("sklearn.datasets.load_diabetes()").
The plot shows the train and test error at each iteration. The train
error at each iteration is stored in the "train_score_" attribute of
the gradient boosting model. The test error at each iterations can be
obtained via the "staged_predict()" method which returns a generator
that yields the predictions at each stage. Plots like these can be
used to determine the optimal number of trees (i.e. "n_estimators") by
early stopping.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_regression.
  py*

* *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_oob.py*

1.11.4.3. Fitting additional weak-learners

py*

Both "GradientBoostingRegressor" and "GradientBoostingClassifier"
support "warm_start=True" which allows you to add more estimators to
an already fitted model.

   >>> _ = est.set_params(n_estimators=200, warm_start=True)  # set warm_start and new nr of trees
   >>> _ = est.fit(X_train, y_train) # fit additional 100 trees to est
   3.84...

1.11.4.4. Controlling the tree size

The size of the regression tree base learners defines the level of
variable interactions that can be captured by the gradient boosting
model. In general, a tree of depth "h" can capture interactions of
order "h" . There are two ways in which the size of the individual
regression trees can be controlled.

If you specify "max_depth=h" then complete binary trees of depth "h"
will be grown. Such trees will have (at most) "2**h" leaf nodes and
"2**h - 1" split nodes.

Alternatively, you can control the tree size by specifying the number
of leaf nodes via the parameter "max_leaf_nodes". In this case, trees
will be grown using best-first search where nodes with the highest
improvement in impurity will be expanded first. A tree with
"max_leaf_nodes=k" has "k - 1" split nodes and thus can model
interactions of up to order "max_leaf_nodes - 1" .

We found that "max_leaf_nodes=k" gives comparable results to
"max_depth=k-1" but is significantly faster to train at the expense of
a slightly higher training error. The parameter "max_leaf_nodes"
corresponds to the variable "J" in the chapter on gradient boosting in
[Friedman2001] and is related to the parameter "interaction.depth" in
R's gbm package where "max_leaf_nodes == interaction.depth + 1" .

1.11.4.5. Mathematical formulation

We first present GBRT for regression, and then detail the
classification case.

1.11.4.5.1. Regression

GBRT regressors are additive models whose prediction \hat{y}_i for a
given input x_i is of the following form:

      \hat{y}_i = F_M(x_i) = \sum_{m=1}^{M} h_m(x_i)

where the h_m are estimators called *weak learners* in the context of
boosting. Gradient Tree Boosting uses decision tree regressors of
fixed size as weak learners. The constant M corresponds to the
*n_estimators* parameter.

Similar to other boosting algorithms, a GBRT is built in a greedy
fashion:

      F_m(x) = F_{m-1}(x) + h_m(x),

where the newly added tree h_m is fitted in order to minimize a sum of
losses L_m, given the previous ensemble F_{m-1}:

      h_m =  \arg\min_{h} L_m = \arg\min_{h} \sum_{i=1}^{n} l(y_i,
      F_{m-1}(x_i) + h(x_i)),

where l(y_i, F(x_i)) is defined by the *loss* parameter, detailed in
the next section.

By default, the initial model F_{0} is chosen as the constant that
minimizes the loss: for a least-squares loss, this is the empirical
mean of the target values. The initial model can also be specified via
the "init" argument.

Using a first-order Taylor approximation, the value of l can be
approximated as follows:

      l(y_i, F_{m-1}(x_i) + h_m(x_i)) \approx l(y_i, F_{m-1}(x_i)) +
      h_m(x_i) \left[ \frac{\partial l(y_i, F(x_i))}{\partial F(x_i)}
      \right]_{F=F_{m - 1}}.

Note:

  Briefly, a first-order Taylor approximation says that l(z) \approx
  l(a) + (z - a) \frac{\partial l}{\partial z}(a). Here, z corresponds
  to F_{m - 1}(x_i) + h_m(x_i), and a corresponds to F_{m-1}(x_i)

The quantity \left[ \frac{\partial l(y_i, F(x_i))}{\partial F(x_i)}
\right]_{F=F_{m - 1}} is the derivative of the loss with respect to
its second parameter, evaluated at F_{m-1}(x). It is easy to compute
for any given F_{m - 1}(x_i) in a closed form since the loss is
differentiable. We will denote it by g_i.

Removing the constant terms, we have:

      h_m \approx \arg\min_{h} \sum_{i=1}^{n} h(x_i) g_i

This is minimized if h(x_i) is fitted to predict a value that is
proportional to the negative gradient -g_i. Therefore, at each

h_m \approx \arg\min_{h} \sum_{i=1}^{n} h(x_i) g_i

iteration, **the estimator** h_m **is fitted to predict the negative
gradients of the samples**. The gradients are updated at each
iteration. This can be considered as some kind of gradient descent in
a functional space.

Note:

  For some losses, e.g. "'absolute_error'" where the gradients are \pm
  1, the values predicted by a fitted h_m are not accurate enough: the
  tree can only output integer values. As a result, the leaves values
  of the tree h_m are modified once the tree is fitted, such that the
  leaves values minimize the loss L_m. The update is loss-dependent:
  for the absolute error loss, the value of a leaf is updated to the
  median of the samples in that leaf.

1.11.4.5.2. Classification
~~~~~~~~~~~~~~~~~~~~~~~~~~

Gradient boosting for classification is very similar to the regression
case. However, the sum of the trees F_M(x_i) = \sum_m h_m(x_i) is not
homogeneous to a prediction: it cannot be a class, since the trees
predict continuous values.

The mapping from the value F_M(x_i) to a class or a probability is
loss-dependent. For the log-loss, the probability that x_i belongs to
the positive class is modeled as p(y_i = 1 | x_i) = \sigma(F_M(x_i))
where \sigma is the sigmoid or expit function.

For multiclass classification, K trees (for K classes) are built at

each of the M iterations. The probability that x_i belongs to class k
is modeled as a softmax of the F_{M,k}(x_i) values.

Note that even for a classification task, the h_m sub-estimator is
still a regressor, not a classifier. This is because the sub-
estimators are trained to predict (negative) *gradients*, which are
always continuous quantities.

1.11.4.6. Loss Functions

The following loss functions are supported and can be specified using
the parameter "loss":

     * Squared error ("'squared_error'"): The natural choice for
       regression due to its superior computational properties. The
       initial model is given by the mean of the target values.

     * Absolute error ("'absolute_error'"): A robust loss function for
       regression. The initial model is given by the median of the
       target values.

     * Huber ("'huber'"): Another robust loss function that combines
       least squares and least absolute deviation; use "alpha" to
       control the sensitivity with regards to outliers (see
       [Friedman2001] for more details).

     * Quantile ("'quantile'"): A loss function for quantile
       regression. Use "0 < alpha < 1" to specify the quantile. This
       loss function can be used to create prediction intervals (see

* Quantile ("'quantile'"): A loss function for quantile
       loss function can be used to create prediction intervals (see 
       *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_quanti
       le.py*).

     * Binary log-loss ("'log-loss'"): The binomial negative log-
       likelihood loss function for binary classification. It provides
       probability estimates.  The initial model is given by the log
       odds-ratio.

     * Multi-class log-loss ("'log-loss'"): The multinomial negative
       log-likelihood loss function for multi-class classification
       with "n_classes" mutually exclusive classes. It provides
       probability estimates.  The initial model is given by the prior
       probability of each class. At each iteration "n_classes"
       regression trees have to be constructed which makes GBRT rather
       inefficient for data sets with a large number of classes.

     * Exponential loss ("'exponential'"): The same loss function as
       "AdaBoostClassifier". Less robust to mislabeled examples than
       "'log-loss'"; can only be used for binary classification.

1.11.4.7. Shrinkage via learning rate

[Friedman2001] proposed a simple regularization strategy that scales
the contribution of each weak learner by a constant factor \nu:

   F_m(x) = F_{m-1}(x) + \nu h_m(x)

The parameter \nu is also called the **learning rate** because it

scales the step length the gradient descent procedure; it can be set
via the "learning_rate" parameter.

The parameter "learning_rate" strongly interacts with the parameter
"n_estimators", the number of weak learners to fit. Smaller values of
"learning_rate" require larger numbers of weak learners to maintain a
constant training error. Empirical evidence suggests that small values
of "learning_rate" favor better test error. [HTF] recommend to set the
learning rate to a small constant (e.g. "learning_rate <= 0.1") and
choose "n_estimators" by early stopping. For a more detailed
discussion of the interaction between "learning_rate" and
"n_estimators" see [R2007].

1.11.4.8. Subsampling

[Friedman2002] proposed stochastic gradient boosting, which combines
gradient boosting with bootstrap averaging (bagging). At each
iteration the base classifier is trained on a fraction "subsample" of
the available training data. The subsample is drawn without
replacement. A typical value of "subsample" is 0.5.

The figure below illustrates the effect of shrinkage and subsampling
on the goodness-of-fit of the model. We can clearly see that shrinkage
outperforms no-shrinkage. Subsampling with shrinkage can further
increase the accuracy of the model. Subsampling without shrinkage, on
the other hand, does poorly.

   [image]

   [image]

Another strategy to reduce the variance is by subsampling the features
analogous to the random splits in "RandomForestClassifier". The number
of subsampled features can be controlled via the "max_features"

Note:

  Using a small "max_features" value can significantly decrease the
  runtime.

Stochastic gradient boosting allows to compute out-of-bag estimates of
the test deviance by computing the improvement in deviance on the
examples that are not included in the bootstrap sample (i.e. the out-
of-bag examples). The improvements are stored in the attribute
"oob_improvement_". "oob_improvement_[i]" holds the improvement in
terms of the loss on the OOB samples if you add the i-th stage to the
current predictions. Out-of-bag estimates can be used for model
selection, for example to determine the optimal number of iterations.
OOB estimates are usually very pessimistic thus we recommend to use
cross-validation instead and only use OOB if cross-validation is too
time consuming.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_regularizat
  ion.py*

* *sphx_glr_auto_examples_ensemble_plot_ensemble_oob.py*

1.11.4.9. Interpretation with feature importance

Individual decision trees can be interpreted easily by simply
visualizing the tree structure. Gradient boosting models, however,
comprise hundreds of regression trees thus they cannot be easily
interpreted by visual inspection of the individual trees. Fortunately,
a number of techniques have been proposed to summarize and interpret
gradient boosting models.

Often features do not contribute equally to predict the target
response; in many situations the majority of the features are in fact
irrelevant. When interpreting a model, the first question usually is:
what are those important features and how do they contributing in
predicting the target response?

Individual decision trees intrinsically perform feature selection by
selecting appropriate split points. This information can be used to
measure the importance of each feature; the basic idea is: the more
often a feature is used in the split points of a tree the more
important that feature is. This notion of importance can be extended
to decision tree ensembles by simply averaging the impurity-based
feature importance of each tree (see Feature importance evaluation for
more details).

The feature importance scores of a fit gradient boosting model can be
accessed via the "feature_importances_" property:

   ...     max_depth=1, random_state=0).fit(X, y)
   >>> clf.feature_importances_
   array([0.10..., 0.10..., 0.11..., ...

Note that this computation of feature importance is based on entropy,
and it is distinct from "sklearn.inspection.permutation_importance()"
which is based on permutation of the features.

Examples:
^^^^^^^^^

  py*

[Friedman2001] Friedman, J.H. (2001). >>:doi:`Greedy function
               approximation: A gradient boosting machine
               <10.1214/aos/1013203451>`<<. Annals of Statistics, 29,
               1189-1232.

[Friedman2002] Friedman, J.H. (2002). Stochastic gradient boosting..
               Computational Statistics & Data Analysis, 38, 367-378.

[R2007] G. Ridgeway (2006). Generalized Boosted Models: A guide to the
        gbm package

1.11.5. Histogram-Based Gradient Boosting

Scikit-learn 0.21 introduced two new implementations of gradient
boosting trees, namely "HistGradientBoostingClassifier" and
"HistGradientBoostingRegressor", inspired by LightGBM (See
[LightGBM]).

These histogram-based estimators can be **orders of magnitude faster**
than "GradientBoostingClassifier" and "GradientBoostingRegressor" when
the number of samples is larger than tens of thousands of samples.

They also have built-in support for missing values, which avoids the
need for an imputer.

These fast estimators first bin the input samples "X" into integer-
valued bins (typically 256 bins) which tremendously reduces the number
of splitting points to consider, and allows the algorithm to leverage
integer-based data structures (histograms) instead of relying on
sorted continuous values when building the trees. The API of these
estimators is slightly different, and some of the features from
"GradientBoostingClassifier" and "GradientBoostingRegressor" are not
yet supported, for instance some loss functions.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_inspection_plot_partial_dependence.py*

1.11.5.1. Usage

Most of the parameters are unchanged from "GradientBoostingClassifier"
and "GradientBoostingRegressor". One exception is the "max_iter"
parameter that replaces "n_estimators", and controls the number of
iterations of the boosting process:

   >>> from sklearn.ensemble import HistGradientBoostingClassifier

   >>> clf = HistGradientBoostingClassifier(max_iter=100).fit(X_train, y_train)
   0.8965

Available losses for regression are 'squared_error', 'absolute_error',
which is less sensitive to outliers, and 'poisson', which is well
suited to model counts and frequencies. For classification, 'log_loss'
is the only option. For binary classification it uses the binary log
loss, also kown as binomial deviance or binary cross-entropy. For
*n_classes >= 3*, it uses the multi-class log loss function, with
multinomial deviance and categorical cross-entropy as alternative
names. The appropriate loss version is selected based on *y* passed to
*fit*.

The size of the trees can be controlled through the "max_leaf_nodes",
"max_depth", and "min_samples_leaf" parameters.

The number of bins used to bin the data is controlled with the
"max_bins" parameter. Using less bins acts as a form of
regularization. It is generally recommended to use as many bins as
possible, which is the default.

The "l2_regularization" parameter is a regularizer on the loss
function and corresponds to \lambda in equation (2) of [XGBoost].

Note that **early-stopping is enabled by default if the number of

samples is larger than 10,000**. The early-stopping behaviour is
controlled via the "early_stopping", "scoring", "validation_fraction",
"n_iter_no_change", and "tol" parameters. It is possible to early-stop
using an arbitrary *scorer*, or just the training or validation loss.
Note that for technical reasons, using a scorer is significantly
slower than using the loss. By default, early-stopping is performed if
there are at least 10,000 samples in the training set, using the
validation loss.

1.11.5.2. Missing values support

"HistGradientBoostingClassifier" and "HistGradientBoostingRegressor"
have built-in support for missing values (NaNs).

During training, the tree grower learns at each split point whether
samples with missing values should go to the left or right child,
based on the potential gain. When predicting, samples with missing
values are assigned to the left or right child consequently:

   >>> X = np.array([0, 1, 2, np.nan]).reshape(-1, 1)
   >>> y = [0, 0, 1, 1]

   >>> gbdt = HistGradientBoostingClassifier(min_samples_leaf=1).fit(X, y)
   >>> gbdt.predict(X)
   array([0, 0, 1, 1])

When the missingness pattern is predictive, the splits can be done on

>>> gbdt = HistGradientBoostingClassifier(min_samples_leaf=1).fit(X, y)

whether the feature value is missing or not:

   >>> X = np.array([0, np.nan, 1, 2, np.nan]).reshape(-1, 1)
   >>> y = [0, 1, 0, 0, 1]
   >>> gbdt = HistGradientBoostingClassifier(min_samples_leaf=1,
   ...                                       max_depth=2,
   ...                                       learning_rate=1,
   ...                                       max_iter=1).fit(X, y)
   array([0, 1, 0, 0, 1])

If no missing values were encountered for a given feature during
training, then samples with missing values are mapped to whichever
child has the most samples.

1.11.5.3. Sample weight support

sample support weights during *fit*.

The following toy example demonstrates how the model ignores the
samples with zero sample weights:

>>> X = [[1, 0],
...      [1, 0],
...      [1, 0],
...      [0, 1]]
>>> y = [0, 0, 1, 0]
>>> # ignore the first 2 training samples by setting their weight to 0
>>> sample_weight = [0, 0, 1, 1]
>>> gb = HistGradientBoostingClassifier(min_samples_leaf=1)
>>> gb.fit(X, y, sample_weight=sample_weight)
HistGradientBoostingClassifier(...)
>>> gb.predict([[1, 0]])
array([1])
>>> gb.predict_proba([[1, 0]])[0, 1]
0.99...

0.99...

As you can see, the *[1, 0]* is comfortably classified as *1* since
the first two samples are ignored due to their sample weights.

Implementation detail: taking sample weights into account amounts to
multiplying the gradients (and the hessians) by the sample weights.
Note that the binning stage (specifically the quantiles computation)
does not take the weights into account.

1.11.5.4. Categorical Features Support

have native support for categorical features: they can consider splits
on non-ordered, categorical data.

For datasets with categorical features, using the native categorical
support is often better than relying on one-hot encoding
("OneHotEncoder"), because one-hot encoding requires more tree depth
to achieve equivalent splits. It is also usually better to rely on the
native categorical support rather than to treat categorical features
as continuous (ordinal), which happens for ordinal-encoded categorical
data, since categories are nominal quantities where order does not
matter.

To enable categorical support, a boolean mask can be passed to the
*categorical_features* parameter, indicating which feature is
categorical. In the following, the first feature will be treated as
categorical and the second feature as numerical:

   >>> gbdt = HistGradientBoostingClassifier(categorical_features=[True, False])

Equivalently, one can pass a list of integers indicating the indices
of the categorical features:

   >>> gbdt = HistGradientBoostingClassifier(categorical_features=[0])

The cardinality of each categorical feature should be less than the
*max_bins* parameter, and each categorical feature is expected to be
encoded in *[0, max_bins - 1]*. To that end, it might be useful to
pre-process the data with an "OrdinalEncoder" as done in *sphx_glr_au
to_examples_ensemble_plot_gradient_boosting_categorical.py*.

If there are missing values during training, the missing values will
be treated as a proper category. If there are no missing values during
training, then at prediction time, missing values are mapped to the
child node that has the most samples (just like for continuous
features). When predicting, categories that were not seen during fit
time will be treated as missing values.

**Split finding with categorical features**: The canonical way of
considering categorical splits in a tree is to consider all of the
2^{K - 1} - 1 partitions, where K is the number of categories. This
can quickly become prohibitive when K is large. Fortunately, since
gradient boosting trees are always regression trees (even for

classification problems), there exist a faster strategy that can yield
equivalent splits. First, the categories of a feature are sorted
according to the variance of the target, for each category *k*. Once
the categories are sorted, one can consider *continuous partitions*,
i.e. treat the categories as if they were ordered continuous values
(see Fisher [Fisher1958] for a formal proof). As a result, only K - 1
splits need to be considered instead of 2^{K - 1} - 1. The initial
sorting is a \mathcal{O}(K \log(K)) operation, leading to a total
complexity of \mathcal{O}(K \log(K) + K), instead of \mathcal{O}(2^K).

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_categorical
  .py*

1.11.5.5. Monotonic Constraints

Depending on the problem at hand, you may have prior knowledge
indicating that a given feature should in general have a positive (or
negative) effect on the target value. For example, all else being
equal, a higher credit score should increase the probability of
getting approved for a loan. Monotonic constraints allow you to
incorporate such prior knowledge into the model.

For a predictor F with two features:

   * a **monotonic increase constraint** is a constraint of the form:
           x_1 \leq x_1' \implies F(x_1, x_2) \leq F(x_1', x_2)

   * a **monotonic decrease constraint** is a constraint of the form:
           x_1 \leq x_1' \implies F(x_1, x_2) \geq F(x_1', x_2)

You can specify a monotonic constraint on each feature using the
*monotonic_cst* parameter. For each feature, a value of 0 indicates no
constraint, while 1 and -1 indicate a monotonic increase and monotonic
decrease constraint, respectively:

   >>> from sklearn.ensemble import HistGradientBoostingRegressor

   ... # monotonic increase, monotonic decrease, and no constraint on the 3 features
   >>> gbdt = HistGradientBoostingRegressor(monotonic_cst=[1, -1, 0])

In a binary classification context, imposing a monotonic increase
(decrease) constraint means that higher values of the feature are
supposed to have a positive (negative) effect on the probability of
samples to belong to the positive class.

Nevertheless, monotonic constraints only marginally constrain feature
effects on the output. For instance, monotonic increase and decrease
constraints cannot be used to enforce the following modelling
constraint:

      x_1 \leq x_1' \implies F(x_1, x_2) \leq F(x_1', x_2')

Also, monotonic constraints are not supported for multiclass
classification.

Note:

  Since categories are unordered quantities, it is not possible to
  enforce monotonic constraints on categorical features.

Examples:
^^^^^^^^^

Note:

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_monotonic_constraints.py*

1.11.5.6. Interaction constraints

A priori, the histogram gradient boosting trees are allowed to use any
feature to split a node into child nodes. This creates so called
interactions between features, i.e. usage of different features as
split along a branch. Sometimes, one wants to restrict the possible
interactions, see [Mayer2022]. This can be done by the parameter
"interaction_cst", where one can specify the indices of features that
are allowed to interact. For instance, with 3 features in total,
"interaction_cst=[{0}, {1}, {2}]" forbids all interactions. The
constraints "[{0, 1}, {1, 2}]" specifies two groups of possibly
interacting features. Features 0 and 1 may interact with each other,
as well as features 1 and 2. But note that features 0 and 2 are
forbidden to interact. The following depicts a tree and the possible
splits of the tree:

      1      <- Both constraint groups could be applied from now on
     / \
    1   2    <- Left split still fulfills both constraint groups.
   / \ / \      Right split at feature 2 has only group {1, 2} from now on.

LightGBM uses the same logic for overlapping groups.

Note that features not listed in "interaction_cst" are automatically

/ \ / \      Right split at feature 2 has only group {1, 2} from now on.

assigned an interaction group for themselves. With again 3 features,
this means that "[{0}]" is equivalent to "[{0}, {1, 2}]".

[Mayer2022] M. Mayer, S.C. Bourassa, M. Hoesli, and D.F. Scognamiglio.
            2022. >>:doi:`Machine Learning Applications to Land and
            Structure Valuation <10.3390/jrfm15050193>`<<. Journal of
            Risk and Financial Management 15, no. 5: 193

1.11.5.7. Low-level parallelism

have implementations that use OpenMP for parallelization through
Cython. For more details on how to control the number of threads,
please refer to our Parallelism notes.

The following parts are parallelized:

* mapping samples from real values to integer-valued bins (finding the
  bin thresholds is however sequential)

* building histograms is parallelized over features

* finding the best split point at a node is parallelized over features

* during fit, mapping samples into the left and right children is
  parallelized over samples

* gradient and hessians computations are parallelized over samples

* predicting is parallelized over samples

1.11.5.8. Why it's faster

The bottleneck of a gradient boosting procedure is building the

decision trees. Building a traditional decision tree (as in the other
GBDTs "GradientBoostingClassifier" and "GradientBoostingRegressor")
requires sorting the samples at each node (for each feature). Sorting
is needed so that the potential gain of a split point can be computed
efficiently. Splitting a single node has thus a complexity of
\mathcal{O}(n_\text{features} \times n \log(n)) where n is the number
of samples at the node.

"HistGradientBoostingClassifier" and "HistGradientBoostingRegressor",
in contrast, do not require sorting the feature values and instead use
a data-structure called a histogram, where the samples are implicitly
ordered. Building a histogram has a \mathcal{O}(n) complexity, so the
node splitting procedure has a \mathcal{O}(n_\text{features} \times n)
complexity, much smaller than the previous one. In addition, instead
of considering n split points, we here consider only "max_bins" split
points, which is much smaller.

In order to build histograms, the input data *X* needs to be binned
into integer-valued bins. This binning procedure does require sorting
the feature values, but it only happens once at the very beginning of
the boosting process (not at each node, like in
"GradientBoostingClassifier" and "GradientBoostingRegressor").

Finally, many parts of the implementation of

are parallelized.

[XGBoost] Tianqi Chen, Carlos Guestrin, >>:arxiv:`"XGBoost: A Scalable
          Tree Boosting System" <1603.02754>`<<

[LightGBM] Ke et. al. "LightGBM: A Highly Efficient Gradient
           BoostingDecision Tree"

[Fisher1958] Fisher, W.D. (1958). "On Grouping for Maximum
             Homogeneity" Journal of the American Statistical
             Association, 53, 789-798.

1.11.6. Voting Classifier

The idea behind the "VotingClassifier" is to combine conceptually
different machine learning classifiers and use a majority vote or the
average predicted probabilities (soft vote) to predict the class
labels. Such a classifier can be useful for a set of equally well
performing models in order to balance out their individual weaknesses.

1.11.6.1. Majority Class Labels (Majority/Hard Voting)

In majority voting, the predicted class label for a particular sample
is the class label that represents the majority (mode) of the class
labels predicted by each individual classifier.

E.g., if the prediction for a given sample is

* classifier 1 -> class 1

* classifier 2 -> class 1

* classifier 3 -> class 2

the VotingClassifier (with "voting='hard'") would classify the sample

as "class 1" based on the majority class label.

In the cases of a tie, the "VotingClassifier" will select the class
based on the ascending sort order. E.g., in the following scenario

* classifier 1 -> class 2

the class label 1 will be assigned to the sample.

1.11.6.2. Usage

The following example shows how to fit the majority rule classifier:

   >>> from sklearn.ensemble import VotingClassifier

   >>> X, y = iris.data[:, 1:3], iris.target

   >>> clf1 = LogisticRegression(random_state=1)
   >>> clf2 = RandomForestClassifier(n_estimators=50, random_state=1)
   >>> clf3 = GaussianNB()

   >>> eclf = VotingClassifier(
   ...     estimators=[('lr', clf1), ('rf', clf2), ('gnb', clf3)],
   ...     voting='hard')

   >>> for clf, label in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'naive Bayes', 'Ensemble']):
   ...     scores = cross_val_score(clf, X, y, scoring='accuracy', cv=5)

>>> for clf, label in zip([clf1, clf2, clf3, eclf], ['Logistic Regression', 'Random Forest', 'naive Bayes', 'Ensemble']):
   ...     print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))
   Accuracy: 0.95 (+/- 0.04) [Logistic Regression]
   Accuracy: 0.94 (+/- 0.04) [Random Forest]
   Accuracy: 0.91 (+/- 0.04) [naive Bayes]
   Accuracy: 0.95 (+/- 0.04) [Ensemble]

1.11.6.3. Weighted Average Probabilities (Soft Voting)

In contrast to majority voting (hard voting), soft voting returns the
class label as argmax of the sum of predicted probabilities.

Specific weights can be assigned to each classifier via the "weights"
parameter. When weights are provided, the predicted class
probabilities for each classifier are collected, multiplied by the
classifier weight, and averaged. The final class label is then derived
from the class label with the highest average probability.

To illustrate this with a simple example, let's assume we have 3
classifiers and a 3-class classification problems where we assign
equal weights to all classifiers: w1=1, w2=1, w3=1.

The weighted average probabilities for a sample would then be
calculated as follows:

+------------------+------------+------------+------------+
| classifier       | class 1    | class 2    | class 3    |
|==================|============|============|============|

| classifier 1     | w1 * 0.2   | w1 * 0.5   | w1 * 0.3   |
| classifier 2     | w2 * 0.6   | w2 * 0.3   | w2 * 0.1   |
| classifier 3     | w3 * 0.3   | w3 * 0.4   | w3 * 0.3   |
| weighted average | 0.37       | 0.4        | 0.23       |

Here, the predicted class label is 2, since it has the highest average
probability.

The following example illustrates how the decision regions may change
when a soft "VotingClassifier" is used based on a linear Support
Vector Machine, a Decision Tree, and a K-nearest neighbor classifier:

   >>> from itertools import product

   >>> # Loading some example data
   >>> X = iris.data[:, [0, 2]]
   >>> y = iris.target

   >>> # Training classifiers
   >>> clf1 = DecisionTreeClassifier(max_depth=4)
   >>> clf2 = KNeighborsClassifier(n_neighbors=7)

>>> X = iris.data[:, [0, 2]]

   >>> clf3 = SVC(kernel='rbf', probability=True)
   >>> eclf = VotingClassifier(estimators=[('dt', clf1), ('knn', clf2), ('svc', clf3)],
   ...                         voting='soft', weights=[2, 1, 2])

   >>> clf1 = clf1.fit(X, y)
   >>> clf2 = clf2.fit(X, y)
   >>> clf3 = clf3.fit(X, y)
   >>> eclf = eclf.fit(X, y)

   [image]

1.11.6.4. Using the *VotingClassifier* with *GridSearchCV*

The "VotingClassifier" can also be used together with "GridSearchCV"
in order to tune the hyperparameters of the individual estimators:

   >>> clf2 = RandomForestClassifier(random_state=1)
   ...     voting='soft'
   ... )

   >>> params = {'lr__C': [1.0, 100.0], 'rf__n_estimators': [20, 200]}

   >>> grid = GridSearchCV(estimator=eclf, param_grid=params, cv=5)
   >>> grid = grid.fit(iris.data, iris.target)

1.11.6.5. Usage

In order to predict the class labels based on the predicted class-
probabilities (scikit-learn estimators in the VotingClassifier must
support "predict_proba" method):

   ... )

Optionally, weights can be provided for the individual classifiers:

   ...     voting='soft', weights=[2,5,1]
   ... )

1.11.7. Voting Regressor

The idea behind the "VotingRegressor" is to combine conceptually
different machine learning regressors and return the average predicted
values. Such a regressor can be useful for a set of equally well

1.11.7.1. Usage

The following example shows how to fit the VotingRegressor:

   >>> from sklearn.datasets import load_diabetes
   >>> from sklearn.ensemble import VotingRegressor

   >>> X, y = load_diabetes(return_X_y=True)

   >>> reg1 = GradientBoostingRegressor(random_state=1)
   >>> reg2 = RandomForestRegressor(random_state=1)
   >>> reg3 = LinearRegression()

>>> # Training classifiers
   >>> ereg = VotingRegressor(estimators=[('gb', reg1), ('rf', reg2), ('lr', reg3)])
   >>> ereg = ereg.fit(X, y)

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_voting_regressor.py*

1.11.8. Stacked generalization

Stacked generalization is a method for combining estimators to reduce
their biases [W1992] [HTF]. More precisely, the predictions of each
individual estimator are stacked together and used as input to a final
estimator to compute the prediction. This final estimator is trained
through cross-validation.

The "StackingClassifier" and "StackingRegressor" provide such
strategies which can be applied to classification and regression
problems.

The *estimators* parameter corresponds to the list of the estimators
which are stacked together in parallel on the input data. It should be
given as a list of names and estimators:

   >>> from sklearn.linear_model import RidgeCV, LassoCV
   >>> from sklearn.neighbors import KNeighborsRegressor
   >>> estimators = [('ridge', RidgeCV()),
   ...               ('lasso', LassoCV(random_state=42)),
   ...               ('knr', KNeighborsRegressor(n_neighbors=20,
   ...                                           metric='euclidean'))]

The *final_estimator* will use the predictions of the *estimators* as

...                                           metric='euclidean'))]

input. It needs to be a classifier or a regressor when using
"StackingClassifier" or "StackingRegressor", respectively:

   >>> from sklearn.ensemble import StackingRegressor
   >>> final_estimator = GradientBoostingRegressor(
   ...     n_estimators=25, subsample=0.5, min_samples_leaf=25, max_features=1,
   ...     random_state=42)
   >>> reg = StackingRegressor(
   ...     estimators=estimators,
   ...     final_estimator=final_estimator)

To train the *estimators* and *final_estimator*, the *fit* method
needs to be called on the training data:

   ...                                                     random_state=42)
   >>> reg.fit(X_train, y_train)
   StackingRegressor(...)

During training, the *estimators* are fitted on the whole training
data *X_train*. They will be used when calling *predict* or
*predict_proba*. To generalize and avoid over-fitting, the
*final_estimator* is trained on out-samples using
"sklearn.model_selection.cross_val_predict()" internally.

For "StackingClassifier", note that the output of the "estimators" is

controlled by the parameter *stack_method* and it is called by each
estimator. This parameter is either a string, being estimator method
names, or *'auto'* which will automatically identify an available
method depending on the availability, tested in the order of
preference: *predict_proba*, *decision_function* and *predict*.

A "StackingRegressor" and "StackingClassifier" can be used as any
other regressor or classifier, exposing a *predict*, *predict_proba*,
and *decision_function* methods, e.g.:

   >>> y_pred = reg.predict(X_test)
   >>> from sklearn.metrics import r2_score
   >>> print('R2 score: {:.2f}'.format(r2_score(y_test, y_pred)))
   R2 score: 0.53

Note that it is also possible to get the output of the stacked
*estimators* using the *transform* method:

   >>> reg.transform(X_test[:5])
   array([[142..., 138..., 146...],
          [179..., 182..., 151...],
          [139..., 132..., 158...],
          [286..., 292..., 225...],
          [126..., 124..., 164...]])

In practice, a stacking predictor predicts as good as the best
predictor of the base layer and even sometimes outperforms it by
combining the different strengths of the these predictors. However,
training a stacking predictor is computationally expensive.

Note:

  For "StackingClassifier", when using

Note:

  *stack_method_='predict_proba'*, the first column is dropped when
  the problem is a binary classification problem. Indeed, both
  probability columns predicted by each estimator are perfectly
  collinear.

Note:

  Multiple stacking layers can be achieved by assigning
  *final_estimator* to a "StackingClassifier" or "StackingRegressor":

     >>> final_layer_rfr = RandomForestRegressor(
     ...     n_estimators=10, max_features=1, max_leaf_nodes=5,random_state=42)
     >>> final_layer_gbr = GradientBoostingRegressor(
     >>> final_layer = StackingRegressor(
     ...     estimators=[('rf', final_layer_rfr),
     ...                 ('gbrt', final_layer_gbr)],
     ...     final_estimator=RidgeCV()
     ...     )
     >>> multi_layer_regressor = StackingRegressor(
     ...     estimators=[('ridge', RidgeCV()),
     ...                 ('lasso', LassoCV(random_state=42)),
     ...                 ('knr', KNeighborsRegressor(n_neighbors=20,
     ...                                             metric='euclidean'))],
     ...     final_estimator=final_layer
     ... )
     >>> multi_layer_regressor.fit(X_train, y_train)
     StackingRegressor(...)
     >>> print('R2 score: {:.2f}'
     ...       .format(multi_layer_regressor.score(X_test, y_test)))

... )
     R2 score: 0.53

[W1992] Wolpert, David H. "Stacked generalization." Neural networks
        5.2 (1992): 241-259.

3.1. Cross-validation: evaluating estimator performance
*******************************************************

Learning the parameters of a prediction function and testing it on the
same data is a methodological mistake: a model that would just repeat
the labels of the samples that it has just seen would have a perfect
score but would fail to predict anything useful on yet-unseen data.
This situation is called **overfitting**. To avoid it, it is common
practice when performing a (supervised) machine learning experiment to
hold out part of the available data as a **test set** "X_test,
y_test". Note that the word "experiment" is not intended to denote
academic use only, because even in commercial settings machine
learning usually starts out experimentally. Here is a flowchart of
typical cross validation workflow in model training. The best
parameters can be determined by grid search techniques.

[image: Grid Search Workflow][image]

In scikit-learn a random split into training and test sets can be
quickly computed with the "train_test_split()" helper function. Let's
load the iris data set to fit a linear support vector machine on it:

   >>> X, y = datasets.load_iris(return_X_y=True)
   >>> X.shape, y.shape
   ((150, 4), (150,))

We can now quickly sample a training set while holding out 40% of the
data for testing (evaluating) our classifier:

>>> X.shape, y.shape

   ...     X, y, test_size=0.4, random_state=0)

   >>> X_train.shape, y_train.shape
   ((90, 4), (90,))
   >>> X_test.shape, y_test.shape
   ((60, 4), (60,))

   >>> clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)
   0.96...

When evaluating different settings ("hyperparameters") for estimators,
such as the "C" setting that must be manually set for an SVM, there is
still a risk of overfitting *on the test set* because the parameters
can be tweaked until the estimator performs optimally. This way,
knowledge about the test set can "leak" into the model and evaluation
metrics no longer report on generalization performance. To solve this
problem, yet another part of the dataset can be held out as a so-
called "validation set": training proceeds on the training set, after
which evaluation is done on the validation set, and when the
experiment seems to be successful, final evaluation can be done on the
test set.

However, by partitioning the available data into three sets, we
drastically reduce the number of samples which can be used for
learning the model, and the results can depend on a particular random
choice for the pair of (train, validation) sets.

A solution to this problem is a procedure called cross-validation (CV

for short). A test set should still be held out for final evaluation,
but the validation set is no longer needed when doing CV. In the basic
approach, called *k*-fold CV, the training set is split into *k*
smaller sets (other approaches are described below, but generally
follow the same principles). The following procedure is followed for
each of the *k* "folds":

   * A model is trained using k-1 of the folds as training data;

   * the resulting model is validated on the remaining part of the
     data (i.e., it is used as a test set to compute a performance
     measure such as accuracy).

The performance measure reported by *k*-fold cross-validation is then
the average of the values computed in the loop. This approach can be
computationally expensive, but does not waste too much data (as is the
case when fixing an arbitrary validation set), which is a major
advantage in problems such as inverse inference where the number of
samples is very small.

[image]

3.1.1. Computing cross-validated metrics

The simplest way to use cross-validation is to call the
"cross_val_score()" helper function on the estimator and the dataset.

The following example demonstrates how to estimate the accuracy of a
linear kernel support vector machine on the iris dataset by splitting

the data, fitting a model and computing the score 5 consecutive times
(with different splits each time):

   >>> clf = svm.SVC(kernel='linear', C=1, random_state=42)
   >>> scores
   array([0.96..., 1. , 0.96..., 0.96..., 1. ])

The mean score and the standard deviation are hence given by:

   >>> print("%0.2f accuracy with a standard deviation of %0.2f" % (scores.mean(), scores.std()))
   0.98 accuracy with a standard deviation of 0.02

By default, the score computed at each CV iteration is the "score"
method of the estimator. It is possible to change this by using the
scoring parameter:

   >>> scores = cross_val_score(
   ...     clf, X, y, cv=5, scoring='f1_macro')
   >>> scores
   array([0.96..., 1.  ..., 0.96..., 0.96..., 1.        ])

See The scoring parameter: defining model evaluation rules for
details. In the case of the Iris dataset, the samples are balanced
across target classes hence the accuracy and the F1-score are almost
equal.

When the "cv" argument is an integer, "cross_val_score()" uses the
"KFold" or "StratifiedKFold" strategies by default, the latter being
used if the estimator derives from "ClassifierMixin".

It is also possible to use other cross validation strategies by

passing a cross validation iterator instead, for instance:

   >>> from sklearn.model_selection import ShuffleSplit
   >>> n_samples = X.shape[0]
   >>> cv = ShuffleSplit(n_splits=5, test_size=0.3, random_state=0)
   >>> cross_val_score(clf, X, y, cv=cv)
   array([0.977..., 0.977..., 1.  ..., 0.955..., 1.        ])

Another option is to use an iterable yielding (train, test) splits as
arrays of indices, for example:

   >>> def custom_cv_2folds(X):
   ...     n = X.shape[0]
   ...     i = 1
   ...     while i <= 2:
   ...         idx = np.arange(n * (i - 1) / 2, n * i / 2, dtype=int)
   ...         yield idx, idx
   ...         i += 1
   ...
   >>> custom_cv = custom_cv_2folds(X)
   >>> cross_val_score(clf, X, y, cv=custom_cv)
   array([1.        , 0.973...])

Data transformation with held out data

Just as it is important to test a predictor on data held-out from
training, preprocessing (such as standardization, feature selection,
etc.) and similar data transformations similarly should be learnt from
a training set and applied to held-out data for prediction:

   >>> from sklearn import preprocessing
   >>> scaler = preprocessing.StandardScaler().fit(X_train)

>>> X_train, X_test, y_train, y_test = train_test_split(
   >>> X_train_transformed = scaler.transform(X_train)
   >>> clf = svm.SVC(C=1).fit(X_train_transformed, y_train)
   >>> clf.score(X_test_transformed, y_test)
   0.9333...

A "Pipeline" makes it easier to compose estimators, providing this
behavior under cross-validation:

   >>> clf = make_pipeline(preprocessing.StandardScaler(), svm.SVC(C=1))
   array([0.977..., 0.933..., 0.955..., 0.933..., 0.977...])

See Pipelines and composite estimators.

3.1.1.1. The cross_validate function and multiple metric evaluation

The "cross_validate()" function differs from "cross_val_score()" in
two ways:

* It allows specifying multiple metrics for evaluation.

* It returns a dict containing fit-times, score-times (and optionally
  training scores as well as fitted estimators) in addition to the
  test score.

For single metric evaluation, where the scoring parameter is a string,
callable or None, the keys will be - "['test_score', 'fit_time',
'score_time']"

And for multiple metric evaluation, the return value is a dict with
the following keys - "['test_<scorer1_name>', 'test_<scorer2_name>',
'test_<scorer...>', 'fit_time', 'score_time']"

"return_train_score" is set to "False" by default to save computation
time. To evaluate the scores on the training set as well you need to
set it to "True".

You may also retain the estimator fitted on each training set by
setting "return_estimator=True".

The multiple metrics can be specified either as a list, tuple or set
of predefined scorer names:

   >>> from sklearn.metrics import recall_score
   >>> scoring = ['precision_macro', 'recall_macro']
   >>> clf = svm.SVC(kernel='linear', C=1, random_state=0)
   >>> scores = cross_validate(clf, X, y, scoring=scoring)
   >>> sorted(scores.keys())
   ['fit_time', 'score_time', 'test_precision_macro', 'test_recall_macro']
   >>> scores['test_recall_macro']

Or as a dict mapping scorer name to a predefined or custom scoring
function:

   >>> from sklearn.metrics import make_scorer
   >>> scoring = {'prec_macro': 'precision_macro',
   ...            'rec_macro': make_scorer(recall_score, average='macro')}
   >>> scores = cross_validate(clf, X, y, scoring=scoring,
   ...                         cv=5, return_train_score=True)
   ['fit_time', 'score_time', 'test_prec_macro', 'test_rec_macro',

...                         cv=5, return_train_score=True)
    'train_prec_macro', 'train_rec_macro']
   >>> scores['train_rec_macro']
   array([0.97..., 0.97..., 0.99..., 0.98..., 0.98...])

Here is an example of "cross_validate" using a single metric:

   >>> scores = cross_validate(clf, X, y,
   ...                         scoring='precision_macro', cv=5,
   ...                         return_estimator=True)
   ['estimator', 'fit_time', 'score_time', 'test_score']

3.1.1.2. Obtaining predictions by cross-validation
--------------------------------------------------

The function "cross_val_predict()" has a similar interface to
"cross_val_score()", but returns, for each element in the input, the
prediction that was obtained for that element when it was in the test
set. Only cross-validation strategies that assign all elements to a
test set exactly once can be used (otherwise, an exception is raised).

Warning:

  Note on inappropriate usage of cross_val_predictThe result of
  "cross_val_predict()" may be different from those obtained using
  "cross_val_score()" as the elements are grouped in different ways.
  The function "cross_val_score()" takes an average over cross-
  validation folds, whereas "cross_val_predict()" simply returns the
  labels (or probabilities) from several distinct models
  undistinguished. Thus, "cross_val_predict()" is not an appropriate

validation folds, whereas "cross_val_predict()" simply returns the
  measure of generalization error.

The function "cross_val_predict()" is appropriate for:
   * Visualization of predictions obtained from different models.

   * Model blending: When predictions of one supervised estimator are
     used to train another estimator in ensemble methods.

The available cross validation iterators are introduced in the
following section.

Examples
^^^^^^^^

* *sphx_glr_auto_examples_model_selection_plot_roc_crossval.py*,

* *sphx_glr_auto_examples_feature_selection_plot_rfe_with_cross_valid
  ation.py*,

* *sphx_glr_auto_examples_model_selection_plot_grid_search_digits.py*,

* *sphx_glr_auto_examples_model_selection_plot_grid_search_text_featu
  re_extraction.py*,

* *sphx_glr_auto_examples_model_selection_plot_cv_predict.py*,

* *sphx_glr_auto_examples_model_selection_plot_nested_cross_validatio
  n_iris.py*.

3.1.2. Cross validation iterators

The following sections list utilities to generate indices that can be
used to generate dataset splits according to different cross
validation strategies.

3.1.2.1. Cross-validation iterators for i.i.d. data

Assuming that some data is Independent and Identically Distributed
(i.i.d.) is making the assumption that all samples stem from the same

generative process and that the generative process is assumed to have
no memory of past generated samples.

The following cross-validators can be used in such cases.

Note:

  While i.i.d. data is a common assumption in machine learning theory,
  it rarely holds in practice. If one knows that the samples have been
  generated using a time-dependent process, it is safer to use a time-
  series aware cross-validation scheme. Similarly, if we know that the
  generative process has a group structure (samples collected from
  different subjects, experiments, measurement devices), it is safer
  to use group-wise cross-validation.

3.1.2.1.1. K-fold

"KFold" divides all the samples in k groups of samples, called folds
(if k = n, this is equivalent to the *Leave One Out* strategy), of
equal sizes (if possible). The prediction function is learned using k
- 1 folds, and the fold left out is used for test.

Example of 2-fold cross-validation on a dataset with 4 samples:

   >>> X = ["a", "b", "c", "d"]
   >>> kf = KFold(n_splits=2)
   >>> for train, test in kf.split(X):
   ...     print("%s %s" % (train, test))
   [2 3] [0 1]
   [0 1] [2 3]

Here is a visualization of the cross-validation behavior. Note that

>>> for train, test in kf.split(X):
   [2 3] [0 1]
   [0 1] [2 3]

"KFold" is not affected by classes or groups.

   [image]

Each fold is constituted by two arrays: the first one is related to
the *training set*, and the second one to the *test set*. Thus, one
can create the training/test sets using numpy indexing:

   >>> X = np.array([[0., 0.], [1., 1.], [-1., -1.], [2., 2.]])
   >>> y = np.array([0, 1, 0, 1])
   >>> X_train, X_test, y_train, y_test = X[train], X[test], y[train], y[test]

3.1.2.1.2. Repeated K-Fold

"RepeatedKFold" repeats K-Fold n times. It can be used when one
requires to run "KFold" n times, producing different splits in each
repetition.

Example of 2-fold K-Fold repeated 2 times:

   >>> from sklearn.model_selection import RepeatedKFold
   >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
   >>> random_state = 12883823
   >>> rkf = RepeatedKFold(n_splits=2, n_repeats=2, random_state=random_state)
   >>> for train, test in rkf.split(X):
   ...
   [2 3] [0 1]
   [0 1] [2 3]
   [0 2] [1 3]
   [1 3] [0 2]

Similarly, "RepeatedStratifiedKFold" repeats Stratified K-Fold n times
with different randomization in each repetition.

3.1.2.1.3. Leave One Out (LOO)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"LeaveOneOut" (or LOO) is a simple cross-validation. Each learning set

is created by taking all the samples except one, the test set being
the sample left out. Thus, for n samples, we have n different training
sets and n different tests set. This cross-validation procedure does
not waste much data as only one sample is removed from the training
set:

   >>> from sklearn.model_selection import LeaveOneOut

   >>> X = [1, 2, 3, 4]
   >>> loo = LeaveOneOut()
   >>> for train, test in loo.split(X):
   [1 2 3] [0]
   [0 2 3] [1]
   [0 1 3] [2]
   [0 1 2] [3]

Potential users of LOO for model selection should weigh a few known
caveats. When compared with k-fold cross validation, one builds n
models from n samples instead of k models, where n > k. Moreover, each
is trained on n - 1 samples rather than (k-1) n / k. In both ways,
assuming k is not too large and k < n, LOO is more computationally
expensive than k-fold cross validation.

In terms of accuracy, LOO often results in high variance as an
estimator for the test error. Intuitively, since n - 1 of the n
samples are used to build each model, models constructed from folds
are virtually identical to each other and to the model built from the
entire training set.

However, if the learning curve is steep for the training size in
question, then 5- or 10- fold cross validation can overestimate the
generalization error.

As a general rule, most authors, and empirical evidence, suggest that
5- or 10- fold cross validation should be preferred to LOO.

* http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-12.html;

* T. Hastie, R. Tibshirani, J. Friedman,  The Elements of Statistical
  Learning, Springer 2009

* L. Breiman, P. Spector Submodel selection and evaluation in
  regression: The X-random case, International Statistical Review
  1992;

* R. Kohavi, A Study of Cross-Validation and Bootstrap for Accuracy
  Estimation and Model Selection, Intl. Jnt. Conf. AI

* R. Bharat Rao, G. Fung, R. Rosales, On the Dangers of Cross-
  Validation. An Experimental Evaluation, SIAM 2008;

* G. James, D. Witten, T. Hastie, R Tibshirani, An Introduction to
  Statistical Learning, Springer 2013.

3.1.2.1.4. Leave P Out (LPO)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"LeavePOut" is very similar to "LeaveOneOut" as it creates all the
possible training/test sets by removing p samples from the complete
set. For n samples, this produces {n \choose p} train-test pairs.
Unlike "LeaveOneOut" and "KFold", the test sets will overlap for p >
1.

Example of Leave-2-Out on a dataset with 4 samples:

   >>> from sklearn.model_selection import LeavePOut

   >>> X = np.ones(4)
   >>> lpo = LeavePOut(p=2)
   >>> for train, test in lpo.split(X):

1.

   [2 3] [0 1]
   [1 3] [0 2]
   [1 2] [0 3]
   [0 3] [1 2]
   [0 2] [1 3]
   [0 1] [2 3]

3.1.2.1.5. Random permutations cross-validation a.k.a. Shuffle & Split
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The "ShuffleSplit" iterator will generate a user defined number of
independent train / test dataset splits. Samples are first shuffled
and then split into a pair of train and test sets.

It is possible to control the randomness for reproducibility of the
results by explicitly seeding the "random_state" pseudo random number
generator.

Here is a usage example:

   >>> X = np.arange(10)
   >>> ss = ShuffleSplit(n_splits=5, test_size=0.25, random_state=0)
   >>> for train_index, test_index in ss.split(X):
   ...     print("%s %s" % (train_index, test_index))
   [9 1 6 7 3 0 5] [2 8 4]
   [2 9 8 0 6 7 4] [3 5 1]
   [4 5 1 0 6 9 7] [2 3 8]
   [2 7 5 8 0 3 4] [6 1 9]
   [4 1 0 6 8 9 3] [5 2 7]

"ShuffleSplit" is not affected by classes or groups.

   [image]

"ShuffleSplit" is thus a good alternative to "KFold" cross validation
that allows a finer control on the number of iterations and the

   [image]

proportion of samples on each side of the train / test split.

3.1.2.2. Cross-validation iterators with stratification based on class labels.
------------------------------------------------------------------------------

Some classification problems can exhibit a large imbalance in the
distribution of the target classes: for instance there could be
several times more negative samples than positive samples. In such
cases it is recommended to use stratified sampling as implemented in
"StratifiedKFold" and "StratifiedShuffleSplit" to ensure that relative
class frequencies is approximately preserved in each train and
validation fold.

3.1.2.2.1. Stratified k-fold

"StratifiedKFold" is a variation of *k-fold* which returns
*stratified* folds: each set contains approximately the same
percentage of samples of each target class as the complete set.

Here is an example of stratified 3-fold cross-validation on a dataset
with 50 samples from two unbalanced classes.  We show the number of
samples in each class and compare with "KFold".

>>> from sklearn.model_selection import StratifiedKFold, KFold
>>> X, y = np.ones((50, 1)), np.hstack(([0] * 45, [1] * 5))
>>> skf = StratifiedKFold(n_splits=3)
>>> for train, test in skf.split(X, y):

...     print('train -  {}   |   test -  {}'.format(
...         np.bincount(y[train]), np.bincount(y[test])))
train -  [30  3]   |   test -  [15  2]
train -  [30  4]   |   test -  [15  1]
>>> kf = KFold(n_splits=3)
>>> for train, test in kf.split(X, y):
train -  [28  5]   |   test -  [17]
train -  [34]   |   test -  [11  5]

We can see that "StratifiedKFold" preserves the class ratios
(approximately 1 / 10) in both train and test dataset.

Here is a visualization of the cross-validation behavior.

   [image]

"RepeatedStratifiedKFold" can be used to repeat Stratified K-Fold n
times with different randomization in each repetition.

3.1.2.2.2. Stratified Shuffle Split

"StratifiedShuffleSplit" is a variation of *ShuffleSplit*, which
returns stratified splits, *i.e* which creates splits by preserving
the same percentage for each target class as in the complete set.

   [image]

3.1.2.3. Cross-validation iterators for grouped data

The i.i.d. assumption is broken if the underlying generative process
yield groups of dependent samples.

Such a grouping of data is domain specific. An example would be when
there is medical data collected from multiple patients, with multiple
samples taken from each patient. And such data is likely to be
dependent on the individual group. In our example, the patient id for
each sample will be its group identifier.

In this case we would like to know if a model trained on a particular
set of groups generalizes well to the unseen groups. To measure this,
we need to ensure that all the samples in the validation fold come
from groups that are not represented at all in the paired training
fold.

The following cross-validation splitters can be used to do that. The
grouping identifier for the samples is specified via the "groups"

3.1.2.3.1. Group k-fold
~~~~~~~~~~~~~~~~~~~~~~~

"GroupKFold" is a variation of k-fold which ensures that the same
group is not represented in both testing and training sets. For
example if the data is obtained from different subjects with several
samples per-subject and if the model is flexible enough to learn from
highly person specific features it could fail to generalize to new
subjects. "GroupKFold" makes it possible to detect this kind of
overfitting situations.

Imagine you have three subjects, each with an associated number from 1
to 3:

   >>> from sklearn.model_selection import GroupKFold

to 3:

   >>> X = [0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 8.8, 9, 10]
   >>> y = ["a", "b", "b", "b", "c", "c", "c", "d", "d", "d"]
   >>> groups = [1, 1, 1, 2, 2, 2, 3, 3, 3, 3]

   >>> gkf = GroupKFold(n_splits=3)
   >>> for train, test in gkf.split(X, y, groups=groups):
   [0 1 2 3 4 5] [6 7 8 9]
   [0 1 2 6 7 8 9] [3 4 5]
   [3 4 5 6 7 8 9] [0 1 2]

Each subject is in a different testing fold, and the same subject is
never in both testing and training. Notice that the folds do not have
exactly the same size due to the imbalance in the data. If class
proportions must be balanced across folds, "StratifiedGroupKFold" is a
better option.

   [image]

Similar to "KFold", the test sets from "GroupKFold" will form a
complete partition of all the data. Unlike "KFold", "GroupKFold" is
not randomized at all, whereas "KFold" is randomized when
"shuffle=True".

3.1.2.3.2. StratifiedGroupKFold
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"StratifiedGroupKFold" is a cross-validation scheme that combines both
"StratifiedKFold" and "GroupKFold". The idea is to try to preserve the
distribution of classes in each split while keeping each group within
a single split. That might be useful when you have an unbalanced
dataset so that using just "GroupKFold" might produce skewed splits.

Example:

Example:

   >>> from sklearn.model_selection import StratifiedGroupKFold
   >>> X = list(range(18))
   >>> y = [1] * 6 + [0] * 12
   >>> groups = [1, 2, 3, 3, 4, 4, 1, 1, 2, 2, 3, 4, 5, 5, 5, 6, 6, 6]
   >>> sgkf = StratifiedGroupKFold(n_splits=3)
   >>> for train, test in sgkf.split(X, y, groups=groups):
   [ 0  2  3  4  5  6  7 10 11 15 16 17] [ 1  8  9 12 13 14]
   [ 0  1  4  5  6  7  8  9 11 12 13 14] [ 2  3 10 15 16 17]
   [ 1  2  3  8  9 10 12 13 14 15 16 17] [ 0  4  5  6  7 11]

Implementation notes:

* With the current implementation full shuffle is not possible in most
  scenarios. When shuffle=True, the following happens:

  1. All groups are shuffled.

  2. Groups are sorted by standard deviation of classes using stable
     sort.

  3. Sorted groups are iterated over and assigned to folds.

  That means that only groups with the same standard deviation of
  class distribution will be shuffled, which might be useful when each
  group has only a single class.

* The algorithm greedily assigns each group to one of n_splits test
  sets, choosing the test set that minimises the variance in class
  distribution across test sets. Group assignment proceeds from groups
  with highest to lowest variance in class frequency, i.e. large
  groups peaked on one or few classes are assigned first.

distribution across test sets. Group assignment proceeds from groups

* This split is suboptimal in a sense that it might produce imbalanced
  splits even if perfect stratification is possible. If you have
  relatively close distribution of classes in each group, using
  "GroupKFold" is better.

Here is a visualization of cross-validation behavior for uneven
groups:

   [image]

3.1.2.3.3. Leave One Group Out

"LeaveOneGroupOut" is a cross-validation scheme where each split holds
out samples belonging to one specific group. Group information is
provided via an array that encodes the group of each sample.

Each training set is thus constituted by all the samples except the
ones related to a specific group. This is the same as
"LeavePGroupsOut" with *n_groups=1* and the same as "GroupKFold" with
*n_splits* equal to the number of unique labels passed to the *groups*

For example, in the cases of multiple experiments, "LeaveOneGroupOut"
can be used to create a cross-validation based on the different
experiments: we create a training set using the samples of all the
experiments except one:

   >>> from sklearn.model_selection import LeaveOneGroupOut

   >>> X = [1, 5, 10, 50, 60, 70, 80]
   >>> y = [0, 1, 1, 2, 2, 2, 2]
   >>> groups = [1, 1, 2, 2, 3, 3, 3]
   >>> logo = LeaveOneGroupOut()
   >>> for train, test in logo.split(X, y, groups=groups):

>>> X = [1, 5, 10, 50, 60, 70, 80]
   [2 3 4 5 6] [0 1]
   [0 1 4 5 6] [2 3]
   [0 1 2 3] [4 5 6]

Another common application is to use time information: for instance
the groups could be the year of collection of the samples and thus
allow for cross-validation against time-based splits.

3.1.2.3.4. Leave P Groups Out
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"LeavePGroupsOut" is similar as "LeaveOneGroupOut", but removes
samples related to P groups for each training/test set. All possible
combinations of P groups are left out, meaning test sets will overlap
for P>1.

Example of Leave-2-Group Out:

   >>> from sklearn.model_selection import LeavePGroupsOut

   >>> X = np.arange(6)
   >>> y = [1, 1, 1, 2, 2, 2]
   >>> groups = [1, 1, 2, 2, 3, 3]
   >>> lpgo = LeavePGroupsOut(n_groups=2)
   >>> for train, test in lpgo.split(X, y, groups=groups):
   [4 5] [0 1 2 3]
   [2 3] [0 1 4 5]
   [0 1] [2 3 4 5]

3.1.2.3.5. Group Shuffle Split

The "GroupShuffleSplit" iterator behaves as a combination of
"ShuffleSplit" and "LeavePGroupsOut", and generates a sequence of
randomized partitions in which a subset of groups are held out for
each split. Each train/test split is performed independently meaning

there is no guaranteed relationship between successive test sets.

   >>> from sklearn.model_selection import GroupShuffleSplit

   >>> X = [0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 0.001]
   >>> y = ["a", "b", "b", "b", "c", "c", "c", "a"]
   >>> groups = [1, 1, 2, 2, 3, 3, 4, 4]
   >>> gss = GroupShuffleSplit(n_splits=4, test_size=0.5, random_state=0)
   >>> for train, test in gss.split(X, y, groups=groups):
   ...
   [0 1 2 3] [4 5 6 7]
   [2 3 6 7] [0 1 4 5]
   [2 3 4 5] [0 1 6 7]
   [4 5 6 7] [0 1 2 3]

   [image]

This class is useful when the behavior of "LeavePGroupsOut" is
desired, but the number of groups is large enough that generating all
possible partitions with P groups withheld would be prohibitively
expensive. In such a scenario, "GroupShuffleSplit" provides a random
sample (with replacement) of the train / test splits generated by
"LeavePGroupsOut".

3.1.2.4. Predefined Fold-Splits / Validation-Sets

For some datasets, a pre-defined split of the data into training- and
validation fold or into several cross-validation folds already exists.
Using "PredefinedSplit" it is possible to use these folds e.g. when

searching for hyperparameters.

For example, when using a validation set, set the "test_fold" to 0 for
all samples that are part of the validation set, and to -1 for all
other samples.

3.1.2.5. Using cross-validation iterators to split train and test

The above group cross-validation functions may also be useful for
splitting a dataset into training and testing subsets. Note that the
convenience function "train_test_split()" is a wrapper around
"ShuffleSplit()" and thus only allows for stratified splitting (using
the class labels) and cannot account for groups.

To perform the train and test split, use the indices for the train and
test subsets yielded by the generator output by the *split()* method
of the cross-validation splitter. For example:

   >>> X = np.array([0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 0.001])
   >>> y = np.array(["a", "b", "b", "b", "c", "c", "c", "a"])
   >>> groups = np.array([1, 1, 2, 2, 3, 3, 4, 4])
   >>> train_indx, test_indx = next(
   ...     GroupShuffleSplit(random_state=7).split(X, y, groups)
   ... )
   >>> X_train, X_test, y_train, y_test = \
   ...     X[train_indx], X[test_indx], y[train_indx], y[test_indx]
   >>> X_train.shape, X_test.shape
   ((6,), (2,))

... )
   >>> np.unique(groups[train_indx]), np.unique(groups[test_indx])
   (array([1, 2, 4]), array([3]))

3.1.2.6. Cross validation of time series data

Time series data is characterized by the correlation between
observations that are near in time (*autocorrelation*). However,
classical cross-validation techniques such as "KFold" and
"ShuffleSplit" assume the samples are independent and identically
distributed, and would result in unreasonable correlation between
training and testing instances (yielding poor estimates of
generalization error) on time series data. Therefore, it is very
important to evaluate our model for time series data on the "future"
observations least like those that are used to train the model. To
achieve this, one solution is provided by "TimeSeriesSplit".

3.1.2.6.1. Time Series Split

"TimeSeriesSplit" is a variation of *k-fold* which returns first k
folds as train set and the (k+1) th fold as test set. Note that unlike
standard cross-validation methods, successive training sets are
supersets of those that come before them. Also, it adds all surplus
data to the first training partition, which is always used to train
the model.

This class can be used to cross-validate time series data samples that
are observed at fixed time intervals.

the model.

Example of 3-split time series cross-validation on a dataset with 6
samples:

   >>> from sklearn.model_selection import TimeSeriesSplit

   >>> X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
   >>> y = np.array([1, 2, 3, 4, 5, 6])
   >>> tscv = TimeSeriesSplit(n_splits=3)
   >>> print(tscv)
   TimeSeriesSplit(gap=0, max_train_size=None, n_splits=3, test_size=None)
   >>> for train, test in tscv.split(X):
   [0 1 2] [3]
   [0 1 2 3] [4]
   [0 1 2 3 4] [5]

   [image]

3.1.3. A note on shuffling

If the data ordering is not arbitrary (e.g. samples with the same
class label are contiguous), shuffling it first may be essential to
get a meaningful cross- validation result. However, the opposite may
be true if the samples are not independently and identically
distributed. For example, if samples correspond to news articles, and
are ordered by their time of publication, then shuffling the data will
likely lead to a model that is overfit and an inflated validation
score: it will be tested on samples that are artificially similar
(close in time) to training samples.

Some cross validation iterators, such as "KFold", have an inbuilt

option to shuffle the data indices before splitting them. Note that:

* This consumes less memory than shuffling the data directly.

* By default no shuffling occurs, including for the (stratified) K
  fold cross- validation performed by specifying "cv=some_integer" to
  "cross_val_score()", grid search, etc. Keep in mind that
  "train_test_split()" still returns a random split.

* The "random_state" parameter defaults to "None", meaning that the
  shuffling will be different every time "KFold(..., shuffle=True)" is
  iterated. However, "GridSearchCV" will use the same shuffling for
  each set of parameters validated by a single call to its "fit"
  method.

* To get identical results for each split, set "random_state" to an
  integer.

For more details on how to control the randomness of cv splitters and
avoid common pitfalls, see Controlling randomness.

3.1.4. Cross validation and model selection

Cross validation iterators can also be used to directly perform model
selection using Grid Search for the optimal hyperparameters of the
model. This is the topic of the next section: Tuning the hyper-
parameters of an estimator.

3.1.5. Permutation test score

"permutation_test_score()" offers another way to evaluate the

performance of classifiers. It provides a permutation-based p-value,
which represents how likely an observed performance of the classifier
would be obtained by chance. The null hypothesis in this test is that
the classifier fails to leverage any statistical dependency between
the features and the labels to make correct predictions on left out
data. "permutation_test_score()" generates a null distribution by
calculating *n_permutations* different permutations of the data. In
each permutation the labels are randomly shuffled, thereby removing
any dependency between the features and the labels. The p-value output
is the fraction of permutations for which the average cross-validation
score obtained by the model is better than the cross-validation score
obtained by the model using the original data. For reliable results
"n_permutations" should typically be larger than 100 and "cv" between
3-10 folds.

A low p-value provides evidence that the dataset contains real
dependency between features and labels and the classifier was able to
utilize this to obtain good results. A high p-value could be due to a
lack of dependency between features and labels (there is no difference
in feature values between the classes) or because the classifier was
not able to use the dependency in the data. In the latter case, using

a more appropriate classifier that is able to utilize the structure in
the data, would result in a lower p-value.

Cross-validation provides information about how well a classifier
generalizes, specifically the range of expected errors of the
classifier. However, a classifier trained on a high dimensional
dataset with no structure may still perform better than expected on
cross-validation, just by chance. This can typically happen with small
datasets with less than a few hundred samples.
"permutation_test_score()" provides information on whether the
classifier has found a real class structure and can help in evaluating
the performance of the classifier.

It is important to note that this test has been shown to produce low
p-values even if there is only weak structure in the data because in
the corresponding permutated datasets there is absolutely no
structure. This test is therefore only able to show when the model
reliably outperforms random guessing.

Finally, "permutation_test_score()" is computed using brute force and
internally fits "(n_permutations + 1) * n_cv" models. It is therefore
only tractable with small datasets for which fitting an individual
model is very fast.

Examples
^^^^^^^^

* *sphx_glr_auto_examples_model_selection_plot_permutation_tests_for_
  classification.py*

* Ojala and Garriga. Permutation Tests for Studying Classifier

Examples
^^^^^^^^

  Performance. J. Mach. Learn. Res. 2010.

1.2. Linear and Quadratic Discriminant Analysis
***********************************************

Linear Discriminant Analysis ("LinearDiscriminantAnalysis") and
Quadratic Discriminant Analysis ("QuadraticDiscriminantAnalysis") are
two classic classifiers, with, as their names suggest, a linear and a
quadratic decision surface, respectively.

These classifiers are attractive because they have closed-form
solutions that can be easily computed, are inherently multiclass, have
proven to work well in practice, and have no hyperparameters to tune.

[image: ldaqda][image]

The plot shows decision boundaries for Linear Discriminant Analysis
and Quadratic Discriminant Analysis. The bottom row demonstrates that
Linear Discriminant Analysis can only learn linear boundaries, while
Quadratic Discriminant Analysis can learn quadratic boundaries and is
therefore more flexible.

Examples:
^^^^^^^^^

*sphx_glr_auto_examples_classification_plot_lda_qda.py*: Comparison of
LDA and QDA on synthetic data.

1.2.1. Dimensionality reduction using Linear Discriminant Analysis
==================================================================

"LinearDiscriminantAnalysis" can be used to perform supervised
dimensionality reduction, by projecting the input data to a linear
subspace consisting of the directions which maximize the separation
between classes (in a precise sense discussed in the mathematics
section below). The dimension of the output is necessarily less than
the number of classes, so this is in general a rather strong

dimensionality reduction, and only makes sense in a multiclass
setting.

This is implemented in the *transform* method. The desired
dimensionality can be set using the "n_components" parameter. This
parameter has no influence on the *fit* and *predict* methods.

Examples:
^^^^^^^^^

*sphx_glr_auto_examples_decomposition_plot_pca_vs_lda.py*: Comparison
of LDA and PCA for dimensionality reduction of the Iris dataset

1.2.2. Mathematical formulation of the LDA and QDA classifiers
==============================================================

Both LDA and QDA can be derived from simple probabilistic models which
model the class conditional distribution of the data P(X|y=k) for each
class k. Predictions can then be obtained by using Bayes' rule, for
each training sample x \in \mathcal{R}^d:

   P(y=k | x) = \frac{P(x | y=k) P(y=k)}{P(x)} = \frac{P(x | y=k) P(y
   = k)}{ \sum_{l} P(x | y=l) \cdot P(y=l)}

and we select the class k which maximizes this posterior probability.

More specifically, for linear and quadratic discriminant analysis,
P(x|y) is modeled as a multivariate Gaussian distribution with
density:

   P(x | y=k) = \frac{1}{(2\pi)^{d/2}
   |\Sigma_k|^{1/2}}\exp\left(-\frac{1}{2} (x-\mu_k)^t \Sigma_k^{-1}
   (x-\mu_k)\right)

where d is the number of features.

1.2.2.1. QDA

density:

According to the model above, the log of the posterior is:

   \log P(y=k | x) &= \log P(x | y=k) + \log P(y = k) + Cst \\ &=
   -\frac{1}{2} \log |\Sigma_k| -\frac{1}{2} (x-\mu_k)^t \Sigma_k^{-1}
   (x-\mu_k) + \log P(y = k) + Cst,

where the constant term Cst corresponds to the denominator P(x), in
addition to other constant terms from the Gaussian. The predicted
class is the one that maximises this log-posterior.

Note:

  **Relation with Gaussian Naive Bayes**If in the QDA model one
  assumes that the covariance matrices are diagonal, then the inputs
  are assumed to be conditionally independent in each class, and the
  resulting classifier is equivalent to the Gaussian Naive Bayes
  classifier "naive_bayes.GaussianNB".

1.2.2.2. LDA

LDA is a special case of QDA, where the Gaussians for each class are
assumed to share the same covariance matrix: \Sigma_k = \Sigma for all
k. This reduces the log posterior to:

   \log P(y=k | x) = -\frac{1}{2} (x-\mu_k)^t \Sigma^{-1} (x-\mu_k) +
   \log P(y = k) + Cst.

The term (x-\mu_k)^t \Sigma^{-1} (x-\mu_k) corresponds to the
Mahalanobis Distance between the sample x and the mean \mu_k. The
Mahalanobis distance tells how close x is from \mu_k, while also
accounting for the variance of each feature. We can thus interpret LDA

as assigning x to the class whose mean is the closest in terms of
Mahalanobis distance, while also accounting for the class prior
probabilities.

The log-posterior of LDA can also be written [3] as:

   \log P(y=k | x) = \omega_k^t x + \omega_{k0} + Cst.

where \omega_k = \Sigma^{-1} \mu_k and \omega_{k0} = -\frac{1}{2}
\mu_k^t\Sigma^{-1}\mu_k + \log P (y = k). These quantities correspond
to the *coef_* and *intercept_* attributes, respectively.

From the above formula, it is clear that LDA has a linear decision
surface. In the case of QDA, there are no assumptions on the
covariance matrices \Sigma_k of the Gaussians, leading to quadratic
decision surfaces. See [1] for more details.

1.2.3. Mathematical formulation of LDA dimensionality reduction
===============================================================

First note that the K means \mu_k are vectors in \mathcal{R}^d, and
they lie in an affine subspace H of dimension at most K - 1 (2 points
lie on a line, 3 points lie on a plane, etc).

As mentioned above, we can interpret LDA as assigning x to the class
whose mean \mu_k is the closest in terms of Mahalanobis distance,
while also accounting for the class prior probabilities.
Alternatively, LDA is equivalent to first *sphering* the data so that

the covariance matrix is the identity, and then assigning x to the
closest mean in terms of Euclidean distance (still accounting for the
class priors).

Computing Euclidean distances in this d-dimensional space is
equivalent to first projecting the data points into H, and computing
the distances there (since the other dimensions will contribute
equally to each class in terms of distance). In other words, if x is
closest to \mu_k in the original space, it will also be the case in H.
This shows that, implicit in the LDA classifier, there is a
dimensionality reduction by linear projection onto a K-1 dimensional
space.

We can reduce the dimension even more, to a chosen L, by projecting
onto the linear subspace H_L which maximizes the variance of the
\mu^*_k after projection (in effect, we are doing a form of PCA for
the transformed class means \mu^*_k). This L corresponds to the
"n_components" parameter used in the "transform()" method. See [1] for
more details.

1.2.4. Shrinkage and Covariance Estimator

Shrinkage is a form of regularization used to improve the estimation
of covariance matrices in situations where the number of training
samples is small compared to the number of features. In this scenario,
the empirical sample covariance is a poor estimator, and shrinkage

helps improving the generalization performance of the classifier.
Shrinkage LDA can be used by setting the "shrinkage" parameter of the
"LinearDiscriminantAnalysis" class to 'auto'. This automatically
determines the optimal shrinkage parameter in an analytic way
following the lemma introduced by Ledoit and Wolf [2]. Note that
currently shrinkage only works when setting the "solver" parameter to
'lsqr' or 'eigen'.

The "shrinkage" parameter can also be manually set between 0 and 1. In
particular, a value of 0 corresponds to no shrinkage (which means the
empirical covariance matrix will be used) and a value of 1 corresponds
to complete shrinkage (which means that the diagonal matrix of
variances will be used as an estimate for the covariance matrix).
Setting this parameter to a value between these two extrema will
estimate a shrunk version of the covariance matrix.

The shrunk Ledoit and Wolf estimator of covariance may not always be
the best choice. For example if the distribution of the data is
normally distributed, the Oracle Shrinkage Approximating estimator
"sklearn.covariance.OAS" yields a smaller Mean Squared Error than the
one given by Ledoit and Wolf's formula used with shrinkage="auto". In
LDA, the data are assumed to be gaussian conditionally to the class.
If these assumptions hold, using LDA with the OAS estimator of

covariance will yield a better classification accuracy than if Ledoit
and Wolf or the empirical covariance estimator is used.

The covariance estimator can be chosen using with the
"covariance_estimator" parameter of the
"discriminant_analysis.LinearDiscriminantAnalysis" class. A covariance
estimator should have a *fit* method and a "covariance_" attribute
like all covariance estimators in the "sklearn.covariance" module.

[image: shrinkage][image]

Examples:
^^^^^^^^^

*sphx_glr_auto_examples_classification_plot_lda.py*: Comparison of LDA
classifiers with Empirical, Ledoit Wolf and OAS covariance estimator.

1.2.5. Estimation algorithms

Using LDA and QDA requires computing the log-posterior which depends
on the class priors P(y=k), the class means \mu_k, and the covariance
matrices.

The 'svd' solver is the default solver used for
"LinearDiscriminantAnalysis", and it is the only available solver for
"QuadraticDiscriminantAnalysis". It can perform both classification
and transform (for LDA). As it does not rely on the calculation of the
covariance matrix, the 'svd' solver may be preferable in situations
where the number of features is large. The 'svd' solver cannot be used
with shrinkage. For QDA, the use of the SVD solver relies on the fact

that the covariance matrix \Sigma_k is, by definition, equal to
\frac{1}{n - 1} X_k^tX_k = \frac{1}{n - 1} V S^2 V^t where V comes
from the SVD of the (centered) matrix: X_k = U S V^t. It turns out
that we can compute the log-posterior above without having to
explicitly compute \Sigma: computing S and V via the SVD of X is
enough. For LDA, two SVDs are computed: the SVD of the centered input
matrix X and the SVD of the class-wise mean vectors.

The 'lsqr' solver is an efficient algorithm that only works for
classification. It needs to explicitly compute the covariance matrix
\Sigma, and supports shrinkage and custom covariance estimators. This
solver computes the coefficients \omega_k = \Sigma^{-1}\mu_k by
solving for \Sigma \omega = \mu_k, thus avoiding the explicit
computation of the inverse \Sigma^{-1}.

The 'eigen' solver is based on the optimization of the between class
scatter to within class scatter ratio. It can be used for both
classification and transform, and it supports shrinkage. However, the
'eigen' solver needs to compute the covariance matrix, so it might not
be suitable for situations with a high number of features.

[1] "The Elements of Statistical Learning", Hastie T., Tibshirani R.,
    Friedman J., Section 4.3, p.106-119, 2008.

[2] Ledoit O, Wolf M. Honey, I Shrunk the Sample Covariance Matrix.

    The Journal of Portfolio Management 30(4), 110-119, 2004.

[3] R. O. Duda, P. E. Hart, D. G. Stork. Pattern Classification
    (Second Edition), section 2.6.2.

6.4. Imputation of missing values

For various reasons, many real world datasets contain missing values,
often encoded as blanks, NaNs or other placeholders. Such datasets
however are incompatible with scikit-learn estimators which assume
that all values in an array are numerical, and that all have and hold
meaning. A basic strategy to use incomplete datasets is to discard
entire rows and/or columns containing missing values. However, this
comes at the price of losing data which may be valuable (even though
incomplete). A better strategy is to impute the missing values, i.e.,
to infer them from the known part of the data. See the glossary entry
on *imputation*.

6.4.1. Univariate vs. Multivariate Imputation

One type of imputation algorithm is univariate, which imputes values
in the i-th feature dimension using only non-missing values in that
feature dimension (e.g. "impute.SimpleImputer"). By contrast,
multivariate imputation algorithms use the entire set of available
feature dimensions to estimate the missing values (e.g.
"impute.IterativeImputer").

6.4.2. Univariate feature imputation

The "SimpleImputer" class provides basic strategies for imputing
missing values. Missing values can be imputed with a provided constant
value, or using the statistics (mean, median or most frequent) of each
column in which the missing values are located. This class also allows

for different missing values encodings.

The following snippet demonstrates how to replace missing values,
encoded as "np.nan", using the mean value of the columns (axis 0) that
contain the missing values:

   >>> from sklearn.impute import SimpleImputer
   >>> imp = SimpleImputer(missing_values=np.nan, strategy='mean')
   >>> imp.fit([[1, 2], [np.nan, 3], [7, 6]])
   SimpleImputer()
   >>> X = [[np.nan, 2], [6, np.nan], [7, 6]]
   >>> print(imp.transform(X))
   [[4.          2.        ]
    [6.          3.666...]
    [7.          6.        ]]

The "SimpleImputer" class also supports sparse matrices:

   >>> import scipy.sparse as sp
   >>> X = sp.csc_matrix([[1, 2], [0, -1], [8, 4]])
   >>> imp = SimpleImputer(missing_values=-1, strategy='mean')
   >>> imp.fit(X)
   SimpleImputer(missing_values=-1)
   >>> X_test = sp.csc_matrix([[-1, 2], [6, -1], [7, 6]])
   >>> print(imp.transform(X_test).toarray())
   [[3. 2.]
    [6. 3.]
    [7. 6.]]

Note that this format is not meant to be used to implicitly store
missing values in the matrix because it would densify it at transform
time. Missing values encoded by 0 must be used with dense input.

The "SimpleImputer" class also supports categorical data represented
as string values or pandas categoricals when using the
"'most_frequent'" or "'constant'" strategy:

   >>> import pandas as pd

   >>> df = pd.DataFrame([["a", "x"],
   ...                    [np.nan, "y"],
   ...                    ["a", np.nan],
   ...                    ["b", "y"]], dtype="category")
   ...
   >>> imp = SimpleImputer(strategy="most_frequent")
   >>> print(imp.fit_transform(df))
   [['a' 'x']
    ['a' 'y']
    ['a' 'y']
    ['b' 'y']]

6.4.3. Multivariate feature imputation

A more sophisticated approach is to use the "IterativeImputer" class,
which models each feature with missing values as a function of other
features, and uses that estimate for imputation. It does so in an
iterated round-robin fashion: at each step, a feature column is
designated as output "y" and the other feature columns are treated as
inputs "X". A regressor is fit on "(X, y)" for known "y". Then, the
regressor is used to predict the missing values of "y".  This is done
for each feature in an iterative fashion, and then is repeated for
"max_iter" imputation rounds. The results of the final imputation
round are returned.

Note:

  This estimator is still **experimental** for now: default parameters
  or details of behaviour might change without any deprecation cycle.
  Resolving the following issues would help stabilize
  "IterativeImputer": convergence criteria (>>:issue:`14338`<<),

or details of behaviour might change without any deprecation cycle.
  default estimators (>>:issue:`13286`<<), and use of random state
  (>>:issue:`15611`<<). To use it, you need to explicitly import
  "enable_iterative_imputer".

   >>> from sklearn.experimental import enable_iterative_imputer
   >>> from sklearn.impute import IterativeImputer
   >>> imp = IterativeImputer(max_iter=10, random_state=0)
   >>> imp.fit([[1, 2], [3, 6], [4, 8], [np.nan, 3], [7, np.nan]])
   IterativeImputer(random_state=0)
   >>> X_test = [[np.nan, 2], [6, np.nan], [np.nan, 6]]
   >>> # the model learns that the second feature is double the first
   >>> print(np.round(imp.transform(X_test)))
   [[ 1.  2.]
    [ 6. 12.]
    [ 3.  6.]]

Both "SimpleImputer" and "IterativeImputer" can be used in a Pipeline
as a way to build a composite estimator that supports imputation. See
*sphx_glr_auto_examples_impute_plot_missing_values.py*.

6.4.3.1. Flexibility of IterativeImputer

There are many well-established imputation packages in the R data
science ecosystem: Amelia, mi, mice, missForest, etc. missForest is
popular, and turns out to be a particular instance of different
sequential imputation algorithms that can all be implemented with
"IterativeImputer" by passing in different regressors to be used for

predicting missing feature values. In the case of missForest, this
regressor is a Random Forest. See *sphx_glr_auto_examples_impute_plot
_iterative_imputer_variants_comparison.py*.

6.4.3.2. Multiple vs. Single Imputation

In the statistics community, it is common practice to perform multiple
imputations, generating, for example, "m" separate imputations for a
single feature matrix. Each of these "m" imputations is then put
through the subsequent analysis pipeline (e.g. feature engineering,
clustering, regression, classification). The "m" final analysis
results (e.g. held-out validation errors) allow the data scientist to
obtain understanding of how analytic results may differ as a
consequence of the inherent uncertainty caused by the missing values.
The above practice is called multiple imputation.

Our implementation of "IterativeImputer" was inspired by the R MICE
package (Multivariate Imputation by Chained Equations) [1], but
differs from it by returning a single imputation instead of multiple
imputations.  However, "IterativeImputer" can also be used for
multiple imputations by applying it repeatedly to the same dataset
with different random seeds when "sample_posterior=True". See [2],
chapter 4 for more discussion on multiple vs. single imputations.

It is still an open problem as to how useful single vs. multiple
imputation is in the context of prediction and classification when the
user is not interested in measuring uncertainty due to missing values.

Note that a call to the "transform" method of "IterativeImputer" is
not allowed to change the number of samples. Therefore multiple
imputations cannot be achieved by a single call to "transform".

6.4.3.3. References

[1] Stef van Buuren, Karin Groothuis-Oudshoorn (2011). "mice:
    Multivariate Imputation by Chained Equations in R". Journal of
    Statistical Software 45: 1-67.

[2] Roderick J A Little and Donald B Rubin (1986). "Statistical
    Analysis with Missing Data". John Wiley & Sons, Inc., New York,
    NY, USA.

6.4.4. Nearest neighbors imputation

The "KNNImputer" class provides imputation for filling in missing
values using the k-Nearest Neighbors approach. By default, a euclidean
distance metric that supports missing values,
"nan_euclidean_distances()", is used to find the nearest neighbors.
Each missing feature is imputed using values from "n_neighbors"
nearest neighbors that have a value for the feature. The feature of
the neighbors are averaged uniformly or weighted by distance to each

neighbor. If a sample has more than one feature missing, then the
neighbors for that sample can be different depending on the particular
feature being imputed. When the number of available neighbors is less
than *n_neighbors* and there are no defined distances to the training
set, the training set average for that feature is used during
imputation. If there is at least one neighbor with a defined distance,
the weighted or unweighted average of the remaining neighbors will be
used during imputation. If a feature is always missing in training, it
is removed during *transform*. For more information on the
methodology, see ref. [OL2001].

encoded as "np.nan", using the mean feature value of the two nearest
neighbors of samples with missing values:

   >>> from sklearn.impute import KNNImputer
   >>> nan = np.nan
   >>> X = [[1, 2, nan], [3, 4, 3], [nan, 6, 5], [8, 8, 7]]
   >>> imputer = KNNImputer(n_neighbors=2, weights="uniform")
   >>> imputer.fit_transform(X)
   array([[1. , 2. , 4. ],
          [3. , 4. , 3. ],
          [5.5, 6. , 5. ],
          [8. , 8. , 7. ]])

[OL2001] Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown,

array([[1. , 2. , 4. ],

         Trevor Hastie, Robert Tibshirani, David Botstein and Russ B.
         Altman, Missing value estimation methods for DNA microarrays,
         BIOINFORMATICS Vol. 17 no. 6, 2001 Pages 520-525.

6.4.5. Keeping the number of features constants

By default, the scikit-learn imputers will drop fully empty features,
i.e. columns containing only missing values. For instance:

   >>> imputer = SimpleImputer()
   >>> X = np.array([[np.nan, 1], [np.nan, 2], [np.nan, 3]])
   array([[1.],
          [2.],
          [3.]])

The first feature in *X* containing only *np.nan* was dropped after
the imputation. While this feature will not help in predictive
setting, dropping the columns will change the shape of *X* which could
be problematic when using imputers in a more complex machine-learning
pipeline. The parameter *keep_empty_features* offers the option to
keep the empty features by imputing with a constant values. In most of
the cases, this constant value is zero:

   >>> imputer.set_params(keep_empty_features=True)
   SimpleImputer(keep_empty_features=True)
   array([[0., 1.],
          [0., 2.],
          [0., 3.]])

6.4.6. Marking imputed values

SimpleImputer(keep_empty_features=True)
          [0., 2.],
          [0., 3.]])

The "MissingIndicator" transformer is useful to transform a dataset
into corresponding binary matrix indicating the presence of missing
values in the dataset. This transformation is useful in conjunction
with imputation. When using imputation, preserving the information
about which values had been missing can be informative. Note that both
the "SimpleImputer" and "IterativeImputer" have the boolean parameter
"add_indicator" ("False" by default) which when set to "True" provides
a convenient way of stacking the output of the "MissingIndicator"
transformer with the output of the imputer.

"NaN" is usually used as the placeholder for missing values. However,
it enforces the data type to be float. The parameter "missing_values"
allows to specify other placeholder such as integer. In the following
example, we will use "-1" as missing values:

   >>> from sklearn.impute import MissingIndicator
   >>> X = np.array([[-1, -1, 1, 3],
   ...               [4, -1, 0, -1],
   ...               [8, -1, 1, 0]])
   >>> indicator = MissingIndicator(missing_values=-1)
   >>> mask_missing_values_only = indicator.fit_transform(X)
   >>> mask_missing_values_only
   array([[ True,  True, False],
          [False,  True,  True],
          [False,  True, False]])

The "features" parameter is used to choose the features for which the

>>> mask_missing_values_only

mask is constructed. By default, it is "'missing-only'" which returns
the imputer mask of the features containing missing values at "fit"
time:

   >>> indicator.features_
   array([0, 1, 3])

The "features" parameter can be set to "'all'" to return all features
whether or not they contain missing values:

   >>> indicator = MissingIndicator(missing_values=-1, features="all")
   >>> mask_all = indicator.fit_transform(X)
   >>> mask_all
   array([[ True,  True, False, False],
          [False,  True, False,  True],
          [False,  True, False, False]])
   array([0, 1, 2, 3])

When using the "MissingIndicator" in a "Pipeline", be sure to use the
"FeatureUnion" or "ColumnTransformer" to add the indicator features to
the regular features. First we obtain the *iris* dataset, and add some
missing values to it.

>>> from sklearn.impute import SimpleImputer, MissingIndicator
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.pipeline import FeatureUnion, make_pipeline
>>> from sklearn.tree import DecisionTreeClassifier
>>> X, y = load_iris(return_X_y=True)
>>> mask = np.random.randint(0, 2, size=X.shape).astype(bool)
>>> X[mask] = np.nan
>>> X_train, X_test, y_train, _ = train_test_split(X, y, test_size=100,

...                                                random_state=0)

Now we create a "FeatureUnion". All features will be imputed using
"SimpleImputer", in order to enable classifiers to work with this
data. Additionally, it adds the indicator variables from
"MissingIndicator".

>>> transformer = FeatureUnion(
...     transformer_list=[
...         ('features', SimpleImputer(strategy='mean')),
...         ('indicators', MissingIndicator())])
>>> transformer = transformer.fit(X_train, y_train)
>>> results = transformer.transform(X_test)
>>> results.shape
(100, 8)

Of course, we cannot use the transformer to make any predictions. We
should wrap this in a "Pipeline" with a classifier (e.g., a
"DecisionTreeClassifier") to be able to make predictions.

>>> clf = make_pipeline(transformer, DecisionTreeClassifier())
>>> clf = clf.fit(X_train, y_train)
>>> results = clf.predict(X_test)
(100,)

6.4.7. Estimators that handle NaN values

Some estimators are designed to handle NaN values without
preprocessing. Below is the list of these estimators, classified by
type (cluster, regressor, classifier, transform):

6.9. Transforming the prediction target ("y")

These are transformers that are not intended to be used on features,
only on supervised learning targets. See also Transforming target in
regression if you want to transform the prediction target for
learning, but evaluate the model in the original (untransformed)
space.

6.9.1. Label binarization

6.9.1.1. LabelBinarizer

"LabelBinarizer" is a utility class to help create a *label indicator
matrix* from a list of *multiclass* labels:

   >>> lb = preprocessing.LabelBinarizer()
   >>> lb.fit([1, 2, 6, 4, 2])
   LabelBinarizer()
   >>> lb.classes_
   array([1, 2, 4, 6])
   >>> lb.transform([1, 6])
          [0, 0, 0, 1]])

Using this format can enable multiclass classification in estimators
that support the label indicator matrix format.

Warning:

  LabelBinarizer is not needed if you are using an estimator that
  already supports *multiclass* data.

For more information about multiclass classification, refer to
Multiclass classification.

6.9.1.2. MultiLabelBinarizer

In *multilabel* learning, the joint set of binary classification tasks
is expressed with a label binary indicator array: each sample is one
row of a 2d array of shape (n_samples, n_classes) with binary values
where the one, i.e. the non zero elements, corresponds to the subset

of labels for that sample. An array such as "np.array([[1, 0, 0], [0,
1, 1], [0, 0, 0]])" represents label 0 in the first sample, labels 1
and 2 in the second sample, and no labels in the third sample.

Producing multilabel data as a list of sets of labels may be more
intuitive. The "MultiLabelBinarizer" transformer can be used to
convert between a collection of collections of labels and the
indicator format:

   >>> y = [[2, 3, 4], [2], [0, 1, 3], [0, 1, 2, 3, 4], [0, 1, 2]]
   >>> MultiLabelBinarizer().fit_transform(y)
   array([[0, 0, 1, 1, 1],
          [0, 0, 1, 0, 0],
          [1, 1, 0, 1, 0],
          [1, 1, 1, 1, 1],
          [1, 1, 1, 0, 0]])

For more information about multilabel classification, refer to
Multilabel classification.

6.9.2. Label encoding

"LabelEncoder" is a utility class to help normalize labels such that
they contain only values between 0 and n_classes-1. This is sometimes
useful for writing efficient Cython routines. "LabelEncoder" can be
used as follows:

   >>> le = preprocessing.LabelEncoder()
   >>> le.fit([1, 2, 2, 6])
   LabelEncoder()
   >>> le.classes_
   array([1, 2, 6])
   >>> le.transform([1, 1, 2, 6])
   array([0, 0, 1, 2])
   >>> le.inverse_transform([0, 0, 1, 2])
   array([1, 1, 2, 6])

>>> le.fit([1, 2, 2, 6])

It can also be used to transform non-numerical labels (as long as they
are hashable and comparable) to numerical labels:

   >>> le.fit(["paris", "paris", "tokyo", "amsterdam"])
   >>> list(le.classes_)
   ['amsterdam', 'paris', 'tokyo']
   >>> le.transform(["tokyo", "tokyo", "paris"])
   array([2, 2, 1])
   >>> list(le.inverse_transform([2, 2, 1]))
   ['tokyo', 'tokyo', 'paris']

6.1. Pipelines and composite estimators
***************************************

Transformers are usually combined with classifiers, regressors or
other estimators to build a composite estimator.  The most common tool
is a Pipeline. Pipeline is often used in combination with FeatureUnion
which concatenates the output of transformers into a composite feature
space.  TransformedTargetRegressor deals with transforming the
*target* (i.e. log-transform *y*). In contrast, Pipelines only
transform the observed data (*X*).

6.1.1. Pipeline: chaining estimators

"Pipeline" can be used to chain multiple estimators into one. This is
useful as there is often a fixed sequence of steps in processing the
data, for example feature selection, normalization and classification.
"Pipeline" serves multiple purposes here:

Convenience and encapsulation
   You only have to call *fit* and *predict* once on your data to fit
   a whole sequence of estimators.

Joint parameter selection
   You can grid search over parameters of all estimators in the
   pipeline at once.

Safety
   Pipelines help avoid leaking statistics from your test data into
   the trained model in cross-validation, by ensuring that the same
   samples are used to train the transformers and predictors.

All estimators in a pipeline, except the last one, must be
transformers (i.e. must have a *transform* method). The last estimator
may be any type (transformer, classifier, etc.).

6.1.1.1. Usage

6.1.1.1.1. Construction

The "Pipeline" is built using a list of "(key, value)" pairs, where
the "key" is a string containing the name you want to give this step
and "value" is an estimator object:

   >>> from sklearn.decomposition import PCA
   >>> estimators = [('reduce_dim', PCA()), ('clf', SVC())]
   >>> pipe = Pipeline(estimators)
   >>> pipe
   Pipeline(steps=[('reduce_dim', PCA()), ('clf', SVC())])

The utility function "make_pipeline()" is a shorthand for constructing
pipelines; it takes a variable number of estimators and returns a
pipeline, filling in the names automatically:

   >>> from sklearn.preprocessing import Binarizer
   >>> make_pipeline(Binarizer(), MultinomialNB())
   Pipeline(steps=[('binarizer', Binarizer()), ('multinomialnb', MultinomialNB())])

6.1.1.1.2. Accessing steps

The estimators of a pipeline are stored as a list in the "steps"
attribute, but can be accessed by index or name by indexing (with
"[idx]") the Pipeline:

   >>> pipe.steps[0]
   ('reduce_dim', PCA())
   >>> pipe[0]
   PCA()
   >>> pipe['reduce_dim']
   PCA()

Pipeline's *named_steps* attribute allows accessing steps by name with

   PCA()
   PCA()

tab completion in interactive environments:

   >>> pipe.named_steps.reduce_dim is pipe['reduce_dim']
   True

A sub-pipeline can also be extracted using the slicing notation
commonly used for Python Sequences such as lists or strings (although
only a step of 1 is permitted). This is convenient for performing only
some of the transformations (or their inverse):

>>> pipe[:1]
Pipeline(steps=[('reduce_dim', PCA())])
>>> pipe[-1:]
Pipeline(steps=[('clf', SVC())])

6.1.1.1.3. Nested parameters

Parameters of the estimators in the pipeline can be accessed using the
"<estimator>__<parameter>" syntax:

   >>> pipe.set_params(clf__C=10)
   Pipeline(steps=[('reduce_dim', PCA()), ('clf', SVC(C=10))])

This is particularly important for doing grid searches:

   >>> param_grid = dict(reduce_dim__n_components=[2, 5, 10],
   ...                   clf__C=[0.1, 10, 100])
   >>> grid_search = GridSearchCV(pipe, param_grid=param_grid)

Individual steps may also be replaced as parameters, and non-final
steps may be ignored by setting them to "'passthrough'":

   >>> param_grid = dict(reduce_dim=['passthrough', PCA(5), PCA(10)],
   ...                   clf=[SVC(), LogisticRegression()],

>>> from sklearn.linear_model import LogisticRegression

The estimators of the pipeline can be retrieved by index:

>>> pipe[0]
PCA()

or by name:

   PCA()

To enable model inspection, "Pipeline" has a "get_feature_names_out()"
method, just like all transformers. You can use pipeline slicing to
get the feature names going into each step:

   >>> iris = load_iris()
   >>> pipe = Pipeline(steps=[
   ...    ('select', SelectKBest(k=2)),
   ...    ('clf', LogisticRegression())])
   >>> pipe.fit(iris.data, iris.target)
   Pipeline(steps=[('select', SelectKBest(...)), ('clf', LogisticRegression(...))])
   >>> pipe[:-1].get_feature_names_out()
   array(['x2', 'x3'], ...)

You can also provide custom feature names for the input data using
"get_feature_names_out":

   >>> pipe[:-1].get_feature_names_out(iris.feature_names)
   array(['petal length (cm)', 'petal width (cm)'], ...)

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_feature_selection_plot_feature_selection_pi
  peline.py*

  re_extraction.py*

* *sphx_glr_auto_examples_compose_plot_digits_pipe.py*

peline.py*

* *sphx_glr_auto_examples_miscellaneous_plot_kernel_approximation.py*

* *sphx_glr_auto_examples_svm_plot_svm_anova.py*

* *sphx_glr_auto_examples_compose_plot_compare_reduction.py*

* *sphx_glr_auto_examples_miscellaneous_plot_pipeline_display.py*

See Also:
^^^^^^^^^

* Composite estimators and parameter spaces

6.1.1.2. Notes

Calling "fit" on the pipeline is the same as calling "fit" on each
estimator in turn, "transform" the input and pass it on to the next
step. The pipeline has all the methods that the last estimator in the
pipeline has, i.e. if the last estimator is a classifier, the
"Pipeline" can be used as a classifier. If the last estimator is a
transformer, again, so is the pipeline.

6.1.1.3. Caching transformers: avoid repeated computation
---------------------------------------------------------

Fitting transformers may be computationally expensive. With its
"memory" parameter set, "Pipeline" will cache each transformer after
calling "fit". This feature is used to avoid computing the fit
transformers within a pipeline if the parameters and input data are
identical. A typical example is the case of a grid search in which the
transformers can be fitted only once and reused for each
configuration.

The parameter "memory" is needed in order to cache the transformers.

"memory" can be either a string containing the directory where to
cache the transformers or a joblib.Memory object:

   >>> from tempfile import mkdtemp
   >>> from shutil import rmtree
   >>> cachedir = mkdtemp()
   >>> pipe = Pipeline(estimators, memory=cachedir)
   >>> pipe
   Pipeline(memory=...,
            steps=[('reduce_dim', PCA()), ('clf', SVC())])
   >>> # Clear the cache directory when you don't need it anymore
   >>> rmtree(cachedir)

Warning:

  **Side effect of caching transformers**Using a "Pipeline" without
  cache enabled, it is possible to inspect the original instance such
  as:

     >>> from sklearn.datasets import load_digits
     >>> X_digits, y_digits = load_digits(return_X_y=True)
     >>> pca1 = PCA()
     >>> svm1 = SVC()
     >>> pipe = Pipeline([('reduce_dim', pca1), ('clf', svm1)])
     >>> pipe.fit(X_digits, y_digits)
     Pipeline(steps=[('reduce_dim', PCA()), ('clf', SVC())])
     >>> # The pca instance can be inspected directly
     >>> print(pca1.components_)
         [[-1.77484909e-19  ... 4.07058917e-18]]

  Enabling caching triggers a clone of the transformers before
  fitting. Therefore, the transformer instance given to the pipeline

[[-1.77484909e-19  ... 4.07058917e-18]]

  cannot be inspected directly. In following example, accessing the
  "PCA" instance "pca2" will raise an "AttributeError" since "pca2"
  will be an unfitted transformer. Instead, use the attribute
  "named_steps" to inspect estimators within the pipeline:

     >>> cachedir = mkdtemp()
     >>> pca2 = PCA()
     >>> svm2 = SVC()
     >>> cached_pipe = Pipeline([('reduce_dim', pca2), ('clf', svm2)],
     ...                        memory=cachedir)
     >>> cached_pipe.fit(X_digits, y_digits)
     Pipeline(memory=...,
             steps=[('reduce_dim', PCA()), ('clf', SVC())])
     >>> print(cached_pipe.named_steps['reduce_dim'].components_)
     >>> # Remove the cache directory
     >>> rmtree(cachedir)

Examples:
^^^^^^^^^

6.1.2. Transforming target in regression

"TransformedTargetRegressor" transforms the targets "y" before fitting
a regression model. The predictions are mapped back to the original
space via an inverse transform. It takes as an argument the regressor
that will be used for prediction, and the transformer that will be
applied to the target variable:

   >>> from sklearn.compose import TransformedTargetRegressor
   >>> from sklearn.preprocessing import QuantileTransformer
   >>> X, y = X[:2000, :], y[:2000]  # select a subset of data
   >>> transformer = QuantileTransformer(output_distribution='normal')
   >>> regressor = LinearRegression()
   >>> regr = TransformedTargetRegressor(regressor=regressor,
   ...                                   transformer=transformer)
   >>> regr.fit(X_train, y_train)
   TransformedTargetRegressor(...)
   >>> print('R2 score: {0:.2f}'.format(regr.score(X_test, y_test)))
   R2 score: 0.61
   >>> raw_target_regr = LinearRegression().fit(X_train, y_train)
   >>> print('R2 score: {0:.2f}'.format(raw_target_regr.score(X_test, y_test)))
   R2 score: 0.59

For simple transformations, instead of a Transformer object, a pair of
functions can be passed, defining the transformation and its inverse
mapping:

   >>> def func(x):
   ...     return np.log(x)
   >>> def inverse_func(x):
   ...     return np.exp(x)

Subsequently, the object is created as:

mapping:

   ...                                   func=func,
   ...                                   inverse_func=inverse_func)
   R2 score: 0.51

By default, the provided functions are checked at each fit to be the
inverse of each other. However, it is possible to bypass this checking
by setting "check_inverse" to "False":

   ...     return x
   ...                                   inverse_func=inverse_func,
   ...                                   check_inverse=False)
   R2 score: -1.57

Note:

  The transformation can be triggered by setting either "transformer"
  or the pair of functions "func" and "inverse_func". However, setting
  both options will raise an error.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_compose_plot_transformed_target.py*

6.1.3. FeatureUnion: composite feature spaces

Examples:
^^^^^^^^^

"FeatureUnion" combines several transformer objects into a new
transformer that combines their output. A "FeatureUnion" takes a list
of transformer objects. During fitting, each of these is fit to the
data independently. The transformers are applied in parallel, and the
feature matrices they output are concatenated side-by-side into a
larger matrix.

When you want to apply different transformations to each field of the
data, see the related class "ColumnTransformer" (see user guide).

"FeatureUnion" serves the same purposes as "Pipeline" - convenience
and joint parameter estimation and validation.

"FeatureUnion" and "Pipeline" can be combined to create complex
models.

(A "FeatureUnion" has no way of checking whether two transformers
might produce identical features. It only produces a union when the
feature sets are disjoint, and making sure they are is the caller's
responsibility.)

6.1.3.1. Usage

A "FeatureUnion" is built using a list of "(key, value)" pairs, where
the "key" is the name you want to give to a given transformation (an
arbitrary string; it only serves as an identifier) and "value" is an
estimator object:

   >>> from sklearn.pipeline import FeatureUnion
   >>> from sklearn.decomposition import KernelPCA

   >>> estimators = [('linear_pca', PCA()), ('kernel_pca', KernelPCA())]
   >>> combined = FeatureUnion(estimators)
   >>> combined
   FeatureUnion(transformer_list=[('linear_pca', PCA()),
                                  ('kernel_pca', KernelPCA())])

Like pipelines, feature unions have a shorthand constructor called
"make_union()" that does not require explicit naming of the
components.

Like "Pipeline", individual steps may be replaced using "set_params",
and ignored by setting to "'drop'":

   >>> combined.set_params(kernel_pca='drop')
                                  ('kernel_pca', 'drop')])

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_compose_plot_feature_union.py*

6.1.4. ColumnTransformer for heterogeneous data

Many datasets contain features of different types, say text, floats,
and dates, where each type of feature requires separate preprocessing
or feature extraction steps.  Often it is easiest to preprocess data
before applying scikit-learn methods, for example using pandas.
Processing your data before passing it to scikit-learn might be
problematic for one of the following reasons:

1. Incorporating statistics from test data into the preprocessors
   makes cross-validation scores unreliable (known as *data leakage*),

   for example in the case of scalers or imputing missing values.

2. You may want to include the parameters of the preprocessors in a
   parameter search.

The "ColumnTransformer" helps performing different transformations for
different columns of the data, within a "Pipeline" that is safe from
data leakage and that can be parametrized. "ColumnTransformer" works
on arrays, sparse matrices, and pandas DataFrames.

To each column, a different transformation can be applied, such as
preprocessing or a specific feature extraction method:

   >>> X = pd.DataFrame(
   ...     {'city': ['London', 'London', 'Paris', 'Sallisaw'],
   ...      'title': ["His Last Bow", "How Watson Learned the Trick",
   ...                "A Moveable Feast", "The Grapes of Wrath"],
   ...      'expert_rating': [5, 3, 4, 5],
   ...      'user_rating': [4, 5, 4, 3]})

For this data, we might want to encode the "'city'" column as a
categorical variable using "OneHotEncoder" but apply a
"CountVectorizer" to the "'title'" column. As we might use multiple
feature extraction methods on the same column, we give each
transformer a unique name, say "'city_category'" and "'title_bow'". By
default, the remaining rating columns are ignored
("remainder='drop'"):

   >>> from sklearn.compose import ColumnTransformer

   >>> from sklearn.preprocessing import OneHotEncoder
   >>> column_trans = ColumnTransformer(
   ...     [('categories', OneHotEncoder(dtype='int'), ['city']),
   ...      ('title_bow', CountVectorizer(), 'title')],
   ...     remainder='drop', verbose_feature_names_out=False)

   >>> column_trans.fit(X)
   ColumnTransformer(transformers=[('categories', OneHotEncoder(dtype='int'),
                                    ['city']),
                                   ('title_bow', CountVectorizer(), 'title')],
                     verbose_feature_names_out=False)

   >>> column_trans.get_feature_names_out()
   array(['city_London', 'city_Paris', 'city_Sallisaw', 'bow', 'feast',
   'grapes', 'his', 'how', 'last', 'learned', 'moveable', 'of', 'the',
    'trick', 'watson', 'wrath'], ...)

   >>> column_trans.transform(X).toarray()
   array([[1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0],
          [1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0],
          [0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],
          [0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1]]...)

In the above example, the "CountVectorizer" expects a 1D array as
input and therefore the columns were specified as a string

[0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1]]...)

("'title'"). However, "OneHotEncoder" as most of other transformers
expects 2D data, therefore in that case you need to specify the column
as a list of strings ("['city']").

Apart from a scalar or a single item list, the column selection can be
specified as a list of multiple items, an integer array, a slice, a
boolean mask, or with a "make_column_selector()". The
"make_column_selector()" is used to select columns based on data type
or column name:

   >>> from sklearn.compose import make_column_selector
   >>> ct = ColumnTransformer([
   ...       ('scale', StandardScaler(),
   ...       make_column_selector(dtype_include=np.number)),
   ...       ('onehot',
   ...       OneHotEncoder(),
   ...       make_column_selector(pattern='city', dtype_include=object))])
   >>> ct.fit_transform(X)
   array([[ 0.904...,  0.      ,  1. ,  0. ,  0. ],
          [-1.507...,  1.414...,  1. ,  0. ,  0. ],
          [-0.301...,  0.      ,  0. ,  1. ,  0. ],
          [ 0.904..., -1.414...,  0. ,  0. ,  1. ]])

Strings can reference columns if the input is a DataFrame, integers
are always interpreted as the positional columns.

We can keep the remaining rating columns by setting
"remainder='passthrough'". The values are appended to the end of the
transformation:

   ...     [('city_category', OneHotEncoder(dtype='int'),['city']),
   ...     remainder='passthrough')

   >>> column_trans.fit_transform(X)
   array([[1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 5, 4],
          [1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 3, 5],
          [0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 4, 4],
          [0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 5, 3]]...)

The "remainder" parameter can be set to an estimator to transform the
remaining rating columns. The transformed values are appended to the
end of the transformation:

   >>> from sklearn.preprocessing import MinMaxScaler
   ...     [('city_category', OneHotEncoder(), ['city']),
   ...     remainder=MinMaxScaler())

   >>> column_trans.fit_transform(X)[:, -2:]
   array([[1. , 0.5],
          [0. , 1. ],
          [0.5, 0.5],
          [1. , 0. ]])

The "make_column_transformer()" function is available to more easily
create a "ColumnTransformer" object. Specifically, the names will be
given automatically. The equivalent for the above example would be:

   >>> from sklearn.compose import make_column_transformer

   >>> column_trans = make_column_transformer(
   ...     (OneHotEncoder(), ['city']),
   ...     (CountVectorizer(), 'title'),
   >>> column_trans
   ColumnTransformer(remainder=MinMaxScaler(),
                     transformers=[('onehotencoder', OneHotEncoder(), ['city']),
                                   ('countvectorizer', CountVectorizer(),
                                    'title')])

If "ColumnTransformer" is fitted with a dataframe and the dataframe
only has string column names, then transforming a dataframe will use
the column names to select the columns:

   >>> ct = ColumnTransformer(
   ...          [("scale", StandardScaler(), ["expert_rating"])]).fit(X)
   >>> X_new = pd.DataFrame({"expert_rating": [5, 6, 1],
   ...                       "ignored_new_col": [1.2, 0.3, -0.1]})
   >>> ct.transform(X_new)
   array([[ 0.9...],
          [ 2.1...],
          [-3.9...]])

6.1.5. Visualizing Composite Estimators

Estimators are displayed with an HTML representation when shown in a
jupyter notebook. This is useful to diagnose or visualize a Pipeline
with many estimators. This visualization is activated by default:

   >>> column_trans  

It can be deactivated by setting the *display* option in
"set_config()" to 'text':

   >>> from sklearn import set_config
   >>> set_config(display='text')  
   >>> # displays text representation in a jupyter context

An example of the HTML output can be seen in the **HTML representation
of Pipeline** section of *sphx_glr_auto_examples_compose_plot_column_
transformer_mixed_types.py*. As an alternative, the HTML can be
written to a file using "estimator_html_repr()":

   >>> from sklearn.utils import estimator_html_repr
   >>> with open('my_estimator.html', 'w') as f:  
   ...     f.write(estimator_html_repr(clf))

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_compose_plot_column_transformer.py*

* *sphx_glr_auto_examples_compose_plot_column_transformer_mixed_types
  .py*

6.7. Kernel Approximation
*************************

This submodule contains functions that approximate the feature
mappings that correspond to certain kernels, as they are used for
example in support vector machines (see Support Vector Machines). The
following feature functions perform non-linear transformations of the
input, which can serve as a basis for linear classification or other

The advantage of using approximate explicit feature maps compared to
the kernel trick, which makes use of feature maps implicitly, is that
explicit mappings can be better suited for online learning and can
significantly reduce the cost of learning with very large datasets.
Standard kernelized SVMs do not scale well to large datasets, but
using an approximate kernel map it is possible to use much more
efficient linear SVMs. In particular, the combination of kernel map
approximations with "SGDClassifier" can make non-linear learning on
large datasets possible.

Since there has not been much empirical work using approximate
embeddings, it is advisable to compare results against exact kernel
methods when possible.

See also:

  Polynomial regression: extending linear models with basis functions
  for an exact polynomial transformation.

6.7.1. Nystroem Method for Kernel Approximation

The Nystroem method, as implemented in "Nystroem" is a general method
for low-rank approximations of kernels. It achieves this by

essentially subsampling the data on which the kernel is evaluated. By
default "Nystroem" uses the "rbf" kernel, but it can use any kernel
function or a precomputed kernel matrix. The number of samples used -
which is also the dimensionality of the features computed - is given
by the parameter "n_components".

6.7.2. Radial Basis Function Kernel

The "RBFSampler" constructs an approximate mapping for the radial
basis function kernel, also known as *Random Kitchen Sinks* [RR2007].
This transformation can be used to explicitly model a kernel map,
prior to applying a linear algorithm, for example a linear SVM:

   >>> from sklearn.kernel_approximation import RBFSampler
   >>> X = [[0, 0], [1, 1], [1, 0], [0, 1]]
   >>> rbf_feature = RBFSampler(gamma=1, random_state=1)
   >>> X_features = rbf_feature.fit_transform(X)
   >>> clf = SGDClassifier(max_iter=5)
   >>> clf.fit(X_features, y)
   >>> clf.score(X_features, y)
   1.0

The mapping relies on a Monte Carlo approximation to the kernel
values. The "fit" function performs the Monte Carlo sampling, whereas
the "transform" method performs the mapping of the data.  Because of
the inherent randomness of the process, results may vary between

different calls to the "fit" function.

The "fit" function takes two arguments: "n_components", which is the
target dimensionality of the feature transform, and "gamma", the
parameter of the RBF-kernel.  A higher "n_components" will result in a
better approximation of the kernel and will yield results more similar
to those produced by a kernel SVM. Note that "fitting" the feature
function does not actually depend on the data given to the "fit"
function. Only the dimensionality of the data is used. Details on the
method can be found in [RR2007].

For a given value of "n_components" "RBFSampler" is often less
accurate as "Nystroem". "RBFSampler" is cheaper to compute, though,
making use of larger feature spaces more efficient.

   [image]Comparing an exact RBF kernel (left) with the approximation
   (right)

Examples:
^^^^^^^^^

6.7.3. Additive Chi Squared Kernel

The additive chi squared kernel is a kernel on histograms, often used
in computer vision.

The additive chi squared kernel as used here is given by

   k(x, y) = \sum_i \frac{2x_iy_i}{x_i+y_i}

This is not exactly the same as
"sklearn.metrics.additive_chi2_kernel()". The authors of [VZ2010]
prefer the version above as it is always positive definite. Since the

kernel is additive, it is possible to treat all components x_i
separately for embedding. This makes it possible to sample the Fourier
transform in regular intervals, instead of approximating using Monte
Carlo sampling.

The class "AdditiveChi2Sampler" implements this component wise
deterministic sampling. Each component is sampled n times, yielding
2n+1 dimensions per input dimension (the multiple of two stems from
the real and complex part of the Fourier transform). In the
literature, n is usually chosen to be 1 or 2, transforming the dataset
to size "n_samples * 5 * n_features" (in the case of n=2).

The approximate feature map provided by "AdditiveChi2Sampler" can be
combined with the approximate feature map provided by "RBFSampler" to
yield an approximate feature map for the exponentiated chi squared
kernel. See the [VZ2010] for details and [VVZ2010] for combination
with the "RBFSampler".

6.7.4. Skewed Chi Squared Kernel

The skewed chi squared kernel is given by:

   k(x,y) = \prod_i \frac{2\sqrt{x_i+c}\sqrt{y_i+c}}{x_i + y_i + 2c}

It has properties that are similar to the exponentiated chi squared
kernel often used in computer vision, but allows for a simple Monte
Carlo approximation of the feature map.

The usage of the "SkewedChi2Sampler" is the same as the usage

described above for the "RBFSampler". The only difference is in the
free parameter, that is called c. For a motivation for this mapping
and the mathematical details see [LS2010].

6.7.5. Polynomial Kernel Approximation via Tensor Sketch

The polynomial kernel is a popular type of kernel function given by:

   k(x, y) = (\gamma x^\top y +c_0)^d

where:

   * "x", "y" are the input vectors

   * "d" is the kernel degree

Intuitively, the feature space of the polynomial kernel of degree *d*
consists of all possible degree-*d* products among input features,
which enables learning algorithms using this kernel to account for
interactions between features.

The TensorSketch [PP2013] method, as implemented in
"PolynomialCountSketch", is a scalable, input data independent method
for polynomial kernel approximation. It is based on the concept of
Count sketch [WIKICS] [CCF2002] , a dimensionality reduction technique
similar to feature hashing, which instead uses several independent
hash functions. TensorSketch obtains a Count Sketch of the outer
product of two vectors (or a vector with itself), which can be used as
an approximation of the polynomial kernel feature space. In
particular, instead of explicitly computing the outer product,

TensorSketch computes the Count Sketch of the vectors and then uses
polynomial multiplication via the Fast Fourier Transform to compute
the Count Sketch of their outer product.

Conveniently, the training phase of TensorSketch simply consists of
initializing some random variables. It is thus independent of the
input data, i.e. it only depends on the number of input features, but
not the data values. In addition, this method can transform samples in
\mathcal{O}(n_{\text{samples}}(n_{\text{features}} +
n_{\text{components}} \log(n_{\text{components}}))) time, where
n_{\text{components}} is the desired output dimension, determined by
"n_components".

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_kernel_approximation_plot_scalable_poly_ker
  nels.py*

6.7.6. Mathematical Details

Kernel methods like support vector machines or kernelized PCA rely on
a property of reproducing kernel Hilbert spaces. For any positive
definite kernel function k (a so called Mercer kernel), it is
guaranteed that there exists a mapping \phi into a Hilbert space
\mathcal{H}, such that

   k(x,y) = \langle \phi(x), \phi(y) \rangle

Where \langle \cdot, \cdot \rangle denotes the inner product in the
Hilbert space.

If an algorithm, such as a linear support vector machine or PCA,

k(x,y) = \langle \phi(x), \phi(y) \rangle

relies only on the scalar product of data points x_i, one may use the
value of k(x_i, x_j), which corresponds to applying the algorithm to
the mapped data points \phi(x_i). The advantage of using k is that the
mapping \phi never has to be calculated explicitly, allowing for
arbitrary large features (even infinite).

One drawback of kernel methods is, that it might be necessary to store
many kernel values k(x_i, x_j) during optimization. If a kernelized
classifier is applied to new data y_j, k(x_i, y_j) needs to be
computed to make predictions, possibly for many different x_i in the
training set.

The classes in this submodule allow to approximate the embedding \phi,
thereby working explicitly with the representations \phi(x_i), which
obviates the need to apply the kernel or store training examples.

[RR2007] "Random features for large-scale kernel machines" Rahimi, A.
         and Recht, B. - Advances in neural information processing
         2007,

[LS2010] "Random Fourier approximations for skewed multiplicative
         histogram kernels" Li, F., Ionescu, C., and Sminchisescu, C.
         - Pattern Recognition,  DAGM 2010, Lecture Notes in Computer
         Science.

[VZ2010] "Efficient additive kernels via explicit feature maps"

- Pattern Recognition,  DAGM 2010, Lecture Notes in Computer
         Science.

         Vedaldi, A. and Zisserman, A. - Computer Vision and Pattern
         Recognition 2010

[VVZ2010] "Generalized RBF feature maps for Efficient Detection"
          Vempati, S. and Vedaldi, A. and Zisserman, A. and Jawahar,
          CV - 2010

[PP2013] >>:doi:`"Fast and scalable polynomial kernels via explicit
         feature maps" <10.1145/2487575.2487591>`<< Pham, N., & Pagh,
         R. - 2013

[CCF2002] "Finding frequent items in data streams" Charikar, M., Chen,
          K., & Farach-Colton - 2002

[WIKICS] "Wikipedia: Count sketch"

1.7. Gaussian Processes

**Gaussian Processes (GP)** are a generic supervised learning method
designed to solve *regression* and *probabilistic classification*
problems.

The advantages of Gaussian processes are:

   * The prediction interpolates the observations (at least for
     regular kernels).

   * The prediction is probabilistic (Gaussian) so that one can
     compute empirical confidence intervals and decide based on those
     if one should refit (online fitting, adaptive fitting) the
     prediction in some region of interest.

   * Versatile: different kernels can be specified. Common kernels are
     provided, but it is also possible to specify custom kernels.

The disadvantages of Gaussian processes include:

   * They are not sparse, i.e., they use the whole samples/features
     information to perform the prediction.

   * They lose efficiency in high dimensional spaces -- namely when
     the number of features exceeds a few dozens.

1.7.1. Gaussian Process Regression (GPR)

The "GaussianProcessRegressor" implements Gaussian processes (GP) for
regression purposes. For this, the prior of the GP needs to be
specified. The prior mean is assumed to be constant and zero (for
"normalize_y=False") or the training data's mean (for
"normalize_y=True"). The prior's covariance is specified by passing a
kernel object. The hyperparameters of the kernel are optimized during
fitting of GaussianProcessRegressor by maximizing the log-marginal-

likelihood (LML) based on the passed "optimizer". As the LML may have
multiple local optima, the optimizer can be started repeatedly by
specifying "n_restarts_optimizer". The first run is always conducted
starting from the initial hyperparameter values of the kernel;
subsequent runs are conducted from hyperparameter values that have
been chosen randomly from the range of allowed values. If the initial
hyperparameters should be kept fixed, *None* can be passed as
optimizer.

The noise level in the targets can be specified by passing it via the
parameter "alpha", either globally as a scalar or per datapoint. Note
that a moderate noise level can also be helpful for dealing with
numeric issues during fitting as it is effectively implemented as
Tikhonov regularization, i.e., by adding it to the diagonal of the
kernel matrix. An alternative to specifying the noise level explicitly
is to include a WhiteKernel component into the kernel, which can
estimate the global noise level from the data (see example below).

The implementation is based on Algorithm 2.1 of [RW2006]. In addition
to the API of standard scikit-learn estimators,
GaussianProcessRegressor:

* allows prediction without prior fitting (based on the GP prior)

* provides an additional method "sample_y(X)", which evaluates samples
  drawn from the GPR (prior or posterior) at given inputs

* exposes a method "log_marginal_likelihood(theta)", which can be used
  externally for other ways of selecting hyperparameters, e.g., via
  Markov chain Monte Carlo.

1.7.2. GPR examples

1.7.2.1. GPR with noise-level estimation

This example illustrates that GPR with a sum-kernel including a
WhiteKernel can estimate the noise level of data. An illustration of
the log-marginal-likelihood (LML) landscape shows that there exist two
local maxima of LML.

   [image]

The first corresponds to a model with a high noise level and a large
length scale, which explains all variations in the data by noise.

   [image]

The second one has a smaller noise level and shorter length scale,
which explains most of the variation by the noise-free functional
relationship. The second model has a higher likelihood; however,
depending on the initial value for the hyperparameters, the gradient-
based optimization might also converge to the high-noise solution. It
is thus important to repeat the optimization several times for
different initializations.

   [image]

1.7.2.2. Comparison of GPR and Kernel Ridge Regression

Both kernel ridge regression (KRR) and GPR learn a target function by

[image]

employing internally the "kernel trick". KRR learns a linear function
in the space induced by the respective kernel which corresponds to a
non-linear function in the original space. The linear function in the
kernel space is chosen based on the mean-squared error loss with ridge
regularization. GPR uses the kernel to define the covariance of a
prior distribution over the target functions and uses the observed
training data to define a likelihood function. Based on Bayes theorem,
a (Gaussian) posterior distribution over target functions is defined,
whose mean is used for prediction.

A major difference is that GPR can choose the kernel's hyperparameters
based on gradient-ascent on the marginal likelihood function while KRR
needs to perform a grid search on a cross-validated loss function
(mean-squared error loss). A further difference is that GPR learns a
generative, probabilistic model of the target function and can thus
provide meaningful confidence intervals and posterior samples along
with the predictions while KRR only provides predictions.

The following figure illustrates both methods on an artificial
noise. The figure compares the learned model of KRR and GPR based on a
ExpSineSquared kernel, which is suited for learning periodic

functions. The kernel's hyperparameters control the smoothness
(length_scale) and periodicity of the kernel (periodicity). Moreover,
the noise level of the data is learned explicitly by GPR by an
additional WhiteKernel component in the kernel and by the
regularization parameter alpha of KRR.

   [image]

The figure shows that both methods learn reasonable models of the
target function. GPR correctly identifies the periodicity of the
function to be roughly 2*\pi (6.28), while KRR chooses the doubled
periodicity 4*\pi . Besides that, GPR provides reasonable confidence
bounds on the prediction which are not available for KRR. A major
difference between the two methods is the time required for fitting
and predicting: while fitting KRR is fast in principle, the grid-
search for hyperparameter optimization scales exponentially with the
number of hyperparameters ("curse of dimensionality"). The gradient-
based optimization of the parameters in GPR does not suffer from this
exponential scaling and is thus considerably faster on this example
with 3-dimensional hyperparameter space. The time for predicting is
similar; however, generating the variance of the predictive
distribution of GPR takes considerably longer than just predicting the
mean.

1.7.2.3. GPR on Mauna Loa CO2 data

mean.

This example is based on Section 5.4.3 of [RW2006]. It illustrates an
example of complex kernel engineering and hyperparameter optimization
using gradient ascent on the log-marginal-likelihood. The data
consists of the monthly average atmospheric CO2 concentrations (in
parts per million by volume (ppmv)) collected at the Mauna Loa
Observatory in Hawaii, between 1958 and 1997. The objective is to
model the CO2 concentration as a function of the time t.

The kernel is composed of several terms that are responsible for
explaining different properties of the signal:

* a long term, smooth rising trend is to be explained by an RBF
  kernel. The RBF kernel with a large length-scale enforces this
  component to be smooth; it is not enforced that the trend is rising
  which leaves this choice to the GP. The specific length-scale and
  the amplitude are free hyperparameters.

* a seasonal component, which is to be explained by the periodic
  ExpSineSquared kernel with a fixed periodicity of 1 year. The
  length-scale of this periodic component, controlling its smoothness,
  is a free parameter. In order to allow decaying away from exact
  periodicity, the product with an RBF kernel is taken. The length-
  scale of this RBF component controls the decay time and is a further
  free parameter.

* smaller, medium term irregularities are to be explained by a

scale of this RBF component controls the decay time and is a further

  RationalQuadratic kernel component, whose length-scale and alpha
  parameter, which determines the diffuseness of the length-scales,
  are to be determined. According to [RW2006], these irregularities
  can better be explained by a RationalQuadratic than an RBF kernel
  component, probably because it can accommodate several length-
  scales.

* a "noise" term, consisting of an RBF kernel contribution, which
  shall explain the correlated noise components such as local weather
  phenomena, and a WhiteKernel contribution for the white noise. The
  relative amplitudes and the RBF's length scale are further free
  parameters.

Maximizing the log-marginal-likelihood after subtracting the target's
mean yields the following kernel with an LML of -83.214:

   34.4**2 * RBF(length_scale=41.8)
   + 3.27**2 * RBF(length_scale=180) * ExpSineSquared(length_scale=1.44,
                                                      periodicity=1)
   + 0.446**2 * RationalQuadratic(alpha=17.7, length_scale=0.957)
   + 0.197**2 * RBF(length_scale=0.138) + WhiteKernel(noise_level=0.0336)

Thus, most of the target signal (34.4ppm) is explained by a long-term
rising trend (length-scale 41.8 years). The periodic component has an
amplitude of 3.27ppm, a decay time of 180 years and a length-scale of
1.44. The long decay time indicates that we have a locally very close

to periodic seasonal component. The correlated noise has an amplitude
of 0.197ppm with a length scale of 0.138 years and a white-noise
contribution of 0.197ppm. Thus, the overall noise level is very small,
indicating that the data can be very well explained by the model. The
figure shows also that the model makes very confident predictions
until around 2015

   [image]

1.7.3. Gaussian Process Classification (GPC)

The "GaussianProcessClassifier" implements Gaussian processes (GP) for
classification purposes, more specifically for probabilistic
classification, where test predictions take the form of class
probabilities. GaussianProcessClassifier places a GP prior on a latent
function f, which is then squashed through a link function to obtain
the probabilistic classification. The latent function f is a so-called
nuisance function, whose values are not observed and are not relevant
by themselves. Its purpose is to allow a convenient formulation of the
model, and f is removed (integrated out) during prediction.
GaussianProcessClassifier implements the logistic link function, for
which the integral cannot be computed analytically but is easily
approximated in the binary case.

In contrast to the regression setting, the posterior of the latent
function f is not Gaussian even for a GP prior since a Gaussian

likelihood is inappropriate for discrete class labels. Rather, a non-
Gaussian likelihood corresponding to the logistic link function
(logit) is used. GaussianProcessClassifier approximates the non-
Gaussian posterior with a Gaussian based on the Laplace approximation.
More details can be found in Chapter 3 of [RW2006].

The GP prior mean is assumed to be zero. The prior's covariance is
specified by passing a kernel object. The hyperparameters of the
kernel are optimized during fitting of GaussianProcessRegressor by
maximizing the log-marginal-likelihood (LML) based on the passed
"optimizer". As the LML may have multiple local optima, the optimizer
can be started repeatedly by specifying "n_restarts_optimizer". The
first run is always conducted starting from the initial hyperparameter
values of the kernel; subsequent runs are conducted from
hyperparameter values that have been chosen randomly from the range of
allowed values. If the initial hyperparameters should be kept fixed,
*None* can be passed as optimizer.

"GaussianProcessClassifier" supports multi-class classification by
performing either one-versus-rest or one-versus-one based training and
prediction.  In one-versus-rest, one binary Gaussian process
classifier is fitted for each class, which is trained to separate this

class from the rest. In "one_vs_one", one binary Gaussian process
classifier is fitted for each pair of classes, which is trained to
separate these two classes. The predictions of these binary predictors
are combined into multi-class predictions. See the section on multi-
class classification for more details.

In the case of Gaussian process classification, "one_vs_one" might be
computationally  cheaper since it has to solve many problems involving
only a subset of the whole training set rather than fewer problems on
the whole dataset. Since Gaussian process classification scales
cubically with the size of the dataset, this might be considerably
faster. However, note that "one_vs_one" does not support predicting
probability estimates but only plain predictions. Moreover, note that
"GaussianProcessClassifier" does not (yet) implement a true multi-
class Laplace approximation internally, but as discussed above is
based on solving several binary classification tasks internally, which
are combined using one-versus-rest or one-versus-one.

1.7.4. GPC examples

1.7.4.1. Probabilistic predictions with GPC

This example illustrates the predicted probability of GPC for an RBF
kernel with different choices of the hyperparameters. The first figure

shows the predicted probability of GPC with arbitrarily chosen
hyperparameters and with the hyperparameters corresponding to the
maximum log-marginal-likelihood (LML).

While the hyperparameters chosen by optimizing LML have a considerably
larger LML, they perform slightly worse according to the log-loss on
test data. The figure shows that this is because they exhibit a steep
change of the class probabilities at the class boundaries (which is
good) but have predicted probabilities close to 0.5 far away from the
class boundaries (which is bad) This undesirable effect is caused by
the Laplace approximation used internally by GPC.

The second figure shows the log-marginal-likelihood for different
choices of the kernel's hyperparameters, highlighting the two choices
of the hyperparameters used in the first figure by black dots.

   [image]

   [image]

1.7.4.2. Illustration of GPC on the XOR dataset
-----------------------------------------------

This example illustrates GPC on XOR data. Compared are a stationary,
isotropic kernel ("RBF") and a non-stationary kernel ("DotProduct").
On this particular dataset, the "DotProduct" kernel obtains
considerably better results because the class-boundaries are linear
and coincide with the coordinate axes. In practice, however,
stationary kernels such as "RBF" often obtain better results.

   [image]

1.7.4.3. Gaussian process classification (GPC) on iris dataset

This example illustrates the predicted probability of GPC for an
isotropic and anisotropic RBF kernel on a two-dimensional version for
the iris-dataset. This illustrates the applicability of GPC to non-
binary classification. The anisotropic RBF kernel obtains slightly
higher log-marginal-likelihood by assigning different length-scales to
the two feature dimensions.

   [image]

1.7.5. Kernels for Gaussian Processes

Kernels (also called "covariance functions" in the context of GPs) are
a crucial ingredient of GPs which determine the shape of prior and
posterior of the GP. They encode the assumptions on the function being
learned by defining the "similarity" of two datapoints combined with
the assumption that similar datapoints should have similar target
values. Two categories of kernels can be distinguished: stationary
kernels depend only on the distance of two datapoints and not on their
absolute values k(x_i, x_j)= k(d(x_i, x_j)) and are thus invariant to
translations in the input space, while non-stationary kernels depend
also on the specific values of the datapoints. Stationary kernels can

further be subdivided into isotropic and anisotropic kernels, where
isotropic kernels are also invariant to rotations in the input space.
For more details, we refer to Chapter 4 of [RW2006]. For guidance on
how to best combine different kernels, we refer to [Duv2014].

1.7.5.1. Gaussian Process Kernel API

The main usage of a "Kernel" is to compute the GP's covariance between
datapoints. For this, the method "__call__" of the kernel can be
called. This method can either be used to compute the "auto-
covariance" of all pairs of datapoints in a 2d array X, or the "cross-
covariance" of all combinations of datapoints of a 2d array X with
datapoints in a 2d array Y. The following identity holds true for all
kernels k (except for the "WhiteKernel"): "k(X) == K(X, Y=X)"

If only the diagonal of the auto-covariance is being used, the method
"diag()" of a kernel can be called, which is more computationally
efficient than the equivalent call to "__call__": "np.diag(k(X, X)) ==
k.diag(X)"

Kernels are parameterized by a vector \theta of hyperparameters. These
hyperparameters can for instance control length-scales or periodicity
of a kernel (see below). All kernels support computing analytic
gradients of the kernel's auto-covariance with respect to log(\theta)
via setting "eval_gradient=True" in the "__call__" method. That is, a

"(len(X), len(X), len(theta))" array is returned where the entry "[i,
j, l]" contains \frac{\partial k_\theta(x_i, x_j)}{\partial
log(\theta_l)}. This gradient is used by the Gaussian process (both
regressor and classifier) in computing the gradient of the log-
marginal-likelihood, which in turn is used to determine the value of
\theta, which maximizes the log-marginal-likelihood, via gradient
ascent. For each hyperparameter, the initial value and the bounds need
to be specified when creating an instance of the kernel. The current
value of \theta can be get and set via the property "theta" of the
kernel object. Moreover, the bounds of the hyperparameters can be
accessed by the property "bounds" of the kernel. Note that both
properties (theta and bounds) return log-transformed values of the
internally used values since those are typically more amenable to
gradient-based optimization. The specification of each hyperparameter
is stored in the form of an instance of "Hyperparameter" in the
respective kernel. Note that a kernel using a hyperparameter with name
"x" must have the attributes self.x and self.x_bounds.

The abstract base class for all kernels is "Kernel". Kernel implements
a similar interface as "Estimator", providing the methods
"get_params()", "set_params()", and "clone()". This allows setting

kernel values also via meta-estimators such as "Pipeline" or
"GridSearch". Note that due to the nested structure of kernels (by
applying kernel operators, see below), the names of kernel parameters
might become relatively complicated. In general, for a binary kernel
operator, parameters of the left operand are prefixed with "k1__" and
parameters of the right operand with "k2__". An additional convenience
method is "clone_with_theta(theta)", which returns a cloned version of
the kernel but with the hyperparameters set to "theta". An
illustrative example:

>>> from sklearn.gaussian_process.kernels import ConstantKernel, RBF
>>> kernel = ConstantKernel(constant_value=1.0, constant_value_bounds=(0.0, 10.0)) * RBF(length_scale=0.5, length_scale_bounds=(0.0, 10.0)) + RBF(length_scale=2.0, length_scale_bounds=(0.0, 10.0))
>>> for hyperparameter in kernel.hyperparameters: print(hyperparameter)
Hyperparameter(name='k1__k1__constant_value', value_type='numeric', bounds=array([[ 0., 10.]]), n_elements=1, fixed=False)
Hyperparameter(name='k1__k2__length_scale', value_type='numeric', bounds=array([[ 0., 10.]]), n_elements=1, fixed=False)
Hyperparameter(name='k2__length_scale', value_type='numeric', bounds=array([[ 0., 10.]]), n_elements=1, fixed=False)
>>> params = kernel.get_params()

>>> for key in sorted(params): print("%s : %s" % (key, params[key]))
k1 : 1**2 * RBF(length_scale=0.5)
k1__k1 : 1**2
k1__k1__constant_value : 1.0
k1__k1__constant_value_bounds : (0.0, 10.0)
k1__k2 : RBF(length_scale=0.5)
k1__k2__length_scale : 0.5
k1__k2__length_scale_bounds : (0.0, 10.0)
k2 : RBF(length_scale=2)
k2__length_scale : 2.0
k2__length_scale_bounds : (0.0, 10.0)
>>> print(kernel.theta)  # Note: log-transformed
[ 0.         -0.69314718  0.69314718]
>>> print(kernel.bounds)  # Note: log-transformed
[[      -inf 2.30258509]
 [      -inf 2.30258509]
 [      -inf 2.30258509]]

All Gaussian process kernels are interoperable with
"sklearn.metrics.pairwise" and vice versa: instances of subclasses of
"Kernel" can be passed as "metric" to "pairwise_kernels" from
"sklearn.metrics.pairwise". Moreover, kernel functions from pairwise
can be used as GP kernels by using the wrapper class "PairwiseKernel".
The only caveat is that the gradient of the hyperparameters is not
analytic but numeric and all those kernels support only isotropic
distances. The parameter "gamma" is considered to be a hyperparameter
and may be optimized. The other kernel parameters are set directly at
initialization and are kept fixed.

1.7.5.2. Basic kernels

The "ConstantKernel" kernel can be used as part of a "Product" kernel

where it scales the magnitude of the other factor (kernel) or as part
of a "Sum" kernel, where it modifies the mean of the Gaussian process.
It depends on a parameter constant\_value. It is defined as:

   k(x_i, x_j) = constant\_value \;\forall\; x_1, x_2

The main use-case of the "WhiteKernel" kernel is as part of a sum-
kernel where it explains the noise-component of the signal. Tuning its
parameter noise\_level corresponds to estimating the noise-level. It
is defined as:

   k(x_i, x_j) = noise\_level \text{ if } x_i == x_j \text{ else } 0

1.7.5.3. Kernel operators

Kernel operators take one or two base kernels and combine them into a
new kernel. The "Sum" kernel takes two kernels k_1 and k_2 and
combines them via k_{sum}(X, Y) = k_1(X, Y) + k_2(X, Y). The
"Product" kernel takes two kernels k_1 and k_2 and combines them via
k_{product}(X, Y) = k_1(X, Y) * k_2(X, Y). The "Exponentiation" kernel
takes one base kernel and a scalar parameter p and combines them via
k_{exp}(X, Y) = k(X, Y)^p. Note that magic methods "__add__",
"__mul___" and "__pow__" are overridden on the Kernel objects, so one
can use e.g. "RBF() + RBF()" as a shortcut for "Sum(RBF(), RBF())".

1.7.5.4. Radial basis function (RBF) kernel

The "RBF" kernel is a stationary kernel. It is also known as the

"squared exponential" kernel. It is parameterized by a length-scale
parameter l>0, which can either be a scalar (isotropic variant of the
kernel) or a vector with the same number of dimensions as the inputs x
(anisotropic variant of the kernel). The kernel is given by:

   k(x_i, x_j) = \text{exp}\left(- \frac{d(x_i, x_j)^2}{2l^2} \right)

where d(\cdot, \cdot) is the Euclidean distance. This kernel is
infinitely differentiable, which implies that GPs with this kernel as
covariance function have mean square derivatives of all orders, and
are thus very smooth. The prior and posterior of a GP resulting from
an RBF kernel are shown in the following figure:

   [image]

1.7.5.5. Matérn kernel

The "Matern" kernel is a stationary kernel and a generalization of the
"RBF" kernel. It has an additional parameter \nu which controls the
smoothness of the resulting function. It is parameterized by a length-
scale parameter l>0, which can either be a scalar (isotropic variant
of the kernel) or a vector with the same number of dimensions as the
inputs x (anisotropic variant of the kernel). The kernel is given by:

   k(x_i, x_j) =
   \frac{1}{\Gamma(\nu)2^{\nu-1}}\Bigg(\frac{\sqrt{2\nu}}{l} d(x_i ,
   x_j )\Bigg)^\nu K_\nu\Bigg(\frac{\sqrt{2\nu}}{l} d(x_i , x_j
   )\Bigg),

where d(\cdot,\cdot) is the Euclidean distance, K_\nu(\cdot) is a

x_j )\Bigg)^\nu K_\nu\Bigg(\frac{\sqrt{2\nu}}{l} d(x_i , x_j
   )\Bigg),

modified Bessel function and \Gamma(\cdot) is the gamma function. As
\nu\rightarrow\infty, the Matérn kernel converges to the RBF kernel.
When \nu = 1/2, the Matérn kernel becomes identical to the absolute
exponential kernel, i.e.,

   k(x_i, x_j) = \exp \Bigg(- \frac{1}{l} d(x_i , x_j ) \Bigg) \quad
   \quad \nu= \tfrac{1}{2}

In particular, \nu = 3/2:

   k(x_i, x_j) =  \Bigg(1 + \frac{\sqrt{3}}{l} d(x_i , x_j )\Bigg)
   \exp \Bigg(-\frac{\sqrt{3}}{l} d(x_i , x_j ) \Bigg) \quad \quad
   \nu= \tfrac{3}{2}

and \nu = 5/2:

   k(x_i, x_j) = \Bigg(1 + \frac{\sqrt{5}}{l} d(x_i , x_j )
   +\frac{5}{3l} d(x_i , x_j )^2 \Bigg) \exp \Bigg(-\frac{\sqrt{5}}{l}
   d(x_i , x_j ) \Bigg) \quad \quad \nu= \tfrac{5}{2}

are popular choices for learning functions that are not infinitely
differentiable (as assumed by the RBF kernel) but at least once (\nu =
3/2) or twice differentiable (\nu = 5/2).

The flexibility of controlling the smoothness of the learned function
via \nu allows adapting to the properties of the true underlying
functional relation. The prior and posterior of a GP resulting from a
Matérn kernel are shown in the following figure:

   [image]

See [RW2006], pp84 for further details regarding the different
variants of the Matérn kernel.

1.7.5.6. Rational quadratic kernel

The "RationalQuadratic" kernel can be seen as a scale mixture (an

infinite sum) of "RBF" kernels with different characteristic length-
scales. It is parameterized by a length-scale parameter l>0 and a
scale mixture parameter  \alpha>0 Only the isotropic variant where l
is a scalar is supported at the moment. The kernel is given by:

   k(x_i, x_j) = \left(1 + \frac{d(x_i, x_j)^2}{2\alpha
   l^2}\right)^{-\alpha}

The prior and posterior of a GP resulting from a "RationalQuadratic"
kernel are shown in the following figure:

   [image]

1.7.5.7. Exp-Sine-Squared kernel

The "ExpSineSquared" kernel allows modeling periodic functions. It is
parameterized by a length-scale parameter l>0 and a periodicity
parameter p>0. Only the isotropic variant where l is a scalar is
supported at the moment. The kernel is given by:

   k(x_i, x_j) = \text{exp}\left(- \frac{ 2\sin^2(\pi d(x_i, x_j) / p)
   }{ l^ 2} \right)

The prior and posterior of a GP resulting from an ExpSineSquared

   [image]

1.7.5.8. Dot-Product kernel

The "DotProduct" kernel is non-stationary and can be obtained from
linear regression by putting N(0, 1) priors on the coefficients of x_d
(d = 1, . . . , D) and a prior of N(0, \sigma_0^2) on the bias. The
"DotProduct" kernel is invariant to a rotation of the coordinates

about the origin, but not translations. It is parameterized by a
parameter \sigma_0^2. For \sigma_0^2 = 0, the kernel is called the
homogeneous linear kernel, otherwise it is inhomogeneous. The kernel
is given by

   k(x_i, x_j) = \sigma_0 ^ 2 + x_i \cdot x_j

The "DotProduct" kernel is commonly combined with exponentiation. An
example with exponent 2 is shown in the following figure:

   [image]

1.7.5.9. References

[RW2006] Carl E. Rasmussen and Christopher K.I. Williams, "Gaussian
         Processes for Machine Learning", MIT Press 2006

[Duv2014] David Duvenaud, "The Kernel Cookbook: Advice on Covariance
          functions", 2014

2.8. Density Estimation

Density estimation walks the line between unsupervised learning,
feature engineering, and data modeling.  Some of the most popular and
useful density estimation techniques are mixture models such as
Gaussian Mixtures ("GaussianMixture"), and neighbor-based approaches
such as the kernel density estimate ("KernelDensity"). Gaussian
Mixtures are discussed more fully in the context of clustering,
because the technique is also useful as an unsupervised clustering
scheme.

Density estimation is a very simple concept, and most people are
already familiar with one common density estimation technique: the
histogram.

2.8.1. Density Estimation: Histograms

A histogram is a simple visualization of data where bins are defined,
and the number of data points within each bin is tallied.  An example
of a histogram can be seen in the upper-left panel of the following
figure:

[image: hist_to_kde][image]

A major problem with histograms, however, is that the choice of
binning can have a disproportionate effect on the resulting
visualization.  Consider the upper-right panel of the above figure.
It shows a histogram over the same data, with the bins shifted right.
The results of the two visualizations look entirely different, and
might lead to different interpretations of the data.

Intuitively, one can also think of a histogram as a stack of blocks,
one block per point.  By stacking the blocks in the appropriate grid

space, we recover the histogram.  But what if, instead of stacking the
blocks on a regular grid, we center each block on the point it
represents, and sum the total height at each location?  This idea
leads to the lower-left visualization.  It is perhaps not as clean as
a histogram, but the fact that the data drive the block locations mean
that it is a much better representation of the underlying data.

This visualization is an example of a *kernel density estimation*, in
this case with a top-hat kernel (i.e. a square block at each point).
We can recover a smoother distribution by using a smoother kernel.
The bottom-right plot shows a Gaussian kernel density estimate, in
which each point contributes a Gaussian curve to the total.  The
result is a smooth density estimate which is derived from the data,
and functions as a powerful non-parametric model of the distribution
of points.

2.8.2. Kernel Density Estimation

Kernel density estimation in scikit-learn is implemented in the
"KernelDensity" estimator, which uses the Ball Tree or KD Tree for
efficient queries (see Nearest Neighbors for a discussion of these).
Though the above example uses a 1D data set for simplicity, kernel
density estimation can be performed in any number of dimensions,

though in practice the curse of dimensionality causes its performance
to degrade in high dimensions.

In the following figure, 100 points are drawn from a bimodal
distribution, and the kernel density estimates are shown for three
choices of kernels:

[image: kde_1d_distribution][image]

It's clear how the kernel shape affects the smoothness of the
resulting distribution.  The scikit-learn kernel density estimator can
be used as follows:

>>> from sklearn.neighbors import KernelDensity
>>> X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
>>> kde = KernelDensity(kernel='gaussian', bandwidth=0.2).fit(X)
>>> kde.score_samples(X)
array([-0.41075698, -0.41075698, -0.41076071, -0.41075698, -0.41075698,
       -0.41076071])

Here we have used "kernel='gaussian'", as seen above. Mathematically,
a kernel is a positive function K(x;h) which is controlled by the
bandwidth parameter h. Given this kernel form, the density estimate at
a point y within a group of points x_i; i=1\cdots N is given by:

   \rho_K(y) = \sum_{i=1}^{N} K(y - x_i; h)

The bandwidth here acts as a smoothing parameter, controlling the
tradeoff between bias and variance in the result.  A large bandwidth
leads to a very smooth (i.e. high-bias) density distribution.  A small

bandwidth leads to an unsmooth (i.e. high-variance) density
distribution.

The parameter *bandwidth* controls this smoothing. One can either set
manually this parameter or use Scott's and Silvermann's estimation
methods.

"KernelDensity" implements several common kernel forms, which are
shown in the following figure:

[image: kde_kernels][image]

The form of these kernels is as follows:

* Gaussian kernel ("kernel = 'gaussian'")

  K(x; h) \propto \exp(- \frac{x^2}{2h^2} )

* Tophat kernel ("kernel = 'tophat'")

  K(x; h) \propto 1 if x < h

* Epanechnikov kernel ("kernel = 'epanechnikov'")

  K(x; h) \propto 1 - \frac{x^2}{h^2}

* Exponential kernel ("kernel = 'exponential'")

  K(x; h) \propto \exp(-x/h)

* Linear kernel ("kernel = 'linear'")

  K(x; h) \propto 1 - x/h if x < h

* Cosine kernel ("kernel = 'cosine'")

  K(x; h) \propto \cos(\frac{\pi x}{2h}) if x < h

The kernel density estimator can be used with any of the valid
distance metrics (see "DistanceMetric" for a list of available
metrics), though the results are properly normalized only for the
Euclidean metric.  One particularly useful metric is the Haversine
distance which measures the angular distance between points on a
sphere.  Here is an example of using a kernel density estimate for a
visualization of geospatial data, in this case the distribution of

observations of two different species on the South American continent:

[image: species_kde][image]

One other useful application of kernel density estimation is to learn
a non-parametric generative model of a dataset in order to efficiently
draw new samples from this generative model. Here is an example of
using this process to create a new set of hand-written digits, using a
Gaussian kernel learned on a PCA projection of the data:

[image: digits_kde][image]

The "new" data consists of linear combinations of the input data, with
weights probabilistically drawn given the KDE model.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neighbors_plot_kde_1d.py*: computation of
  simple kernel density estimates in one dimension.

* *sphx_glr_auto_examples_neighbors_plot_digits_kde_sampling.py*: an
  example of using Kernel Density estimation to learn a generative
  model of the hand-written digits data, and drawing new samples from
  this model.

* *sphx_glr_auto_examples_neighbors_plot_species_kde.py*: an example
  of Kernel Density estimation using the Haversine distance metric to
  visualize geospatial data

2.9. Neural network models (unsupervised)

2.9.1. Restricted Boltzmann machines

Restricted Boltzmann machines (RBM) are unsupervised nonlinear feature
learners based on a probabilistic model. The features extracted by an
RBM or a hierarchy of RBMs often give good results when fed into a
linear classifier such as a linear SVM or a perceptron.

The model makes assumptions regarding the distribution of inputs. At
the moment, scikit-learn only provides "BernoulliRBM", which assumes
the inputs are either binary values or values between 0 and 1, each
encoding the probability that the specific feature would be turned on.

The RBM tries to maximize the likelihood of the data using a
particular graphical model. The parameter learning algorithm used
(Stochastic Maximum Likelihood) prevents the representations from
straying far from the input data, which makes them capture interesting
regularities, but makes the model less useful for small datasets, and
usually not useful for density estimation.

The method gained popularity for initializing deep neural networks
with the weights of independent RBMs. This method is known as
unsupervised pre-training.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neural_networks_plot_rbm_logistic_classific
  ation.py*

2.9.1.1. Graphical model and parametrization

The graphical model of an RBM is a fully-connected bipartite graph.

[image]

ation.py*

[image]

The nodes are random variables whose states depend on the state of the
other nodes they are connected to. The model is therefore
parameterized by the weights of the connections, as well as one
intercept (bias) term for each visible and hidden unit, omitted from
the image for simplicity.

The energy function measures the quality of a joint assignment:

   E(\mathbf{v}, \mathbf{h}) = -\sum_i \sum_j w_{ij}v_ih_j - \sum_i
   b_iv_i   - \sum_j c_jh_j

In the formula above, \mathbf{b} and \mathbf{c} are the intercept
vectors for the visible and hidden layers, respectively. The joint
probability of the model is defined in terms of the energy:

   P(\mathbf{v}, \mathbf{h}) = \frac{e^{-E(\mathbf{v},
   \mathbf{h})}}{Z}

The word *restricted* refers to the bipartite structure of the model,
which prohibits direct interaction between hidden units, or between
visible units. This means that the following conditional
independencies are assumed:

   h_i \bot h_j | \mathbf{v} \\ v_i \bot v_j | \mathbf{h}

The bipartite structure allows for the use of efficient block Gibbs
sampling for inference.

2.9.1.2. Bernoulli Restricted Boltzmann machines

In the "BernoulliRBM", all units are binary stochastic units. This
means that the input data should either be binary, or real-valued

between 0 and 1 signifying the probability that the visible unit would
turn on or off. This is a good model for character recognition, where
the interest is on which pixels are active and which aren't. For
images of natural scenes it no longer fits because of background,
depth and the tendency of neighbouring pixels to take the same values.

The conditional probability distribution of each unit is given by the
logistic sigmoid activation function of the input it receives:

   P(v_i=1|\mathbf{h}) = \sigma(\sum_j w_{ij}h_j + b_i) \\
   P(h_i=1|\mathbf{v}) = \sigma(\sum_i w_{ij}v_i + c_j)

where \sigma is the logistic sigmoid function:

   \sigma(x) = \frac{1}{1 + e^{-x}}

2.9.1.3. Stochastic Maximum Likelihood learning

The training algorithm implemented in "BernoulliRBM" is known as
Stochastic Maximum Likelihood (SML) or Persistent Contrastive
Divergence (PCD). Optimizing maximum likelihood directly is infeasible
because of the form of the data likelihood:

   \log P(v) = \log \sum_h e^{-E(v, h)} - \log \sum_{x, y} e^{-E(x,
   y)}

For simplicity the equation above is written for a single training
example. The gradient with respect to the weights is formed of two
terms corresponding to the ones above. They are usually known as the

positive gradient and the negative gradient, because of their
respective signs.  In this implementation, the gradients are estimated
over mini-batches of samples.

In maximizing the log-likelihood, the positive gradient makes the
model prefer hidden states that are compatible with the observed
training data. Because of the bipartite structure of RBMs, it can be
computed efficiently. The negative gradient, however, is intractable.
Its goal is to lower the energy of joint states that the model
prefers, therefore making it stay true to the data. It can be
approximated by Markov chain Monte Carlo using block Gibbs sampling by
iteratively sampling each of v and h given the other, until the chain
mixes. Samples generated in this way are sometimes referred as fantasy
particles. This is inefficient and it is difficult to determine
whether the Markov chain mixes.

The Contrastive Divergence method suggests to stop the chain after a
small number of iterations, k, usually even 1. This method is fast and
has low variance, but the samples are far from the model distribution.

Persistent Contrastive Divergence addresses this. Instead of starting
a new chain each time the gradient is needed, and performing only one
Gibbs sampling step, in PCD we keep a number of chains (fantasy

particles) that are updated k Gibbs steps after each weight update.
This allows the particles to explore the space more thoroughly.

* "A fast learning algorithm for deep belief nets" G. Hinton, S.
  Osindero, Y.-W. Teh, 2006

* "Training Restricted Boltzmann Machines using Approximations to the
  Likelihood Gradient" T. Tieleman, 2008

1.4. Support Vector Machines

**Support vector machines (SVMs)** are a set of supervised learning
methods used for classification, regression and outliers detection.

The advantages of support vector machines are:

   * Effective in high dimensional spaces.

   * Still effective in cases where number of dimensions is greater
     than the number of samples.

   * Uses a subset of training points in the decision function (called
     support vectors), so it is also memory efficient.

   * Versatile: different Kernel functions can be specified for the
     decision function. Common kernels are provided, but it is also
     possible to specify custom kernels.

The disadvantages of support vector machines include:

   * If the number of features is much greater than the number of
     samples, avoid over-fitting in choosing Kernel functions and
     regularization term is crucial.

   * SVMs do not directly provide probability estimates, these are
     calculated using an expensive five-fold cross-validation (see
     Scores and probabilities, below).

The support vector machines in scikit-learn support both dense
("numpy.ndarray" and convertible to that by "numpy.asarray") and
sparse (any "scipy.sparse") sample vectors as input. However, to use
an SVM to make predictions for sparse data, it must have been fit on
such data. For optimal performance, use C-ordered "numpy.ndarray"
(dense) or "scipy.sparse.csr_matrix" (sparse) with "dtype=float64".

1.4.1. Classification

"SVC", "NuSVC" and "LinearSVC" are classes capable of performing
binary and multi-class classification on a dataset.

   [image]

"SVC" and "NuSVC" are similar methods, but accept slightly different
sets of parameters and have different mathematical formulations (see
section Mathematical formulation). On the other hand, "LinearSVC" is
another (faster) implementation of Support Vector Classification for
the case of a linear kernel. Note that "LinearSVC" does not accept
parameter "kernel", as this is assumed to be linear. It also lacks
some of the attributes of "SVC" and "NuSVC", like "support_".

As other classifiers, "SVC", "NuSVC" and "LinearSVC" take as input two
arrays: an array *X* of shape *(n_samples, n_features)* holding the
training samples, and an array *y* of class labels (strings or
integers), of shape *(n_samples)*:

   SVC()

SVMs decision function (detailed in the Mathematical formulation)
depends on some subset of the training data, called the support
vectors. Some properties of these support vectors can be found in
attributes "support_vectors_", "support_" and "n_support_":

   >>> # get support vectors
   >>> clf.support_vectors_
   array([[0., 0.],
          [1., 1.]])
   >>> # get indices of support vectors
   >>> clf.support_
   array([0, 1]...)
   >>> # get number of support vectors for each class
   >>> clf.n_support_
   array([1, 1]...)

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_svm_plot_separating_hyperplane.py*,

* *sphx_glr_auto_examples_svm_plot_svm_nonlinear.py*

* *sphx_glr_auto_examples_svm_plot_svm_anova.py*,

1.4.1.1. Multi-class classification

"SVC" and "NuSVC" implement the "one-versus-one" approach for multi-
class classification. In total, "n_classes * (n_classes - 1) / 2"
classifiers are constructed and each one trains data from two classes.
To provide a consistent interface with other classifiers, the
"decision_function_shape" option allows to monotonically transform the
results of the "one-versus-one" classifiers to a "one-vs-rest"
decision function of shape "(n_samples, n_classes)".

>>> X = [[0], [1], [2], [3]]
>>> Y = [0, 1, 2, 3]
>>> clf = svm.SVC(decision_function_shape='ovo')
>>> clf.fit(X, Y)
SVC(decision_function_shape='ovo')
>>> dec = clf.decision_function([[1]])
>>> dec.shape[1] # 4 classes: 4*3/2 = 6
6
>>> clf.decision_function_shape = "ovr"
>>> dec.shape[1] # 4 classes
4

6
4

On the other hand, "LinearSVC" implements "one-vs-the-rest" multi-
class strategy, thus training *n_classes* models.

>>> lin_clf = svm.LinearSVC()
>>> lin_clf.fit(X, Y)
LinearSVC()
>>> dec = lin_clf.decision_function([[1]])
>>> dec.shape[1]
4

See Mathematical formulation for a complete description of the
decision function.

Note that the "LinearSVC" also implements an alternative multi-class
strategy, the so-called multi-class SVM formulated by Crammer and
Singer [16], by using the option "multi_class='crammer_singer'". In
practice, one-vs-rest classification is usually preferred, since the
results are mostly similar, but the runtime is significantly less.

For "one-vs-rest" "LinearSVC" the attributes "coef_" and "intercept_"
have the shape "(n_classes, n_features)" and "(n_classes,)"
respectively. Each row of the coefficients corresponds to one of the
"n_classes" "one-vs-rest" classifiers and similar for the intercepts,
in the order of the "one" class.

In the case of "one-vs-one" "SVC" and "NuSVC", the layout of the
attributes is a little more involved. In the case of a linear kernel,
the attributes "coef_" and "intercept_" have the shape "(n_classes *
(n_classes - 1) / 2, n_features)" and "(n_classes * (n_classes - 1) /
2)" respectively. This is similar to the layout for "LinearSVC"

described above, with each row now corresponding to a binary
classifier. The order for classes 0 to n is "0 vs 1", "0 vs 2" , ...
"0 vs n", "1 vs 2", "1 vs 3", "1 vs n", . . . "n-1 vs n".

The shape of "dual_coef_" is "(n_classes-1, n_SV)" with a somewhat
hard to grasp layout. The columns correspond to the support vectors
involved in any of the "n_classes * (n_classes - 1) / 2" "one-vs-one"
classifiers. Each support vector "v" has a dual coefficient in each of
the "n_classes - 1" classifiers comparing the class of "v" against
another class. Note that some, but not all, of these dual
coefficients, may be zero. The "n_classes - 1" entries in each column
are these dual coefficients, ordered by the opposing class.

This might be clearer with an example: consider a three class problem
with class 0 having three support vectors v^{0}_0, v^{1}_0, v^{2}_0
and class 1 and 2 having two support vectors v^{0}_1, v^{1}_1 and
v^{0}_2, v^{1}_2 respectively.  For each support vector v^{j}_i, there
are two dual coefficients.  Let's call the coefficient of support
vector v^{j}_i in the classifier between classes i and k
\alpha^{j}_{i,k}. Then "dual_coef_" looks like this:

+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+--------------------------+
| \alpha^{0}_{0,1}         | \alpha^{1}_{0,1}         | \alpha^{2}_{0,1}         | \alpha^{0}_{1,0}         | \alpha^{1}_{1,0}         | \alpha^{0}_{2,0}         | \alpha^{1}_{2,0}         |
| \alpha^{0}_{0,2}         | \alpha^{1}_{0,2}         | \alpha^{2}_{0,2}         | \alpha^{0}_{1,2}         | \alpha^{1}_{1,2}         | \alpha^{0}_{2,1}         | \alpha^{1}_{2,1}         |
| Coefficients for SVs of class 0                                                | Coefficients for SVs of class 1                     | Coefficients for SVs of class 2                     |

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_svm_plot_iris_svc.py*,

1.4.1.2. Scores and probabilities

The "decision_function" method of "SVC" and "NuSVC" gives per-class
scores for each sample (or a single score per sample in the binary
case). When the constructor option "probability" is set to "True",
class membership probability estimates (from the methods
"predict_proba" and "predict_log_proba") are enabled. In the binary
case, the probabilities are calibrated using Platt scaling [9]:
logistic regression on the SVM's scores, fit by an additional cross-
validation on the training data. In the multiclass case, this is
extended as per [10].

Note:

  The same probability calibration procedure is available for all
  estimators via the "CalibratedClassifierCV" (see Probability
  calibration). In the case of "SVC" and "NuSVC", this procedure is
  builtin in libsvm which is used under the hood, so it does not rely
  on scikit-learn's "CalibratedClassifierCV".

The cross-validation involved in Platt scaling is an expensive
operation for large datasets. In addition, the probability estimates

on scikit-learn's "CalibratedClassifierCV".

may be inconsistent with the scores:

* the "argmax" of the scores may not be the argmax of the
  probabilities

* in binary classification, a sample may be labeled by "predict" as
  belonging to the positive class even if the output of
  *predict_proba* is less than 0.5; and similarly, it could be labeled
  as negative even if the output of *predict_proba* is more than 0.5.

Platt's method is also known to have theoretical issues. If confidence
scores are required, but these do not have to be probabilities, then
it is advisable to set "probability=False" and use "decision_function"
instead of "predict_proba".

Please note that when "decision_function_shape='ovr'" and "n_classes >
2", unlike "decision_function", the "predict" method does not try to
break ties by default. You can set "break_ties=True" for the output of
"predict" to be the same as "np.argmax(clf.decision_function(...),
axis=1)", otherwise the first class among the tied classes will always
be returned; but have in mind that it comes with a computational cost.
See *sphx_glr_auto_examples_svm_plot_svm_tie_breaking.py* for an
example on tie breaking.

1.4.1.3. Unbalanced problems

In problems where it is desired to give more importance to certain
classes or certain individual samples, the parameters "class_weight"

and "sample_weight" can be used.

"SVC" (but not "NuSVC") implements the parameter "class_weight" in the
"fit" method. It's a dictionary of the form "{class_label : value}",
where value is a floating point number > 0 that sets the parameter "C"
of class "class_label" to "C * value". The figure below illustrates
the decision boundary of an unbalanced problem, with and without
weight correction.

   [image]

"SVC", "NuSVC", "SVR", "NuSVR", "LinearSVC", "LinearSVR" and
"OneClassSVM" implement also weights for individual samples in the
*fit* method through the "sample_weight" parameter. Similar to
"class_weight", this sets the parameter "C" for the i-th example to "C
* sample_weight[i]", which will encourage the classifier to get these
samples right. The figure below illustrates the effect of sample
weighting on the decision boundary. The size of the circles is
proportional to the sample weights:

   [image]

Examples:
^^^^^^^^^

  y*

* *sphx_glr_auto_examples_svm_plot_weighted_samples.py*,

1.4.2. Regression

The method of Support Vector Classification can be extended to solve
regression problems. This method is called Support Vector Regression.

The model produced by support vector classification (as described

above) depends only on a subset of the training data, because the cost
function for building the model does not care about training points
that lie beyond the margin. Analogously, the model produced by Support
Vector Regression depends only on a subset of the training data,
because the cost function ignores samples whose prediction is close to
their target.

There are three different implementations of Support Vector
Regression: "SVR", "NuSVR" and "LinearSVR". "LinearSVR" provides a
faster implementation than "SVR" but only considers the linear kernel,
while "NuSVR" implements a slightly different formulation than "SVR"
and "LinearSVR". See Implementation details for further details.

As with classification classes, the fit method will take as argument
vectors X, y, only that in this case y is expected to have floating
point values instead of integer values:

   >>> X = [[0, 0], [2, 2]]
   >>> y = [0.5, 2.5]
   >>> regr = svm.SVR()
   >>> regr.fit(X, y)
   SVR()
   >>> regr.predict([[1, 1]])
   array([1.5])

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_svm_plot_svm_regression.py*

1.4.3. Density estimation, novelty detection

The class "OneClassSVM" implements a One-Class SVM which is used in
outlier detection.

See Novelty and Outlier Detection for the description and usage of
OneClassSVM.

1.4.4. Complexity

Support Vector Machines are powerful tools, but their compute and
storage requirements increase rapidly with the number of training
vectors. The core of an SVM is a quadratic programming problem (QP),
separating support vectors from the rest of the training data. The QP
solver used by the libsvm-based implementation scales between
O(n_{features} \times n_{samples}^2) and O(n_{features} \times
n_{samples}^3) depending on how efficiently the libsvm cache is used
in practice (dataset dependent). If the data is very sparse
n_{features} should be replaced by the average number of non-zero
features in a sample vector.

For the linear case, the algorithm used in "LinearSVC" by the
liblinear implementation is much more efficient than its libsvm-based
"SVC" counterpart and can scale almost linearly to millions of samples
and/or features.

1.4.5. Tips on Practical Use

   * **Avoiding data copy**: For "SVC", "SVR", "NuSVC" and "NuSVR", if
     the data passed to certain methods is not C-ordered contiguous
     and double precision, it will be copied before calling the
     underlying C implementation. You can check whether a given numpy
     array is C-contiguous by inspecting its "flags" attribute.

     For "LinearSVC" (and "LogisticRegression") any input passed as a

array is C-contiguous by inspecting its "flags" attribute.

     numpy array will be copied and converted to the liblinear
     internal sparse data representation (double precision floats and
     int32 indices of non-zero components). If you want to fit a
     large-scale linear classifier without copying a dense numpy
     C-contiguous double precision array as input, we suggest to use
     the "SGDClassifier" class instead.  The objective function can be
     configured to be almost the same as the "LinearSVC" model.

   * **Kernel cache size**: For "SVC", "SVR", "NuSVC" and "NuSVR", the
     size of the kernel cache has a strong impact on run times for
     larger problems.  If you have enough RAM available, it is
     recommended to set "cache_size" to a higher value than the
     default of 200(MB), such as 500(MB) or 1000(MB).

   * **Setting C**: "C" is "1" by default and it's a reasonable
     default choice.  If you have a lot of noisy observations you
     should decrease it: decreasing C corresponds to more
     regularization.

     "LinearSVC" and "LinearSVR" are less sensitive to "C" when it
     becomes large, and prediction results stop improving after a
     certain threshold. Meanwhile, larger "C" values will take more
     time to train, sometimes up to 10 times longer, as shown in [11].

   * Support Vector Machine algorithms are not scale invariant, so

time to train, sometimes up to 10 times longer, as shown in [11].

     **it is highly recommended to scale your data**. For example,
     scale each attribute on the input vector X to [0,1] or [-1,+1],
     or standardize it to have mean 0 and variance 1. Note that the
     meaningful results. This can be done easily by using a
     "Pipeline":

        >>> from sklearn.pipeline import make_pipeline
        >>> from sklearn.preprocessing import StandardScaler
        >>> from sklearn.svm import SVC

        >>> clf = make_pipeline(StandardScaler(), SVC())

     See section Preprocessing data for more details on scaling and
     normalization.

   * Regarding the *shrinking* parameter, quoting [12]: *We found that
     if the number of iterations is large, then shrinking can shorten
     the training time. However, if we loosely solve the optimization
     problem (e.g., by using a large stopping tolerance), the code
     without using shrinking may be much faster*

   * Parameter "nu" in "NuSVC"/"OneClassSVM"/"NuSVR" approximates the
     fraction of training errors and support vectors.

   * In "SVC", if the data is unbalanced (e.g. many positive and few
     negative), set "class_weight='balanced'" and/or try different
     penalty parameters "C".

   * **Randomness of the underlying implementations**: The underlying

negative), set "class_weight='balanced'" and/or try different

     implementations of "SVC" and "NuSVC" use a random number
     generator only to shuffle the data for probability estimation
     (when "probability" is set to "True"). This randomness can be
     controlled with the "random_state" parameter. If "probability" is
     set to "False" these estimators are not random and "random_state"
     has no effect on the results. The underlying "OneClassSVM"
     implementation is similar to the ones of "SVC" and "NuSVC". As no
     probability estimation is provided for "OneClassSVM", it is not
     random.

     The underlying "LinearSVC" implementation uses a random number
     generator to select features when fitting the model with a dual
     coordinate descent (i.e when "dual" is set to "True"). It is thus
     not uncommon to have slightly different results for the same
     input data. If that happens, try with a smaller *tol* parameter.
     This randomness can also be controlled with the "random_state"
     parameter. When "dual" is set to "False" the underlying
     implementation of "LinearSVC" is not random and "random_state"
     has no effect on the results.

   * Using L1 penalization as provided by "LinearSVC(penalty='l1',
     dual=False)" yields a sparse solution, i.e. only a subset of
     feature weights is different from zero and contribute to the

* Using L1 penalization as provided by "LinearSVC(penalty='l1',
     decision function. Increasing "C" yields a more complex model
     (more features are selected). The "C" value that yields a "null"
     model (all weights equal to zero) can be calculated using
     "l1_min_c()".

1.4.6. Kernel functions

The *kernel function* can be any of the following:

   * linear: \langle x, x'\rangle.

   * polynomial: (\gamma \langle x, x'\rangle + r)^d, where d is
     specified by parameter "degree", r by "coef0".

   * rbf: \exp(-\gamma \|x-x'\|^2), where \gamma is specified by
     parameter "gamma", must be greater than 0.

   * sigmoid \tanh(\gamma \langle x,x'\rangle + r), where r is
     specified by "coef0".

Different kernels are specified by the *kernel* parameter:

   >>> linear_svc = svm.SVC(kernel='linear')
   >>> linear_svc.kernel
   'linear'
   >>> rbf_svc = svm.SVC(kernel='rbf')
   >>> rbf_svc.kernel
   'rbf'

See also Kernel Approximation for a solution to use RBF kernels that
is much faster and more scalable.

1.4.6.1. Parameters of the RBF Kernel

When training an SVM with the *Radial Basis Function* (RBF) kernel,
two parameters must be considered: "C" and "gamma".  The parameter
"C", common to all SVM kernels, trades off misclassification of
training examples against simplicity of the decision surface. A low

"C" makes the decision surface smooth, while a high "C" aims at
classifying all training examples correctly.  "gamma" defines how much
influence a single training example has. The larger "gamma" is, the
closer other examples must be to be affected.

Proper choice of "C" and "gamma" is critical to the SVM's performance.
One is advised to use "GridSearchCV" with "C" and "gamma" spaced
exponentially far apart to choose good values.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_svm_plot_rbf_parameters.py*

1.4.6.2. Custom Kernels

You can define your own kernels by either giving the kernel as a
python function or by precomputing the Gram matrix.

Classifiers with custom kernels behave the same way as any other
classifiers, except that:

   * Field "support_vectors_" is now empty, only indices of support
     vectors are stored in "support_"

   * A reference (and not a copy) of the first argument in the "fit()"
     method is stored for future reference. If that array changes
     between the use of "fit()" and "predict()" you will have
     unexpected results.

1.4.6.2.1. Using Python functions as kernels
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can use your own defined kernels by passing a function to the
"kernel" parameter.

unexpected results.

Your kernel must take as arguments two matrices of shape
"(n_samples_1, n_features)", "(n_samples_2, n_features)" and return a
kernel matrix of shape "(n_samples_1, n_samples_2)".

The following code defines a linear kernel and creates a classifier
instance that will use that kernel:

   >>> def my_kernel(X, Y):
   ...     return np.dot(X, Y.T)
   ...
   >>> clf = svm.SVC(kernel=my_kernel)

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_svm_plot_custom_kernel.py*.

1.4.6.2.2. Using the Gram matrix
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can pass pre-computed kernels by using the "kernel='precomputed'"
option. You should then pass Gram matrix instead of X to the *fit* and
*predict* methods. The kernel values between *all* training vectors
and the test vectors must be provided:

>>> from sklearn.datasets import make_classification
>>> from sklearn import svm
>>> X, y = make_classification(n_samples=10, random_state=0)
>>> X_train , X_test , y_train, y_test = train_test_split(X, y, random_state=0)
>>> clf = svm.SVC(kernel='precomputed')
>>> # linear kernel computation
>>> gram_train = np.dot(X_train, X_train.T)
>>> clf.fit(gram_train, y_train)
SVC(kernel='precomputed')

>>> # predict on training examples
>>> gram_test = np.dot(X_test, X_train.T)
>>> clf.predict(gram_test)
array([0, 1, 0])

1.4.7. Mathematical formulation

A support vector machine constructs a hyper-plane or set of hyper-
planes in a high or infinite dimensional space, which can be used for
classification, regression or other tasks. Intuitively, a good
separation is achieved by the hyper-plane that has the largest
distance to the nearest training data points of any class (so-called
functional margin), since in general the larger the margin the lower
the generalization error of the classifier. The figure below shows the
decision function for a linearly separable problem, with three samples
on the margin boundaries, called "support vectors":

   [image]

In general, when the problem isn't linearly separable, the support
vectors are the samples *within* the margin boundaries.

We recommend [13] and [14] as good references for the theory and
practicalities of SVMs.

1.4.7.1. SVC

Given training vectors x_i \in \mathbb{R}^p, i=1,..., n, in two
classes, and a vector y \in \{1, -1\}^n, our goal is to find w \in
\mathbb{R}^p and b \in \mathbb{R} such that the prediction given by
\text{sign} (w^T\phi(x) + b) is correct for most samples.

SVC solves the following primal problem:

   \min_ {w, b, \zeta} \frac{1}{2} w^T w + C \sum_{i=1}^{n} \zeta_i
   \textrm {subject to } & y_i (w^T \phi (x_i) + b) \geq 1 -
   \zeta_i,\\ & \zeta_i \geq 0, i=1, ..., n

Intuitively, we're trying to maximize the margin (by minimizing
||w||^2 = w^Tw), while incurring a penalty when a sample is
misclassified or within the margin boundary. Ideally, the value y_i
(w^T \phi (x_i) + b) would be \geq 1 for all samples, which indicates
a perfect prediction. But problems are usually not always perfectly
separable with a hyperplane, so we allow some samples to be at a
distance \zeta_i from their correct margin boundary. The penalty term
*C* controls the strength of this penalty, and as a result, acts as an
inverse regularization parameter (see note below).

The dual problem to the primal is

   \min_{\alpha} \frac{1}{2} \alpha^T Q \alpha - e^T \alpha   \textrm
   {subject to } & y^T \alpha = 0\\ & 0 \leq \alpha_i \leq C, i=1,
   ..., n

where e is the vector of all ones, and Q is an n by n positive
semidefinite matrix, Q_{ij} \equiv y_i y_j K(x_i, x_j), where K(x_i,
x_j) = \phi (x_i)^T \phi (x_j) is the kernel. The terms \alpha_i are
called the dual coefficients, and they are upper-bounded by C. This
dual representation highlights the fact that training vectors are
implicitly mapped into a higher (maybe infinite) dimensional space by

the function \phi: see kernel trick.

Once the optimization problem is solved, the output of
*decision_function* for a given sample x becomes:

   \sum_{i\in SV} y_i \alpha_i K(x_i, x) + b,

and the predicted class correspond to its sign. We only need to sum
over the support vectors (i.e. the samples that lie within the margin)
because the dual coefficients \alpha_i are zero for the other samples.

These parameters can be accessed through the attributes "dual_coef_"
which holds the product y_i \alpha_i, "support_vectors_" which holds
the support vectors, and "intercept_" which holds the independent term
b

Note:

  While SVM models derived from libsvm and liblinear use "C" as
  regularization parameter, most other estimators use "alpha". The
  exact equivalence between the amount of regularization of two models
  depends on the exact objective function optimized by the model. For
  example, when the estimator used is "Ridge" regression, the relation
  between them is given as C = \frac{1}{alpha}.

1.4.7.2. LinearSVC

The primal problem can be equivalently formulated as

   \min_ {w, b} \frac{1}{2} w^T w + C \sum_{i=1}^{n}\max(0, 1 - y_i
   (w^T \phi(x_i) + b)),

where we make use of the hinge loss. This is the form that is directly
optimized by "LinearSVC", but unlike the dual form, this one does not

(w^T \phi(x_i) + b)),

involve inner products between samples, so the famous kernel trick
cannot be applied. This is why only the linear kernel is supported by
"LinearSVC" (\phi is the identity function).

1.4.7.3. NuSVC

The \nu-SVC formulation [15] is a reparameterization of the C-SVC and
therefore mathematically equivalent.

We introduce a new parameter \nu (instead of C) which controls the
number of support vectors and *margin errors*: \nu \in (0, 1] is an
upper bound on the fraction of margin errors and a lower bound of the
fraction of support vectors. A margin error corresponds to a sample
that lies on the wrong side of its margin boundary: it is either
misclassified, or it is correctly classified but does not lie beyond
the margin.

1.4.7.4. SVR

Given training vectors x_i \in \mathbb{R}^p, i=1,..., n, and a vector
y \in \mathbb{R}^n \varepsilon-SVR solves the following primal
problem:

   \min_ {w, b, \zeta, \zeta^*} \frac{1}{2} w^T w + C \sum_{i=1}^{n}
   (\zeta_i + \zeta_i^*)    \textrm {subject to } & y_i - w^T \phi
   (x_i) - b \leq \varepsilon + \zeta_i,\\                       & w^T
   \phi (x_i) + b - y_i \leq \varepsilon + \zeta_i^*,\\
   & \zeta_i, \zeta_i^* \geq 0, i=1, ..., n

Here, we are penalizing samples whose prediction is at least
\varepsilon away from their true target. These samples penalize the

& \zeta_i, \zeta_i^* \geq 0, i=1, ..., n

objective by \zeta_i or \zeta_i^*, depending on whether their
predictions lie above or below the \varepsilon tube.

The dual problem is

   \min_{\alpha, \alpha^*} \frac{1}{2} (\alpha - \alpha^*)^T Q (\alpha
   - \alpha^*) + \varepsilon e^T (\alpha + \alpha^*) - y^T (\alpha -
   \alpha^*)   \textrm {subject to } & e^T (\alpha - \alpha^*) = 0\\ &
   0 \leq \alpha_i, \alpha_i^* \leq C, i=1, ..., n

where e is the vector of all ones, Q is an n by n positive
semidefinite matrix, Q_{ij} \equiv K(x_i, x_j) = \phi (x_i)^T \phi
(x_j) is the kernel. Here training vectors are implicitly mapped into
a higher (maybe infinite) dimensional space by the function \phi.

The prediction is:

   \sum_{i \in SV}(\alpha_i - \alpha_i^*) K(x_i, x) + b

which holds the difference \alpha_i - \alpha_i^*, "support_vectors_"
which holds the support vectors, and "intercept_" which holds the
independent term b

1.4.7.5. LinearSVR

   \min_ {w, b} \frac{1}{2} w^T w + C \sum_{i=1}^{n}\max(0, |y_i -
   (w^T \phi(x_i) + b)| - \varepsilon),

where we make use of the epsilon-insensitive loss, i.e. errors of less
than \varepsilon are ignored. This is the form that is directly
optimized by "LinearSVR".

1.4.8. Implementation details

Internally, we use libsvm [12] and liblinear [11] to handle all
computations. These libraries are wrapped using C and Cython. For a
description of the implementation and details of the algorithms used,
please refer to their respective papers.

[9] Platt "Probabilistic outputs for SVMs and comparisons to
    regularized likelihood methods".

[10] Wu, Lin and Weng, "Probability estimates for multi-class
     classification by pairwise coupling", JMLR 5:975-1005, 2004.

[11] Fan, Rong-En, et al., "LIBLINEAR: A library for large linear
     classification.", Journal of machine learning research 9.Aug
     (2008): 1871-1874.

[12] Chang and Lin, LIBSVM: A Library for Support Vector Machines.

[13] Bishop, Pattern recognition and machine learning, chapter 7
     Sparse Kernel Machines

[14] >>:doi:`"A Tutorial on Support Vector Regression"
     <10.1023/B:STCO.0000035301.49549.88>`<< Alex J. Smola, Bernhard
     Schölkopf - Statistics and Computing archive Volume 14 Issue 3,
     August 2004, p. 199-222.

[15] Schölkopf et. al New Support Vector Algorithms

[16] Crammer and Singer On the Algorithmic Implementation ofMulticlass
     Kernel-based Vector Machines, JMLR 2001.

3.4. Validation curves: plotting scores to evaluate models

Every estimator has its advantages and drawbacks. Its generalization
error can be decomposed in terms of bias, variance and noise. The
**bias** of an estimator is its average error for different training
sets. The **variance** of an estimator indicates how sensitive it is
to varying training sets. Noise is a property of the data.

In the following plot, we see a function f(x) = \cos (\frac{3}{2} \pi
x) and some noisy samples from that function. We use three different
estimators to fit the function: linear regression with polynomial
features of degree 1, 4 and 15. We see that the first estimator can at
best provide only a poor fit to the samples and the true function
because it is too simple (high bias), the second estimator
approximates it almost perfectly and the last estimator approximates
the training data perfectly but does not fit the true function very
well, i.e. it is very sensitive to varying training data (high
variance).

   [image]

Bias and variance are inherent properties of estimators and we usually
have to select learning algorithms and hyperparameters so that both
bias and variance are as low as possible (see Bias-variance dilemma).
Another way to reduce the variance of a model is to use more training
data. However, you should only collect more training data if the true
function is too complex to be approximated by an estimator with a
lower variance.

In the simple one-dimensional problem that we have seen in the example
it is easy to see whether the estimator suffers from bias or variance.
However, in high-dimensional spaces, models can become very difficult
to visualize. For this reason, it is often helpful to use the tools
described below.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_model_selection_plot_underfitting_overfitti
  ng.py*

* *sphx_glr_auto_examples_model_selection_plot_validation_curve.py*

* *sphx_glr_auto_examples_model_selection_plot_learning_curve.py*

3.4.1. Validation curve

To validate a model we need a scoring function (see Metrics and
scoring: quantifying the quality of predictions), for example accuracy
for classifiers. The proper way of choosing multiple hyperparameters
of an estimator is of course grid search or similar methods (see
Tuning the hyper-parameters of an estimator) that select the
hyperparameter with the maximum score on a validation set or multiple
validation sets. Note that if we optimize the hyperparameters based on
a validation score the validation score is biased and not a good
estimate of the generalization any longer. To get a proper estimate of
the generalization we have to compute the score on another test set.

However, it is sometimes helpful to plot the influence of a single

hyperparameter on the training score and the validation score to find
out whether the estimator is overfitting or underfitting for some
hyperparameter values.

The function "validation_curve()" can help in this case:

   >>> from sklearn.model_selection import validation_curve
   >>> from sklearn.linear_model import Ridge

   >>> indices = np.arange(y.shape[0])
   >>> np.random.shuffle(indices)
   >>> X, y = X[indices], y[indices]

   >>> train_scores, valid_scores = validation_curve(
   ...     Ridge(), X, y, param_name="alpha", param_range=np.logspace(-7, 3, 3),
   ...     cv=5)
   >>> train_scores
   array([[0.93..., 0.94..., 0.92..., 0.91..., 0.92...],
          [0.93..., 0.94..., 0.92..., 0.91..., 0.92...],
          [0.51..., 0.52..., 0.49..., 0.47..., 0.49...]])
   >>> valid_scores
   array([[0.90..., 0.84..., 0.94..., 0.96..., 0.93...],
          [0.90..., 0.84..., 0.94..., 0.96..., 0.93...],
          [0.46..., 0.25..., 0.50..., 0.49..., 0.52...]])

If the training score and the validation score are both low, the
estimator will be underfitting. If the training score is high and the
validation score is low, the estimator is overfitting and otherwise it
is working very well. A low training score and a high validation score

is usually not possible. Underfitting, overfitting, and a working
model are shown in the in the plot below where we vary the parameter
\gamma of an SVM on the digits dataset.

   [image]

3.4.2. Learning curve

A learning curve shows the validation and training score of an
estimator for varying numbers of training samples. It is a tool to
find out how much we benefit from adding more training data and
whether the estimator suffers more from a variance error or a bias
error. Consider the following example where we plot the learning curve
of a naive Bayes classifier and an SVM.

For the naive Bayes, both the validation score and the training score
converge to a value that is quite low with increasing size of the
training set. Thus, we will probably not benefit much from more
training data.

In contrast, for small amounts of data, the training score of the SVM
is much greater than the validation score. Adding more training
samples will most likely increase generalization.

   [image]

We can use the function "learning_curve()" to generate the values that
are required to plot such a learning curve (number of samples that
have been used, the average scores on the training sets and the
average scores on the validation sets):

   >>> from sklearn.model_selection import learning_curve

   >>> train_sizes, train_scores, valid_scores = learning_curve(
   ...     SVC(kernel='linear'), X, y, train_sizes=[50, 80, 110], cv=5)
   >>> train_sizes
   array([ 50, 80, 110])
   array([[0.98..., 0.98 , 0.98..., 0.98..., 0.98...],
          [0.98..., 1.   , 0.98..., 0.98..., 0.98...],
          [0.98..., 1.   , 0.98..., 0.98..., 0.99...]])
   array([[1. ,  0.93...,  1. ,  1. ,  0.96...],
          [1. ,  0.96...,  1. ,  1. ,  0.96...],
          [1. ,  0.96...,  1. ,  1. ,  0.96...]])

If you intend to plot the learning curves only, the class
"LearningCurveDisplay" will be easier to use. You can use the method
"from_estimator()" similarly to "learning_curve()" to generate and
plot the learning curve:

1.15. Isotonic regression

The class "IsotonicRegression" fits a non-decreasing real function to
1-dimensional data. It solves the following problem:

   minimize \sum_i w_i (y_i - \hat{y}_i)^2

   subject to \hat{y}_i \le \hat{y}_j whenever X_i \le X_j,

where the weights w_i are strictly positive, and both *X* and *y* are
arbitrary real quantities.

The *increasing* parameter changes the constraint to \hat{y}_i \ge
\hat{y}_j whenever X_i \le X_j. Setting it to 'auto' will
automatically choose the constraint based on Spearman's rank
correlation coefficient.

"IsotonicRegression" produces a series of predictions \hat{y}_i for
the training data which are the closest to the targets y in terms of
mean squared error. These predictions are interpolated for predicting
to unseen data. The predictions of "IsotonicRegression" thus form a
function that is piecewise linear:

   [image]

2.2. Manifold learning

   Look for the bare necessities
   The simple bare necessities
   Forget about your worries and your strife
   I mean the bare necessities
   Old Mother Nature's recipes
   That bring the bare necessities of life

      -- Baloo's song [The Jungle Book]

   [image]

[image: manifold_img3][image] [image: manifold_img4][image] [image:
manifold_img5][image] [image: manifold_img6][image]

Manifold learning is an approach to non-linear dimensionality
reduction. Algorithms for this task are based on the idea that the
dimensionality of many data sets is only artificially high.

2.2.1. Introduction

High-dimensional datasets can be very difficult to visualize.  While
data in two or three dimensions can be plotted to show the inherent
structure of the data, equivalent high-dimensional plots are much less
intuitive.  To aid visualization of the structure of a dataset, the
dimension must be reduced in some way.

The simplest way to accomplish this dimensionality reduction is by
taking a random projection of the data.  Though this allows some
degree of visualization of the data structure, the randomness of the
choice leaves much to be desired.  In a random projection, it is
likely that the more interesting structure within the data will be
lost.

[image: digits_img][image] [image: projected_img][image]

To address this concern, a number of supervised and unsupervised
linear dimensionality reduction frameworks have been designed, such as

lost.

Principal Component Analysis (PCA), Independent Component Analysis,
Linear Discriminant Analysis, and others.  These algorithms define
specific rubrics to choose an "interesting" linear projection of the
data. These methods can be powerful, but often miss important non-
linear structure in the data.

[image: PCA_img][image] [image: LDA_img][image]

Manifold Learning can be thought of as an attempt to generalize linear
frameworks like PCA to be sensitive to non-linear structure in data.
Though supervised variants exist, the typical manifold learning
problem is unsupervised: it learns the high-dimensional structure of
the data from the data itself, without the use of predetermined
classifications.

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_manifold_plot_lle_digits.py* for an
  example of dimensionality reduction on handwritten digits.

* See *sphx_glr_auto_examples_manifold_plot_compare_methods.py* for an
  example of dimensionality reduction on a toy "S-curve" dataset.

The manifold learning implementations available in scikit-learn are
summarized below

2.2.2. Isomap

One of the earliest approaches to manifold learning is the Isomap
algorithm, short for Isometric Mapping.  Isomap can be viewed as an
extension of Multi-dimensional Scaling (MDS) or Kernel PCA. Isomap

seeks a lower-dimensional embedding which maintains geodesic distances
between all points.  Isomap can be performed with the object "Isomap".

   [image]

2.2.2.1. Complexity

The Isomap algorithm comprises three stages:

1. **Nearest neighbor search.**  Isomap uses "BallTree" for efficient
   neighbor search. The cost is approximately O[D \log(k) N \log(N)],
   for k nearest neighbors of N points in D dimensions.

2. **Shortest-path graph search.**  The most efficient known
   algorithms for this are *Dijkstra's Algorithm*, which is
   approximately O[N^2(k + \log(N))], or the *Floyd-Warshall
   algorithm*, which is O[N^3].  The algorithm can be selected by the
   user with the "path_method" keyword of "Isomap".  If unspecified,
   the code attempts to choose the best algorithm for the input data.

3. **Partial eigenvalue decomposition.**  The embedding is encoded in
   the eigenvectors corresponding to the d largest eigenvalues of the
   N \times N isomap kernel.  For a dense solver, the cost is
   approximately O[d N^2].  This cost can often be improved using the
   "ARPACK" solver.  The eigensolver can be specified by the user with
   the "eigen_solver" keyword of "Isomap".  If unspecified, the code
   attempts to choose the best algorithm for the input data.

"ARPACK" solver.  The eigensolver can be specified by the user with

The overall complexity of Isomap is O[D \log(k) N \log(N)] + O[N^2(k +
\log(N))] + O[d N^2].

* N : number of training data points

* D : input dimension

* k : number of nearest neighbors

* d : output dimension

* "A global geometric framework for nonlinear dimensionality
  reduction" Tenenbaum, J.B.; De Silva, V.; & Langford, J.C.  Science
  290 (5500)

2.2.3. Locally Linear Embedding

Locally linear embedding (LLE) seeks a lower-dimensional projection of
the data which preserves distances within local neighborhoods.  It can
be thought of as a series of local Principal Component Analyses which
are globally compared to find the best non-linear embedding.

Locally linear embedding can be performed with function
"locally_linear_embedding()" or its object-oriented counterpart
"LocallyLinearEmbedding".

   [image]

2.2.3.1. Complexity

The standard LLE algorithm comprises three stages:

1. **Nearest Neighbors Search**.  See discussion under Isomap above.

2. **Weight Matrix Construction**. O[D N k^3]. The construction of the
   LLE weight matrix involves the solution of a k \times k linear
   equation for each of the N local neighborhoods

3. **Partial Eigenvalue Decomposition**. See discussion under Isomap
   above.

LLE weight matrix involves the solution of a k \times k linear

   above.

The overall complexity of standard LLE is O[D \log(k) N \log(N)] + O[D
N k^3] + O[d N^2].

* "Nonlinear dimensionality reduction by locally linear embedding"
  Roweis, S. & Saul, L.  Science 290:2323 (2000)

2.2.4. Modified Locally Linear Embedding

One well-known issue with LLE is the regularization problem.  When the
number of neighbors is greater than the number of input dimensions,
the matrix defining each local neighborhood is rank-deficient.  To
address this, standard LLE applies an arbitrary regularization
parameter r, which is chosen relative to the trace of the local weight
matrix.  Though it can be shown formally that as r \to 0, the solution
converges to the desired embedding, there is no guarantee that the
optimal solution will be found for r > 0.  This problem manifests
itself in embeddings which distort the underlying geometry of the
manifold.

One method to address the regularization problem is to use multiple
weight vectors in each neighborhood.  This is the essence of *modified
locally linear embedding* (MLLE).  MLLE can be  performed with
function "locally_linear_embedding()" or its object-oriented

counterpart "LocallyLinearEmbedding", with the keyword "method =
'modified'". It requires "n_neighbors > n_components".

   [image]

2.2.4.1. Complexity

The MLLE algorithm comprises three stages:

1. **Nearest Neighbors Search**.  Same as standard LLE

2. **Weight Matrix Construction**. Approximately O[D N k^3] + O[N
   (k-D) k^2].  The first term is exactly equivalent to that of
   standard LLE.  The second term has to do with constructing the
   weight matrix from multiple weights.  In practice, the added cost
   of constructing the MLLE weight matrix is relatively small compared
   to the cost of stages 1 and 3.

3. **Partial Eigenvalue Decomposition**. Same as standard LLE

The overall complexity of MLLE is O[D \log(k) N \log(N)] + O[D N k^3]
+ O[N (k-D) k^2] + O[d N^2].

* "MLLE: Modified Locally Linear Embedding Using Multiple Weights"
  Zhang, Z. & Wang, J.

2.2.5. Hessian Eigenmapping

Hessian Eigenmapping (also known as Hessian-based LLE: HLLE) is
another method of solving the regularization problem of LLE.  It
revolves around a hessian-based quadratic form at each neighborhood

which is used to recover the locally linear structure.  Though other
implementations note its poor scaling with data size, "sklearn"
implements some algorithmic improvements which make its cost
comparable to that of other LLE variants for small output dimension.
HLLE can be  performed with function "locally_linear_embedding()" or
its object-oriented counterpart "LocallyLinearEmbedding", with the
keyword "method = 'hessian'". It requires "n_neighbors > n_components
* (n_components + 3) / 2".

   [image]

2.2.5.1. Complexity

The HLLE algorithm comprises three stages:

   d^6].  The first term reflects a similar cost to that of standard
   LLE.  The second term comes from a QR decomposition of the local
   hessian estimator.

The overall complexity of standard HLLE is O[D \log(k) N \log(N)] +
O[D N k^3] + O[N d^6] + O[d N^2].

* "Hessian Eigenmaps: Locally linear embedding techniques for high-
  dimensional data" Donoho, D. & Grimes, C. Proc Natl Acad Sci USA.
  100:5591 (2003)

2.2.6. Spectral Embedding

Spectral Embedding is an approach to calculating a non-linear
embedding. Scikit-learn implements Laplacian Eigenmaps, which finds a
low dimensional representation of the data using a spectral
decomposition of the graph Laplacian. The graph generated can be
considered as a discrete approximation of the low dimensional manifold
in the high dimensional space. Minimization of a cost function based
on the graph ensures that points close to each other on the manifold
are mapped close to each other in the low dimensional space,
preserving local distances. Spectral embedding can be  performed with
the function "spectral_embedding()" or its object-oriented counterpart
"SpectralEmbedding".

2.2.6.1. Complexity

The Spectral Embedding (Laplacian Eigenmaps) algorithm comprises three
stages:

1. **Weighted Graph Construction**. Transform the raw input data into
   graph representation using affinity (adjacency) matrix
   representation.

2. **Graph Laplacian Construction**. unnormalized Graph Laplacian is
   constructed as L = D - A for and normalized one as L =
   D^{-\frac{1}{2}} (D - A) D^{-\frac{1}{2}}.

3. **Partial Eigenvalue Decomposition**. Eigenvalue decomposition is
   done on graph Laplacian

constructed as L = D - A for and normalized one as L =

The overall complexity of spectral embedding is O[D \log(k) N \log(N)]
+ O[D N k^3] + O[d N^2].

* "Laplacian Eigenmaps for Dimensionality Reduction and Data
  Representation" M. Belkin, P. Niyogi, Neural Computation, June 2003;
  15 (6):1373-1396

2.2.7. Local Tangent Space Alignment

Though not technically a variant of LLE, Local tangent space alignment
(LTSA) is algorithmically similar enough to LLE that it can be put in
this category. Rather than focusing on preserving neighborhood
distances as in LLE, LTSA seeks to characterize the local geometry at
each neighborhood via its tangent space, and performs a global
optimization to align these local tangent spaces to learn the
embedding.  LTSA can be performed with function
"LocallyLinearEmbedding", with the keyword "method = 'ltsa'".

   [image]

2.2.7.1. Complexity

The LTSA algorithm comprises three stages:

2. **Weight Matrix Construction**. Approximately O[D N k^3] + O[k^2
   d].  The first term reflects a similar cost to that of standard
   LLE.

   LLE.

The overall complexity of standard LTSA is O[D \log(k) N \log(N)] +
O[D N k^3] + O[k^2 d] + O[d N^2].

* >>:arxiv:`"Principal manifolds and nonlinear dimensionality
  reduction via tangent space alignment" <cs/0212008>`<< Zhang, Z. &
  Zha, H. Journal of Shanghai Univ. 8:406 (2004)

2.2.8. Multi-dimensional Scaling (MDS)

Multidimensional scaling ("MDS") seeks a low-dimensional
representation of the data in which the distances respect well the
distances in the original high-dimensional space.

In general, "MDS" is a technique used for analyzing similarity or
dissimilarity data. It attempts to model similarity or dissimilarity
data as distances in a geometric spaces. The data can be ratings of
similarity between objects, interaction frequencies of molecules, or
trade indices between countries.

There exists two types of MDS algorithm: metric and non metric. In
scikit-learn, the class "MDS" implements both. In Metric MDS, the
input similarity matrix arises from a metric (and thus respects the
triangular inequality), the distances between output two points are

then set to be as close as possible to the similarity or dissimilarity
data. In the non-metric version, the algorithms will try to preserve
the order of the distances, and hence seek for a monotonic
relationship between the distances in the embedded space and the
similarities/dissimilarities.

   [image]

Let S be the similarity matrix, and X the coordinates of the n input
points. Disparities \hat{d}_{ij} are transformation of the
similarities chosen in some optimal ways. The objective, called the
stress, is then defined by \sum_{i < j} d_{ij}(X) - \hat{d}_{ij}(X)

2.2.8.1. Metric MDS

The simplest metric "MDS" model, called *absolute MDS*, disparities
are defined by \hat{d}_{ij} = S_{ij}. With absolute MDS, the value
S_{ij} should then correspond exactly to the distance between point i
and j in the embedding point.

Most commonly, disparities are set to \hat{d}_{ij} = b S_{ij}.

2.2.8.2. Nonmetric MDS

Non metric "MDS" focuses on the ordination of the data. If S_{ij} >
S_{jk}, then the embedding should enforce d_{ij} < d_{jk}. For this
reason, we discuss it in terms of dissimilarities (\delta_{ij})
instead of similarities (S_{ij}). Note that dissimilarities can easily
be obtained from similarities through a simple transform, e.g.

\delta_{ij}=c_1-c_2 S_{ij} for some real constants c_1, c_2. A simple
algorithm to enforce proper ordination is to use a monotonic
regression of d_{ij} on \delta_{ij}, yielding disparities \hat{d}_{ij}
in the same order as \delta_{ij}.

A trivial solution to this problem is to set all the points on the
origin. In order to avoid that, the disparities \hat{d}_{ij} are
normalized. Note that since we only care about relative ordering, our
objective should be invariant to simple translation and scaling,
however the stress used in metric MDS is sensitive to scaling. To
address this, non-metric MDS may use a normalized stress, known as
Stress-1 defined as

   \sqrt{\frac{\sum_{i < j} (d_{ij} - \hat{d}_{ij})^2}{\sum_{i < j}
   d_{ij}^2}}.

The use of normalized Stress-1 can be enabled by setting
*normalized_stress=True*, however it is only compatible with the non-
metric MDS problem and will be ignored in the metric case.

   [image]

* "Modern Multidimensional Scaling - Theory and Applications" Borg,
  I.; Groenen P. Springer Series in Statistics (1997)

* "Nonmetric multidimensional scaling: a numerical method" Kruskal, J.
  Psychometrika, 29 (1964)

* "Multidimensional scaling by optimizing goodness of fit to a
  nonmetric hypothesis" Kruskal, J. Psychometrika, 29, (1964)

Psychometrika, 29 (1964)

2.2.9. t-distributed Stochastic Neighbor Embedding (t-SNE)

t-SNE ("TSNE") converts affinities of data points to probabilities.
The affinities in the original space are represented by Gaussian joint
probabilities and the affinities in the embedded space are represented
by Student's t-distributions. This allows t-SNE to be particularly
sensitive to local structure and has a few other advantages over
existing techniques:

* Revealing the structure at many scales on a single map

* Revealing data that lie in multiple, different, manifolds or
  clusters

* Reducing the tendency to crowd points together at the center

While Isomap, LLE and variants are best suited to unfold a single
continuous low dimensional manifold, t-SNE will focus on the local
structure of the data and will tend to extract clustered local groups
of samples as highlighted on the S-curve example. This ability to
group samples based on the local structure might be beneficial to
visually disentangle a dataset that comprises several manifolds at
once as is the case in the digits dataset.

The Kullback-Leibler (KL) divergence of the joint probabilities in the
original space and the embedded space will be minimized by gradient
descent. Note that the KL divergence is not convex, i.e. multiple

restarts with different initializations will end up in local minima of
the KL divergence. Hence, it is sometimes useful to try different
seeds and select the embedding with the lowest KL divergence.

The disadvantages to using t-SNE are roughly:

* t-SNE is computationally expensive, and can take several hours on
  million-sample datasets where PCA will finish in seconds or minutes

* The Barnes-Hut t-SNE method is limited to two or three dimensional
  embeddings.

* The algorithm is stochastic and multiple restarts with different
  seeds can yield different embeddings. However, it is perfectly
  legitimate to pick the embedding with the least error.

* Global structure is not explicitly preserved. This problem is
  mitigated by initializing points with PCA (using *init='pca'*).

   [image]

2.2.9.1. Optimizing t-SNE

The main purpose of t-SNE is visualization of high-dimensional data.
Hence, it works best when the data will be embedded on two or three
dimensions.

Optimizing the KL divergence can be a little bit tricky sometimes.
There are five parameters that control the optimization of t-SNE and
therefore possibly the quality of the resulting embedding:

* perplexity

* early exaggeration factor

* learning rate

* maximum number of iterations

* angle (not used in the exact method)

The perplexity is defined as k=2^{(S)} where S is the Shannon entropy

of the conditional probability distribution. The perplexity of a
k-sided die is k, so that k is effectively the number of nearest
neighbors t-SNE considers when generating the conditional
probabilities. Larger perplexities lead to more nearest neighbors and
less sensitive to small structure. Conversely a lower perplexity
considers a smaller number of neighbors, and thus ignores more global
information in favour of the local neighborhood. As dataset sizes get
larger more points will be required to get a reasonable sample of the
local neighborhood, and hence larger perplexities may be required.
Similarly noisier datasets will require larger perplexity values to
encompass enough local neighbors to see beyond the background noise.

The maximum number of iterations is usually high enough and does not
need any tuning. The optimization consists of two phases: the early
exaggeration phase and the final optimization. During early
exaggeration the joint probabilities in the original space will be
artificially increased by multiplication with a given factor. Larger
factors result in larger gaps between natural clusters in the data. If
the factor is too high, the KL divergence could increase during this
phase. Usually it does not have to be tuned. A critical parameter is

the learning rate. If it is too low gradient descent will get stuck in
a bad local minimum. If it is too high the KL divergence will increase
during optimization. A heuristic suggested in Belkina et al. (2019) is
to set the learning rate to the sample size divided by the early
exaggeration factor. We implement this heuristic as
*learning_rate='auto'* argument. More tips can be found in Laurens van
der Maaten's FAQ (see references). The last parameter, angle, is a
tradeoff between performance and accuracy. Larger angles imply that we
can approximate larger regions by a single point, leading to better
speed but less accurate results.

"How to Use t-SNE Effectively" provides a good discussion of the
effects of the various parameters, as well as interactive plots to
explore the effects of different parameters.

2.2.9.2. Barnes-Hut t-SNE

The Barnes-Hut t-SNE that has been implemented here is usually much
slower than other manifold learning algorithms. The optimization is
quite difficult and the computation of the gradient is O[d N log(N)],
where d is the number of output dimensions and N is the number of
samples. The Barnes-Hut method improves on the exact method where
t-SNE complexity is O[d N^2], but has several other notable
differences:

* The Barnes-Hut implementation only works when the target

  dimensionality is 3 or less. The 2D case is typical when building
  visualizations.

* Barnes-Hut only works with dense input data. Sparse data matrices
  can only be embedded with the exact method or can be approximated by
  a dense low rank projection for instance using "TruncatedSVD"

* Barnes-Hut is an approximation of the exact method. The
  approximation is parameterized with the angle parameter, therefore
  the angle parameter is unused when method="exact"

* Barnes-Hut is significantly more scalable. Barnes-Hut can be used to
  embed hundred of thousands of data points while the exact method can
  handle thousands of samples before becoming computationally
  intractable

For visualization purpose (which is the main use case of t-SNE), using
the Barnes-Hut method is strongly recommended. The exact t-SNE method
is useful for checking the theoretically properties of the embedding
possibly in higher dimensional space but limit to small datasets due
to computational constraints.

Also note that the digits labels roughly match the natural grouping
found by t-SNE while the linear 2D projection of the PCA model yields
a representation where label regions largely overlap. This is a strong
clue that this data can be well separated by non linear methods that

focus on the local structure (e.g. an SVM with a Gaussian RBF kernel).
However, failing to visualize well separated homogeneously labeled
groups with t-SNE in 2D does not necessarily imply that the data
cannot be correctly classified by a supervised model. It might be the
case that 2 dimensions are not high enough to accurately represent the
internal structure of the data.

* "Visualizing High-Dimensional Data Using t-SNE" van der Maaten,
  L.J.P.; Hinton, G. Journal of Machine Learning Research (2008)

* "t-Distributed Stochastic Neighbor Embedding" van der Maaten, L.J.P.

* "Accelerating t-SNE using Tree-Based Algorithms" van der Maaten,
  L.J.P.; Journal of Machine Learning Research 15(Oct):3221-3245,
  2014.

* "Automated optimized parameters for T-distributed stochastic
  neighbor embedding improve visualization and analysis of large
  datasets" Belkina, A.C., Ciccolella, C.O., Anno, R., Halpert, R.,
  Spidlen, J., Snyder-Cappione, J.E., Nature Communications 10, 5415
  (2019).

2.2.10. Tips on practical use

* Make sure the same scale is used over all features. Because manifold
  learning methods are based on a nearest-neighbor search, the
  algorithm may perform poorly otherwise.  See StandardScaler for
  convenient ways of scaling heterogeneous data.

learning methods are based on a nearest-neighbor search, the

* The reconstruction error computed by each routine can be used to
  choose the optimal output dimension.  For a d-dimensional manifold
  embedded in a D-dimensional parameter space, the reconstruction
  error will decrease as "n_components" is increased until
  "n_components == d".

* Note that noisy data can "short-circuit" the manifold, in essence
  acting as a bridge between parts of the manifold that would
  otherwise be well-separated.  Manifold learning on noisy and/or
  incomplete data is an active area of research.

* Certain input configurations can lead to singular weight matrices,
  for example when more than two points in the dataset are identical,
  or when the data is split into disjointed groups.  In this case,
  "solver='arpack'" will fail to find the null space.  The easiest way
  to address this is to use "solver='dense'" which will work on a
  singular matrix, though it may be very slow depending on the number
  of input points.  Alternatively, one can attempt to understand the
  source of the singularity: if it is due to disjoint sets, increasing
  "n_neighbors" may help.  If it is due to identical points in the
  dataset, removing these points may help.

See also:

  Totally Random Trees Embedding can also be useful to derive non-
  linear representations of feature space, also it does not perform

dataset, removing these points may help.

See also:

  dimensionality reduction.

2.7. Novelty and Outlier Detection
**********************************

Many applications require being able to decide whether a new
observation belongs to the same distribution as existing observations
(it is an *inlier*), or should be considered as different (it is an
*outlier*). Often, this ability is used to clean real data sets. Two
important distinctions must be made:

outlier detection:
   The training data contains outliers which are defined as
   observations that are far from the others. Outlier detection
   estimators thus try to fit the regions where the training data is
   the most concentrated, ignoring the deviant observations.

novelty detection:
   The training data is not polluted by outliers and we are interested
   in detecting whether a **new** observation is an outlier. In this
   context an outlier is also called a novelty.

Outlier detection and novelty detection are both used for anomaly
detection, where one is interested in detecting abnormal or unusual
observations. Outlier detection is then also known as unsupervised
anomaly detection and novelty detection as semi-supervised anomaly
detection. In the context of outlier detection, the outliers/anomalies
cannot form a dense cluster as available estimators assume that the
outliers/anomalies are located in low density regions. On the
contrary, in the context of novelty detection, novelties/anomalies can
form a dense cluster as long as they are in a low density region of
the training data, considered as normal in this context.

The scikit-learn project provides a set of machine learning tools that
can be used both for novelty or outlier detection. This strategy is
implemented with objects learning in an unsupervised way from the
data:

   estimator.fit(X_train)

new observations can then be sorted as inliers or outliers with a
"predict" method:

   estimator.predict(X_test)

Inliers are labeled 1, while outliers are labeled -1. The predict
method makes use of a threshold on the raw scoring function computed
by the estimator. This scoring function is accessible through the
"score_samples" method, while the threshold can be controlled by the
"contamination" parameter.

The "decision_function" method is also defined from the scoring
function, in such a way that negative values are outliers and non-
negative ones are inliers:

   estimator.decision_function(X_test)

Note that "neighbors.LocalOutlierFactor" does not support "predict",
"decision_function" and "score_samples" methods by default but only a
"fit_predict" method, as this estimator was originally meant to be
applied for outlier detection. The scores of abnormality of the
training samples are accessible through the "negative_outlier_factor_"
attribute.

If you really want to use "neighbors.LocalOutlierFactor" for novelty

detection, i.e. predict labels or compute the score of abnormality of
new unseen data, you can instantiate the estimator with the "novelty"
parameter set to "True" before fitting the estimator. In this case,
"fit_predict" is not available.

Warning:

  **Novelty detection with Local Outlier Factor**When "novelty" is set
  to "True" be aware that you must only use "predict",
  "decision_function" and "score_samples" on new unseen data and not
  on the training samples as this would lead to wrong results. I.e.,
  the result of "predict" will not be the same as "fit_predict". The
  scores of abnormality of the training samples are always accessible
  through the "negative_outlier_factor_" attribute.

The behavior of "neighbors.LocalOutlierFactor" is summarized in the
following table.

+------------------------------+----------------------------------+-----------------------+
| Method                       | Outlier detection                | Novelty detection     |
|==============================|==================================|=======================|
| "fit_predict"                | OK                               | Not available         |
| "predict"                    | Not available                    | Use only on new data  |

| "decision_function"          | Not available                    | Use only on new data  |
| "score_samples"              | Use "negative_outlier_factor_"   | Use only on new data  |
| "negative_outlier_factor_"   | OK                               | OK                    |

2.7.1. Overview of outlier detection methods

A comparison of the outlier detection algorithms in scikit-learn.
Local Outlier Factor (LOF) does not show a decision boundary in black
as it has no predict method to be applied on new data when it is used
for outlier detection.

   [image]

"ensemble.IsolationForest" and "neighbors.LocalOutlierFactor" perform
reasonably well on the data sets considered here. The
"svm.OneClassSVM" is known to be sensitive to outliers and thus does
not perform very well for outlier detection. That being said, outlier
detection in high-dimension, or without any assumptions on the

distribution of the inlying data is very challenging.
"svm.OneClassSVM" may still be used with outlier detection but
requires fine-tuning of its hyperparameter *nu* to handle outliers and
prevent overfitting. "linear_model.SGDOneClassSVM" provides an
implementation of a linear One-Class SVM with a linear complexity in
the number of samples. This implementation is here used with a kernel
approximation technique to obtain results similar to "svm.OneClassSVM"
which uses a Gaussian kernel by default. Finally,
"covariance.EllipticEnvelope" assumes the data is Gaussian and learns
an ellipse. For more details on the different estimators refer to the
example
*sphx_glr_auto_examples_miscellaneous_plot_anomaly_comparison.py* and
the sections hereunder.

Examples:
^^^^^^^^^

* See
  *sphx_glr_auto_examples_miscellaneous_plot_anomaly_comparison.py*
  for a comparison of the "svm.OneClassSVM", the
  "ensemble.IsolationForest", the "neighbors.LocalOutlierFactor" and
  "covariance.EllipticEnvelope".

* See *sphx_glr_auto_examples_miscellaneous_plot_outlier_detection_be
  nch.py* for an example showing how to evaluate outlier detection
  estimators, the "neighbors.LocalOutlierFactor" and the
  "ensemble.IsolationForest", using ROC curves from
  "metrics.RocCurveDisplay".

2.7.2. Novelty Detection

estimators, the "neighbors.LocalOutlierFactor" and the

Consider a data set of n observations from the same distribution
described by p features.  Consider now that we add one more
observation to that data set. Is the new observation so different from
the others that we can doubt it is regular? (i.e. does it come from
the same distribution?) Or on the contrary, is it so similar to the
other that we cannot distinguish it from the original observations?
This is the question addressed by the novelty detection tools and
methods.

In general, it is about to learn a rough, close frontier delimiting
the contour of the initial observations distribution, plotted in
embedding p-dimensional space. Then, if further observations lay
within the frontier-delimited subspace, they are considered as coming
from the same population than the initial observations. Otherwise, if
they lay outside the frontier, we can say that they are abnormal with
a given confidence in our assessment.

The One-Class SVM has been introduced by Schölkopf et al. for that
purpose and implemented in the Support Vector Machines module in the
"svm.OneClassSVM" object. It requires the choice of a kernel and a
scalar parameter to define a frontier.  The RBF kernel is usually
chosen although there exists no exact formula or algorithm to set its
bandwidth parameter. This is the default in the scikit-learn

implementation. The *nu* parameter, also known as the margin of the
One-Class SVM, corresponds to the probability of finding a new, but
regular, observation outside the frontier.

* Estimating the support of a high-dimensional distribution Schölkopf,
  Bernhard, et al. Neural computation 13.7 (2001): 1443-1471.

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_svm_plot_oneclass.py* for visualizing
  the frontier learned around some data by a "svm.OneClassSVM" object.

* *sphx_glr_auto_examples_applications_plot_species_distribution_mode
  ling.py*

   [image]

2.7.2.1. Scaling up the One-Class SVM

An online linear version of the One-Class SVM is implemented in
"linear_model.SGDOneClassSVM". This implementation scales linearly
with the number of samples and can be used with a kernel approximation
to approximate the solution of a kernelized "svm.OneClassSVM" whose
complexity is at best quadratic in the number of samples. See section
Online One-Class SVM for more details.

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_linear_model_plot_sgdocsvm_vs_ocsvm.py*
  for an illustration of the approximation of a kernelized One-Class
  SVM with the *linear_model.SGDOneClassSVM* combined with kernel
  approximation.

2.7.3. Outlier Detection

for an illustration of the approximation of a kernelized One-Class

Outlier detection is similar to novelty detection in the sense that
the goal is to separate a core of regular observations from some
polluting ones, called *outliers*. Yet, in the case of outlier
detection, we don't have a clean data set representing the population
of regular observations that can be used to train any tool.

2.7.3.1. Fitting an elliptic envelope

One common way of performing outlier detection is to assume that the
regular data come from a known distribution (e.g. data are Gaussian
distributed). From this assumption, we generally try to define the
"shape" of the data, and can define outlying observations as
observations which stand far enough from the fit shape.

The scikit-learn provides an object "covariance.EllipticEnvelope" that
fits a robust covariance estimate to the data, and thus fits an
ellipse to the central data points, ignoring points outside the
central mode.

For instance, assuming that the inlier data are Gaussian distributed,
it will estimate the inlier location and covariance in a robust way
(i.e. without being influenced by outliers). The Mahalanobis distances
obtained from this estimate is used to derive a measure of
outlyingness. This strategy is illustrated below.

   [image]

Examples:
^^^^^^^^^

* See

   [image]

Examples:
^^^^^^^^^

* See
  *sphx_glr_auto_examples_covariance_plot_mahalanobis_distances.py*
  for an illustration of the difference between using a standard
  ("covariance.EmpiricalCovariance") or a robust estimate
  ("covariance.MinCovDet") of location and covariance to assess the
  degree of outlyingness of an observation.

* Rousseeuw, P.J., Van Driessen, K. "A fast algorithm for the minimum
  covariance determinant estimator" Technometrics 41(3), 212 (1999)

2.7.3.2. Isolation Forest

One efficient way of performing outlier detection in high-dimensional
datasets is to use random forests. The "ensemble.IsolationForest"
'isolates' observations by randomly selecting a feature and then
randomly selecting a split value between the maximum and minimum
values of the selected feature.

Since recursive partitioning can be represented by a tree structure,
the number of splittings required to isolate a sample is equivalent to
the path length from the root node to the terminating node.

This path length, averaged over a forest of such random trees, is a
measure of normality and our decision function.

Random partitioning produces noticeably shorter paths for anomalies.
Hence, when a forest of random trees collectively produce shorter path
lengths for particular samples, they are highly likely to be
anomalies.

The implementation of "ensemble.IsolationForest" is based on an
ensemble of "tree.ExtraTreeRegressor". Following Isolation Forest
original paper, the maximum depth of each tree is set to \lceil
\log_2(n) \rceil where n is the number of samples used to build the
tree (see (Liu et al., 2008) for more details).

This algorithm is illustrated below.

   [image]

The "ensemble.IsolationForest" supports "warm_start=True" which allows
you to add more trees to an already fitted model:

   >>> from sklearn.ensemble import IsolationForest
   >>> X = np.array([[-1, -1], [-2, -1], [-3, -2], [0, 0], [-20, 50], [3, 5]])
   >>> clf = IsolationForest(n_estimators=10, warm_start=True)
   >>> clf.fit(X)  # fit 10 trees  
   >>> clf.set_params(n_estimators=20)  # add 10 more trees  
   >>> clf.fit(X)  # fit the added trees  

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_ensemble_plot_isolation_forest.py* for
  an illustration of the use of IsolationForest.

* See
  for a comparison of "ensemble.IsolationForest" with
  "neighbors.LocalOutlierFactor", "svm.OneClassSVM" (tuned to perform
  like an outlier detection method), "linear_model.SGDOneClassSVM",
  and a covariance-based outlier detection with

like an outlier detection method), "linear_model.SGDOneClassSVM",

* Liu, Fei Tony, Ting, Kai Ming and Zhou, Zhi-Hua. "Isolation forest."
  Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on.

2.7.3.3. Local Outlier Factor

Another efficient way to perform outlier detection on moderately high
dimensional datasets is to use the Local Outlier Factor (LOF)
algorithm.

The "neighbors.LocalOutlierFactor" (LOF) algorithm computes a score
(called local outlier factor) reflecting the degree of abnormality of
the observations. It measures the local density deviation of a given
data point with respect to its neighbors. The idea is to detect the
samples that have a substantially lower density than their neighbors.

In practice the local density is obtained from the k-nearest
neighbors. The LOF score of an observation is equal to the ratio of
the average local density of its k-nearest neighbors, and its own
local density: a normal instance is expected to have a local density
similar to that of its neighbors, while abnormal data are expected to
have much smaller local density.

The number k of neighbors considered, (alias parameter n_neighbors) is
typically chosen 1) greater than the minimum number of objects a
cluster has to contain, so that other objects can be local outliers
relative to this cluster, and 2) smaller than the maximum number of

close by objects that can potentially be local outliers. In practice,
such information is generally not available, and taking n_neighbors=20
appears to work well in general. When the proportion of outliers is
high (i.e. greater than 10 %, as in the example below), n_neighbors
should be greater (n_neighbors=35 in the example below).

The strength of the LOF algorithm is that it takes both local and
global properties of datasets into consideration: it can perform well
even in datasets where abnormal samples have different underlying
densities. The question is not, how isolated the sample is, but how
isolated it is with respect to the surrounding neighborhood.

When applying LOF for outlier detection, there are no "predict",
"decision_function" and "score_samples" methods but only a
"fit_predict" method. The scores of abnormality of the training
samples are accessible through the "negative_outlier_factor_"
attribute. Note that "predict", "decision_function" and
"score_samples" can be used on new unseen data when LOF is applied for
novelty detection, i.e. when the "novelty" parameter is set to "True",
but the result of "predict" may differ from that of "fit_predict". See
Novelty detection with Local Outlier Factor.

This strategy is illustrated below.

   [image]

Examples:
^^^^^^^^^

   [image]

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_neighbors_plot_lof_outlier_detection.py*
  for an illustration of the use of "neighbors.LocalOutlierFactor".

* See
  for a comparison with other anomaly detection methods.

* Breunig, Kriegel, Ng, and Sander (2000) LOF: identifying density-
  based local outliers. Proc. ACM SIGMOD

2.7.4. Novelty detection with Local Outlier Factor

To use "neighbors.LocalOutlierFactor" for novelty detection, i.e.
predict labels or compute the score of abnormality of new unseen data,
you need to instantiate the estimator with the "novelty" parameter set
to "True" before fitting the estimator:

   lof = LocalOutlierFactor(novelty=True)
   lof.fit(X_train)

Note that "fit_predict" is not available in this case to avoid
inconsistencies.

Warning:

  **Novelty detection with Local Outlier Factor`**When "novelty" is
  set to "True" be aware that you must only use "predict",

the result of "predict" will not be the same as "fit_predict". The

Novelty detection with Local Outlier Factor is illustrated below.

      [image]

1.13. Feature selection

The classes in the "sklearn.feature_selection" module can be used for
feature selection/dimensionality reduction on sample sets, either to
improve estimators' accuracy scores or to boost their performance on
very high-dimensional datasets.

1.13.1. Removing features with low variance

"VarianceThreshold" is a simple baseline approach to feature
selection. It removes all features whose variance doesn't meet some
threshold. By default, it removes all zero-variance features, i.e.
features that have the same value in all samples.

As an example, suppose that we have a dataset with boolean features,
and we want to remove all features that are either one or zero (on or
off) in more than 80% of the samples. Boolean features are Bernoulli
random variables, and the variance of such variables is given by

   \mathrm{Var}[X] = p(1 - p)

so we can select using the threshold ".8 * (1 - .8)":

   >>> from sklearn.feature_selection import VarianceThreshold
   >>> X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
   >>> sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
   >>> sel.fit_transform(X)
          [1, 0],
          [0, 0],
          [1, 1],
          [1, 0],
          [1, 1]])

As expected, "VarianceThreshold" has removed the first column, which
has a probability p = 5/6 > .8 of containing a zero.

1.13.2. Univariate feature selection

Univariate feature selection works by selecting the best features
based on univariate statistical tests. It can be seen as a
preprocessing step to an estimator. Scikit-learn exposes feature
selection routines as objects that implement the "transform" method:

   * "SelectKBest" removes all but the k highest scoring features

   * "SelectPercentile" removes all but a user-specified highest
     scoring percentage of features

   * using common univariate statistical tests for each feature: false
     positive rate "SelectFpr", false discovery rate "SelectFdr", or
     family wise error "SelectFwe".

   * "GenericUnivariateSelect" allows to perform univariate feature
     selection with a configurable strategy. This allows to select the
     best univariate selection strategy with hyper-parameter search
     estimator.

For instance, we can use a F-test to retrieve the two best features
for a dataset as follows:

>>> from sklearn.feature_selection import SelectKBest
>>> from sklearn.feature_selection import f_classif
>>> X.shape
(150, 4)
>>> X_new = SelectKBest(f_classif, k=2).fit_transform(X, y)
>>> X_new.shape
(150, 2)

These objects take as input a scoring function that returns univariate

(150, 4)
(150, 2)

scores and p-values (or only scores for "SelectKBest" and
"SelectPercentile"):

   * For regression: "r_regression()", "f_regression()",
     "mutual_info_regression()"

   * For classification: "chi2()", "f_classif()",
     "mutual_info_classif()"

The methods based on F-test estimate the degree of linear dependency
between two random variables. On the other hand, mutual information
methods can capture any kind of statistical dependency, but being
nonparametric, they require more samples for accurate estimation. Note
that the \chi^2-test should only be applied to non-negative features,
such as frequencies.

Feature selection with sparse data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you use sparse data (i.e. data represented as sparse matrices),
"chi2()", "mutual_info_regression()", "mutual_info_classif()" will
deal with the data without making it dense.

Warning:

  Beware not to use a regression scoring function with a
  classification problem, you will get useless results.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_feature_selection_plot_feature_selection.py*

* *sphx_glr_auto_examples_feature_selection_plot_f_test_vs_mi.py*

1.13.3. Recursive feature elimination

Given an external estimator that assigns weights to features (e.g.,
the coefficients of a linear model), the goal of recursive feature

elimination ("RFE") is to select features by recursively considering
smaller and smaller sets of features. First, the estimator is trained
on the initial set of features and the importance of each feature is
obtained either through any specific attribute (such as "coef_",
"feature_importances_") or callable. Then, the least important
features are pruned from current set of features. That procedure is
recursively repeated on the pruned set until the desired number of
features to select is eventually reached.

"RFECV" performs RFE in a cross-validation loop to find the optimal
number of features.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_feature_selection_plot_rfe_digits.py*: A
  recursive feature elimination example showing the relevance of
  pixels in a digit classification task.

  ation.py*: A recursive feature elimination example with automatic
  tuning of the number of features selected with cross-validation.

1.13.4. Feature selection using SelectFromModel

"SelectFromModel" is a meta-transformer that can be used alongside any
estimator that assigns importance to each feature through a specific
attribute (such as "coef_", "feature_importances_") or via an

*importance_getter* callable after fitting. The features are
considered unimportant and removed if the corresponding importance of
the feature values are below the provided "threshold" parameter. Apart
from specifying the threshold numerically, there are built-in
heuristics for finding a threshold using a string argument. Available
heuristics are "mean", "median" and float multiples of these like
"0.1*mean". In combination with the *threshold* criteria, one can use
the *max_features* parameter to set a limit on the number of features
to select.

For examples on how it is to be used refer to the sections below.

Examples
^^^^^^^^

* *sphx_glr_auto_examples_feature_selection_plot_select_from_model_di
  abetes.py*

1.13.4.1. L1-based feature selection

Linear models penalized with the L1 norm have sparse solutions: many
of their estimated coefficients are zero. When the goal is to reduce
the dimensionality of the data to use with another classifier, they
can be used along with "SelectFromModel" to select the non-zero
coefficients. In particular, sparse estimators useful for this purpose
are the "Lasso" for regression, and of "LogisticRegression" and
"LinearSVC" for classification:

   >>> from sklearn.svm import LinearSVC

   >>> from sklearn.feature_selection import SelectFromModel
   >>> X.shape
   (150, 4)
   >>> lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y)
   >>> model = SelectFromModel(lsvc, prefit=True)
   >>> X_new = model.transform(X)
   >>> X_new.shape
   (150, 3)

With SVMs and logistic-regression, the parameter C controls the
sparsity: the smaller C the fewer features selected. With Lasso, the
higher the alpha parameter, the fewer features selected.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_lasso_dense_vs_sparse_dat
  a.py*.

**L1-recovery and compressive sensing**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For a good choice of alpha, the Lasso can fully recover the exact set
of non-zero variables using only few observations, provided certain
specific conditions are met. In particular, the number of samples
should be "sufficiently large", or L1 models will perform at random,
where "sufficiently large" depends on the number of non-zero
coefficients, the logarithm of the number of features, the amount of
noise, the smallest absolute value of non-zero coefficients, and the
structure of the design matrix X. In addition, the design matrix must
display certain specific properties, such as not being too correlated.

There is no general rule to select an alpha parameter for recovery of
non-zero coefficients. It can by set by cross-validation ("LassoCV" or
"LassoLarsCV"), though this may lead to under-penalized models:
including a small number of non-relevant variables is not detrimental
to prediction score. BIC ("LassoLarsIC") tends, on the opposite, to
set high values of alpha.

**Reference** Richard G. Baraniuk "Compressive Sensing", IEEE Signal
Processing Magazine [120] July 2007
http://users.isr.ist.utl.pt/~aguiar/CS_notes.pdf

1.13.4.2. Tree-based feature selection

Tree-based estimators (see the "sklearn.tree" module and forest of
trees in the "sklearn.ensemble" module) can be used to compute
impurity-based feature importances, which in turn can be used to
discard irrelevant features (when coupled with the "SelectFromModel"
meta-transformer):

   (150, 4)
   >>> clf = ExtraTreesClassifier(n_estimators=50)
   >>> clf = clf.fit(X, y)
   >>> clf.feature_importances_  
   array([ 0.04...,  0.05...,  0.4...,  0.4...])
   >>> model = SelectFromModel(clf, prefit=True)

>>> clf = clf.fit(X, y)
   >>> X_new.shape               
   (150, 2)

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_ensemble_plot_forest_importances.py*:
  example on synthetic data showing the recovery of the actually
  meaningful features.

* *sphx_glr_auto_examples_ensemble_plot_forest_importances_faces.py*:
  example on face recognition data.

1.13.5. Sequential Feature Selection

Sequential Feature Selection [sfs] (SFS) is available in the
"SequentialFeatureSelector" transformer. SFS can be either forward or
backward:

Forward-SFS is a greedy procedure that iteratively finds the best new
feature to add to the set of selected features. Concretely, we
initially start with zero features and find the one feature that
maximizes a cross-validated score when an estimator is trained on this
single feature. Once that first feature is selected, we repeat the
procedure by adding a new feature to the set of selected features. The
procedure stops when the desired number of selected features is
reached, as determined by the *n_features_to_select* parameter.

Backward-SFS follows the same idea but works in the opposite
direction: instead of starting with no features and greedily adding
features, we start with *all* the features and greedily *remove*
features from the set. The *direction* parameter controls whether

forward or backward SFS is used.

In general, forward and backward selection do not yield equivalent
results. Also, one may be much faster than the other depending on the
requested number of selected features: if we have 10 features and ask
for 7 selected features, forward selection would need to perform 7
iterations while backward selection would only need to perform 3.

SFS differs from "RFE" and "SelectFromModel" in that it does not
require the underlying model to expose a *coef_* or
*feature_importances_* attribute. It may however be slower considering
that more models need to be evaluated, compared to the other
approaches. For example in backward selection, the iteration going
from *m* features to *m - 1* features using k-fold cross-validation
requires fitting *m * k* models, while "RFE" would require only a
single fit, and "SelectFromModel" always just does a single fit and
requires no iterations.

Examples
^^^^^^^^

[sfs] Ferri et al, Comparative study of techniques for large-scale
      feature selection.

1.13.6. Feature selection as part of a pipeline

Feature selection is usually used as a pre-processing step before

feature selection.

doing the actual learning. The recommended way to do this in scikit-
learn is to use a "Pipeline":

   clf = Pipeline([
     ('feature_selection', SelectFromModel(LinearSVC(penalty="l1"))),
     ('classification', RandomForestClassifier())
   ])
   clf.fit(X, y)

In this snippet we make use of a "LinearSVC" coupled with
"SelectFromModel" to evaluate feature importances and select the most
relevant features. Then, a "RandomForestClassifier" is trained on the
transformed output, i.e. using only relevant features. You can perform
similar operations with the other feature selection methods and also
classifiers that provide a way to evaluate feature importances of
course. See the "Pipeline" examples for more details.

6.2. Feature extraction

The "sklearn.feature_extraction" module can be used to extract
features in a format supported by machine learning algorithms from
datasets consisting of formats such as text and image.

Note:

  Feature extraction is very different from Feature selection: the
  former consists in transforming arbitrary data, such as text or
  images, into numerical features usable for machine learning. The
  latter is a machine learning technique applied on these features.

6.2.1. Loading features from dicts

The class "DictVectorizer" can be used to convert feature arrays
represented as lists of standard Python "dict" objects to the
NumPy/SciPy representation used by scikit-learn estimators.

While not particularly fast to process, Python's "dict" has the
advantages of being convenient to use, being sparse (absent features
need not be stored) and storing feature names in addition to values.

"DictVectorizer" implements what is called one-of-K or "one-hot"
coding for categorical (aka nominal, discrete) features. Categorical
features are "attribute-value" pairs where the value is restricted to
a list of discrete possibilities without ordering (e.g. topic
identifiers, types of objects, tags, names...).

In the following, "city" is a categorical attribute while
"temperature" is a traditional numerical feature:

   >>> measurements = [
   ...     {'city': 'Dubai', 'temperature': 33.},
   ...     {'city': 'London', 'temperature': 12.},

   ...     {'city': 'San Francisco', 'temperature': 18.},
   ... ]

   >>> from sklearn.feature_extraction import DictVectorizer
   >>> vec = DictVectorizer()

   >>> vec.fit_transform(measurements).toarray()
   array([[ 1.,  0.,  0., 33.],
          [ 0.,  1.,  0., 12.],
          [ 0.,  0.,  1., 18.]])

   >>> vec.get_feature_names_out()
   array(['city=Dubai', 'city=London', 'city=San Francisco', 'temperature'], ...)

"DictVectorizer" accepts multiple string values for one feature, like,
e.g., multiple categories for a movie.

Assume a database classifies each movie using some categories (not
mandatories) and its year of release.

>>> movie_entry = [{'category': ['thriller', 'drama'], 'year': 2003},
...                {'category': ['animation', 'family'], 'year': 2011},
...                {'year': 1974}]
>>> vec.fit_transform(movie_entry).toarray()
array([[0.000e+00, 1.000e+00, 0.000e+00, 1.000e+00, 2.003e+03],
       [1.000e+00, 0.000e+00, 1.000e+00, 0.000e+00, 2.011e+03],
       [0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 1.974e+03]])
>>> vec.get_feature_names_out()
array(['category=animation', 'category=drama', 'category=family',
       'category=thriller', 'year'], ...)
>>> vec.transform({'category': ['thriller'],
...                'unseen_feature': '3'}).toarray()
array([[0., 0., 0., 1., 0.]])

'category=thriller', 'year'], ...)

"DictVectorizer" is also a useful representation transformation for
training sequence classifiers in Natural Language Processing models
that typically work by extracting feature windows around a particular
word of interest.

For example, suppose that we have a first algorithm that extracts Part
of Speech (PoS) tags that we want to use as complementary tags for
training a sequence classifier (e.g. a chunker). The following dict
could be such a window of features extracted around the word 'sat' in
the sentence 'The cat sat on the mat.':

   >>> pos_window = [
   ...     {
   ...         'word-2': 'the',
   ...         'pos-2': 'DT',
   ...         'word-1': 'cat',
   ...         'pos-1': 'NN',
   ...         'word+1': 'on',
   ...         'pos+1': 'PP',
   ...     },
   ...     # in a real application one would extract many such dictionaries
   ... ]

This description can be vectorized into a sparse two-dimensional
matrix suitable for feeding into a classifier (maybe after being piped
into a "TfidfTransformer" for normalization):

   >>> pos_vectorized = vec.fit_transform(pos_window)
   >>> pos_vectorized
   <1x6 sparse matrix of type '<... 'numpy.float64'>'
       with 6 stored elements in Compressed Sparse ... format>
   >>> pos_vectorized.toarray()
   array([[1., 1., 1., 1., 1., 1.]])

<1x6 sparse matrix of type '<... 'numpy.float64'>'
   array(['pos+1=PP', 'pos-1=NN', 'pos-2=DT', 'word+1=on', 'word-1=cat',
          'word-2=the'], ...)

As you can imagine, if one extracts such a context around each
individual word of a corpus of documents the resulting matrix will be
very wide (many one-hot-features) with most of them being valued to
zero most of the time. So as to make the resulting data structure able
to fit in memory the "DictVectorizer" class uses a "scipy.sparse"
matrix by default instead of a "numpy.ndarray".

6.2.2. Feature hashing

The class "FeatureHasher" is a high-speed, low-memory vectorizer that
uses a technique known as feature hashing, or the "hashing trick".
Instead of building a hash table of the features encountered in
training, as the vectorizers do, instances of "FeatureHasher" apply a
hash function to the features to determine their column index in
sample matrices directly. The result is increased speed and reduced
memory usage, at the expense of inspectability; the hasher does not
remember what the input features looked like and has no
"inverse_transform" method.

Since the hash function might cause collisions between (unrelated)
features, a signed hash function is used and the sign of the hash
value determines the sign of the value stored in the output matrix for

a feature. This way, collisions are likely to cancel out rather than
accumulate error, and the expected mean of any output feature's value
is zero. This mechanism is enabled by default with
"alternate_sign=True" and is particularly useful for small hash table
sizes ("n_features < 10000"). For large hash table sizes, it can be
disabled, to allow the output to be passed to estimators like
"MultinomialNB" or "chi2" feature selectors that expect non-negative
inputs.

"FeatureHasher" accepts either mappings (like Python's "dict" and its
variants in the "collections" module), "(feature, value)" pairs, or
strings, depending on the constructor parameter "input_type". Mapping
are treated as lists of "(feature, value)" pairs, while single strings
have an implicit value of 1, so "['feat1', 'feat2', 'feat3']" is
interpreted as "[('feat1', 1), ('feat2', 1), ('feat3', 1)]". If a
single feature occurs multiple times in a sample, the associated
values will be summed (so "('feat', 2)" and "('feat', 3.5)" become
"('feat', 5.5)"). The output from "FeatureHasher" is always a
"scipy.sparse" matrix in the CSR format.

Feature hashing can be employed in document classification, but unlike
"CountVectorizer", "FeatureHasher" does not do word splitting or any
other preprocessing except Unicode-to-UTF-8 encoding; see Vectorizing

a large text corpus with the hashing trick, below, for a combined
tokenizer/hasher.

As an example, consider a word-level natural language processing task
that needs features extracted from "(token, part_of_speech)" pairs.
One could use a Python generator function to extract features:

   def token_features(token, part_of_speech):
       if token.isdigit():
           yield "numeric"
       else:
           yield "token={}".format(token.lower())
           yield "token,pos={},{}".format(token, part_of_speech)
       if token[0].isupper():
           yield "uppercase_initial"
       if token.isupper():
           yield "all_uppercase"
       yield "pos={}".format(part_of_speech)

Then, the "raw_X" to be fed to "FeatureHasher.transform" can be
constructed using:

   raw_X = (token_features(tok, pos_tagger(tok)) for tok in corpus)

and fed to a hasher with:

   hasher = FeatureHasher(input_type='string')
   X = hasher.transform(raw_X)

to get a "scipy.sparse" matrix "X".

Note the use of a generator comprehension, which introduces laziness
into the feature extraction: tokens are only processed on demand from
the hasher.

6.2.2.1. Implementation details

"FeatureHasher" uses the signed 32-bit variant of MurmurHash3. As a
result (and because of limitations in "scipy.sparse"), the maximum
number of features supported is currently 2^{31} - 1.

The original formulation of the hashing trick by Weinberger et al.
used two separate hash functions h and \xi to determine the column
index and sign of a feature, respectively. The present implementation
works under the assumption that the sign bit of MurmurHash3 is
independent of its other bits.

Since a simple modulo is used to transform the hash function to a
column index, it is advisable to use a power of two as the
"n_features" parameter; otherwise the features will not be mapped
evenly to the columns.

* Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola and
  Josh Attenberg (2009). Feature hashing for large scale multitask
  learning. Proc. ICML.

* MurmurHash3.

6.2.3. Text feature extraction

6.2.3.1. The Bag of Words representation

Text Analysis is a major application field for machine learning
algorithms. However the raw data, a sequence of symbols cannot be fed
directly to the algorithms themselves as most of them expect numerical
feature vectors with a fixed size rather than the raw text documents
with variable length.

In order to address this, scikit-learn provides utilities for the most
common ways to extract numerical features from text content, namely:

* **tokenizing** strings and giving an integer id for each possible
  token, for instance by using white-spaces and punctuation as token
  separators.

* **counting** the occurrences of tokens in each document.

* **normalizing** and weighting with diminishing importance tokens
  that occur in the majority of samples / documents.

In this scheme, features and samples are defined as follows:

* each **individual token occurrence frequency** (normalized or not)
  is treated as a **feature**.

* the vector of all the token frequencies for a given **document** is
  considered a multivariate **sample**.

A corpus of documents can thus be represented by a matrix with one row
per document and one column per token (e.g. word) occurring in the
corpus.

We call **vectorization** the general process of turning a collection
of text documents into numerical feature vectors. This specific
strategy (tokenization, counting and normalization) is called the
**Bag of Words** or "Bag of n-grams" representation. Documents are
described by word occurrences while completely ignoring the relative
position information of the words in the document.

6.2.3.2. Sparsity

As most documents will typically use a very small subset of the words
used in the corpus, the resulting matrix will have many feature values
that are zeros (typically more than 99% of them).

For instance a collection of 10,000 short text documents (such as
emails) will use a vocabulary with a size in the order of 100,000
unique words in total while each document will use 100 to 1000 unique
words individually.

In order to be able to store such a matrix in memory but also to speed
up algebraic operations matrix / vector, implementations will
typically use a sparse representation such as the implementations
available in the "scipy.sparse" package.

6.2.3.3. Common Vectorizer usage

"CountVectorizer" implements both tokenization and occurrence counting
in a single class:

This model has many parameters, however the default values are quite
reasonable (please see  the reference documentation for the details):

   >>> vectorizer = CountVectorizer()
   >>> vectorizer
   CountVectorizer()

Let's use it to tokenize and count the word occurrences of a
minimalistic corpus of text documents:

   >>> corpus = [
   ...     'This is the first document.',
   ...     'This is the second second document.',
   ...     'And the third one.',
   ...     'Is this the first document?',
   ... ]
   >>> X = vectorizer.fit_transform(corpus)
   >>> X
   <4x9 sparse matrix of type '<... 'numpy.int64'>'

...     'And the third one.',
   ... ]
   >>> X
       with 19 stored elements in Compressed Sparse ... format>

The default configuration tokenizes the string by extracting words of
at least 2 letters. The specific function that does this step can be
requested explicitly:

   >>> analyze = vectorizer.build_analyzer()
   >>> analyze("This is a text document to analyze.") == (
   ...     ['this', 'is', 'text', 'document', 'to', 'analyze'])
   True

Each term found by the analyzer during the fit is assigned a unique
integer index corresponding to a column in the resulting matrix. This
interpretation of the columns can be retrieved as follows:

   >>> vectorizer.get_feature_names_out()
   array(['and', 'document', 'first', 'is', 'one', 'second', 'the',
          'third', 'this'], ...)

   >>> X.toarray()
   array([[0, 1, 1, 1, 0, 0, 1, 0, 1],
          [0, 1, 0, 1, 0, 2, 1, 0, 1],
          [1, 0, 0, 0, 1, 0, 1, 1, 0],
          [0, 1, 1, 1, 0, 0, 1, 0, 1]]...)

The converse mapping from feature name to column index is stored in
the "vocabulary_" attribute of the vectorizer:

   >>> vectorizer.vocabulary_.get('document')
   1

Hence words that were not seen in the training corpus will be
completely ignored in future calls to the transform method:

   >>> vectorizer.transform(['Something completely new.']).toarray()
   array([[0, 0, 0, 0, 0, 0, 0, 0, 0]]...)

Note that in the previous corpus, the first and the last documents
have exactly the same words hence are encoded in equal vectors. In
particular we lose the information that the last document is an
interrogative form. To preserve some of the local ordering information
we can extract 2-grams of words in addition to the 1-grams (individual
words):

   >>> bigram_vectorizer = CountVectorizer(ngram_range=(1, 2),
   ...                                     token_pattern=r'\b\w+\b', min_df=1)
   >>> analyze = bigram_vectorizer.build_analyzer()
   >>> analyze('Bi-grams are cool!') == (
   ...     ['bi', 'grams', 'are', 'cool', 'bi grams', 'grams are', 'are cool'])
   True

The vocabulary extracted by this vectorizer is hence much bigger and
can now resolve ambiguities encoded in local positioning patterns:

   >>> X_2 = bigram_vectorizer.fit_transform(corpus).toarray()
   >>> X_2
   array([[0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0],
          [0, 0, 1, 0, 0, 1, 1, 0, 0, 2, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0],
          [1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0],
          [0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1]]...)

In particular the interrogative form "Is this" is only present in the
last document:

   >>> feature_index = bigram_vectorizer.vocabulary_.get('is this')

   >>> X_2[:, feature_index]
   array([0, 0, 0, 1]...)

6.2.3.3.1. Using stop words
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Stop words are words like "and", "the", "him", which are presumed to
be uninformative in representing the content of a text, and which may
be removed to avoid them being construed as signal for prediction.
Sometimes, however, similar words are useful for prediction, such as
in classifying writing style or personality.

There are several known issues in our provided 'english' stop word
list. It does not aim to be a general, 'one-size-fits-all' solution as
some tasks may require a more custom solution. See [NQY18] for more
details.

Please take care in choosing a stop word list. Popular stop word lists
may include words that are highly informative to some tasks, such as
*computer*.

You should also make sure that the stop word list has had the same
preprocessing and tokenization applied as the one used in the
vectorizer. The word *we've* is split into *we* and *ve* by
CountVectorizer's default tokenizer, so if *we've* is in "stop_words",
but *ve* is not, *ve* will be retained from *we've* in transformed
text.  Our vectorizers will try to identify and warn about some kinds
of inconsistencies.

[NQY18] J. Nothman, H. Qin and R. Yurchak (2018). "Stop Word Lists in
        Free Open-source Software Packages". In *Proc. Workshop for

        NLP Open Source Software*.

6.2.3.4. Tf–idf term weighting

In a large text corpus, some words will be very present (e.g. "the",
"a", "is" in English) hence carrying very little meaningful
information about the actual contents of the document. If we were to
feed the direct count data directly to a classifier those very
frequent terms would shadow the frequencies of rarer yet more
interesting terms.

In order to re-weight the count features into floating point values
suitable for usage by a classifier it is very common to use the tf–idf
transform.

Tf means **term-frequency** while tf–idf means term-frequency times
**inverse document-frequency**: \text{tf-idf(t,d)}=\text{tf(t,d)}
\times \text{idf(t)}.

Using the "TfidfTransformer"'s default settings,
"TfidfTransformer(norm='l2', use_idf=True, smooth_idf=True,
sublinear_tf=False)" the term frequency, the number of times a term
occurs in a given document, is multiplied with idf component, which is
computed as

\text{idf}(t) = \log{\frac{1 + n}{1+\text{df}(t)}} + 1,

where n is the total number of documents in the document set, and
\text{df}(t) is the number of documents in the document set that
contain term t. The resulting tf-idf vectors are then normalized by
the Euclidean norm:

v_{norm} = \frac{v}{||v||_2} = \frac{v}{\sqrt{v{_1}^2 + v{_2}^2 +

\dots + v{_n}^2}}.

This was originally a term weighting scheme developed for information
retrieval (as a ranking function for search engines results) that has
also found good use in document classification and clustering.

The following sections contain further explanations and examples that
illustrate how the tf-idfs are computed exactly and how the tf-idfs
computed in scikit-learn's "TfidfTransformer" and "TfidfVectorizer"
differ slightly from the standard textbook notation that defines the
idf as

\text{idf}(t) = \log{\frac{n}{1+\text{df}(t)}}.

In the "TfidfTransformer" and "TfidfVectorizer" with
"smooth_idf=False", the "1" count is added to the idf instead of the
idf's denominator:

\text{idf}(t) = \log{\frac{n}{\text{df}(t)}} + 1

This normalization is implemented by the "TfidfTransformer" class:

   >>> transformer = TfidfTransformer(smooth_idf=False)
   >>> transformer
   TfidfTransformer(smooth_idf=False)

Again please see the reference documentation for the details on all
the parameters.

Let's take an example with the following counts. The first term is
present 100% of the time hence not very interesting. The two other
features only in less than 50% of the time hence probably more
representative of the content of the documents:

   >>> counts = [[3, 0, 1],
   ...           [2, 0, 0],

   ...           [3, 0, 0],
   ...           [4, 0, 0],
   ...           [3, 2, 0],
   ...           [3, 0, 2]]
   ...
   >>> tfidf = transformer.fit_transform(counts)
   >>> tfidf
   <6x3 sparse matrix of type '<... 'numpy.float64'>'
       with 9 stored elements in Compressed Sparse ... format>

   >>> tfidf.toarray()
   array([[0.81940995, 0.        , 0.57320793],
          [1.        , 0.        , 0.        ],
          [0.47330339, 0.88089948, 0.        ],
          [0.58149261, 0.        , 0.81355169]])

Each row is normalized to have unit Euclidean norm:

\dots + v{_n}^2}}

For example, we can compute the tf-idf of the first term in the first
document in the *counts* array as follows:

n = 6

\text{df}(t)_{\text{term1}} = 6

\text{idf}(t)_{\text{term1}} = \log \frac{n}{\text{df}(t)} + 1 =
\log(1)+1 = 1

\text{tf-idf}_{\text{term1}} = \text{tf} \times \text{idf} = 3 \times
1 = 3

Now, if we repeat this computation for the remaining 2 terms in the
document, we get

\text{tf-idf}_{\text{term2}} = 0 \times (\log(6/1)+1) = 0

\text{tf-idf}_{\text{term3}} = 1 \times (\log(6/2)+1) \approx 2.0986

and the vector of raw tf-idfs:

\text{tf-idf}_{\text{raw}} = [3, 0, 2.0986].

Then, applying the Euclidean (L2) norm, we obtain the following tf-
idfs for document 1:

\frac{[3, 0, 2.0986]}{\sqrt{\big(3^2 + 0^2 + 2.0986^2\big)}} = [
0.819,  0,  0.573].

Furthermore, the default parameter "smooth_idf=True" adds "1" to the
numerator and  denominator as if an extra document was seen containing
every term in the collection exactly once, which prevents zero
divisions:

\text{idf}(t) = \log{\frac{1 + n}{1+\text{df}(t)}} + 1

Using this modification, the tf-idf of the third term in document 1
changes to 1.8473:

\text{tf-idf}_{\text{term3}} = 1 \times \log(7/3)+1 \approx 1.8473

And the L2-normalized tf-idf changes to

\frac{[3, 0, 1.8473]}{\sqrt{\big(3^2 + 0^2 + 1.8473^2\big)}} =
[0.8515, 0, 0.5243]:

   >>> transformer = TfidfTransformer()
   >>> transformer.fit_transform(counts).toarray()
   array([[0.85151335, 0.        , 0.52433293],
          [0.55422893, 0.83236428, 0.        ],
          [0.63035731, 0.        , 0.77630514]])

The weights of each feature computed by the "fit" method call are
stored in a model attribute:

   >>> transformer.idf_
   array([1. ..., 2.25..., 1.84...])

As tf–idf is very often used for text features, there is also another

class called "TfidfVectorizer" that combines all the options of
"CountVectorizer" and "TfidfTransformer" in a single model:

   >>> from sklearn.feature_extraction.text import TfidfVectorizer
   >>> vectorizer = TfidfVectorizer()
   >>> vectorizer.fit_transform(corpus)
   <4x9 sparse matrix of type '<... 'numpy.float64'>'

While the tf–idf normalization is often very useful, there might be
cases where the binary occurrence markers might offer better features.
This can be achieved by using the "binary" parameter of
"CountVectorizer". In particular, some estimators such as Bernoulli
Naive Bayes explicitly model discrete boolean random variables. Also,
very short texts are likely to have noisy tf–idf values while the
binary occurrence info is more stable.

As usual the best way to adjust the feature extraction parameters is
to use a cross-validated grid search, for instance by pipelining the
feature extractor with a classifier:

   * *sphx_glr_auto_examples_model_selection_plot_grid_search_text_fe
     ature_extraction.py*

6.2.3.5. Decoding text files

Text is made of characters, but files are made of bytes. These bytes
represent characters according to some *encoding*. To work with text
files in Python, their bytes must be *decoded* to a character set

called Unicode. Common encodings are ASCII, Latin-1 (Western Europe),
KOI8-R (Russian) and the universal encodings UTF-8 and UTF-16. Many
others exist.

Note:

  An encoding can also be called a 'character set', but this term is
  less accurate: several encodings can exist for a single character
  set.

The text feature extractors in scikit-learn know how to decode text
files, but only if you tell them what encoding the files are in. The
"CountVectorizer" takes an "encoding" parameter for this purpose. For
modern text files, the correct encoding is probably UTF-8, which is
therefore the default ("encoding="utf-8"").

If the text you are loading is not actually encoded with UTF-8,
however, you will get a "UnicodeDecodeError". The vectorizers can be
told to be silent about decoding errors by setting the "decode_error"
parameter to either ""ignore"" or ""replace"". See the documentation
for the Python function "bytes.decode" for more details (type
"help(bytes.decode)" at the Python prompt).

If you are having trouble decoding text, here are some things to try:

* Find out what the actual encoding of the text is. The file might
  come with a header or README that tells you the encoding, or there
  might be some standard encoding you can assume based on where the
  text comes from.

* You may be able to find out what kind of encoding it is in general

might be some standard encoding you can assume based on where the

  using the UNIX command "file". The Python "chardet" module comes
  with a script called "chardetect.py" that will guess the specific
  encoding, though you cannot rely on its guess being correct.

* You could try UTF-8 and disregard the errors. You can decode byte
  strings with "bytes.decode(errors='replace')" to replace all
  decoding errors with a meaningless character, or set
  "decode_error='replace'" in the vectorizer. This may damage the
  usefulness of your features.

* Real text may come from a variety of sources that may have used
  different encodings, or even be sloppily decoded in a different
  encoding than the one it was encoded with. This is common in text
  retrieved from the Web. The Python package ftfy can automatically
  sort out some classes of decoding errors, so you could try decoding
  the unknown text as "latin-1" and then using "ftfy" to fix errors.

* If the text is in a mish-mash of encodings that is simply too hard
  to sort out (which is the case for the 20 Newsgroups dataset), you
  can fall back on a simple single-byte encoding such as "latin-1".
  Some text may display incorrectly, but at least the same sequence of
  bytes will always represent the same feature.

For example, the following snippet uses "chardet" (not shipped with
scikit-learn, must be installed separately) to figure out the encoding

bytes will always represent the same feature.

of three texts. It then vectorizes the texts and prints the learned
vocabulary. The output is not shown here.

>>> import chardet    
>>> text1 = b"Sei mir gegr\xc3\xbc\xc3\x9ft mein Sauerkraut"
>>> text2 = b"holdselig sind deine Ger\xfcche"
>>> text3 = b"\xff\xfeA\x00u\x00f\x00 \x00F\x00l\x00\xfc\x00g\x00e\x00l\x00n\x00 \x00d\x00e\x00s\x00 \x00G\x00e\x00s\x00a\x00n\x00g\x00e\x00s\x00,\x00 \x00H\x00e\x00r\x00z\x00l\x00i\x00e\x00b\x00c\x00h\x00e\x00n\x00,\x00 \x00t\x00r\x00a\x00g\x00 \x00i\x00c\x00h\x00 \x00d\x00i\x00c\x00h\x00 \x00f\x00o\x00r\x00t\x00"
>>> decoded = [x.decode(chardet.detect(x)['encoding'])
...            for x in (text1, text2, text3)]        
>>> v = CountVectorizer().fit(decoded).vocabulary_    
>>> for term in v: print(v)                           

(Depending on the version of "chardet", it might get the first one
wrong.)

For an introduction to Unicode and character encodings in general, see
Joel Spolsky's Absolute Minimum Every Software Developer Must Know
About Unicode.

6.2.3.6. Applications and examples

The bag of words representation is quite simplistic but surprisingly
useful in practice.

In particular in a **supervised setting** it can be successfully
combined with fast and scalable linear models to train **document
classifiers**, for instance:

   * *sphx_glr_auto_examples_text_plot_document_classification_20news
     groups.py*

In an **unsupervised setting** it can be used to group similar
documents together by applying clustering algorithms such as K-means:

   * *sphx_glr_auto_examples_text_plot_document_clustering.py*

Finally it is possible to discover the main topics of a corpus by
relaxing the hard assignment constraint of clustering, for instance by
using Non-negative matrix factorization (NMF or NNMF):

   * *sphx_glr_auto_examples_applications_plot_topics_extraction_with
     _nmf_lda.py*

6.2.3.7. Limitations of the Bag of Words representation
-------------------------------------------------------

A collection of unigrams (what bag of words is) cannot capture phrases
and multi-word expressions, effectively disregarding any word order
dependence. Additionally, the bag of words model doesn't account for
potential misspellings or word derivations.

N-grams to the rescue! Instead of building a simple collection of
unigrams (n=1), one might prefer a collection of bigrams (n=2), where
occurrences of pairs of consecutive words are counted.

One might alternatively consider a collection of character n-grams, a
representation resilient against misspellings and derivations.

For example, let's say we're dealing with a corpus of two documents:

"['words', 'wprds']". The second document contains a misspelling of
the word 'words'. A simple bag of words representation would consider
these two as very distinct documents, differing in both of the two
possible features. A character 2-gram representation, however, would
find the documents matching in 4 out of 8 features, which may help the
preferred classifier decide better:

   >>> ngram_vectorizer = CountVectorizer(analyzer='char_wb', ngram_range=(2, 2))
   >>> counts = ngram_vectorizer.fit_transform(['words', 'wprds'])
   >>> ngram_vectorizer.get_feature_names_out()
   array([' w', 'ds', 'or', 'pr', 'rd', 's ', 'wo', 'wp'], ...)
   >>> counts.toarray().astype(int)
   array([[1, 1, 1, 0, 1, 1, 1, 0],
          [1, 1, 0, 1, 1, 1, 0, 1]])

In the above example, "char_wb" analyzer is used, which creates
n-grams only from characters inside word boundaries (padded with space
on each side). The "char" analyzer, alternatively, creates n-grams
that span across words:

   >>> ngram_vectorizer = CountVectorizer(analyzer='char_wb', ngram_range=(5, 5))
   >>> ngram_vectorizer.fit_transform(['jumpy fox'])
   <1x4 sparse matrix of type '<... 'numpy.int64'>'
      with 4 stored elements in Compressed Sparse ... format>

<1x4 sparse matrix of type '<... 'numpy.int64'>'
   array([' fox ', ' jump', 'jumpy', 'umpy '], ...)

   >>> ngram_vectorizer = CountVectorizer(analyzer='char', ngram_range=(5, 5))
   <1x5 sparse matrix of type '<... 'numpy.int64'>'
       with 5 stored elements in Compressed Sparse ... format>
   array(['jumpy', 'mpy f', 'py fo', 'umpy ', 'y fox'], ...)

The word boundaries-aware variant "char_wb" is especially interesting
for languages that use white-spaces for word separation as it
generates significantly less noisy features than the raw "char"
variant in that case. For such languages it can increase both the
predictive accuracy and convergence speed of classifiers trained using
such features while retaining the robustness with regards to
misspellings and word derivations.

While some local positioning information can be preserved by
extracting n-grams instead of individual words, bag of words and bag
of n-grams destroy most of the inner structure of the document and
hence most of the meaning carried by that internal structure.

In order to address the wider task of Natural Language Understanding,
the local structure of sentences and paragraphs should thus be taken
into account. Many such models will thus be casted as "Structured

output" problems which are currently outside of the scope of scikit-
learn.

6.2.3.8. Vectorizing a large text corpus with the hashing trick
---------------------------------------------------------------

The above vectorization scheme is simple but the fact that it holds an
**in- memory mapping from the string tokens to the integer feature
indices** (the "vocabulary_" attribute) causes several **problems when
dealing with large datasets**:

* the larger the corpus, the larger the vocabulary will grow and hence
  the memory use too,

* fitting requires the allocation of intermediate data structures of
  size proportional to that of the original dataset.

* building the word-mapping requires a full pass over the dataset
  hence it is not possible to fit text classifiers in a strictly
  online manner.

* pickling and un-pickling vectorizers with a large "vocabulary_" can
  be very slow (typically much slower than pickling / un-pickling flat
  data structures such as a NumPy array of the same size),

* it is not easily possible to split the vectorization work into
  concurrent sub tasks as the "vocabulary_" attribute would have to be
  a shared state with a fine grained synchronization barrier: the
  mapping from token string to feature index is dependent on ordering
  of the first occurrence of each token hence would have to be shared,

mapping from token string to feature index is dependent on ordering
  potentially harming the concurrent workers' performance to the point
  of making them slower than the sequential variant.

It is possible to overcome those limitations by combining the "hashing
trick" (Feature hashing) implemented by the "FeatureHasher" class and
the text preprocessing and tokenization features of the
"CountVectorizer".

This combination is implementing in "HashingVectorizer", a transformer
class that is mostly API compatible with "CountVectorizer".
"HashingVectorizer" is stateless, meaning that you don't have to call
"fit" on it:

   >>> from sklearn.feature_extraction.text import HashingVectorizer
   >>> hv = HashingVectorizer(n_features=10)
   >>> hv.transform(corpus)
   <4x10 sparse matrix of type '<... 'numpy.float64'>'
       with 16 stored elements in Compressed Sparse ... format>

You can see that 16 non-zero feature tokens were extracted in the
vector output: this is less than the 19 non-zeros extracted previously
by the "CountVectorizer" on the same toy corpus. The discrepancy comes
from hash function collisions because of the low value of the
"n_features" parameter.

In a real world setting, the "n_features" parameter can be left to its
default value of "2 ** 20" (roughly one million possible features). If
memory or downstream models size is an issue selecting a lower value

such as "2 ** 18" might help without introducing too many additional
collisions on typical text classification tasks.

Note that the dimensionality does not affect the CPU training time of
algorithms which operate on CSR matrices ("LinearSVC(dual=True)",
"Perceptron", "SGDClassifier", "PassiveAggressive") but it does for
algorithms that work with CSC matrices ("LinearSVC(dual=False)",
"Lasso()", etc).

Let's try again with the default setting:

   >>> hv = HashingVectorizer()
   <4x1048576 sparse matrix of type '<... 'numpy.float64'>'

We no longer get the collisions, but this comes at the expense of a
much larger dimensionality of the output space. Of course, other terms
than the 19 used here might still collide with each other.

The "HashingVectorizer" also comes with the following limitations:

* it is not possible to invert the model (no "inverse_transform"
  method), nor to access the original string representation of the
  features, because of the one-way nature of the hash function that
  performs the mapping.

* it does not provide IDF weighting as that would introduce
  statefulness in the model. A "TfidfTransformer" can be appended to
  it in a pipeline if required.

6.2.3.9. Performing out-of-core scaling with HashingVectorizer

statefulness in the model. A "TfidfTransformer" can be appended to

An interesting development of using a "HashingVectorizer" is the
ability to perform out-of-core scaling. This means that we can learn
from data that does not fit into the computer's main memory.

A strategy to implement out-of-core scaling is to stream data to the
estimator in mini-batches. Each mini-batch is vectorized using
"HashingVectorizer" so as to guarantee that the input space of the
estimator has always the same dimensionality. The amount of memory
used at any time is thus bounded by the size of a mini-batch. Although
there is no limit to the amount of data that can be ingested using
such an approach, from a practical point of view the learning time is
often limited by the CPU time one wants to spend on the task.

For a full-fledged example of out-of-core scaling in a text
classification task see *sphx_glr_auto_examples_applications_plot_out
_of_core_classification.py*.

6.2.3.10. Customizing the vectorizer classes

It is possible to customize the behavior by passing a callable to the
vectorizer constructor:

   >>> def my_tokenizer(s):
   ...     return s.split()
   ...
   >>> vectorizer = CountVectorizer(tokenizer=my_tokenizer)
   >>> vectorizer.build_analyzer()(u"Some... punctuation!") == (

>>> def my_tokenizer(s):
   ...
   ...     ['some...', 'punctuation!'])
   True

In particular we name:

   * "preprocessor": a callable that takes an entire document as input
     (as a single string), and returns a possibly transformed version
     of the document, still as an entire string. This can be used to
     remove HTML tags, lowercase the entire document, etc.

   * "tokenizer": a callable that takes the output from the
     preprocessor and splits it into tokens, then returns a list of
     these.

   * "analyzer": a callable that replaces the preprocessor and
     tokenizer. The default analyzers all call the preprocessor and
     tokenizer, but custom analyzers will skip this. N-gram extraction
     and stop word filtering take place at the analyzer level, so a
     custom analyzer may have to reproduce these steps.

(Lucene users might recognize these names, but be aware that scikit-
learn concepts may not map one-to-one onto Lucene concepts.)

To make the preprocessor, tokenizer and analyzers aware of the model
parameters it is possible to derive from the class and override the
"build_preprocessor", "build_tokenizer" and "build_analyzer" factory
methods instead of passing custom functions.

Some tips and tricks:

   * If documents are pre-tokenized by an external package, then store

     them in files (or strings) with the tokens separated by
     whitespace and pass "analyzer=str.split"

   * Fancy token-level analysis such as stemming, lemmatizing,
     compound splitting, filtering based on part-of-speech, etc. are
     not included in the scikit-learn codebase, but can be added by
     customizing either the tokenizer or the analyzer. Here's a
     "CountVectorizer" with a tokenizer and lemmatizer using NLTK:

        >>> from nltk import word_tokenize          
        >>> from nltk.stem import WordNetLemmatizer 
        >>> class LemmaTokenizer:
        ...     def __init__(self):
        ...         self.wnl = WordNetLemmatizer()
        ...     def __call__(self, doc):
        ...         return [self.wnl.lemmatize(t) for t in word_tokenize(doc)]
        ...
        >>> vect = CountVectorizer(tokenizer=LemmaTokenizer())  

     (Note that this will not filter out punctuation.)

     The following example will, for instance, transform some British
     spelling to American spelling:

        >>> import re
        >>> def to_british(tokens):
        ...     for t in tokens:
        ...         t = re.sub(r"(...)our$", r"\1or", t)
        ...         t = re.sub(r"([bt])re$", r"\1er", t)
        ...         t = re.sub(r"([iy])s(e$|ing|ation)", r"\1z\2", t)
        ...         t = re.sub(r"ogue$", "og", t)
        ...         yield t
        ...

...         t = re.sub(r"([iy])s(e$|ing|ation)", r"\1z\2", t)
        ...         yield t
        ...
        >>> class CustomVectorizer(CountVectorizer):
        ...     def build_tokenizer(self):
        ...         tokenize = super().build_tokenizer()
        ...         return lambda doc: list(to_british(tokenize(doc)))
        ...
        >>> print(CustomVectorizer().build_analyzer()(u"color colour"))
        [...'color', ...'color']

     for other styles of preprocessing; examples include stemming,
     lemmatization, or normalizing numerical tokens, with the latter
     illustrated in:

        * *sphx_glr_auto_examples_bicluster_plot_bicluster_newsgroups
          .py*

Customizing the vectorizer can also be useful when handling Asian
languages that do not use an explicit word separator such as
whitespace.

6.2.4. Image feature extraction

6.2.4.1. Patch extraction

The "extract_patches_2d()" function extracts patches from an image
stored as a two-dimensional array, or three-dimensional with color
information along the third axis. For rebuilding an image from all its
patches, use "reconstruct_from_patches_2d()". For example let us
generate a 4x4 pixel picture with 3 color channels (e.g. in RGB
format):

   >>> from sklearn.feature_extraction import image

   >>> one_image = np.arange(4 * 4 * 3).reshape((4, 4, 3))

format):

   >>> one_image[:, :, 0]  # R channel of a fake RGB picture
   array([[ 0,  3,  6,  9],
          [12, 15, 18, 21],
          [24, 27, 30, 33],
          [36, 39, 42, 45]])

   >>> patches = image.extract_patches_2d(one_image, (2, 2), max_patches=2,
   >>> patches.shape
   (2, 2, 2, 3)
   >>> patches[:, :, :, 0]
   array([[[ 0,  3],
           [12, 15]],

          [[15, 18],
           [27, 30]]])
   >>> patches = image.extract_patches_2d(one_image, (2, 2))
   (9, 2, 2, 3)
   >>> patches[4, :, :, 0]
   array([[15, 18],
          [27, 30]])

Let us now try to reconstruct the original image from the patches by
averaging on overlapping areas:

   >>> reconstructed = image.reconstruct_from_patches_2d(patches, (4, 4, 3))
   >>> np.testing.assert_array_equal(one_image, reconstructed)

The "PatchExtractor" class works in the same way as
"extract_patches_2d()", only it supports multiple images as input. It
is implemented as an estimator, so it can be used in pipelines. See:

   >>> five_images = np.arange(5 * 4 * 4 * 3).reshape(5, 4, 4, 3)
   >>> patches = image.PatchExtractor(patch_size=(2, 2)).transform(five_images)
   (45, 2, 2, 3)

6.2.4.2. Connectivity graph of an image

Several estimators in the scikit-learn can use connectivity

>>> patches.shape

information between features or samples. For instance Ward clustering
(Hierarchical clustering) can cluster together only neighboring pixels
of an image, thus forming contiguous patches:

   [image]

For this purpose, the estimators use a 'connectivity' matrix, giving
which samples are connected.

The function "img_to_graph()" returns such a matrix from a 2D or 3D
image. Similarly, "grid_to_graph()" build a connectivity matrix for
images given the shape of these image.

These matrices can be used to impose connectivity in estimators that
use connectivity information, such as Ward clustering (Hierarchical
clustering), but also to build precomputed kernels, or similarity
matrices.

Note:

  **Examples**

  * *sphx_glr_auto_examples_cluster_plot_coin_ward_segmentation.py*

  * *sphx_glr_auto_examples_cluster_plot_segmentation_toy.py*

  * *sphx_glr_auto_examples_cluster_plot_feature_agglomeration_vs_uni
    variate_selection.py*

3.2. Tuning the hyper-parameters of an estimator
************************************************

Hyper-parameters are parameters that are not directly learnt within
estimators. In scikit-learn they are passed as arguments to the
constructor of the estimator classes. Typical examples include "C",
"kernel" and "gamma" for Support Vector Classifier, "alpha" for Lasso,
etc.

It is possible and recommended to search the hyper-parameter space for
the best cross validation score.

Any parameter provided when constructing an estimator may be optimized
in this manner. Specifically, to find the names and current values for
all parameters for a given estimator, use:

   estimator.get_params()

A search consists of:

* an estimator (regressor or classifier such as "sklearn.svm.SVC()");

* a parameter space;

* a method for searching or sampling candidates;

* a cross-validation scheme; and

* a score function.

Two generic approaches to parameter search are provided in scikit-
learn: for given values, "GridSearchCV" exhaustively considers all
parameter combinations, while "RandomizedSearchCV" can sample a given
number of candidates from a parameter space with a specified
distribution. Both these tools have successive halving counterparts
"HalvingGridSearchCV" and "HalvingRandomSearchCV", which can be much
faster at finding a good parameter combination.

After describing these tools we detail best practices applicable to
these approaches. Some models allow for specialized, efficient

parameter search strategies, outlined in Alternatives to brute force
parameter search.

Note that it is common that a small subset of those parameters can
have a large impact on the predictive or computation performance of
the model while others can be left to their default values. It is
recommended to read the docstring of the estimator class to get a
finer understanding of their expected behavior, possibly by reading
the enclosed reference to the literature.

3.2.1. Exhaustive Grid Search

The grid search provided by "GridSearchCV" exhaustively generates
candidates from a grid of parameter values specified with the
"param_grid" parameter. For instance, the following "param_grid":

   param_grid = [
     {'C': [1, 10, 100, 1000], 'kernel': ['linear']},
     {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
    ]

specifies that two grids should be explored: one with a linear kernel
and C values in [1, 10, 100, 1000], and the second one with an RBF
kernel, and the cross-product of C values ranging in [1, 10, 100,
1000] and gamma values in [0.001, 0.0001].

The "GridSearchCV" instance implements the usual estimator API: when
"fitting" it on a dataset all the possible combinations of parameter
values are evaluated and the best combination is retained.

Examples:
^^^^^^^^^

* See

Examples:
^^^^^^^^^

* See
  *sphx_glr_auto_examples_model_selection_plot_grid_search_digits.py*
  for an example of Grid Search computation on the digits dataset.

* See *sphx_glr_auto_examples_model_selection_plot_grid_search_text_f
  eature_extraction.py* for an example of Grid Search coupling
  parameters from a text documents feature extractor (n-gram count
  vectorizer and TF-IDF transformer) with a classifier (here a linear
  SVM trained with SGD with either elastic net or L2 penalty) using a
  "pipeline.Pipeline" instance.

* See *sphx_glr_auto_examples_model_selection_plot_nested_cross_valid
  ation_iris.py* for an example of Grid Search within a cross
  validation loop on the iris dataset. This is the best practice for
  evaluating the performance of a model with grid search.

* See *sphx_glr_auto_examples_model_selection_plot_multi_metric_evalu
  ation.py* for an example of "GridSearchCV" being used to evaluate
  multiple metrics simultaneously.

* See *sphx_glr_auto_examples_model_selection_plot_grid_search_refit_
  callable.py* for an example of using "refit=callable" interface in
  "GridSearchCV". The example shows how this interface adds certain
  amount of flexibility in identifying the "best" estimator. This
  interface can also be used in multiple metrics evaluation.

* See
  *sphx_glr_auto_examples_model_selection_plot_grid_search_stats.py*

amount of flexibility in identifying the "best" estimator. This

* See
  for an example of how to do a statistical comparison on the outputs
  of "GridSearchCV".

3.2.2. Randomized Parameter Optimization

While using a grid of parameter settings is currently the most widely
used method for parameter optimization, other search methods have more
favorable properties. "RandomizedSearchCV" implements a randomized
search over parameters, where each setting is sampled from a
distribution over possible parameter values. This has two main
benefits over an exhaustive search:

* A budget can be chosen independent of the number of parameters and
  possible values.

* Adding parameters that do not influence the performance does not
  decrease efficiency.

Specifying how parameters should be sampled is done using a
dictionary, very similar to specifying parameters for "GridSearchCV".
Additionally, a computation budget, being the number of sampled
candidates or sampling iterations, is specified using the "n_iter"
parameter. For each parameter, either a distribution over possible
values or a list of discrete choices (which will be sampled uniformly)
can be specified:

   {'C': scipy.stats.expon(scale=100), 'gamma': scipy.stats.expon(scale=.1),
     'kernel': ['rbf'], 'class_weight':['balanced', None]}

This example uses the "scipy.stats" module, which contains many useful
distributions for sampling parameters, such as "expon", "gamma",
"uniform" or "randint".

In principle, any function can be passed that provides a "rvs" (random
variate sample) method to sample a value. A call to the "rvs" function
should provide independent random samples from possible parameter
values on consecutive calls.

   Warning:

     The distributions in "scipy.stats" prior to version scipy 0.16 do
     not allow specifying a random state. Instead, they use the global
     numpy random state, that can be seeded via "np.random.seed" or
     set using "np.random.set_state". However, beginning scikit-learn
     0.18, the "sklearn.model_selection" module sets the random state
     provided by the user if scipy >= 0.16 is also available.

For continuous parameters, such as "C" above, it is important to
specify a continuous distribution to take full advantage of the
randomization. This way, increasing "n_iter" will always lead to a
finer search.

A continuous log-uniform random variable is available through
"loguniform". This is a continuous version of log-spaced parameters.
For example to specify "C" above, "loguniform(1, 100)" can be used
instead of "[1, 10, 100]" or "np.logspace(0, 2, num=1000)". This is an
alias to scipy.stats.loguniform.

Mirroring the example above in grid search, we can specify a
continuous random variable that is log-uniformly distributed between
"1e0" and "1e3":

   from sklearn.utils.fixes import loguniform
   {'C': loguniform(1e0, 1e3),
    'gamma': loguniform(1e-4, 1e-3),
    'kernel': ['rbf'],
    'class_weight':['balanced', None]}

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_model_selection_plot_randomized_search.py*
  compares the usage and efficiency of randomized search and grid
  search.

* Bergstra, J. and Bengio, Y., Random search for hyper-parameter
  optimization, The Journal of Machine Learning Research (2012)

3.2.3. Searching for optimal parameters with successive halving

Scikit-learn also provides the "HalvingGridSearchCV" and
"HalvingRandomSearchCV" estimators that can be used to search a
parameter space using successive halving [1] [2]. Successive halving
(SH) is like a tournament among candidate parameter combinations. SH
is an iterative selection process where all candidates (the parameter
combinations) are evaluated with a small amount of resources at the
first iteration. Only some of these candidates are selected for the
next iteration, which will be allocated more resources. For parameter

tuning, the resource is typically the number of training samples, but
it can also be an arbitrary numeric parameter such as *n_estimators*
in a random forest.

As illustrated in the figure below, only a subset of candidates
'survive' until the last iteration. These are the candidates that have
consistently ranked among the top-scoring candidates across all
iterations. Each iteration is allocated an increasing amount of
resources per candidate, here the number of samples.

   [image]

We here briefly describe the main parameters, but each parameter and
their interactions are described in more details in the sections
below. The "factor" (> 1) parameter controls the rate at which the
resources grow, and the rate at which the number of candidates
decreases. In each iteration, the number of resources per candidate is
multiplied by "factor" and the number of candidates is divided by the
same factor. Along with "resource" and "min_resources", "factor" is
the most important parameter to control the search in our
implementation, though a value of 3 usually works well. "factor"
effectively controls the number of iterations in "HalvingGridSearchCV"
and the number of candidates (by default) and iterations in
"HalvingRandomSearchCV". "aggressive_elimination=True" can also be
used if the number of available resources is small. More control is

available through tuning the "min_resources" parameter.

These estimators are still **experimental**: their predictions and
their API might change without any deprecation cycle. To use them, you
need to explicitly import "enable_halving_search_cv":

   >>> # explicitly require this experimental feature
   >>> from sklearn.experimental import enable_halving_search_cv  # noqa
   >>> # now you can import normally from model_selection
   >>> from sklearn.model_selection import HalvingGridSearchCV
   >>> from sklearn.model_selection import HalvingRandomSearchCV

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_model_selection_plot_successive_halving_hea
  tmap.py*

* *sphx_glr_auto_examples_model_selection_plot_successive_halving_ite
  rations.py*

3.2.3.1. Choosing "min_resources" and the number of candidates

Beside "factor", the two main parameters that influence the behaviour
of a successive halving search are the "min_resources" parameter, and
the number of candidates (or parameter combinations) that are
evaluated. "min_resources" is the amount of resources allocated at the
first iteration for each candidate. The number of candidates is
specified directly in "HalvingRandomSearchCV", and is determined from

the "param_grid" parameter of "HalvingGridSearchCV".

Consider a case where the resource is the number of samples, and where
we have 1000 samples. In theory, with "min_resources=10" and
"factor=2", we are able to run **at most** 7 iterations with the
following number of samples: "[10, 20, 40, 80, 160, 320, 640]".

But depending on the number of candidates, we might run less than 7
iterations: if we start with a **small** number of candidates, the
last iteration might use less than 640 samples, which means not using
all the available resources (samples). For example if we start with 5
candidates, we only need 2 iterations: 5 candidates for the first
iteration, then *5 // 2 = 2* candidates at the second iteration, after
which we know which candidate performs the best (so we don't need a
third one). We would only be using at most 20 samples which is a waste
since we have 1000 samples at our disposal. On the other hand, if we
start with a **high** number of candidates, we might end up with a lot
of candidates at the last iteration, which may not always be ideal: it
means that many candidates will run with the full resources, basically
reducing the procedure to standard search.

In the case of "HalvingRandomSearchCV", the number of candidates is
set by default such that the last iteration uses as much of the

available resources as possible. For "HalvingGridSearchCV", the number
of candidates is determined by the *param_grid* parameter. Changing
the value of "min_resources" will impact the number of possible
iterations, and as a result will also have an effect on the ideal
number of candidates.

Another consideration when choosing "min_resources" is whether or not
it is easy to discriminate between good and bad candidates with a
small amount of resources. For example, if you need a lot of samples
to distinguish between good and bad parameters, a high "min_resources"
is recommended. On the other hand if the distinction is clear even
with a small amount of samples, then a small "min_resources" may be
preferable since it would speed up the computation.

Notice in the example above that the last iteration does not use the
maximum amount of resources available: 1000 samples are available, yet
only 640 are used, at most. By default, both "HalvingRandomSearchCV"
and "HalvingGridSearchCV" try to use as many resources as possible in
the last iteration, with the constraint that this amount of resources
must be a multiple of both *min_resources* and *factor* (this
constraint will be clear in the next section). "HalvingRandomSearchCV"
achieves this by sampling the right amount of candidates, while

"HalvingGridSearchCV" achieves this by properly setting
*min_resources*. Please see Exhausting the available resources for
details.

3.2.3.2. Amount of resource and number of candidates at each iteration
----------------------------------------------------------------------

At any iteration *i*, each candidate is allocated a given amount of
resources which we denote *n_resources_i*. This quantity is controlled
by the parameters "factor" and "min_resources" as follows (*factor* is
strictly greater than 1):

   n_resources_i = factor**i * min_resources,

or equivalently:

   n_resources_{i+1} = n_resources_i * factor

where "min_resources == n_resources_0" is the amount of resources used
at the first iteration. "factor" also defines the proportions of
candidates that will be selected for the next iteration:

   n_candidates_i = n_candidates // (factor ** i)

   n_candidates_0 = n_candidates
   n_candidates_{i+1} = n_candidates_i // factor

So in the first iteration, we use "min_resources" resources
"n_candidates" times. In the second iteration, we use "min_resources *
factor" resources "n_candidates // factor" times. The third again
multiplies the resources per candidate and divides the number of
candidates. This process stops when the maximum amount of resource per

candidate is reached, or when we have identified the best candidate.
The best candidate is identified at the iteration that is evaluating
*factor* or less candidates (see just below for an explanation).

Here is an example with "min_resources=3" and "factor=2", starting
with 70 candidates:

+-------------------------+-------------------------+
| "n_resources_i"         | "n_candidates_i"        |
|=========================|=========================|
| 3 (=min_resources)      | 70 (=n_candidates)      |
| 3 * 2 = 6               | 70 // 2 = 35            |
| 6 * 2 = 12              | 35 // 2 = 17            |
| 12 * 2 = 24             | 17 // 2 = 8             |
| 24 * 2 = 48             | 8 // 2 = 4              |
| 48 * 2 = 96             | 4 // 2 = 2              |

We can note that:

* the process stops at the first iteration which evaluates *factor=2*
  candidates: the best candidate is the best out of these 2
  candidates. It is not necessary to run an additional iteration,

  since it would only evaluate one candidate (namely the best one,
  which we have already identified). For this reason, in general, we
  want the last iteration to run at most "factor" candidates. If the
  last iteration evaluates more than *factor* candidates, then this
  last iteration reduces to a regular search (as in
  "RandomizedSearchCV" or "GridSearchCV").

* each "n_resources_i" is a multiple of both "factor" and
  "min_resources" (which is confirmed by its definition above).

The amount of resources that is used at each iteration can be found in
the *n_resources_* attribute.

3.2.3.3. Choosing a resource

By default, the resource is defined in terms of number of samples.
That is, each iteration will use an increasing amount of samples to
train on. You can however manually specify a parameter to use as the
resource with the "resource" parameter. Here is an example where the
resource is defined in terms of the number of estimators of a random
forest:

   >>>
   >>> param_grid = {'max_depth': [3, 5, 10],

>>> from sklearn.model_selection import HalvingGridSearchCV
   >>>
   ...               'min_samples_split': [2, 5, 10]}
   >>> base_estimator = RandomForestClassifier(random_state=0)
   >>> X, y = make_classification(n_samples=1000, random_state=0)
   >>> sh = HalvingGridSearchCV(base_estimator, param_grid, cv=5,
   ...                          factor=2, resource='n_estimators',
   ...                          max_resources=30).fit(X, y)
   >>> sh.best_estimator_
   RandomForestClassifier(max_depth=5, n_estimators=24, random_state=0)

Note that it is not possible to budget on a parameter that is part of
the parameter grid.

3.2.3.4. Exhausting the available resources

As mentioned above, the number of resources that is used at each
iteration depends on the *min_resources* parameter. If you have a lot
of resources available but start with a low number of resources, some
of them might be wasted (i.e. not used):

   >>> param_grid= {'kernel': ('linear', 'rbf'),
   ...              'C': [1, 10, 100]}
   >>> base_estimator = SVC(gamma='scale')
   >>> X, y = make_classification(n_samples=1000)

>>> param_grid= {'kernel': ('linear', 'rbf'),
   ...                          factor=2, min_resources=20).fit(X, y)
   >>> sh.n_resources_
   [20, 40, 80]

The search process will only use 80 resources at most, while our
maximum amount of available resources is "n_samples=1000". Here, we
have "min_resources = r_0 = 20".

For "HalvingGridSearchCV", by default, the *min_resources* parameter
is set to 'exhaust'. This means that *min_resources* is automatically
set such that the last iteration can use as many resources as
possible, within the *max_resources* limit:

   ...                          factor=2, min_resources='exhaust').fit(X, y)
   [250, 500, 1000]

*min_resources* was here automatically set to 250, which results in
the last iteration using all the resources. The exact value that is
used depends on the number of candidate parameter, on *max_resources*
and on *factor*.

For "HalvingRandomSearchCV", exhausting the resources can be done in 2
ways:

* by setting *min_resources='exhaust'*, just like for
  "HalvingGridSearchCV";

* by setting *n_candidates='exhaust'*.

Both options are mutally exclusive: using *min_resources='exhaust'*
requires knowing the number of candidates, and symmetrically

"HalvingGridSearchCV";

*n_candidates='exhaust'* requires knowing *min_resources*.

In general, exhausting the total number of resources leads to a better
final candidate parameter, and is slightly more time-intensive.

3.2.3.5. Aggressive elimination of candidates

Ideally, we want the last iteration to evaluate "factor" candidates
(see Amount of resource and number of candidates at each iteration).
We then just have to pick the best one. When the number of available
resources is small with respect to the number of candidates, the last
iteration may have to evaluate more than "factor" candidates:

   >>>
   >>>
   >>> param_grid = {'kernel': ('linear', 'rbf'),
   ...               'C': [1, 10, 100]}
   ...                          factor=2, max_resources=40,
   ...                          aggressive_elimination=False).fit(X, y)
   [20, 40]
   >>> sh.n_candidates_

...                          factor=2, max_resources=40,
   [20, 40]
   [6, 3]

Since we cannot use more than "max_resources=40" resources, the
process has to stop at the second iteration which evaluates more than
"factor=2" candidates.

Using the "aggressive_elimination" parameter, you can force the search
process to end up with less than "factor" candidates at the last
iteration. To do this, the process will eliminate as many candidates
as necessary using "min_resources" resources:

   ...                            factor=2,
   ...                            max_resources=40,
   ...                            aggressive_elimination=True,
   ...                            ).fit(X, y)
   [20, 20,  40]
   [6, 3, 2]

Notice that we end with 2 candidates at the last iteration since we
have eliminated enough candidates during the first iterations, using
"n_resources = min_resources = 20".

3.2.3.6. Analyzing results with the *cv_results_* attribute
-----------------------------------------------------------

The "cv_results_" attribute contains useful information for analyzing
the results of a search. It can be converted to a pandas dataframe
with "df = pd.DataFrame(est.cv_results_)". The "cv_results_" attribute
of "HalvingGridSearchCV" and "HalvingRandomSearchCV" is similar to

that of "GridSearchCV" and "RandomizedSearchCV", with additional
information related to the successive halving process.

Here is an example with some of the columns of a (truncated)
dataframe:

+------+--------+-----------------+-------------------+------------------------------------------------------------------------------------------+
|      | iter   | n_resources     | mean_test_score   | params                                                                                   |
|======|========|=================|===================|==========================================================================================|
| 0    | 0      | 125             | 0.983667          | {'criterion': 'log_loss', 'max_depth': None, 'max_features': 9, 'min_samples_split': 5}  |
| 1    | 0      | 125             | 0.983667          | {'criterion': 'gini', 'max_depth': None, 'max_features': 8, 'min_samples_split': 7}      |
| 2    | 0      | 125             | 0.983667          | {'criterion': 'gini', 'max_depth': None, 'max_features': 10, 'min_samples_split': 10}    |

| 3    | 0      | 125             | 0.983667          | {'criterion': 'log_loss', 'max_depth': None, 'max_features': 6, 'min_samples_split': 6}  |
| ...  | ...    | ...             | ...               | ...                                                                                      |
| 15   | 2      | 500             | 0.951958          | {'criterion': 'log_loss', 'max_depth': None, 'max_features': 9, 'min_samples_split': 10} |
| 16   | 2      | 500             | 0.947958          | {'criterion': 'gini', 'max_depth': None, 'max_features': 10, 'min_samples_split': 10}    |

| 17   | 2      | 500             | 0.951958          | {'criterion': 'gini', 'max_depth': None, 'max_features': 10, 'min_samples_split': 4}     |
| 18   | 3      | 1000            | 0.961009          | {'criterion': 'log_loss', 'max_depth': None, 'max_features': 9, 'min_samples_split': 10} |
| 19   | 3      | 1000            | 0.955989          | {'criterion': 'gini', 'max_depth': None, 'max_features': 10, 'min_samples_split': 4}     |

Each row corresponds to a given parameter combination (a candidate)
and a given iteration. The iteration is given by the "iter" column.
The "n_resources" column tells you how many resources were used.

In the example above, the best parameter combination is "{'criterion':
'log_loss', 'max_depth': None, 'max_features': 9, 'min_samples_split':
10}" since it has reached the last iteration (3) with the highest
score: 0.96.

[1] K. Jamieson, A. Talwalkar, Non-stochastic Best Arm Identification
    and Hyperparameter Optimization, in proc. of Machine Learning
    Research, 2016.

[2] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, A. Talwalkar,
    >>:arxiv:`Hyperband: A Novel Bandit-Based Approach to
    Hyperparameter Optimization <1603.06560>`<<, in Machine Learning
    Research 18, 2018.

3.2.4. Tips for parameter search

3.2.4.1. Specifying an objective metric

By default, parameter search uses the "score" function of the
estimator to evaluate a parameter setting. These are the
"sklearn.metrics.accuracy_score()" for classification and
"sklearn.metrics.r2_score()" for regression.  For some applications,
other scoring functions are better suited (for example in unbalanced
classification, the accuracy score is often uninformative). An
alternative scoring function can be specified via the "scoring"
parameter of most parameter search tools. See The scoring parameter:
defining model evaluation rules for more details.

3.2.4.2. Specifying multiple metrics for evaluation

"GridSearchCV" and "RandomizedSearchCV" allow specifying multiple
metrics for the "scoring" parameter.

Multimetric scoring can either be specified as a list of strings of

predefined scores names or a dict mapping the scorer name to the
scorer function and/or the predefined scorer name(s). See Using
multiple metric evaluation for more details.

When specifying multiple metrics, the "refit" parameter must be set to
the metric (string) for which the "best_params_" will be found and
used to build the "best_estimator_" on the whole dataset. If the
search should not be refit, set "refit=False". Leaving refit to the
default value "None" will result in an error when using multiple
metrics.

See *sphx_glr_auto_examples_model_selection_plot_multi_metric_evaluat
ion.py* for an example usage.

"HalvingRandomSearchCV" and "HalvingGridSearchCV" do not support
multimetric scoring.

3.2.4.3. Composite estimators and parameter spaces

"GridSearchCV" and "RandomizedSearchCV" allow searching over
parameters of composite or nested estimators such as "Pipeline",
"ColumnTransformer", "VotingClassifier" or "CalibratedClassifierCV"
using a dedicated "<estimator>__<parameter>" syntax:

   >>> from sklearn.calibration import CalibratedClassifierCV
   >>> from sklearn.datasets import make_moons
   >>> X, y = make_moons()
   >>> calibrated_forest = CalibratedClassifierCV(

>>> from sklearn.ensemble import RandomForestClassifier
   ...    estimator=RandomForestClassifier(n_estimators=10))
   >>> param_grid = {
   ...    'estimator__max_depth': [2, 4, 6, 8]}
   >>> search = GridSearchCV(calibrated_forest, param_grid, cv=5)
   >>> search.fit(X, y)
   GridSearchCV(cv=5,
                estimator=CalibratedClassifierCV(...),
                param_grid={'estimator__max_depth': [2, 4, 6, 8]})

Here, "<estimator>" is the parameter name of the nested estimator, in
this case "estimator". If the meta-estimator is constructed as a
collection of estimators as in *pipeline.Pipeline*, then "<estimator>"
refers to the name of the estimator, see Nested parameters.  In
practice, there can be several levels of nesting:

   >>> pipe = Pipeline([
   ...    ('select', SelectKBest()),
   ...    ('model', calibrated_forest)])
   ...    'select__k': [1, 2],
   ...    'model__estimator__max_depth': [2, 4, 6, 8]}
   >>> search = GridSearchCV(pipe, param_grid, cv=5).fit(X, y)

Please refer to Pipeline: chaining estimators for performing parameter
searches over pipelines.

3.2.4.4. Model selection: development and evaluation

Model selection by evaluating various parameter settings can be seen

as a way to use the labeled data to "train" the parameters of the
grid.

When evaluating the resulting model it is important to do it on held-
out samples that were not seen during the grid search process: it is
recommended to split the data into a **development set** (to be fed to
the "GridSearchCV" instance) and an **evaluation set** to compute
performance metrics.

This can be done by using the "train_test_split()" utility function.

3.2.4.5. Parallelism

The parameter search tools evaluate each parameter combination on each
data fold independently. Computations can be run in parallel by using
the keyword "n_jobs=-1". See function signature for more details, and
also the Glossary entry for *n_jobs*.

3.2.4.6. Robustness to failure

Some parameter settings may result in a failure to "fit" one or more
folds of the data.  By default, this will cause the entire search to
fail, even if some parameter settings could be fully evaluated.
Setting "error_score=0" (or *=np.NaN*) will make the procedure robust
to such failure, issuing a warning and setting the score for that fold
to 0 (or *NaN*), but completing the search.

3.2.5. Alternatives to brute force parameter search

3.2.5.1. Model specific cross-validation

Some models can fit data for a range of values of some parameter
almost as efficiently as fitting the estimator for a single value of
the parameter. This feature can be leveraged to perform a more
efficient cross-validation used for model selection of this parameter.

The most common parameter amenable to this strategy is the parameter
encoding the strength of the regularizer. In this case we say that we
compute the **regularization path** of the estimator.

Here is the list of such models:

3.2.5.2. Information Criterion

Some models can offer an information-theoretic closed-form formula of
the optimal estimate of the regularization parameter by computing a
single regularization path (instead of several when using cross-
validation).

Here is the list of models benefiting from the Akaike Information
Criterion (AIC) or the Bayesian Information Criterion (BIC) for
automated model selection:

3.2.5.3. Out of Bag Estimates

When using ensemble methods base upon bagging, i.e. generating new
training sets using sampling with replacement, part of the training
set remains unused.  For each classifier in the ensemble, a different
part of the training set is left out.

This left out portion can be used to estimate the generalization error
without having to rely on a separate validation set.  This estimate
comes "for free" as no additional data is needed and can be used for
model selection.

This is currently implemented in the following classes:

6.5. Unsupervised dimensionality reduction
******************************************

If your number of features is high, it may be useful to reduce it with
an unsupervised step prior to supervised steps. Many of the
Unsupervised learning methods implement a "transform" method that can
be used to reduce the dimensionality. Below we discuss two specific
example of this pattern that are heavily used.

**Pipelining**

The unsupervised data reduction and the supervised estimator can be
chained in one step. See Pipeline: chaining estimators.

6.5.1. PCA: principal component analysis

"decomposition.PCA" looks for a combination of features that capture
well the variance of the original features. See Decomposing signals in
components (matrix factorization problems).

**Examples**

6.5.2. Random projections

The module: "random_projection" provides several tools for data
reduction by random projections. See the relevant section of the
documentation: Random Projection.

* *sphx_glr_auto_examples_miscellaneous_plot_johnson_lindenstrauss_bo
  und.py*

6.5.3. Feature agglomeration

"cluster.FeatureAgglomeration" applies Hierarchical clustering to
group together features that behave similarly.

  riate_selection.py*

* *sphx_glr_auto_examples_cluster_plot_digits_agglomeration.py*

**Feature scaling**

Note that if features have very different scaling or statistical
properties, "cluster.FeatureAgglomeration" may not be able to capture
the links between related features. Using a
"preprocessing.StandardScaler" can be useful in these settings.

6.6. Random Projection

The "sklearn.random_projection" module implements a simple and
computationally efficient way to reduce the dimensionality of the data
by trading a controlled amount of accuracy (as additional variance)
for faster processing times and smaller model sizes. This module
implements two types of unstructured random matrix: Gaussian random
matrix and sparse random matrix.

The dimensions and distribution of random projections matrices are
controlled so as to preserve the pairwise distances between any two
samples of the dataset. Thus random projection is a suitable
approximation technique for distance based method.

* Sanjoy Dasgupta. 2000. Experiments with random projection. In
  Proceedings of the Sixteenth conference on Uncertainty in artificial
  intelligence (UAI'00), Craig Boutilier and Moisés Goldszmidt (Eds.).
  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 143-151.

* Ella Bingham and Heikki Mannila. 2001. Random projection in
  dimensionality reduction: applications to image and text data. In
  Proceedings of the seventh ACM SIGKDD international conference on
  Knowledge discovery and data mining (KDD '01). ACM, New York, NY,
  USA, 245-250.

6.6.1. The Johnson-Lindenstrauss lemma

The main theoretical result behind the efficiency of random projection
is the Johnson-Lindenstrauss lemma (quoting Wikipedia):

   In mathematics, the Johnson-Lindenstrauss lemma is a result

   concerning low-distortion embeddings of points from high-
   dimensional into low-dimensional Euclidean space. The lemma states
   that a small set of points in a high-dimensional space can be
   embedded into a space of much lower dimension in such a way that
   distances between the points are nearly preserved. The map used for
   the embedding is at least Lipschitz, and can even be taken to be an
   orthogonal projection.

Knowing only the number of samples, the
"johnson_lindenstrauss_min_dim()" estimates conservatively the minimal
size of the random subspace to guarantee a bounded distortion
introduced by the random projection:

   >>> from sklearn.random_projection import johnson_lindenstrauss_min_dim
   >>> johnson_lindenstrauss_min_dim(n_samples=1e6, eps=0.5)
   663
   >>> johnson_lindenstrauss_min_dim(n_samples=1e6, eps=[0.5, 0.1, 0.01])
   array([    663,   11841, 1112658])
   >>> johnson_lindenstrauss_min_dim(n_samples=[1e4, 1e5, 1e6], eps=0.1)
   array([ 7894,  9868, 11841])

   [image]

   [image]

Example:
^^^^^^^^

* See *sphx_glr_auto_examples_miscellaneous_plot_johnson_lindenstraus
  s_bound.py* for a theoretical explication on the Johnson-
  Lindenstrauss lemma and an empirical validation using sparse random
  matrices.

s_bound.py* for a theoretical explication on the Johnson-
  matrices.

* Sanjoy Dasgupta and Anupam Gupta, 1999. An elementary proof of the
  Johnson-Lindenstrauss Lemma.

6.6.2. Gaussian random projection

The "GaussianRandomProjection" reduces the dimensionality by
projecting the original input space on a randomly generated matrix
where components are drawn from the following distribution N(0,
\frac{1}{n_{components}}).

Here a small excerpt which illustrates how to use the Gaussian random
projection transformer:

   >>> from sklearn import random_projection
   >>> X = np.random.rand(100, 10000)
   >>> transformer = random_projection.GaussianRandomProjection()
   (100, 3947)

6.6.3. Sparse random projection

The "SparseRandomProjection" reduces the dimensionality by projecting
the original input space using a sparse random matrix.

Sparse random matrices are an alternative to dense Gaussian random
projection matrix that guarantees similar embedding quality while
being much more memory efficient and allowing faster computation of
the projected data.

If we define "s = 1 / density", the elements of the random matrix are
drawn from

   \left\{ \begin{array}{c c l}
   -\sqrt{\frac{s}{n_{\text{components}}}} & & 1 / 2s\\ 0 &\text{with
   probability}  & 1 - 1 / s \\

drawn from

   +\sqrt{\frac{s}{n_{\text{components}}}} & & 1 / 2s\\ \end{array}
   \right.

where n_{\text{components}} is the size of the projected subspace. By
default the density of non zero elements is set to the minimum density
as recommended by Ping Li et al.: 1 / \sqrt{n_{\text{features}}}.

Here a small excerpt which illustrates how to use the sparse random

   >>> transformer = random_projection.SparseRandomProjection()

* D. Achlioptas. 2003. Database-friendly random projections: Johnson-
  Lindenstrauss  with binary coins. Journal of Computer and System
  Sciences 66 (2003) 671–687

* Ping Li, Trevor J. Hastie, and Kenneth W. Church. 2006. Very sparse
  random projections. In Proceedings of the 12th ACM SIGKDD
  international conference on Knowledge discovery and data mining (KDD
  '06). ACM, New York, NY, USA, 287-296.

6.6.4. Inverse Transform

The random projection transformers have "compute_inverse_components"
parameter. When set to True, after creating the random "components_"
matrix during fitting, the transformer computes the pseudo-inverse of
this matrix and stores it as "inverse_components_". The

"inverse_components_" matrix has shape n_{features} \times
n_{components}, and it is always a dense matrix, regardless of whether
the components matrix is sparse or dense. So depending on the number
of features and components, it may use a lot of memory.

When the "inverse_transform" method is called, it computes the product
of the input "X" and the transpose of the inverse components. If the
inverse components have been computed during fit, they are reused at
each call to "inverse_transform". Otherwise they are recomputed each
time, which can be costly. The result is always dense, even if "X" is
sparse.

Here a small code example which illustrates how to use the inverse
transform feature:

   >>> from sklearn.random_projection import SparseRandomProjection
   >>> transformer = SparseRandomProjection(
   ...   compute_inverse_components=True
   ... )
   ...
   >>> X_new_inversed = transformer.inverse_transform(X_new)
   >>> X_new_inversed.shape
   (100, 10000)
   >>> X_new_again = transformer.transform(X_new_inversed)
   >>> np.allclose(X_new, X_new_again)
   True

4.2. Permutation feature importance
***********************************

Permutation feature importance is a model inspection technique that
can be used for any *fitted* *estimator* when the data is tabular.
This is especially useful for non-linear or opaque *estimators*. The
permutation feature importance is defined to be the decrease in a
model score when a single feature value is randomly shuffled [1]. This
procedure breaks the relationship between the feature and the target,
thus the drop in the model score is indicative of how much the model
depends on the feature. This technique benefits from being model
agnostic and can be calculated many times with different permutations
of the feature.

Warning:

  Features that are deemed of **low importance for a bad model** (low
  cross-validation score) could be **very important for a good
  model**. Therefore it is always important to evaluate the predictive
  power of a model using a held-out set (or better with cross-
  validation) prior to computing importances. Permutation importance
  does not reflect to the intrinsic predictive value of a feature by
  itself but **how important this feature is for a particular model**.

The "permutation_importance()" function calculates the feature
importance of *estimators* for a given dataset. The "n_repeats"
parameter sets the number of times a feature is randomly shuffled and
returns a sample of feature importances.

Let's consider the following trained regression model:

   >>> diabetes = load_diabetes()
   >>> X_train, X_val, y_train, y_val = train_test_split(
   ...     diabetes.data, diabetes.target, random_state=0)
   ...
   >>> model = Ridge(alpha=1e-2).fit(X_train, y_train)
   >>> model.score(X_val, y_val)
   0.356...

Its validation performance, measured via the R^2 score, is
significantly larger than the chance level. This makes it possible to
use the "permutation_importance()" function to probe which features
are most predictive:

   >>> from sklearn.inspection import permutation_importance
   >>> r = permutation_importance(model, X_val, y_val,
   ...                            n_repeats=30,
   ...                            random_state=0)
   ...
   >>> for i in r.importances_mean.argsort()[::-1]:
   ...     if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
   ...         print(f"{diabetes.feature_names[i]:<8}"
   ...               f"{r.importances_mean[i]:.3f}"
   ...               f" +/- {r.importances_std[i]:.3f}")
   ...
   s5      0.204 +/- 0.050
   bmi     0.176 +/- 0.048
   bp      0.088 +/- 0.033
   sex     0.056 +/- 0.023

Note that the importance values for the top features represent a large
fraction of the reference score of 0.356.

bmi     0.176 +/- 0.048

Permutation importances can be computed either on the training set or
on a held-out testing or validation set. Using a held-out set makes it
possible to highlight which features contribute the most to the
generalization power of the inspected model. Features that are
important on the training set but not on the held-out set might cause
the model to overfit.

The permutation feature importance is the decrease in a model score
when a single feature value is randomly shuffled. The score function
to be used for the computation of importances can be specified with
the *scoring* argument, which also accepts multiple scorers. Using
multiple scorers is more computationally efficient than sequentially
calling "permutation_importance()" several times with a different
scorer, as it reuses model predictions.

An example of using multiple scorers is shown below, employing a list
of metrics, but more input formats are possible, as documented in
Using multiple metric evaluation.

>>> scoring = ['r2', 'neg_mean_absolute_percentage_error', 'neg_mean_squared_error']
>>> r_multi = permutation_importance(
...     model, X_val, y_val, n_repeats=30, random_state=0, scoring=scoring)
...
>>> for metric in r_multi:
...     print(f"{metric}")
...     r = r_multi[metric]
...     for i in r.importances_mean.argsort()[::-1]:

...
...         if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
...             print(f"    {diabetes.feature_names[i]:<8}"
...                   f"{r.importances_mean[i]:.3f}"
...                   f" +/- {r.importances_std[i]:.3f}")
...
r2
    s5      0.204 +/- 0.050
    bmi     0.176 +/- 0.048
    bp      0.088 +/- 0.033
    sex     0.056 +/- 0.023
neg_mean_absolute_percentage_error
    s5      0.081 +/- 0.020
    bmi     0.064 +/- 0.015
    bp      0.029 +/- 0.010
neg_mean_squared_error
    s5      1013.866 +/- 246.445
    bmi     872.726 +/- 240.298
    bp      438.663 +/- 163.022
    sex     277.376 +/- 115.123

The ranking of the features is approximately the same for different
metrics even if the scales of the importance values are very
different. However, this is not guaranteed and different metrics might
lead to significantly different feature importances, in particular for
models trained for imbalanced classification problems, for which the
choice of the classification metric can be critical.

4.2.1. Outline of the permutation importance algorithm

* Inputs: fitted predictive model m, tabular dataset (training or
  validation) D.

* Compute the reference score s of the model m on data D (for instance
  the accuracy for a classifier or the R^2 for a regressor).

* For each feature j (column of D):

validation) D.

  * For each repetition k in {1, ..., K}:

    * Randomly shuffle column j of dataset D to generate a corrupted
      version of the data named \tilde{D}_{k,j}.

    * Compute the score s_{k,j} of model m on corrupted data
      \tilde{D}_{k,j}.

  * Compute importance i_j for feature f_j defined as:

       i_j = s - \frac{1}{K} \sum_{k=1}^{K} s_{k,j}

4.2.2. Relation to impurity-based importance in trees
=====================================================

Tree-based models provide an alternative measure of feature
importances based on the mean decrease in impurity (MDI). Impurity is
quantified by the splitting criterion of the decision trees (Gini, Log
Loss or Mean Squared Error). However, this method can give high
importance to features that may not be predictive on unseen data when
the model is overfitting. Permutation-based feature importance, on the
other hand, avoids this issue, since it can be computed on unseen
data.

Furthermore, impurity-based feature importance for trees are
**strongly biased** and **favor high cardinality features** (typically
numerical features) over low cardinality features such as binary
features or categorical variables with a small number of possible

Permutation-based feature importances do not exhibit such a bias.

Additionally, the permutation feature importance may be computed
performance metric on the model predictions and can be used to analyze
any model class (not just tree-based models).

The following example highlights the limitations of impurity-based
feature importance in contrast to permutation-based feature
importance:
*sphx_glr_auto_examples_inspection_plot_permutation_importance.py*.

4.2.3. Misleading values on strongly correlated features

When two features are correlated and one of the features is permuted,
the model will still have access to the feature through its correlated
feature. This will result in a lower importance value for both
features, where they might *actually* be important.

One way to handle this is to cluster features that are correlated and
only keep one feature from each cluster. This strategy is explored in
the following example: *sphx_glr_auto_examples_inspection_plot_permut
ation_importance_multicollinear.py*.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_inspection_plot_permutation_importance.py*

* *sphx_glr_auto_examples_inspection_plot_permutation_importance_mult
  icollinear.py*

[1] L. Breiman, >>:doi:`"Random Forests" <10.1023/A:1010933404324>`<<,
    Machine Learning, 45(1), 5-32, 2001.

1.16. Probability calibration
*****************************

When performing classification you often want not only to predict the
class label, but also obtain a probability of the respective label.
This probability gives you some kind of confidence on the prediction.
Some models can give you poor estimates of the class probabilities and
some even do not support probability prediction (e.g., some instances
of "SGDClassifier"). The calibration module allows you to better
calibrate the probabilities of a given model, or to add support for
probability prediction.

Well calibrated classifiers are probabilistic classifiers for which
the output of the *predict_proba* method can be directly interpreted
as a confidence level. For instance, a well calibrated (binary)
classifier should classify the samples such that among the samples to
which it gave a *predict_proba* value close to 0.8, approximately 80%
actually belong to the positive class.

1.16.1. Calibration curves

Calibration curves (also known as reliability diagrams) compare how
well the probabilistic predictions of a binary classifier are
calibrated. It plots the true frequency of the positive label against
its predicted probability, for binned predictions. The x axis
represents the average predicted probability in each bin. The y axis
is the *fraction of positives*, i.e. the proportion of samples whose
class is the positive class (in each bin). The top calibration curve

plot is created with "CalibrationDisplay.from_estimators()", which
uses "calibration_curve()" to calculate the per bin average predicted
probabilities and fraction of positives.
"CalibrationDisplay.from_estimator()" takes as input a fitted
classifier, which is used to calculate the predicted probabilities.
The classifier thus must have *predict_proba* method. For the few
classifiers that do not have a *predict_proba* method, it is possible
to use "CalibratedClassifierCV" to calibrate the classifier outputs to

The bottom histogram gives some insight into the behavior of each
classifier by showing the number of samples in each predicted
probability bin.

   [image]

"LogisticRegression" returns well calibrated predictions by default as
it directly optimizes Log loss. In contrast, the other methods return
biased probabilities; with different biases per method:

"GaussianNB" tends to push probabilities to 0 or 1 (note the counts in
the histograms). This is mainly because it makes the assumption that
features are conditionally independent given the class, which is not
the case in this dataset which contains 2 redundant features.

"RandomForestClassifier" shows the opposite behavior: the histograms
show peaks at approximately 0.2 and 0.9 probability, while
probabilities close to 0 or 1 are very rare. An explanation for this

is given by Niculescu-Mizil and Caruana [1]: "Methods such as bagging
and random forests that average predictions from a base set of models
can have difficulty making predictions near 0 and 1 because variance
in the underlying base models will bias predictions that should be
near zero or one away from these values. Because predictions are
restricted to the interval [0,1], errors caused by variance tend to be
one-sided near zero and one. For example, if a model should predict p
= 0 for a case, the only way bagging can achieve this is if all bagged
trees predict zero. If we add noise to the trees that bagging is
averaging over, this noise will cause some trees to predict values
larger than 0 for this case, thus moving the average prediction of the
bagged ensemble away from 0. We observe this effect most strongly with
random forests because the base-level trees trained with random
forests have relatively high variance due to feature subsetting." As a
result, the calibration curve also referred to as the reliability
diagram (Wilks 1995 [2]) shows a characteristic sigmoid shape,
indicating that the classifier could trust its "intuition" more and
return probabilities closer to 0 or 1 typically.

Linear Support Vector Classification ("LinearSVC") shows an even more

sigmoid curve than "RandomForestClassifier", which is typical for
maximum-margin methods (compare Niculescu-Mizil and Caruana [1]),
which focus on difficult to classify samples that are close to the
decision boundary (the support vectors).

1.16.2. Calibrating a classifier

Calibrating a classifier consists of fitting a regressor (called a
*calibrator*) that maps the output of the classifier (as given by
*decision_function* or *predict_proba*) to a calibrated probability in
[0, 1]. Denoting the output of the classifier for a given sample by
f_i, the calibrator tries to predict p(y_i = 1 | f_i).

The samples that are used to fit the calibrator should not be the same
samples used to fit the classifier, as this would introduce bias. This
is because performance of the classifier on its training data would be
better than for novel data. Using the classifier output of training
data to fit the calibrator would thus result in a biased calibrator
that maps to probabilities closer to 0 and 1 than it should.

1.16.3. Usage

The "CalibratedClassifierCV" class is used to calibrate a classifier.

"CalibratedClassifierCV" uses a cross-validation approach to ensure
unbiased data is always used to fit the calibrator. The data is split

into k *(train_set, test_set)* couples (as determined by *cv*). When
*ensemble=True* (default), the following procedure is repeated
independently for each cross-validation split: a clone of
*base_estimator* is first trained on the train subset. Then its
predictions on the test subset are used to fit a calibrator (either a
sigmoid or isotonic regressor). This results in an ensemble of k
*(classifier, calibrator)* couples where each calibrator maps the
output of its corresponding classifier into [0, 1]. Each couple is
exposed in the *calibrated_classifiers_* attribute, where each entry
is a calibrated classifier with a *predict_proba* method that outputs
calibrated probabilities. The output of *predict_proba* for the main
"CalibratedClassifierCV" instance corresponds to the average of the
predicted probabilities of the *k* estimators in the
*calibrated_classifiers_* list. The output of *predict* is the class
that has the highest probability.

When *ensemble=False*, cross-validation is used to obtain 'unbiased'
predictions for all the data, via "cross_val_predict()". These
unbiased predictions are then used to train the calibrator. The
attribute *calibrated_classifiers_* consists of only one *(classifier,
calibrator)* couple where the classifier is the *base_estimator*
trained on all the data. In this case the output of *predict_proba*

for "CalibratedClassifierCV" is the predicted probabilities obtained
from the single *(classifier, calibrator)* couple.

The main advantage of *ensemble=True* is to benefit from the
traditional ensembling effect (similar to Bagging meta-estimator). The
resulting ensemble should both be well calibrated and slightly more
accurate than with *ensemble=False*. The main advantage of using
*ensemble=False* is computational: it reduces the overall fit time by
training only a single base classifier and calibrator pair, decreases
the final model size and increases prediction speed.

Alternatively an already fitted classifier can be calibrated by
setting *cv="prefit"*. In this case, the data is not split and all of
it is used to fit the regressor. It is up to the user to make sure
that the data used for fitting the classifier is disjoint from the
data used for fitting the regressor.

"sklearn.metrics.brier_score_loss()" may be used to assess how well a
classifier is calibrated. However, this metric should be used with
care because a lower Brier score does not always mean a better
calibrated model. This is because the Brier score metric is a
combination of calibration loss and refinement loss. Calibration loss
is defined as the mean squared deviation from empirical probabilities
derived from the slope of ROC segments. Refinement loss can be defined

as the expected optimal loss as measured by the area under the optimal
cost curve. As refinement loss can change independently from
calibration loss, a lower Brier score does not necessarily mean a
better calibrated model.

"CalibratedClassifierCV" supports the use of two 'calibration'
regressors: 'sigmoid' and 'isotonic'.

1.16.3.1. Sigmoid

The sigmoid regressor is based on Platt's logistic model [3]:

   p(y_i = 1 | f_i) = \frac{1}{1 + \exp(A f_i + B)}

where y_i is the true label of sample i and f_i is the output of the
un-calibrated classifier for sample i. A and B are real numbers to be
determined when fitting the regressor via maximum likelihood.

The sigmoid method assumes the calibration curve can be corrected by
applying a sigmoid function to the raw predictions. This assumption
has been empirically justified in the case of Support Vector Machines
with common kernel functions on various benchmark datasets in section
2.1 of Platt 1999 [3] but does not necessarily hold in general.
Additionally, the logistic model works best if the calibration error
is symmetrical, meaning the classifier output for each binary class is
normally distributed with the same variance [6]. This can be a problem
for highly imbalanced classification problems, where outputs do not
have equal variance.

In general this method is most effective when the un-calibrated model
is under-confident and has similar calibration errors for both high
and low outputs.

1.16.3.2. Isotonic

The 'isotonic' method fits a non-parametric isotonic regressor, which
outputs a step-wise non-decreasing function (see "sklearn.isotonic").
It minimizes:

   \sum_{i=1}^{n} (y_i - \hat{f}_i)^2

subject to \hat{f}_i >= \hat{f}_j whenever f_i >= f_j. y_i is the true
label of sample i and \hat{f}_i is the output of the calibrated
classifier for sample i (i.e., the calibrated probability). This
method is more general when compared to 'sigmoid' as the only
restriction is that the mapping function is monotonically increasing.
It is thus more powerful as it can correct any monotonic distortion of
the un-calibrated model. However, it is more prone to overfitting,
especially on small datasets [5].

Overall, 'isotonic' will perform as well as or better than 'sigmoid'
when there is enough data (greater than ~ 1000 samples) to avoid
overfitting [1].

1.16.3.3. Multiclass support

Both isotonic and sigmoid regressors only support 1-dimensional data
(e.g., binary classification output) but are extended for multiclass
classification if the *base_estimator* supports multiclass
predictions. For multiclass predictions, "CalibratedClassifierCV"

calibrates for each class separately in a OneVsRestClassifier fashion
[4]. When predicting probabilities, the calibrated probabilities for
each class are predicted separately. As those probabilities do not
necessarily sum to one, a postprocessing is performed to normalize
them.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_calibration_plot_calibration_curve.py*

* *sphx_glr_auto_examples_calibration_plot_calibration_multiclass.py*

* *sphx_glr_auto_examples_calibration_plot_calibration.py*

[1] Predicting Good Probabilities with Supervised Learning, A.
    Niculescu-Mizil & R. Caruana, ICML 2005

[2] On the combination of forecast probabilities for consecutive
    precipitation periods. Wea. Forecasting, 5, 640–650., Wilks, D.
    S., 1990a

[3] Probabilistic Outputs for Support Vector Machines and Comparisons
    to Regularized Likelihood Methods. J. Platt, (1999)

[4] Transforming Classifier Scores into Accurate Multiclass
    Probability Estimates. B. Zadrozny & C. Elkan, (KDD 2002)

[5] Predicting accurate probabilities with a ranking loss. Menon AK,
    Jiang XJ, Vembu S, Elkan C, Ohno-Machado L. Proc Int Conf Mach
    Learn. 2012;2012:703-710

[6] Beyond sigmoids: How to obtain well-calibrated probabilities from

Jiang XJ, Vembu S, Elkan C, Ohno-Machado L. Proc Int Conf Mach

    binary classifiers with beta calibration Kull, M., Silva Filho, T.
    M., & Flach, P. (2017).

6.3. Preprocessing data

The "sklearn.preprocessing" package provides several common utility
functions and transformer classes to change raw feature vectors into a
representation that is more suitable for the downstream estimators.

In general, learning algorithms benefit from standardization of the
data set. If some outliers are present in the set, robust scalers or
transformers are more appropriate. The behaviors of the different
scalers, transformers, and normalizers on a dataset containing
marginal outliers is highlighted in
*sphx_glr_auto_examples_preprocessing_plot_all_scaling.py*.

6.3.1. Standardization, or mean removal and variance scaling
============================================================

**Standardization** of datasets is a **common requirement for many
machine learning estimators** implemented in scikit-learn; they might
behave badly if the individual features do not more or less look like
standard normally distributed data: Gaussian with **zero mean and unit
variance**.

In practice we often ignore the shape of the distribution and just
transform the data to center it by removing the mean value of each
feature, then scale it by dividing non-constant features by their
standard deviation.

For instance, many elements used in the objective function of a
learning algorithm (such as the RBF kernel of Support Vector Machines
or the l1 and l2 regularizers of linear models) may assume that all
features are centered around zero or have variance in the same order.

If a feature has a variance that is orders of magnitude larger than
others, it might dominate the objective function and make the
estimator unable to learn from other features correctly as expected.

The "preprocessing" module provides the "StandardScaler" utility
class, which is a quick and easy way to perform the following
operation on an array-like dataset:

   >>> X_train = np.array([[ 1., -1.,  2.],
   ...                     [ 2.,  0.,  0.],
   ...                     [ 0.,  1., -1.]])
   >>> scaler
   StandardScaler()

   >>> scaler.mean_
   array([1. ..., 0. ..., 0.33...])

   >>> scaler.scale_
   array([0.81..., 0.81..., 1.24...])

   >>> X_scaled = scaler.transform(X_train)
   >>> X_scaled
   array([[ 0.  ..., -1.22...,  1.33...],
          [ 1.22...,  0.  ..., -0.26...],
          [-1.22...,  1.22..., -1.06...]])

Scaled data has zero mean and unit variance:

   >>> X_scaled.mean(axis=0)
   array([0., 0., 0.])

   >>> X_scaled.std(axis=0)
   array([1., 1., 1.])

This class implements the "Transformer" API to compute the mean and
standard deviation on a training set so as to be able to later re-
apply the same transformation on the testing set. This class is hence
suitable for use in the early steps of a "Pipeline":

   >>> X, y = make_classification(random_state=42)
   >>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
   >>> pipe = make_pipeline(StandardScaler(), LogisticRegression())
   >>> pipe.fit(X_train, y_train)  # apply scaling on training data

   >>> pipe.score(X_test, y_test)  # apply scaling on testing data, without leaking training data.
   0.96

It is possible to disable either centering or scaling by either
passing "with_mean=False" or "with_std=False" to the constructor of
"StandardScaler".

6.3.1.1. Scaling features to a range

An alternative standardization is scaling features to lie between a
given minimum and maximum value, often between zero and one, or so
that the maximum absolute value of each feature is scaled to unit
size. This can be achieved using "MinMaxScaler" or "MaxAbsScaler",

The motivation to use this scaling include robustness to very small
standard deviations of features and preserving zero entries in sparse
data.

Here is an example to scale a toy data matrix to the "[0, 1]" range:

   ...
   >>> min_max_scaler = preprocessing.MinMaxScaler()
   >>> X_train_minmax = min_max_scaler.fit_transform(X_train)
   >>> X_train_minmax
   array([[0.5       , 0.        , 1.        ],
          [1.        , 0.5       , 0.33333333],
          [0.        , 1.        , 0.        ]])

The same instance of the transformer can then be applied to some new
test data unseen during the fit call: the same scaling and shifting
operations will be applied to be consistent with the transformation
performed on the train data:

   >>> X_test = np.array([[-3., -1.,  4.]])
   >>> X_test_minmax = min_max_scaler.transform(X_test)
   >>> X_test_minmax
   array([[-1.5       ,  0.        ,  1.66666667]])

It is possible to introspect the scaler attributes to find about the
exact nature of the transformation learned on the training data:

   >>> min_max_scaler.scale_
   array([0.5       , 0.5       , 0.33...])

   >>> min_max_scaler.min_
   array([0.        , 0.5       , 0.33...])

>>> min_max_scaler.scale_

If "MinMaxScaler" is given an explicit "feature_range=(min, max)" the
full formula is:

   X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))

   X_scaled = X_std * (max - min) + min

"MaxAbsScaler" works in a very similar fashion, but scales in a way
that the training data lies within the range "[-1, 1]" by dividing
through the largest maximum value in each feature. It is meant for
data that is already centered at zero or sparse data.

Here is how to use the toy data from the previous example with this
scaler:

   ...
   >>> max_abs_scaler = preprocessing.MaxAbsScaler()
   >>> X_train_maxabs = max_abs_scaler.fit_transform(X_train)
   >>> X_train_maxabs
   array([[ 0.5, -1. ,  1. ],
          [ 1. ,  0. ,  0. ],
          [ 0. ,  1. , -0.5]])
   >>> X_test = np.array([[ -3., -1.,  4.]])
   >>> X_test_maxabs = max_abs_scaler.transform(X_test)
   >>> X_test_maxabs
   array([[-1.5, -1. ,  2. ]])
   >>> max_abs_scaler.scale_
   array([2.,  1.,  2.])

6.3.1.2. Scaling sparse data

Centering sparse data would destroy the sparseness structure in the
data, and thus rarely is a sensible thing to do. However, it can make
sense to scale sparse inputs, especially if features are on different
scales.

scales.

"MaxAbsScaler" was specifically designed for scaling sparse data, and
is the recommended way to go about this. However, "StandardScaler" can
accept "scipy.sparse" matrices  as input, as long as "with_mean=False"
is explicitly passed to the constructor. Otherwise a "ValueError" will
be raised as silently centering would break the sparsity and would
often crash the execution by allocating excessive amounts of memory
unintentionally. "RobustScaler" cannot be fitted to sparse inputs, but
you can use the "transform" method on sparse inputs.

Note that the scalers accept both Compressed Sparse Rows and
Compressed Sparse Columns format (see "scipy.sparse.csr_matrix" and
"scipy.sparse.csc_matrix"). Any other sparse input will be **converted
to the Compressed Sparse Rows representation**.  To avoid unnecessary
memory copies, it is recommended to choose the CSR or CSC
representation upstream.

Finally, if the centered data is expected to be small enough,
explicitly converting the input to an array using the "toarray" method
of sparse matrices is another option.

6.3.1.3. Scaling data with outliers

If your data contains many outliers, scaling using the mean and
variance of the data is likely to not work very well. In these cases,
you can use "RobustScaler" as a drop-in replacement instead. It uses

more robust estimates for the center and range of your data.

Further discussion on the importance of centering and scaling data is
available on this FAQ: Should I normalize/standardize/rescale the
data?

Scaling vs Whitening

It is sometimes not enough to center and scale the features
independently, since a downstream model can further make some
assumption on the linear independence of the features.

To address this issue you can use "PCA" with "whiten=True" to further
remove the linear correlation across features.

6.3.1.4. Centering kernel matrices

If you have a kernel matrix of a kernel K that computes a dot product
in a feature space (possibly implicitly) defined by a function
\phi(\cdot), a "KernelCenterer" can transform the kernel matrix so
that it contains inner products in the feature space defined by \phi
followed by the removal of the mean in that space. In other words,
"KernelCenterer" computes the centered Gram matrix associated to a
positive semidefinite kernel K.

**Mathematical formulation**

We can have a look at the mathematical formulation now that we have
the intuition. Let K be a kernel matrix of shape *(n_samples,
n_samples)* computed from X, a data matrix of shape *(n_samples,

n_features)*, during the *fit* step. K is defined by

   K(X, X) = \phi(X) . \phi(X)^{T}

\phi(X) is a function mapping of X to a Hilbert space. A centered
kernel \tilde{K} is defined as:

   \tilde{K}(X, X) = \tilde{\phi}(X) . \tilde{\phi}(X)^{T}

where \tilde{\phi}(X) results from centering \phi(X) in the Hilbert
space.

Thus, one could compute \tilde{K} by mapping X using the function
\phi(\cdot) and center the data in this new space. However, kernels
are often used because they allows some algebra calculations that
avoid computing explicitly this mapping using \phi(\cdot). Indeed, one
can implicitly center as shown in Appendix B in [Scholkopf1998]:

   \tilde{K} = K - 1_{\text{n}_{samples}} K - K 1_{\text{n}_{samples}}
   + 1_{\text{n}_{samples}} K 1_{\text{n}_{samples}}

1_{\text{n}_{samples}} is a matrix of *(n_samples, n_samples)* where
all entries are equal to \frac{1}{\text{n}_{samples}}. In the
*transform* step, the kernel becomes K_{test}(X, Y) defined as:

   K_{test}(X, Y) = \phi(Y) . \phi(X)^{T}

Y is the test dataset of shape *(n_samples_test, n_features)* and thus
K_{test} is of shape *(n_samples_test, n_samples)*. In this case,
centering K_{test} is done as:

   \tilde{K}_{test}(X, Y) = K_{test} - 1'_{\text{n}_{samples}} K -
   K_{test} 1_{\text{n}_{samples}} + 1'_{\text{n}_{samples}} K

   1_{\text{n}_{samples}}

1'_{\text{n}_{samples}} is a matrix of shape *(n_samples_test,
n_samples)* where all entries are equal to
\frac{1}{\text{n}_{samples}}.

[Scholkopf1998] B. Schölkopf, A. Smola, and K.R. Müller, "Nonlinear
                component analysis as a kernel eigenvalue problem."
                Neural computation 10.5 (1998): 1299-1319.

6.3.2. Non-linear transformation

Two types of transformations are available: quantile transforms and
power transforms. Both quantile and power transforms are based on
monotonic transformations of the features and thus preserve the rank
of the values along each feature.

Quantile transforms put all features into the same desired
distribution based on the formula G^{-1}(F(X)) where F is the
cumulative distribution function of the feature and G^{-1} the
quantile function of the desired output distribution G. This formula
is using the two following facts: (i) if X is a random variable with a
continuous cumulative distribution function F then F(X) is uniformly
distributed on [0,1]; (ii) if U is a random variable with uniform
distribution on [0,1] then G^{-1}(U) has distribution G. By performing
a rank transformation, a quantile transform smooths out unusual
distributions and is less influenced by outliers than scaling methods.

It does, however, distort correlations and distances within and across
features.

Power transforms are a family of parametric transformations that aim
to map data from any distribution to as close to a Gaussian

6.3.2.1. Mapping to a Uniform distribution

"QuantileTransformer" provides a non-parametric transformation to map
the data to a uniform distribution with values between 0 and 1:

   >>> quantile_transformer = preprocessing.QuantileTransformer(random_state=0)
   >>> X_train_trans = quantile_transformer.fit_transform(X_train)
   >>> X_test_trans = quantile_transformer.transform(X_test)
   >>> np.percentile(X_train[:, 0], [0, 25, 50, 75, 100]) 
   array([ 4.3,  5.1,  5.8,  6.5,  7.9])

This feature corresponds to the sepal length in cm. Once the quantile
transformation applied, those landmarks approach closely the
percentiles previously defined:

   >>> np.percentile(X_train_trans[:, 0], [0, 25, 50, 75, 100])
   ... 
   array([ 0.00... ,  0.24...,  0.49...,  0.73...,  0.99... ])

This can be confirmed on a independent testing set with similar
remarks:

... 

remarks:

   >>> np.percentile(X_test[:, 0], [0, 25, 50, 75, 100])
   ... 
   array([ 4.4  ,  5.125,  5.75 ,  6.175,  7.3  ])
   >>> np.percentile(X_test_trans[:, 0], [0, 25, 50, 75, 100])
   ... 
   array([ 0.01...,  0.25...,  0.46...,  0.60... ,  0.94...])

6.3.2.2. Mapping to a Gaussian distribution

In many modeling scenarios, normality of the features in a dataset is
desirable. Power transforms are a family of parametric, monotonic
transformations that aim to map data from any distribution to as close
to a Gaussian distribution as possible in order to stabilize variance
and minimize skewness.

"PowerTransformer" currently provides two such power transformations,
the Yeo-Johnson transform and the Box-Cox transform.

The Yeo-Johnson transform is given by:

   x_i^{(\lambda)} = \begin{cases}  [(x_i + 1)^\lambda - 1] / \lambda
   & \text{if } \lambda \neq 0, x_i \geq 0, \\[8pt] \ln{(x_i + 1)} &
   \text{if } \lambda = 0, x_i \geq 0 \\[8pt] -[(-x_i + 1)^{2 -
   \lambda} - 1] / (2 - \lambda) & \text{if } \lambda \neq 2, x_i < 0,
   \\[8pt]  - \ln (- x_i + 1) & \text{if } \lambda = 2, x_i < 0

while the Box-Cox transform is given by:

   x_i^{(\lambda)} = \begin{cases} \dfrac{x_i^\lambda - 1}{\lambda} &
   \text{if } \lambda \neq 0, \\[8pt] \ln{(x_i)} & \text{if } \lambda
   = 0, \end{cases}

Box-Cox can only be applied to strictly positive data. In both
methods, the transformation is parameterized by \lambda, which is
determined through maximum likelihood estimation. Here is an example
of using Box-Cox to map samples drawn from a lognormal distribution to
a normal distribution:

   >>> pt = preprocessing.PowerTransformer(method='box-cox', standardize=False)
   >>> X_lognormal = np.random.RandomState(616).lognormal(size=(3, 3))
   >>> X_lognormal
   array([[1.28..., 1.18..., 0.84...],
          [0.94..., 1.60..., 0.38...],
          [1.35..., 0.21..., 1.09...]])
   >>> pt.fit_transform(X_lognormal)
   array([[ 0.49...,  0.17..., -0.15...],
          [-0.05...,  0.58..., -0.57...],
          [ 0.69..., -0.84...,  0.10...]])

While the above example sets the *standardize* option to *False*,
"PowerTransformer" will apply zero-mean, unit-variance normalization
to the transformed output by default.

Below are examples of Box-Cox and Yeo-Johnson applied to various
probability distributions.  Note that when applied to certain
distributions, the power transforms achieve very Gaussian-like
results, but with others, they are ineffective. This highlights the
importance of visualizing the data before and after transformation.

   [image]

It is also possible to map data to a normal distribution using

   [image]

"QuantileTransformer" by setting "output_distribution='normal'". Using
the earlier example with the iris dataset:

   >>> quantile_transformer = preprocessing.QuantileTransformer(
   ...     output_distribution='normal', random_state=0)
   >>> X_trans = quantile_transformer.fit_transform(X)
   >>> quantile_transformer.quantiles_
   array([[4.3, 2. , 1. , 0.1],
          [4.4, 2.2, 1.1, 0.1],
          [4.4, 2.2, 1.2, 0.1],
          ...,
          [7.7, 4.1, 6.7, 2.5],
          [7.7, 4.2, 6.7, 2.5],
          [7.9, 4.4, 6.9, 2.5]])

Thus the median of the input becomes the mean of the output, centered
at 0. The normal output is clipped so that the input's minimum and
maximum --- corresponding to the 1e-7 and 1 - 1e-7 quantiles
respectively --- do not become infinite under the transformation.

6.3.3. Normalization

**Normalization** is the process of **scaling individual samples to
have unit norm**. This process can be useful if you plan to use a
quadratic form such as the dot-product or any other kernel to quantify
the similarity of any pair of samples.

This assumption is the base of the Vector Space Model often used in
text classification and clustering contexts.

The function "normalize()" provides a quick and easy way to perform
this operation on a single array-like dataset, either using the "l1",
"l2", or "max" norms:

   >>> X = [[ 1., -1.,  2.],
   ...      [ 2.,  0.,  0.],
   ...      [ 0.,  1., -1.]]
   >>> X_normalized = preprocessing.normalize(X, norm='l2')

   >>> X_normalized
   array([[ 0.40..., -0.40...,  0.81...],
          [ 1.  ...,  0.  ...,  0.  ...],
          [ 0.  ...,  0.70..., -0.70...]])

The "preprocessing" module further provides a utility class
"Normalizer" that implements the same operation using the
"Transformer" API (even though the "fit" method is useless in this
case: the class is stateless as this operation treats samples
independently).

This class is hence suitable for use in the early steps of a
"Pipeline":

   >>> normalizer = preprocessing.Normalizer().fit(X)  # fit does nothing
   >>> normalizer
   Normalizer()

The normalizer instance can then be used on sample vectors as any
transformer:

   >>> normalizer.transform(X)

   >>> normalizer.transform([[-1.,  1., 0.]])
   array([[-0.70...,  0.70...,  0.  ...]])

Note: L2 normalization is also known as spatial sign preprocessing.

Sparse input

"normalize()" and "Normalizer" accept **both dense array-like and
sparse matrices from scipy.sparse as input**.

For sparse input the data is **converted to the Compressed Sparse Rows

representation** (see "scipy.sparse.csr_matrix") before being fed to
efficient Cython routines. To avoid unnecessary memory copies, it is
recommended to choose the CSR representation upstream.

6.3.4. Encoding categorical features

Often features are not given as continuous values but categorical. For
example a person could have features "["male", "female"]", "["from
Europe", "from US", "from Asia"]", "["uses Firefox", "uses Chrome",
"uses Safari", "uses Internet Explorer"]". Such features can be
efficiently coded as integers, for instance "["male", "from US", "uses
Internet Explorer"]" could be expressed as "[0, 1, 3]" while
"["female", "from Asia", "uses Chrome"]" would be "[1, 2, 1]".

To convert categorical features to such integer codes, we can use the
"OrdinalEncoder". This estimator transforms each categorical feature
to one new feature of integers (0 to n_categories - 1):

   >>> enc = preprocessing.OrdinalEncoder()
   >>> X = [['male', 'from US', 'uses Safari'], ['female', 'from Europe', 'uses Firefox']]
   >>> enc.fit(X)
   OrdinalEncoder()
   >>> enc.transform([['female', 'from US', 'uses Safari']])
   array([[0., 1., 1.]])

Such integer representation can, however, not be used directly with
all scikit-learn estimators, as these expect continuous input, and

array([[0., 1., 1.]])

would interpret the categories as being ordered, which is often not
desired (i.e. the set of browsers was ordered arbitrarily).

By default, "OrdinalEncoder" will also passthrough missing values that
are indicated by *np.nan*.

>>> enc = preprocessing.OrdinalEncoder()
>>> X = [['male'], ['female'], [np.nan], ['female']]
>>> enc.fit_transform(X)
array([[ 1.],
       [ 0.],
       [nan],
       [ 0.]])

"OrdinalEncoder" provides a parameter *encoded_missing_value* to
encode the missing values without the need to create a pipeline and
using "SimpleImputer".

>>> enc = preprocessing.OrdinalEncoder(encoded_missing_value=-1)
       [ 0.],
       [-1.],
       [ 0.]])

The above processing is equivalent to the following pipeline:

   >>> enc = Pipeline(steps=[
   ...     ("encoder", preprocessing.OrdinalEncoder()),
   ...     ("imputer", SimpleImputer(strategy="constant", fill_value=-1)),
   ... ])
   >>> enc.fit_transform(X)
   array([[ 1.],
          [ 0.],
          [-1.],
          [ 0.]])

Another possibility to convert categorical features to features that
can be used with scikit-learn estimators is to use a one-of-K, also

          [ 0.],
          [-1.],
          [ 0.]])

known as one-hot or dummy encoding. This type of encoding can be
obtained with the "OneHotEncoder", which transforms each categorical
feature with "n_categories" possible values into "n_categories" binary
features, with one of them 1, and all others 0.

Continuing the example above:

   >>> enc = preprocessing.OneHotEncoder()
   OneHotEncoder()
   >>> enc.transform([['female', 'from US', 'uses Safari'],
   ...                ['male', 'from Europe', 'uses Safari']]).toarray()
   array([[1., 0., 0., 1., 0., 1.],
          [0., 1., 1., 0., 0., 1.]])

By default, the values each feature can take is inferred automatically
from the dataset and can be found in the "categories_" attribute:

   >>> enc.categories_
   [array(['female', 'male'], dtype=object), array(['from Europe', 'from US'], dtype=object), array(['uses Firefox', 'uses Safari'], dtype=object)]

It is possible to specify this explicitly using the parameter
"categories". There are two genders, four possible continents and four
web browsers in our dataset:

   >>> genders = ['female', 'male']
   >>> locations = ['from Africa', 'from Asia', 'from Europe', 'from US']
   >>> browsers = ['uses Chrome', 'uses Firefox', 'uses IE', 'uses Safari']

>>> genders = ['female', 'male']
   >>> enc = preprocessing.OneHotEncoder(categories=[genders, locations, browsers])
   >>> # Note that for there are missing categorical values for the 2nd and 3rd
   >>> # feature
   OneHotEncoder(categories=[['female', 'male'],
                             ['from Africa', 'from Asia', 'from Europe',
                              'from US'],
                             ['uses Chrome', 'uses Firefox', 'uses IE',
                              'uses Safari']])
   >>> enc.transform([['female', 'from Asia', 'uses Chrome']]).toarray()
   array([[1., 0., 0., 1., 0., 0., 1., 0., 0., 0.]])

If there is a possibility that the training data might have missing
categorical features, it can often be better to specify
*handle_unknown='infrequent_if_exist'* instead of setting the
*categories* manually as above. When
*handle_unknown='infrequent_if_exist'* is specified and unknown
categories are encountered during transform, no error will be raised
but the resulting one-hot encoded columns for this feature will be all
zeros or considered as an infrequent category if enabled.
(*handle_unknown='infrequent_if_exist'* is only supported for one-hot
encoding):

   >>> enc = preprocessing.OneHotEncoder(handle_unknown='infrequent_if_exist')
   OneHotEncoder(handle_unknown='infrequent_if_exist')
   array([[1., 0., 0., 0., 0., 0.]])

It is also possible to encode each column into "n_categories - 1"
columns instead of "n_categories" columns by using the "drop"
parameter. This parameter allows the user to specify a category for
each feature to be dropped. This is useful to avoid co-linearity in
the input matrix in some classifiers. Such functionality is useful,
for example, when using non-regularized regression
("LinearRegression"), since co-linearity would cause the covariance
matrix to be non-invertible:

   >>> X = [['male', 'from US', 'uses Safari'],
   ...      ['female', 'from Europe', 'uses Firefox']]
   >>> drop_enc = preprocessing.OneHotEncoder(drop='first').fit(X)
   >>> drop_enc.categories_
   [array(['female', 'male'], dtype=object), array(['from Europe', 'from US'], dtype=object),
    array(['uses Firefox', 'uses Safari'], dtype=object)]
   >>> drop_enc.transform(X).toarray()
   array([[1., 1., 1.],
          [0., 0., 0.]])

One might want to drop one of the two columns only for features with 2

>>> drop_enc.transform(X).toarray()

categories. In this case, you can set the parameter
*drop='if_binary'*.

>>> X = [['male', 'US', 'Safari'],
...      ['female', 'Europe', 'Firefox'],
...      ['female', 'Asia', 'Chrome']]
>>> drop_enc = preprocessing.OneHotEncoder(drop='if_binary').fit(X)
>>> drop_enc.categories_
[array(['female', 'male'], dtype=object), array(['Asia', 'Europe', 'US'], dtype=object),
 array(['Chrome', 'Firefox', 'Safari'], dtype=object)]
array([[1., 0., 0., 1., 0., 0., 1.],
       [0., 0., 1., 0., 0., 1., 0.],
       [0., 1., 0., 0., 1., 0., 0.]])

In the transformed *X*, the first column is the encoding of the
feature with categories "male"/"female", while the remaining 6 columns
is the encoding of the 2 features with respectively 3 categories each.

When *handle_unknown='ignore'* and *drop* is not None, unknown
categories will be encoded as all zeros:

   >>> drop_enc = preprocessing.OneHotEncoder(drop='first',
   ...                                        handle_unknown='ignore').fit(X)
   >>> X_test = [['unknown', 'America', 'IE']]
   >>> drop_enc.transform(X_test).toarray()
   array([[0., 0., 0., 0., 0.]])

All the categories in *X_test* are unknown during transform and will
be mapped to all zeros. This means that unknown categories will have
the same mapping as the dropped category.

"OneHotEncoder.inverse_transform()" will map all zeros to the dropped
category if a category is dropped and *None* if a category is not
dropped:

   >>> drop_enc = preprocessing.OneHotEncoder(drop='if_binary', sparse_output=False,
   >>> X_trans = drop_enc.transform(X_test)
   >>> X_trans
   array([[0., 0., 0., 0., 0., 0., 0.]])
   >>> drop_enc.inverse_transform(X_trans)
   array([['female', None, None]], dtype=object)

"OneHotEncoder" supports categorical features with missing values by
considering the missing values as an additional category:

   >>> X = [['male', 'Safari'],
   ...      ['female', None],
   ...      [np.nan, 'Firefox']]
   >>> enc = preprocessing.OneHotEncoder(handle_unknown='error').fit(X)
   [array(['female', 'male', nan], dtype=object),
    array(['Firefox', 'Safari', None], dtype=object)]
   >>> enc.transform(X).toarray()
   array([[0., 1., 0., 0., 1., 0.],
          [1., 0., 0., 0., 0., 1.],
          [0., 0., 1., 1., 0., 0.]])

If a feature contains both *np.nan* and *None*, they will be
considered separate categories:

   >>> X = [['Safari'], [None], [np.nan], ['Firefox']]

   [array(['Firefox', 'Safari', None, nan], dtype=object)]
   array([[0., 1., 0., 0.],
          [0., 0., 1., 0.],
          [0., 0., 0., 1.],
          [1., 0., 0., 0.]])

See Loading features from dicts for categorical features that are
represented as a dict, not as scalars.

6.3.4.1. Infrequent categories

"OneHotEncoder" supports aggregating infrequent categories into a
single output for each feature. The parameters to enable the gathering
of infrequent categories are *min_frequency* and *max_categories*.

1. *min_frequency* is either an  integer greater or equal to 1, or a
   float in the interval *(0.0, 1.0)*. If *min_frequency* is an
   integer, categories with a cardinality smaller than *min_frequency*
   will be considered infrequent. If *min_frequency* is a float,
   categories with a cardinality smaller than this fraction of the
   total number of samples will be considered infrequent. The default
   value is 1, which means every category is encoded separately.

2. *max_categories* is either *None* or any integer greater than 1.
   This parameter sets an upper limit to the number of output features
   for each input feature. *max_categories* includes the feature that
   combines infrequent categories.

This parameter sets an upper limit to the number of output features

In the following example, the categories, *'dog', 'snake'* are
considered infrequent:

   >>> X = np.array([['dog'] * 5 + ['cat'] * 20 + ['rabbit'] * 10 +
   ...               ['snake'] * 3], dtype=object).T
   >>> enc = preprocessing.OneHotEncoder(min_frequency=6, sparse_output=False).fit(X)
   >>> enc.infrequent_categories_
   [array(['dog', 'snake'], dtype=object)]
   >>> enc.transform(np.array([['dog'], ['cat'], ['rabbit'], ['snake']]))
   array([[0., 0., 1.],
          [1., 0., 0.],
          [0., 1., 0.],
          [0., 0., 1.]])

By setting handle_unknown to *'infrequent_if_exist'*, unknown
categories will be considered infrequent:

   >>> enc = preprocessing.OneHotEncoder(
   ...    handle_unknown='infrequent_if_exist', sparse_output=False, min_frequency=6)
   >>> enc = enc.fit(X)
   >>> enc.transform(np.array([['dragon']]))
   array([[0., 0., 1.]])

"OneHotEncoder.get_feature_names_out()" uses 'infrequent' as the
infrequent feature name:

   >>> enc.get_feature_names_out()
   array(['x0_cat', 'x0_rabbit', 'x0_infrequent_sklearn'], dtype=object)

When *'handle_unknown'* is set to *'infrequent_if_exist'* and an
unknown category is encountered in transform:

1. If infrequent category support was not configured or there was no
   infrequent category during training, the resulting one-hot encoded

   columns for this feature will be all zeros. In the inverse
   transform, an unknown category will be denoted as *None*.

2. If there is an infrequent category during training, the unknown
   category will be considered infrequent. In the inverse transform,
   'infrequent_sklearn' will be used to represent the infrequent
   category.

Infrequent categories can also be configured using *max_categories*.
In the following example, we set *max_categories=2* to limit the
number of features in the output. This will result in all but the
*'cat'* category to be considered infrequent, leading to two features,
one for *'cat'* and one for infrequent categories - which are all the
others:

   >>> enc = preprocessing.OneHotEncoder(max_categories=2, sparse_output=False)
   >>> enc.transform([['dog'], ['cat'], ['rabbit'], ['snake']])
          [1., 0.],
          [0., 1.],
          [0., 1.]])

If both *max_categories* and *min_frequency* are non-default values,
then categories are selected based on *min_frequency* first and
*max_categories* categories are kept. In the following example,
*min_frequency=4* considers only *snake* to be infrequent, but
*max_categories=3*, forces *dog* to also be infrequent:

   >>> enc = preprocessing.OneHotEncoder(min_frequency=4, max_categories=3, sparse_output=False)

If there are infrequent categories with the same cardinality at the
cutoff of *max_categories*, then then the first *max_categories* are
taken based on lexicon ordering. In the following example, "b", "c",
and "d", have the same cardinality and with *max_categories=2*, "b"
and "c" are infrequent because they have a higher lexicon order.

>>> X = np.asarray([["a"] * 20 + ["b"] * 10 + ["c"] * 10 + ["d"] * 10], dtype=object).T
>>> enc = preprocessing.OneHotEncoder(max_categories=3).fit(X)
>>> enc.infrequent_categories_
[array(['b', 'c'], dtype=object)]

6.3.5. Discretization

Discretization (otherwise known as quantization or binning) provides a
way to partition continuous features into discrete values. Certain
datasets with continuous features may benefit from discretization,
because discretization can transform the dataset of continuous
attributes to one with only nominal attributes.

One-hot encoded discretized features can make a model more expressive,
while maintaining interpretability. For instance, pre-processing with

a discretizer can introduce nonlinearity to linear models. For more
advanced possibilities, in particular smooth ones, see Generating
polynomial features further below.

6.3.5.1. K-bins discretization

"KBinsDiscretizer" discretizes features into "k" bins:

   >>> X = np.array([[ -3., 5., 15 ],
   ...               [  0., 6., 14 ],
   ...               [  6., 3., 11 ]])
   >>> est = preprocessing.KBinsDiscretizer(n_bins=[3, 2, 2], encode='ordinal').fit(X)

By default the output is one-hot encoded into a sparse matrix (See
Encoding categorical features) and this can be configured with the
"encode" parameter. For each feature, the bin edges are computed
during "fit" and together with the number of bins, they will define
the intervals. Therefore, for the current example, these intervals are
defined as:

   * feature 1: {[-\infty, -1), [-1, 2), [2, \infty)}

   * feature 2: {[-\infty, 5), [5, \infty)}

   * feature 3: {[-\infty, 14), [14, \infty)}

Based on these bin intervals, "X" is transformed as follows:

   >>> est.transform(X)                      
   array([[ 0., 1., 1.],
          [ 1., 1., 1.],
          [ 2., 0., 0.]])

The resulting dataset contains ordinal attributes which can be further
used in a "Pipeline".

Discretization is similar to constructing histograms for continuous

[ 2., 0., 0.]])

data. However, histograms focus on counting features which fall into
particular bins, whereas discretization focuses on assigning feature
values to these bins.

"KBinsDiscretizer" implements different binning strategies, which can
be selected with the "strategy" parameter. The 'uniform' strategy uses
constant-width bins. The 'quantile' strategy uses the quantiles values
to have equally populated bins in each feature. The 'kmeans' strategy
defines bins based on a k-means clustering procedure performed on each
feature independently.

Be aware that one can specify custom bins by passing a callable
defining the discretization strategy to "FunctionTransformer". For
instance, we can use the Pandas function "pandas.cut()":

   >>> bins = [0, 1, 13, 20, 60, np.inf]
   >>> labels = ['infant', 'kid', 'teen', 'adult', 'senior citizen']
   >>> transformer = preprocessing.FunctionTransformer(
   ...     pd.cut, kw_args={'bins': bins, 'labels': labels, 'retbins': False}
   ... )
   >>> X = np.array([0.2, 2, 15, 25, 97])
   >>> transformer.fit_transform(X)
   ['infant', 'kid', 'teen', 'adult', 'senior citizen']
   Categories (5, object): ['infant' < 'kid' < 'teen' < 'adult' < 'senior citizen']

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_preprocessing_plot_discretization.py*

Categories (5, object): ['infant' < 'kid' < 'teen' < 'adult' < 'senior citizen']

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_preprocessing_plot_discretization_classific
  ation.py*

* *sphx_glr_auto_examples_preprocessing_plot_discretization_strategie
  s.py*

6.3.5.2. Feature binarization

**Feature binarization** is the process of **thresholding numerical
features to get boolean values**. This can be useful for downstream
probabilistic estimators that make assumption that the input data is
distributed according to a multi-variate Bernoulli distribution. For
instance, this is the case for the "BernoulliRBM".

It is also common among the text processing community to use binary
feature values (probably to simplify the probabilistic reasoning) even
if normalized counts (a.k.a. term frequencies) or TF-IDF valued
features often perform slightly better in practice.

As for the "Normalizer", the utility class "Binarizer" is meant to be
used in the early stages of "Pipeline". The "fit" method does nothing
as each sample is treated independently of others:

   >>> binarizer = preprocessing.Binarizer().fit(X)  # fit does nothing
   >>> binarizer
   Binarizer()

   >>> binarizer.transform(X)
   array([[1., 0., 1.],
          [0., 1., 0.]])

It is possible to adjust the threshold of the binarizer:

>>> binarizer

   >>> binarizer = preprocessing.Binarizer(threshold=1.1)

As for the "Normalizer" class, the preprocessing module provides a
companion function "binarize()" to be used when the transformer API is
not necessary.

Note that the "Binarizer" is similar to the "KBinsDiscretizer" when "k
= 2", and when the bin edge is at the value "threshold".

"binarize()" and "Binarizer" accept **both dense array-like and sparse
matrices from scipy.sparse as input**.

representation** (see "scipy.sparse.csr_matrix"). To avoid unnecessary
memory copies, it is recommended to choose the CSR representation
upstream.

6.3.6. Imputation of missing values

Tools for imputing missing values are discussed at Imputation of
missing values.

6.3.7. Generating polynomial features

Often it's useful to add complexity to a model by considering
nonlinear features of the input data. We show two possibilities that
are both based on polynomials: The first one uses pure polynomials,
the second one uses splines, i.e. piecewise polynomials.

6.3.7.1. Polynomial features

A simple and common method to use is polynomial features, which can
get features' high-order and interaction terms. It is implemented in
"PolynomialFeatures":

   >>> X
          [2, 3],
          [4, 5]])
   >>> poly = PolynomialFeatures(2)

The features of X have been transformed from (X_1, X_2) to (1, X_1,
X_2, X_1^2, X_1X_2, X_2^2).

In some cases, only interaction terms among features are required, and
it can be gotten with the setting "interaction_only=True":

   >>> X = np.arange(9).reshape(3, 3)
   >>> X
   array([[0, 1, 2],
          [3, 4, 5],
          [6, 7, 8]])
   >>> poly = PolynomialFeatures(degree=3, interaction_only=True)
   array([[  1.,   0.,   1.,   2.,   0.,   0.,   2.,   0.],
          [  1.,   3.,   4.,   5.,  12.,  15.,  20.,  60.],
          [  1.,   6.,   7.,   8.,  42.,  48.,  56., 336.]])

The features of X have been transformed from (X_1, X_2, X_3) to (1,
X_1, X_2, X_3, X_1X_2, X_1X_3, X_2X_3, X_1X_2X_3).

Note that polynomial features are used implicitly in kernel methods

(e.g., "SVC", "KernelPCA") when using polynomial Kernel functions.

See
*sphx_glr_auto_examples_linear_model_plot_polynomial_interpolation.py*
for Ridge regression using created polynomial features.

6.3.7.2. Spline transformer

Another way to add nonlinear terms instead of pure polynomials of
features is to generate spline basis functions for each feature with
the "SplineTransformer". Splines are piecewise polynomials,
parametrized by their polynomial degree and the positions of the
knots. The "SplineTransformer" implements a B-spline basis, cf. the
references below.

Note:

  The "SplineTransformer" treats each feature separately, i.e. it
  won't give you interaction terms.

Some of the advantages of splines over polynomials are:

   * B-splines are very flexible and robust if you keep a fixed low
     degree, usually 3, and parsimoniously adapt the number of knots.
     Polynomials would need a higher degree, which leads to the next
     point.

   * B-splines do not have oscillatory behaviour at the boundaries as
     have polynomials (the higher the degree, the worse). This is
     known as Runge's phenomenon.

   * B-splines provide good options for extrapolation beyond the
     boundaries, i.e. beyond the range of fitted values. Have a look
     at the option "extrapolation".

known as Runge's phenomenon.

   * B-splines generate a feature matrix with a banded structure. For
     a single feature, every row contains only "degree + 1" non-zero
     elements, which occur consecutively and are even positive. This
     results in a matrix with good numerical properties, e.g. a low
     condition number, in sharp contrast to a matrix of polynomials,
     which goes under the name Vandermonde matrix. A low condition
     number is important for stable algorithms of linear models.

The following code snippet shows splines in action:

   >>> from sklearn.preprocessing import SplineTransformer
   >>> X = np.arange(5).reshape(5, 1)
   >>> X
          [1],
          [2],
          [3],
          [4]])
   >>> spline = SplineTransformer(degree=2, n_knots=3)
   >>> spline.fit_transform(X)
   array([[0.5  , 0.5  , 0.   , 0.   ],
          [0.125, 0.75 , 0.125, 0.   ],
          [0.   , 0.5  , 0.5  , 0.   ],
          [0.   , 0.125, 0.75 , 0.125],
          [0.   , 0.   , 0.5  , 0.5  ]])

As the "X" is sorted, one can easily see the banded matrix output.
Only the three middle diagonals are non-zero for "degree=2". The
higher the degree, the more overlapping of the splines.

Interestingly, a "SplineTransformer" of "degree=0" is the same as

"KBinsDiscretizer" with "encode='onehot-dense'" and "n_bins = n_knots
- 1" if "knots = strategy".

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_linear_model_plot_polynomial_interpolation.
  py*

* *sphx_glr_auto_examples_applications_plot_cyclical_feature_engineer
  ing.py*

* Eilers, P., & Marx, B. (1996). >>:doi:`Flexible Smoothing with
  B-splines and Penalties <10.1214/ss/1038425655>`<<. Statist. Sci. 11
  (1996), no. 2, 89--121.

* Perperoglou, A., Sauerbrei, W., Abrahamowicz, M. et al. >>:doi:`A
  review of spline function procedures in R
  <10.1186/s12874-019-0666-3>`<<. BMC Med Res Methodol 19, 46 (2019).

6.3.8. Custom transformers

Often, you will want to convert an existing Python function into a
transformer to assist in data cleaning or processing. You can
implement a transformer from an arbitrary function with
"FunctionTransformer". For example, to build a transformer that
applies a log transformation in a pipeline, do:

   >>> from sklearn.preprocessing import FunctionTransformer
   >>> transformer = FunctionTransformer(np.log1p, validate=True)
   >>> X = np.array([[0, 1], [2, 3]])
   >>> # Since FunctionTransformer is no-op during fit, we can call transform directly
   >>> transformer.transform(X)
   array([[0.        , 0.69314718],

>>> X = np.array([[0, 1], [2, 3]])
          [1.09861229, 1.38629436]])

You can ensure that "func" and "inverse_func" are the inverse of each
other by setting "check_inverse=True" and calling "fit" before
"transform". Please note that a warning is raised and can be turned
into an error with a "filterwarnings":

   >>> import warnings
   >>> warnings.filterwarnings("error", message=".*check_inverse*.",
   ...                         category=UserWarning, append=False)

For a full code example that demonstrates using a
"FunctionTransformer" to extract features from text data see
*sphx_glr_auto_examples_compose_plot_column_transformer.py* and *sphx
_glr_auto_examples_applications_plot_cyclical_feature_engineering.py*.

1.6. Nearest Neighbors

"sklearn.neighbors" provides functionality for unsupervised and
supervised neighbors-based learning methods.  Unsupervised nearest
neighbors is the foundation of many other learning methods, notably
manifold learning and spectral clustering.  Supervised neighbors-based
learning comes in two flavors: classification for data with discrete
labels, and regression for data with continuous labels.

The principle behind nearest neighbor methods is to find a predefined
number of training samples closest in distance to the new point, and
predict the label from these.  The number of samples can be a user-
defined constant (k-nearest neighbor learning), or vary based on the
local density of points (radius-based neighbor learning). The distance
can, in general, be any metric measure: standard Euclidean distance is
the most common choice. Neighbors-based methods are known as *non-
generalizing* machine learning methods, since they simply "remember"
all of its training data (possibly transformed into a fast indexing
structure such as a Ball Tree or KD Tree).

Despite its simplicity, nearest neighbors has been successful in a
large number of classification and regression problems, including
handwritten digits and satellite image scenes. Being a non-parametric
method, it is often successful in classification situations where the
decision boundary is very irregular.

The classes in "sklearn.neighbors" can handle either NumPy arrays or

*scipy.sparse* matrices as input.  For dense matrices, a large number
of possible distance metrics are supported.  For sparse matrices,
arbitrary Minkowski metrics are supported for searches.

There are many learning routines which rely on nearest neighbors at
their core.  One example is kernel density estimation, discussed in
the density estimation section.

1.6.1. Unsupervised Nearest Neighbors

"NearestNeighbors" implements unsupervised nearest neighbors learning.
It acts as a uniform interface to three different nearest neighbors
algorithms: "BallTree", "KDTree", and a brute-force algorithm based on
routines in "sklearn.metrics.pairwise". The choice of neighbors search
algorithm is controlled through the keyword "'algorithm'", which must
be one of "['auto', 'ball_tree', 'kd_tree', 'brute']".  When the
default value "'auto'" is passed, the algorithm attempts to determine
the best approach from the training data.  For a discussion of the
strengths and weaknesses of each option, see Nearest Neighbor
Algorithms.

   Warning:

     Regarding the Nearest Neighbors algorithms, if two neighbors k+1
     and k have identical distances but different labels, the result
     will depend on the ordering of the training data.

1.6.1.1. Finding the Nearest Neighbors

and k have identical distances but different labels, the result

For the simple task of finding the nearest neighbors between two sets
of data, the unsupervised algorithms within "sklearn.neighbors" can be
used:

>>> from sklearn.neighbors import NearestNeighbors
>>> nbrs = NearestNeighbors(n_neighbors=2, algorithm='ball_tree').fit(X)
>>> distances, indices = nbrs.kneighbors(X)
>>> indices
array([[0, 1],
       [1, 0],
       [2, 1],
       [3, 4],
       [4, 3],
       [5, 4]]...)
>>> distances
array([[0.        , 1.        ],
       [0.        , 1.        ],
       [0.        , 1.41421356],
       [0.        , 1.        ],
       [0.        , 1.        ],
       [0.        , 1.41421356]])

Because the query set matches the training set, the nearest neighbor
of each point is the point itself, at a distance of zero.

It is also possible to efficiently produce a sparse graph showing the
connections between neighboring points:

>>> nbrs.kneighbors_graph(X).toarray()
array([[1., 1., 0., 0., 0., 0.],
       [1., 1., 0., 0., 0., 0.],
       [0., 1., 1., 0., 0., 0.],
       [0., 0., 0., 1., 1., 0.],
       [0., 0., 0., 0., 1., 1.]])

The dataset is structured such that points nearby in index order are

[0., 1., 1., 0., 0., 0.],

nearby in parameter space, leading to an approximately block-diagonal
matrix of K-nearest neighbors.  Such a sparse graph is useful in a
variety of circumstances which make use of spatial relationships
between points for unsupervised learning: in particular, see "Isomap",
"LocallyLinearEmbedding", and "SpectralClustering".

1.6.1.2. KDTree and BallTree Classes

Alternatively, one can use the "KDTree" or "BallTree" classes directly
to find nearest neighbors.  This is the functionality wrapped by the
"NearestNeighbors" class used above.  The Ball Tree and KD Tree have
the same interface; we'll show an example of using the KD Tree here:

>>> from sklearn.neighbors import KDTree
>>> kdt = KDTree(X, leaf_size=30, metric='euclidean')
>>> kdt.query(X, k=2, return_distance=False)
       [1, 0],
       [2, 1],
       [3, 4],
       [4, 3],

Refer to the "KDTree" and "BallTree" class documentation for more
information on the options available for nearest neighbors searches,
including specification of query strategies, distance metrics, etc.
For a list of available metrics, see the documentation of the
"DistanceMetric" class and the metrics listed in

*sklearn.metrics.pairwise.PAIRWISE_DISTANCE_FUNCTIONS*. Note that the
"cosine" metric uses "cosine_distances()".

1.6.2. Nearest Neighbors Classification

Neighbors-based classification is a type of *instance-based learning*
or *non-generalizing learning*: it does not attempt to construct a
general internal model, but simply stores instances of the training
data. Classification is computed from a simple majority vote of the
nearest neighbors of each point: a query point is assigned the data
class which has the most representatives within the nearest neighbors
of the point.

scikit-learn implements two different nearest neighbors classifiers:
"KNeighborsClassifier" implements learning based on the k nearest
neighbors of each query point, where k is an integer value specified
by the user.  "RadiusNeighborsClassifier" implements learning based on
the number of neighbors within a fixed radius r of each training
point, where r is a floating-point value specified by the user.

The k-neighbors classification in "KNeighborsClassifier" is the most
commonly used technique. The optimal choice of the value k is highly
data-dependent: in general a larger k suppresses the effects of noise,
but makes the classification boundaries less distinct.

In cases where the data is not uniformly sampled, radius-based
neighbors classification in "RadiusNeighborsClassifier" can be a
better choice. The user specifies a fixed radius r, such that points
in sparser neighborhoods use fewer nearest neighbors for the
classification.  For high-dimensional parameter spaces, this method
becomes less effective due to the so-called "curse of dimensionality".

The basic nearest neighbors classification uses uniform weights: that
is, the value assigned to a query point is computed from a simple
majority vote of the nearest neighbors.  Under some circumstances, it
is better to weight the neighbors such that nearer neighbors
contribute more to the fit.  This can be accomplished through the
"weights" keyword.  The default value, "weights = 'uniform'", assigns
uniform weights to each neighbor. "weights = 'distance'" assigns
weights proportional to the inverse of the distance from the query
point.  Alternatively, a user-defined function of the distance can be
supplied to compute the weights.

[image: classification_1][image] [image: classification_2][image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neighbors_plot_classification.py*: an
  example of classification using nearest neighbors.

1.6.3. Nearest Neighbors Regression

^^^^^^^^^

Neighbors-based regression can be used in cases where the data labels
are continuous rather than discrete variables.  The label assigned to
a query point is computed based on the mean of the labels of its
nearest neighbors.

scikit-learn implements two different neighbors regressors:
"KNeighborsRegressor" implements learning based on the k nearest
by the user.  "RadiusNeighborsRegressor" implements learning based on
the neighbors within a fixed radius r of the query point, where r is a
floating-point value specified by the user.

The basic nearest neighbors regression uses uniform weights: that is,
each point in the local neighborhood contributes uniformly to the
classification of a query point.  Under some circumstances, it can be
advantageous to weight points such that nearby points contribute more
to the regression than faraway points.  This can be accomplished
through the "weights" keyword.  The default value, "weights =
'uniform'", assigns equal weights to all points.  "weights =
'distance'" assigns weights proportional to the inverse of the
distance from the query point. Alternatively, a user-defined function
of the distance can be supplied, which will be used to compute the
weights.

   [image]

weights.

   [image]

The use of multi-output nearest neighbors for regression is
demonstrated in *sphx_glr_auto_examples_miscellaneous_plot_multioutpu
t_face_completion.py*. In this example, the inputs X are the pixels of
the upper half of faces and the outputs Y are the pixels of the lower
half of those faces.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neighbors_plot_regression.py*: an example of
  regression using nearest neighbors.

  ion.py*: an example of multi-output regression using nearest
  neighbors.

1.6.4. Nearest Neighbor Algorithms

1.6.4.1. Brute Force

Fast computation of nearest neighbors is an active area of research in
machine learning. The most naive neighbor search implementation
involves the brute-force computation of distances between all pairs of
points in the dataset: for N samples in D dimensions, this approach
scales as O[D N^2].  Efficient brute-force neighbors searches can be
very competitive for small data samples. However, as the number of
samples N grows, the brute-force approach quickly becomes infeasible.
In the classes within "sklearn.neighbors", brute-force neighbors
searches are specified using the keyword "algorithm = 'brute'", and
are computed using the routines available in

"sklearn.metrics.pairwise".

1.6.4.2. K-D Tree

To address the computational inefficiencies of the brute-force
approach, a variety of tree-based data structures have been invented.
In general, these structures attempt to reduce the required number of
distance calculations by efficiently encoding aggregate distance
information for the sample. The basic idea is that if point A is very
distant from point B, and point B is very close to point C, then we
know that points A and C are very distant, *without having to
explicitly calculate their distance*. In this way, the computational
cost of a nearest neighbors search can be reduced to O[D N \log(N)] or
better. This is a significant improvement over brute-force for large
N.

An early approach to taking advantage of this aggregate information
was the *KD tree* data structure (short for *K-dimensional tree*),
which generalizes two-dimensional *Quad-trees* and 3-dimensional *Oct-
trees* to an arbitrary number of dimensions.  The KD tree is a binary
tree structure which recursively partitions the parameter space along
the data axes, dividing it into nested orthotropic regions into which
data points are filed.  The construction of a KD tree is very fast:
because partitioning is performed only along the data axes, no

D-dimensional distances need to be computed. Once constructed, the
nearest neighbor of a query point can be determined with only
O[\log(N)] distance computations. Though the KD tree approach is very
fast for low-dimensional (D < 20) neighbors searches, it becomes
inefficient as D grows very large: this is one manifestation of the
so-called "curse of dimensionality". In scikit-learn, KD tree
neighbors searches are specified using the keyword "algorithm =
'kd_tree'", and are computed using the class "KDTree".

* "Multidimensional binary search trees used for associative
  searching", Bentley, J.L., Communications of the ACM (1975)

1.6.4.3. Ball Tree

To address the inefficiencies of KD Trees in higher dimensions, the
*ball tree* data structure was developed.  Where KD trees partition
data along Cartesian axes, ball trees partition data in a series of
nesting hyper-spheres.  This makes tree construction more costly than
that of the KD tree, but results in a data structure which can be very
efficient on highly structured data, even in very high dimensions.

A ball tree recursively divides the data into nodes defined by a
centroid C and radius r, such that each point in the node lies within
the hyper-sphere defined by r and C. The number of candidate points

for a neighbor search is reduced through use of the *triangle
inequality*:

   |x+y| \leq |x| + |y|

With this setup, a single distance calculation between a test point
and the centroid is sufficient to determine a lower and upper bound on
the distance to all points within the node. Because of the spherical
geometry of the ball tree nodes, it can out-perform a *KD-tree* in
high dimensions, though the actual performance is highly dependent on
the structure of the training data. In scikit-learn, ball-tree-based
'ball_tree'", and are computed using the class "BallTree".
Alternatively, the user can work with the "BallTree" class directly.

* "Five Balltree Construction Algorithms", Omohundro, S.M.,
  International Computer Science Institute Technical Report (1989)

1.6.4.4. Choice of Nearest Neighbors Algorithm

The optimal algorithm for a given dataset is a complicated choice, and
depends on a number of factors:

* number of samples N (i.e. "n_samples") and dimensionality D (i.e.
  "n_features").

  * *Brute force* query time grows as O[D N]

  * *Ball tree* query time grows as approximately O[D \log(N)]

  * *KD tree* query time changes with D in a way that is difficult to

"n_features").

    precisely characterise.  For small D (less than 20 or so) the cost
    is approximately O[D\log(N)], and the KD tree query can be very
    efficient. For larger D, the cost increases to nearly O[DN], and
    the overhead due to the tree structure can lead to queries which
    are slower than brute force.

  For small data sets (N less than 30 or so), \log(N) is comparable to
  N, and brute force algorithms can be more efficient than a tree-
  based approach.  Both "KDTree" and "BallTree" address this through
  providing a *leaf size* parameter: this controls the number of
  samples at which a query switches to brute-force.  This allows both
  algorithms to approach the efficiency of a brute-force computation
  for small N.

* data structure: *intrinsic dimensionality* of the data and/or
  *sparsity* of the data. Intrinsic dimensionality refers to the
  dimension d \le D of a manifold on which the data lies, which can be
  linearly or non-linearly embedded in the parameter space. Sparsity
  refers to the degree to which the data fills the parameter space
  (this is to be distinguished from the concept as used in "sparse"
  matrices.  The data matrix may have no zero entries, but the
  **structure** can still be "sparse" in this sense).

  * *Brute force* query time is unchanged by data structure.

matrices.  The data matrix may have no zero entries, but the

  * *Ball tree* and *KD tree* query times can be greatly influenced by
    data structure.  In general, sparser data with a smaller intrinsic
    dimensionality leads to faster query times.  Because the KD tree
    internal representation is aligned with the parameter axes, it
    will not generally show as much improvement as ball tree for
    arbitrarily structured data.

  Datasets used in machine learning tend to be very structured, and
  are very well-suited for tree-based queries.

* number of neighbors k requested for a query point.

  * *Brute force* query time is largely unaffected by the value of k

  * *Ball tree* and *KD tree* query time will become slower as k
    increases.  This is due to two effects: first, a larger k leads to
    the necessity to search a larger portion of the parameter space.
    Second, using k > 1 requires internal queueing of results as the
    tree is traversed.

  As k becomes large compared to N, the ability to prune branches in a
  tree-based query is reduced.  In this situation, Brute force queries
  can be more efficient.

* number of query points.  Both the ball tree and the KD Tree require
  a construction phase.  The cost of this construction becomes
  negligible when amortized over many queries.  If only a small number

a construction phase.  The cost of this construction becomes
  of queries will be performed, however, the construction can make up
  a significant fraction of the total cost.  If very few query points
  will be required, brute force is better than a tree-based method.

Currently, "algorithm = 'auto'" selects "'brute'" if any of the
following conditions are verified:

* input data is sparse

* "metric = 'precomputed'"

* D > 15

* k >= N/2

* "effective_metric_" isn't in the "VALID_METRICS" list for either
  "'kd_tree'" or "'ball_tree'"

Otherwise, it selects the first out of "'kd_tree'" and "'ball_tree'"
that has "effective_metric_" in its "VALID_METRICS" list. This
heuristic is based on the following assumptions:

* the number of query points is at least the same order as the number
  of training points

* "leaf_size" is close to its default value of "30"

* when D > 15, the intrinsic dimensionality of the data is generally
  too high for tree-based methods

1.6.4.5. Effect of "leaf_size"

As noted above, for small sample sizes a brute force search can be
more efficient than a tree-based query.  This fact is accounted for in
the ball tree and KD tree by internally switching to brute force
searches within leaf nodes.  The level of this switch can be specified
with the parameter "leaf_size".  This parameter choice has many
effects:

**construction time**

effects:

   A larger "leaf_size" leads to a faster tree construction time,
   because fewer nodes need to be created

**query time**
   Both a large or small "leaf_size" can lead to suboptimal query
   cost. For "leaf_size" approaching 1, the overhead involved in
   traversing nodes can significantly slow query times.  For
   "leaf_size" approaching the size of the training set, queries
   become essentially brute force. A good compromise between these is
   "leaf_size = 30", the default value of the parameter.

**memory**
   As "leaf_size" increases, the memory required to store a tree
   structure decreases.  This is especially important in the case of
   ball tree, which stores a D-dimensional centroid for each node.
   The required storage space for "BallTree" is approximately "1 /
   leaf_size" times the size of the training set.

"leaf_size" is not referenced for brute force queries.

1.6.4.6. Valid Metrics for Nearest Neighbor Algorithms

A list of valid metrics for any of the above algorithms can be
obtained by using their "valid_metric" attribute. For example, valid

metrics for "KDTree" can be generated by:

>>> print(sorted(KDTree.valid_metrics))
['chebyshev', 'cityblock', 'euclidean', 'infinity', 'l1', 'l2', 'manhattan', 'minkowski', 'p']

1.6.5. Nearest Centroid Classifier

The "NearestCentroid" classifier is a simple algorithm that represents
each class by the centroid of its members. In effect, this makes it
similar to the label updating phase of the "KMeans" algorithm. It also
has no parameters to choose, making it a good baseline classifier. It
does, however, suffer on non-convex classes, as well as when classes
have drastically different variances, as equal variance in all
dimensions is assumed. See Linear Discriminant Analysis
("LinearDiscriminantAnalysis") and Quadratic Discriminant Analysis
("QuadraticDiscriminantAnalysis") for more complex methods that do not
make this assumption. Usage of the default "NearestCentroid" is
simple:

>>> from sklearn.neighbors import NearestCentroid
>>> y = np.array([1, 1, 1, 2, 2, 2])
>>> clf = NearestCentroid()
>>> clf.fit(X, y)
NearestCentroid()
>>> print(clf.predict([[-0.8, -1]]))
[1]

1.6.5.1. Nearest Shrunken Centroid

[1]

The "NearestCentroid" classifier has a "shrink_threshold" parameter,
which implements the nearest shrunken centroid classifier. In effect,
the value of each feature for each centroid is divided by the within-
class variance of that feature. The feature values are then reduced by
"shrink_threshold". Most notably, if a particular feature value
crosses zero, it is set to zero. In effect, this removes the feature
from affecting the classification. This is useful, for example, for
removing noisy features.

In the example below, using a small shrink threshold increases the
accuracy of the model from 0.81 to 0.82.

[image: nearest_centroid_1][image] [image: nearest_centroid_2][image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neighbors_plot_nearest_centroid.py*: an
  example of classification using nearest centroid with different
  shrink thresholds.

1.6.6. Nearest Neighbors Transformer

Many scikit-learn estimators rely on nearest neighbors: Several
classifiers and regressors such as "KNeighborsClassifier" and
"KNeighborsRegressor", but also some clustering methods such as
"DBSCAN" and "SpectralClustering", and some manifold embeddings such
as "TSNE" and "Isomap".

All these estimators can compute internally the nearest neighbors, but
most of them also accept precomputed nearest neighbors *sparse graph*,

as given by "kneighbors_graph()" and "radius_neighbors_graph()". With
mode *mode='connectivity'*, these functions return a binary adjacency
sparse graph as required, for instance, in "SpectralClustering".
Whereas with *mode='distance'*, they return a distance sparse graph as
required, for instance, in "DBSCAN". To include these functions in a
scikit-learn pipeline, one can also use the corresponding classes
"KNeighborsTransformer" and "RadiusNeighborsTransformer". The benefits
of this sparse graph API are multiple.

First, the precomputed graph can be re-used multiple times, for
instance while varying a parameter of the estimator. This can be done
manually by the user, or using the caching properties of the scikit-
learn pipeline:

>>> import tempfile
>>> from sklearn.manifold import Isomap
>>> from sklearn.neighbors import KNeighborsTransformer
>>> from sklearn.pipeline import make_pipeline
>>> from sklearn.datasets import make_regression
>>> cache_path = tempfile.gettempdir()  # we use a temporary folder here
>>> X, _ = make_regression(n_samples=50, n_features=25, random_state=0)
>>> estimator = make_pipeline(
...     KNeighborsTransformer(mode='distance'),
...     Isomap(n_components=3, metric='precomputed'),
...     memory=cache_path)
>>> X_embedded = estimator.fit_transform(X)
>>> X_embedded.shape
(50, 3)

(50, 3)

Second, precomputing the graph can give finer control on the nearest
neighbors estimation, for instance enabling multiprocessing though the
parameter *n_jobs*, which might not be available in all estimators.

Finally, the precomputation can be performed by custom estimators to
use different implementations, such as approximate nearest neighbors
methods, or implementation with special data types. The precomputed
neighbors *sparse graph* needs to be formatted as in
"radius_neighbors_graph()" output:

* a CSR matrix (although COO, CSC or LIL will be accepted).

* only explicitly store nearest neighborhoods of each sample with
  respect to the training data. This should include those at 0
  distance from a query point, including the matrix diagonal when
  computing the nearest neighborhoods between the training data and
  itself.

* each row's *data* should store the distance in increasing order
  (optional. Unsorted data will be stable-sorted, adding a
  computational overhead).

* all values in data should be non-negative.

* there should be no duplicate *indices* in any row (see
  https://github.com/scipy/scipy/issues/5807).

* if the algorithm being passed the precomputed matrix uses k nearest
  neighbors (as opposed to radius neighborhood), at least k neighbors

https://github.com/scipy/scipy/issues/5807).

  must be stored in each row (or k+1, as explained in the following
  note).

Note:

  When a specific number of neighbors is queried (using
  "KNeighborsTransformer"), the definition of *n_neighbors* is
  ambiguous since it can either include each training point as its own
  neighbor, or exclude them. Neither choice is perfect, since
  including them leads to a different number of non-self neighbors
  during training and testing, while excluding them leads to a
  difference between *fit(X).transform(X)* and *fit_transform(X)*,
  which is against scikit-learn API. In "KNeighborsTransformer" we use
  the definition which includes each training point as its own
  neighbor in the count of *n_neighbors*. However, for compatibility
  reasons with other estimators which use the other definition, one
  extra neighbor will be computed when *mode == 'distance'*. To
  maximise compatibility with all estimators, a safe choice is to
  always include one extra neighbor in a custom nearest neighbors
  estimator, since unnecessary neighbors will be filtered by following

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neighbors_approximate_nearest_neighbors.py*:
  an example of pipelining "KNeighborsTransformer" and "TSNE". Also
  proposes two custom nearest neighbors estimators based on external
  packages.

an example of pipelining "KNeighborsTransformer" and "TSNE". Also
  packages.

* *sphx_glr_auto_examples_neighbors_plot_caching_nearest_neighbors.py
  *: an example of pipelining "KNeighborsTransformer" and
  "KNeighborsClassifier" to enable caching of the neighbors graph
  during a hyper-parameter grid-search.

1.6.7. Neighborhood Components Analysis

Neighborhood Components Analysis (NCA,
"NeighborhoodComponentsAnalysis") is a distance metric learning
algorithm which aims to improve the accuracy of nearest neighbors
classification compared to the standard Euclidean distance. The
algorithm directly maximizes a stochastic variant of the leave-one-out
k-nearest neighbors (KNN) score on the training set. It can also learn
a low-dimensional linear projection of data that can be used for data
visualization and fast classification.

[image: nca_illustration_1][image] [image: nca_illustration_2][image]

In the above illustrating figure, we consider some points from a
randomly generated dataset. We focus on the stochastic KNN
classification of point no. 3. The thickness of a link between sample
3 and another point is proportional to their distance, and can be seen
as the relative weight (or probability) that a stochastic nearest
neighbor prediction rule would assign to this point. In the original
space, sample 3 has many stochastic neighbors from various classes, so

the right class is not very likely. However, in the projected space
learned by NCA, the only stochastic neighbors with non-negligible
weight are from the same class as sample 3, guaranteeing that the
latter will be well classified. See the mathematical formulation for

1.6.7.1. Classification

Combined with a nearest neighbors classifier ("KNeighborsClassifier"),
NCA is attractive for classification because it can naturally handle
multi-class problems without any increase in the model size, and does
not introduce additional parameters that require fine-tuning by the
user.

NCA classification has been shown to work well in practice for data
sets of varying size and difficulty. In contrast to related methods
such as Linear Discriminant Analysis, NCA does not make any
assumptions about the class distributions. The nearest neighbor
classification can naturally produce highly irregular decision
boundaries.

To use this model for classification, one needs to combine a
"NeighborhoodComponentsAnalysis" instance that learns the optimal
transformation with a "KNeighborsClassifier" instance that performs
the classification in the projected space. Here is an example using
the two classes:

>>> from sklearn.neighbors import (NeighborhoodComponentsAnalysis,
... KNeighborsClassifier)

>>> from sklearn.pipeline import Pipeline
>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
... stratify=y, test_size=0.7, random_state=42)
>>> nca = NeighborhoodComponentsAnalysis(random_state=42)
>>> knn = KNeighborsClassifier(n_neighbors=3)
>>> nca_pipe = Pipeline([('nca', nca), ('knn', knn)])
>>> nca_pipe.fit(X_train, y_train)
Pipeline(...)
>>> print(nca_pipe.score(X_test, y_test))
0.96190476...

[image: nca_classification_1][image] [image:
nca_classification_2][image]

The plot shows decision boundaries for Nearest Neighbor Classification
and Neighborhood Components Analysis classification on the iris
dataset, when training and scoring on only two features, for
visualisation purposes.

1.6.7.2. Dimensionality reduction

NCA can be used to perform supervised dimensionality reduction. The
input data are projected onto a linear subspace consisting of the
directions which minimize the NCA objective. The desired
dimensionality can be set using the parameter "n_components". For
instance, the following figure shows a comparison of dimensionality
reduction with Principal Component Analysis ("PCA"), Linear
Discriminant Analysis ("LinearDiscriminantAnalysis") and Neighborhood

Component Analysis ("NeighborhoodComponentsAnalysis") on the Digits
dataset, a dataset with size n_{samples} = 1797 and n_{features} = 64.
The data set is split into a training and a test set of equal size,
then standardized. For evaluation the 3-nearest neighbor
classification accuracy is computed on the 2-dimensional projected
points found by each method. Each data sample belongs to one of 10
classes.

[image: nca_dim_reduction_1][image] [image:
nca_dim_reduction_2][image] [image: nca_dim_reduction_3][image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_neighbors_plot_nca_classification.py*

* *sphx_glr_auto_examples_neighbors_plot_nca_dim_reduction.py*

* *sphx_glr_auto_examples_manifold_plot_lle_digits.py*

1.6.7.3. Mathematical formulation

The goal of NCA is to learn an optimal linear transformation matrix of
size "(n_components, n_features)", which maximises the sum over all
samples i of the probability p_i that i is correctly classified, i.e.:

   \underset{L}{\arg\max} \sum\limits_{i=0}^{N - 1} p_{i}

with N = "n_samples" and p_i the probability of sample i being
correctly classified according to a stochastic nearest neighbors rule
in the learned embedded space:

   p_{i}=\sum\limits_{j \in C_i}{p_{i j}}

where C_i is the set of points in the same class as sample i, and p_{i
j} is the softmax over Euclidean distances in the embedded space:

   p_{i j} = \frac{\exp(-||L x_i - L x_j||^2)}{\sum\limits_{k \ne
   i} {\exp{-(||L x_i - L x_k||^2)}}} , \quad p_{i i} = 0

1.6.7.3.1. Mahalanobis distance

NCA can be seen as learning a (squared) Mahalanobis distance metric:

   || L(x_i - x_j)||^2 = (x_i - x_j)^TM(x_i - x_j),

where M = L^T L is a symmetric positive semi-definite matrix of size
"(n_features, n_features)".

1.6.7.4. Implementation

This implementation follows what is explained in the original paper
[1]. For the optimisation method, it currently uses scipy's L-BFGS-B
with a full gradient computation at each iteration, to avoid to tune
the learning rate and provide stable learning.

See the examples below and the docstring of
"NeighborhoodComponentsAnalysis.fit()" for further information.

1.6.7.5. Complexity

1.6.7.5.1. Training
~~~~~~~~~~~~~~~~~~~

NCA stores a matrix of pairwise distances, taking "n_samples ** 2"
memory. Time complexity depends on the number of iterations done by
the optimisation algorithm. However, one can set the maximum number of
iterations with the argument "max_iter". For each iteration, time
complexity is "O(n_components x n_samples x min(n_samples,
n_features))".

1.6.7.5.2. Transform

Here the "transform" operation returns LX^T, therefore its time
complexity equals "n_components * n_features * n_samples_test". There
is no added space complexity in the operation.

[1] "Neighbourhood Components Analysis", J. Goldberger, S. Roweis, G.
    Hinton, R. Salakhutdinov, Advances in Neural Information
    Processing Systems, Vol. 17, May 2005, pp. 513-520.

Wikipedia entry on Neighborhood Components Analysis

1.12. Multiclass and multioutput algorithms

This section of the user guide covers functionality related to multi-
learning problems, including *multiclass*, *multilabel*, and
*multioutput* classification and regression.

The modules in this section implement *meta-estimators*, which require
a base estimator to be provided in their constructor. Meta-estimators
extend the functionality of the base estimator to support multi-
learning problems, which is accomplished by transforming the multi-
learning problem into a set of simpler problems, then fitting one
estimator per problem.

This section covers two modules: "sklearn.multiclass" and
"sklearn.multioutput". The chart below demonstrates the problem types
that each module is responsible for, and the corresponding meta-
estimators that each module provides.

[image]

The table below provides a quick reference on the differences between
problem types. More detailed explanations can be found in subsequent
sections of this guide.

+--------------------------------+-------------------------+---------------------------+----------------------------------------------------+
|                                | Number of targets       | Target cardinality        | Valid "type_of_target()"                           |
|================================|=========================|===========================|====================================================|

| Multiclass classification      | 1                       | >2                        | 'multiclass'                                       |
| Multilabel classification      | >1                      | 2 (0 or 1)                | 'multilabel-indicator'                             |
| Multiclass-multioutput         | >1                      | >2                        | 'multiclass-multioutput'                           |
| classification                 |                         |                           |                                                    |
| Multioutput regression         | >1                      | Continuous                | 'continuous-multioutput'                           |

Below is a summary of scikit-learn estimators that have multi-learning

support built-in, grouped by strategy. You don't need the meta-
estimators provided by this section if you're using one of these
estimators. However, meta-estimators can provide additional strategies
beyond what is built-in:

* **Inherently multiclass:**

  * "naive_bayes.BernoulliNB"

  * "tree.DecisionTreeClassifier"

  * "tree.ExtraTreeClassifier"

  * "ensemble.ExtraTreesClassifier"

  * "naive_bayes.GaussianNB"

  * "neighbors.KNeighborsClassifier"

  * "semi_supervised.LabelPropagation"

  * "semi_supervised.LabelSpreading"

  * "discriminant_analysis.LinearDiscriminantAnalysis"

  * "svm.LinearSVC" (setting multi_class="crammer_singer")

  * "linear_model.LogisticRegression" (setting
    multi_class="multinomial")

  * "linear_model.LogisticRegressionCV" (setting

  * "neural_network.MLPClassifier"

  * "neighbors.NearestCentroid"

  * "discriminant_analysis.QuadraticDiscriminantAnalysis"

  * "neighbors.RadiusNeighborsClassifier"

  * "ensemble.RandomForestClassifier"

  * "linear_model.RidgeClassifier"

  * "linear_model.RidgeClassifierCV"

* **Multiclass as One-Vs-One:**

  * "svm.NuSVC"

  * "svm.SVC".

  * "gaussian_process.GaussianProcessClassifier" (setting multi_class
    = "one_vs_one")

* **Multiclass as One-Vs-The-Rest:**

  * "ensemble.GradientBoostingClassifier"

    = "one_vs_rest")

  * "svm.LinearSVC" (setting multi_class="ovr")

  * "linear_model.LogisticRegression" (setting multi_class="ovr")

  * "linear_model.LogisticRegressionCV" (setting multi_class="ovr")

  * "linear_model.SGDClassifier"

  * "linear_model.Perceptron"

  * "linear_model.PassiveAggressiveClassifier"

* **Support multilabel:**

* **Support multiclass-multioutput:**

1.12.1. Multiclass classification

Warning:

  All classifiers in scikit-learn do multiclass classification out-of-
  the-box. You don't need to use the "sklearn.multiclass" module
  unless you want to experiment with different multiclass strategies.

**Multiclass classification** is a classification task with more than
two classes. Each sample can only be labeled as one class.

For example, classification using features extracted from a set of

images of fruit, where each image may either be of an orange, an
apple, or a pear. Each image is one sample and is labeled as one of
the 3 possible classes. Multiclass classification makes the assumption
that each sample is assigned to one and only one label - one sample
cannot, for example, be both a pear and an apple.

While all scikit-learn classifiers are capable of multiclass
classification, the meta-estimators offered by "sklearn.multiclass"
permit changing the way they handle more than two classes because this
may have an effect on classifier performance (either in terms of
generalization error or required computational resources).

1.12.1.1. Target format

Valid *multiclass* representations for "type_of_target()" (*y*) are:

   * 1d or column vector containing more than two discrete values. An
     example of a vector "y" for 4 samples:

     >>> import numpy as np
     >>> y = np.array(['apple', 'pear', 'apple', 'orange'])
     >>> print(y)
     ['apple' 'pear' 'apple' 'orange']

   * Dense or sparse *binary* matrix of shape "(n_samples, n_classes)"
     with a single sample per row, where each column represents one
     class. An example of both a dense and sparse *binary* matrix "y"
     for 4 samples, where the columns, in order, are apple, orange,
     and pear:

class. An example of both a dense and sparse *binary* matrix "y"
     and pear:

     >>> from sklearn.preprocessing import LabelBinarizer
     >>> y_dense = LabelBinarizer().fit_transform(y)
     >>> print(y_dense)
       [[1 0 0]
        [0 0 1]
        [1 0 0]
        [0 1 0]]
     >>> from scipy import sparse
     >>> y_sparse = sparse.csr_matrix(y_dense)
     >>> print(y_sparse)
         (0, 0)        1
         (1, 2)        1
         (2, 0)        1
         (3, 1)        1

For more information about "LabelBinarizer", refer to Transforming the
prediction target (y).

1.12.1.2. OneVsRestClassifier

The **one-vs-rest** strategy, also known as **one-vs-all**, is
implemented in "OneVsRestClassifier".  The strategy consists in
fitting one classifier per class. For each classifier, the class is
fitted against all the other classes. In addition to its computational
efficiency (only *n_classes* classifiers are needed), one advantage of
this approach is its interpretability. Since each class is represented
by one and only one classifier, it is possible to gain knowledge about
the class by inspecting its corresponding classifier. This is the most
commonly used strategy and is a fair default choice.

Below is an example of multiclass learning using OvR:

   >>> OneVsRestClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)
   array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
          2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 2, 2, 2, 2,
          2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

"OneVsRestClassifier" also supports multilabel classification. To use
this feature, feed the classifier an indicator matrix, in which cell
[i, j] indicates the presence of label j in sample i.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_miscellaneous_plot_multilabel.py*

1.12.1.3. OneVsOneClassifier

"OneVsOneClassifier" constructs one classifier per pair of classes. At
prediction time, the class which received the most votes is selected.
In the event of a tie (among two classes with an equal number of
votes), it selects the class with the highest aggregate classification

confidence by summing over the pair-wise classification confidence
levels computed by the underlying binary classifiers.

Since it requires to fit "n_classes * (n_classes - 1) / 2"
classifiers, this method is usually slower than one-vs-the-rest, due
to its O(n_classes^2) complexity. However, this method may be
advantageous for algorithms such as kernel algorithms which don't
scale well with "n_samples". This is because each individual learning
problem only involves a small subset of the data whereas, with one-vs-
the-rest, the complete dataset is used "n_classes" times. The decision
function is the result of a monotonic transformation of the one-
versus-one classification.

Below is an example of multiclass learning using OvO:

   >>> from sklearn.multiclass import OneVsOneClassifier
   >>> OneVsOneClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)
          1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1,

0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,

* "Pattern Recognition and Machine Learning. Springer", Christopher M.
  Bishop, page 183, (First Edition)

1.12.1.4. OutputCodeClassifier

Error-Correcting Output Code-based strategies are fairly different
from one-vs-the-rest and one-vs-one. With these strategies, each class
is represented in a Euclidean space, where each dimension can only be
0 or 1. Another way to put it is that each class is represented by a
binary code (an array of 0 and 1). The matrix which keeps track of the
location/code of each class is called the code book. The code size is
the dimensionality of the aforementioned space. Intuitively, each
class should be represented by a code as unique as possible and a good
code book should be designed to optimize classification accuracy. In
this implementation, we simply use a randomly-generated code book as
advocated in [3] although more elaborate methods may be added in the
future.

At fitting time, one binary classifier per bit in the code book is
fitted. At prediction time, the classifiers are used to project new

future.

points in the class space and the class closest to the points is
chosen.

In "OutputCodeClassifier", the "code_size" attribute allows the user
to control the number of classifiers which will be used. It is a
percentage of the total number of classes.

A number between 0 and 1 will require fewer classifiers than one-vs-
the-rest. In theory, "log2(n_classes) / n_classes" is sufficient to
represent each class unambiguously. However, in practice, it may not
lead to good accuracy since "log2(n_classes)" is much smaller than
*n_classes*.

A number greater than 1 will require more classifiers than one-vs-the-
rest. In this case, some classifiers will in theory correct for the
mistakes made by other classifiers, hence the name "error-correcting".
In practice, however, this may not happen as classifier mistakes will
typically be correlated. The error-correcting output codes have a
similar effect to bagging.

Below is an example of multiclass learning using Output-Codes:

   >>> from sklearn.multiclass import OutputCodeClassifier
   >>> clf = OutputCodeClassifier(LinearSVC(random_state=0),
   ...                            code_size=2, random_state=0)
   >>> clf.fit(X, y).predict(X)

>>> clf = OutputCodeClassifier(LinearSVC(random_state=0),
          0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1,
          1, 2, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1,
          2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 2, 2,

* "Solving multiclass learning problems via error-correcting output
  codes", Dietterich T., Bakiri G., Journal of Artificial Intelligence
  Research 2, 1995.

[3] "The error coding method and PICTs", James G., Hastie T., Journal
    of Computational and Graphical statistics 7, 1998.

* "The Elements of Statistical Learning", Hastie T., Tibshirani R.,
  Friedman J., page 606 (second-edition) 2008.

1.12.2. Multilabel classification

**Multilabel classification** (closely related to **multioutput**
**classification**) is a classification task labeling each sample with
"m" labels from "n_classes" possible classes, where "m" can be 0 to
"n_classes" inclusive. This can be thought of as predicting properties
of a sample that are not mutually exclusive. Formally, a binary output

is assigned to each class, for every sample. Positive classes are
indicated with 1 and negative classes with 0 or -1. It is thus
comparable to running "n_classes" binary classification tasks, for
example with "MultiOutputClassifier". This approach treats each label
independently whereas multilabel classifiers *may* treat the multiple
classes simultaneously, accounting for correlated behavior among them.

For example, prediction of the topics relevant to a text document or
video. The document or video may be about one of 'religion',
'politics', 'finance' or 'education', several of the topic classes or
all of the topic classes.

1.12.2.1. Target format

A valid representation of *multilabel* *y* is an either dense or
sparse *binary* matrix of shape "(n_samples, n_classes)". Each column
represents a class. The "1"'s in each row denote the positive classes
a sample has been labeled with. An example of a dense matrix "y" for 3
samples:

>>> y = np.array([[1, 0, 0, 1], [0, 0, 1, 1], [0, 0, 0, 0]])
>>> print(y)
[[1 0 0 1]
 [0 0 1 1]
 [0 0 0 0]]

Dense binary matrices can also be created using "MultiLabelBinarizer".
For more information, refer to Transforming the prediction target (y).

An example of the same "y" in sparse matrix form:

>>> y_sparse = sparse.csr_matrix(y)
>>> print(y_sparse)
  (0, 0)      1
  (0, 3)      1
  (1, 2)      1
  (1, 3)      1

  (0, 0)      1
  (0, 3)      1
  (1, 2)      1
  (1, 3)      1

1.12.2.2. MultiOutputClassifier

Multilabel classification support can be added to any classifier with
"MultiOutputClassifier". This strategy consists of fitting one
classifier per target.  This allows multiple target variable
classifications. The purpose of this class is to extend estimators to
be able to estimate a series of target functions (f1,f2,f3...,fn) that
are trained on a single X predictor matrix to predict a series of
responses (y1,y2,y3...,yn).

You can find a usage example for "MultiOutputClassifier" as part of
the section on Multiclass-multioutput classification since it is a
generalization of multilabel classification to multiclass outputs
instead of binary outputs.

1.12.2.3. ClassifierChain

Classifier chains (see "ClassifierChain") are a way of combining a
number of binary classifiers into a single multi-label model that is
capable of exploiting correlations among targets.

For a multi-label classification problem with N classes, N binary
classifiers are assigned an integer between 0 and N-1. These integers
define the order of models in the chain. Each classifier is then fit
on the available training data plus the true labels of the classes
whose models were assigned a lower number.

When predicting, the true labels will not be available. Instead the

predictions of each model are passed on to the subsequent models in
the chain to be used as features.

Clearly the order of the chain is important. The first model in the
chain has no information about the other labels while the last model
in the chain has features indicating the presence of all of the other
labels. In general one does not know the optimal ordering of the
models in the chain so typically many randomly ordered chains are fit
and their predictions are averaged together.

Jesse Read, Bernhard Pfahringer, Geoff Holmes, Eibe Frank,
   "Classifier Chains for Multi-label Classification", 2009.

1.12.3. Multiclass-multioutput classification

**Multiclass-multioutput classification** (also known as **multitask
classification**) is a classification task which labels each sample
with a set of **non-binary** properties. Both the number of properties
and the number of classes per property is greater than 2. A single
estimator thus handles several joint classification tasks. This is
both a generalization of the multi*label* classification task, which
only considers binary attributes, as well as a generalization of the
multi*class* classification task, where only one property is
considered.

For example, classification of the properties "type of fruit" and

"colour" for a set of images of fruit. The property "type of fruit"
has the possible classes: "apple", "pear" and "orange". The property
"colour" has the possible classes: "green", "red", "yellow" and
"orange". Each sample is an image of a fruit, a label is output for
both properties and each label is one of the possible classes of the
corresponding property.

Note that all classifiers handling multiclass-multioutput (also known
as multitask classification) tasks, support the multilabel
classification task as a special case. Multitask classification is
similar to the multioutput classification task with different model
formulations. For more information, see the relevant estimator
documentat

Below is an example of multiclass-multioutput classification:

>>> from sklearn.multioutput import MultiOutputClassifier
>>> from sklearn.utils import shuffle
>>> X, y1 = make_classification(n_samples=10, n_features=100,
...                             n_informative=30, n_classes=3,
...                             random_state=1)
>>> y2 = shuffle(y1, random_state=1)
>>> y3 = shuffle(y1, random_state=2)
>>> Y = np.vstack((y1, y2, y3)).T
>>> n_samples, n_features = X.shape # 10,100
>>> n_outputs = Y.shape[1] # 3
>>> n_classes = 3

>>> forest = RandomForestClassifier(random_state=1)
>>> multi_target_forest = MultiOutputClassifier(forest, n_jobs=2)
>>> multi_target_forest.fit(X, Y).predict(X)
array([[2, 2, 0],
       [1, 2, 1],
       [2, 1, 0],
       [0, 0, 2],
       [0, 2, 1],
       [0, 0, 2],
       [1, 1, 0],
       [1, 1, 1],
       [0, 0, 2],
       [2, 0, 0]])

Warning:

  At present, no metric in "sklearn.metrics" supports the multiclass-
  multioutput classification task.

1.12.3.1. Target format

A valid representation of *multioutput* *y* is a dense matrix of shape
"(n_samples, n_classes)" of class labels. A column wise concatenation
of 1d *multiclass* variables. An example of "y" for 3 samples:

>>> y = np.array([['apple', 'green'], ['orange', 'orange'], ['pear', 'green']])
[['apple' 'green']
 ['orange' 'orange']
 ['pear' 'green']]

1.12.4. Multioutput regression

**Multioutput regression** predicts multiple numerical properties for
each sample. Each property is a numerical variable and the number of
properties to be predicted for each sample is greater than or equal to
2. Some estimators that support multioutput regression are faster than
just running "n_output" estimators.

For example, prediction of both wind speed and wind direction, in

degrees, using data obtained at a certain location. Each sample would
be data obtained at one location and both wind speed and direction
would be output for each sample.

1.12.4.1. Target format

"(n_samples, n_output)" of floats. A column wise concatenation of
*continuous* variables. An example of "y" for 3 samples:

>>> y = np.array([[31.4, 94], [40.5, 109], [25.0, 30]])
[[ 31.4  94. ]
 [ 40.5 109. ]
 [ 25.   30. ]]

1.12.4.2. MultiOutputRegressor

Multioutput regression support can be added to any regressor with
"MultiOutputRegressor".  This strategy consists of fitting one
regressor per target. Since each target is represented by exactly one
regressor it is possible to gain knowledge about the target by
inspecting its corresponding regressor. As "MultiOutputRegressor" fits
one regressor per target it can not take advantage of correlations
between targets.

Below is an example of multioutput regression:

>>> from sklearn.multioutput import MultiOutputRegressor
>>> from sklearn.ensemble import GradientBoostingRegressor
>>> X, y = make_regression(n_samples=10, n_targets=3, random_state=1)

>>> MultiOutputRegressor(GradientBoostingRegressor(random_state=0)).fit(X, y).predict(X)
array([[-154.75474165, -147.03498585,  -50.03812219],
       [   7.12165031,    5.12914884,  -81.46081961],
       [-187.8948621 , -100.44373091,   13.88978285],
       [-141.62745778,   95.02891072, -191.48204257],
       [  97.03260883,  165.34867495,  139.52003279],
       [ 123.92529176,   21.25719016,   -7.84253   ],
       [-122.25193977,  -85.16443186, -107.12274212],
       [ -30.170388  ,  -94.80956739,   12.16979946],
       [ 140.72667194,  176.50941682,  -17.50447799],
       [ 149.37967282,  -81.15699552,   -5.72850319]])

1.12.4.3. RegressorChain

Regressor chains (see "RegressorChain") is analogous to
"ClassifierChain" as a way of combining a number of regressions into a
single multi-target model that is capable of exploiting correlations
among targets.

4.1. Partial Dependence and Individual Conditional Expectation plots
********************************************************************

Partial dependence plots (PDP) and individual conditional expectation
(ICE) plots can be used to visualize and analyze interaction between
the target response [1] and a set of input features of interest.

Both PDPs [H2009] and ICEs [G2015] assume that the input features of
interest are independent from the complement features, and this
assumption is often violated in practice. Thus, in the case of
correlated features, we will create absurd data points to compute the
PDP/ICE [M2019].

4.1.1. Partial dependence plots

Partial dependence plots (PDP) show the dependence between the target
response and a set of input features of interest, marginalizing over
the values of all other input features (the 'complement' features).
Intuitively, we can interpret the partial dependence as the expected
target response as a function of the input features of interest.

Due to the limits of human perception, the size of the set of input
features of interest must be small (usually, one or two) thus the
input features of interest are usually chosen among the most important
features.

The figure below shows two one-way and one two-way partial dependence
plots for the bike sharing dataset, with a
"HistGradientBoostingRegressor":

   [image]

One-way PDPs tell us about the interaction between the target response

   [image]

and an input feature of interest (e.g. linear, non-linear). The left
plot in the above figure shows the effect of the temperature on the
number of bike rentals; we can clearly see that a higher temperature
is related with a higher number of bike rentals. Similarly, we could
analyze the effect of the humidity on the number of bike rentals
(middle plot). Thus, these interpretations are marginal, considering a
feature at a time.

PDPs with two input features of interest show the interactions among
the two features. For example, the two-variable PDP in the above
figure shows the dependence of the number of bike rentals on joint
values of temperature and humidity. We can clearly see an interaction
between the two features: with a temperature higher than 20 degrees
Celsius, mainly the humidity has a strong impact on the number of bike
rentals. For lower temperatures, both the temperature and the humidity
have an impact on the number of bike rentals.

The "sklearn.inspection" module provides a convenience function
"from_estimator()" to create one-way and two-way partial dependence
plots. In the below example we show how to create a grid of partial
dependence plots: two one-way PDPs for the features "0" and "1" and a
two-way PDP between the two features:

   >>> from sklearn.inspection import PartialDependenceDisplay

   >>> features = [0, 1, (0, 1)]
   >>> PartialDependenceDisplay.from_estimator(clf, X, features)
   <...>

You can access the newly created figure and Axes objects using
"plt.gcf()" and "plt.gca()".

To make a partial dependence plot with categorical features, you need
to specify which features are categorical using the parameter
*categorical_features*. This parameter takes a list of indices, names
of the categorical features or a boolean mask. The graphical
representation of partial dependence for categorical features is a bar
plot or a 2D heatmap.

For multi-class classification, you need to set the class label for
which the PDPs should be created via the "target" argument:

   >>> mc_clf = GradientBoostingClassifier(n_estimators=10,
   ...     max_depth=1).fit(iris.data, iris.target)
   >>> features = [3, 2, (3, 2)]
   >>> PartialDependenceDisplay.from_estimator(mc_clf, X, features, target=0)
   <...>

The same parameter "target" is used to specify the target in multi-

>>> features = [3, 2, (3, 2)]
   <...>

output regression settings.

If you need the raw values of the partial dependence function rather
than the plots, you can use the
"sklearn.inspection.partial_dependence()" function:

   >>> from sklearn.inspection import partial_dependence

   >>> results = partial_dependence(clf, X, [0])
   >>> results["average"]
   array([[ 2.466...,  2.466..., ...
   >>> results["values"]
   [array([-1.624..., -1.592..., ...

The values at which the partial dependence should be evaluated are
directly generated from "X". For 2-way partial dependence, a 2D-grid
of values is generated. The "values" field returned by
"sklearn.inspection.partial_dependence()" gives the actual values used
in the grid for each input feature of interest. They also correspond
to the axis of the plots.

4.1.2. Individual conditional expectation (ICE) plot

Similar to a PDP, an individual conditional expectation (ICE) plot
shows the dependence between the target function and an input feature
of interest. However, unlike a PDP, which shows the average effect of
the input feature, an ICE plot visualizes the dependence of the
prediction on a feature for each sample separately with one line per
sample. Due to the limits of human perception, only one input feature
of interest is supported for ICE plots.

The figures below show two ICE plots for the bike sharing dataset,
with a "HistGradientBoostingRegressor":. The figures plot the
corresponding PD line overlaid on ICE lines.

   [image]

While the PDPs are good at showing the average effect of the target
features, they can obscure a heterogeneous relationship created by
interactions. When interactions are present the ICE plot will provide
many more insights. For example, we see that the ICE for the
temperature feature gives us some additional information: Some of the
ICE lines are flat while some others shows a decrease of the
dependence for temperature above 35 degrees Celsius. We observe a
similar pattern for the humidity feature: some of the ICE lines show a
sharp decrease when the humidity is above 80%.

The "sklearn.inspection" module's
"PartialDependenceDisplay.from_estimator()" convenience function can
be used to create ICE plots by setting "kind='individual'". In the
example below, we show how to create a grid of ICE plots:

>>> from sklearn.datasets import make_hastie_10_2
>>> from sklearn.inspection import PartialDependenceDisplay

>>> X, y = make_hastie_10_2(random_state=0)
>>> clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,
...     max_depth=1, random_state=0).fit(X, y)

>>> features = [0, 1]
>>> PartialDependenceDisplay.from_estimator(clf, X, features,
...     kind='individual')
<...>

In ICE plots it might not be easy to see the average effect of the
input feature of interest. Hence, it is recommended to use ICE plots
alongside PDPs. They can be plotted together with "kind='both'".

...     kind='both')
<...>

If there are too many lines in an ICE plot, it can be difficult to see
differences between individual samples and interpret the model.
Centering the ICE at the first value on the x-axis, produces centered
Individual Conditional Expectation (cICE) plots [G2015]. This puts
emphasis on the divergence of individual conditional expectations from
the mean line, thus making it easier to explore heterogeneous
relationships. cICE plots can be plotted by setting *centered=True*:

...     kind='both', centered=True)
<...>

4.1.3. Mathematical Definition

Let X_S be the set of input features of interest (i.e. the *features*
parameter) and let X_C be its complement.

The partial dependence of the response f at a point x_S is defined as:

   pd_{X_S}(x_S) &\overset{def}{=} \mathbb{E}_{X_C}\left[ f(x_S, X_C)
   \right]\\               &= \int f(x_S, x_C) p(x_C) dx_C,

where f(x_S, x_C) is the response function (*predict*, *predict_proba*
or *decision_function*) for a given sample whose values are defined by
x_S for the features in X_S, and by x_C for the features in X_C. Note
that x_S and x_C may be tuples.

Computing this integral for various values of x_S produces a PDP plot
as above. An ICE line is defined as a single f(x_{S}, x_{C}^{(i)})
evaluated at x_{S}.

4.1.4. Computation methods

There are two main methods to approximate the integral above, namely
the 'brute' and 'recursion' methods. The *method* parameter controls
which method to use.

The 'brute' method is a generic method that works with any estimator.
Note that computing ICE plots is only supported with the 'brute'
method. It approximates the above integral by computing an average
over the data *X*:

   pd_{X_S}(x_S) \approx \frac{1}{n_\text{samples}} \sum_{i=1}^n
   f(x_S, x_C^{(i)}),

where x_C^{(i)} is the value of the i-th sample for the features in
X_C. For each value of x_S, this method requires a full pass over the
dataset *X* which is computationally intensive.

Each of the f(x_{S}, x_{C}^{(i)}) corresponds to one ICE line
evaluated at x_{S}. Computing this for multiple values of x_{S}, one
obtains a full ICE line. As one can see, the average of the ICE lines

correspond to the partial dependence line.

The 'recursion' method is faster than the 'brute' method, but it is
only supported for PDP plots by some tree-based estimators. It is
computed as follows. For a given point x_S, a weighted tree traversal
is performed: if a split node involves an input feature of interest,
the corresponding left or right branch is followed; otherwise both
branches are followed, each branch being weighted by the fraction of
training samples that entered that branch. Finally, the partial
dependence is given by a weighted average of all the visited leaves
values.

With the 'brute' method, the parameter *X* is used both for generating
the grid of values x_S and the complement feature values x_C. However
with the 'recursion' method, *X* is only used for the grid values:
implicitly, the x_C values are those of the training data.

By default, the 'recursion' method is used for plotting PDPs on tree-
based estimators that support it, and 'brute' is used for the rest.

Note:

  While both methods should be close in general, they might differ in
  some specific settings. The 'brute' method assumes the existence of
  the data points (x_S, x_C^{(i)}). When the features are correlated,
  such artificial samples may have a very low probability mass. The

the data points (x_S, x_C^{(i)}). When the features are correlated,
  'brute' and 'recursion' methods will likely disagree regarding the
  value of the partial dependence, because they will treat these
  unlikely samples differently. Remember, however, that the primary
  assumption for interpreting PDPs is that the features should be
  independent.

Examples:
^^^^^^^^^

[1] For classification, the target response may be the probability of
    a class (the positive class for binary classification), or the
    decision function.

[H2009] T. Hastie, R. Tibshirani and J. Friedman, The Elements of
        Statistical Learning, Second Edition, Section 10.13.2,
        Springer, 2009.

[M2019] C. Molnar, Interpretable Machine Learning, Section 5.1, 2019.

[G2015] >>:arxiv:`A. Goldstein, A. Kapelner, J. Bleich, and E. Pitkin,
        "Peeking Inside the Black Box: Visualizing Statistical
        Learning With Plots of Individual Conditional Expectation"
        Journal of Computational and Graphical Statistics, 24(1):
        44-65, Springer, 2015. <1309.6392>`<<

2.1. Gaussian mixture models

"sklearn.mixture" is a package which enables one to learn Gaussian
Mixture Models (diagonal, spherical, tied and full covariance matrices
supported), sample them, and estimate them from data. Facilities to
help determine the appropriate number of components are also provided.

      [image]**Two-component Gaussian mixture model:** *data points,
      and equi-probability surfaces of the model.*

A Gaussian mixture model is a probabilistic model that assumes all the
data points are generated from a mixture of a finite number of
Gaussian distributions with unknown parameters. One can think of
mixture models as generalizing k-means clustering to incorporate
information about the covariance structure of the data as well as the
centers of the latent Gaussians.

Scikit-learn implements different classes to estimate Gaussian mixture
models, that correspond to different estimation strategies, detailed
below.

2.1.1. Gaussian Mixture

The "GaussianMixture" object implements the expectation-maximization
(EM) algorithm for fitting mixture-of-Gaussian models. It can also
draw confidence ellipsoids for multivariate models, and compute the
Bayesian Information Criterion to assess the number of clusters in the
data. A "GaussianMixture.fit()" method is provided that learns a
Gaussian Mixture Model from train data. Given test data, it can assign
to each sample the Gaussian it most probably belongs to using the

"GaussianMixture.predict()" method.

The "GaussianMixture" comes with different options to constrain the
covariance of the difference classes estimated: spherical, diagonal,
tied or full covariance.

   [image]

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_mixture_plot_gmm_covariances.py* for an
  example of using the Gaussian mixture as clustering on the iris
  dataset.

* See *sphx_glr_auto_examples_mixture_plot_gmm_pdf.py* for an example
  on plotting the density estimation.

2.1.1.1. Pros and cons of class "GaussianMixture"

2.1.1.1.1. Pros
~~~~~~~~~~~~~~~

Speed:
   It is the fastest algorithm for learning mixture models

Agnostic:
   As this algorithm maximizes only the likelihood, it will not bias
   the means towards zero, or bias the cluster sizes to have specific
   structures that might or might not apply.

2.1.1.1.2. Cons

Singularities:
   When one has insufficiently many points per mixture, estimating the
   covariance matrices becomes difficult, and the algorithm is known
   to diverge and find solutions with infinite likelihood unless one
   regularizes the covariances artificially.

Number of components:
   This algorithm will always use all the components it has access to,

regularizes the covariances artificially.

   needing held-out data or information theoretical criteria to decide
   how many components to use in the absence of external cues.

2.1.1.2. Selecting the number of components in a classical Gaussian Mixture Model
---------------------------------------------------------------------------------

The BIC criterion can be used to select the number of components in a
Gaussian Mixture in an efficient way. In theory, it recovers the true
number of components only in the asymptotic regime (i.e. if much data
is available and assuming that the data was actually generated i.i.d.
from a mixture of Gaussian distribution). Note that using a
Variational Bayesian Gaussian mixture avoids the specification of the
number of components for a Gaussian mixture model.

   [image]

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_mixture_plot_gmm_selection.py* for an
  example of model selection performed with classical Gaussian
  mixture.

2.1.1.3. Estimation algorithm Expectation-maximization

The main difficulty in learning Gaussian mixture models from unlabeled
data is that one usually doesn't know which points came from which
latent component (if one has access to this information it gets very
easy to fit a separate Gaussian distribution to each set of points).
Expectation-maximization is a well-founded statistical algorithm to

get around this problem by an iterative process. First one assumes
random components (randomly centered on data points, learned from
k-means, or even just normally distributed around the origin) and
computes for each point a probability of being generated by each
component of the model. Then, one tweaks the parameters to maximize
the likelihood of the data given those assignments. Repeating this
process is guaranteed to always converge to a local optimum.

2.1.1.4. Choice of the Initialization Method

There is a choice of four initialization methods (as well as inputting
user defined initial means) to generate the initial centers for the
model components:

k-means (default)
   This applies a traditional k-means clustering algorithm. This can
   be computationally expensive compared to other initialization
   methods.

k-means++
   This uses the initialization method of k-means clustering:
   k-means++. This will pick the first center at random from the data.
   Subsequent centers will be chosen from a weighted distribution of
   the data favouring points further away from existing centers.
   k-means++ is the default initialization for k-means so will be
   quicker than running a full k-means but can still take a
   significant amount of time for large data sets with many
   components.

random_from_data

quicker than running a full k-means but can still take a

   This will pick random data points from the input data as the
   initial centers. This is a very fast method of initialization but
   can produce non-convergent results if the chosen points are too
   close to each other.

random
   Centers are chosen as a small perturbation away from the mean of
   all data. This method is simple but can lead to the model taking
   longer to converge.

   [image]

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_mixture_plot_gmm_init.py* for an example
  of using different initializations in Gaussian Mixture.

2.1.2. Variational Bayesian Gaussian Mixture

The "BayesianGaussianMixture" object implements a variant of the
Gaussian mixture model with variational inference algorithms. The API
is similar to the one defined by "GaussianMixture".

2.1.2.1. Estimation algorithm: variational inference

Variational inference is an extension of expectation-maximization that
maximizes a lower bound on model evidence (including priors) instead
of data likelihood. The principle behind variational methods is the
same as expectation-maximization (that is both are iterative
algorithms that alternate between finding the probabilities for each
point to be generated by each mixture and fitting the mixture to these

assigned points), but variational methods add regularization by
integrating information from prior distributions. This avoids the
singularities often found in expectation-maximization solutions but
introduces some subtle biases to the model. Inference is often notably
slower, but not usually as much so as to render usage unpractical.

Due to its Bayesian nature, the variational algorithm needs more
hyperparameters than expectation-maximization, the most important of
these being the concentration parameter "weight_concentration_prior".
Specifying a low value for the concentration prior will make the model
put most of the weight on a few components and set the remaining
components' weights very close to zero. High values of the
concentration prior will allow a larger number of components to be
active in the mixture.

The parameters implementation of the "BayesianGaussianMixture" class
proposes two types of prior for the weights distribution: a finite
mixture model with Dirichlet distribution and an infinite mixture
model with the Dirichlet Process. In practice Dirichlet Process
inference algorithm is approximated and uses a truncated distribution
with a fixed maximum number of components (called the Stick-breaking
representation). The number of components actually used almost always
depends on the data.

The next figure compares the results obtained for the different type
of the weight concentration prior (parameter
"weight_concentration_prior_type") for different values of
"weight_concentration_prior". Here, we can see the value of the
"weight_concentration_prior" parameter has a strong impact on the
effective number of active components obtained. We can also notice
that large values for the concentration weight prior lead to more
uniform weights when the type of prior is 'dirichlet_distribution'
while this is not necessarily the case for the 'dirichlet_process'
type (used by default).

[image: plot_bgmm][image] [image: plot_dpgmm][image]

The examples below compare Gaussian mixture models with a fixed number
of components, to the variational Gaussian mixture models with a
Dirichlet process prior. Here, a classical Gaussian mixture is fitted
with 5 components on a dataset composed of 2 clusters. We can see that
the variational Gaussian mixture with a Dirichlet process prior is
able to limit itself to only 2 components whereas the Gaussian mixture
fits the data with a fixed number of components that has to be set a
priori by the user. In this case the user has selected
"n_components=5" which does not match the true generative distribution
of this toy dataset. Note that with very little observations, the

variational Gaussian mixture models with a Dirichlet process prior can
take a conservative stand, and fit only one component.

   [image]

On the following figure we are fitting a dataset not well-depicted by
a Gaussian mixture. Adjusting the "weight_concentration_prior",
parameter of the "BayesianGaussianMixture" controls the number of
components used to fit this data. We also present on the last two
plots a random sampling generated from the two resulting mixtures.

   [image]

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_mixture_plot_gmm.py* for an example on
  plotting the confidence ellipsoids for both "GaussianMixture" and
  "BayesianGaussianMixture".

* *sphx_glr_auto_examples_mixture_plot_gmm_sin.py* shows using
  "GaussianMixture" and "BayesianGaussianMixture" to fit a sine wave.

* See *sphx_glr_auto_examples_mixture_plot_concentration_prior.py* for
  an example plotting the confidence ellipsoids for the
  "BayesianGaussianMixture" with different
  "weight_concentration_prior_type" for different values of the
  parameter "weight_concentration_prior".

2.1.2.2. Pros and cons of variational inference with "BayesianGaussianMixture"

2.1.2.2.1. Pros

Automatic selection:

   when "weight_concentration_prior" is small enough and
   "n_components" is larger than what is found necessary by the model,
   the Variational Bayesian mixture model has a natural tendency to
   set some mixture weights values close to zero. This makes it
   possible to let the model choose a suitable number of effective
   components automatically. Only an upper bound of this number needs
   to be provided. Note however that the "ideal" number of active
   components is very application specific and is typically ill-
   defined in a data exploration setting.

Less sensitivity to the number of parameters:
   unlike finite models, which will almost always use all components
   as much as they can, and hence will produce wildly different
   solutions for different numbers of components, the variational
   inference with a Dirichlet process prior
   ("weight_concentration_prior_type='dirichlet_process'") won't
   change much with changes to the parameters, leading to more
   stability and less tuning.

Regularization:
   due to the incorporation of prior information, variational
   solutions have less pathological special cases than expectation-
   maximization solutions.

2.1.2.2.2. Cons

Speed:
   the extra parametrization necessary for variational inference makes
   inference slower, although not by much.

Hyperparameters:

maximization solutions.

Speed:

   this algorithm needs an extra hyperparameter that might need
   experimental tuning via cross-validation.

Bias:
   there are many implicit biases in the inference algorithms (and
   also in the Dirichlet process if used), and whenever there is a
   mismatch between these biases and the data it might be possible to
   fit better models using a finite mixture.

2.1.2.3. The Dirichlet Process

Here we describe variational inference algorithms on Dirichlet process
mixture. The Dirichlet process is a prior probability distribution on
*clusterings with an infinite, unbounded, number of partitions*.
Variational techniques let us incorporate this prior structure on
Gaussian mixture models at almost no penalty in inference time,
comparing with a finite Gaussian mixture model.

An important question is how can the Dirichlet process use an
infinite, unbounded number of clusters and still be consistent. While
a full explanation doesn't fit this manual, one can think of its stick
breaking process analogy to help understanding it. The stick breaking
process is a generative story for the Dirichlet process. We start with
a unit-length stick and in each step we break off a portion of the
remaining stick. Each time, we associate the length of the piece of

the stick to the proportion of points that falls into a group of the
mixture. At the end, to represent the infinite mixture, we associate
the last remaining piece of the stick to the proportion of points that
don't fall into all the other groups. The length of each piece is a
random variable with probability proportional to the concentration
parameter. Smaller values of the concentration will divide the unit-
length into larger pieces of the stick (defining more concentrated
distribution). Larger concentration values will create smaller pieces
of the stick (increasing the number of components with non zero
weights).

Variational inference techniques for the Dirichlet process still work
with a finite approximation to this infinite mixture model, but
instead of having to specify a priori how many components one wants to
use, one just specifies the concentration parameter and an upper bound
on the number of mixture components (this upper bound, assuming it is
higher than the "true" number of components, affects only algorithmic
complexity, not the actual number of components used).

1.10. Decision Trees

**Decision Trees (DTs)** are a non-parametric supervised learning
method used for classification and regression. The goal is to create a
model that predicts the value of a target variable by learning simple
decision rules inferred from the data features. A tree can be seen as
a piecewise constant approximation.

For instance, in the example below, decision trees learn from data to
approximate a sine curve with a set of if-then-else decision rules.
The deeper the tree, the more complex the decision rules and the
fitter the model.

   [image]

Some advantages of decision trees are:

   * Simple to understand and to interpret. Trees can be visualized.

   * Requires little data preparation. Other techniques often require
     data normalization, dummy variables need to be created and blank
     values to be removed. Note however that this module does not
     support missing values.

   * The cost of using the tree (i.e., predicting data) is logarithmic
     in the number of data points used to train the tree.

   * Able to handle both numerical and categorical data. However, the
     scikit-learn implementation does not support categorical
     variables for now. Other techniques are usually specialized in
     analyzing datasets that have only one type of variable. See
     algorithms for more information.

   * Able to handle multi-output problems.

   * Uses a white box model. If a given situation is observable in a

algorithms for more information.

     model, the explanation for the condition is easily explained by
     boolean logic. By contrast, in a black box model (e.g., in an
     artificial neural network), results may be more difficult to
     interpret.

   * Possible to validate a model using statistical tests. That makes
     it possible to account for the reliability of the model.

   * Performs well even if its assumptions are somewhat violated by
     the true model from which the data were generated.

The disadvantages of decision trees include:

   * Decision-tree learners can create over-complex trees that do not
     generalize the data well. This is called overfitting. Mechanisms
     such as pruning, setting the minimum number of samples required
     at a leaf node or setting the maximum depth of the tree are
     necessary to avoid this problem.

   * Decision trees can be unstable because small variations in the
     data might result in a completely different tree being generated.
     This problem is mitigated by using decision trees within an
     ensemble.

   * Predictions of decision trees are neither smooth nor continuous,
     but piecewise constant approximations as seen in the above
     figure. Therefore, they are not good at extrapolation.

   * The problem of learning an optimal decision tree is known to be

but piecewise constant approximations as seen in the above

     NP-complete under several aspects of optimality and even for
     simple concepts. Consequently, practical decision-tree learning
     algorithms are based on heuristic algorithms such as the greedy
     algorithm where locally optimal decisions are made at each node.
     Such algorithms cannot guarantee to return the globally optimal
     decision tree.  This can be mitigated by training multiple trees
     in an ensemble learner, where the features and samples are
     randomly sampled with replacement.

   * There are concepts that are hard to learn because decision trees
     do not express them easily, such as XOR, parity or multiplexer
     problems.

   * Decision tree learners create biased trees if some classes
     dominate. It is therefore recommended to balance the dataset
     prior to fitting with the decision tree.

1.10.1. Classification

"DecisionTreeClassifier" is a class capable of performing multi-class
classification on a dataset.

As with other classifiers, "DecisionTreeClassifier" takes as input two
arrays: an array X, sparse or dense, of shape "(n_samples,
n_features)" holding the training samples, and an array Y of integer
values, shape "(n_samples,)", holding the class labels for the
training samples:

   >>> from sklearn import tree

   >>> clf = tree.DecisionTreeClassifier()

After being fitted, the model can then be used to predict the class of
samples:

In case that there are multiple classes with the same and highest
probability, the classifier will predict the class with the lowest
index amongst those classes.

As an alternative to outputting a specific class, the probability of
each class can be predicted, which is the fraction of training samples
of the class in a leaf:

   >>> clf.predict_proba([[2., 2.]])
   array([[0., 1.]])

"DecisionTreeClassifier" is capable of both binary (where the labels
are [-1, 1]) classification and multiclass (where the labels are [0,
..., K-1]) classification.

Using the Iris dataset, we can construct a tree as follows:

   >>> X, y = iris.data, iris.target

Once trained, you can plot the tree with the "plot_tree()" function:

   >>> tree.plot_tree(clf)
   [...]

   [image]

We can also export the tree in Graphviz format using the
"export_graphviz()" exporter. If you use the conda package manager,
the graphviz binaries and the python package can be installed with
*conda install python-graphviz*.

Alternatively binaries for graphviz can be downloaded from the
graphviz project homepage, and the Python wrapper installed from pypi
with *pip install graphviz*.

Below is an example graphviz export of the above tree trained on the
entire iris dataset; the results are saved in an output file
*iris.pdf*:

   >>> import graphviz 
   >>> dot_data = tree.export_graphviz(clf, out_file=None) 
   >>> graph = graphviz.Source(dot_data) 
   >>> graph.render("iris") 

The "export_graphviz()" exporter also supports a variety of aesthetic
options, including coloring nodes by their class (or value for
regression) and using explicit variable and class names if desired.
Jupyter notebooks also render these plots inline automatically:

   >>> dot_data = tree.export_graphviz(clf, out_file=None, 
   ...                      feature_names=iris.feature_names,  
   ...                      class_names=iris.target_names,  
   ...                      filled=True, rounded=True,  
   ...                      special_characters=True)  
   >>> graph = graphviz.Source(dot_data)  
   >>> graph 

   [image]

Alternatively, the tree can also be exported in textual format with
the function "export_text()". This method doesn't require the
installation of external libraries and is more compact:

>>> from sklearn.tree import export_text
>>> iris = load_iris()
>>> decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
>>> decision_tree = decision_tree.fit(iris.data, iris.target)
>>> r = export_text(decision_tree, feature_names=iris['feature_names'])
>>> print(r)
|--- petal width (cm) <= 0.80
|   |--- class: 0
|--- petal width (cm) >  0.80
|   |--- petal width (cm) <= 1.75
|   |   |--- class: 1
|   |--- petal width (cm) >  1.75
|   |   |--- class: 2

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_tree_plot_iris_dtc.py*

* *sphx_glr_auto_examples_tree_plot_unveil_tree_structure.py*

1.10.2. Regression

   [image]

Decision trees can also be applied to regression problems, using the
"DecisionTreeRegressor" class.

As in the classification setting, the fit method will take as argument
arrays X and y, only that in this case y is expected to have floating

   >>> clf = tree.DecisionTreeRegressor()
   >>> clf.predict([[1, 1]])
   array([0.5])

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_tree_plot_tree_regression.py*

1.10.3. Multi-output problems

Examples:
^^^^^^^^^

A multi-output problem is a supervised learning problem with several
outputs to predict, that is when Y is a 2d array of shape "(n_samples,
n_outputs)".

When there is no correlation between the outputs, a very simple way to
solve this kind of problem is to build n independent models, i.e. one
for each output, and then to use those models to independently predict
each one of the n outputs. However, because it is likely that the
output values related to the same input are themselves correlated, an
often better way is to build a single model capable of predicting
simultaneously all n outputs. First, it requires lower training time
since only a single estimator is built. Second, the generalization
accuracy of the resulting estimator may often be increased.

With regard to decision trees, this strategy can readily be used to
support multi-output problems. This requires the following changes:

   * Store n output values in leaves, instead of 1;

   * Use splitting criteria that compute the average reduction across
     all n outputs.

This module offers support for multi-output problems by implementing
this strategy in both "DecisionTreeClassifier" and
"DecisionTreeRegressor". If a decision tree is fit on an output array
Y of shape "(n_samples, n_outputs)" then the resulting estimator will:

   * Output n_output values upon "predict";

   * Output a list of n_output arrays of class probabilities upon
     "predict_proba".

The use of multi-output trees for regression is demonstrated in
*sphx_glr_auto_examples_tree_plot_tree_regression_multioutput.py*. In
this example, the input X is a single real value and the outputs Y are
the sine and cosine of X.

   [image]

The use of multi-output trees for classification is demonstrated in *
sphx_glr_auto_examples_miscellaneous_plot_multioutput_face_completion
.py*. In this example, the inputs X are the pixels of the upper half
of faces and the outputs Y are the pixels of the lower half of those
faces.

   [image]

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_tree_plot_tree_regression_multioutput.py*

  ion.py*

* M. Dumont et al,  Fast multi-class image annotation with random
  subwindows and multiple output randomized trees, International
  Conference on Computer Vision Theory and Applications 2009

1.10.4. Complexity

In general, the run time cost to construct a balanced binary tree is
O(n_{samples}n_{features}\log(n_{samples})) and query time
O(\log(n_{samples})).  Although the tree construction algorithm

attempts to generate balanced trees, they will not always be balanced.
Assuming that the subtrees remain approximately balanced, the cost at
each node consists of searching through O(n_{features}) to find the
feature that offers the largest reduction in the impurity criterion,
e.g. log loss (which is equivalent to an information gain). This has a
cost of O(n_{features}n_{samples}\log(n_{samples})) at each node,
leading to a total cost over the entire trees (by summing the cost at
each node) of O(n_{features}n_{samples}^{2}\log(n_{samples})).

1.10.5. Tips on practical use

   * Decision trees tend to overfit on data with a large number of
     features. Getting the right ratio of samples to number of
     features is important, since a tree with few samples in high
     dimensional space is very likely to overfit.

   * Consider performing  dimensionality reduction (PCA, ICA, or
     Feature selection) beforehand to give your tree a better chance
     of finding features that are discriminative.

   * *sphx_glr_auto_examples_tree_plot_unveil_tree_structure.py* will
     help in gaining more insights about how the decision tree makes
     predictions, which is important for understanding the important
     features in the data.

help in gaining more insights about how the decision tree makes

   * Visualize your tree as you are training by using the "export"
     function.  Use "max_depth=3" as an initial tree depth to get a
     feel for how the tree is fitting to your data, and then increase
     the depth.

   * Remember that the number of samples required to populate the tree
     doubles for each additional level the tree grows to.  Use
     "max_depth" to control the size of the tree to prevent
     overfitting.

   * Use "min_samples_split" or "min_samples_leaf" to ensure that
     multiple samples inform every decision in the tree, by
     controlling which splits will be considered. A very small number
     will usually mean the tree will overfit, whereas a large number
     will prevent the tree from learning the data. Try
     "min_samples_leaf=5" as an initial value. If the sample size
     varies greatly, a float number can be used as percentage in these
     two parameters. While "min_samples_split" can create arbitrarily
     small leaves, "min_samples_leaf" guarantees that each leaf has a
     minimum size, avoiding low-variance, over-fit leaf nodes in
     regression problems.  For classification with few classes,
     "min_samples_leaf=1" is often the best choice.

     Note that "min_samples_split" considers samples directly and
     independent of "sample_weight", if provided (e.g. a node with m

"min_samples_leaf=1" is often the best choice.

     weighted samples is still treated as having exactly m samples).
     Consider "min_weight_fraction_leaf" or "min_impurity_decrease" if
     accounting for sample weights is required at splits.

   * Balance your dataset before training to prevent the tree from
     being biased toward the classes that are dominant. Class
     balancing can be done by sampling an equal number of samples from
     each class, or preferably by normalizing the sum of the sample
     weights ("sample_weight") for each class to the same value. Also
     note that weight-based pre-pruning criteria, such as
     "min_weight_fraction_leaf", will then be less biased toward
     dominant classes than criteria that are not aware of the sample
     weights, like "min_samples_leaf".

   * If the samples are weighted, it will be easier to optimize the
     tree structure using weight-based pre-pruning criterion such as
     "min_weight_fraction_leaf", which ensure that leaf nodes contain
     at least a fraction of the overall sum of the sample weights.

   * All decision trees use "np.float32" arrays internally. If
     training data is not in this format, a copy of the dataset will
     be made.

   * If the input matrix X is very sparse, it is recommended to
     convert to sparse "csc_matrix" before calling fit and sparse

be made.

     "csr_matrix" before calling predict. Training time can be orders
     of magnitude faster for a sparse matrix input compared to a dense
     matrix when features have zero values in most of the samples.

1.10.6. Tree algorithms: ID3, C4.5, C5.0 and CART

What are all the various decision tree algorithms and how do they
differ from each other? Which one is implemented in scikit-learn?

ID3 (Iterative Dichotomiser 3) was developed in 1986 by Ross Quinlan.
The algorithm creates a multiway tree, finding for each node (i.e. in
a greedy manner) the categorical feature that will yield the largest
information gain for categorical targets. Trees are grown to their
maximum size and then a pruning step is usually applied to improve the
ability of the tree to generalize to unseen data.

C4.5 is the successor to ID3 and removed the restriction that features
must be categorical by dynamically defining a discrete attribute
(based on numerical variables) that partitions the continuous
attribute value into a discrete set of intervals. C4.5 converts the
trained trees (i.e. the output of the ID3 algorithm) into sets of if-
then rules. The accuracy of each rule is then evaluated to determine
the order in which they should be applied. Pruning is done by removing

a rule's precondition if the accuracy of the rule improves without it.

C5.0 is Quinlan's latest version release under a proprietary license.
It uses less memory and builds smaller rulesets than C4.5 while being
more accurate.

CART (Classification and Regression Trees) is very similar to C4.5,
but it differs in that it supports numerical target variables
(regression) and does not compute rule sets. CART constructs binary
trees using the feature and threshold that yield the largest
information gain at each node.

scikit-learn uses an optimized version of the CART algorithm; however,
the scikit-learn implementation does not support categorical variables
for now.

1.10.7. Mathematical formulation

Given training vectors x_i \in R^n, i=1,..., l and a label vector y
\in R^l, a decision tree recursively partitions the feature space such
that the samples with the same labels or similar target values are
grouped together.

Let the data at node m be represented by Q_m with n_m samples. For
each candidate split \theta = (j, t_m) consisting of a feature j and
threshold t_m, partition the data into Q_m^{left}(\theta) and
Q_m^{right}(\theta) subsets

   Q_m^{left}(\theta) = \{(x, y) | x_j \leq t_m\}  Q_m^{right}(\theta)
   = Q_m \setminus Q_m^{left}(\theta)

The quality of a candidate split of node m is then computed using an

Q_m^{left}(\theta) = \{(x, y) | x_j \leq t_m\}  Q_m^{right}(\theta)

impurity function or loss function H(), the choice of which depends on
the task being solved (classification or regression)

   G(Q_m, \theta) = \frac{n_m^{left}}{n_m} H(Q_m^{left}(\theta)) +
   \frac{n_m^{right}}{n_m} H(Q_m^{right}(\theta))

Select the parameters that minimises the impurity

   \theta^* = \operatorname{argmin}_\theta  G(Q_m, \theta)

Recurse for subsets Q_m^{left}(\theta^*) and Q_m^{right}(\theta^*)
until the maximum allowable depth is reached, n_m < \min_{samples} or
n_m = 1.

1.10.7.1. Classification criteria

If a target is a classification outcome taking on values 0,1,...,K-1,
for node m, let

   p_{mk} = \frac{1}{n_m} \sum_{y \in Q_m} I(y = k)

be the proportion of class k observations in node m. If m is a
terminal node, *predict_proba* for this region is set to p_{mk}.
Common measures of impurity are the following.

Gini:

   H(Q_m) = \sum_k p_{mk} (1 - p_{mk})

Log Loss or Entropy:

   H(Q_m) = - \sum_k p_{mk} \log(p_{mk})

Note:

  The entropy criterion computes the Shannon entropy of the possible
  classes. It takes the class frequencies of the training data points
  that reached a given leaf m as their probability. Using the
  **Shannon entropy as tree node splitting criterion is equivalent to
  minimizing the log loss** (also known as cross-entropy and

that reached a given leaf m as their probability. Using the
  multinomial deviance) between the true labels y_i and the
  probalistic predictions T_k(x_i) of the tree model T for class k.To
  see this, first recall that the log loss of a tree model T computed
  on a dataset D is defined as follows:

     \mathrm{LL}(D, T) = -\frac{1}{n} \sum_{(x_i, y_i) \in D} \sum_k
     I(y_i = k) \log(T_k(x_i))

  where D is a training dataset of n pairs (x_i, y_i).In a
  classification tree, the predicted class probabilities within leaf
  nodes are constant, that is: for all (x_i, y_i) \in Q_m, one has:
  T_k(x_i) = p_{mk} for each class k.This property makes it possible
  to rewrite \mathrm{LL}(D, T) as the sum of the Shannon entropies
  computed for each leaf of T weighted by the number of training data
  points that reached each leaf:

     \mathrm{LL}(D, T) = \sum_{m \in T} \frac{n_m}{n} H(Q_m)

1.10.7.2. Regression criteria

If the target is a continuous value, then for node m, common criteria
to minimize as for determining locations for future splits are Mean
Squared Error (MSE or L2 error), Poisson deviance as well as Mean
Absolute Error (MAE or L1 error). MSE and Poisson deviance both set
the predicted value of terminal nodes to the learned mean value
\bar{y}_m of the node whereas the MAE sets the predicted value of

terminal nodes to the median median(y)_m.

Mean Squared Error:

   \bar{y}_m = \frac{1}{n_m} \sum_{y \in Q_m} y  H(Q_m) =
   \frac{1}{n_m} \sum_{y \in Q_m} (y - \bar{y}_m)^2

Half Poisson deviance:

   H(Q_m) = \frac{1}{n_m} \sum_{y \in Q_m} (y \log\frac{y}{\bar{y}_m}
   - y + \bar{y}_m)

Setting *criterion="poisson"* might be a good choice if your target is
a count or a frequency (count per some unit). In any case, y >= 0 is a
necessary condition to use this criterion. Note that it fits much
slower than the MSE criterion.

Mean Absolute Error:

   median(y)_m = \underset{y \in Q_m}{\mathrm{median}}(y)  H(Q_m) =
   \frac{1}{n_m} \sum_{y \in Q_m} |y - median(y)_m|

Note that it fits much slower than the MSE criterion.

1.10.8. Minimal Cost-Complexity Pruning

Minimal cost-complexity pruning is an algorithm used to prune a tree
to avoid over-fitting, described in Chapter 3 of [BRE]. This algorithm
is parameterized by \alpha\ge0 known as the complexity parameter. The
complexity parameter is used to define the cost-complexity measure,
R_\alpha(T) of a given tree T:

   R_\alpha(T) = R(T) + \alpha|\widetilde{T}|

where |\widetilde{T}| is the number of terminal nodes in T and R(T) is
traditionally defined as the total misclassification rate of the

R_\alpha(T) = R(T) + \alpha|\widetilde{T}|

terminal nodes. Alternatively, scikit-learn uses the total sample
weighted impurity of the terminal nodes for R(T). As shown above, the
impurity of a node depends on the criterion. Minimal cost-complexity
pruning finds the subtree of T that minimizes R_\alpha(T).

The cost complexity measure of a single node is
R_\alpha(t)=R(t)+\alpha. The branch, T_t, is defined to be a tree
where node t is its root. In general, the impurity of a node is
greater than the sum of impurities of its terminal nodes, R(T_t)<R(t).
However, the cost complexity measure of a node, t, and its branch,
T_t, can be equal depending on \alpha. We define the effective \alpha
of a node to be the value where they are equal,
R_\alpha(T_t)=R_\alpha(t) or
\alpha_{eff}(t)=\frac{R(t)-R(T_t)}{|T|-1}. A non-terminal node with
the smallest value of \alpha_{eff} is the weakest link and will be
pruned. This process stops when the pruned tree's minimal \alpha_{eff}
is greater than the "ccp_alpha" parameter.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py*

[BRE] L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification
      and Regression Trees. Wadsworth, Belmont, CA, 1984.

* https://en.wikipedia.org/wiki/Decision_tree_learning

* https://en.wikipedia.org/wiki/Predictive_analytics

and Regression Trees. Wadsworth, Belmont, CA, 1984.

* J.R. Quinlan. C4. 5: programs for machine learning. Morgan Kaufmann,
  1993.

* T. Hastie, R. Tibshirani and J. Friedman. Elements of Statistical
  Learning, Springer, 2009.

1.8. Cross decomposition

The cross decomposition module contains **supervised** estimators for
dimensionality reduction and regression, belonging to the "Partial
Least Squares" family.

   [image]

Cross decomposition algorithms find the fundamental relations between
two matrices (X and Y). They are latent variable approaches to
modeling the covariance structures in these two spaces. They will try
to find the multidimensional direction in the X space that explains
the maximum multidimensional variance direction in the Y space. In
other words, PLS projects both *X* and *Y* into a lower-dimensional
subspace such that the covariance between *transformed(X)* and
*transformed(Y)* is maximal.

PLS draws similarities with Principal Component Regression (PCR),
where the samples are first projected into a lower-dimensional
subspace, and the targets *y* are predicted using *transformed(X)*.
One issue with PCR is that the dimensionality reduction is
unsupervized, and may lose some important variables: PCR would keep
the features with the most variance, but it's possible that features
with a small variances are relevant from predicting the target. In a
way, PLS allows for the same kind of dimensionality reduction, but by
taking into account the targets *y*. An illustration of this fact is
given in the following example: *
*sphx_glr_auto_examples_cross_decomposition_plot_pcr_vs_pls.py*.

Apart from CCA, the PLS estimators are particularly suited when the

matrix of predictors has more variables than observations, and when
there is multicollinearity among the features. By contrast, standard
linear regression would fail in these cases unless it is regularized.

Classes included in this module are "PLSRegression", "PLSCanonical",
"CCA" and "PLSSVD"

1.8.1. PLSCanonical

We here describe the algorithm used in "PLSCanonical". The other
estimators use variants of this algorithm, and are detailed below. We
recommend section [1] for more details and comparisons between these
algorithms. In [1], "PLSCanonical" corresponds to "PLSW2A".

Given two centered matrices X \in \mathbb{R}^{n \times d} and Y \in
\mathbb{R}^{n \times t}, and a number of components K, "PLSCanonical"
proceeds as follows:

Set X_1 to X and Y_1 to Y. Then, for each k \in [1, K]:

* a) compute u_k \in \mathbb{R}^d and v_k \in \mathbb{R}^t, the first
  left and right singular vectors of the cross-covariance matrix C =
  X_k^T Y_k. u_k and v_k are called the *weights*. By definition, u_k
  and v_k are chosen so that they maximize the covariance between the
  projected X_k and the projected target, that is \text{Cov}(X_k u_k,
  Y_k v_k).

* b) Project X_k and Y_k on the singular vectors to obtain *scores*:
  \xi_k = X_k u_k and \omega_k = Y_k v_k

* c) Regress X_k on \xi_k, i.e. find a vector \gamma_k \in

Y_k v_k).

  \mathbb{R}^d such that the rank-1 matrix \xi_k \gamma_k^T is as
  close as possible to X_k. Do the same on Y_k with \omega_k to obtain
  \delta_k. The vectors \gamma_k and \delta_k are called the
  *loadings*.

* d) *deflate* X_k and Y_k, i.e. subtract the rank-1 approximations:
  X_{k+1} = X_k - \xi_k \gamma_k^T, and Y_{k + 1} = Y_k - \omega_k
  \delta_k^T.

At the end, we have approximated X as a sum of rank-1 matrices: X =
\Xi \Gamma^T where \Xi \in \mathbb{R}^{n \times K} contains the scores
in its columns, and \Gamma^T \in \mathbb{R}^{K \times d} contains the
loadings in its rows. Similarly for Y, we have Y = \Omega \Delta^T.

Note that the scores matrices \Xi and \Omega correspond to the
projections of the training data X and Y, respectively.

Step *a)* may be performed in two ways: either by computing the whole
SVD of C and only retain the singular vectors with the biggest
singular values, or by directly computing the singular vectors using
the power method (cf section 11.3 in [1]), which corresponds to the
*'nipals'* option of the *algorithm* parameter.

1.8.1.1. Transforming data

To transform X into \bar{X}, we need to find a projection matrix P
such that \bar{X} = XP. We know that for the training data, \Xi = XP,
and X = \Xi \Gamma^T. Setting P = U(\Gamma^T U)^{-1} where U is the

matrix with the u_k in the columns, we have XP = X U(\Gamma^T U)^{-1}
= \Xi (\Gamma^T U) (\Gamma^T U)^{-1} = \Xi as desired. The rotation
matrix P can be accessed from the *x_rotations_* attribute.

Similarly, Y can be transformed using the rotation matrix V(\Delta^T
V)^{-1}, accessed via the *y_rotations_* attribute.

1.8.1.2. Predicting the targets Y

To predict the targets of some data X, we are looking for a
coefficient matrix \beta \in R^{d \times t} such that Y = X\beta.

The idea is to try to predict the transformed targets \Omega as a
function of the transformed samples \Xi, by computing \alpha \in
\mathbb{R} such that \Omega = \alpha \Xi.

Then, we have Y = \Omega \Delta^T = \alpha \Xi \Delta^T, and since \Xi
is the transformed training data we have that Y = X \alpha P \Delta^T,
and as a result the coefficient matrix \beta = \alpha P \Delta^T.

\beta can be accessed through the *coef_* attribute.

1.8.2. PLSSVD

"PLSSVD" is a simplified version of "PLSCanonical" described earlier:
instead of iteratively deflating the matrices X_k and Y_k, "PLSSVD"
computes the SVD of C = X^TY only *once*, and stores the
*n_components* singular vectors corresponding to the biggest singular
values in the matrices *U* and *V*, corresponding to the *x_weights_*
and *y_weights_* attributes. Here, the transformed data is simply

*transformed(X) = XU* and *transformed(Y) = YV*.

If *n_components == 1*, "PLSSVD" and "PLSCanonical" are strictly

1.8.3. PLSRegression

The "PLSRegression" estimator is similar to "PLSCanonical" with
*algorithm='nipals'*, with 2 significant differences:

* at step a) in the power method to compute u_k and v_k, v_k is never
  normalized.

* at step c), the targets Y_k are approximated using the projection of
  X_k (i.e. \xi_k) instead of the projection of Y_k (i.e. \omega_k).
  In other words, the loadings computation is different. As a result,
  the deflation in step d) will also be affected.

These two modifications affect the output of *predict* and
*transform*, which are not the same as for "PLSCanonical". Also, while
the number of components is limited by *min(n_samples, n_features,
n_targets)* in "PLSCanonical", here the limit is the rank of X^TX,
i.e. *min(n_samples, n_features)*.

"PLSRegression" is also known as PLS1 (single targets) and PLS2
(multiple targets). Much like "Lasso", "PLSRegression" is a form of
regularized linear regression where the number of components controls
the strength of the regularization.

1.8.4. Canonical Correlation Analysis

Canonical Correlation Analysis was developed prior and independently
to PLS. But it turns out that "CCA" is a special case of PLS, and

corresponds to PLS in "Mode B" in the literature.

"CCA" differs from "PLSCanonical" in the way the weights u_k and v_k
are computed in the power method of step a). Details can be found in
section 10 of [1].

Since "CCA" involves the inversion of X_k^TX_k and Y_k^TY_k, this
estimator can be unstable if the number of features or targets is
greater than the number of samples.

Reference:

[1] A survey of Partial Least Squares (PLS) methods, with emphasis on
    the two-block case JA Wegelin

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_cross_decomposition_plot_compare_cross_deco
  mposition.py*

* *sphx_glr_auto_examples_cross_decomposition_plot_pcr_vs_pls.py*

1.9. Naive Bayes

Naive Bayes methods are a set of supervised learning algorithms based
on applying Bayes' theorem with the "naive" assumption of conditional
independence between every pair of features given the value of the
class variable. Bayes' theorem states the following relationship,
given class variable y and dependent feature vector x_1 through x_n, :

   P(y \mid x_1, \dots, x_n) = \frac{P(y) P(x_1, \dots, x_n \mid y)}
   {P(x_1, \dots, x_n)}

Using the naive conditional independence assumption that

   P(x_i | y, x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n) = P(x_i | y),

for all i, this relationship is simplified to

   P(y \mid x_1, \dots, x_n) = \frac{P(y) \prod_{i=1}^{n} P(x_i \mid
   y)}                                  {P(x_1, \dots, x_n)}

Since P(x_1, \dots, x_n) is constant given the input, we can use the
following classification rule:

   P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid
   y)  \Downarrow  \hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i
   \mid y),

and we can use Maximum A Posteriori (MAP) estimation to estimate P(y)
and P(x_i \mid y); the former is then the relative frequency of class
y in the training set.

The different naive Bayes classifiers differ mainly by the assumptions
they make regarding the distribution of P(x_i \mid y).

In spite of their apparently over-simplified assumptions, naive Bayes
classifiers have worked quite well in many real-world situations,
famously document classification and spam filtering. They require a

small amount of training data to estimate the necessary parameters.
(For theoretical reasons why naive Bayes works well, and on which
types of data it does, see the references below.)

Naive Bayes learners and classifiers can be extremely fast compared to
more sophisticated methods. The decoupling of the class conditional
feature distributions means that each distribution can be
independently estimated as a one dimensional distribution. This in
turn helps to alleviate problems stemming from the curse of
dimensionality.

On the flip side, although naive Bayes is known as a decent
classifier, it is known to be a bad estimator, so the probability
outputs from "predict_proba" are not to be taken too seriously.

* H. Zhang (2004). The optimality of Naive Bayes. Proc. FLAIRS.

1.9.1. Gaussian Naive Bayes

"GaussianNB" implements the Gaussian Naive Bayes algorithm for
classification. The likelihood of the features is assumed to be
Gaussian:

   P(x_i \mid y) = \frac{1}{\sqrt{2\pi\sigma^2_y}}
   \exp\left(-\frac{(x_i - \mu_y)^2}{2\sigma^2_y}\right)

The parameters \sigma_y and \mu_y are estimated using maximum
likelihood.

>>> from sklearn.naive_bayes import GaussianNB

>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
>>> gnb = GaussianNB()
>>> y_pred = gnb.fit(X_train, y_train).predict(X_test)
>>> print("Number of mislabeled points out of a total %d points : %d"
...       % (X_test.shape[0], (y_test != y_pred).sum()))
Number of mislabeled points out of a total 75 points : 4

1.9.2. Multinomial Naive Bayes

"MultinomialNB" implements the naive Bayes algorithm for multinomially
distributed data, and is one of the two classic naive Bayes variants
used in text classification (where the data are typically represented
as word vector counts, although tf-idf vectors are also known to work
well in practice). The distribution is parametrized by vectors
\theta_y = (\theta_{y1},\ldots,\theta_{yn}) for each class y, where n
is the number of features (in text classification, the size of the
vocabulary) and \theta_{yi} is the probability P(x_i \mid y) of
feature i appearing in a sample belonging to class y.

The parameters \theta_y is estimated by a smoothed version of maximum
likelihood, i.e. relative frequency counting:

   \hat{\theta}_{yi} = \frac{ N_{yi} + \alpha}{N_y + \alpha n}

where N_{yi} = \sum_{x \in T} x_i is the number of times feature i
appears in a sample of class y in the training set T, and N_{y} =

\hat{\theta}_{yi} = \frac{ N_{yi} + \alpha}{N_y + \alpha n}

\sum_{i=1}^{n} N_{yi} is the total count of all features for class y.

The smoothing priors \alpha \ge 0 accounts for features not present in
the learning samples and prevents zero probabilities in further
computations. Setting \alpha = 1 is called Laplace smoothing, while
\alpha < 1 is called Lidstone smoothing.

1.9.3. Complement Naive Bayes

"ComplementNB" implements the complement naive Bayes (CNB) algorithm.
CNB is an adaptation of the standard multinomial naive Bayes (MNB)
algorithm that is particularly suited for imbalanced data sets.
Specifically, CNB uses statistics from the *complement* of each class
to compute the model's weights. The inventors of CNB show empirically
that the parameter estimates for CNB are more stable than those for
MNB. Further, CNB regularly outperforms MNB (often by a considerable
margin) on text classification tasks. The procedure for calculating
the weights is as follows:

   \hat{\theta}_{ci} = \frac{\alpha_i + \sum_{j:y_j \neq c} d_{ij}}
   {\alpha + \sum_{j:y_j \neq c} \sum_{k} d_{kj}}  w_{ci} = \log
   \hat{\theta}_{ci}  w_{ci} = \frac{w_{ci}}{\sum_{j} |w_{cj}|}

where the summations are over all documents j not in class c, d_{ij}
is either the count or tf-idf value of term i in document j, \alpha_i

\hat{\theta}_{ci}  w_{ci} = \frac{w_{ci}}{\sum_{j} |w_{cj}|}

is a smoothing hyperparameter like that found in MNB, and \alpha =
\sum_{i} \alpha_i. The second normalization addresses the tendency for
longer documents to dominate parameter estimates in MNB. The
classification rule is:

   \hat{c} = \arg\min_c \sum_{i} t_i w_{ci}

i.e., a document is assigned to the class that is the *poorest*
complement match.

* Rennie, J. D., Shih, L., Teevan, J., & Karger, D. R. (2003).
  Tackling the poor assumptions of naive bayes text classifiers. In
  ICML (Vol. 3, pp. 616-623).

1.9.4. Bernoulli Naive Bayes

"BernoulliNB" implements the naive Bayes training and classification
algorithms for data that is distributed according to multivariate
Bernoulli distributions; i.e., there may be multiple features but each
one is assumed to be a binary-valued (Bernoulli, boolean) variable.
Therefore, this class requires samples to be represented as binary-
valued feature vectors; if handed any other kind of data, a
"BernoulliNB" instance may binarize its input (depending on the
"binarize" parameter).

The decision rule for Bernoulli naive Bayes is based on

   P(x_i \mid y) = P(x_i = 1 \mid y) x_i + (1 - P(x_i = 1 \mid y)) (1
   - x_i)

which differs from multinomial NB's rule in that it explicitly

   - x_i)

penalizes the non-occurrence of a feature i that is an indicator for
class y, where the multinomial variant would simply ignore a non-
occurring feature.

In the case of text classification, word occurrence vectors (rather
than word count vectors) may be used to train and use this classifier.
"BernoulliNB" might perform better on some datasets, especially those
with shorter documents. It is advisable to evaluate both models, if
time permits.

* C.D. Manning, P. Raghavan and H. Schütze (2008). Introduction to
  Information Retrieval. Cambridge University Press, pp. 234-265.

* A. McCallum and K. Nigam (1998). A comparison of event models for
  Naive Bayes text classification. Proc. AAAI/ICML-98 Workshop on
  Learning for Text Categorization, pp. 41-48.

* V. Metsis, I. Androutsopoulos and G. Paliouras (2006). Spam
  filtering with Naive Bayes -- Which Naive Bayes? 3rd Conf. on Email
  and Anti-Spam (CEAS).

1.9.5. Categorical Naive Bayes

"CategoricalNB" implements the categorical naive Bayes algorithm for
categorically distributed data. It assumes that each feature, which is
described by the index i, has its own categorical distribution.

For each feature i in the training set X, "CategoricalNB" estimates a

categorical distribution for each feature i of X conditioned on the
class y. The index set of the samples is defined as J = \{ 1, \dots, m
\}, with m as the number of samples.

The probability of category t in feature i given class c is estimated
as:

   P(x_i = t \mid y = c \: ;\, \alpha) = \frac{ N_{tic} +
   \alpha}{N_{c} +                                        \alpha n_i},

where N_{tic} = |\{j \in J \mid x_{ij} = t, y_j = c\}| is the number
of times category t appears in the samples x_{i}, which belong to
class c, N_{c} = |\{ j \in J\mid y_j = c\}| is the number of samples
with class c, \alpha is a smoothing parameter and n_i is the number of
available categories of feature i.

"CategoricalNB" assumes that the sample matrix X is encoded (for
instance with the help of "OrdinalEncoder") such that all categories
for each feature i are represented with numbers 0, ..., n_i - 1 where
n_i is the number of available categories of feature i.

1.9.6. Out-of-core naive Bayes model fitting

Naive Bayes models can be used to tackle large scale classification
problems for which the full training set might not fit in memory. To
handle this case, "MultinomialNB", "BernoulliNB", and "GaussianNB"
expose a "partial_fit" method that can be used incrementally as done
with other classifiers as demonstrated in *sphx_glr_auto_examples_app

lications_plot_out_of_core_classification.py*. All naive Bayes
classifiers support sample weighting.

Contrary to the "fit" method, the first call to "partial_fit" needs to
be passed the list of all the expected class labels.

For an overview of available strategies in scikit-learn, see also the
out-of-core learning documentation.

Note:

  The "partial_fit" method call of naive Bayes models introduces some
  computational overhead. It is recommended to use data chunk sizes
  that are as large as possible, that is as the available RAM allows.

2.4. Biclustering

Biclustering can be performed with the module
"sklearn.cluster.bicluster". Biclustering algorithms simultaneously
cluster rows and columns of a data matrix. These clusters of rows and
columns are known as biclusters. Each determines a submatrix of the
original data matrix with some desired properties.

For instance, given a matrix of shape "(10, 10)", one possible
bicluster with three rows and two columns induces a submatrix of shape
"(3, 2)":

   >>> data = np.arange(100).reshape(10, 10)
   >>> rows = np.array([0, 2, 3])[:, np.newaxis]
   >>> columns = np.array([1, 2])
   >>> data[rows, columns]
   array([[ 1,  2],
          [21, 22],
          [31, 32]])

For visualization purposes, given a bicluster, the rows and columns of
the data matrix may be rearranged to make the bicluster contiguous.

Algorithms differ in how they define biclusters. Some of the common
types include:

* constant values, constant rows, or constant columns

* unusually high or low values

* submatrices with low variance

* correlated rows or columns

Algorithms also differ in how rows and columns may be assigned to
biclusters, which leads to different bicluster structures. Block
diagonal or checkerboard structures occur when rows and columns are
divided into partitions.

If each row and each column belongs to exactly one bicluster, then
rearranging the rows and columns of the data matrix reveals the
biclusters on the diagonal. Here is an example of this structure where

biclusters have higher average values than the other rows and columns:

   [image]An example of biclusters formed by partitioning rows and
   columns.

In the checkerboard case, each row belongs to all column clusters, and
each column belongs to all row clusters. Here is an example of this
structure where the variance of the values within each bicluster is
small:

   [image]An example of checkerboard biclusters.

After fitting a model, row and column cluster membership can be found
in the "rows_" and "columns_" attributes. "rows_[i]" is a binary
vector with nonzero entries corresponding to rows that belong to
bicluster "i". Similarly, "columns_[i]" indicates which columns belong
to bicluster "i".

Some models also have "row_labels_" and "column_labels_" attributes.
These models partition the rows and columns, such as in the block
diagonal and checkerboard bicluster structures.

Note:

  Biclustering has many other names in different fields including co-
  clustering, two-mode clustering, two-way clustering, block
  clustering, coupled two-way clustering, etc. The names of some
  algorithms, such as the Spectral Co-Clustering algorithm, reflect
  these alternate names.

2.4.1. Spectral Co-Clustering

The "SpectralCoclustering" algorithm finds biclusters with values

these alternate names.

higher than those in the corresponding other rows and columns. Each
row and each column belongs to exactly one bicluster, so rearranging
the rows and columns to make partitions contiguous reveals these high
values along the diagonal:

Note:

  The algorithm treats the input data matrix as a bipartite graph: the
  rows and columns of the matrix correspond to the two sets of
  vertices, and each entry corresponds to an edge between a row and a
  column. The algorithm approximates the normalized cut of this graph
  to find heavy subgraphs.

2.4.1.1. Mathematical formulation

An approximate solution to the optimal normalized cut may be found via
the generalized eigenvalue decomposition of the Laplacian of the
graph. Usually this would mean working directly with the Laplacian
matrix. If the original data matrix A has shape m \times n, the
Laplacian matrix for the corresponding bipartite graph has shape (m +
n) \times (m + n). However, in this case it is possible to work
directly with A, which is smaller and more efficient.

The input matrix A is preprocessed as follows:

   A_n = R^{-1/2} A C^{-1/2}

Where R is the diagonal matrix with entry i equal to \sum_{j} A_{ij}
and C is the diagonal matrix with entry j equal to \sum_{i} A_{ij}.

The singular value decomposition, A_n = U \Sigma V^\top, provides the

partitions of the rows and columns of A. A subset of the left singular
vectors gives the row partitions, and a subset of the right singular
vectors gives the column partitions.

The \ell = \lceil \log_2 k \rceil singular vectors, starting from the
second, provide the desired partitioning information. They are used to
form the matrix Z:

   Z = \begin{bmatrix} R^{-1/2} U \\\\                     C^{-1/2} V
   \end{bmatrix}

where the columns of U are u_2, \dots, u_{\ell + 1}, and similarly for
V.

Then the rows of Z are clustered using k-means. The first "n_rows"
labels provide the row partitioning, and the remaining "n_columns"
labels provide the column partitioning.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_bicluster_plot_spectral_coclustering.py*: A
  simple example showing how to generate a data matrix with biclusters
  and apply this method to it.

* *sphx_glr_auto_examples_bicluster_plot_bicluster_newsgroups.py*: An
  example of finding biclusters in the twenty newsgroup dataset.

* Dhillon, Inderjit S, 2001. >>:doi:`Co-clustering documents and words
  using bipartite spectral graph partitioning
  <10.1145/502512.502550>`<<

2.4.2. Spectral Biclustering

The "SpectralBiclustering" algorithm assumes that the input data
matrix has a hidden checkerboard structure. The rows and columns of a

matrix with this structure may be partitioned so that the entries of
any bicluster in the Cartesian product of row clusters and column
clusters are approximately constant. For instance, if there are two
row partitions and three column partitions, each row will belong to
three biclusters, and each column will belong to two biclusters.

The algorithm partitions the rows and columns of a matrix so that a
corresponding blockwise-constant checkerboard matrix provides a good
approximation to the original matrix.

2.4.2.1. Mathematical formulation

The input matrix A is first normalized to make the checkerboard
pattern more obvious. There are three possible methods:

1. *Independent row and column normalization*, as in Spectral Co-
   Clustering. This method makes the rows sum to a constant and the
   columns sum to a different constant.

2. **Bistochastization**: repeated row and column normalization until
   convergence. This method makes both rows and columns sum to the
   same constant.

3. **Log normalization**: the log of the data matrix is computed: L =
   \log A. Then the column mean \overline{L_{i \cdot}}, row mean
   \overline{L_{\cdot j}}, and overall mean \overline{L_{\cdot \cdot}}
   of L are computed. The final matrix is computed according to the
   formula

\overline{L_{\cdot j}}, and overall mean \overline{L_{\cdot \cdot}}
   formula

   K_{ij} = L_{ij} - \overline{L_{i \cdot}} - \overline{L_{\cdot j}} +
   \overline{L_{\cdot \cdot}}

After normalizing, the first few singular vectors are computed, just
as in the Spectral Co-Clustering algorithm.

If log normalization was used, all the singular vectors are
meaningful. However, if independent normalization or bistochastization
were used, the first singular vectors, u_1 and v_1. are discarded.
From now on, the "first" singular vectors refers to u_2 \dots u_{p+1}
and v_2 \dots v_{p+1} except in the case of log normalization.

Given these singular vectors, they are ranked according to which can
be best approximated by a piecewise-constant vector. The
approximations for each vector are found using one-dimensional k-means
and scored using the Euclidean distance. Some subset of the best left
and right singular vector are selected. Next, the data is projected to
this best subset of singular vectors and clustered.

For instance, if p singular vectors were calculated, the q best are
found as described, where q<p. Let U be the matrix with columns the q
best left singular vectors, and similarly V for the right. To
partition the rows, the rows of A are projected to a q dimensional
space: A * V. Treating the m rows of this m \times q matrix as samples
and clustering using k-means yields the row labels. Similarly,

projecting the columns to A^{\top} * U and clustering this n \times q
matrix yields the column labels.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_bicluster_plot_spectral_biclustering.py*: a
  simple example showing how to generate a checkerboard matrix and
  bicluster it.

* Kluger, Yuval, et. al., 2003. >>:doi:`Spectral biclustering of
  microarray data: coclustering genes and conditions
  <10.1101/gr.648603>`<<

2.4.3. Biclustering evaluation

There are two ways of evaluating a biclustering result: internal and
external. Internal measures, such as cluster stability, rely only on
the data and the result themselves. Currently there are no internal
bicluster measures in scikit-learn. External measures refer to an
external source of information, such as the true solution. When
working with real data the true solution is usually unknown, but
biclustering artificial data may be useful for evaluating algorithms
precisely because the true solution is known.

To compare a set of found biclusters to the set of true biclusters,
two similarity measures are needed: a similarity measure for
individual biclusters, and a way to combine these individual
similarities into an overall score.

To compare individual biclusters, several measures have been used. For

now, only the Jaccard index is implemented:

   J(A, B) = \frac{|A \cap B|}{|A| + |B| - |A \cap B|}

where A and B are biclusters, |A \cap B| is the number of elements in
their intersection. The Jaccard index achieves its minimum of 0 when
the biclusters to not overlap at all and its maximum of 1 when they
are identical.

Several methods have been developed to compare two sets of biclusters.
For now, only "consensus_score()" (Hochreiter et. al., 2010) is
available:

1. Compute bicluster similarities for pairs of biclusters, one in each
   set, using the Jaccard index or a similar measure.

2. Assign biclusters from one set to another in a one-to-one fashion
   to maximize the sum of their similarities. This step is performed
   using the Hungarian algorithm.

3. The final sum of similarities is divided by the size of the larger
   set.

The minimum consensus score, 0, occurs when all pairs of biclusters
are totally dissimilar. The maximum score, 1, occurs when both sets

* Hochreiter, Bodenhofer, et. al., 2010. FABIA: factor analysis for
  bicluster acquisition.

6.8. Pairwise metrics, Affinities and Kernels

The "sklearn.metrics.pairwise" submodule implements utilities to
evaluate pairwise distances or affinity of sets of samples.

This module contains both distance metrics and kernels. A brief
summary is given on the two here.

Distance metrics are functions "d(a, b)" such that "d(a, b) < d(a, c)"
if objects "a" and "b" are considered "more similar" than objects "a"
and "c". Two objects exactly alike would have a distance of zero. One
of the most popular examples is Euclidean distance. To be a 'true'
metric, it must obey the following four conditions:

   1. d(a, b) >= 0, for all a and b
   2. d(a, b) == 0, if and only if a = b, positive definiteness
   3. d(a, b) == d(b, a), symmetry
   4. d(a, c) <= d(a, b) + d(b, c), the triangle inequality

Kernels are measures of similarity, i.e. "s(a, b) > s(a, c)" if
objects "a" and "b" are considered "more similar" than objects "a" and
"c". A kernel must also be positive semi-definite.

There are a number of ways to convert between a distance metric and a
similarity measure, such as a kernel. Let "D" be the distance, and "S"
be the kernel:

   1. "S = np.exp(-D * gamma)", where one heuristic for choosing
      "gamma" is "1 / num_features"

   2. "S = 1. / (D / np.max(D))"

The distances between the row vectors of "X" and the row vectors of
"Y" can be evaluated using "pairwise_distances()". If "Y" is omitted
the pairwise distances of the row vectors of "X" are calculated.

Similarly, "pairwise.pairwise_kernels()" can be used to calculate the
kernel between *X* and *Y* using different kernel functions. See the
API reference for more details.

>>> from sklearn.metrics.pairwise import pairwise_kernels
>>> X = np.array([[2, 3], [3, 5], [5, 8]])
>>> Y = np.array([[1, 0], [2, 1]])
>>> pairwise_distances(X, Y, metric='manhattan')
array([[ 4.,  2.],
       [ 7.,  5.],
       [12., 10.]])
>>> pairwise_distances(X, metric='manhattan')
array([[0., 3., 8.],
       [3., 0., 5.],
       [8., 5., 0.]])
>>> pairwise_kernels(X, Y, metric='linear')
array([[ 2.,  7.],
       [ 3., 11.],
       [ 5., 18.]])

6.8.1. Cosine similarity

"cosine_similarity()" computes the L2-normalized dot product of
vectors. That is, if x and y are row vectors, their cosine similarity
k is defined as:

   k(x, y) = \frac{x y^\top}{\|x\| \|y\|}

This is called cosine similarity, because Euclidean (L2) normalization
projects the vectors onto the unit sphere, and their dot product is
then the cosine of the angle between the points denoted by the
vectors.

This kernel is a popular choice for computing the similarity of
documents represented as tf-idf vectors. "cosine_similarity()" accepts

vectors.

"scipy.sparse" matrices. (Note that the tf-idf functionality in
"sklearn.feature_extraction.text" can produce normalized vectors, in
which case "cosine_similarity()" is equivalent to "linear_kernel()",
only slower.)

  Information Retrieval. Cambridge University Press.
  https://nlp.stanford.edu/IR-book/html/htmledition/the-vector-space-
  model-for-scoring-1.html

6.8.2. Linear kernel

The function "linear_kernel()" computes the linear kernel, that is, a
special case of "polynomial_kernel()" with "degree=1" and "coef0=0"
(homogeneous). If "x" and "y" are column vectors, their linear kernel
is:

   k(x, y) = x^\top y

6.8.3. Polynomial kernel

The function "polynomial_kernel()" computes the degree-d polynomial
kernel between two vectors. The polynomial kernel represents the
similarity between two vectors. Conceptually, the polynomial kernels
considers not only the similarity between vectors under the same
dimension, but also across dimensions. When used in machine learning
algorithms, this allows to account for feature interaction.

The polynomial kernel is defined as:

where:

If c_0 = 0 the kernel is said to be homogeneous.

where:

6.8.4. Sigmoid kernel

The function "sigmoid_kernel()" computes the sigmoid kernel between
two vectors. The sigmoid kernel is also known as hyperbolic tangent,
or Multilayer Perceptron (because, in the neural network field, it is
often used as neuron activation function). It is defined as:

   k(x, y) = \tanh( \gamma x^\top y + c_0)

where:

   * \gamma is known as slope

   * c_0 is known as intercept

6.8.5. RBF kernel

The function "rbf_kernel()" computes the radial basis function (RBF)
kernel between two vectors. This kernel is defined as:

   k(x, y) = \exp( -\gamma \| x-y \|^2)

where "x" and "y" are the input vectors. If \gamma = \sigma^{-2} the
kernel is known as the Gaussian kernel of variance \sigma^2.

6.8.6. Laplacian kernel

The function "laplacian_kernel()" is a variant on the radial basis
function kernel defined as:

   k(x, y) = \exp( -\gamma \| x-y \|_1)

where "x" and "y" are the input vectors and \|x-y\|_1 is the Manhattan
distance between the input vectors.

It has proven useful in ML applied to noiseless data. See e.g. Machine
learning for quantum mechanics in a nutshell.

6.8.7. Chi-squared kernel

The chi-squared kernel is a very popular choice for training non-

linear SVMs in computer vision applications. It can be computed using
"chi2_kernel()" and then passed to an "SVC" with
"kernel="precomputed"":

   >>> from sklearn.metrics.pairwise import chi2_kernel
   >>> X = [[0, 1], [1, 0], [.2, .8], [.7, .3]]
   >>> y = [0, 1, 0, 1]
   >>> K = chi2_kernel(X, gamma=.5)
   >>> K
   array([[1.        , 0.36787944, 0.89483932, 0.58364548],
          [0.36787944, 1.        , 0.51341712, 0.83822343],
          [0.89483932, 0.51341712, 1.        , 0.7768366 ],
          [0.58364548, 0.83822343, 0.7768366 , 1.        ]])

   >>> svm = SVC(kernel='precomputed').fit(K, y)
   >>> svm.predict(K)
   array([0, 1, 0, 1])

It can also be directly used as the "kernel" argument:

   >>> svm = SVC(kernel=chi2_kernel).fit(X, y)
   >>> svm.predict(X)

The chi squared kernel is given by

   k(x, y) = \exp \left (-\gamma \sum_i \frac{(x[i] - y[i]) ^ 2}{x[i]
   + y[i]} \right )

The data is assumed to be non-negative, and is often normalized to
have an L1-norm of one. The normalization is rationalized with the
connection to the chi squared distance, which is a distance between
discrete probability distributions.

The chi squared kernel is most commonly used on histograms (bags) of
visual words.

* Zhang, J. and Marszalek, M. and Lazebnik, S. and Schmid, C. Local

  features and kernels for classification of texture and object
  categories: A comprehensive study International Journal of Computer
  Vision 2007 https://hal.archives-ouvertes.fr/hal-00171412/document

2.6. Covariance estimation

Many statistical problems require the estimation of a population's
covariance matrix, which can be seen as an estimation of data set
scatter plot shape. Most of the time, such an estimation has to be
done on a sample whose properties (size, structure, homogeneity) have
a large influence on the estimation's quality. The
"sklearn.covariance" package provides tools for accurately estimating
a population's covariance matrix under various settings.

We assume that the observations are independent and identically
distributed (i.i.d.).

2.6.1. Empirical covariance

The covariance matrix of a data set is known to be well approximated
by the classical *maximum likelihood estimator* (or "empirical
covariance"), provided the number of observations is large enough
compared to the number of features (the variables describing the
observations). More precisely, the Maximum Likelihood Estimator of a
sample is an asymptotically unbiased estimator of the corresponding
population's covariance matrix.

The empirical covariance matrix of a sample can be computed using the
"empirical_covariance()" function of the package, or by fitting an
"EmpiricalCovariance" object to the data sample with the
"EmpiricalCovariance.fit()" method. Be careful that results depend on
whether the data are centered, so one may want to use the
"assume_centered" parameter accurately. More precisely, if
"assume_centered=False", then the test set is supposed to have the

same mean vector as the training set. If not, both should be centered
by the user, and "assume_centered=True" should be used.

Examples:
^^^^^^^^^

* See
  *sphx_glr_auto_examples_covariance_plot_covariance_estimation.py*
  for an example on how to fit an "EmpiricalCovariance" object to
  data.

2.6.2. Shrunk Covariance

2.6.2.1. Basic shrinkage

Despite being an asymptotically unbiased estimator of the covariance
matrix, the Maximum Likelihood Estimator is not a good estimator of
the eigenvalues of the covariance matrix, so the precision matrix
obtained from its inversion is not accurate. Sometimes, it even occurs
that the empirical covariance matrix cannot be inverted for numerical
reasons. To avoid such an inversion problem, a transformation of the
empirical covariance matrix has been introduced: the "shrinkage".

In scikit-learn, this transformation (with a user-defined shrinkage
coefficient) can be directly applied to a pre-computed covariance with
the "shrunk_covariance()" method. Also, a shrunk estimator of the
covariance can be fitted to data with a "ShrunkCovariance" object and
its "ShrunkCovariance.fit()" method. Again, results depend on whether
the data are centered, so one may want to use the "assume_centered"
parameter accurately.

Mathematically, this shrinkage consists in reducing the ratio between
the smallest and the largest eigenvalues of the empirical covariance
matrix. It can be done by simply shifting every eigenvalue according
to a given offset, which is equivalent of finding the l2-penalized
Maximum Likelihood Estimator of the covariance matrix. In practice,
shrinkage boils down to a simple a convex transformation : \Sigma_{\rm
shrunk} = (1-\alpha)\hat{\Sigma} + \alpha\frac{{\rm
Tr}\hat{\Sigma}}{p}\rm Id.

Choosing the amount of shrinkage, \alpha amounts to setting a
bias/variance trade-off, and is discussed below.

Examples:
^^^^^^^^^

* See
  for an example on how to fit a "ShrunkCovariance" object to data.

2.6.2.2. Ledoit-Wolf shrinkage

In their 2004 paper [1], O. Ledoit and M. Wolf propose a formula to
compute the optimal shrinkage coefficient \alpha that minimizes the
Mean Squared Error between the estimated and the real covariance
matrix.

The Ledoit-Wolf estimator of the covariance matrix can be computed on
a sample with the "ledoit_wolf()" function of the "sklearn.covariance"
package, or it can be otherwise obtained by fitting a "LedoitWolf"
object to the same sample.

Note:

  **Case when population covariance matrix is isotropic**It is

Note:

  important to note that when the number of samples is much larger
  than the number of features, one would expect that no shrinkage
  would be necessary. The intuition behind this is that if the
  population covariance is full rank, when the number of sample grows,
  the sample covariance will also become positive definite. As a
  result, no shrinkage would necessary and the method should
  automatically do this.This, however, is not the case in the Ledoit-
  Wolf procedure when the population covariance happens to be a
  multiple of the identity matrix. In this case, the Ledoit-Wolf
  shrinkage estimate approaches 1 as the number of samples increases.
  This indicates that the optimal estimate of the covariance matrix in
  the Ledoit-Wolf sense is multiple of the identity. Since the
  population covariance is already a multiple of the identity matrix,
  the Ledoit-Wolf solution is indeed a reasonable estimate.

Examples:
^^^^^^^^^

* See
  for an example on how to fit a "LedoitWolf" object to data and for
  visualizing the performances of the Ledoit-Wolf estimator in terms
  of likelihood.

[1] O. Ledoit and M. Wolf, "A Well-Conditioned Estimator for Large-
    Dimensional Covariance Matrices", Journal of Multivariate

of likelihood.

    Analysis, Volume 88, Issue 2, February 2004, pages 365-411.

2.6.2.3. Oracle Approximating Shrinkage

Under the assumption that the data are Gaussian distributed, Chen et
al. [2] derived a formula aimed at choosing a shrinkage coefficient
that yields a smaller Mean Squared Error than the one given by Ledoit
and Wolf's formula. The resulting estimator is known as the Oracle
Shrinkage Approximating estimator of the covariance.

The OAS estimator of the covariance matrix can be computed on a sample
with the "oas()" function of the "sklearn.covariance" package, or it
can be otherwise obtained by fitting an "OAS" object to the same
sample.

   [image]Bias-variance trade-off when setting the shrinkage:
   comparing the choices of Ledoit-Wolf and OAS estimators

[2] >>:arxiv:`"Shrinkage algorithms for MMSE covariance estimation.",
    Chen, Y., Wiesel, A., Eldar, Y. C., & Hero, A. O. IEEE
    Transactions on Signal Processing, 58(10), 5016-5029, 2010.
    <0907.4698>`<<

Examples:
^^^^^^^^^

* See
  for an example on how to fit an "OAS" object to data.

* See *sphx_glr_auto_examples_covariance_plot_lw_vs_oas.py* to
  visualize the Mean Squared Error difference between a "LedoitWolf"

for an example on how to fit an "OAS" object to data.

  and an "OAS" estimator of the covariance.

   [image]

2.6.3. Sparse inverse covariance

The matrix inverse of the covariance matrix, often called the
precision matrix, is proportional to the partial correlation matrix.
It gives the partial independence relationship. In other words, if two
features are independent conditionally on the others, the
corresponding coefficient in the precision matrix will be zero. This
is why it makes sense to estimate a sparse precision matrix: the
estimation of the covariance matrix is better conditioned by learning
independence relations from the data. This is known as *covariance
selection*.

In the small-samples situation, in which "n_samples" is on the order
of "n_features" or smaller, sparse inverse covariance estimators tend
to work better than shrunk covariance estimators. However, in the
opposite situation, or for very correlated data, they can be
numerically unstable. In addition, unlike shrinkage estimators, sparse
estimators are able to recover off-diagonal structure.

The "GraphicalLasso" estimator uses an l1 penalty to enforce sparsity
on the precision matrix: the higher its "alpha" parameter, the more
sparse the precision matrix. The corresponding "GraphicalLassoCV"
object uses cross-validation to automatically set the "alpha"

   [image]*A comparison of maximum likelihood, shrinkage and sparse
   estimates of the covariance and precision matrix in the very small
   samples settings.*

Note:

  **Structure recovery**Recovering a graphical structure from
  correlations in the data is a challenging thing. If you are
  interested in such recovery keep in mind that:

  * Recovery is easier from a correlation matrix than a covariance
    matrix: standardize your observations before running
    "GraphicalLasso"

  * If the underlying graph has nodes with much more connections than
    the average node, the algorithm will miss some of these
    connections.

  * If your number of observations is not large compared to the number
    of edges in your underlying graph, you will not recover it.

  * Even if you are in favorable recovery conditions, the alpha
    parameter chosen by cross-validation (e.g. using the
    "GraphicalLassoCV" object) will lead to selecting too many edges.
    However, the relevant edges will have heavier weights than the
    irrelevant ones.

The mathematical formulation is the following:

   \hat{K} = \mathrm{argmin}_K \big(             \mathrm{tr} S K -
   \mathrm{log} \mathrm{det} K             + \alpha \|K\|_1
   \big)

Where K is the precision matrix to be estimated, and S is the sample
covariance matrix. \|K\|_1 is the sum of the absolute values of off-

\big)

diagonal coefficients of K. The algorithm employed to solve this
problem is the GLasso algorithm, from the Friedman 2008 Biostatistics
paper. It is the same algorithm as in the R "glasso" package.

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_covariance_plot_sparse_cov.py*: example on
  synthetic data showing some recovery of a structure, and comparing
  to other covariance estimators.

* *sphx_glr_auto_examples_applications_plot_stock_market.py*: example
  on real stock market data, finding which symbols are most linked.

* Friedman et al, "Sparse inverse covariance estimation with the
  graphical lasso", Biostatistics 9, pp 432, 2008

2.6.4. Robust Covariance Estimation

Real data sets are often subject to measurement or recording errors.
Regular but uncommon observations may also appear for a variety of
reasons. Observations which are very uncommon are called outliers. The
empirical covariance estimator and the shrunk covariance estimators
presented above are very sensitive to the presence of outliers in the
data. Therefore, one should use robust covariance estimators to
estimate the covariance of its real data sets. Alternatively, robust
covariance estimators can be used to perform outlier detection and
discard/downweight some observations according to further processing
of the data.

The "sklearn.covariance" package implements a robust estimator of
covariance, the Minimum Covariance Determinant [3].

2.6.4.1. Minimum Covariance Determinant

The Minimum Covariance Determinant estimator is a robust estimator of
a data set's covariance introduced by P.J. Rousseeuw in [3].  The idea
is to find a given proportion (h) of "good" observations which are not
outliers and compute their empirical covariance matrix.  This
empirical covariance matrix is then rescaled to compensate the
performed selection of observations ("consistency step").  Having
computed the Minimum Covariance Determinant estimator, one can give
weights to observations according to their Mahalanobis distance,
leading to a reweighted estimate of the covariance matrix of the data
set ("reweighting step").

Rousseeuw and Van Driessen [4] developed the FastMCD algorithm in
order to compute the Minimum Covariance Determinant. This algorithm is
used in scikit-learn when fitting an MCD object to data. The FastMCD
algorithm also computes a robust estimate of the data set location at
the same time.

Raw estimates can be accessed as "raw_location_" and "raw_covariance_"
attributes of a "MinCovDet" robust covariance estimator object.

[3] P. J. Rousseeuw. Least median of squares regression. J. Am Stat
    Ass, 79:871, 1984.

[4] A Fast Algorithm for the Minimum Covariance Determinant Estimator,
    1999, American Statistical Association and the American Society
    for Quality, TECHNOMETRICS.

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_covariance_plot_robust_vs_empirical_cov
  ariance.py* for an example on how to fit a "MinCovDet" object to
  data and see how the estimate remains accurate despite the presence
  of outliers.

* See
  *sphx_glr_auto_examples_covariance_plot_mahalanobis_distances.py* to
  visualize the difference between "EmpiricalCovariance" and
  "MinCovDet" covariance estimators in terms of Mahalanobis distance
  (so we get a better estimate of the precision matrix too).

| Influence of outliers on location and covariance   | Separating inliers from outliers using a           |
| estimates                                          | Mahalanobis distance                               |
|====================================================|====================================================|
| [image: robust_vs_emp][image]                      | [image: mahalanobis][image]                        |

3.3. Metrics and scoring: quantifying the quality of predictions
****************************************************************

There are 3 different APIs for evaluating the quality of a model's
predictions:

* **Estimator score method**: Estimators have a "score" method
  providing a default evaluation criterion for the problem they are
  designed to solve. This is not discussed on this page, but in each
  estimator's documentation.

* **Scoring parameter**: Model-evaluation tools using cross-validation
  (such as "model_selection.cross_val_score()" and
  "model_selection.GridSearchCV") rely on an internal *scoring*
  strategy. This is discussed in the section The scoring parameter:
  defining model evaluation rules.

* **Metric functions**: The "sklearn.metrics" module implements
  functions assessing prediction error for specific purposes. These
  metrics are detailed in sections on Classification metrics,
  Multilabel ranking metrics, Regression metrics and Clustering
  metrics.

Finally, Dummy estimators are useful to get a baseline value of those
metrics for random predictions.

See also:

  For "pairwise" metrics, between *samples* and not estimators or
  predictions, see the Pairwise metrics, Affinities and Kernels
  section.

3.3.1. The "scoring" parameter: defining model evaluation rules

Model selection and evaluation using tools, such as
"model_selection.GridSearchCV" and

"model_selection.cross_val_score()", take a "scoring" parameter that
controls what metric they apply to the estimators evaluated.

3.3.1.1. Common cases: predefined values

For the most common use cases, you can designate a scorer object with
the "scoring" parameter; the table below shows all possible values.
All scorer objects follow the convention that **higher return values
are better than lower return values**.  Thus metrics which measure the
distance between the model and the data, like
"metrics.mean_squared_error()", are available as
neg_mean_squared_error which return the negated value of the metric.

+--------------------------------------+------------------------------------------------+------------------------------------+
| Scoring                              | Function                                       | Comment                            |
|======================================|================================================|====================================|
| **Classification**                   |                                                |                                    |

| 'accuracy'                           | "metrics.accuracy_score()"                     |                                    |
| 'balanced_accuracy'                  | "metrics.balanced_accuracy_score()"            |                                    |
| 'top_k_accuracy'                     | "metrics.top_k_accuracy_score()"               |                                    |
| 'average_precision'                  | "metrics.average_precision_score()"            |                                    |
| 'neg_brier_score'                    | "metrics.brier_score_loss()"                   |                                    |

| 'f1'                                 | "metrics.f1_score()"                           | for binary targets                 |
| 'f1_micro'                           | "metrics.f1_score()"                           | micro-averaged                     |
| 'f1_macro'                           | "metrics.f1_score()"                           | macro-averaged                     |
| 'f1_weighted'                        | "metrics.f1_score()"                           | weighted average                   |
| 'f1_samples'                         | "metrics.f1_score()"                           | by multilabel sample               |

| 'neg_log_loss'                       | "metrics.log_loss()"                           | requires "predict_proba" support   |
| 'precision' etc.                     | "metrics.precision_score()"                    | suffixes apply as with 'f1'        |
| 'recall' etc.                        | "metrics.recall_score()"                       | suffixes apply as with 'f1'        |
| 'jaccard' etc.                       | "metrics.jaccard_score()"                      | suffixes apply as with 'f1'        |
| 'roc_auc'                            | "metrics.roc_auc_score()"                      |                                    |

| 'roc_auc_ovr'                        | "metrics.roc_auc_score()"                      |                                    |
| 'roc_auc_ovo'                        | "metrics.roc_auc_score()"                      |                                    |
| 'roc_auc_ovr_weighted'               | "metrics.roc_auc_score()"                      |                                    |
| 'roc_auc_ovo_weighted'               | "metrics.roc_auc_score()"                      |                                    |
| **Clustering**                       |                                                |                                    |

| 'adjusted_mutual_info_score'         | "metrics.adjusted_mutual_info_score()"         |                                    |
| 'adjusted_rand_score'                | "metrics.adjusted_rand_score()"                |                                    |
| 'completeness_score'                 | "metrics.completeness_score()"                 |                                    |
| 'fowlkes_mallows_score'              | "metrics.fowlkes_mallows_score()"              |                                    |
| 'homogeneity_score'                  | "metrics.homogeneity_score()"                  |                                    |

| 'mutual_info_score'                  | "metrics.mutual_info_score()"                  |                                    |
| 'normalized_mutual_info_score'       | "metrics.normalized_mutual_info_score()"       |                                    |
| 'rand_score'                         | "metrics.rand_score()"                         |                                    |
| 'v_measure_score'                    | "metrics.v_measure_score()"                    |                                    |
| **Regression**                       |                                                |                                    |

| 'explained_variance'                 | "metrics.explained_variance_score()"           |                                    |
| 'max_error'                          | "metrics.max_error()"                          |                                    |
| 'neg_mean_absolute_error'            | "metrics.mean_absolute_error()"                |                                    |
| 'neg_mean_squared_error'             | "metrics.mean_squared_error()"                 |                                    |
| 'neg_root_mean_squared_error'        | "metrics.mean_squared_error()"                 |                                    |

| 'neg_mean_squared_log_error'         | "metrics.mean_squared_log_error()"             |                                    |
| 'neg_median_absolute_error'          | "metrics.median_absolute_error()"              |                                    |
| 'r2'                                 | "metrics.r2_score()"                           |                                    |
| 'neg_mean_poisson_deviance'          | "metrics.mean_poisson_deviance()"              |                                    |
| 'neg_mean_gamma_deviance'            | "metrics.mean_gamma_deviance()"                |                                    |

| 'neg_mean_absolute_percentage_error' | "metrics.mean_absolute_percentage_error()"     |                                    |
| 'd2_absolute_error_score'            | "metrics.d2_absolute_error_score()"            |                                    |
| 'd2_pinball_score'                   | "metrics.d2_pinball_score()"                   |                                    |
| 'd2_tweedie_score'                   | "metrics.d2_tweedie_score()"                   |                                    |

Usage examples:

>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import cross_val_score
>>> clf = svm.SVC(random_state=0)
>>> cross_val_score(clf, X, y, cv=5, scoring='recall_macro')
array([0.96..., 0.96..., 0.96..., 0.93..., 1.        ])
>>> model = svm.SVC()

>>> cross_val_score(model, X, y, cv=5, scoring='wrong_choice')
Traceback (most recent call last):
ValueError: 'wrong_choice' is not a valid scoring value. Use
sklearn.metrics.get_scorer_names() to get valid options.

Note:

  The values listed by the "ValueError" exception correspond to the
  functions measuring prediction accuracy described in the following
  sections. You can retrieve the names of all available scorers by
  calling "get_scorer_names()".

3.3.1.2. Defining your scoring strategy from metric functions

The module "sklearn.metrics" also exposes a set of simple functions
measuring a prediction error given ground truth and prediction:

* functions ending with "_score" return a value to maximize, the
  higher the better.

* functions ending with "_error" or "_loss" return a value to
  minimize, the lower the better.  When converting into a scorer
  object using "make_scorer()", set the "greater_is_better" parameter
  to "False" ("True" by default; see the parameter description below).

Metrics available for various machine learning tasks are detailed in
sections below.

Many metrics are not given names to be used as "scoring" values,
sometimes because they require additional parameters, such as
"fbeta_score()". In such cases, you need to generate an appropriate

scoring object.  The simplest way to generate a callable object for
scoring is by using "make_scorer()". That function converts metrics
into callables that can be used for model evaluation.

One typical use case is to wrap an existing metric function from the
library with non-default values for its parameters, such as the "beta"
parameter for the "fbeta_score()" function:

   >>> from sklearn.metrics import fbeta_score, make_scorer
   >>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
   >>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
   ...                     scoring=ftwo_scorer, cv=5)

The second use case is to build a completely custom scorer object from
a simple python function using "make_scorer()", which can take several
parameters:

* the python function you want to use ("my_custom_loss_func" in the
  example below)

* whether the python function returns a score
  ("greater_is_better=True", the default) or a loss
  ("greater_is_better=False").  If a loss, the output of the python
  function is negated by the scorer object, conforming to the cross
  validation convention that scorers return higher values for better
  models.

* for classification metrics only: whether the python function you

validation convention that scorers return higher values for better
  models.

  provided requires continuous decision certainties
  ("needs_threshold=True").  The default value is False.

* any additional parameters, such as "beta" or "labels" in
  "f1_score()".

Here is an example of building custom scorers, and of using the
"greater_is_better" parameter:

   >>> def my_custom_loss_func(y_true, y_pred):
   ...     diff = np.abs(y_true - y_pred).max()
   ...     return np.log1p(diff)
   ...
   >>> # score will negate the return value of my_custom_loss_func,
   >>> # which will be np.log(2), 0.693, given the values for X
   >>> # and y defined below.
   >>> score = make_scorer(my_custom_loss_func, greater_is_better=False)
   >>> X = [[1], [1]]
   >>> from sklearn.dummy import DummyClassifier
   >>> clf = DummyClassifier(strategy='most_frequent', random_state=0)
   >>> my_custom_loss_func(y, clf.predict(X))
   0.69...
   >>> score(clf, X, y)
   -0.69...

3.3.1.3. Implementing your own scoring object

You can generate even more flexible model scorers by constructing your
own scoring object from scratch, without using the "make_scorer()"
factory. For a callable to be a scorer, it needs to meet the protocol
specified by the following two rules:

* It can be called with parameters "(estimator, X, y)", where

  "estimator" is the model that should be evaluated, "X" is validation
  data, and "y" is the ground truth target for "X" (in the supervised
  case) or "None" (in the unsupervised case).

* It returns a floating point number that quantifies the "estimator"
  prediction quality on "X", with reference to "y". Again, by
  convention higher numbers are better, so if your scorer returns
  loss, that value should be negated.

Note:

  **Using custom scorers in functions where n_jobs > 1**While defining
  the custom scoring function alongside the calling function should
  work out of the box with the default joblib backend (loky),
  importing it from another module will be a more robust approach and
  work independently of the joblib backend.For example, to use
  "n_jobs" greater than 1 in the example below,
  "custom_scoring_function" function is saved in a user-created module
  ("custom_scorer_module.py") and imported:

     >>> from custom_scorer_module import custom_scoring_function 
     >>> cross_val_score(model,
     ...  X_train,
     ...  y_train,
     ...  scoring=make_scorer(custom_scoring_function, greater_is_better=False),
     ...  cv=5,
     ...  n_jobs=-1) 

3.3.1.4. Using multiple metric evaluation

Scikit-learn also permits evaluation of multiple metrics in

...  cv=5,

"GridSearchCV", "RandomizedSearchCV" and "cross_validate".

There are three ways to specify multiple scoring metrics for the
"scoring" parameter:

* As an iterable of string metrics::
     >>> scoring = ['accuracy', 'precision']

* As a "dict" mapping the scorer name to the scoring function::
     >>> from sklearn.metrics import accuracy_score
     >>> from sklearn.metrics import make_scorer
     >>> scoring = {'accuracy': make_scorer(accuracy_score),
     ...            'prec': 'precision'}

  Note that the dict values can either be scorer functions or one of
  the predefined metric strings.

* As a callable that returns a dictionary of scores:

     >>> from sklearn.model_selection import cross_validate
     >>> from sklearn.metrics import confusion_matrix
     >>> # A sample toy binary classification dataset
     >>> X, y = datasets.make_classification(n_classes=2, random_state=0)
     >>> svm = LinearSVC(random_state=0)
     >>> def confusion_matrix_scorer(clf, X, y):
     ...      y_pred = clf.predict(X)
     ...      cm = confusion_matrix(y, y_pred)
     ...      return {'tn': cm[0, 0], 'fp': cm[0, 1],
     ...              'fn': cm[1, 0], 'tp': cm[1, 1]}
     >>> cv_results = cross_validate(svm, X, y, cv=5,
     ...                             scoring=confusion_matrix_scorer)

...              'fn': cm[1, 0], 'tp': cm[1, 1]}
     >>> # Getting the test set true positive scores
     >>> print(cv_results['test_tp'])
     [10  9  8  7  8]
     >>> # Getting the test set false negative scores
     >>> print(cv_results['test_fn'])
     [0 1 2 3 2]

3.3.2. Classification metrics

The "sklearn.metrics" module implements several loss, score, and
utility functions to measure classification performance. Some metrics
might require probability estimates of the positive class, confidence
values, or binary decisions values. Most implementations allow each
sample to provide a weighted contribution to the overall score,
through the "sample_weight" parameter.

Some of these are restricted to the binary classification case:

Others also work in the multiclass case:

Some also work in the multilabel case:

And some work with binary and multilabel (but not multiclass)
problems:

In the following sub-sections, we will describe each of those
functions, preceded by some notes on common API and metric definition.

3.3.2.1. From binary to multiclass and multilabel

Some metrics are essentially defined for binary classification tasks
(e.g. "f1_score()", "roc_auc_score()"). In these cases, by default
only the positive label is evaluated, assuming by default that the

positive class is labelled "1" (though this may be configurable
through the "pos_label" parameter).

In extending a binary metric to multiclass or multilabel problems, the
data is treated as a collection of binary problems, one for each
class. There are then a number of ways to average binary metric
calculations across the set of classes, each of which may be useful in
some scenario. Where available, you should select among these using
the "average" parameter.

* ""macro"" simply calculates the mean of the binary metrics, giving
  equal weight to each class.  In problems where infrequent classes
  are nonetheless important, macro-averaging may be a means of
  highlighting their performance. On the other hand, the assumption
  that all classes are equally important is often untrue, such that
  macro-averaging will over-emphasize the typically low performance on
  an infrequent class.

* ""weighted"" accounts for class imbalance by computing the average
  of binary metrics in which each class's score is weighted by its
  presence in the true data sample.

* ""micro"" gives each sample-class pair an equal contribution to the
  overall metric (except as a result of sample-weight). Rather than
  summing the metric per class, this sums the dividends and divisors

overall metric (except as a result of sample-weight). Rather than
  that make up the per-class metrics to calculate an overall quotient.
  Micro-averaging may be preferred in multilabel settings, including
  multiclass classification where a majority class is to be ignored.

* ""samples"" applies only to multilabel problems. It does not
  calculate a per-class measure, instead calculating the metric over
  the true and predicted classes for each sample in the evaluation
  data, and returning their ("sample_weight"-weighted) average.

* Selecting "average=None" will return an array with the score for
  each class.

While multiclass data is provided to the metric, like binary targets,
as an array of class labels, multilabel data is specified as an
indicator matrix, in which cell "[i, j]" has value 1 if sample "i" has
label "j" and value 0 otherwise.

3.3.2.2. Accuracy score

The "accuracy_score()" function computes the accuracy, either the
fraction (default) or the count (normalize=False) of correct

In multilabel classification, the function returns the subset
accuracy. If the entire set of predicted labels for a sample strictly
match with the true set of labels, then the subset accuracy is 1.0;
otherwise it is 0.0.

If \hat{y}_i is the predicted value of the i-th sample and y_i is the
corresponding true value, then the fraction of correct predictions
over n_\text{samples} is defined as

   \texttt{accuracy}(y, \hat{y}) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples}-1} 1(\hat{y}_i = y_i)

where 1(x) is the indicator function.

>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2

In the multilabel case with binary label indicators:

   >>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
   0.5

Example:
^^^^^^^^

* See *sphx_glr_auto_examples_model_selection_plot_permutation_tests_
  for_classification.py* for an example of accuracy score usage using
  permutations of the dataset.

3.3.2.3. Top-k accuracy score

The "top_k_accuracy_score()" function is a generalization of
"accuracy_score()". The difference is that a prediction is considered
correct as long as the true label is associated with one of the "k"
highest predicted scores. "accuracy_score()" is the special case of *k
= 1*.

The function covers the binary and multiclass classification cases but
not the multilabel case.

If \hat{f}_{i,j} is the predicted class for the i-th sample
corresponding to the j-th largest predicted score and y_i is the

   \texttt{top-k accuracy}(y, \hat{f}) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples}-1} \sum_{j=1}^{k} 1(\hat{f}_{i,j} =
   y_i)

where k is the number of guesses allowed and 1(x) is the indicator
function.

>>> from sklearn.metrics import top_k_accuracy_score
>>> y_true = np.array([0, 1, 2, 2])
>>> y_score = np.array([[0.5, 0.2, 0.2],
...                     [0.3, 0.4, 0.2],
...                     [0.2, 0.4, 0.3],
...                     [0.7, 0.2, 0.1]])
>>> top_k_accuracy_score(y_true, y_score, k=2)
0.75
>>> # Not normalizing gives the number of "correctly" classified samples
>>> top_k_accuracy_score(y_true, y_score, k=2, normalize=False)
3

3.3.2.4. Balanced accuracy score

The "balanced_accuracy_score()" function computes the balanced
accuracy, which avoids inflated performance estimates on imbalanced
datasets. It is the macro-average of recall scores per class or,
equivalently, raw accuracy where each sample is weighted according to
the inverse prevalence of its true class. Thus for balanced datasets,
the score is equal to accuracy.

In the binary case, balanced accuracy is equal to the arithmetic mean
of sensitivity (true positive rate) and specificity (true negative

rate), or the area under the ROC curve with binary predictions rather
than scores:

   \texttt{balanced-accuracy} = \frac{1}{2}\left( \frac{TP}{TP + FN} +
   \frac{TN}{TN + FP}\right )

If the classifier performs equally well on either class, this term
reduces to the conventional accuracy (i.e., the number of correct
predictions divided by the total number of predictions).

In contrast, if the conventional accuracy is above chance only because
the classifier takes advantage of an imbalanced test set, then the
balanced accuracy, as appropriate, will drop to \frac{1}{n\_classes}.

The score ranges from 0 to 1, or when "adjusted=True" is used, it
rescaled to the range \frac{1}{1 - n\_classes} to 1, inclusive, with
performance at random scoring 0.

If y_i is the true value of the i-th sample, and w_i is the
corresponding sample weight, then we adjust the sample weight to:

   \hat{w}_i = \frac{w_i}{\sum_j{1(y_j = y_i) w_j}}

where 1(x) is the indicator function. Given predicted \hat{y}_i for
sample i, balanced accuracy is defined as:

   \texttt{balanced-accuracy}(y, \hat{y}, w) =
   \frac{1}{\sum{\hat{w}_i}} \sum_i 1(\hat{y}_i = y_i) \hat{w}_i

With "adjusted=True", balanced accuracy reports the relative increase
from \texttt{balanced-accuracy}(y, \mathbf{0}, w) =
\frac{1}{n\_classes}.  In the binary case, this is also known as

*Youden's J statistic*, or *informedness*.

Note:

  The multiclass definition here seems the most reasonable extension
  of the metric used in binary classification, though there is no
  certain consensus in the literature:

  * Our definition: [Mosley2013], [Kelleher2015] and [Guyon2015],
    where [Guyon2015] adopt the adjusted version to ensure that random
    predictions have a score of 0 and perfect predictions have a score
    of 1..

  * Class balanced accuracy as described in [Mosley2013]: the minimum
    between the precision and the recall for each class is computed.
    Those values are then averaged over the total number of classes to
    get the balanced accuracy.

  * Balanced Accuracy as described in [Urbanowicz2015]: the average of
    sensitivity and specificity is computed for each class and then
    averaged over total number of classes.

[Guyon2015] I. Guyon, K. Bennett, G. Cawley, H.J. Escalante, S.
            Escalera, T.K. Ho, N. Macià, B. Ray, M. Saeed, A.R.
            Statnikov, E. Viegas, Design of the 2015 ChaLearn AutoML
            Challenge, IJCNN 2015.

[Mosley2013] L. Mosley, A balanced approach to the multi-class
             imbalance problem, IJCV 2010.

[Kelleher2015] John. D. Kelleher, Brian Mac Namee, Aoife D'Arcy,

               Fundamentals of Machine Learning for Predictive Data
               Analytics: Algorithms, Worked Examples, and Case
               Studies, 2015.

[Urbanowicz2015] Urbanowicz R.J.,  Moore, J.H. >>:doi:`ExSTraCS 2.0:
                 description and evaluation of a scalable learning
                 classifier system <10.1007/s12065-015-0128-8>`<<,
                 Evol. Intel. (2015) 8: 89.

3.3.2.5. Cohen's kappa

The function "cohen_kappa_score()" computes Cohen's kappa statistic.
This measure is intended to compare labelings by different human
annotators, not a classifier versus a ground truth.

The kappa score (see docstring) is a number between -1 and 1. Scores
above .8 are generally considered good agreement; zero or lower means
no agreement (practically random labels).

Kappa scores can be computed for binary or multiclass problems, but
not for multilabel problems (except by manually computing a per-label
score) and not for more than two annotators.

>>> from sklearn.metrics import cohen_kappa_score
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> cohen_kappa_score(y_true, y_pred)
0.4285714285714286

3.3.2.6. Confusion matrix

The "confusion_matrix()" function evaluates classification accuracy by

computing the confusion matrix with each row corresponding to the true
class (Wikipedia and other references may use different convention for
axes).

By definition, entry i, j in a confusion matrix is the number of
observations actually in group i, but predicted to be in group j. Here
is an example:

   >>> from sklearn.metrics import confusion_matrix
   >>> y_true = [2, 0, 2, 2, 0, 1]
   >>> y_pred = [0, 0, 2, 2, 0, 2]
   >>> confusion_matrix(y_true, y_pred)
   array([[2, 0, 0],
          [0, 0, 1],
          [1, 0, 2]])

"ConfusionMatrixDisplay" can be used to visually represent a confusion
matrix as shown in the
*sphx_glr_auto_examples_model_selection_plot_confusion_matrix.py*
example, which creates the following figure:

[image]

The parameter "normalize" allows to report ratios instead of counts.
The confusion matrix can be normalized in 3 different ways: "'pred'",
"'true'", and "'all'" which will divide the counts by the sum of each
columns, rows, or the entire matrix, respectively.

>>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]
>>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
>>> confusion_matrix(y_true, y_pred, normalize='all')
array([[0.25 , 0.125],
       [0.25 , 0.375]])

For binary problems, we can get counts of true negatives, false
positives, false negatives and true positives as follows:

   >>> y_true = [0, 0, 0, 1, 1, 1, 1, 1]

[0.25 , 0.375]])

   >>> y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
   >>> tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
   >>> tn, fp, fn, tp
   (2, 1, 2, 3)

Example:
^^^^^^^^

* See
  *sphx_glr_auto_examples_model_selection_plot_confusion_matrix.py*
  for an example of using a confusion matrix to evaluate classifier
  output quality.

* See *sphx_glr_auto_examples_classification_plot_digits_classificati
  on.py* for an example of using a confusion matrix to classify hand-
  written digits.

* See *sphx_glr_auto_examples_text_plot_document_classification_20new
  sgroups.py* for an example of using a confusion matrix to classify
  text documents.

3.3.2.7. Classification report

The "classification_report()" function builds a text report showing
the main classification metrics. Here is a small example with custom
"target_names" and inferred labels:

   >>> from sklearn.metrics import classification_report
   >>> y_true = [0, 1, 2, 2, 0]
   >>> y_pred = [0, 0, 2, 1, 0]
   >>> target_names = ['class 0', 'class 1', 'class 2']
   >>> print(classification_report(y_true, y_pred, target_names=target_names))
                 precision    recall  f1-score   support

        class 0       0.67      1.00      0.80         2
        class 1       0.00      0.00      0.00         1

precision    recall  f1-score   support

        class 2       1.00      0.50      0.67         2

       accuracy                           0.60         5
      macro avg       0.56      0.50      0.49         5
   weighted avg       0.67      0.60      0.59         5

Example:
^^^^^^^^

  on.py* for an example of classification report usage for hand-

* See
  for an example of classification report usage for grid search with
  nested cross-validation.

3.3.2.8. Hamming loss

The "hamming_loss()" computes the average Hamming loss or Hamming
distance between two sets of samples.

If \hat{y}_{i,j} is the predicted value for the j-th label of a given
sample i, y_{i,j} is the corresponding true value, n_\text{samples} is
the number of samples and n_\text{labels} is the number of labels,
then the Hamming loss L_{Hamming} is defined as:

   L_{Hamming}(y, \hat{y}) = \frac{1}{n_\text{samples} *
   n_\text{labels}} \sum_{i=0}^{n_\text{samples}-1}
   \sum_{j=0}^{n_\text{labels} - 1} 1(\hat{y}_{i,j} \not= y_{i,j})

The equation above does not hold true in the case of multiclass
classification. Please refer to the note below for more information.

   >>> from sklearn.metrics import hamming_loss
   >>> y_pred = [1, 2, 3, 4]
   >>> y_true = [2, 2, 3, 4]
   >>> hamming_loss(y_true, y_pred)
   0.25

   >>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2)))
   0.75

Note:

  In multiclass classification, the Hamming loss corresponds to the
  Hamming distance between "y_true" and "y_pred" which is similar to
  the Zero one loss function.  However, while zero-one loss penalizes
  prediction sets that do not strictly match true sets, the Hamming
  loss penalizes individual labels.  Thus the Hamming loss, upper
  bounded by the zero-one loss, is always between zero and one,
  inclusive; and predicting a proper subset or superset of the true
  labels will give a Hamming loss between zero and one, exclusive.

3.3.2.9. Precision, recall and F-measures

Intuitively, precision is the ability of the classifier not to label
as positive a sample that is negative, and recall is the ability of
the classifier to find all the positive samples.

The  F-measure (F_\beta and F_1 measures) can be interpreted as a
weighted harmonic mean of the precision and recall. A F_\beta measure
reaches its best value at 1 and its worst score at 0. With \beta = 1,

F_\beta and F_1  are equivalent, and the recall and the precision are
equally important.

The "precision_recall_curve()" computes a precision-recall curve from
the ground truth label and a score given by the classifier by varying
a decision threshold.

The "average_precision_score()" function computes the average
precision (AP) from prediction scores. The value is between 0 and 1
and higher is better. AP is defined as

   \text{AP} = \sum_n (R_n - R_{n-1}) P_n

where P_n and R_n are the precision and recall at the nth threshold.
With random predictions, the AP is the fraction of positive samples.

References [Manning2008] and [Everingham2010] present alternative
variants of AP that interpolate the precision-recall curve. Currently,
"average_precision_score()" does not implement any interpolated
variant. References [Davis2006] and [Flach2015] describe why a linear
interpolation of points on the precision-recall curve provides an
overly-optimistic measure of classifier performance. This linear
interpolation is used when computing area under the curve with the
trapezoidal rule in "auc()".

Several functions allow you to analyze the precision, recall and
F-measures score:

Note that the "precision_recall_curve()" function is restricted to the
binary case. The "average_precision_score()" function works only in
binary classification and multilabel indicator format. The

"PredictionRecallDisplay.from_estimator()" and
"PredictionRecallDisplay.from_predictions()" functions will plot the
precision-recall curve as follows.

[image]

Examples:
^^^^^^^^^

* See
  for an example of "precision_score()" and "recall_score()" usage to
  estimate parameters using grid search with nested cross-validation.

* See
  *sphx_glr_auto_examples_model_selection_plot_precision_recall.py*
  for an example of "precision_recall_curve()" usage to evaluate
  classifier output quality.

[Manning2008] C.D. Manning, P. Raghavan, H. Schütze, Introduction to
              Information Retrieval, 2008.

[Everingham2010] M. Everingham, L. Van Gool, C.K.I. Williams, J. Winn,
                 A. Zisserman, The Pascal Visual Object Classes (VOC)
                 Challenge, IJCV 2010.

[Davis2006] J. Davis, M. Goadrich, The Relationship Between Precision-
            Recall and ROC Curves, ICML 2006.

[Flach2015] P.A. Flach, M. Kull, Precision-Recall-Gain Curves: PR
            Analysis Done Right, NIPS 2015.

3.3.2.9.1. Binary classification

In a binary classification task, the terms ''positive'' and
''negative'' refer to the classifier's prediction, and the terms

''true'' and ''false'' refer to whether that prediction corresponds to
the external judgment (sometimes known as the ''observation''). Given
these definitions, we can formulate the following table:

+---------------------+-----------------------+----------------------------+
|                     | Actual class (observation)                         |
| Predicted class     | tp (true positive)    | fp (false positive)        |
| (expectation)       | Correct result        | Unexpected result          |
|                     +-----------------------+----------------------------+
|                     | fn (false negative)   | tn (true negative) Correct |
|                     | Missing result        | absence of result          |

In this context, we can define the notions of precision, recall and
F-measure:

   \text{precision} = \frac{tp}{tp + fp},

   \text{recall} = \frac{tp}{tp + fn},

   F_\beta = (1 + \beta^2) \frac{\text{precision} \times
   \text{recall}}{\beta^2 \text{precision} + \text{recall}}.

Sometimes recall is also called ''sensitivity''.

Here are some small examples in binary classification:

\text{recall}}{\beta^2 \text{precision} + \text{recall}}.

   >>> y_pred = [0, 1, 0, 0]
   >>> y_true = [0, 1, 0, 1]
   >>> metrics.precision_score(y_true, y_pred)
   1.0
   >>> metrics.recall_score(y_true, y_pred)
   0.5
   >>> metrics.f1_score(y_true, y_pred)
   0.66...
   >>> metrics.fbeta_score(y_true, y_pred, beta=0.5)
   0.83...
   >>> metrics.fbeta_score(y_true, y_pred, beta=1)
   0.66...
   >>> metrics.fbeta_score(y_true, y_pred, beta=2)
   0.55...
   >>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5)
   (array([0.66..., 1.        ]), array([1. , 0.5]), array([0.71..., 0.83...]), array([2, 2]))

   >>> from sklearn.metrics import precision_recall_curve
   >>> from sklearn.metrics import average_precision_score
   >>> y_true = np.array([0, 0, 1, 1])
   >>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
   >>> precision, recall, threshold = precision_recall_curve(y_true, y_scores)
   >>> precision
   array([0.5       , 0.66..., 0.5       , 1.        , 1.        ])
   >>> recall
   array([1. , 1. , 0.5, 0.5, 0. ])
   >>> threshold
   array([0.1 , 0.35, 0.4 , 0.8 ])
   >>> average_precision_score(y_true, y_scores)
   0.83...

3.3.2.9.2. Multiclass and multilabel classification

In a multiclass and multilabel classification task, the notions of

0.83...

precision, recall, and F-measures can be applied to each label
independently. There are a few ways to combine results across labels,
specified by the "average" argument to the "average_precision_score()"
(multilabel only), "f1_score()", "fbeta_score()",
"precision_recall_fscore_support()", "precision_score()" and
"recall_score()" functions, as described above. Note that if all
labels are included, "micro"-averaging in a multiclass setting will
produce precision, recall and F that are all identical to accuracy.
Also note that "weighted" averaging may produce an F-score that is not
between precision and recall.

To make this more explicit, consider the following notation:

* y the set of *true* (sample, label) pairs

* \hat{y} the set of *predicted* (sample, label) pairs

* L the set of labels

* S the set of samples

* y_s the subset of y with sample s, i.e. y_s := \left\{(s', l) \in y
  | s' = s\right\}

* y_l the subset of y with label l

* similarly, \hat{y}_s and \hat{y}_l are subsets of \hat{y}

* P(A, B) := \frac{\left| A \cap B \right|}{\left|B\right|} for some
  sets A and B

* R(A, B) := \frac{\left| A \cap B \right|}{\left|A\right|}
  (Conventions vary on handling A = \emptyset; this implementation
  uses R(A, B):=0, and similar for P.)

* F_\beta(A, B) := \left(1 + \beta^2\right) \frac{P(A, B) \times R(A,

(Conventions vary on handling A = \emptyset; this implementation

  B)}{\beta^2 P(A, B) + R(A, B)}

Then the metrics are defined as:

+-----------------+--------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| "average"       | Precision                                                                                                          | Recall                                                                                                             | F_beta                                                                                                                 |
|=================|====================================================================================================================|====================================================================================================================|========================================================================================================================|

| ""micro""       | P(y, \hat{y})                                                                                                      | R(y, \hat{y})                                                                                                      | F_\beta(y, \hat{y})                                                                                                    |
| ""samples""     | \frac{1}{\left|S\right|} \sum_{s \in S} P(y_s, \hat{y}_s)                                                          | \frac{1}{\left|S\right|} \sum_{s \in S} R(y_s, \hat{y}_s)                                                          | \frac{1}{\left|S\right|} \sum_{s \in S} F_\beta(y_s, \hat{y}_s)                                                        |

| ""macro""       | \frac{1}{\left|L\right|} \sum_{l \in L} P(y_l, \hat{y}_l)                                                          | \frac{1}{\left|L\right|} \sum_{l \in L} R(y_l, \hat{y}_l)                                                          | \frac{1}{\left|L\right|} \sum_{l \in L} F_\beta(y_l, \hat{y}_l)                                                        |
| ""weighted""    | \frac{1}{\sum_{l \in L} \left|y_l\right|} \sum_{l \in L} \left|y_l\right| P(y_l, \hat{y}_l)                        | \frac{1}{\sum_{l \in L} \left|y_l\right|} \sum_{l \in L} \left|y_l\right| R(y_l, \hat{y}_l)                        | \frac{1}{\sum_{l \in L} \left|y_l\right|} \sum_{l \in L} \left|y_l\right| F_\beta(y_l, \hat{y}_l)                      |

| "None"          | \langle P(y_l, \hat{y}_l) | l \in L \rangle                                                                        | \langle R(y_l, \hat{y}_l) | l \in L \rangle                                                                        | \langle F_\beta(y_l, \hat{y}_l) | l \in L \rangle                                                                      |

>>> y_true = [0, 1, 2, 0, 1, 2]
>>> y_pred = [0, 2, 1, 0, 0, 1]
>>> metrics.precision_score(y_true, y_pred, average='macro')
0.22...
>>> metrics.recall_score(y_true, y_pred, average='micro')
0.33...
>>> metrics.f1_score(y_true, y_pred, average='weighted')
0.26...
>>> metrics.fbeta_score(y_true, y_pred, average='macro', beta=0.5)
0.23...
>>> metrics.precision_recall_fscore_support(y_true, y_pred, beta=0.5, average=None)
(array([0.66..., 0.        , 0.        ]), array([1., 0., 0.]), array([0.71..., 0.        , 0.        ]), array([2, 2, 2]...))

For multiclass classification with a "negative class", it is possible
to exclude some labels:

>>> metrics.recall_score(y_true, y_pred, labels=[1, 2], average='micro')

... # excluding 0, no labels were correctly recalled
0.0

Similarly, labels not present in the data sample may be accounted for
in macro-averaging.

>>> metrics.precision_score(y_true, y_pred, labels=[0, 1, 2, 3], average='macro')
0.166...

3.3.2.10. Jaccard similarity coefficient score

The "jaccard_score()" function computes the average of Jaccard
similarity coefficients, also called the Jaccard index, between pairs
of label sets.

The Jaccard similarity coefficient with a ground truth label set y and
predicted label set \hat{y}, is defined as

   J(y, \hat{y}) = \frac{|y \cap \hat{y}|}{|y \cup \hat{y}|}.

The "jaccard_score()" (like "precision_recall_fscore_support()")
applies natively to binary targets. By computing it set-wise it can be
extended to apply to multilabel and multiclass through the use of
*average* (see above).

In the binary case:

   >>> from sklearn.metrics import jaccard_score
   >>> y_true = np.array([[0, 1, 1],
   ...                    [1, 1, 0]])
   >>> y_pred = np.array([[1, 1, 1],
   ...                    [1, 0, 0]])
   >>> jaccard_score(y_true[0], y_pred[0])
   0.6666...

In the 2D comparison case (e.g. image similarity):

>>> jaccard_score(y_true, y_pred, average="micro")
0.6

0.6666...

0.6

   >>> jaccard_score(y_true, y_pred, average='samples')
   0.5833...
   >>> jaccard_score(y_true, y_pred, average='macro')
   0.6666...
   >>> jaccard_score(y_true, y_pred, average=None)
   array([0.5, 0.5, 1. ])

Multiclass problems are binarized and treated like the corresponding
multilabel problem:

   >>> y_pred = [0, 2, 1, 2]
   >>> y_true = [0, 1, 2, 2]
   array([1. , 0. , 0.33...])
   0.44...
   >>> jaccard_score(y_true, y_pred, average='micro')
   0.33...

3.3.2.11. Hinge loss

The "hinge_loss()" function computes the average distance between the
model and the data using hinge loss, a one-sided metric that considers
only prediction errors. (Hinge loss is used in maximal margin
classifiers such as support vector machines.)

If the true label y_i of a binary classification task is encoded as
y_i=\left\{-1, +1\right\} for every sample i; and w_i is the
corresponding predicted decision (an array of shape (*n_samples*,) as
output by the *decision_function* method), then the hinge loss is

   L_\text{Hinge}(y, w) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples}-1} \max\left\{1 - w_i y_i, 0\right\}

If there are more than two labels, "hinge_loss()" uses a multiclass

variant due to Crammer & Singer. Here is the paper describing it.

In this case the predicted decision is an array of shape (*n_samples*,
*n_labels*). If w_{i, y_i} is the predicted decision for the true
label y_i of the i-th sample; and \hat{w}_{i, y_i} = \max\left\{w_{i,
y_j}~|~y_j \ne y_i \right\} is the maximum of the predicted decisions
for all the other labels, then the multi-class hinge loss is defined
by:

   \sum_{i=0}^{n_\text{samples}-1} \max\left\{1 + \hat{w}_{i, y_i} -
   w_{i, y_i}, 0\right\}

Here is a small example demonstrating the use of the "hinge_loss()"
function with a svm classifier in a binary class problem:

   >>> from sklearn.metrics import hinge_loss
   >>> X = [[0], [1]]
   >>> y = [-1, 1]
   >>> est = svm.LinearSVC(random_state=0)
   >>> est.fit(X, y)
   LinearSVC(random_state=0)
   >>> pred_decision = est.decision_function([[-2], [3], [0.5]])
   >>> pred_decision
   array([-2.18...,  2.36...,  0.09...])
   >>> hinge_loss([-1, 1, 1], pred_decision)
   0.3...

Here is an example demonstrating the use of the "hinge_loss()"
function with a svm classifier in a multiclass problem:

   >>> X = np.array([[0], [1], [2], [3]])
   >>> Y = np.array([0, 1, 2, 3])
   >>> labels = np.array([0, 1, 2, 3])

   >>> est = svm.LinearSVC()
   >>> est.fit(X, Y)
   LinearSVC()
   >>> pred_decision = est.decision_function([[-1], [2], [3]])
   >>> y_true = [0, 2, 3]
   >>> hinge_loss(y_true, pred_decision, labels=labels)
   0.56...

3.3.2.12. Log loss

Log loss, also called logistic regression loss or cross-entropy loss,
is defined on probability estimates.  It is commonly used in
(multinomial) logistic regression and neural networks, as well as in
some variants of expectation-maximization, and can be used to evaluate
the probability outputs ("predict_proba") of a classifier instead of
its discrete predictions.

For binary classification with a true label y \in \{0,1\} and a
probability estimate p = \operatorname{Pr}(y = 1), the log loss per
sample is the negative log-likelihood of the classifier given the true
label:

   L_{\log}(y, p) = -\log \operatorname{Pr}(y|p) = -(y \log (p) + (1 -
   y) \log (1 - p))

This extends to the multiclass case as follows. Let the true labels
for a set of samples be encoded as a 1-of-K binary indicator matrix Y,
i.e., y_{i,k} = 1 if sample i has label k taken from a set of K
labels. Let P be a matrix of probability estimates, with p_{i,k} =
\operatorname{Pr}(y_{i,k} = 1). Then the log loss of the whole set is

   L_{\log}(Y, P) = -\log \operatorname{Pr}(Y|P) = - \frac{1}{N}

   \sum_{i=0}^{N-1} \sum_{k=0}^{K-1} y_{i,k} \log p_{i,k}

To see how this generalizes the binary log loss given above, note that
in the binary case, p_{i,0} = 1 - p_{i,1} and y_{i,0} = 1 - y_{i,1},
so expanding the inner sum over y_{i,k} \in \{0,1\} gives the binary
log loss.

The "log_loss()" function computes log loss given a list of ground-
truth labels and a probability matrix, as returned by an estimator's

>>> from sklearn.metrics import log_loss
>>> y_true = [0, 0, 1, 1]
>>> y_pred = [[.9, .1], [.8, .2], [.3, .7], [.01, .99]]
>>> log_loss(y_true, y_pred)
0.1738...

The first "[.9, .1]" in "y_pred" denotes 90% probability that the
first sample has label 0.  The log loss is non-negative.

3.3.2.13. Matthews correlation coefficient

The "matthews_corrcoef()" function computes the Matthew's correlation
coefficient (MCC) for binary classes.  Quoting Wikipedia:

   "The Matthews correlation coefficient is used in machine learning
   as a measure of the quality of binary (two-class) classifications.
   It takes into account true and false positives and negatives and is
   generally regarded as a balanced measure which can be used even if
   the classes are of very different sizes. The MCC is in essence a

generally regarded as a balanced measure which can be used even if
   correlation coefficient value between -1 and +1. A coefficient of
   +1 represents a perfect prediction, 0 an average random prediction
   and -1 an inverse prediction. The statistic is also known as the
   phi coefficient."

In the binary (two-class) case, tp, tn, fp and fn are respectively the
number of true positives, true negatives, false positives and false
negatives, the MCC is defined as

   MCC = \frac{tp \times tn - fp \times fn}{\sqrt{(tp + fp)(tp +
   fn)(tn + fp)(tn + fn)}}.

In the multiclass case, the Matthews correlation coefficient can be
defined in terms of a "confusion_matrix()" C for K classes.  To
simplify the definition consider the following intermediate variables:

* t_k=\sum_{i}^{K} C_{ik} the number of times class k truly occurred,

* p_k=\sum_{i}^{K} C_{ki} the number of times class k was predicted,

* c=\sum_{k}^{K} C_{kk} the total number of samples correctly
  predicted,

* s=\sum_{i}^{K} \sum_{j}^{K} C_{ij} the total number of samples.

Then the multiclass MCC is defined as:

   MCC = \frac{     c \times s - \sum_{k}^{K} p_k \times t_k }{\sqrt{
   (s^2 - \sum_{k}^{K} p_k^2) \times     (s^2 - \sum_{k}^{K} t_k^2) }}

When there are more than two labels, the value of the MCC will no
longer range between -1 and +1. Instead the minimum value will be
somewhere between -1 and 0 depending on the number and distribution of

ground true labels. The maximum value is always +1.

Here is a small example illustrating the usage of the
"matthews_corrcoef()" function:

>>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)
-0.33...

3.3.2.14. Multi-label confusion matrix

The "multilabel_confusion_matrix()" function computes class-wise
(default) or sample-wise (samplewise=True) multilabel confusion matrix
to evaluate the accuracy of a classification.
multilabel_confusion_matrix also treats multiclass data as if it were
multilabel, as this is a transformation commonly applied to evaluate
multiclass problems with binary classification metrics (such as
precision, recall, etc.).

When calculating class-wise multilabel confusion matrix C, the count
of true negatives for class i is C_{i,0,0}, false negatives is
C_{i,1,0}, true positives is C_{i,1,1} and false positives is
C_{i,0,1}.

Here is an example demonstrating the use of the
"multilabel_confusion_matrix()" function with *multilabel indicator
matrix* input:

   >>> from sklearn.metrics import multilabel_confusion_matrix
   >>> y_true = np.array([[1, 0, 1],
   ...                    [0, 1, 0]])
   >>> y_pred = np.array([[1, 0, 0],

   ...                    [0, 1, 1]])
   >>> multilabel_confusion_matrix(y_true, y_pred)
   array([[[1, 0],
           [0, 1]],

          [[1, 0],
           [0, 1]],

          [[0, 1],
           [1, 0]]])

Or a confusion matrix can be constructed for each sample's labels:

>>> multilabel_confusion_matrix(y_true, y_pred, samplewise=True)
array([[[1, 0],
        [1, 1]],

       [[1, 1],
        [0, 1]]])

"multilabel_confusion_matrix()" function with *multiclass* input:

   >>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
   >>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
   >>> multilabel_confusion_matrix(y_true, y_pred,
   ...                             labels=["ant", "bird", "cat"])
   array([[[3, 1],
           [0, 2]],

          [[5, 0],
           [1, 0]],

          [[2, 1],
           [1, 2]]])

Here are some examples demonstrating the use of the
"multilabel_confusion_matrix()" function to calculate recall (or
sensitivity), specificity, fall out and miss rate for each class in a
problem with multilabel indicator matrix input.

Calculating recall (also called the true positive rate or the
sensitivity) for each class:

   >>> y_true = np.array([[0, 0, 1],
   ...                    [0, 1, 0],

   >>> y_pred = np.array([[0, 1, 0],
   ...                    [0, 0, 1],
   >>> mcm = multilabel_confusion_matrix(y_true, y_pred)
   >>> tn = mcm[:, 0, 0]
   >>> tp = mcm[:, 1, 1]
   >>> fn = mcm[:, 1, 0]
   >>> fp = mcm[:, 0, 1]
   >>> tp / (tp + fn)
   array([1. , 0.5, 0. ])

Calculating specificity (also called the true negative rate) for each
class:

   >>> tn / (tn + fp)
   array([1. , 0. , 0.5])

Calculating fall out (also called the false positive rate) for each
class:

   >>> fp / (fp + tn)
   array([0. , 1. , 0.5])

Calculating miss rate (also called the false negative rate) for each
class:

   >>> fn / (fn + tp)
   array([0. , 0.5, 1. ])

3.3.2.15. Receiver operating characteristic (ROC)

The function "roc_curve()" computes the receiver operating
characteristic curve, or ROC curve. Quoting Wikipedia :

   "A receiver operating characteristic (ROC), or simply ROC curve, is
   a graphical plot which illustrates the performance of a binary
   classifier system as its discrimination threshold is varied. It is
   created by plotting the fraction of true positives out of the
   positives (TPR = true positive rate) vs. the fraction of false
   positives out of the negatives (FPR = false positive rate), at

created by plotting the fraction of true positives out of the
   various threshold settings. TPR is also known as sensitivity, and
   FPR is one minus the specificity or true negative rate."

This function requires the true binary value and the target scores,
which can either be probability estimates of the positive class,
confidence values, or binary decisions. Here is a small example of how
to use the "roc_curve()" function:

   >>> from sklearn.metrics import roc_curve
   >>> y = np.array([1, 1, 2, 2])
   >>> scores = np.array([0.1, 0.4, 0.35, 0.8])
   >>> fpr, tpr, thresholds = roc_curve(y, scores, pos_label=2)
   >>> fpr
   array([0. , 0. , 0.5, 0.5, 1. ])
   >>> tpr
   array([0. , 0.5, 0.5, 1. , 1. ])
   >>> thresholds
   array([1.8 , 0.8 , 0.4 , 0.35, 0.1 ])

Compared to metrics such as the subset accuracy, the Hamming loss, or
the F1 score, ROC doesn't require optimizing a threshold for each
label.

The "roc_auc_score()" function, denoted by ROC-AUC or AUROC, computes
the area under the ROC curve. By doing so, the curve information is
summarized in one number.

The following figure shows the ROC curve and ROC-AUC score for a
classifier aimed to distinguish the virginica flower from the rest of
the species in the *iris_dataset*:

[image]

For more information see the Wikipedia article on AUC.

3.3.2.15.1. Binary case

[image]

In the **binary case**, you can either provide the probability
estimates, using the *classifier.predict_proba()* method, or the non-
thresholded decision values given by the
*classifier.decision_function()* method. In the case of providing the
probability estimates, the probability of the class with the "greater
label" should be provided. The "greater label" corresponds to
*classifier.classes_[1]* and thus *classifier.predict_proba(X)[:, 1]*.
Therefore, the *y_score* parameter is of size (n_samples,).

>>> from sklearn.datasets import load_breast_cancer
>>> from sklearn.metrics import roc_auc_score
>>> X, y = load_breast_cancer(return_X_y=True)
>>> clf = LogisticRegression(solver="liblinear").fit(X, y)
>>> clf.classes_
array([0, 1])

We can use the probability estimates corresponding to
*clf.classes_[1]*.

>>> y_score = clf.predict_proba(X)[:, 1]
>>> roc_auc_score(y, y_score)
0.99...

Otherwise, we can use the non-thresholded decision values

>>> roc_auc_score(y, clf.decision_function(X))
0.99...

3.3.2.15.2. Multi-class case

The "roc_auc_score()" function can also be used in **multi-class
classification**. Two averaging strategies are currently supported:
the one-vs-one algorithm computes the average of the pairwise ROC AUC

scores, and the one-vs-rest algorithm computes the average of the ROC
AUC scores for each class against all other classes. In both cases,
the predicted labels are provided in an array with values from 0 to
"n_classes", and the scores correspond to the probability estimates
that a sample belongs to a particular class. The OvO and OvR
algorithms support weighting uniformly ("average='macro'") and by
prevalence ("average='weighted'").

**One-vs-one Algorithm**: Computes the average AUC of all possible
pairwise combinations of classes. [HT2001] defines a multiclass AUC
metric weighted uniformly:

   \frac{1}{c(c-1)}\sum_{j=1}^{c}\sum_{k > j}^c (\text{AUC}(j | k) +
   \text{AUC}(k | j))

where c is the number of classes and \text{AUC}(j | k) is the AUC with
class j as the positive class and class k as the negative class. In
general, \text{AUC}(j | k) \neq \text{AUC}(k | j)) in the multiclass
case. This algorithm is used by setting the keyword argument
"multiclass" to "'ovo'" and "average" to "'macro'".

The [HT2001] multiclass AUC metric can be extended to be weighted by
the prevalence:

   \frac{1}{c(c-1)}\sum_{j=1}^{c}\sum_{k > j}^c p(j \cup k)(
   \text{AUC}(j | k) + \text{AUC}(k | j))

where c is the number of classes. This algorithm is used by setting

the keyword argument "multiclass" to "'ovo'" and "average" to
"'weighted'". The "'weighted'" option returns a prevalence-weighted
average as described in [FC2009].

**One-vs-rest Algorithm**: Computes the AUC of each class against the
rest [PD2000]. The algorithm is functionally the same as the
multilabel case. To enable this algorithm set the keyword argument
"multiclass" to "'ovr'". Additionally to "'macro'" [F2006] and
"'weighted'" [F2001] averaging, OvR supports "'micro'" averaging.

In applications where a high false positive rate is not tolerable the
parameter "max_fpr" of "roc_auc_score()" can be used to summarize the
ROC curve up to the given limit.

The following figure shows the micro-averaged ROC curve and its
corresponding ROC-AUC score for a classifier aimed to distinguish the
the different species in the *iris_dataset*:

[image]

3.3.2.15.3. Multi-label case

In **multi-label classification**, the "roc_auc_score()" function is
extended by averaging over the labels as above. In this case, you
should provide a *y_score* of shape *(n_samples, n_classes)*. Thus,
when using the probability estimates, one needs to select the
probability of the class with the greater label for each output.

>>> from sklearn.datasets import make_multilabel_classification

>>> X, y = make_multilabel_classification(random_state=0)
>>> inner_clf = LogisticRegression(solver="liblinear", random_state=0)
>>> clf = MultiOutputClassifier(inner_clf).fit(X, y)
>>> y_score = np.transpose([y_pred[:, 1] for y_pred in clf.predict_proba(X)])
>>> roc_auc_score(y, y_score, average=None)
array([0.82..., 0.86..., 0.94..., 0.85... , 0.94...])

And the decision values do not require such processing.

>>> from sklearn.linear_model import RidgeClassifierCV
>>> clf = RidgeClassifierCV().fit(X, y)
>>> y_score = clf.decision_function(X)
array([0.81..., 0.84... , 0.93..., 0.87..., 0.94...])

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_model_selection_plot_roc.py* for an
  example of using ROC to evaluate the quality of the output of a

* See *sphx_glr_auto_examples_model_selection_plot_roc_crossval.py*
  for an example of using ROC to evaluate classifier output quality,
  using cross-validation.

* See *sphx_glr_auto_examples_applications_plot_species_distribution_
  modeling.py* for an example of using ROC to model species

[HT2001] Hand, D.J. and Till, R.J., (2001). A simple generalisation of
         the area under the ROC curve for multiple class

         classification problems. Machine learning, 45(2), pp.171-186.

[FC2009] Ferri, Cèsar & Hernandez-Orallo, Jose & Modroiu, R. (2009).
         An Experimental Comparison of Performance Measures for
         Classification. Pattern Recognition Letters. 30. 27-38.

[PD2000] Provost, F., Domingos, P. (2000). Well-trained PETs:
         Improving probability estimation trees (Section 6.2), CeDER
         Working Paper #IS-00-04, Stern School of Business, New York
         University.

[F2006] Fawcett, T., 2006. An introduction to ROC analysis. Pattern
        Recognition Letters, 27(8), pp. 861-874.

[F2001] Fawcett, T., 2001. Using rule sets to maximize ROC performance
        In Data Mining, 2001. Proceedings IEEE International
        Conference, pp. 131-138.

3.3.2.16. Detection error tradeoff (DET)

The function "det_curve()" computes the detection error tradeoff curve
(DET) curve [WikipediaDET2017]. Quoting Wikipedia:

   "A detection error tradeoff (DET) graph is a graphical plot of
   error rates for binary classification systems, plotting false
   reject rate vs. false accept rate. The x- and y-axes are scaled
   non-linearly by their standard normal deviates (or just by
   logarithmic transformation), yielding tradeoff curves that are more

reject rate vs. false accept rate. The x- and y-axes are scaled
   linear than ROC curves, and use most of the image area to highlight
   the differences of importance in the critical operating region."

DET curves are a variation of receiver operating characteristic (ROC)
curves where False Negative Rate is plotted on the y-axis instead of
True Positive Rate. DET curves are commonly plotted in normal deviate
scale by transformation with \phi^{-1} (with \phi being the cumulative
distribution function). The resulting performance curves explicitly
visualize the tradeoff of error types for given classification
algorithms. See [Martin1997] for examples and further motivation.

This figure compares the ROC and DET curves of two example classifiers
on the same classification task:

[image]

**Properties:**

* DET curves form a linear curve in normal deviate scale if the
  detection scores are normally (or close-to normally) distributed. It
  was shown by [Navratil2007] that the reverse is not necessarily true
  and even more general distributions are able to produce linear DET
  curves.

* The normal deviate scale transformation spreads out the points such
  that a comparatively larger space of plot is occupied. Therefore
  curves with similar classification performance might be easier to
  distinguish on a DET plot.

that a comparatively larger space of plot is occupied. Therefore

* With False Negative Rate being "inverse" to True Positive Rate the
  point of perfection for DET curves is the origin (in contrast to the
  top left corner for ROC curves).

**Applications and limitations:**

DET curves are intuitive to read and hence allow quick visual
assessment of a classifier's performance. Additionally DET curves can
be consulted for threshold analysis and operating point selection.
This is particularly helpful if a comparison of error types is
required.

On the other hand DET curves do not provide their metric as a single
number. Therefore for either automated evaluation or comparison to
other classification tasks metrics like the derived area under ROC
curve might be better suited.

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_model_selection_plot_det.py* for an
  example comparison between receiver operating characteristic (ROC)
  curves and Detection error tradeoff (DET) curves.

[WikipediaDET2017] Wikipedia contributors. Detection error tradeoff.
                   Wikipedia, The Free Encyclopedia. September 4,
                   2017, 23:33 UTC. Available at: https://en.wikipedi
                   a.org/w/index.php?title=Detection_error_tradeoff&o
                   ldid=798982054. Accessed February 19, 2018.

[Martin1997] A. Martin, G. Doddington, T. Kamm, M. Ordowski, and M.

a.org/w/index.php?title=Detection_error_tradeoff&o

             Przybocki, The DET Curve in Assessment of Detection Task
             Performance, NIST 1997.

[Navratil2007] J. Navractil and D. Klusacek, "On Linear DETs," 2007
               IEEE International Conference on Acoustics, Speech and
               Signal Processing - ICASSP '07, Honolulu, HI, 2007, pp.
               IV-229-IV-232.

3.3.2.17. Zero one loss

The "zero_one_loss()" function computes the sum or the average of the
0-1 classification loss (L_{0-1}) over n_{\text{samples}}. By default,
the function normalizes over the sample. To get the sum of the
L_{0-1}, set "normalize" to "False".

In multilabel classification, the "zero_one_loss()" scores a subset as
one if its labels strictly match the predictions, and as a zero if
there are any errors.  By default, the function returns the percentage
of imperfectly predicted subsets.  To get the count of such subsets
instead, set "normalize" to "False"

corresponding true value, then the 0-1 loss L_{0-1} is defined as:

   L_{0-1}(y, \hat{y}) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples}-1} 1(\hat{y}_i \not= y_i)

where 1(x) is the indicator function. The zero one loss can also be
computed as zero-one loss = 1 - accuracy.

\sum_{i=0}^{n_\text{samples}-1} 1(\hat{y}_i \not= y_i)

>>> from sklearn.metrics import zero_one_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1

In the multilabel case with binary label indicators, where the first
label set [0,1] has an error:

   >>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
   0.5

   >>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)),  normalize=False)
   1

Example:
^^^^^^^^

* See *sphx_glr_auto_examples_feature_selection_plot_rfe_with_cross_v
  alidation.py* for an example of zero one loss usage to perform
  recursive feature elimination with cross-validation.

3.3.2.18. Brier score loss

The "brier_score_loss()" function computes the Brier score for binary
classes [Brier1950]. Quoting Wikipedia:

   "The Brier score is a proper score function that measures the
   accuracy of probabilistic predictions. It is applicable to tasks in
   which predictions must assign probabilities to a set of mutually
   exclusive discrete outcomes."

This function returns the mean squared error of the actual outcome y
\in \{0,1\} and the predicted probability estimate p =
\operatorname{Pr}(y = 1) (*predict_proba*) as outputted by:

   BS = \frac{1}{n_{\text{samples}}} \sum_{i=0}^{n_{\text{samples}} -
   1}(y_i - p_i)^2

The Brier score loss is also between 0 to 1 and the lower the value
(the mean square difference is smaller), the more accurate the
prediction is.

Here is a small example of usage of this function:

   >>> from sklearn.metrics import brier_score_loss
   >>> y_true = np.array([0, 1, 1, 0])
   >>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
   >>> y_prob = np.array([0.1, 0.9, 0.8, 0.4])
   >>> y_pred = np.array([0, 1, 1, 0])
   >>> brier_score_loss(y_true, y_prob)
   0.055
   >>> brier_score_loss(y_true, 1 - y_prob, pos_label=0)
   0.055
   >>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
   0.055
   >>> brier_score_loss(y_true, y_prob > 0.5)
   0.0

The Brier score can be used to assess how well a classifier is
calibrated. However, a lower Brier score loss does not always mean a
better calibration. This is because, by analogy with the bias-variance
decomposition of the mean squared error, the Brier score loss can be
decomposed as the sum of calibration loss and refinement loss
[Bella2012]. Calibration loss is defined as the mean squared deviation
from empirical probabilities derived from the slope of ROC segments.
Refinement loss can be defined as the expected optimal loss as
measured by the area under the optimal cost curve. Refinement loss can

change independently from calibration loss, thus a lower Brier score
loss does not necessarily mean a better calibrated model. "Only when
refinement loss remains the same does a lower Brier score loss always
mean better calibration" [Bella2012], [Flach2008].

Example:
^^^^^^^^

* See *sphx_glr_auto_examples_calibration_plot_calibration.py* for an
  example of Brier score loss usage to perform probability calibration
  of classifiers.

[Brier1950] G. Brier, Verification of forecasts expressed in terms of
            probability, Monthly weather review 78.1 (1950)

[Bella2012] Bella, Ferri, Hernández-Orallo, and Ramírez-Quintana
            "Calibration of Machine Learning Models" in Khosrow-Pour,
            M. "Machine learning: concepts, methodologies, tools and
            applications." Hershey, PA: Information Science Reference
            (2012).

[Flach2008] Flach, Peter, and Edson Matsubara. "On classification,
            ranking, and probability estimation." Dagstuhl Seminar
            Proceedings. Schloss Dagstuhl-Leibniz-Zentrum fr
            Informatik (2008).

3.3.2.19. Class likelihood ratios

The "class_likelihood_ratios()" function computes the positive and
negative likelihood ratios LR_\pm for binary classes, which can be

interpreted as the ratio of post-test to pre-test odds as explained
below. As a consequence, this metric is invariant w.r.t. the class
prevalence (the number of samples in the positive class divided by the
total number of samples) and **can be extrapolated between populations
regardless of any possible class imbalance.**

The LR_\pm metrics are therefore very useful in settings where the
data available to learn and evaluate a classifier is a study
population with nearly balanced classes, such as a case-control study,
while the target application, i.e. the general population, has very
low prevalence.

The positive likelihood ratio LR_+ is the probability of a classifier
to correctly predict that a sample belongs to the positive class
divided by the probability of predicting the positive class for a
sample belonging to the negative class:

   LR_+ = \frac{\text{PR}(P+|T+)}{\text{PR}(P+|T-)}.

The notation here refers to predicted (P) or true (T) label and the
sign + and - refer to the positive and negative class, respectively,
e.g. P+ stands for "predicted positive".

Analogously, the negative likelihood ratio LR_- is the probability of
a sample of the positive class being classified as belonging to the
negative class divided by the probability of a sample of the negative
class being correctly classified:

   LR_- = \frac{\text{PR}(P-|T+)}{\text{PR}(P-|T-)}.

For classifiers above chance LR_+ above 1 **higher is better**, while
LR_- ranges from 0 to 1 and **lower is better**. Values of
LR_\pm\approx 1 correspond to chance level.

Notice that probabilities differ from counts, for instance
\operatorname{PR}(P+|T+) is not equal to the number of true positive
counts "tp" (see the wikipedia page for the actual formulas).

**Interpretation across varying prevalence:**

Both class likelihood ratios are interpretable in terms of an odds
ratio (pre-test and post-tests):

   \text{post-test odds} = \text{Likelihood ratio} \times \text{pre-
   test odds}.

Odds are in general related to probabilities via

   \text{odds} = \frac{\text{probability}}{1 - \text{probability}},

or equivalently

   \text{probability} = \frac{\text{odds}}{1 + \text{odds}}.

On a given population, the pre-test probability is given by the
prevalence. By converting odds to probabilities, the likelihood ratios
can be translated into a probability of truly belonging to either
class before and after a classifier prediction:

   \text{post-test odds} = \text{Likelihood ratio} \times \frac{\text
   {pre-test probability}}{1 - \text{pre-test probability}},

   \text{post-test probability} = \frac{\text{post-test odds}}{1 +
   \text{post-test odds}}.

**Mathematical divergences:**

{pre-test probability}}{1 - \text{pre-test probability}},

The positive likelihood ratio is undefined when fp = 0, which can be
interpreted as the classifier perfectly identifying positive cases. If
fp = 0 and additionally tp = 0, this leads to a zero/zero division.
This happens, for instance, when using a *DummyClassifier* that always
predicts the negative class and therefore the interpretation as a
perfect classifier is lost.

The negative likelihood ratio is undefined when tn = 0. Such
divergence is invalid, as LR_- > 1 would indicate an increase in the
odds of a sample belonging to the positive class after being
classified as negative, as if the act of classifying caused the
positive condition. This includes the case of a *DummyClassifier* that
always predicts the positive class (i.e. when tn=fn=0).

Both class likelihood ratios are undefined when tp=fn=0, which means
that no samples of the positive class were present in the testing set.
This can also happen when cross-validating highly imbalanced data.

In all the previous cases the "class_likelihood_ratios()" function
raises by default an appropriate warning message and returns *nan* to
avoid pollution when averaging over cross-validation folds.

For a worked-out demonstration of the "class_likelihood_ratios()"
function, see the example below.

Examples:
^^^^^^^^^

Examples:
^^^^^^^^^

* *sphx_glr_auto_examples_model_selection_plot_likelihood_ratios.py*

* Wikipedia entry for Likelihood ratios in diagnostic testing

* Brenner, H., & Gefeller, O. (1997). Variation of sensitivity,
  specificity, likelihood ratios and predictive values with disease
  prevalence. Statistics in medicine, 16(9), 981-991.

3.3.3. Multilabel ranking metrics

In multilabel learning, each sample can have any number of ground
truth labels associated with it. The goal is to give high scores and
better rank to the ground truth labels.

3.3.3.1. Coverage error

The "coverage_error()" function computes the average number of labels
that have to be included in the final prediction such that all true
labels are predicted. This is useful if you want to know how many top-
scored-labels you have to predict in average without missing any true
one. The best value of this metrics is thus the average number of true
labels.

Note:

  Our implementation's score is 1 greater than the one given in
  Tsoumakas et al., 2010. This extends it to handle the degenerate
  case in which an instance has 0 true labels.

Formally, given a binary indicator matrix of the ground truth labels y
\in \left\{0, 1\right\}^{n_\text{samples} \times n_\text{labels}} and

case in which an instance has 0 true labels.

the score associated with each label \hat{f} \in
\mathbb{R}^{n_\text{samples} \times n_\text{labels}}, the coverage is
defined as

   coverage(y, \hat{f}) = \frac{1}{n_{\text{samples}}}
   \sum_{i=0}^{n_{\text{samples}} - 1} \max_{j:y_{ij} = 1}
   \text{rank}_{ij}

with \text{rank}_{ij} = \left|\left\{k: \hat{f}_{ik} \geq \hat{f}_{ij}
\right\}\right|. Given the rank definition, ties in "y_scores" are
broken by giving the maximal rank that would have been assigned to all
tied values.

   >>> from sklearn.metrics import coverage_error
   >>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
   >>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
   >>> coverage_error(y_true, y_score)
   2.5

3.3.3.2. Label ranking average precision

The "label_ranking_average_precision_score()" function implements
label ranking average precision (LRAP). This metric is linked to the
"average_precision_score()" function, but is based on the notion of
label ranking instead of precision and recall.

Label ranking average precision (LRAP) averages over the samples the
answer to the following question: for each ground truth label, what
fraction of higher-ranked labels were true labels? This performance

measure will be higher if you are able to give better rank to the
labels associated with each sample. The obtained score is always
strictly greater than 0, and the best value is 1. If there is exactly
one relevant label per sample, label ranking average precision is
equivalent to the mean reciprocal rank.

\mathbb{R}^{n_\text{samples} \times n_\text{labels}}, the average
precision is defined as

   LRAP(y, \hat{f}) = \frac{1}{n_{\text{samples}}}
   \sum_{i=0}^{n_{\text{samples}} - 1} \frac{1}{||y_i||_0}
   \sum_{j:y_{ij} = 1} \frac{|\mathcal{L}_{ij}|}{\text{rank}_{ij}}

where \mathcal{L}_{ij} = \left\{k: y_{ik} = 1, \hat{f}_{ik} \geq
\hat{f}_{ij} \right\}, \text{rank}_{ij} = \left|\left\{k: \hat{f}_{ik}
\geq \hat{f}_{ij} \right\}\right|, |\cdot| computes the cardinality of
the set (i.e., the number of elements in the set), and ||\cdot||_0 is
the \ell_0 "norm" (which computes the number of nonzero elements in a
vector).

   >>> from sklearn.metrics import label_ranking_average_precision_score

>>> from sklearn.metrics import label_ranking_average_precision_score
   >>> label_ranking_average_precision_score(y_true, y_score)
   0.416...

3.3.3.3. Ranking loss

The "label_ranking_loss()" function computes the ranking loss which
averages over the samples the number of label pairs that are
incorrectly ordered, i.e. true labels have a lower score than false
labels, weighted by the inverse of the number of ordered pairs of
false and true labels. The lowest achievable ranking loss is zero.

\mathbb{R}^{n_\text{samples} \times n_\text{labels}}, the ranking loss
is defined as

   ranking\_loss(y, \hat{f}) =  \frac{1}{n_{\text{samples}}}
   \sum_{i=0}^{n_{\text{samples}} - 1}
   \frac{1}{||y_i||_0(n_\text{labels} - ||y_i||_0)}   \left|\left\{(k,
   l): \hat{f}_{ik} \leq \hat{f}_{il}, y_{ik} = 1, y_{il} = 0
   \right\}\right|

where |\cdot| computes the cardinality of the set (i.e., the number of
elements in the set) and ||\cdot||_0 is the \ell_0 "norm" (which
computes the number of nonzero elements in a vector).

   >>> from sklearn.metrics import label_ranking_loss

   >>> label_ranking_loss(y_true, y_score)
   0.75...
   >>> # With the following prediction, we have perfect and minimal loss
   >>> y_score = np.array([[1.0, 0.1, 0.2], [0.1, 0.2, 0.9]])
   0.0

* Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-
  label data. In Data mining and knowledge discovery handbook (pp.
  667-685). Springer US.

3.3.3.4. Normalized Discounted Cumulative Gain

Discounted Cumulative Gain (DCG) and Normalized Discounted Cumulative
Gain (NDCG) are ranking metrics implemented in "dcg_score()" and
"ndcg_score()" ; they compare a predicted order to ground-truth
scores, such as the relevance of answers to a query.

From the Wikipedia page for Discounted Cumulative Gain:

"Discounted cumulative gain (DCG) is a measure of ranking quality. In
information retrieval, it is often used to measure effectiveness of
web search engine algorithms or related applications. Using a graded
relevance scale of documents in a search-engine result set, DCG
measures the usefulness, or gain, of a document based on its position
in the result list. The gain is accumulated from the top of the result

list to the bottom, with the gain of each result discounted at lower
ranks"

DCG orders the true targets (e.g. relevance of query answers) in the
predicted order, then multiplies them by a logarithmic decay and sums
the result. The sum can be truncated after the first K results, in
which case we call it DCG@K. NDCG, or NDCG@K is DCG divided by the DCG
obtained by a perfect prediction, so that it is always between 0 and
1. Usually, NDCG is preferred to DCG.

Compared with the ranking loss, NDCG can take into account relevance
scores, rather than a ground-truth ranking. So if the ground-truth
consists only of an ordering, the ranking loss should be preferred; if
the ground-truth consists of actual usefulness scores (e.g. 0 for
irrelevant, 1 for relevant, 2 for very relevant), NDCG can be used.

For one sample, given the vector of continuous ground-truth values for
each target y \in \mathbb{R}^{M}, where M is the number of outputs,
and the prediction \hat{y}, which induces the ranking function f, the
DCG score is

   \sum_{r=1}^{\min(K, M)}\frac{y_{f(r)}}{\log(1 + r)}

and the NDCG score is the DCG score divided by the DCG score obtained
for y.

* Wikipedia entry for Discounted Cumulative Gain

* Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based
  evaluation of IR techniques. ACM Transactions on Information Systems

  (TOIS), 20(4), 422-446.

* Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013,
  May). A theoretical analysis of NDCG ranking measures. In
  Proceedings of the 26th Annual Conference on Learning Theory (COLT
  2013)

* McSherry, F., & Najork, M. (2008, March). Computing information
  retrieval performance measures efficiently in the presence of tied
  scores. In European conference on information retrieval (pp.
  414-421). Springer, Berlin, Heidelberg.

3.3.4. Regression metrics

utility functions to measure regression performance. Some of those
have been enhanced to handle the multioutput case:
"mean_squared_error()", "mean_absolute_error()", "r2_score()",
"explained_variance_score()", "mean_pinball_loss()",
"d2_pinball_score()" and "d2_absolute_error_score()".

These functions have a "multioutput" keyword argument which specifies
the way the scores or losses for each individual target should be
averaged. The default is "'uniform_average'", which specifies a
uniformly weighted mean over outputs. If an "ndarray" of shape
"(n_outputs,)" is passed, then its entries are interpreted as weights
and an according weighted average is returned. If "multioutput" is

"'raw_values'", then all unaltered individual scores or losses will be
returned in an array of shape "(n_outputs,)".

The "r2_score()" and "explained_variance_score()" accept an additional
value "'variance_weighted'" for the "multioutput" parameter. This
option leads to a weighting of each individual score by the variance
of the corresponding target variable. This setting quantifies the
globally captured unscaled variance. If the target variables are of
different scale, then this score puts more importance on explaining
the higher variance variables. "multioutput='variance_weighted'" is
the default value for "r2_score()" for backward compatibility. This
will be changed to "uniform_average" in the future.

3.3.4.1. R² score, the coefficient of determination

The "r2_score()" function computes the coefficient of determination,
usually denoted as R^2.

It represents the proportion of variance (of y) that has been
explained by the independent variables in the model. It provides an
indication of goodness of fit and therefore a measure of how well
unseen samples are likely to be predicted by the model, through the
proportion of explained variance.

As such variance is dataset dependent, R^2 may not be meaningfully

comparable across different datasets. Best possible score is 1.0 and
it can be negative (because the model can be arbitrarily worse). A
constant model that always predicts the expected (average) value of y,
disregarding the input features, would get an R^2 score of 0.0.

Note: when the prediction residuals have zero mean, the R^2 score and
the Explained variance score are identical.

corresponding true value for total n samples, the estimated R^2 is

   R^2(y, \hat{y}) = 1 - \frac{\sum_{i=1}^{n} (y_i -
   \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}

where \bar{y} = \frac{1}{n} \sum_{i=1}^{n} y_i and \sum_{i=1}^{n} (y_i
- \hat{y}_i)^2 = \sum_{i=1}^{n} \epsilon_i^2.

Note that "r2_score()" calculates unadjusted R^2 without correcting
for bias in sample variance of y.

In the particular case where the true target is constant, the R^2
score is not finite: it is either "NaN" (perfect predictions) or
"-Inf" (imperfect predictions). Such non-finite scores may prevent
correct model optimization such as grid-search cross-validation to be
performed correctly. For this reason the default behaviour of
"r2_score()" is to replace them with 1.0 (perfect predictions) or 0.0
(imperfect predictions). If "force_finite" is set to "False", this

score falls back on the original R^2 definition.

Here is a small example of usage of the "r2_score()" function:

   >>> y_true = [3, -0.5, 2, 7]
   >>> y_pred = [2.5, 0.0, 2, 8]
   >>> r2_score(y_true, y_pred)
   0.948...
   >>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
   >>> y_pred = [[0, 2], [-1, 2], [8, -5]]
   >>> r2_score(y_true, y_pred, multioutput='variance_weighted')
   0.938...
   >>> r2_score(y_true, y_pred, multioutput='uniform_average')
   0.936...
   >>> r2_score(y_true, y_pred, multioutput='raw_values')
   array([0.965..., 0.908...])
   >>> r2_score(y_true, y_pred, multioutput=[0.3, 0.7])
   0.925...
   >>> y_true = [-2, -2, -2]
   >>> y_pred = [-2, -2, -2]
   1.0
   >>> r2_score(y_true, y_pred, force_finite=False)
   nan
   >>> y_pred = [-2, -2, -2 + 1e-8]
   0.0
   -inf

Example:
^^^^^^^^

* See
  *sphx_glr_auto_examples_linear_model_plot_lasso_and_elasticnet.py*
  for an example of R² score usage to evaluate Lasso and Elastic Net
  on sparse signals.

3.3.4.2. Mean absolute error

for an example of R² score usage to evaluate Lasso and Elastic Net

The "mean_absolute_error()" function computes mean absolute error, a
risk metric corresponding to the expected value of the absolute error
loss or l1-norm loss.

If \hat{y}_i is the predicted value of the i-th sample, and y_i is the
corresponding true value, then the mean absolute error (MAE) estimated
over n_{\text{samples}} is defined as

   \text{MAE}(y, \hat{y}) = \frac{1}{n_{\text{samples}}}
   \sum_{i=0}^{n_{\text{samples}}-1} \left| y_i - \hat{y}_i \right|.

Here is a small example of usage of the "mean_absolute_error()"
function:

   >>> from sklearn.metrics import mean_absolute_error
   >>> mean_absolute_error(y_true, y_pred)
   0.5
   0.75
   >>> mean_absolute_error(y_true, y_pred, multioutput='raw_values')
   array([0.5, 1. ])
   >>> mean_absolute_error(y_true, y_pred, multioutput=[0.3, 0.7])
   0.85...

3.3.4.3. Mean squared error

The "mean_squared_error()" function computes mean square error, a risk
metric corresponding to the expected value of the squared (quadratic)
error or loss.

corresponding true value, then the mean squared error (MSE) estimated

   \text{MSE}(y, \hat{y}) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples} - 1} (y_i - \hat{y}_i)^2.

Here is a small example of usage of the "mean_squared_error()"
function:

   >>> mean_squared_error(y_true, y_pred)
   0.375
   0.7083...

Examples:
^^^^^^^^^

* See *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_regress
  ion.py* for an example of mean squared error usage to evaluate
  gradient boosting regression.

3.3.4.4. Mean squared logarithmic error

The "mean_squared_log_error()" function computes a risk metric
corresponding to the expected value of the squared logarithmic
(quadratic) error or loss.

corresponding true value, then the mean squared logarithmic error
(MSLE) estimated over n_{\text{samples}} is defined as

   \text{MSLE}(y, \hat{y}) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples} - 1} (\log_e (1 + y_i) - \log_e (1 +
   \hat{y}_i) )^2.

Where \log_e (x) means the natural logarithm of x. This metric is best

\sum_{i=0}^{n_\text{samples} - 1} (\log_e (1 + y_i) - \log_e (1 +

to use when targets having exponential growth, such as population
counts, average sales of a commodity over a span of years etc. Note
that this metric penalizes an under-predicted estimate greater than an
over-predicted estimate.

Here is a small example of usage of the "mean_squared_log_error()"
function:

   >>> from sklearn.metrics import mean_squared_log_error
   >>> y_true = [3, 5, 2.5, 7]
   >>> y_pred = [2.5, 5, 4, 8]
   >>> mean_squared_log_error(y_true, y_pred)
   0.039...
   >>> y_true = [[0.5, 1], [1, 2], [7, 6]]
   >>> y_pred = [[0.5, 2], [1, 2.5], [8, 8]]
   0.044...

3.3.4.5. Mean absolute percentage error

The "mean_absolute_percentage_error()" (MAPE), also known as mean
absolute percentage deviation (MAPD), is an evaluation metric for
regression problems. The idea of this metric is to be sensitive to
relative errors. It is for example not changed by a global scaling of
the target variable.

corresponding true value, then the mean absolute percentage error
(MAPE) estimated over n_{\text{samples}} is defined as

   \text{MAPE}(y, \hat{y}) = \frac{1}{n_{\text{samples}}}
   \sum_{i=0}^{n_{\text{samples}}-1} \frac{{}\left| y_i - \hat{y}_i
   \right|}{\max(\epsilon, \left| y_i \right|)}

\text{MAPE}(y, \hat{y}) = \frac{1}{n_{\text{samples}}}

where \epsilon is an arbitrary small yet strictly positive number to
avoid undefined results when y is zero.

The "mean_absolute_percentage_error()" function supports multioutput.

Here is a small example of usage of the
"mean_absolute_percentage_error()" function:

   >>> from sklearn.metrics import mean_absolute_percentage_error
   >>> y_true = [1, 10, 1e6]
   >>> y_pred = [0.9, 15, 1.2e6]
   >>> mean_absolute_percentage_error(y_true, y_pred)
   0.2666...

In above example, if we had used *mean_absolute_error*, it would have
ignored the small magnitude values and only reflected the error in
prediction of highest magnitude value. But that problem is resolved in
case of MAPE because it calculates relative percentage error with
respect to actual output.

3.3.4.6. Median absolute error

The "median_absolute_error()" is particularly interesting because it
is robust to outliers. The loss is calculated by taking the median of
all absolute differences between the target and the prediction.

corresponding true value, then the median absolute error (MedAE)
estimated over n_{\text{samples}} is defined as

   \text{MedAE}(y, \hat{y}) = \text{median}(\mid y_1 - \hat{y}_1 \mid,
   \ldots, \mid y_n - \hat{y}_n \mid).

The "median_absolute_error()" does not support multioutput.

Here is a small example of usage of the "median_absolute_error()"
function:

   >>> from sklearn.metrics import median_absolute_error
   >>> median_absolute_error(y_true, y_pred)
   0.5

3.3.4.7. Max error

The "max_error()" function computes the maximum residual error , a
metric that captures the worst case error between the predicted value
and the true value. In a perfectly fitted single output regression
model, "max_error" would be "0" on the training set and though this
would be highly unlikely in the real world, this metric shows the
extent of error that the model had when it was fitted.

corresponding true value, then the max error is defined as

   \text{Max Error}(y, \hat{y}) = \max(| y_i - \hat{y}_i |)

Here is a small example of usage of the "max_error()" function:

   >>> from sklearn.metrics import max_error
   >>> y_true = [3, 2, 7, 1]
   >>> y_pred = [9, 2, 7, 1]
   >>> max_error(y_true, y_pred)
   6

The "max_error()" does not support multioutput.

3.3.4.8. Explained variance score

The "explained_variance_score()" computes the explained variance
regression score.

6

If \hat{y} is the estimated target output, y the corresponding
(correct) target output, and Var is Variance, the square of the
standard deviation, then the explained variance is estimated as
follow:

   explained\_{}variance(y, \hat{y}) = 1 - \frac{Var\{ y -
   \hat{y}\}}{Var\{y\}}

The best possible score is 1.0, lower values are worse.

Link to R² score, the coefficient of determination
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The difference between the explained variance score and the R² score,
the coefficient of determination is that when the explained variance
score does not account for systematic offset in the prediction. For
this reason, the R² score, the coefficient of determination should be
preferred in general.

In the particular case where the true target is constant, the
Explained Variance score is not finite: it is either "NaN" (perfect
predictions) or "-Inf" (imperfect predictions). Such non-finite scores
may prevent correct model optimization such as grid-search cross-
validation to be performed correctly. For this reason the default
behaviour of "explained_variance_score()" is to replace them with 1.0
(perfect predictions) or 0.0 (imperfect predictions). You can set the
"force_finite" parameter to "False" to prevent this fix from happening

and fallback on the original Explained Variance score.

Here is a small example of usage of the "explained_variance_score()"
function:

   >>> from sklearn.metrics import explained_variance_score
   >>> explained_variance_score(y_true, y_pred)
   0.957...
   >>> explained_variance_score(y_true, y_pred, multioutput='raw_values')
   array([0.967..., 1.        ])
   >>> explained_variance_score(y_true, y_pred, multioutput=[0.3, 0.7])
   0.990...
   1.0
   >>> explained_variance_score(y_true, y_pred, force_finite=False)
   nan
   0.0
   -inf

3.3.4.9. Mean Poisson, Gamma, and Tweedie deviances

The "mean_tweedie_deviance()" function computes the mean Tweedie
deviance error with a "power" parameter (p). This is a metric that
elicits predicted expectation values of regression targets.

Following special cases exist,

* when "power=0" it is equivalent to "mean_squared_error()".

* when "power=1" it is equivalent to "mean_poisson_deviance()".

* when "power=2" it is equivalent to "mean_gamma_deviance()".

corresponding true value, then the mean Tweedie deviance error (D) for
power p, estimated over n_{\text{samples}} is defined as

   \text{D}(y, \hat{y}) = \frac{1}{n_\text{samples}}
   \sum_{i=0}^{n_\text{samples} - 1} \begin{cases} (y_i-\hat{y}_i)^2,
   & \text{for }p=0\text{ (Normal)}\\ 2(y_i \log(y_i/\hat{y}_i) +
   \hat{y}_i - y_i),  & \text{for }p=1\text{ (Poisson)}\\
   2(\log(\hat{y}_i/y_i) + y_i/\hat{y}_i - 1),  & \text{for }p=2\text{
   (Gamma)}\\ 2\left(\frac{\max(y_i,0)^{2-p}}{(1-p)(2-p)}- \frac{y_i\
   ,\hat{y}_i^{1-p}}{1-p}+\frac{\hat{y}_i^{2-p}}{2-p}\right), &
   \text{otherwise} \end{cases}

Tweedie deviance is a homogeneous function of degree "2-power". Thus,
Gamma distribution with "power=2" means that simultaneously scaling
"y_true" and "y_pred" has no effect on the deviance. For Poisson
distribution "power=1" the deviance scales linearly, and for Normal
distribution ("power=0"), quadratically.  In general, the higher
"power" the less weight is given to extreme deviations between true
and predicted targets.

For instance, let's compare the two predictions 1.5 and 150 that are
both 50% larger than their corresponding true value.

The mean squared error ("power=0") is very sensitive to the prediction
difference of the second point,:

   >>> from sklearn.metrics import mean_tweedie_deviance
   >>> mean_tweedie_deviance([1.0], [1.5], power=0)
   0.25
   >>> mean_tweedie_deviance([100.], [150.], power=0)
   2500.0

If we increase "power" to 1,:

   >>> mean_tweedie_deviance([1.0], [1.5], power=1)
   0.18...
   >>> mean_tweedie_deviance([100.], [150.], power=1)
   18.9...

the difference in errors decreases. Finally, by setting, "power=2":

   >>> mean_tweedie_deviance([1.0], [1.5], power=2)
   0.14...
   >>> mean_tweedie_deviance([100.], [150.], power=2)
   0.14...

we would get identical errors. The deviance when "power=2" is thus
only sensitive to relative errors.

3.3.4.10. Pinball loss

The "mean_pinball_loss()" function is used to evaluate the predictive
performance of quantile regression models.

   \text{pinball}(y, \hat{y}) = \frac{1}{n_{\text{samples}}}
   \sum_{i=0}^{n_{\text{samples}}-1}  \alpha \max(y_i - \hat{y}_i, 0)
   + (1 - \alpha) \max(\hat{y}_i - y_i, 0)

The value of pinball loss is equivalent to half of
"mean_absolute_error()" when the quantile parameter "alpha" is set to
0.5.

Here is a small example of usage of the "mean_pinball_loss()"
function:

   >>> from sklearn.metrics import mean_pinball_loss
   >>> y_true = [1, 2, 3]

0.5.

function:

   >>> mean_pinball_loss(y_true, [0, 2, 3], alpha=0.1)
   0.03...
   >>> mean_pinball_loss(y_true, [1, 2, 4], alpha=0.1)
   0.3...
   >>> mean_pinball_loss(y_true, [0, 2, 3], alpha=0.9)
   0.3...
   >>> mean_pinball_loss(y_true, [1, 2, 4], alpha=0.9)
   0.03...
   >>> mean_pinball_loss(y_true, y_true, alpha=0.1)
   0.0
   >>> mean_pinball_loss(y_true, y_true, alpha=0.9)
   0.0

It is possible to build a scorer object with a specific choice of
"alpha":

   >>> mean_pinball_loss_95p = make_scorer(mean_pinball_loss, alpha=0.95)

Such a scorer can be used to evaluate the generalization performance
of a quantile regressor via cross-validation:

>>>
>>> X, y = make_regression(n_samples=100, random_state=0)
>>> estimator = GradientBoostingRegressor(
...     loss="quantile",
...     alpha=0.95,
...     random_state=0,
... )
>>> cross_val_score(estimator, X, y, cv=5, scoring=mean_pinball_loss_95p)
array([13.6..., 9.7..., 23.3..., 9.5..., 10.4...])

It is also possible to build scorer objects for hyper-parameter
tuning. The sign of the loss must be switched to ensure that greater
means better as explained in the example linked below.

Example:
^^^^^^^^

* See

Example:
^^^^^^^^

* See
  *sphx_glr_auto_examples_ensemble_plot_gradient_boosting_quantile.py*
  for an example of using the pinball loss to evaluate and tune the
  hyper-parameters of quantile regression models on data with non-
  symmetric noise and outliers.

3.3.4.11. D² score

The D² score computes the fraction of deviance explained. It is a
generalization of R², where the squared error is generalized and
replaced by a deviance of choice \text{dev}(y, \hat{y}) (e.g.,
Tweedie, pinball or mean absolute error). D² is a form of a *skill
score*. It is calculated as

   D^2(y, \hat{y}) = 1 - \frac{\text{dev}(y, \hat{y})}{\text{dev}(y,
   y_{\text{null}})} \,.

Where y_{\text{null}} is the optimal prediction of an intercept-only
model (e.g., the mean of *y_true* for the Tweedie case, the median for
absolute error and the alpha-quantile for pinball loss).

Like R², the best possible score is 1.0 and it can be negative
(because the model can be arbitrarily worse). A constant model that
always predicts y_{\text{null}}, disregarding the input features,
would get a D² score of 0.0.

3.3.4.11.1. D² Tweedie score

The "d2_tweedie_score()" function implements the special case of D²
where \text{dev}(y, \hat{y}) is the Tweedie deviance, see Mean
Poisson, Gamma, and Tweedie deviances. It is also known as D² Tweedie

and is related to McFadden's likelihood ratio index.

The argument "power" defines the Tweedie power as for
"mean_tweedie_deviance()". Note that for *power=0*,
"d2_tweedie_score()" equals "r2_score()" (for single targets).

A scorer object with a specific choice of "power" can be built by:

   >>> from sklearn.metrics import d2_tweedie_score, make_scorer
   >>> d2_tweedie_score_15 = make_scorer(d2_tweedie_score, power=1.5)

3.3.4.11.2. D² pinball score

The "d2_pinball_score()" function implements the special case of D²
with the pinball loss, see Pinball loss, i.e.:

   \text{dev}(y, \hat{y}) = \text{pinball}(y, \hat{y}).

The argument "alpha" defines the slope of the pinball loss as for
"mean_pinball_loss()" (Pinball loss). It determines the quantile level
"alpha" for which the pinball loss and also D² are optimal. Note that
for *alpha=0.5* (the default) "d2_pinball_score()" equals
"d2_absolute_error_score()".

A scorer object with a specific choice of "alpha" can be built by:

   >>> from sklearn.metrics import d2_pinball_score, make_scorer
   >>> d2_pinball_score_08 = make_scorer(d2_pinball_score, alpha=0.8)

3.3.4.11.3. D² absolute error score

The "d2_absolute_error_score()" function implements the special case
of the Mean absolute error:

   \text{dev}(y, \hat{y}) = \text{MAE}(y, \hat{y}).

Here are some usage examples of the "d2_absolute_error_score()"
function:

   >>> from sklearn.metrics import d2_absolute_error_score
   >>> d2_absolute_error_score(y_true, y_pred)
   0.764...
   >>> y_pred = [1, 2, 3]
   1.0
   >>> y_pred = [2, 2, 2]
   0.0

3.3.4.12. Visual evaluation of regression models

Among methods to assess the quality of regression models, scikit-learn
provides the "PredictionErrorDisplay" class. It allows to visually
inspect the prediction errors of a model in two different manners.

[image]

The plot on the left shows the actual values vs predicted values. For
a noise-free regression task aiming to predict the (conditional)
expectation of *y*, a perfect regression model would display data
points on the diagonal defined by predicted equal to actual values.
The further away from this optimal line, the larger the error of the
model. In a more realistic setting with irreducible noise, that is,
when not all the variations of *y* can be explained by features in

*X*, then the best model would lead to a cloud of points densely
arranged around the diagonal.

Note that the above only holds when the predicted values is the
expected value of *y* given *X*. This is typically the case for
regression models that minimize the mean squared error objective
function or more generally the mean Tweedie deviance for any value of
its "power" parameter.

When plotting the predictions of an estimator that predicts a quantile
of *y* given *X*, e.g. "QuantileRegressor" or any other model
minimizing the pinball loss, a fraction of the points are either
expected to lie above or below the diagonal depending on the estimated
quantile level.

All in all, while intuitive to read, this plot does not really inform
us on what to do to obtain a better model.

The right-hand side plot shows the residuals (i.e. the difference
between the actual and the predicted values) vs. the predicted values.

This plot makes it easier to visualize if the residuals follow and
homoscedastic or heteroschedastic distribution.

In particular, if the true distribution of *y|X* is Poisson or Gamma
distributed, it is expected that the variance of the residuals of the
optimal model would grow with the predicted value of *E[y|X]* (either
linearly for Poisson or quadratically for Gamma).

When fitting a linear least squares regression model (see

"LinearRegression" and "Ridge"), we can use this plot to check if some
of the model assumptions are met, in particular that the residuals
should be uncorrelated, their expected value should be null and that
their variance should be constant (homoschedasticity).

If this is not the case, and in particular if the residuals plot show
some banana-shaped structure, this is a hint that the model is likely
mis-specified and that non-linear feature engineering or switching to
a non-linear regression model might be useful.

Refer to the example below to see a model evaluation that makes use of
this display.

Example:
^^^^^^^^

* See *sphx_glr_auto_examples_compose_plot_transformed_target.py* for
  an example on how to use "PredictionErrorDisplay" to visualize the
  prediction quality improvement of a regression model obtained by
  transforming the target before learning.

3.3.5. Clustering metrics

utility functions. For more information see the Clustering performance
evaluation section for instance clustering, and Biclustering
evaluation for biclustering.

3.3.6. Dummy estimators

When doing supervised learning, a simple sanity check consists of
comparing one's estimator against simple rules of thumb.

"DummyClassifier" implements several such simple strategies for
classification:

* "stratified" generates random predictions by respecting the training
  set class distribution.

* "most_frequent" always predicts the most frequent label in the
  training set.

* "prior" always predicts the class that maximizes the class prior
  (like "most_frequent") and "predict_proba" returns the class prior.

* "uniform" generates predictions uniformly at random.

* "constant" always predicts a constant label that is provided by the
  user.
     A major motivation of this method is F1-scoring, when the
     positive class is in the minority.

Note that with all these strategies, the "predict" method completely
ignores the input data!

To illustrate "DummyClassifier", first let's create an imbalanced
dataset:

   >>> y[y != 1] = -1

Next, let's compare the accuracy of "SVC" and "most_frequent":

   >>> clf = SVC(kernel='linear', C=1).fit(X_train, y_train)
   0.63...

>>> from sklearn.dummy import DummyClassifier
   0.63...
   >>> clf.fit(X_train, y_train)
   DummyClassifier(random_state=0, strategy='most_frequent')
   0.57...

We see that "SVC" doesn't do much better than a dummy classifier. Now,
let's change the kernel:

   >>> clf = SVC(kernel='rbf', C=1).fit(X_train, y_train)
   0.94...

We see that the accuracy was boosted to almost 100%.  A cross
validation strategy is recommended for a better estimate of the
accuracy, if it is not too CPU costly. For more information see the
Cross-validation: evaluating estimator performance section. Moreover
if you want to optimize over the parameter space, it is highly
recommended to use an appropriate methodology; see the Tuning the
hyper-parameters of an estimator section for details.

More generally, when the accuracy of a classifier is too close to
random, it probably means that something went wrong: features are not
helpful, a hyperparameter is not correctly tuned, the classifier is
suffering from class imbalance, etc...

"DummyRegressor" also implements four simple rules of thumb for

* "mean" always predicts the mean of the training targets.

* "median" always predicts the median of the training targets.

* "quantile" always predicts a user provided quantile of the training
  targets.

* "constant" always predicts a constant value that is provided by the
  user.

In all these strategies, the "predict" method completely ignores the
input data.

This project is a community effort, and everyone is welcome to
contribute.

The project is hosted on https://github.com/scikit-learn/scikit-learn

is laid out in the governance document: Scikit-learn governance and
decision-making.

Scikit-learn is somewhat selective when it comes to adding new
algorithms, and the best way to contribute and to help the project is
to start working on known issues. See Issues for New Contributors to
get started.

**Our community, our values**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We are a community based on openness and friendly, didactic,
discussions.

We aspire to treat everybody equally, and value their contributions.
We are particularly seeking people from underrepresented backgrounds
in Open Source Software and scikit-learn in particular to participate
and contribute their expertise and experience.

Decisions are made based on technical merit and consensus.

Code is not the only way to help the project. Reviewing pull requests,
answering questions to help others on mailing lists or issues,
organizing and teaching tutorials, working on the website, improving
the documentation, are all priceless contributions.

We abide by the principles of openness, respect, and consideration of
others of the Python Software Foundation:
https://www.python.org/psf/codeofconduct/

In case you experience issues using this package, do not hesitate to

submit a ticket to the GitHub issue tracker. You are also welcome to
post feature requests or pull requests.

Ways to contribute

There are many ways to contribute to scikit-learn, with the most
common ones being contribution of code or documentation to the
project. Improving the documentation is no less important than
improving the library itself.  If you find a typo in the
documentation, or have made improvements, do not hesitate to send an
email to the mailing list or preferably submit a GitHub pull request.
Full documentation can be found under the doc/ directory.

But there are many other ways to help. In particular helping to
improve, triage, and investigate issues and reviewing other
developers' pull requests are very valuable contributions that
decrease the burden on the project maintainers.

Another way to contribute is to report issues you're facing, and give
a "thumbs up" on issues that others reported and that are relevant to
you.  It also helps us if you spread the word: reference the project
from your blog and articles, link to it from your website, or simply
star to say "I use it":

In case a contribution/issue involves changes to the API principles or
changes to dependencies or supported versions, it must be backed by a
Enhancement proposals (SLEPs), where a SLEP must be submitted as a

pull-request to enhancement proposals using the SLEP template and
follows the decision-making process outlined in Scikit-learn
governance and decision-making.

Contributing to related projects

Scikit-learn thrives in an ecosystem of several related projects,
which also may have relevant issues to work on, including smaller
projects such as:

* scikit-learn-contrib

* joblib

* sphinx-gallery

* numpydoc

* liac-arff

and larger projects:

* numpy

* scipy

* matplotlib

* and so on.

Look for issues marked "help wanted" or similar. Helping these
projects may help Scikit-learn too. See also Related Projects.

Submitting a bug report or a feature request

We use GitHub issues to track all bugs and feature requests; feel free
to open an issue if you have found a bug or wish to see a feature
implemented.

submit a ticket to the Bug Tracker. You are also welcome to post
feature requests or pull requests.

It is recommended to check that your issue complies with the following
rules before submitting:

* Verify that your issue is not being currently addressed by other
  issues or pull requests.

* If you are submitting an algorithm or feature request, please verify
  that the algorithm fulfills our new algorithm requirements.

issues or pull requests.

* If you are submitting a bug report, we strongly encourage you to
  follow the guidelines in How to make a good bug report.

How to make a good bug report

When you submit an issue to Github, please do your best to follow
these guidelines! This will make it a lot easier to provide you with
good feedback:

* The ideal bug report contains a short reproducible code snippet,
  this way anyone can try to reproduce the bug easily (see this for
  more details). If your snippet is longer than around 50 lines,
  please link to a gist or a github repo.

* If not feasible to include a reproducible snippet, please be
  specific about what **estimators and/or functions are involved and
  the shape of the data**.

* If an exception is raised, please **provide the full traceback**.

* Please include your **operating system type and version number**, as
  well as your **Python, scikit-learn, numpy, and scipy versions**.
  This information can be found by running the following code snippet:

     >>> import sklearn
     >>> sklearn.show_versions()  

* Please ensure all **code snippets and error messages are formatted
  in appropriate code blocks**.  See Creating and highlighting code
  blocks for more details.

If you want to help curate issues, read the following.

Contributing code

Note:

in appropriate code blocks**.  See Creating and highlighting code

Note:

  To avoid duplicating work, it is highly advised that you search
  through the issue tracker and the PR list. If in doubt about
  duplicated work, or if you want to work on a non-trivial feature,
  it's recommended to first open an issue in the issue tracker to get
  some feedbacks from core developers.One easy way to find an issue to
  work on is by applying the "help wanted" label in your search. This
  lists all the issues that have been unclaimed so far. In order to
  claim an issue for yourself, please comment exactly "/take" on it
  for the CI to automatically assign the issue to you.

Video resources

These videos are step-by-step introductions on how to contribute to
scikit-learn, and are a great companion to the following text
guidelines. Please make sure to still check our guidelines below,
since they describe our latest up-to-date workflow.

* Crash Course in Contributing to Scikit-Learn & Open Source Projects:
  Video, Transcript

* Example of Submitting a Pull Request to scikit-learn: Video,
  Transcript

* Sprint-specific instructions and practical tips: Video, Transcript

* 3 Components of Reviewing a Pull Request: Video, Transcript

Note:

  In January 2021, the default branch name changed from "master" to
  "main" for the scikit-learn GitHub repository to use more inclusive

Note:

  terms. These videos were created prior to the renaming of the
  branch. For contributors who are viewing these videos to set up
  their working environment and submitting a PR, "master" should be
  replaced to "main".

How to contribute

The preferred way to contribute to scikit-learn is to fork the main
repository on GitHub, then submit a "pull request" (PR).

In the first few steps, we explain how to locally install scikit-
learn, and how to set up your git repository:

1. Create an account on GitHub if you do not already have one.

2. Fork the project repository: click on the 'Fork' button near the
   top of the page. This creates a copy of the code under your account
   on the GitHub user account. For more details on how to fork a
   repository see this guide.

3. Clone your fork of the scikit-learn repo from your GitHub account
   to your local disk:

3. Follow steps 2-7 in Building from source to build scikit-learn in
   development mode and return to this document.

4. Install the development dependencies:

5. Add the "upstream" remote. This saves a reference to the main
   scikit-learn repository, which you can use to keep your repository
   synchronized with the latest changes:

6. Check that the *upstream* and *origin* remote aliases are
   configured correctly by running *git remote -v* which should
   display:

synchronized with the latest changes:

   display:

      origin  git@github.com:YourLogin/scikit-learn.git (fetch)
      origin  git@github.com:YourLogin/scikit-learn.git (push)
      upstream        git@github.com:scikit-learn/scikit-learn.git (fetch)
      upstream        git@github.com:scikit-learn/scikit-learn.git (push)

You should now have a working installation of scikit-learn, and your
git repository properly configured. The next steps now describe the
process of modifying code and submitting a PR:

7. Synchronize your "main" branch with the "upstream/main" branch,
   more details on GitHub Docs:

8. Create a feature branch to hold your development changes:

   and start making changes. Always use a feature branch. It's good
   practice to never work on the "main" branch!

9. (**Optional**) Install pre-commit to run code style checks before
   each commit:

   pre-commit checks can be disabled for a particular commit with *git
   commit -n*.

10. Develop the feature on your feature branch on your computer, using
    Git to do the version control. When you're done editing, add
    changed files using "git add" and then "git commit":

    to record your changes in Git, then push the changes to your
    GitHub account with:

11. Follow these instructions to create a pull request from your fork.
    This will send an email to the committers. You may want to

GitHub account with:

    consider sending an email to the mailing list for more visibility.

Note:

  If you are modifying a Cython module, you have to re-compile after
  modifications and before testing them:Use the "--no-build-isolation"
  flag to avoid compiling the whole project each time, only the files
  you have modified.

It is often helpful to keep your local feature branch synchronized
with the latest changes of the main scikit-learn repository:

Subsequently, you might need to solve the conflicts. You can refer to
the Git documentation related to resolving merge conflict using the
command line.

Learning git:
^^^^^^^^^^^^^

The Git documentation and http://try.github.io are excellent resources
to get started with git, and understanding all of the commands shown
here.

Pull request checklist

Before a PR can be merged, it needs to be approved by two core
developers. Please prefix the title of your pull request with "[MRG]"
if the contribution is complete and should be subjected to a detailed
review. An incomplete contribution -- where you expect to do more work
before receiving a full review -- should be prefixed "[WIP]" (to
indicate a work in progress) and changed to "[MRG]" when it matures.
WIPs may be useful to: indicate you are working on something to avoid
duplicated work, request broad review of functionality or API, or seek

collaborators. WIPs often benefit from the inclusion of a task list in
the PR description.

In order to ease the reviewing process, we recommend that your
contribution complies with the following rules before marking a PR as
"[MRG]". The **bolded** ones are especially important:

1. **Give your pull request a helpful title** that summarizes what
   your contribution does. This title will often become the commit
   message once merged so it should summarize your contribution for
   posterity. In some cases "Fix <ISSUE TITLE>" is enough. "Fix
   #<ISSUE NUMBER>" is never a good title.

2. **Make sure your code passes the tests**. The whole test suite can
   be run with *pytest*, but it is usually not recommended since it
   takes a long time. It is often enough to only run the test related
   to your changes: for example, if you changed something in
   *sklearn/linear_model/_logistic.py*, running the following commands
   will usually be enough:

   * *pytest sklearn/linear_model/_logistic.py* to make sure the
     doctest examples are correct

   * *pytest sklearn/linear_model/tests/test_logistic.py* to run the
     tests specific to the file

   * *pytest sklearn/linear_model* to test the whole "linear_model"
     module

   * *pytest doc/modules/linear_model.rst* to make sure the user guide
     examples are correct.

* *pytest sklearn/linear_model* to test the whole "linear_model"
     module

   * *pytest sklearn/tests/test_common.py -k LogisticRegression* to
     run all our estimator checks (specifically for
     *LogisticRegression*, if that's the estimator you changed).

   There may be other failing tests, but they will be caught by the CI
   so you don't need to run the whole test suite locally. For
   guidelines on how to use "pytest" efficiently, see the Useful
   pytest aliases and flags.

3. **Make sure your code is properly commented and documented**, and
   **make sure the documentation renders properly**. To build the
   documentation, please refer to our Documentation guidelines. The CI
   will also build the docs: please refer to Generated documentation
   on GitHub Actions.

4. **Tests are necessary for enhancements to be accepted**. Bug-fixes
   or new features should be provided with non-regression tests. These
   tests verify the correct behavior of the fix or feature. In this
   manner, further modifications on the code base are granted to be
   consistent with the desired behavior. In the case of bug fixes, at
   the time of the PR, the non-regression tests should fail for the
   code base in the "main" branch and pass for the PR code.

5. Run *black* to auto-format your code.

   See black's editor integration documentation to configure your
   editor to run *black*.

code base in the "main" branch and pass for the PR code.

6. Run *flake8* to make sure you followed the project coding
   conventions.

7. Follow the Coding guidelines.

8. When applicable, use the validation tools and scripts in the
   "sklearn.utils" submodule.  A list of utility routines available
   for developers can be found in the Utilities for Developers page.

9. Often pull requests resolve one or more other issues (or pull
   requests). If merging your pull request means that some other
   issues/PRs should be closed, you should use keywords to create link
   to them (e.g., "Fixes #1234"; multiple issues/PRs are allowed as
   long as each one is preceded by a keyword). Upon merging, those
   issues/PRs will automatically be closed by GitHub. If your pull
   request is simply related to some other issues/PRs, create a link
   to them without using the keywords (e.g., "See also #1234").

10. PRs should often substantiate the change, through benchmarks of
    performance and efficiency (see Monitoring performance) or through
    examples of usage. Examples also illustrate the features and
    intricacies of the library to users. Have a look at other examples
    in the examples/ directory for reference. Examples should
    demonstrate why the new functionality is useful in practice and,
    if possible, compare it to other methods available in scikit-
    learn.

demonstrate why the new functionality is useful in practice and,
    learn.

11. New features have some maintenance overhead. We expect PR authors
    to take part in the maintenance for the code they submit, at least
    initially. New features need to be illustrated with narrative
    documentation in the user guide, with small code snippets. If
    relevant, please also add references in the literature, with PDF
    links when possible.

12. The user guide should also include expected time and space
    complexity of the algorithm and scalability, e.g. "this algorithm
    can scale to a large number of samples > 100000, but does not
    scale in dimensionality: n_features is expected to be lower than
    100".

You can also check our Code Review Guidelines to get an idea of what
reviewers will expect.

You can check for common programming errors with the following tools:

* Code with a good unittest coverage (at least 80%, better 100%),
  check with:

  see also Testing and improving test coverage

  Run static analysis with *mypy*:

  must not produce new errors in your pull request. Using *# type:
  ignore* annotation can be a workaround for a few cases that are not
  supported by mypy, in particular,

  * when importing C or Cython modules

  * on properties with decorators

Bonus points for contributions that include a performance analysis
with a benchmark script and profiling output (see Monitoring
performance).

* on properties with decorators

Also check out the How to optimize for speed guide for more details on
profiling and Cython optimizations.

Note:

  The current state of the scikit-learn code base is not compliant
  with all of those guidelines, but we expect that enforcing those
  constraints on all new contributions will get the overall code base
  quality in the right direction.

Note:

  For two very well documented and more detailed guides on development
  workflow, please pay a visit to the Scipy Development Workflow - and
  the Astropy Workflow for Developers sections.

Continuous Integration (CI)

* Azure pipelines are used for testing scikit-learn on Linux, Mac and
  Windows, with different dependencies and settings.

* CircleCI is used to build the docs for viewing, for linting with
  flake8, and for testing with ARM64 / aarch64 on Linux

Please note that if one of the following markers appear in the latest
commit message, the following actions are taken.

   +------------------------+--------------------------------------------------------------------------------------------------------------+
   | Commit Message Marker  | Action Taken by CI                                                                                           |

| Commit Message Marker  | Action Taken by CI                                                                                           |
   | [ci skip]              | CI is skipped completely                                                                                     |
   | [cd build]             | CD is run (wheels and source distribution are built)                                                         |
   | [cd build gh]          | CD is run only for GitHub Actions                                                                            |
   | [cd build cirrus]      | CD is run only for Cirrus CI                                                                                 |

+------------------------+--------------------------------------------------------------------------------------------------------------+
   | [lint skip]            | Azure pipeline skips linting                                                                                 |
   | [scipy-dev]            | Build & test with our dependencies (numpy, scipy, etc ...) development builds                                |
   | [nogil]                | Build & test with the nogil experimental branches of CPython, Cython, NumPy, SciPy...                        |
   | [pypy]                 | Build & test with PyPy                                                                                       |
   | [azure parallel]       | Run Azure CI jobs in parallel                                                                                |

| [azure parallel]       | Run Azure CI jobs in parallel                                                                                |
   | [float32]              | Run float32 tests by setting *SKLEARN_RUN_FLOAT32_TESTS=1*. See Environment variables for more details       |
   | [doc skip]             | Docs are not built                                                                                           |
   | [doc quick]            | Docs built, but excludes example gallery plots                                                               |
   | [doc build]            | Docs built including example gallery plots (very long)                                                       |

Note that, by default, the documentation is built but only the

examples that are directly modified by the pull request are executed.

Stalled pull requests

As contributing a feature can be a lengthy process, some pull requests
appear inactive but unfinished. In such a case, taking them over is a
great service for the project.

A good etiquette to take over is:

* **Determine if a PR is stalled**

  * A pull request may have the label "stalled" or "help wanted" if we
    have already identified it as a candidate for other contributors.

  * To decide whether an inactive PR is stalled, ask the contributor
    if she/he plans to continue working on the PR in the near future.
    Failure to respond within 2 weeks with an activity that moves the
    PR forward suggests that the PR is stalled and will result in
    tagging that PR with "help wanted".

    Note that if a PR has received earlier comments on the
    contribution that have had no reply in a month, it is safe to
    assume that the PR is stalled and to shorten the wait time to one
    day.

    After a sprint, follow-up for un-merged PRs opened during sprint
    will be communicated to participants at the sprint, and those PRs
    will be tagged "sprint". PRs tagged with "sprint" can be
    reassigned or declared stalled by sprint leaders.

* **Taking over a stalled PR**: To take over a PR, it is important to
  comment on the stalled PR that you are taking over and to link from

reassigned or declared stalled by sprint leaders.

  the new PR to the old one. The new PR should be created by pulling
  from the old one.

Stalled and Unclaimed Issues

Generally speaking, issues which are up for grabs will have a "help
wanted". tag. However, not all issues which need contributors will
have this tag, as the "help wanted" tag is not always up-to-date with
the state of the issue. Contributors can find issues which are still
up for grabs using the following guidelines:

* First, to **determine if an issue is claimed**:

  * Check for linked pull requests

  * Check the conversation to see if anyone has said that they're
    working on creating a pull request

* If a contributor comments on an issue to say they are working on it,
  a pull request is expected within 2 weeks (new contributor) or 4
  weeks (contributor or core dev), unless an larger time frame is
  explicitly given. Beyond that time, another contributor can take the
  issue and make a pull request for it. We encourage contributors to
  comment directly on the stalled or unclaimed issue to let community
  members know that they will be working on it.

* If the issue is linked to a stalled pull request, we recommend that
  contributors follow the procedure described in the Stalled pull
  requests section rather than working directly on the issue.

Issues for New Contributors

New contributors should look for the following tags when looking for
issues.  We strongly recommend that new contributors tackle "easy"
issues first: this helps the contributor become familiar with the
contribution workflow, and for the core devs to become acquainted with
the contributor; besides which, we frequently underestimate how easy
an issue is to solve!

good first issue tag

A great way to start contributing to scikit-learn is to pick an item
from the list of good first issues in the issue tracker. Resolving
these issues allow you to start contributing to the project without
much prior knowledge. If you have already contributed to scikit-learn,
you should look at Easy issues instead.

Easy tag
^^^^^^^^

If you have already contributed to scikit-learn, another great way to
contribute to scikit-learn is to pick an item from the list of Easy
issues in the issue tracker. Your assistance in this area will be
greatly appreciated by the more experienced developers as it helps
free up their time to concentrate on other issues.

help wanted tag
^^^^^^^^^^^^^^^

We often use the help wanted tag to mark issues regardless of
difficulty. Additionally, we use the help wanted tag to mark Pull

Requests which have been abandoned by their original contributor and
are available for someone to pick up where the original contributor
left off. The list of issues with the help wanted tag can be found
here.

Note that not all issues which need contributors will have this tag.

We are glad to accept any sort of documentation:

* **function/method/class docstrings** (also known as "API
  documentation") - these describe what the object does and details
  any parameters, attributes and methods. Docstrings live alongside
  the code in sklearn/.

* **user guide** - these provide more detailed information about the
  algorithms implemented in scikit-learn and generally live in the
  root doc/ directory and doc/modules/.

* **tutorials** - these introduce various statistical learning and
  machine learning concepts and are located in doc/tutorial.

* **examples** - these provide full code examples that may demonstrate
  the use of scikit-learn modules, compare different algorithms or
  discuss their interpretation etc. Examples live in examples/

* **other reStructuredText documents** (like this one) - provide
  various other useful information (e.g., our guide to contributing)
  and live in doc/.

You can edit the documentation using any text editor, and then
generate the HTML output by following Building the documentation. The

and live in doc/.

resulting HTML files will be placed in "_build/html/stable" and are
viewable in a web browser, for instance by opening the local
"_build/html/stable/index.html" file.

Building the documentation

First, make sure you have properly installed the development version.

Building the documentation requires installing some additional
packages:

To build the documentation, you need to be in the "doc" folder:

In the vast majority of cases, you only need to generate the full web
site, without the example gallery:

The documentation will be generated in the "_build/html/stable"
directory and are viewable in a web browser, for instance by opening
the local "_build/html/stable/index.html" file. To also generate the
example gallery you can use:

This will run all the examples, which takes a while. If you only want
to generate a few examples, you can use:

This is particularly useful if you are modifying a few examples.

Set the environment variable *NO_MATHJAX=1* if you intend to view the
documentation in an offline setting.

To build the PDF manual, run:

Warning:

  **Sphinx version**While we do our best to have the documentation
  build under as many versions of Sphinx as possible, the different
  versions tend to behave slightly differently. To get the best
  results, you should use the same version as the one we used on

build under as many versions of Sphinx as possible, the different
  CircleCI. Look at this github search to know the exact version.

Guidelines for writing documentation

It is important to keep a good compromise between mathematical and
algorithmic details, and give intuition to the reader on what the
algorithm does.

Basically, to elaborate on the above, it is best to always start with
a small paragraph with a hand-waving explanation of what the method
does to the data. Then, it is very helpful to point out why the
feature is useful and when it should be used - the latter also
including "big O" (O\left(g\left(n\right)\right)) complexities of the
algorithm, as opposed to just *rules of thumb*, as the latter can be
very machine-dependent. If those complexities are not available, then
rules of thumb may be provided instead.

Secondly, a generated figure from an example (as mentioned in the
previous paragraph) should then be included to further provide some
intuition.

Next, one or two small code examples to show its use can be added.

Next, any math and equations, followed by references, can be added to
further the documentation. Not starting the documentation with the
maths makes it more friendly towards users that are just interested in
what the feature will do, as opposed to how it works "under the hood".

Finally, follow the formatting rules below to make it consistently
good:

* Add "See Also" in docstrings for related classes/functions.

* "See Also" in docstrings should be one line per reference, with a
  colon and an explanation, for example:

     See Also
     --------
     SelectKBest : Select features based on the k highest scores.
     SelectFpr : Select features based on a false positive rate test.

* When documenting the parameters and attributes, here is a list of
  some well-formatted examples:

     n_clusters : int, default=3
         The number of clusters detected by the algorithm.

     some_param : {'hello', 'goodbye'}, bool or int, default=True
         The parameter description goes here, which can be either a string
         literal (either `hello` or `goodbye`), a bool, or an int. The default
         value is True.

     array_parameter : {array-like, sparse matrix} of shape (n_samples, n_features) or (n_samples,)
         This parameter accepts data in either of the mentioned forms, with one
         of the mentioned shapes. The default value is
         `np.ones(shape=(n_samples,))`.

     list_param : list of int

     typed_ndarray : ndarray of shape (n_samples,), dtype=np.int32

     sample_weight : array-like of shape (n_samples,), default=None

     multioutput_array : ndarray of shape (n_samples, n_classes) or list of such arrays

sample_weight : array-like of shape (n_samples,), default=None

  In general have the following in mind:

     1. Use Python basic types. ("bool" instead of "boolean")

     2. Use parenthesis for defining shapes: "array-like of shape
        (n_samples,)" or "array-like of shape (n_samples, n_features)"

     3. For strings with multiple options, use brackets: "input:
        {'log', 'squared', 'multinomial'}"

     4. 1D or 2D data can be a subset of "{array-like, ndarray, sparse
        matrix, dataframe}". Note that "array-like" can also be a
        "list", while "ndarray" is explicitly only a "numpy.ndarray".

     5. Specify "dataframe" when "frame-like" features are being used,
        such as the column names.

     6. When specifying the data type of a list, use "of" as a
        delimiter: "list of int". When the parameter supports arrays
        giving details about the shape and/or data type and a list of
        such arrays, you can use one of "array-like of shape
        (n_samples,) or list of such arrays".

     7. When specifying the dtype of an ndarray, use e.g.
        "dtype=np.int32" after defining the shape: "ndarray of shape
        (n_samples,), dtype=np.int32". You can specify multiple dtype
        as a set: "array-like of shape (n_samples,),
        dtype={np.float64, np.float32}". If one wants to mention
        arbitrary precision, use *integral* and *floating* rather than

as a set: "array-like of shape (n_samples,),
        the Python dtype *int* and *float*. When both *int* and
        *floating* are supported, there is no need to specify the
        dtype.

     8. When the default is "None", "None" only needs to be specified
        at the end with "default=None". Be sure to include in the
        docstring, what it means for the parameter or attribute to be
        "None".

* For unwritten formatting rules, try to follow existing good works:

     * When bibliographic references are available with arxiv or
       Digital Object Identifier identification numbers, use the
       sphinx directives *:arxiv:* or *:doi:*. For example, see
       references in Spectral Clustering Graphs.

     * For "References" in docstrings, see the Silhouette Coefficient
       ("sklearn.metrics.silhouette_score()").

* When editing reStructuredText (".rst") files, try to keep line
  length under 80 characters when possible (exceptions include links
  and tables).

* In scikit-learn reStructuredText files both single and double
  backticks surrounding text will render as inline literal (often used
  for code, e.g., *list*). This is due to specific configurations we
  have set. Single backticks should be used nowadays.

* Before submitting your pull request check if your modifications have
  introduced new sphinx warnings and try to fix them.

have set. Single backticks should be used nowadays.

Cross-referencing

It is often useful to cross-reference to other pages in the scikit-
learn documentation. This should be done with reStructuredText cross-
referencing syntax:

* Section - to link to an arbitrary section in the documentation, use
  reference labels (see Sphinx docs). For example:

     .. _my-section:

     My section
     ----------

     This is the text of the section.

     To refer to itself use :ref:`my-section`.

  You should not modify existing sphinx reference labels as this would
  break existing cross references and external links pointing to
  specific sections in the scikit-learn documentation.

* Glossary - linking to a term in the Glossary of Common Terms and API
  Elements:

     :term:`cross_validation`

* Function - to link to the documentation of a function, use the full
  import path to the function:

     :func:`~sklearn.model_selection.cross_val_score`

  However, if there is a 'currentmodule' directive above you in the
  document, you will only need to use the path to the function
  succeeding the current module specified. For example:

     .. currentmodule:: sklearn.model_selection

     :func:`cross_val_score`

* Class - to link to documentation of a class, use the full import
  path to the class, unless there is a 'currentmodule' directive in
  the document above (see above):

:func:`cross_val_score`

     :class:`~sklearn.preprocessing.StandardScaler`

Generated documentation on GitHub Actions

When you change the documentation in a pull request, GitHub Actions
automatically builds it. To view the documentation generated by GitHub
Actions, simply go to the bottom of your PR page, look for the item
"Check the rendered docs here!" and click on 'details' next to it:

[image]

Testing and improving test coverage

High-quality unit testing is a corner-stone of the scikit-learn
development process. For this purpose, we use the pytest package. The
tests are functions appropriately named, located in *tests*
subdirectories, that check the validity of the algorithms and the
different options of the code.

Running *pytest* in a folder will run all the tests of the
corresponding subpackages. For a more detailed *pytest* workflow,
please refer to the Pull request checklist.

We expect code coverage of new features to be at least around 90%.

Writing matplotlib related tests

Test fixtures ensure that a set of tests will be executing with the
appropriate initialization and cleanup. The scikit-learn test suite
implements a fixture which can be used with "matplotlib".

"pyplot"

"pyplot"
   The "pyplot" fixture should be used when a test function is dealing
   with "matplotlib". "matplotlib" is a soft dependency and is not
   required. This fixture is in charge of skipping the tests if
   "matplotlib" is not installed. In addition, figures created during
   the tests will be automatically closed once the test function has
   been executed.

To use this fixture in a test function, one needs to pass it as an
argument:

   def test_requiring_mpl_fixture(pyplot):
       # you can now safely use matplotlib

Workflow to improve test coverage

To test code coverage, you need to install the coverage package in
addition to pytest.

1. Run 'make test-coverage'. The output lists for each file the line
      numbers that are not tested.

2. Find a low hanging fruit, looking at which lines are not tested,
      write or adapt a test specifically for these lines.

3. Loop.

Monitoring performance

*This section is heavily inspired from the* pandas documentation.

When proposing changes to the existing code base, it's important to
make sure that they don't introduce performance regressions. Scikit-
learn uses asv benchmarks to monitor the performance of a selection of
common estimators and functions. You can view these benchmarks on the

scikit-learn benchmark page. The corresponding benchmark suite can be
found in the *scikit-learn/asv_benchmarks* directory.

To use all features of asv, you will need either *conda* or
*virtualenv*. For more details please check the asv installation
webpage.

First of all you need to install the development version of asv:

and change your directory to *asv_benchmarks/*:

The benchmark suite is configured to run against your local clone of
scikit-learn. Make sure it is up to date:

In the benchmark suite, the benchmarks are organized following the
same structure as scikit-learn. For example, you can compare the
performance of a specific estimator between "upstream/main" and the
branch you are working on:

The command uses conda by default for creating the benchmark
environments. If you want to use virtualenv instead, use the *-E*
flag:

You can also specify a whole module to benchmark:

You can replace *HEAD* by any local branch. By default it will only
report the benchmarks that have change by at least 10%. You can
control this ratio with the *-f* flag.

To run the full benchmark suite, simply remove the *-b* flag :

However this can take up to two hours. The *-b* flag also accepts a
regular expression for a more complex subset of benchmarks to run.

To run the benchmarks without comparing to another branch, use the
*run* command:

You can also run the benchmark suite using the version of scikit-learn
already installed in your current Python environment:

It's particularly useful when you installed scikit-learn in editable
mode to avoid creating a new environment each time you run the
benchmarks. By default the results are not saved when using an
existing installation. To save the results you must specify a commit
hash:

Benchmarks are saved and organized by machine, environment and commit.
To see the list of all saved benchmarks:

and to see the report of a specific run:

When running benchmarks for a pull request you're working on please
report the results on github.

The benchmark suite supports additional configurable options which can
be set in the *benchmarks/config.json* configuration file. For
example, the benchmarks can run for a provided list of values for the
*n_jobs* parameter.

More information on how to write a benchmark and how to use asv can be
found in the asv documentation.

Issue Tracker Tags

All issues and pull requests on the GitHub issue tracker should have
(at least) one of the following tags:

Bug / Crash:
   Something is happening that clearly shouldn't happen. Wrong results
   as well as unexpected errors from estimators go here.

Cleanup / Enhancement:
   Improving performance, usability, consistency.

Documentation:

as well as unexpected errors from estimators go here.

   Missing, incorrect or sub-standard documentations and examples.

New Feature:
   Feature requests and pull requests implementing a new feature.

There are four other tags to help new contributors:

good first issue:
   This issue is ideal for a first contribution to scikit-learn. Ask
   for help if the formulation is unclear. If you have already
   contributed to scikit-learn, look at Easy issues instead.

Easy:
   This issue can be tackled without much prior experience.

Moderate:
   Might need some knowledge of machine learning or the package, but
   is still approachable for someone new to the project.

help wanted:
   This tag marks an issue which currently lacks a contributor or a PR
   that needs another contributor to take over the work. These issues
   can range in difficulty, and may not be approachable for new
   contributors. Note that not all issues which need contributors will
   have this tag.

Maintaining backwards compatibility

Deprecation

If any publicly accessible method, function, attribute or parameter is
renamed, we still support the old one for two releases and issue a
deprecation warning when it is called/passed/accessed. E.g., if the
function "zero_one" is renamed to "zero_one_loss", we add the
decorator "deprecated" (from "sklearn.utils") to "zero_one" and call
"zero_one_loss" from that function:

   from ..utils import deprecated

   def zero_one_loss(y_true, y_pred, normalize=True):
       # actual implementation
       pass

   @deprecated("Function 'zero_one' was renamed to 'zero_one_loss' "
               "in version 0.13 and will be removed in release 0.15. "
               "Default behavior is changed from 'normalize=False' to "
               "'normalize=True'")
   def zero_one(y_true, y_pred, normalize=False):
       return zero_one_loss(y_true, y_pred, normalize)

If an attribute is to be deprecated, use the decorator "deprecated" on
a property. Please note that the "property" decorator should be placed
before the "deprecated" decorator for the docstrings to be rendered
properly. E.g., renaming an attribute "labels_" to "classes_" can be
done as:

   @deprecated("Attribute `labels_` was deprecated in version 0.13 and "
               "will be removed in 0.15. Use `classes_` instead")
   @property
   def labels_(self):
       return self.classes_

If a parameter has to be deprecated, a "FutureWarning" warning must be
raised too. In the following example, k is deprecated and renamed to
n_clusters:

   import warnings

   def example_function(n_clusters=8, k='deprecated'):
       if k != 'deprecated':
           warnings.warn("'k' was renamed to n_clusters in version 0.13 and "
                         "will be removed in 0.15.",

if k != 'deprecated':
                         FutureWarning)
           n_clusters = k

When the change is in a class, we validate and raise warning in "fit":

   class ExampleEstimator(BaseEstimator):
       def __init__(self, n_clusters=8, k='deprecated'):
           self.n_clusters = n_clusters
           self.k = k

       def fit(self, X, y):
           if self.k != 'deprecated':
               warnings.warn("'k' was renamed to n_clusters in version 0.13 and "
                             "will be removed in 0.15.",
                             FutureWarning)
               self._n_clusters = self.k
           else:
               self._n_clusters = self.n_clusters

As in these examples, the warning message should always give both the
version in which the deprecation happened and the version in which the
old behavior will be removed. If the deprecation happened in version
0.x-dev, the message should say deprecation occurred in version 0.x
and the removal will be in 0.(x+2), so that users will have enough
time to adapt their code to the new behaviour. For example, if the
deprecation happened in version 0.18-dev, the message should say it
happened in version 0.18 and the old behavior will be removed in
version 0.20.

In addition, a deprecation note should be added in the docstring,

recalling the same information as the deprecation warning as explained
above. Use the ".. deprecated::" directive:

   .. deprecated:: 0.13
      ``k`` was renamed to ``n_clusters`` in version 0.13 and will be removed
      in 0.15.

What's more, a deprecation requires a test which ensures that the
warning is raised in relevant cases but not in other cases. The
warning should be caught in all other tests (using e.g.,
"@pytest.mark.filterwarnings"), and there should be no warning in the
examples.

Change the default value of a parameter

If the default value of a parameter needs to be changed, please
replace the default value with a specific value (e.g., "warn") and
raise "FutureWarning" when users are using the default value. The
following example assumes that the current version is 0.20 and that we
change the default value of "n_clusters" from 5 (old default for 0.20)
to 10 (new default for 0.22):

   def example_function(n_clusters='warn'):
       if n_clusters == 'warn':
           warnings.warn("The default value of n_clusters will change from "
                         "5 to 10 in 0.22.", FutureWarning)
           n_clusters = 5

   class ExampleEstimator:
       def __init__(self, n_clusters='warn'):

n_clusters = 5

           if self.n_clusters == 'warn':
             warnings.warn("The default value of n_clusters will change from "
                           "5 to 10 in 0.22.", FutureWarning)
             self._n_clusters = 5

Similar to deprecations, the warning message should always give both
the version in which the change happened and the version in which the
old behavior will be removed.

The parameter description in the docstring needs to be updated
accordingly by adding a *versionchanged* directive with the old and
new default value, pointing to the version when the change will be
effective:

   .. versionchanged:: 0.22
      The default value for `n_clusters` will change from 5 to 10 in version 0.22.

Finally, we need a test which ensures that the warning is raised in
relevant cases but not in other cases. The warning should be caught in
all other tests (using e.g., "@pytest.mark.filterwarnings"), and there
should be no warning in the examples.

Code Review Guidelines

Reviewing code contributed to the project as PRs is a crucial
component of scikit-learn development. We encourage anyone to start
reviewing code of other developers. The code review process is often
highly educational for everybody involved. This is particularly

appropriate if it is a feature you would like to use, and so can
respond critically about whether the PR meets your needs. While each
pull request needs to be signed off by two core developers, you can
speed up this process by providing your feedback.

Note:

  The difference between an objective improvement and a subjective nit
  isn't always clear. Reviewers should recall that code review is
  primarily about reducing risk in the project. When reviewing code,
  one should aim at preventing situations which may require a bug fix,
  a deprecation, or a retraction. Regarding docs: typos, grammar
  issues and disambiguations are better addressed immediately.

Here are a few important aspects that need to be covered in any code
review, from high-level questions to a more detailed check-list.

* Do we want this in the library? Is it likely to be used? Do you, as
  a scikit-learn user, like the change and intend to use it? Is it in
  the scope of scikit-learn? Will the cost of maintaining a new
  feature be worth its benefits?

* Is the code consistent with the API of scikit-learn? Are public
  functions/classes/parameters well named and intuitively designed?

* Are all public functions/classes and their parameters, return types,
  and stored attributes named according to scikit-learn conventions

  and documented clearly?

* Is any new functionality described in the user-guide and illustrated
  with examples?

* Is every public function/class tested? Are a reasonable set of
  parameters, their values, value types, and combinations tested? Do
  the tests validate that the code is correct, i.e. doing what the
  documentation says it does? If the change is a bug-fix, is a non-
  regression test included? Look at this to get started with testing
  in Python.

* Do the tests pass in the continuous integration build? If
  appropriate, help the contributor understand why tests failed.

* Do the tests cover every line of code (see the coverage report in
  the build log)? If not, are the lines missing coverage good
  exceptions?

* Is the code easy to read and low on redundancy? Should variable
  names be improved for clarity or consistency? Should comments be
  added? Should comments be removed as unhelpful or extraneous?

* Could the code easily be rewritten to run much more efficiently for
  relevant settings?

* Is the code backwards compatible with previous versions? (or is a
  deprecation cycle necessary?)

* Will the new code add any dependencies on other libraries? (this is
  unlikely to be accepted)

* Does the documentation render properly (see the Documentation
  section for more details), and are the plots instructive?

unlikely to be accepted)

Standard replies for reviewing includes some frequent comments that
reviewers may make.

Communication Guidelines

Reviewing open pull requests (PRs) helps move the project forward. It
is a great way to get familiar with the codebase and should motivate
the contributor to keep involved in the project. [1]

* Every PR, good or bad, is an act of generosity. Opening with a
  positive comment will help the author feel rewarded, and your
  subsequent remarks may be heard more clearly. You may feel good
  also.

* Begin if possible with the large issues, so the author knows they've
  been understood. Resist the temptation to immediately go line by
  line, or to open with small pervasive issues.

* Do not let perfect be the enemy of the good. If you find yourself
  making many small suggestions that don't fall into the Code Review
  Guidelines, consider the following approaches:

  * refrain from submitting these;

  * prefix them as "Nit" so that the contributor knows it's OK not to
    address;

  * follow up in a subsequent PR, out of courtesy, you may want to let
    the original contributor know.

* Do not rush, take the time to make your comments clear and justify
  your suggestions.

* You are the face of the project. Bad days occur to everyone, in that
  occasion you deserve a break: try to take your time and stay
  offline.

your suggestions.

  offline.

[1] Adapted from the numpy communication guidelines.

Reading the existing code base

Reading and digesting an existing code base is always a difficult
exercise that takes time and experience to master. Even though we try
to write simple code in general, understanding the code can seem
overwhelming at first, given the sheer size of the project. Here is a
list of tips that may help make this task easier and faster (in no
particular order).

* Get acquainted with the APIs of scikit-learn objects: understand
  what *fit*, *predict*, *transform*, etc. are used for.

* Before diving into reading the code of a function / class, go
  through the docstrings first and try to get an idea of what each
  parameter / attribute is doing. It may also help to stop a minute
  and think *how would I do this myself if I had to?*

* The trickiest thing is often to identify which portions of the code
  are relevant, and which are not. In scikit-learn **a lot** of input
  checking is performed, especially at the beginning of the *fit*
  methods. Sometimes, only a very small portion of the code is doing
  the actual job. For example looking at the "fit()" method of
  "LinearRegression", what you're looking for might just be the call
  the "scipy.linalg.lstsq", but it is buried into multiple lines of

the actual job. For example looking at the "fit()" method of
  input checking and the handling of different kinds of parameters.

* Due to the use of Inheritance, some methods may be implemented in
  parent classes. All estimators inherit at least from
  "BaseEstimator", and from a "Mixin" class (e.g. "ClassifierMixin")
  that enables default behaviour depending on the nature of the
  estimator (classifier, regressor, transformer, etc.).

* Sometimes, reading the tests for a given function will give you an
  idea of what its intended purpose is. You can use "git grep" (see
  below) to find all the tests written for a function. Most tests for
  a specific function/class are placed under the "tests/" folder of
  the module

* You'll often see code looking like this: "out =
  Parallel(...)(delayed(some_function)(param) for param in
  some_iterable)". This runs "some_function" in parallel using Joblib.
  "out" is then an iterable containing the values returned by
  "some_function" for each call.

* We use Cython to write fast code. Cython code is located in ".pyx"
  and ".pxd" files. Cython code has a more C-like flavor: we use
  pointers, perform manual memory allocation, etc. Having some minimal
  experience in C / C++ is pretty much mandatory here.

* Master your tools.

  * With such a big project, being efficient with your favorite editor

experience in C / C++ is pretty much mandatory here.

    or IDE goes a long way towards digesting the code base. Being able
    to quickly jump (or *peek*) to a function/class/attribute
    definition helps a lot. So does being able to quickly see where a
    given name is used in a file.

  * git also has some built-in killer features. It is often useful to
    understand how a file changed over time, using e.g. "git blame"
    (manual). This can also be done directly on GitHub. "git grep"
    (examples) is also extremely useful to see every occurrence of a
    pattern (e.g. a function call or a variable) in the code base.

* Configure *git blame* to ignore the commit that migrated the code
  style to *black*.

  Find out more information in black's documentation for avoiding
  ruining git blame.

Developer's Guide

  * Ways to contribute

  * Submitting a bug report or a feature request

    * How to make a good bug report

  * Contributing code

    * Video resources

    * How to contribute

    * Pull request checklist

      * Continuous Integration (CI)

      * Stalled pull requests

      * Stalled and Unclaimed Issues

    * Issues for New Contributors

  * Documentation

    * Building the documentation

    * Guidelines for writing documentation

    * Cross-referencing

    * Generated documentation on GitHub Actions

  * Testing and improving test coverage

    * Writing matplotlib related tests

    * Workflow to improve test coverage

  * Monitoring performance

  * Issue Tracker Tags

  * Maintaining backwards compatibility

    * Deprecation

    * Change the default value of a parameter

  * Code Review Guidelines

    * Communication Guidelines

  * Reading the existing code base

* Crafting a minimal reproducer for scikit-learn

  * Good practices

    * Provide a failing code example with minimal comments

    * Boil down your script to something as small as possible

    * **DO NOT** report your data unless it is extremely necessary

    * Use markdown formatting

  * Synthetic dataset

    * NumPy

    * Pandas

    * *make_regression*

    * *make_classification*

    * *make_blobs*

    * Dataset loading utilities

* Developing scikit-learn estimators

  * APIs of scikit-learn objects

    * Different objects

    * Estimators

      * Instantiation

      * Fitting

      * Estimated Attributes

      * Fitting

      * Optional Arguments

      * Universal attributes

  * Rolling your own estimator

    * get_params and set_params

    * Parameters and init

    * Cloning

    * Pipeline compatibility

    * Estimator types

    * Specific models

    * Estimator Tags

  * Developer API for *set_output*

  * Coding guidelines

    * Input validation

    * Random Numbers

    * Numerical assertions in tests

* Developers' Tips and Tricks

  * Productivity and sanity-preserving tips

    * Folding and unfolding outdated diffs on pull requests

    * Checking out pull requests as remote-tracking branches

    * Display code coverage in pull requests

    * Useful pytest aliases and flags

    * Standard replies for reviewing

  * Debugging memory errors in Cython with valgrind

  * Building and testing for the ARM64 platform on a x86_64 machine

* Utilities for Developers

  * Validation Tools

  * Efficient Linear Algebra & Array Operations

  * Efficient Random Sampling

  * Efficient Routines for Sparse Matrices

  * Graph Routines

  * Testing Functions

  * Multiclass and multilabel utility function

  * Helper Functions

  * Hash Functions

  * Warnings and Exceptions

* How to optimize for speed

  * Python, Cython or C/C++?

  * Profiling Python code

  * Memory usage profiling

  * Performance tips for the Cython developer

    * Using OpenMP

  * Profiling compiled extensions

* Python, Cython or C/C++?

    * Using yep and gperftools

    * Using gprof

    * Using valgrind / callgrind / kcachegrind

      * kcachegrind

  * Multi-core parallelism using "joblib.Parallel"

  * A simple algorithmic trick: warm restarts

* Installing the development version of scikit-learn

  * Installing nightly builds

  * Building from source

    * Dependencies

      * Runtime dependencies

      * Build dependencies

      * Test dependencies

    * Building a specific version from a tag

    * Editable mode

  * Platform-specific instructions

    * Windows

    * macOS

      * macOS compilers from conda-forge

      * macOS compilers from Homebrew

    * Linux

      * Linux compilers from the system

      * Linux compilers from conda-forge

    * FreeBSD

  * Parallel builds

* Bug triaging and issue curation

  * Working on issues to improve them

  * Working on PRs to help review

  * Triaging operations for members of the core and contributor
    experience teams

  * A typical workflow for triaging issues

* Maintainer / core-developer information

  * Releasing

    * Before a release

    * Preparing a release PR

      * Major version release

      * Minor version release

    * Making a release

    * Release checklist

  * Merging Pull Requests

  * The scikit-learn.org web site

  * Travis Cron jobs

  * Experimental features

* Developing with the Plotting API

* Making a release

  * Plotting API Overview

  * Plotting with Multiple Axes

Installing the development version of scikit-learn
**************************************************

This section introduces how to install the **main branch** of scikit-
learn. This can be done by either installing a nightly build or
building from source.

Installing nightly builds

The continuous integration servers of the scikit-learn project build,
test and upload wheel packages for the most recent Python version on a
nightly basis.

Installing a nightly build is the quickest way to:

* try a new feature that will be shipped in the next release (that is,
  a feature from a pull-request that was recently merged to the main
  branch);

* check whether a bug you encountered has been fixed since the last
  release.

You can install the nightly build of scikit-learn using the *scipy-
wheels-nightly* index from the PyPI registry of *anaconda.org*:

Note that first uninstalling scikit-learn might be required to be able
to install nightly builds of scikit-learn.

Building from source

Building from source is required to work on a contribution (bug fix,
new feature, code or documentation improvement).

1. Use Git to check out the latest source from the scikit-learn
   repository on Github.:

   If you plan on submitting a pull-request, you should clone from
   your fork instead.

2. Install a recent version of Python (3.9 is recommended at the time
   of writing) for instance using Miniforge3. Miniforge provides a

your fork instead.

   conda-based distribution of Python and the most popular scientific
   libraries.

   If you installed Python with conda, we recommend to create a
   dedicated conda environment with all the build dependencies of
   scikit-learn (namely NumPy, SciPy, and Cython):

3. **Alternative to conda:** If you run Linux or similar, you can
   instead use your system's Python provided it is recent enough (3.8
   or higher at the time of writing). In this case, we recommend to
   create a dedicated virtualenv and install the scikit-learn build
   dependencies with pip:

4. Install a compiler with OpenMP support for your platform. See
   instructions for Windows, macOS, Linux and FreeBSD.

5. Build the project with pip in Editable mode:

6. Check that the installed scikit-learn has a version number ending
   with *.dev0*:

7. Please refer to the Developer's Guide and Useful pytest aliases and
   flags to run the tests on the module of your choice.

Note:

  You will have to run the "pip install --no-build-isolation
  --editable ." command every time the source code of a Cython file is
  updated (ending in *.pyx* or *.pxd*). Use the "--no-build-isolation"

Dependencies

Runtime dependencies

flag to avoid compiling the whole project each time, only the files

Scikit-learn requires the following dependencies both at build time
and at runtime:

* Python (>= 3.8),

* NumPy (>= >>|NumpyMinVersion|<<),

* SciPy (>= >>|ScipyMinVersion|<<),

* Joblib (>= >>|JoblibMinVersion|<<),

* threadpoolctl (>= >>|ThreadpoolctlMinVersion|<<).

Build dependencies
~~~~~~~~~~~~~~~~~~

Building Scikit-learn also requires:

* Cython >= >>|CythonMinVersion|<<

* A C/C++ compiler and a matching OpenMP runtime library. See the
  platform system specific instructions for more details.

Note:

  If OpenMP is not supported by the compiler, the build will be done
  with OpenMP functionalities disabled. This is not recommended since
  it will force some estimators to run in sequential mode instead of
  leveraging thread-based parallelism. Setting the
  "SKLEARN_FAIL_NO_OPENMP" environment variable (before cythonization)
  will force the build to fail if OpenMP is not supported.

Since version 0.21, scikit-learn automatically detects and uses the
linear algebra library used by SciPy **at runtime**. Scikit-learn has
therefore no build dependency on BLAS/LAPACK implementations such as
OpenBlas, Atlas, Blis or MKL.

Test dependencies

Running tests requires:

* pytest >= >>|PytestMinVersion|<<

Some tests also require pandas.

Building a specific version from a tag

If you want to build a stable version, you can "git checkout
<VERSION>" to get the code for that particular version, or download an
zip archive of the version from github.

Editable mode

If you run the development version, it is cumbersome to reinstall the
package each time you update the sources. Therefore it is recommended
that you install in with the "pip install --no-build-isolation
--editable ." command, which allows you to edit the code in-place.
This builds the extension in place and creates a link to the
development directory (see the pip docs).

As the doc aboves explains, this is fundamentally similar to using the
command "python setup.py develop". (see the setuptool docs). It is
however preferred to use pip.

On Unix-like systems, you can equivalently type "make in" from the
top-level folder. Have a look at the "Makefile" for additional
utilities.

Platform-specific instructions

Here are instructions to install a working C/C++ compiler with OpenMP
support to build scikit-learn Cython extensions for each supported
platform.

Windows
-------

First, download the Build Tools for Visual Studio 2019 installer.

Run the downloaded *vs_buildtools.exe* file, during the installation
you will need to make sure you select "Desktop development with C++",

similarly to this screenshot:

[image]

Secondly, find out if you are running 64-bit or 32-bit Python. The
building command depends on the architecture of the Python
interpreter. You can check the architecture by running the following
in "cmd" or "powershell" console:

For 64-bit Python, configure the build environment by running the
following commands in "cmd" or an Anaconda Prompt (if you use
Anaconda):

      $ SET DISTUTILS_USE_SDK=1
      $ "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvarsall.bat" x64

Replace "x64" by "x86" to build for 32-bit Python.

Please be aware that the path above might be different from user to
user. The aim is to point to the "vcvarsall.bat" file that will set
the necessary environment variables in the current command prompt.

Finally, build scikit-learn from this command prompt:

macOS
-----

The default C compiler on macOS, Apple clang (confusingly aliased as
*/usr/bin/gcc*), does not directly support OpenMP. We present two
alternatives to enable OpenMP support:

* either install *conda-forge::compilers* with conda;

* or install *libomp* with Homebrew to extend the default Apple clang
  compiler.

For Apple Silicon M1 hardware, only the conda-forge method below is
known to work at the time of writing (January 2021). You can install
the *macos/arm64* distribution of conda using the miniforge installer

macOS compilers from conda-forge

If you use the conda package manager (version >= 4.7), you can install
the "compilers" meta-package from the conda-forge channel, which
provides OpenMP-enabled C/C++ compilers based on the llvm toolchain.

First install the macOS command line tools:

It is recommended to use a dedicated conda environment to build
scikit-learn from source:

Note:

  If you get any conflicting dependency error message, try commenting
  out any custom conda configuration in the "$HOME/.condarc" file. In
  particular the "channel_priority: strict" directive is known to
  cause problems for this setup.

You can check that the custom compilers are properly installed from
conda forge using the following command:

which should include "compilers" and "llvm-openmp".

Note:

  If you installed these packages after creating and activating a new
  conda environment, you will need to first deactivate and then
  reactivate the environment for these changes to take effect.

The compilers meta-package will automatically set custom environment
variables:

They point to files and folders from your "sklearn-dev" conda
environment (in particular in the bin/, include/ and lib/ subfolders).
For instance "-L/path/to/conda/envs/sklearn-dev/lib" should appear in
"LDFLAGS".

In the log, you should see the compiled extension being built with the

clang and clang++ compilers installed by conda with the "-fopenmp"
command line flag.

macOS compilers from Homebrew

Another solution is to enable OpenMP support for the clang compiler
shipped by default on macOS.

Install the Homebrew package manager for macOS.

Install the LLVM OpenMP library:

Set the following environment variables:

Finally, build scikit-learn in verbose mode (to check for the presence
of the "-fopenmp" flag in the compiler commands):

Linux
-----

Linux compilers from the system

Installing scikit-learn from source without using conda requires you
to have installed the scikit-learn Python development headers and a
working C/C++ compiler with OpenMP support (typically the GCC
toolchain).

Install build dependencies for Debian-based operating systems, e.g.
Ubuntu:

then proceed as usual:

Cython and the pre-compiled wheels for the runtime dependencies
(numpy, scipy and joblib) should automatically be installed in
"$HOME/.local/lib/pythonX.Y/site-packages". Alternatively you can run
the above commands from a virtualenv or a conda environment to get
full isolation from the Python packages installed via the system
packager. When using an isolated environment, "pip3" should be
replaced by "pip" in the above commands.

When precompiled wheels of the runtime dependencies are not available
for your architecture (e.g. ARM), you can install the system versions:

On Red Hat and clones (e.g. CentOS), install the dependencies using:

Linux compilers from conda-forge

Alternatively, install a recent version of the GNU C Compiler
toolchain (GCC) in the user folder using conda:

FreeBSD
-------

The clang compiler included in FreeBSD 12.0 and 11.2 base systems does
not include OpenMP support. You need to install the *openmp* library
from packages (or ports):

This will install header files in "/usr/local/include" and libs in
"/usr/local/lib". Since these directories are not searched by default,
you can set the environment variables to these locations:

Finally, build the package using the standard command:

For the upcoming FreeBSD 12.1 and 11.3 versions, OpenMP will be
included in the base system and these steps will not be necessary.

Parallel builds

It is possible to build scikit-learn compiled extensions in parallel
by setting and environment variable as follows before calling the "pip
install" or "python setup.py build_ext" commands:

   export SKLEARN_BUILD_PARALLEL=3
   pip install --verbose --no-build-isolation --editable .

On a machine with 2 CPU cores, it can be beneficial to use a

export SKLEARN_BUILD_PARALLEL=3

parallelism level of 3 to overlap IO bound tasks (reading and writing
files on disk) with CPU bound tasks (actually compiling).

How to optimize for speed

The following gives some practical guidelines to help you write
efficient code for the scikit-learn project.

Note:

  While it is always useful to profile your code so as to **check
  performance assumptions**, it is also highly recommended to **review
  the literature** to ensure that the implemented algorithm is the
  state of the art for the task before investing into costly
  implementation optimization.Times and times, hours of efforts
  invested in optimizing complicated implementation details have been
  rendered irrelevant by the subsequent discovery of simple
  **algorithmic tricks**, or by using another algorithm altogether
  that is better suited to the problem.The section A simple
  algorithmic trick: warm restarts gives an example of such a trick.

Python, Cython or C/C++?

In general, the scikit-learn project emphasizes the **readability** of
the source code to make it easy for the project users to dive into the
source code so as to understand how the algorithm behaves on their
data but also for ease of maintainability (by the developers).

When implementing a new algorithm is thus recommended to **start
implementing it in Python using Numpy and Scipy** by taking care of
avoiding looping code using the vectorized idioms of those libraries.
In practice this means trying to **replace any nested for loops by
calls to equivalent Numpy array methods**. The goal is to avoid the

CPU wasting time in the Python interpreter rather than crunching
numbers to fit your statistical model. It's generally a good idea to
consider NumPy and SciPy performance tips: https://scipy.github.io
/old-wiki/pages/PerformanceTips

Sometimes however an algorithm cannot be expressed efficiently in
simple vectorized Numpy code. In this case, the recommended strategy
is the following:

   1. **Profile** the Python implementation to find the main
      bottleneck and isolate it in a **dedicated module level
      function**. This function will be reimplemented as a compiled
      extension module.

   2. If there exists a well maintained BSD or MIT **C/C++**
      implementation of the same algorithm that is not too big, you
      can write a **Cython wrapper** for it and include a copy of the
      source code of the library in the scikit-learn source tree: this
      strategy is used for the classes "svm.LinearSVC", "svm.SVC" and
      "linear_model.LogisticRegression" (wrappers for liblinear and
      libsvm).

   3. Otherwise, write an optimized version of your Python function
      using **Cython** directly. This strategy is used for the
      "linear_model.ElasticNet" and "linear_model.SGDClassifier"
      classes for instance.

   4. **Move the Python version of the function in the tests** and use
      it to check that the results of the compiled extension are

classes for instance.

      consistent with the gold standard, easy to debug Python version.

   5. Once the code is optimized (not simple bottleneck spottable by
      profiling), check whether it is possible to have **coarse
      grained parallelism** that is amenable to **multi-processing**
      by using the "joblib.Parallel" class.

When using Cython, use either

to generate C files. You are responsible for adding .c/.cpp extensions
along with build parameters in each submodule "setup.py".

C/C++ generated files are embedded in distributed stable packages. The
goal is to make it possible to install scikit-learn stable version on
any machine with Python, Numpy, Scipy and C/C++ compiler.

Profiling Python code

In order to profile Python code we recommend to write a script that
loads and prepare you data and then use the IPython integrated
profiler for interactively exploring the relevant part for the code.

Suppose we want to profile the Non Negative Matrix Factorization
module of scikit-learn. Let us setup a new IPython session and load
the digits dataset and as in the
*sphx_glr_auto_examples_classification_plot_digits_classification.py*
example:

   In [1]: from sklearn.decomposition import NMF

   In [2]: from sklearn.datasets import load_digits

   In [3]: X, _ = load_digits(return_X_y=True)

example:

Before starting the profiling session and engaging in tentative
optimization iterations, it is important to measure the total
execution time of the function we want to optimize without any kind of
profiler overhead and save it somewhere for later reference:

   In [4]: %timeit NMF(n_components=16, tol=1e-2).fit(X)
   1 loops, best of 3: 1.7 s per loop

To have a look at the overall performance profile using the "%prun"
magic command:

   In [5]: %prun -l nmf.py NMF(n_components=16, tol=1e-2).fit(X)
            14496 function calls in 1.682 CPU seconds

      Ordered by: internal time
      List reduced from 90 to 9 due to restriction <'nmf.py'>

      ncalls  tottime  percall  cumtime  percall filename:lineno(function)
          36    0.609    0.017    1.499    0.042 nmf.py:151(_nls_subproblem)
        1263    0.157    0.000    0.157    0.000 nmf.py:18(_pos)
           1    0.053    0.053    1.681    1.681 nmf.py:352(fit_transform)
         673    0.008    0.000    0.057    0.000 nmf.py:28(norm)
           1    0.006    0.006    0.047    0.047 nmf.py:42(_initialize_nmf)
          36    0.001    0.000    0.010    0.000 nmf.py:36(_sparseness)
          30    0.001    0.000    0.001    0.000 nmf.py:23(_neg)
           1    0.000    0.000    0.000    0.000 nmf.py:337(__init__)
           1    0.000    0.000    1.681    1.681 nmf.py:461(fit)

30    0.001    0.000    0.001    0.000 nmf.py:23(_neg)

The "tottime" column is the most interesting: it gives to total time
spent executing the code of a given function ignoring the time spent
in executing the sub-functions. The real total time (local code + sub-
function calls) is given by the "cumtime" column.

Note the use of the "-l nmf.py" that restricts the output to lines
that contains the "nmf.py" string. This is useful to have a quick look
at the hotspot of the nmf Python module it-self ignoring anything
else.

Here is the beginning of the output of the same command without the
"-l nmf.py" filter:

   In [5] %prun NMF(n_components=16, tol=1e-2).fit(X)
            16159 function calls in 1.840 CPU seconds

        2833    0.653    0.000    0.653    0.000 {numpy.core._dotblas.dot}
          46    0.651    0.014    1.636    0.036 nmf.py:151(_nls_subproblem)
        1397    0.171    0.000    0.171    0.000 nmf.py:18(_pos)
        2780    0.167    0.000    0.167    0.000 {method 'sum' of 'numpy.ndarray' objects}
           1    0.064    0.064    1.840    1.840 nmf.py:352(fit_transform)
        1542    0.043    0.000    0.043    0.000 {method 'flatten' of 'numpy.ndarray' objects}

1    0.064    0.064    1.840    1.840 nmf.py:352(fit_transform)
         337    0.019    0.000    0.019    0.000 {method 'all' of 'numpy.ndarray' objects}
        2734    0.011    0.000    0.181    0.000 fromnumeric.py:1185(sum)
           2    0.010    0.005    0.010    0.005 {numpy.linalg.lapack_lite.dgesdd}
         748    0.009    0.000    0.065    0.000 nmf.py:28(norm)
   ...

The above results show that the execution is largely dominated by dot
products operations (delegated to blas). Hence there is probably no
huge gain to expect by rewriting this code in Cython or C/C++: in this
case out of the 1.7s total execution time, almost 0.7s are spent in
compiled code we can consider optimal. By rewriting the rest of the
Python code and assuming we could achieve a 1000% boost on this
portion (which is highly unlikely given the shallowness of the Python
loops), we would not gain more than a 2.4x speed-up globally.

Hence major improvements can only be achieved by **algorithmic
improvements** in this particular example (e.g. trying to find
operation that are both costly and useless to avoid computing then
rather than trying to optimize their implementation).

It is however still interesting to check what's happening inside the
"_nls_subproblem" function which is the hotspot if we only consider
Python code: it takes around 100% of the accumulated time of the

module. In order to better understand the profile of this specific
function, let us install "line_profiler" and wire it to IPython:

* **Under IPython 0.13+**, first create a configuration profile:

Then register the line_profiler extension in
"~/.ipython/profile_default/ipython_config.py":

   c.TerminalIPythonApp.extensions.append('line_profiler')
   c.InteractiveShellApp.extensions.append('line_profiler')

This will register the "%lprun" magic command in the IPython terminal
application and the other frontends such as qtconsole and notebook.

Now restart IPython and let us use this new toy:

   In [1]: from sklearn.datasets import load_digits

   In [2]: from sklearn.decomposition import NMF
     ... : from sklearn.decomposition._nmf import _nls_subproblem

   In [4]: %lprun -f _nls_subproblem NMF(n_components=16, tol=1e-2).fit(X)
   Timer unit: 1e-06 s

   File: sklearn/decomposition/nmf.py
   Function: _nls_subproblem at line 137
   Total time: 1.73153 s

   Line #      Hits         Time  Per Hit   % Time  Line Contents
   ==============================================================
      137                                           def _nls_subproblem(V, W, H_init, tol, max_iter):

      138                                               """Non-negative least square solver
      ...
      170                                               """
      171        48         5863    122.1      0.3      if (H_init < 0).any():
      172                                                   raise ValueError("Negative values in H_init passed to NLS solver.")
      173
      174        48          139      2.9      0.0      H = H_init
      175        48       112141   2336.3      5.8      WtV = np.dot(W.T, V)
      176        48        16144    336.3      0.8      WtW = np.dot(W.T, W)
      177
      178                                               # values justified in the paper
      179        48          144      3.0      0.0      alpha = 1
      180        48          113      2.4      0.0      beta = 0.1
      181       638         1880      2.9      0.1      for n_iter in range(1, max_iter + 1):
      182       638       195133    305.9     10.2          grad = np.dot(WtW, H) - WtV
      183       638       495761    777.1     25.9          proj_gradient = norm(grad[np.logical_or(grad < 0, H > 0)])
      184       638         2449      3.8      0.1          if proj_gradient < tol:
      185        48          130      2.7      0.0              break
      186

184       638         2449      3.8      0.1          if proj_gradient < tol:
      186
      187      1474         4474      3.0      0.2          for inner_iter in range(1, 20):
      188      1474        83833     56.9      4.4              Hn = H - alpha * grad
      189                                                       # Hn = np.where(Hn > 0, Hn, 0)
      190      1474       194239    131.8     10.1              Hn = _pos(Hn)
      191      1474        48858     33.1      2.5              d = Hn - H
      192      1474       150407    102.0      7.8              gradd = np.sum(grad * d)
      193      1474       515390    349.7     26.9              dQd = np.sum(np.dot(WtW, d) * d)
      ...

By looking at the top values of the "% Time" column it is really easy
to pin-point the most expensive expressions that would deserve
additional care.

Memory usage profiling

You can analyze in detail the memory usage of any Python code with the
help of memory_profiler. First, install the latest version:

Then, setup the magics in a manner similar to "line_profiler".

* **Under IPython 0.11+**, first create a configuration profile:

Then register the extension in
"~/.ipython/profile_default/ipython_config.py" alongside the line
profiler:

   c.TerminalIPythonApp.extensions.append('memory_profiler')
   c.InteractiveShellApp.extensions.append('memory_profiler')

profiler:

This will register the "%memit" and "%mprun" magic commands in the
IPython terminal application and the other frontends such as qtconsole
and   notebook.

"%mprun" is useful to examine, line-by-line, the memory usage of key
functions in your program. It is very similar to "%lprun", discussed
in the previous section. For example, from the "memory_profiler"
"examples" directory:

   In [1] from example import my_func

   In [2] %mprun -f my_func my_func()
   Filename: example.py

   Line #    Mem usage  Increment   Line Contents
   ==============================================
        3                           @profile
        4      5.97 MB    0.00 MB   def my_func():
        5     13.61 MB    7.64 MB       a = [1] * (10 ** 6)
        6    166.20 MB  152.59 MB       b = [2] * (2 * 10 ** 7)
        7     13.61 MB -152.59 MB       del b
        8     13.61 MB    0.00 MB       return a

Another useful magic that "memory_profiler" defines is "%memit", which
is analogous to "%timeit". It can be used as follows:

   In [1]: import numpy as np

   In [2]: %memit np.zeros(1e7)
   maximum of 3: 76.402344 MB per loop

For more details, see the docstrings of the magics, using "%memit?"
and "%mprun?".

Performance tips for the Cython developer

If profiling of the Python code reveals that the Python interpreter
overhead is larger by one order of magnitude or more than the cost of
the actual numerical computation (e.g. "for" loops over vector
components, nested evaluation of conditional expression, scalar
arithmetic...), it is probably adequate to extract the hotspot portion
of the code as a standalone function in a ".pyx" file, add static type
declarations and then use Cython to generate a C program suitable to
be compiled as a Python extension module.

The official documentation available at http://docs.cython.org/
contains a tutorial and reference guide for developing such a module.
In the following we will just highlight a couple of tricks that we
found important in practice on the existing cython codebase in the
scikit-learn project.

TODO: html report, type declarations, bound checks, division by zero
checks, memory alignment, direct blas calls...

* https://www.youtube.com/watch?v=gMvkiQ-gOW8

* https://conference.scipy.org/proceedings/SciPy2009/paper_1/

* https://conference.scipy.org/proceedings/SciPy2009/paper_2/

Using OpenMP

Since scikit-learn can be built without OpenMP, it's necessary to
protect each direct call to OpenMP.

There are some helpers in sklearn/utils/_openmp_helpers.pyx that you
can reuse for the main useful functionalities and already have the

necessary protection to be built without OpenMP.

If the helpers are not enough, you need to protect your OpenMP code
using the following syntax:

   # importing OpenMP
   IF SKLEARN_OPENMP_PARALLELISM_ENABLED:
       cimport openmp

   # calling OpenMP
       max_threads = openmp.omp_get_max_threads()
   ELSE:
       max_threads = 1

Note:

  Protecting the parallel loop, "prange", is already done by cython.

Profiling compiled extensions

When working with compiled extensions (written in C/C++ with a wrapper
or directly as Cython extension), the default Python profiler is
useless: we need a dedicated tool to introspect what's happening
inside the compiled extension it-self.

Using yep and gperftools

Easy profiling without special compilation options use yep:

* https://pypi.org/project/yep/

* https://fa.bianp.net/blog/2011/a-profiler-for-python-extensions

Using gprof

In order to profile compiled Python extensions one could use "gprof"
after having recompiled the project with "gcc -pg" and using the
"python-dbg" variant of the interpreter on debian / ubuntu: however
this approach requires to also have "numpy" and "scipy" recompiled
with "-pg" which is rather complicated to get working.

Fortunately there exist two alternative profilers that don't require
you to recompile everything.

Using valgrind / callgrind / kcachegrind

kcachegrind
~~~~~~~~~~~

"yep" can be used to create a profiling report. "kcachegrind" provides
a graphical environment to visualize this report:

Note:

  "yep" can be executed with the argument "--lines" or "-l" to compile
  a profiling report 'line by line'.

Multi-core parallelism using "joblib.Parallel"

See joblib documentation

A simple algorithmic trick: warm restarts

See the glossary entry for *warm_start*

Bug triaging and issue curation
*******************************

The issue tracker is important to the communication in the project: it
helps developers identify major projects to work on, as well as to
discuss priorities. For this reason, it is important to curate it,
adding labels to issues and closing issues that are not necessary.

Working on issues to improve them

Improving issues increases their chances of being successfully
resolved. Guidelines on submitting good issues can be found here. A
third party can give useful feedback or even add comments on the
issue. The following actions are typically useful:

   * documenting issues that are missing elements to reproduce the
     problem such as code samples

   * suggesting better use of code formatting

   * suggesting to reformulate the title and description to make them
     more explicit about the problem to be solved

   * linking to related issues or discussions while briefly describing
     how they are related, for instance "See also #xyz for a similar
     attempt at this" or "See also #xyz where the same thing happened
     in SomeEstimator" provides context and helps the discussion.

Fruitful discussions

Online discussions may be harder than it seems at first glance, in
particular given that a person new to open-source may have a very
different understanding of the process than a seasoned maintainer.

Overall, it is useful to stay positive and assume good will. The

following article explores how to lead online discussions in the
context of open source.

Working on PRs to help review

Reviewing code is also encouraged. Contributors and users are welcome
to participate to the review process following our review guidelines.

Triaging operations for members of the core and contributor experience teams
============================================================================

In addition to the above, members of the core team and the contributor
experience team can do the following important tasks:

* Update labels for issues and PRs: see the list of the available
  github labels.

* Determine if a PR must be relabeled as stalled or needs help (this
  is typically very important in the context of sprints, where the
  risk is to create many unfinished PRs)

* If a stalled PR is taken over by a newer PR, then label the stalled
  PR as "Superseded", leave a comment on the stalled PR linking to the
  new PR, and likely close the stalled PR.

* Triage issues:

  * **close usage questions** and politely point the reporter to use
    Stack Overflow instead.

  * **close duplicate issues**, after checking that they are indeed
    duplicate. Ideally, the original submitter moves the discussion to
    the older, duplicate issue

Stack Overflow instead.

  * **close issues that cannot be replicated**, after leaving time (at
    least a week) to add extra information

Saved replies are useful to gain time and yet be welcoming and polite
when triaging.

See the github description for roles in the organization.

Closing issues: a tough call

When uncertain on whether an issue should be closed or not, it is best
to strive for consensus with the original poster, and possibly to seek
relevant expertise. However, when the issue is a usage question, or
when it has been considered as unclear for many years it should be
closed.

A typical workflow for triaging issues

The following workflow [1] is a good way to approach issue triaging:

1. Thank the reporter for opening an issue

   The issue tracker is many people's first interaction with the
   scikit-learn project itself, beyond just using the library. As
   such, we want it to be a welcoming, pleasant experience.

2. Is this a usage question? If so close it with a polite message
   (here is an example).

3. Is the necessary information provided?

   If crucial information (like the version of scikit-learn used), is
   missing feel free to ask for that and label the issue with "Needs
   info".

4. Is this a duplicate issue?

If crucial information (like the version of scikit-learn used), is
   info".

   We have many open issues. If a new issue seems to be a duplicate,
   point to the original issue. If it is a clear duplicate, or
   consensus is that it is redundant, close it. Make sure to still
   thank the reporter, and encourage them to chime in on the original
   issue, and perhaps try to fix it.

   If the new issue provides relevant information, such as a better or
   slightly different example, add it to the original issue as a
   comment or an edit to the original post.

5. Make sure that the title accurately reflects the issue. If you have
   the necessary permissions edit it yourself if it's not clear.

6. Is the issue minimal and reproducible?

   For bug reports, we ask that the reporter provide a minimal
   reproducible example. See this useful post by Matthew Rocklin for a
   good explanation. If the example is not reproducible, or if it's
   clearly not minimal, feel free to ask the reporter if they can
   provide and example or simplify the provided one. Do acknowledge
   that writing minimal reproducible examples is hard work. If the
   reporter is struggling, you can try to write one yourself.

   If a reproducible example is provided, but you see a
   simplification, add your simpler reproducible example.

7. Add the relevant labels, such as "Documentation" when the issue is

If a reproducible example is provided, but you see a

   about documentation, "Bug" if it is clearly a bug, "Enhancement" if
   it is an enhancement request, ...

   If the issue is clearly defined and the fix seems relatively
   straightforward, label the issue as “Good first issue”.

   An additional useful step can be to tag the corresponding module
   e.g. *sklearn.linear_models* when relevant.

8. Remove the "Needs Triage" label from the issue if the label exists.

[1] Adapted from the pandas project maintainers guide

Crafting a minimal reproducer for scikit-learn
**********************************************

Whether submitting a bug report, designing a suite of tests, or simply
posting a question in the discussions, being able to craft minimal,
reproducible examples (or minimal, workable examples) is the key to
communicating effectively and efficiently with the community.

There are very good guidelines on the internet such as this
StackOverflow document or this blogpost by Matthew Rocklin on crafting
Minimal Complete Verifiable Examples (referred below as MCVE). Our
goal is not to be repetitive with those references but rather to
provide a step-by-step guide on how to narrow down a bug until you
have reached the shortest possible code to reproduce it.

The first step before submitting a bug report to scikit-learn is to
read the Issue template. It is already quite informative about the
information you will be asked to provide.

Good practices

In this section we will focus on the **Steps/Code to Reproduce**
section of the Issue template. We will start with a snippet of code
that already provides a failing example but that has room for
readability improvement. We then craft a MCVE from it.

**Example**

   # I am currently working in a ML project and when I tried to fit a
   # GradientBoostingRegressor instance to my_data.csv I get a UserWarning:
   # "X has feature names, but DecisionTreeRegressor was fitted without
   # feature names". You can get a copy of my dataset from

# "X has feature names, but DecisionTreeRegressor was fitted without
   # https://example.com/my_data.csv and verify my features do have
   # names. The problem seems to arise during fit when I pass an integer
   # to the n_iter_no_change parameter.

   df = pd.read_csv('my_data.csv')
   X = df[["feature_name"]] # my features do have names
   y = df["target"]

   # We set random_state=42 for the train_test_split
   X_train, X_test, y_train, y_test = train_test_split(
       X, y, test_size=0.33, random_state=42
   )

   scaler = StandardScaler(with_mean=False)
   X_train = scaler.fit_transform(X_train)
   X_test = scaler.transform(X_test)

   # An instance with default n_iter_no_change raises no error nor warnings
   gbdt = GradientBoostingRegressor(random_state=0)
   gbdt.fit(X_train, y_train)
   default_score = gbdt.score(X_test, y_test)

   # the bug appears when I change the value for n_iter_no_change
   gbdt = GradientBoostingRegressor(random_state=0, n_iter_no_change=5)
   other_score = gbdt.score(X_test, y_test)

Provide a failing code example with minimal comments

Writing instructions to reproduce the problem in English is often
ambiguous. Better make sure that all the necessary details to
reproduce the problem are illustrated in the Python code snippet to

avoid any ambiguity. Besides, by this point you already provided a
concise description in the **Describe the bug** section of the Issue
template.

The following code, while **still not minimal**, is already **much
better** because it can be copy-pasted in a Python terminal to
reproduce the problem in one step. In particular:

   * it contains **all necessary imports statements**;

   * it can fetch the public dataset without having to manually
     download a file and put it in the expected location on the disk.

**Improved example**

   import pandas as pd

   df = pd.read_csv("https://example.com/my_data.csv")
   X = df[["feature_name"]]

   from sklearn.model_selection import train_test_split

   )

   from sklearn.preprocessing import StandardScaler

   from sklearn.ensemble import GradientBoostingRegressor

   gbdt.fit(X_train, y_train)  # no warning

   gbdt.fit(X_train, y_train)  # raises warning

default_score = gbdt.score(X_test, y_test)

Boil down your script to something as small as possible

You have to ask yourself which lines of code are relevant and which
are not for reproducing the bug. Deleting unnecessary lines of code or
simplifying the function calls by omitting unrelated non-default
options will help you and other contributors narrow down the cause of
the bug.

In particular, for this specific example:

* the warning has nothing to do with the *train_test_split* since it
  already appears in the training step, before we use the test set.

* similarly, the lines that compute the scores on the test set are not
  necessary;

* the bug can be reproduced for any value of *random_state* so leave
  it to its default;

* the bug can be reproduced without preprocessing the data with the
  *StandardScaler*.

   gbdt = GradientBoostingRegressor()
   gbdt.fit(X, y)  # no warning

   gbdt = GradientBoostingRegressor(n_iter_no_change=5)
   gbdt.fit(X, y)  # raises warning

**DO NOT** report your data unless it is extremely necessary

gbdt.fit(X, y)  # no warning

------------------------------------------------------------

The idea is to make the code as self-contained as possible. For doing
so, you can use a Synthetic dataset. It can be generated using numpy,
pandas or the "sklearn.datasets" module. Most of the times the bug is
not related to a particular structure of your data. Even if it is, try
to find an available dataset that has similar characteristics to yours
and that reproduces the problem. In this particular case, we are
interested in data that has labeled feature names.

   df = pd.DataFrame(
       {
           "feature_name": [-12.32, 1.43, 30.01, 22.17],
           "target": [72, 55, 32, 43],
       }
   )

   gbdt.fit(X, y) # no warning
   gbdt.fit(X, y) # raises warning

As already mentioned, the key to communication is the readability of
the code and good formatting can really be a plus. Notice that in the
previous snippet we:

   * try to limit all lines to a maximum of 79 characters to avoid
     horizontal scrollbars in the code snippets blocks rendered on the
     GitHub issue;

   * use blank lines to separate groups of related functions;

   * place all the imports in their own group at the beginning.

The simplification steps presented in this guide can be implemented in
a different order than the progression we have shown here. The
important points are:

   * a minimal reproducer should be runnable by a simple copy-and-
     paste in a python terminal;

   * it should be simplified as much as possible by removing any code
     steps that are not strictly needed to reproducing the original
     problem;

   * it should ideally only rely on a minimal dataset generated on-
     the-fly by running the code instead of relying on external data,
     if possible.

Use markdown formatting

To format code or text into its own distinct block, use triple
backticks. Markdown supports an optional language identifier to enable
syntax highlighting in your fenced code block. For example:

   ```python
   from sklearn.datasets import make_blobs

   n_samples = 100
   n_components = 3
   X, y = make_blobs(n_samples=n_samples, centers=n_components)
   ```

will render a python formatted snippet as follows

```

It is not necessary to create several blocks of code when submitting a
bug report. Remember other reviewers are going to copy-paste your code
and having a single cell will make their task easier.

In the section named **Actual results** of the Issue template you are
asked to provide the error message including the full traceback of the
exception. In this case, use the *python-traceback* qualifier. For
example:

   ```python-traceback
   ---------------------------------------------------------------------------
   TypeError                                 Traceback (most recent call last)
   <ipython-input-1-a674e682c281> in <module>
       4 vectorizer = CountVectorizer(input=docs, analyzer='word')
       5 lda_features = vectorizer.fit_transform(docs)
   ----> 6 lda_model = LatentDirichletAllocation(
       7     n_topics=10,
       8     learning_method='online',

   TypeError: __init__() got an unexpected keyword argument 'n_topics'
   ```

yields the following when rendered:

<ipython-input-1-a674e682c281> in <module>

Synthetic dataset

Before choosing a particular synthetic dataset, first you have to
identify the type of problem you are solving: Is it a classification,
a regression, a clustering, etc?

Once that you narrowed down the type of problem, you need to provide a
synthetic dataset accordingly. Most of the times you only need a
minimalistic dataset. Here is a non-exhaustive list of tools that may
help you.

NumPy
-----

NumPy tools such as numpy.random.randn and numpy.random.randint can be
used to create dummy numeric data.

* regression

     Regressions take continuous numeric data as features and target.

        import numpy as np

        rng = np.random.RandomState(0)
        n_samples, n_features = 5, 5
        X = rng.randn(n_samples, n_features)
        y = rng.randn(n_samples)

A similar snippet can be used as synthetic data when testing scaling
tools such as "sklearn.preprocessing.StandardScaler".

* classification

     If the bug is not raised during when encoding a categorical
     variable, you can feed numeric data to a classifier. Just
     remember to ensure that the target is indeed an integer.

variable, you can feed numeric data to a classifier. Just

        y = rng.randint(0, 2, n_samples)  # binary target with values in {0, 1}

     If the bug only happens with non-numeric class labels, you might
     want to generate a random target with numpy.random.choice.

        n_samples, n_features = 50, 5
        y = np.random.choice(
            ["male", "female", "other"], size=n_samples, p=[0.49, 0.49, 0.02]
        )

Pandas
------

Some scikit-learn objects expect pandas dataframes as input. In this
case you can transform numpy arrays into pandas objects using
pandas.DataFrame, or pandas.Series.

   import numpy as np

   rng = np.random.RandomState(0)
   n_samples, n_features = 5, 5
   X = pd.DataFrame(
       {
           "continuous_feature": rng.randn(n_samples),
           "positive_feature": rng.uniform(low=0.0, high=100.0, size=n_samples),
           "categorical_feature": rng.choice(["a", "b", "c"], size=n_samples),
       }
   )
   y = pd.Series(rng.randn(n_samples))

In addition, scikit-learn includes various Generated datasets that can
be used to build artificial datasets of controlled size and

*make_regression*

As hinted by the name, "sklearn.datasets.make_regression" produces
regression targets with noise as an optionally-sparse random linear
combination of random features.

   from sklearn.datasets import make_regression

   X, y = make_regression(n_samples=1000, n_features=20)

*make_classification*

"sklearn.datasets.make_classification" creates multiclass datasets
with multiple Gaussian clusters per class. Noise can be introduced by
means of correlated, redundant or uninformative features.

   from sklearn.datasets import make_classification

   X, y = make_classification(
       n_features=2, n_redundant=0, n_informative=2, n_clusters_per_class=1
   )

*make_blobs*

Similarly to *make_classification*, "sklearn.datasets.make_blobs"
creates multiclass datasets using normally-distributed clusters of
points. It provides greater control regarding the centers and standard
deviations of each cluster, and therefore it is useful to demonstrate

   X, y = make_blobs(n_samples=10, centers=3, n_features=2)

Dataset loading utilities

You can use the Dataset loading utilities to load and fetch several
popular reference datasets. This option is useful when the bug relates
to the particular structure of the data, e.g. dealing with missing

values or image recognition.

   from sklearn.datasets import load_breast_cancer

   X, y = load_breast_cancer(return_X_y=True)

Developers' Tips and Tricks

Productivity and sanity-preserving tips

In this section we gather some useful advice and tools that may
increase your quality-of-life when reviewing pull requests, running
unit tests, and so forth. Some of these tricks consist of userscripts
that require a browser extension such as TamperMonkey or GreaseMonkey;
to set up userscripts you must have one of these extensions installed,
enabled and running.  We provide userscripts as GitHub gists; to
install them, click on the "Raw" button on the gist page.

Folding and unfolding outdated diffs on pull requests

GitHub hides discussions on PRs when the corresponding lines of code
have been changed in the mean while. This userscript provides a
shortcut (Control-Alt-P at the time of writing but look at the code to
be sure) to unfold all such hidden discussions at once, so you can
catch up.

Checking out pull requests as remote-tracking branches

In your local fork, add to your ".git/config", under the "[remote
"upstream"]" heading, the line:

   fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*

You may then use "git checkout pr/PR_NUMBER" to navigate to the code
of the pull-request with the given number. (Read more in this gist.)

Display code coverage in pull requests

To overlay the code coverage reports generated by the CodeCov
continuous integration, consider this browser extension. The coverage
of each line will be displayed as a color background behind the line
number.

Useful pytest aliases and flags

The full test suite takes fairly long to run. For faster iterations,
it is possibly to select a subset of tests using pytest selectors. In
particular, one can run a single test based on its node ID:

or use the -k pytest parameter to select tests based on their name.
For instance,:

will run all *common tests* for the "LogisticRegression" estimator.

When a unit test fails, the following tricks can make debugging
easier:

   1. The command line argument "pytest -l" instructs pytest to print
      the local variables when a failure occurs.

   2. The argument "pytest --pdb" drops into the Python debugger on
      failure. To instead drop into the rich IPython debugger "ipdb",
      you may set up a shell alias to:

Other *pytest* options that may become useful include:

   * "-x" which exits on the first failed test

   * "--lf" to rerun the tests that failed on the previous run

   * "--ff" to rerun all previous tests, running the ones that failed
     first

   * "-s" so that pytest does not capture the output of "print()"
     statements

* "--ff" to rerun all previous tests, running the ones that failed
     first

   * "--tb=short" or "--tb=line" to control the length of the logs

   * "--runxfail" also run tests marked as a known failure (XFAIL) and
     report errors.

Since our continuous integration tests will error if "FutureWarning"
isn't properly caught, it is also recommended to run "pytest" along
with the "-Werror::FutureWarning" flag.

Standard replies for reviewing

It may be helpful to store some of these in GitHub's saved replies for
reviewing:

Issue: Usage questions
      You are asking a usage question. The issue tracker is for bugs and new features. For usage questions, it is recommended to try [Stack Overflow](https://stackoverflow.com/questions/tagged/scikit-learn) or [the Mailing List](https://mail.python.org/mailman/listinfo/scikit-learn).

      Unfortunately, we need to close this issue as this issue tracker is a communication tool used for the development of scikit-learn. The additional activity created by usage questions crowds it too much and impedes this development. The conversation can continue here, however there is no guarantee that is will receive attention from core developers.

Issue: You're welcome to update the docs
      Please feel free to offer a pull request updating the documentation if you feel it could be improved.

Issue: Self-contained example for bug

      Please provide [self-contained example code](https://stackoverflow.com/help/mcve), including imports and data (if possible), so that other contributors can just run it and reproduce your issue. Ideally your example code should be minimal.

Issue: Software versions
      To help diagnose your issue, please paste the output of:
      ```py
      import sklearn; sklearn.show_versions()
      ```
      Thanks.

Issue: Code blocks
      Readability can be greatly improved if you [format](https://help.github.com/articles/creating-and-highlighting-code-blocks/) your code snippets and complete error messages appropriately. For example:

          ```python
          print(something)
          ```
      generates:
      ```python
      print(something)
      ```
      And:

          ```pytb
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          ImportError: No module named 'hello'
          ```
      ```pytb
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      ImportError: No module named 'hello'
      ```
      You can edit your issue descriptions and comments at any time to improve readability. This helps maintainers a lot. Thanks!

Issue/Comment: Linking to code

```

      Friendly advice: for clarity's sake, you can link to code like [this](https://help.github.com/articles/creating-a-permanent-link-to-a-code-snippet/).

Issue/Comment: Linking to comments
      Please use links to comments, which make it a lot easier to see what you are referring to, rather than just linking to the issue. See [this](https://stackoverflow.com/questions/25163598/how-do-i-reference-a-specific-issue-comment-on-github) for more details.

PR-NEW: Better description and title
      Thanks for the pull request! Please make the title of the PR more descriptive. The title will become the commit message when this is merged. You should state what issue (or PR) it fixes/resolves in the description using the syntax described [here](https://scikit-learn.org/dev/developers/contributing.html#contributing-pull-requests).

PR-NEW: Fix #
      Please use "Fix #issueNumber" in your PR description (and you can do it more than once). This way the associated issue gets closed automatically when the PR is merged. For more details, look at [this](https://github.com/blog/1506-closing-issues-via-pull-requests).

PR-NEW or Issue: Maintenance cost

      Every feature we include has a [maintenance cost](https://scikit-learn.org/dev/faq.html#why-are-you-so-selective-on-what-algorithms-you-include-in-scikit-learn). Our maintainers are mostly volunteers. For a new feature to be included, we need evidence that it is often useful and, ideally, [well-established](https://scikit-learn.org/dev/faq.html#what-are-the-inclusion-criteria-for-new-algorithms) in the literature or in practice. Also, we expect PR authors to take part in the maintenance for the code they submit, at least initially. That doesn't stop you implementing it for yourself and publishing it in a separate repository, or even [scikit-learn-contrib](https://scikit-learn-contrib.github.io).

PR-WIP: What's needed before merge?
      Please clarify (perhaps as a TODO list in the PR description) what work you believe still needs to be done before it can be reviewed for merge. When it is ready, please prefix the PR title with `[MRG]`.

PR-WIP: Regression test needed
      Please add a [non-regression test](https://en.wikipedia.org/wiki/Non-regression_testing) that would fail at main but pass in this PR.

PR-WIP: PEP8
      You have some [PEP8](https://www.python.org/dev/peps/pep-0008/) violations, whose details you can see in the Circle CI `lint` job. It might be worth configuring your code editor to check for such errors on the fly, so you can catch them before committing.

PR-MRG: Patience

      Before merging, we generally require two core developers to agree that your pull request is desirable and ready. [Please be patient](https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention), as we mostly rely on volunteered time from busy core developers. (You are also welcome to help us out with [reviewing other PRs](https://scikit-learn.org/dev/developers/contributing.html#code-review-guidelines).)

PR-MRG: Add to what's new
      Please add an entry to the change log at `doc/whats_new/v*.rst`. Like the other entries there, please reference this pull request with `:pr:` and credit yourself (and other contributors if applicable) with `:user:`.

PR: Don't change unrelated
      Please do not change unrelated lines. It makes your contribution harder to review and may introduce merge conflicts to other pull requests.

Debugging memory errors in Cython with valgrind

While python/numpy's built-in memory management is relatively robust,
it can lead to performance penalties for some routines. For this
reason, much of the high-performance code in scikit-learn is written
in cython. This performance gain comes with a tradeoff, however: it is
very easy for memory bugs to crop up in cython code, especially in
situations where that code relies heavily on pointer arithmetic.

Memory errors can manifest themselves a number of ways. The easiest

ones to debug are often segmentation faults and related glibc errors.
Uninitialized variables can lead to unexpected behavior that is
difficult to track down. A very useful tool when debugging these sorts
of errors is valgrind.

Valgrind is a command-line tool that can trace memory errors in a
variety of code. Follow these steps:

   1. Install valgrind on your system.

   2. Download the python valgrind suppression file: valgrind-
      python.supp.

   3. Follow the directions in the README.valgrind file to customize
      your python suppressions. If you don't, you will have spurious
      output coming related to the python interpreter instead of your
      own code.

   4. Run valgrind as follows:

The result will be a list of all the memory-related errors, which
reference lines in the C-code generated by cython from your .pyx file.
If you examine the referenced lines in the .c file, you will see
comments which indicate the corresponding location in your .pyx source
file. Hopefully the output will give you clues as to the source of
your memory error.

For more information on valgrind and the array of options it has, see
the tutorials and documentation on the valgrind web site.

Building and testing for the ARM64 platform on a x86_64 machine

ARM-based machines are a popular target for mobile, edge or other low-
energy deployments (including in the cloud, for instance on Scaleway
or AWS Graviton).

Here are instructions to setup a local dev environment to reproduce
ARM-specific bugs or test failures on a x86_64 host laptop or
workstation. This is based on QEMU user mode emulation using docker
for convenience (see https://github.com/multiarch/qemu-user-static).

Note:

  The following instructions are illustrated for ARM64 but they also
  apply to ppc64le, after changing the Docker image and Miniforge
  paths appropriately.

Prepare a folder on the host filesystem and download the necessary
tools and source code:

Use docker to install QEMU user mode and run an ARM64v8 container with
access to your shared folder under the */io* mount point:

In the container, install miniforge3 for the ARM64 (a.k.a. aarch64)
architecture:

Whenever you restart a new container, you will need to reinit the
conda env previously installed under */io/miniforge3*:

as the */root* home folder is part of the ephemeral docker container.
Every file or directory stored under */io* is persistent on the other
hand.

You can then build scikit-learn as usual (you will need to install
compiler tools and dependencies using apt or conda as usual). Building

hand.

scikit-learn takes a lot of time because of the emulation layer,
however it needs to be done only once if you put the scikit-learn
folder under the */io* mount point.

Then use pytest to run only the tests of the module you are interested
in debugging.

Maintainer / core-developer information

Releasing
=========

This section is about preparing a major release, incrementing the
minor version, or a bug fix release incrementing the patch version.
Our convention is that we release one or more release candidates
(0.RRrcN) before releasing the final distributions. We follow the
PEP101 to indicate release candidates, post, and minor releases.

Before a release

1. Update authors table:

   and commit. This is only needed if the authors have changed since
   the last release. This step is sometimes done independent of the
   release. This updates the maintainer list and is not the
   contributor list for the release.

2. Confirm any blockers tagged for the milestone are resolved, and
   that other issues tagged for the milestone can be postponed.

3. Ensure the change log and commits correspond (within reason!), and
   that the change log is reasonably well curated. Some tools for
   these tasks include:

   * "maint_tools/sort_whats_new.py" can put what's new entries into
     sections. It's not perfect, and requires manual checking of the
     changes. If the what's new list is well curated, it may not be
     necessary.

   * The "maint_tools/whats_missing.sh" script may be used to identify
     pull requests that were merged but likely missing from What's
     New.

4. Make sure the deprecations, FIXME and TODOs tagged for the release
   have been taken care of.

**Permissions**

pull requests that were merged but likely missing from What's
     New.

The release manager requires a set of permissions on top of the usual
permissions given to maintainers, which includes:

* *maintainer* role on "scikit-learn" projects on "pypi.org" and
  "test.pypi.org", separately.

* become a member of the *scikit-learn* team on conda-forge by editing
  the "recipe/meta.yaml" file on "https://github.com/conda-forge
  /scikit-learn-feedstock"

Preparing a release PR

Major version release

Prior to branching please do not forget to prepare a Release
Highlights page as a runnable example and check that its HTML
rendering looks correct. These release highlights should be linked
from the "doc/whats_new/v0.99.rst" file for the new version of scikit-
learn.

Releasing the first RC of e.g. version *0.99.0* involves creating the
release branch *0.99.X* directly on the main repo, where *X* really is
the letter X, **not a placeholder**. The development for the major and
minor releases of *0.99* should **also** happen under *0.99.X*. Each
release (rc, major, or minor) is a tag under that branch.

This is done only once, as the major and minor releases happen on the
same branch:

   Again, *X* is literal here, and *99* is replaced by the release
   number. The branches are called "0.19.X", "0.20.X", etc.

In terms of including changes, the first RC ideally counts as a
*feature freeze*. Each coming release candidate and the final release
afterwards will include only minor documentation changes and bug
fixes. Any major enhancement or feature should be excluded.

Then you can prepare a local branch for the release itself, for
instance: "release-0.99.0rc1", push it to your github fork and open a
PR **to the** *scikit-learn/0.99.X* **branch**. Copy the Release
checklist templates in the description of the Pull Request to track
progress.

This PR will be used to push commits related to the release as
explained in Making a release.

You can also create a second PR from main and targeting main to
increment the "__version__" variable in *sklearn/__init__.py* to
increment the dev version. This means while we're in the release
candidate period, the latest stable is two versions behind the main
branch, instead of one. In this PR targeting main you should also
include a new file for the matching version under the "doc/whats_new/"
folder so PRs that target the next version can contribute their
changelog entries to this file in parallel to the release process.

Minor version release

The minor releases should include bug fixes and some relevant
documentation changes only. Any PR resulting in a behavior change
which is not a bug fix should be excluded.

First, create a branch, **on your own fork** (to release e.g.
*0.99.3*):

Then, create a PR **to the** *scikit-learn/0.99.X* **branch** (not to
main!) with all the desired changes:

Copy the Release checklist templates in the description of the Pull
Request to track progress.

Do not forget to add a commit updating "sklearn.__version__".

It's nice to have a copy of the "git rebase -i" log in the PR to help
others understand what's included.

Making a release

* Ensure that you have checked out the branch of the release PR as
  explained in Preparing a release PR above.

* Update docs. Note that this is for the final release, not
  necessarily for the RC releases. These changes should be made in
  main and cherry-picked into the release branch, only before the
  final release.

  * Edit the "doc/whats_new/v0.99.rst" file to add release title and
    list of contributors. You can retrieve the list of contributor
    names with:

       $ git shortlog -s 0.98.33.. | cut -f2- | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s

    * For major releases, link the release highlights example from the
      "doc/whats_new/v0.99.rst" file.

  * Update the release date in "whats_new.rst"

  * Edit the "doc/templates/index.html" to change the 'News' entry of

"doc/whats_new/v0.99.rst" file.

    the front page (with the release month as well).

* On the branch for releasing, update the version number in
  "sklearn/__init__.py", the "__version__".

  For major releases, please add a 0 at the end: *0.99.0* instead of
  *0.99*.

  For the first release candidate, use the *rc1* suffix on the
  expected final release number: *0.99.0rc1*.

* Trigger the wheel builder with the "[cd build]" commit marker using
  the command:

  The wheel building workflow is managed by GitHub Actions and the
  results be browsed at: https://github.com/scikit-learn/scikit-
  learn/actions?query=workflow%3A%22Wheel+builder%22

Note:

  Before building the wheels, make sure that the "pyproject.toml" file
  is up to date and using the oldest version of "numpy" for each
  Python version to avoid ABI incompatibility issues. Moreover, a new
  line have to be included in the "pyproject.toml" file for each new
  supported version of Python.

Note:

  The acronym CD in *[cd build]* stands for Continuous Delivery and
  refers to the automation used to generate the release artifacts
  (binary and source packages). This can be seen as an extension to CI
  which stands for Continuous Integration. The CD workflow on GitHub
  Actions is also used to automatically create nightly builds and
  publish packages for the development branch of scikit-learn. See
  Installing nightly builds.

Actions is also used to automatically create nightly builds and

4. Once all the CD jobs have completed successfully in the PR, merge
   it, again with the *[cd build]* marker in the commit message. This
   time the results will be uploaded to the staging area.

   You should then be able to upload the generated artifacts (.tar.gz
   and .whl files) to https://test.pypi.org using the "Run workflow"
   form for the following GitHub Actions workflow:

   https://github.com/scikit-learn/scikit-
   learn/actions?query=workflow%3A%22Publish+to+Pypi%22

5. If this went fine, you can proceed with tagging. Proceed with
   caution. Ideally, tags should be created when you're almost certain
   that the release is ready, since adding a tag to the main repo can
   trigger certain automated processes.

   Create the tag and push it (if it's an RC, it can be "0.xx.0rc1"
   for instance):

6. Confirm that the bot has detected the tag on the conda-forge
   feedstock repo: https://github.com/conda-forge/scikit-learn-
   feedstock. If not, submit a PR for the release. If you want to
   publish an RC release on conda-forge, the PR should target the *rc*
   branch as opposed to the *main* branch. The two branches need to be
   kept sync together otherwise.

7. Trigger the GitHub Actions workflow again but this time to upload
   the artifacts to the real https://pypi.org (replace "testpypi" by

kept sync together otherwise.

   "pypi" in the "Run workflow" form).

8. **Alternative to step 7**: it's possible to collect locally the
   generated binary wheel packages and source tarball and upload them
   all to PyPI by running the following commands in the scikit-learn
   source folder (checked out at the release tag):

   This command will download all the binary packages accumulated in
   the staging area on the anaconda.org hosting service and put them
   in your local *./dist* folder.

   Check the content of the *./dist* folder: it should contain all the
   wheels along with the source tarball ("scikit-learn-RRR.tar.gz").

   Make sure that you do not have developer versions or older versions
   of the scikit-learn package in that folder.

   Before uploading to pypi, you can test upload to test.pypi.org:

   Upload everything at once to https://pypi.org:

9. For major/minor (not bug-fix release or release candidates), update
   the symlink for "stable" and the "latestStable" variable in
   https://github.com/scikit-learn/scikit-learn.github.io:

10. Update "SECURITY.md" to reflect the latest supported version.

Release checklist

The following GitHub checklist might be helpful in a release PR:

   * [ ] update news and what's new date in release branch
   * [ ] update news and what's new date and sklearn dev0 version in main branch

* [ ] update news and what's new date in release branch
   * [ ] check that the wheels for the release can be built successfully
   * [ ] merge the PR with `[cd build]` commit message to upload wheels to the staging repo
   * [ ] upload the wheels and source tarball to https://test.pypi.org
   * [ ] create tag on the main github repo
   * [ ] confirm bot detected at
     https://github.com/conda-forge/scikit-learn-feedstock and wait for merge
   * [ ] upload the wheels and source tarball to PyPI
   * [ ] https://github.com/scikit-learn/scikit-learn/releases publish (except for RC)
   * [ ] announce on mailing list and on Twitter, and LinkedIn
   * [ ] update symlink for stable in
     https://github.com/scikit-learn/scikit-learn.github.io (only major/minor)
   * [ ] update SECURITY.md in main branch (except for RC)

Merging Pull Requests

Individual commits are squashed when a Pull Request (PR) is merged on
Github. Before merging,

* the resulting commit title can be edited if necessary. Note that
  this will rename the PR title by default.

* the detailed description, containing the titles of all the commits,
  can be edited or deleted.

* for PRs with multiple code contributors care must be taken to keep
  the *Co-authored-by: name <name@example.com>* tags in the detailed
  description. This will mark the PR as having multiple co-authors.

the *Co-authored-by: name <name@example.com>* tags in the detailed
  Whether code contributions are significanly enough to merit co-
  authorship is left to the maintainer's discretion, same as for the
  "what's new" entry.

The scikit-learn.org web site

The scikit-learn web site (https://scikit-learn.org) is hosted at
GitHub, but should rarely be updated manually by pushing to the
https://github.com/scikit-learn/scikit-learn.github.io repository.
Most updates can be made by pushing to master (for /dev) or a release
branch like 0.99.X, from which Circle CI builds and uploads the
documentation automatically.

Travis Cron jobs

From https://docs.travis-ci.com/user/cron-jobs: Travis CI cron jobs
work similarly to the cron utility, they run builds at regular
scheduled intervals independently of whether any commits were pushed
to the repository. Cron jobs always fetch the most recent commit on a
particular branch and build the project at that state. Cron jobs can
run daily, weekly or monthly, which in practice means up to an hour
after the selected time span, and you cannot set them to run at a
specific time.

For scikit-learn, Cron jobs are used for builds that we do not want to
run in each PR. As an example the build with the dev versions of numpy
and scipy is run as a Cron job. Most of the time when this numpy-dev
build fail, it is related to a numpy change and not a scikit-learn

one, so it would not make sense to blame the PR author for the Travis
failure.

The definition of what gets run in the Cron job is done in the
.travis.yml config file, exactly the same way as the other Travis
jobs. We use a "if: type = cron" filter in order for the build to be
run only in Cron jobs.

The branch targeted by the Cron job and the frequency of the Cron job
is set via the web UI at https://www.travis-ci.org/scikit-learn
/scikit-learn/settings.

Experimental features

The "sklearn.experimental" module was introduced in 0.21 and contains
experimental features / estimators that are subject to change without
deprecation cycle.

To create an experimental module, you can just copy and modify the
content of enable_hist_gradient_boosting.py, or
enable_iterative_imputer.py.

Note:

  These are permalink as in 0.24, where these estimators are still
  experimental. They might be stable at the time of reading - hence
  the permalink. See below for instructions on the transition from
  experimental to stable.

Note that the public import path must be to a public subpackage (like
"sklearn/ensemble" or "sklearn/impute"), not just a ".py" module.
Also, the (private) experimental features that are imported must be in
a submodule/subpackage of the public subpackage, e.g.
"sklearn/ensemble/_hist_gradient_boosting/" or

"sklearn/impute/_iterative.py". This is needed so that pickles still
work in the future when the features aren't experimental anymore.

To avoid type checker (e.g. mypy) errors a direct import of
experimental estimators should be done in the parent module, protected
by the "if typing.TYPE_CHECKING" check. See
sklearn/ensemble/__init__.py, or sklearn/impute/__init__.py for an
example.

Please also write basic tests following those in
test_enable_hist_gradient_boosting.py.

Make sure every user-facing code you write explicitly mentions that
the feature is experimental, and add a "# noqa" comment to avoid
pep8-related warnings:

   # To use this experimental feature, we need to explicitly ask for it:
   from sklearn.experimental import enable_hist_gradient_boosting  # noqa
   from sklearn.ensemble import HistGradientBoostingRegressor

For the docs to render properly, please also import
"enable_my_experimental_feature" in "doc/conf.py", else sphinx won't
be able to import the corresponding modules. Note that using "from
sklearn.experimental import *" **does not work**.

Note that some experimental classes / functions are not included in
the "sklearn.experimental" module: "sklearn.datasets.fetch_openml".

Once the feature become stable, remove all
*enable_my_experimental_feature* in the scikit-learn code (even

feature highlights etc.) and make the *enable_my_experimental_feature*
a no-op that just raises a warning: enable_hist_gradient_boosting.py.
The file should stay there indefinitely as we don't want to break
users code: we just incentivize them to remove that import with the
warning.

Also update the tests accordingly:

Developing scikit-learn estimators

Whether you are proposing an estimator for inclusion in scikit-learn,
developing a separate package compatible with scikit-learn, or
implementing custom components for your own projects, this chapter
details how to develop objects that safely interact with scikit-learn
Pipelines and model selection tools.

APIs of scikit-learn objects

To have a uniform API, we try to have a common basic API for all the
objects. In addition, to avoid the proliferation of framework code, we
try to adopt simple conventions and limit to a minimum the number of
methods an object must implement.

Elements of the scikit-learn API are described more definitively in
the Glossary of Common Terms and API Elements.

Different objects

The main objects in scikit-learn are (one class can implement multiple
interfaces):

Estimator:
   The base object, implements a "fit" method to learn from data,
   either:

      estimator = estimator.fit(data, targets)

   or:

      estimator = estimator.fit(data)

Predictor:
   For supervised learning, or some unsupervised problems, implements:

      prediction = predictor.predict(data)

   Classification algorithms usually also offer a way to quantify
   certainty of a prediction, either using "decision_function" or
   "predict_proba":

      probability = predictor.predict_proba(data)

Transformer:
   For filtering or modifying the data, in a supervised or
   unsupervised way, implements:

"predict_proba":

      new_data = transformer.transform(data)

   When fitting and transforming can be performed much more
   efficiently together than separately, implements:

      new_data = transformer.fit_transform(data)

Model:
   A model that can give a goodness of fit measure or a likelihood of
   unseen data, implements (higher is better):

      score = model.score(data)

The API has one predominant object: the estimator. An estimator is an
object that fits a model based on some training data and is capable of
inferring some properties on new data. It can be, for instance, a
classifier or a regressor. All estimators implement the fit method:

   estimator.fit(X, y)

All built-in estimators also have a "set_params" method, which sets
data-independent parameters (overriding previous parameter values
passed to "__init__").

All estimators in the main scikit-learn codebase should inherit from
"sklearn.base.BaseEstimator".

Instantiation

This concerns the creation of an object. The object's "__init__"
method might accept constants as arguments that determine the
estimator's behavior (like the C constant in SVMs). It should not,
however, take the actual training data as an argument, as this is left
to the "fit()" method:

   clf2 = SVC(C=2.3)
   clf3 = SVC([[1, 2], [2, 3]], [-1, 1]) # WRONG!

The arguments accepted by "__init__" should all be keyword arguments
with a default value. In other words, a user should be able to
instantiate an estimator without passing any arguments to it. The
arguments should all correspond to hyperparameters describing the
model or the optimisation problem the estimator tries to solve. These
initial arguments (or parameters) are always remembered by the
estimator. Also note that they should not be documented under the
"Attributes" section, but rather under the "Parameters" section for
that estimator.

In addition, **every keyword argument accepted by** "__init__"
**should correspond to an attribute on the instance**. Scikit-learn
relies on this to find the relevant attributes to set on an estimator
when doing model selection.

To summarize, an "__init__" should look like:

   def __init__(self, param1=1, param2=2):
       self.param1 = param1
       self.param2 = param2

There should be no logic, not even input validation, and the
parameters should not be changed. The corresponding logic should be
put where the parameters are used, typically in "fit". The following
is wrong:

   def __init__(self, param1=1, param2=2, param3=3):
       # WRONG: parameters should not be modified
       if param1 > 1:
           param2 += 1

is wrong:

           param2 += 1
       # WRONG: the object's attributes should have exactly the name of
       # the argument in the constructor
       self.param3 = param2

The reason for postponing the validation is that the same validation
would have to be performed in "set_params", which is used in
algorithms like "GridSearchCV".

Fitting
~~~~~~~

The next thing you will probably want to do is to estimate some
parameters in the model. This is implemented in the "fit()" method.

The "fit()" method takes the training data as arguments, which can be
one array in the case of unsupervised learning, or two arrays in the
case of supervised learning.

Note that the model is fitted using "X" and "y", but the object holds
no reference to "X" and "y". There are, however, some exceptions to
this, as in the case of precomputed kernels where this data must be
stored for use by the predict method.

+---------------+--------------------------------------------------------+
| Parameters    |                                                        |
|===============|========================================================|
| X             | array-like of shape (n_samples, n_features)            |
| y             | array-like of shape (n_samples,)                       |

| kwargs        | optional data-dependent parameters                     |

"X.shape[0]" should be the same as "y.shape[0]". If this requisite is
not met, an exception of type "ValueError" should be raised.

"y" might be ignored in the case of unsupervised learning. However, to
make it possible to use the estimator as part of a pipeline that can
mix both supervised and unsupervised transformers, even unsupervised
estimators need to accept a "y=None" keyword argument in the second
position that is just ignored by the estimator. For the same reason,
"fit_predict", "fit_transform", "score" and "partial_fit" methods need
to accept a "y" argument in the second place if they are implemented.

The method should return the object ("self"). This pattern is useful
to be able to implement quick one liners in an IPython session such
as:

   y_predicted = SVC(C=100).fit(X_train, y_train).predict(X_test)

Depending on the nature of the algorithm, "fit" can sometimes also
accept additional keywords arguments. However, any parameter that can
have a value assigned prior to having access to the data should be an
"__init__" keyword argument. **fit parameters should be restricted to

directly data dependent variables**. For instance a Gram matrix or an
affinity matrix which are precomputed from the data matrix "X" are
data dependent. A tolerance stopping criterion "tol" is not directly
data dependent (although the optimal value according to some scoring
function probably is).

When "fit" is called, any previous call to "fit" should be ignored. In
general, calling "estimator.fit(X1)" and then "estimator.fit(X2)"
should be the same as only calling "estimator.fit(X2)". However, this
may not be true in practice when "fit" depends on some random process,
see *random_state*. Another exception to this rule is when the hyper-
parameter "warm_start" is set to "True" for estimators that support
it. "warm_start=True" means that the previous state of the trainable
parameters of the estimator are reused instead of using the default
initialization strategy.

Estimated Attributes

Attributes that have been estimated from the data must always have a
name ending with trailing underscore, for example the coefficients of
some regression estimator would be stored in a "coef_" attribute after
"fit" has been called.

The estimated attributes are expected to be overridden when you call
"fit" a second time.

Optional Arguments

In iterative algorithms, the number of iterations should be specified
by an integer called "n_iter".

Universal attributes

Estimators that expect tabular input should set a *n_features_in_*
attribute at *fit* time to indicate the number of features that the
estimator expects for subsequent calls to *predict* or *transform*.
See SLEP010 for details.

Rolling your own estimator

If you want to implement a new estimator that is scikit-learn-
compatible, whether it is just for you or for contributing it to
scikit-learn, there are several internals of scikit-learn that you
should be aware of in addition to the scikit-learn API outlined above.
You can check whether your estimator adheres to the scikit-learn
interface and standards by running "check_estimator()" on an instance.
The "parametrize_with_checks()" pytest decorator can also be used (see
its docstring for details and possible interactions with *pytest*):

   >>> from sklearn.utils.estimator_checks import check_estimator
   >>> check_estimator(LinearSVC())  # passes

The main motivation to make a class compatible to the scikit-learn
estimator interface might be that you want to use it together with
model evaluation and selection tools such as
"model_selection.GridSearchCV" and "pipeline.Pipeline".

Before detailing the required interface below, we describe two ways to
achieve the correct interface more easily.

Project template:

We provide a project template which helps in the creation of Python
packages containing scikit-learn compatible estimators. It provides:

* an initial git repository with Python package directory structure

* a template of a scikit-learn estimator

* an initial test suite including use of "check_estimator"

* directory structures and scripts to compile documentation and
  example galleries

* scripts to manage continuous integration (testing on Linux and
  Windows)

* instructions from getting started to publishing on PyPi

"BaseEstimator" and mixins:

We tend to use "duck typing", so building an estimator which follows
the API suffices for compatibility, without needing to inherit from or
even import any scikit-learn classes.

However, if a dependency on scikit-learn is acceptable in your code,
you can prevent a lot of boilerplate code by deriving a class from
"BaseEstimator" and optionally the mixin classes in "sklearn.base".
For example, below is a custom classifier, with more examples included
in the scikit-learn-contrib project template.

>>> from sklearn.base import BaseEstimator, ClassifierMixin
>>> from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
>>> from sklearn.utils.multiclass import unique_labels

>>> from sklearn.metrics import euclidean_distances
>>> class TemplateClassifier(BaseEstimator, ClassifierMixin):
...
...     def __init__(self, demo_param='demo'):
...         self.demo_param = demo_param
...
...     def fit(self, X, y):
...
...         # Check that X and y have correct shape
...         X, y = check_X_y(X, y)
...         # Store the classes seen during fit
...         self.classes_ = unique_labels(y)
...
...         self.X_ = X
...         self.y_ = y
...         # Return the classifier
...         return self
...
...     def predict(self, X):
...
...         # Check if fit has been called
...         check_is_fitted(self)
...
...         # Input validation
...         X = check_array(X)
...
...         closest = np.argmin(euclidean_distances(X, self.X_), axis=1)
...         return self.y_[closest]

get_params and set_params

All scikit-learn estimators have "get_params" and "set_params"
functions. The "get_params" function takes no arguments and returns a
dict of the "__init__" parameters of the estimator, together with
their values.

It must take one keyword argument, "deep", which receives a boolean
value that determines whether the method should return the parameters
of sub-estimators (for most estimators, this can be ignored). The

default value for "deep" should be *True*. For instance considering
the following estimator:

   >>> from sklearn.base import BaseEstimator
   >>> class MyEstimator(BaseEstimator):
   ...     def __init__(self, subestimator=None, my_extra_param="random"):
   ...         self.subestimator = subestimator
   ...         self.my_extra_param = my_extra_param

The parameter *deep* will control whether or not the parameters of the
*subestimator* should be reported. Thus when *deep=True*, the output
will be:

   >>> my_estimator = MyEstimator(subestimator=LogisticRegression())
   >>> for param, value in my_estimator.get_params(deep=True).items():
   ...     print(f"{param} -> {value}")
   my_extra_param -> random
   subestimator__C -> 1.0
   subestimator__class_weight -> None
   subestimator__dual -> False
   subestimator__fit_intercept -> True
   subestimator__intercept_scaling -> 1
   subestimator__l1_ratio -> None
   subestimator__max_iter -> 100
   subestimator__multi_class -> auto
   subestimator__n_jobs -> None
   subestimator__penalty -> l2
   subestimator__random_state -> None
   subestimator__solver -> lbfgs
   subestimator__tol -> 0.0001
   subestimator__verbose -> 0
   subestimator__warm_start -> False
   subestimator -> LogisticRegression()

Often, the *subestimator* has a name (as e.g. named steps in a

subestimator__tol -> 0.0001

"Pipeline" object), in which case the key should become *<name>__C*,
*<name>__class_weight*, etc.

While when *deep=False*, the output will be:

   >>> for param, value in my_estimator.get_params(deep=False).items():

On the other hand, "set_params" takes the parameters of "__init__" as
keyword arguments, unpacks them into a dict of the form "'parameter':
value" and sets the parameters of the estimator using this dict.
Return value must be the estimator itself.

While the "get_params" mechanism is not essential (see Cloning below),
the "set_params" function is necessary as it is used to set parameters
during grid searches.

The easiest way to implement these functions, and to get a sensible
"__repr__" method, is to inherit from "sklearn.base.BaseEstimator". If
you do not want to make your code dependent on scikit-learn, the
easiest way to implement the interface is:

   def get_params(self, deep=True):
       # suppose this estimator has parameters "alpha" and "recursive"
       return {"alpha": self.alpha, "recursive": self.recursive}

   def set_params(self, **parameters):
       for parameter, value in parameters.items():
           setattr(self, parameter, value)
       return self

Parameters and init

def set_params(self, **parameters):

As "model_selection.GridSearchCV" uses "set_params" to apply parameter
setting to estimators, it is essential that calling "set_params" has
the same effect as setting parameters using the "__init__" method. The
easiest and recommended way to accomplish this is to **not do any
parameter validation in** "__init__". All logic behind estimator
parameters, like translating string arguments into functions, should
be done in "fit".

Also it is expected that parameters with trailing "_" are **not to be
set inside the** "__init__" **method**. All and only the public
attributes set by fit have a trailing "_". As a result the existence
of parameters with trailing "_" is used to check if the estimator has
been fitted.

Cloning
-------

For use with the "model_selection" module, an estimator must support
the "base.clone" function to replicate an estimator. This can be done
by providing a "get_params" method. If "get_params" is present, then
"clone(estimator)" will be an instance of "type(estimator)" on which
"set_params" has been called with clones of the result of
"estimator.get_params()".

Objects that do not provide this method will be deep-copied (using the
Python standard function "copy.deepcopy") if "safe=False" is passed to
"clone".

Estimators can customize the behavior of "base.clone()" by defining a

"clone".

*__sklearn_clone__* method. *__sklearn_clone__* must return an
instance of the estimator. *__sklearn_clone__* is useful when an
estimator needs to hold on to some state when "base.clone()" is called
on the estimator. For example, a frozen meta-estimator for
transformers can be defined as follows:

   class FrozenTransformer(BaseEstimator):
       def __init__(self, fitted_transformer):
           self.fitted_transformer = fitted_transformer

       def __getattr__(self, name):
           # `fitted_transformer`'s attributes are now accessible
           return getattr(self.fitted_transformer, name)

       def __sklearn_clone__(self):
           return self

           # Fitting does not change the state of the estimator

       def fit_transform(self, X, y=None):
           # fit_transform only transforms the data
           return self.fitted_transformer.transform(X, y)

Pipeline compatibility

For an estimator to be usable together with "pipeline.Pipeline" in any
but the last step, it needs to provide a "fit" or "fit_transform"
function. To be able to evaluate the pipeline on any data but the
training set, it also needs to provide a "transform" function. There
are no special requirements for the last step in a pipeline, except

that it has a "fit" function. All "fit" and "fit_transform" functions
must take arguments "X, y", even if y is not used. Similarly, for
"score" to be usable, the last step of the pipeline needs to have a
"score" function that accepts an optional "y".

Estimator types

Some common functionality depends on the kind of estimator passed. For
example, cross-validation in "model_selection.GridSearchCV" and
"model_selection.cross_val_score()" defaults to being stratified when
used on a classifier, but not otherwise. Similarly, scorers for
average precision that take a continuous prediction need to call
"decision_function" for classifiers, but "predict" for regressors.
This distinction between classifiers and regressors is implemented
using the "_estimator_type" attribute, which takes a string value. It
should be ""classifier"" for classifiers and ""regressor"" for
regressors and ""clusterer"" for clustering methods, to work as
expected. Inheriting from "ClassifierMixin", "RegressorMixin" or
"ClusterMixin" will set the attribute automatically.  When a meta-
estimator needs to distinguish among estimator types, instead of
checking "_estimator_type" directly, helpers like
"base.is_classifier()" should be used.

Specific models

Classifiers should accept "y" (target) arguments to "fit" that are
sequences (lists, arrays) of either strings or integers.  They should
not assume that the class labels are a contiguous range of integers;
instead, they should store a list of classes in a "classes_" attribute
or property.  The order of class labels in this attribute should match
the order in which "predict_proba", "predict_log_proba" and
"decision_function" return their values.  The easiest way to achieve
this is to put:

   self.classes_, y = np.unique(y, return_inverse=True)

in "fit".  This returns a new "y" that contains class indexes, rather
than labels, in the range [0, "n_classes").

A classifier's "predict" method should return arrays containing class
labels from "classes_". In a classifier that implements
"decision_function", this can be achieved with:

   def predict(self, X):
       D = self.decision_function(X)
       return self.classes_[np.argmax(D, axis=1)]

In linear models, coefficients are stored in an array called "coef_",
and the independent term is stored in "intercept_".
"sklearn.linear_model._base" contains a few base classes and mixins
that implement common linear model patterns.

The "sklearn.utils.multiclass" module contains useful functions for
working with multiclass and multilabel problems.

Estimator Tags

Warning:

Warning:

  The estimator tags are experimental and the API is subject to
  change.

Scikit-learn introduced estimator tags in version 0.21. These are
annotations of estimators that allow programmatic inspection of their
capabilities, such as sparse matrix support, supported output types
and supported methods. The estimator tags are a dictionary returned by
the method "_get_tags()". These tags are used in the common checks run
by the "check_estimator()" function and the
"parametrize_with_checks()" decorator. Tags determine which checks to
run and what input data is appropriate. Tags can depend on estimator
parameters or even system architecture and can in general only be
determined at runtime.

The current set of estimator tags are:

allow_nan (default=False)
   whether the estimator supports data with missing values encoded as
   np.NaN

binary_only (default=False)
   whether estimator supports binary classification but lacks multi-
   class classification support.

multilabel (default=False)
   whether the estimator supports multilabel output

multioutput (default=False)
   whether a regressor supports multi-target outputs or a classifier
   supports multi-class multi-output.

multioutput_only (default=False)
   whether estimator supports only multi-output classification or
   regression.

no_validation (default=False)

supports multi-class multi-output.

   whether the estimator skips input-validation. This is only meant
   for stateless and dummy transformers!

non_deterministic (default=False)
   whether the estimator is not deterministic given a fixed
   "random_state"

pairwise (default=False)
   This boolean attribute indicates whether the data (*X*) *fit* and
   similar methods consists of pairwise measures over samples rather
   than a feature representation for each sample.  It is usually
   *True* where an estimator has a *metric* or *affinity* or *kernel*
   parameter with value 'precomputed'. Its primary purpose is to
   support a *meta-estimator* or a cross validation procedure that
   extracts a sub-sample of data intended for a pairwise estimator,
   where the data needs to be indexed on both axes. Specifically, this
   tag is used by "_safe_split()" to slice rows and columns.

preserves_dtype (default=``[np.float64]``)
   applies only on transformers. It corresponds to the data types
   which will be preserved such that *X_trans.dtype* is the same as
   *X.dtype* after calling *transformer.transform(X)*. If this list is
   empty, then the transformer is not expected to preserve the data
   type. The first value in the list is considered as the default data
   type, corresponding to the data type of the output when the input

type. The first value in the list is considered as the default data
   data type is not going to be preserved.

poor_score (default=False)
   whether the estimator fails to provide a "reasonable" test-set
   score, which currently for regression is an R2 of 0.5 on
   "make_regression(n_samples=200, n_features=10, n_informative=1,
   bias=5.0, noise=20, random_state=42)", and for classification an
   accuracy of 0.83 on "make_blobs(n_samples=300, random_state=0)".
   These datasets and values are based on current estimators in
   sklearn and might be replaced by something more systematic.

requires_fit (default=True)
   whether the estimator requires to be fitted before calling one of
   *transform*, *predict*, *predict_proba*, or *decision_function*.

requires_positive_X (default=False)
   whether the estimator requires positive X.

requires_y (default=False)
   whether the estimator requires y to be passed to *fit*,
   *fit_predict* or *fit_transform* methods. The tag is True for
   estimators inheriting from *~sklearn.base.RegressorMixin* and
   *~sklearn.base.ClassifierMixin*.

requires_positive_y (default=False)
   whether the estimator requires a positive y (only applicable for
   regression).

_skip_test (default=False)
   whether to skip common tests entirely. Don't use this unless you
   have a *very good* reason.

_xfail_checks (default=False)
   dictionary "{check_name: reason}" of common checks that will be

whether to skip common tests entirely. Don't use this unless you

   marked as *XFAIL* for pytest, when using
   "parametrize_with_checks()". These checks will be simply ignored
   and not run by "check_estimator()", but a *SkipTestWarning* will be
   raised. Don't use this unless there is a *very good* reason for
   your estimator not to pass the check. Also note that the usage of
   this tag is highly subject to change because we are trying to make
   it more flexible: be prepared for breaking changes in the future.

stateless (default=False)
   whether the estimator needs access to data for fitting. Even though
   an estimator is stateless, it might still need a call to "fit" for
   initialization.

X_types (default=['2darray'])
   Supported input types for X as list of strings. Tests are currently
   only run if '2darray' is contained in the list, signifying that the
   estimator takes continuous 2d numpy arrays as input. The default
   value is ['2darray']. Other possible types are "'string'",
   "'sparse'", "'categorical'", "dict", "'1dlabels'" and "'2dlabels'".
   The goal is that in the future the supported input type will
   determine the data used during testing, in particular for
   "'string'", "'sparse'" and "'categorical'" data. For now, the test
   for sparse data do not make use of the "'sparse'" tag.

determine the data used during testing, in particular for

It is unlikely that the default values for each tag will suit the
needs of your specific estimator. Additional tags can be created or
default tags can be overridden by defining a *_more_tags()* method
which returns a dict with the desired overridden tags or new tags. For
example:

   class MyMultiOutputEstimator(BaseEstimator):

       def _more_tags(self):
           return {'multioutput_only': True,
                   'non_deterministic': True}

Any tag that is not in *_more_tags()* will just fall-back to the
default values documented above.

Even if it is not recommended, it is possible to override the method
*_get_tags()*. Note however that **all tags must be present in the
dict**. If any of the keys documented above is not present in the
output of *_get_tags()*, an error will occur.

In addition to the tags, estimators also need to declare any non-
optional parameters to "__init__" in the "_required_parameters" class
attribute, which is a list or tuple.  If "_required_parameters" is
only "["estimator"]" or "["base_estimator"]", then the estimator will
be instantiated with an instance of "LogisticRegression" (or
"RidgeRegression" if the estimator is a regressor) in the tests. The
choice of these two models is somewhat idiosyncratic but both should
provide robust closed-form solutions.

Developer API for *set_output*

With SLEP018, scikit-learn introduces the *set_output* API for
configuring transformers to output pandas DataFrames. The *set_output*
API is automatically defined if the transformer defines
*get_feature_names_out* and subclasses "base.TransformerMixin".
*get_feature_names_out* is used to get the column names of pandas
output.

"base.OneToOneFeatureMixin" and "base.ClassNamePrefixFeaturesOutMixin"
are helpful mixins for defining *get_feature_names_out*.
"base.OneToOneFeatureMixin" is useful when the transformer has a one-
to-one correspondence between input features and output features, such
as "StandardScaler". "base.ClassNamePrefixFeaturesOutMixin" is useful
when the transformer needs to generate its own feature names out, such
as "PCA".

You can opt-out of the *set_output* API by setting
*auto_wrap_output_keys=None* when defining a custom subclass:

   class MyTransformer(TransformerMixin, BaseEstimator, auto_wrap_output_keys=None):

       def fit(self, X, y=None):
       def transform(self, X, y=None):
           return X
       def get_feature_names_out(self, input_features=None):
           ...

The default value for *auto_wrap_output_keys* is *("transform",)*,
which automatically wraps *fit_transform* and *transform*. The
*TransformerMixin* uses the *__init_subclass__* mechanism to consume

*auto_wrap_output_keys* and pass all other keyword arguments to it's
super class. Super classes' *__init_subclass__* should **not** depend
on *auto_wrap_output_keys*.

For transformers that return multiple arrays in *transform*, auto
wrapping will only wrap the first array and not alter the other
arrays.

See *sphx_glr_auto_examples_miscellaneous_plot_set_output.py* for an
example on how to use the API.

Coding guidelines

The following are some guidelines on how new code should be written
for inclusion in scikit-learn, and which may be appropriate to adopt
in external projects. Of course, there are special cases and there
will be exceptions to these rules. However, following these rules when
submitting new code makes the review easier so new code can be
integrated in less time.

Uniformly formatted code makes it easier to share code ownership. The
scikit-learn project tries to closely follow the official Python
guidelines detailed in PEP8 that detail how code should be formatted
and indented. Please read it and follow it.

In addition, we add the following guidelines:

* Use underscores to separate words in non class names: "n_samples"
  rather than "nsamples".

* Avoid multiple statements on one line. Prefer a line return after a
  control flow statement ("if"/"for").

* Use relative imports for references inside scikit-learn.

* Unit tests are an exception to the previous rule; they should use
  absolute imports, exactly as client code would. A corollary is that,
  if "sklearn.foo" exports a class or function that is implemented in
  "sklearn.foo.bar.baz", the test should import it from "sklearn.foo".

* **Please don't use** "import *" **in any case**. It is considered
  harmful by the official Python recommendations. It makes the code
  harder to read as the origin of symbols is no longer explicitly
  referenced, but most important, it prevents using a static analysis
  tool like pyflakes to automatically find bugs in scikit-learn.

* Use the numpy docstring standard in all your docstrings.

A good example of code that we like can be found here.

Input validation

The module "sklearn.utils" contains various functions for doing input
validation and conversion. Sometimes, "np.asarray" suffices for
validation; do *not* use "np.asanyarray" or "np.atleast_2d", since
those let NumPy's "np.matrix" through, which has a different API
(e.g., "*" means dot product on "np.matrix", but Hadamard product on
"np.ndarray").

In other cases, be sure to call "check_array()" on any array-like
argument passed to a scikit-learn API function. The exact parameters

to use depends mainly on whether and which "scipy.sparse" matrices
must be accepted.

For more information, refer to the Utilities for Developers page.

Random Numbers

If your code depends on a random number generator, do not use
"numpy.random.random()" or similar routines.  To ensure repeatability
in error checking, the routine should accept a keyword "random_state"
and use this to construct a "numpy.random.RandomState" object. See
"sklearn.utils.check_random_state()" in Utilities for Developers.

Here's a simple example of code using some of the above guidelines:

   from sklearn.utils import check_array, check_random_state

   def choose_random_sample(X, random_state=0):
       """Choose a random point from X.

       Parameters
       ----------
       X : array-like of shape (n_samples, n_features)
           An array representing the data.
       random_state : int or RandomState instance, default=0
           The seed of the pseudo random number generator that selects a
           random sample. Pass an int for reproducible output across multiple
           function calls.
           See :term:`Glossary <random_state>`.

       Returns
       -------
       x : ndarray of shape (n_features,)
           A random point selected from X.
       """
       X = check_array(X)
       random_state = check_random_state(random_state)

Returns
       -------
       """
       i = random_state.randint(X.shape[0])
       return X[i]

If you use randomness in an estimator instead of a freestanding
function, some additional guidelines apply.

First off, the estimator should take a "random_state" argument to its
"__init__" with a default value of "None". It should store that
argument's value, **unmodified**, in an attribute "random_state".
"fit" can call "check_random_state" on that attribute to get an actual
random number generator. If, for some reason, randomness is needed
after "fit", the RNG should be stored in an attribute "random_state_".
The following example should make this clear:

   class GaussianNoise(BaseEstimator, TransformerMixin):
       """This estimator ignores its input and returns random Gaussian noise.

       It also does not adhere to all scikit-learn conventions,
       but showcases how to handle randomness.
       """

       def __init__(self, n_components=100, random_state=None):
           self.random_state = random_state
           self.n_components = n_components

       # the arguments are ignored anyway, so we make them optional
       def fit(self, X=None, y=None):
           self.random_state_ = check_random_state(self.random_state)

       def transform(self, X):
           n_samples = X.shape[0]

def fit(self, X=None, y=None):

           return self.random_state_.randn(n_samples, self.n_components)

The reason for this setup is reproducibility: when an estimator is
"fit" twice to the same data, it should produce an identical model
both times, hence the validation in "fit", not "__init__".

Numerical assertions in tests

When asserting the quasi-equality of arrays of continuous values, do
use "sklearn.utils._testing.assert_allclose()".

The relative tolerance is automatically inferred from the provided
arrays dtypes (for float32 and float64 dtypes in particular) but you
can override via "rtol".

When comparing arrays of zero-elements, please do provide a non-zero
value for the absolute tolerance via "atol".

For more information, please refer to the docstring of
"sklearn.utils._testing.assert_allclose()".

Utilities for Developers

Scikit-learn contains a number of utilities to help with development.
These are located in "sklearn.utils", and include tools in a number of
categories. All the following functions and classes are in the module
"sklearn.utils".

Warning:

  These utilities are meant to be used internally within the scikit-
  learn package.  They are not guaranteed to be stable between
  versions of scikit-learn.  Backports, in particular, will be removed
  as the scikit-learn dependencies evolve.

Validation Tools

These are tools used to check and validate input.  When you write a
function which accepts arrays, matrices, or sparse matrices as
arguments, the following should be used when applicable.

* "assert_all_finite()": Throw an error if array contains NaNs or
  Infs.

* "as_float_array()": convert input to an array of floats.  If a
  sparse matrix is passed, a sparse matrix will be returned.

* "check_array()": check that input is a 2D array, raise error on
  sparse matrices. Allowed sparse matrix formats can be given
  optionally, as well as allowing 1D or N-dimensional arrays. Calls
  "assert_all_finite()" by default.

* "check_X_y()": check that X and y have consistent length, calls
  check_array on X, and column_or_1d on y. For multilabel
  classification or multitarget regression, specify multi_output=True,
  in which case check_array will be called on y.

* "indexable()": check that all input arrays have consistent length

classification or multitarget regression, specify multi_output=True,

  and can be sliced or indexed using safe_index.  This is used to
  validate input for cross-validation.

* "validation.check_memory()" checks that input is
  "joblib.Memory"-like, which means that it can be converted into a
  "sklearn.utils.Memory" instance (typically a str denoting the
  "cachedir") or has the same interface.

If your code relies on a random number generator, it should never use
functions like "numpy.random.random" or "numpy.random.normal".  This
approach can lead to repeatability issues in unit tests.  Instead, a
"numpy.random.RandomState" object should be used, which is built from
a "random_state" argument passed to the class or function.  The
function "check_random_state()", below, can then be used to create a
random number generator object.

* "check_random_state()": create a "np.random.RandomState" object from
  a parameter "random_state".

  * If "random_state" is "None" or "np.random", then a randomly-
    initialized "RandomState" object is returned.

  * If "random_state" is an integer, then it is used to seed a new
    "RandomState" object.

  * If "random_state" is a "RandomState" object, then it is passed
    through.

For example:

   >>> from sklearn.utils import check_random_state
   >>> random_state = 0
   >>> random_state = check_random_state(random_state)
   >>> random_state.rand(4)

through.

   array([0.5488135 , 0.71518937, 0.60276338, 0.54488318])

When developing your own scikit-learn compatible estimator, the
following helpers are available.

* "validation.check_is_fitted()": check that the estimator has been
  fitted before calling "transform", "predict", or similar methods.
  This helper allows to raise a standardized error message across
  estimator.

* "validation.has_fit_parameter()": check that a given parameter is
  supported in the "fit" method of a given estimator.

Efficient Linear Algebra & Array Operations

* "extmath.randomized_range_finder()": construct an orthonormal matrix
  whose range approximates the range of the input.  This is used in
  "extmath.randomized_svd()", below.

* "extmath.randomized_svd()": compute the k-truncated randomized SVD.
  This algorithm finds the exact truncated singular values
  decomposition using randomization to speed up the computations. It
  is particularly fast on large matrices on which you wish to extract
  only a small number of components.

* "arrayfuncs.cholesky_delete()": (used in "lars_path()")  Remove an
  item from a cholesky factorization.

* "arrayfuncs.min_pos()": (used in "sklearn.linear_model.least_angle")
  Find the minimum of the positive values within an array.

item from a cholesky factorization.

* "extmath.fast_logdet()": efficiently compute the log of the
  determinant of a matrix.

* "extmath.density()": efficiently compute the density of a sparse
  vector

* "extmath.safe_sparse_dot()": dot product which will correctly handle
  "scipy.sparse" inputs.  If the inputs are dense, it is equivalent to
  "numpy.dot".

* "extmath.weighted_mode()": an extension of "scipy.stats.mode" which
  allows each item to have a real-valued weight.

* "resample()": Resample arrays or sparse matrices in a consistent
  way. used in "shuffle()", below.

* "shuffle()": Shuffle arrays or sparse matrices in a consistent way.
  Used in "k_means()".

Efficient Random Sampling

* "random.sample_without_replacement()": implements efficient
  algorithms for sampling "n_samples" integers from a population of
  size "n_population" without replacement.

Efficient Routines for Sparse Matrices

The "sklearn.utils.sparsefuncs" cython module hosts compiled
extensions to efficiently process "scipy.sparse" data.

* "sparsefuncs.mean_variance_axis()": compute the means and variances
  along a specified axis of a CSR matrix. Used for normalizing the
  tolerance stopping criterion in "KMeans".

* "sparsefuncs_fast.inplace_csr_row_normalize_l1()" and
  "sparsefuncs_fast.inplace_csr_row_normalize_l2()": can be used to

tolerance stopping criterion in "KMeans".

  normalize individual sparse samples to unit L1 or L2 norm as done in
  "Normalizer".

* "sparsefuncs.inplace_csr_column_scale()": can be used to multiply
  the columns of a CSR matrix by a constant scale (one scale per
  column). Used for scaling features to unit standard deviation in
  "StandardScaler".

* "sort_graph_by_row_values()": can be used to sort a CSR sparse
  matrix such that each row is stored with increasing values. This is
  useful to improve efficiency when using precomputed sparse distance
  matrices in estimators relying on nearest neighbors graph.

Graph Routines

* "graph.single_source_shortest_path_length()": (not currently used in
  scikit-learn) Return the shortest path from a single source to all
  connected nodes on a graph.  Code is adapted from networkx. If this
  is ever needed again, it would be far faster to use a single
  iteration of Dijkstra's algorithm from "graph_shortest_path".

Testing Functions

* "discovery.all_estimators()" : returns a list of all estimators in
  scikit-learn to test for consistent behavior and interfaces.

* "discovery.all_displays()" : returns a list of all displays (related
  to plotting API) in scikit-learn to test for consistent behavior and
  interfaces.

* "discovery.all_functions()" : returns a list all functions in

to plotting API) in scikit-learn to test for consistent behavior and

Multiclass and multilabel utility function

* "multiclass.is_multilabel()": Helper function to check if the task
  is a multi-label classification one.

* "multiclass.unique_labels()": Helper function to extract an ordered
  array of unique labels from different formats of target.

Helper Functions

* "gen_even_slices": generator to create "n"-packs of slices going up
  to "n".  Used in "dict_learning()" and "k_means()".

* "gen_batches": generator to create slices containing batch size
  elements from 0 to "n"

* "safe_mask()": Helper function to convert a mask to the format
  expected by the numpy array or scipy sparse matrix on which to use
  it (sparse matrices support integer indices only while numpy arrays
  support both boolean masks and integer indices).

* "safe_sqr()": Helper function for unified squaring ("**2") of array-
  likes, matrices and sparse matrices.

Hash Functions

* "murmurhash3_32()" provides a python wrapper for the
  "MurmurHash3_x86_32" C++ non cryptographic hash function. This hash
  function is suitable for implementing lookup tables, Bloom filters,
  Count Min Sketch, feature hashing and implicitly defined sparse
  random projections:

     >>> from sklearn.utils import murmurhash3_32

Count Min Sketch, feature hashing and implicitly defined sparse

     >>> murmurhash3_32("some feature", seed=0) == -384616559
     True

     >>> murmurhash3_32("some feature", seed=0, positive=True) == 3910350737
     True

  The "sklearn.utils.murmurhash" module can also be "cimported" from
  other cython modules so as to benefit from the high performance of
  MurmurHash while skipping the overhead of the Python interpreter.

Warnings and Exceptions

* "deprecated": Decorator to mark a function or class as deprecated.

* "ConvergenceWarning": Custom warning to catch convergence problems.
  Used in "sklearn.covariance.graphical_lasso".

Developing with the Plotting API

machine learning. The key features of this API is to run calculations
once and to have the flexibility to adjust the visualizations after
the fact. This section is intended for developers who wish to develop
or maintain plotting tools. For usage, users should refer to the
:ref`User Guide <visualizations>`.

Plotting API Overview

This logic is encapsulated into a display object where the computed
data is stored and the plotting is done in a *plot* method. The
display object's *__init__* method contains only the data needed to
create the visualization. The *plot* method takes in parameters that
only have to do with visualization, such as a matplotlib axes. The
*plot* method will store the matplotlib artists as attributes allowing
for style adjustments through the display object. The *Display* class
should define one or both class methods: *from_estimator* and
*from_predictions*. These methods allows to create the *Display*
object from the estimator and some data or from the true and predicted
values. After these class methods create the display object with the
computed values, then call the display's plot method. Note that the
*plot* method defines attributes related to matplotlib, such as the
line artist. This allows for customizations after calling the *plot*
method.

For example, the *RocCurveDisplay* defines the following methods and
attributes:

method.

   class RocCurveDisplay:
       def __init__(self, fpr, tpr, roc_auc, estimator_name):
           ...
           self.fpr = fpr
           self.tpr = tpr
           self.roc_auc = roc_auc
           self.estimator_name = estimator_name

       @classmethod
       def from_estimator(cls, estimator, X, y):
           # get the predictions
           y_pred = estimator.predict_proba(X)[:, 1]
           return cls.from_predictions(y, y_pred, estimator.__class__.__name__)

       def from_predictions(cls, y, y_pred, estimator_name):
           # do ROC computation from y and y_pred
           fpr, tpr, roc_auc = ...
           viz = RocCurveDisplay(fpr, tpr, roc_auc, estimator_name)
           return viz.plot()

       def plot(self, ax=None, name=None, **kwargs):
           ...
           self.line_ = ...
           self.ax_ = ax
           self.figure_ = ax.figure_

Read more in *sphx_glr_auto_examples_miscellaneous_plot_roc_curve_vis
ualization_api.py* and the User Guide.

Plotting with Multiple Axes

Some of the plotting tools like "from_estimator()" and
"PartialDependenceDisplay" support plotting on multiple axes. Two
different scenarios are supported:

1. If a list of axes is passed in, *plot* will check if the number of
axes is consistent with the number of axes it expects and then draws

on those axes. 2. If a single axes is passed in, that axes defines a
space for multiple axes to be placed. In this case, we suggest using
matplotlib's *~matplotlib.gridspec.GridSpecFromSubplotSpec* to split
up the space:

   import matplotlib.pyplot as plt
   from matplotlib.gridspec import GridSpecFromSubplotSpec

   fig, ax = plt.subplots()
   gs = GridSpecFromSubplotSpec(2, 2, subplot_spec=ax.get_subplotspec())

   ax_top_left = fig.add_subplot(gs[0, 0])
   ax_top_right = fig.add_subplot(gs[0, 1])
   ax_bottom = fig.add_subplot(gs[1, :])

By default, the *ax* keyword in *plot* is *None*. In this case, the
single axes is created and the gridspec api is used to create the
regions to plot in.

See for example, " The axes defining the bounding box is saved in a
`bounding_ax_()" attribute. The individual axes created are stored in
an *axes_* ndarray, corresponding to the axes position on the grid.
Positions that are not used are set to *None*. Furthermore, the
matplotlib Artists are stored in *lines_* and *contours_* where the
key is the position on the grid. When a list of axes is passed in, the
*axes_*, *lines_*, and *contours_* is a 1d ndarray corresponding to
the list of axes passed in.