.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/parallel/distributed_backend_simple.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_parallel_distributed_backend_simple.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_parallel_distributed_backend_simple.py:


Using Dask for single-machine parallel computing
================================================

This example shows the simplest usage of the
`Dask <https://docs.dask.org/en/stable/>`_
backend on your local machine.

This is useful for prototyping a solution, to later be run on a truly
`distributed Dask cluster
<https://docs.dask.org/en/stable/deploying.html#distributed-computing>`_,
as the only change needed is the cluster class.

Another realistic usage scenario: combining dask code with joblib code,
for instance using dask for preprocessing data, and scikit-learn for
machine learning. In such a setting, it may be interesting to use
distributed as a backend scheduler for both dask and joblib, to
orchestrate the computation.

.. GENERATED FROM PYTHON SOURCE LINES 23-25

Setup the distributed client
##############################################################################

.. GENERATED FROM PYTHON SOURCE LINES 25-36

.. code-block:: Python

    from dask.distributed import Client, LocalCluster

    # replace with whichever cluster class you're using
    # https://docs.dask.org/en/stable/deploying.html#distributed-computing
    cluster = LocalCluster()
    # connect client to your cluster
    client = Client(cluster)

    # Monitor your computation with the Dask dashboard
    print(client.dashboard_link)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    http://127.0.0.1:8787/status




.. GENERATED FROM PYTHON SOURCE LINES 37-39

Run parallel computation using dask.distributed
##############################################################################

.. GENERATED FROM PYTHON SOURCE LINES 39-49

.. code-block:: Python


    import time
    import joblib


    def long_running_function(i):
        time.sleep(0.1)
        return i









.. GENERATED FROM PYTHON SOURCE LINES 50-52

The verbose messages below show that the backend is indeed the
dask.distributed one

.. GENERATED FROM PYTHON SOURCE LINES 52-57

.. code-block:: Python

    with joblib.parallel_config(backend="dask"):
        joblib.Parallel(verbose=100)(
            joblib.delayed(long_running_function)(i) for i in range(10)
        )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    [Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 2 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   1 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Batch computation too fast (0.15661358833312988s.) Setting batch_size=2.
    [Parallel(n_jobs=-1)]: Done   2 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Done   3 tasks      | elapsed:    0.3s
    [Parallel(n_jobs=-1)]: Done   4 tasks      | elapsed:    0.3s
    [Parallel(n_jobs=-1)]: Done   6 tasks      | elapsed:    0.5s
    [Parallel(n_jobs=-1)]: Done   8 out of  10 | elapsed:    0.5s remaining:    0.1s
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.6s finished





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 1.903 seconds)


.. _sphx_glr_download_auto_examples_parallel_distributed_backend_simple.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: distributed_backend_simple.ipynb <distributed_backend_simple.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: distributed_backend_simple.py <distributed_backend_simple.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_