.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/parallel/parallel_with_tqdm.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_parallel_parallel_with_tqdm.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_parallel_parallel_with_tqdm.py:


=================================================
Using tqdm to track progress with joblib.Parallel
=================================================

This example demonstrates how to use a ``tqdm`` progress bar together with
:class:`joblib.Parallel`.

We present three main approaches:

1.  :ref:`solution_generator`, via ``return_as='generator'`` to update the
    progress bar as tasks finish. This works well for homogeneous tasks but
    has some limitations for heterogeneous ones.
2.  :ref:`solution_unordered_generator`, which works for heterogeneous tasks
    but does not preserve the tasks' order.
3.  :ref:`solution_subclass`, which requires more code but allows accurate
    progress reporting independently of the ``return_as`` option.

Note that ``return_as='generator'`` is not available for the ``'multiprocessing'``
backend.

.. GENERATED FROM PYTHON SOURCE LINES 24-31

Task definition
===============

We first define a task that sleeps for a given amount of time and returns
that value. We also define a list of sleep durations that will be used.
Note that the task durations are heterogeneous, simulating a situation
where not all tasks take the same amount of time to complete.

.. GENERATED FROM PYTHON SOURCE LINES 31-43

.. code-block:: Python


    import time


    def task(t):
        time.sleep(0.1 * t)
        return t


    times = [7, 2, 3, 5, 6, 4, 1]









.. GENERATED FROM PYTHON SOURCE LINES 44-59

A first naive solution
======================

.. warning::

    This solution reports the number of *dispatched* tasks rather than the
    number of *completed* tasks. It is presented for pedagogical purposes
    and should not be used in practice.

A ``tqdm`` progress bar takes an iterable as input and returns an iterable.
A :class:`~joblib.Parallel` call also consumes an iterable.

A straightforward solution is therefore to wrap the input iterable with
``tqdm`` before passing it to :class:`~joblib.Parallel`. The progress bar
then reflects the number of tasks dispatched to the workers.

.. GENERATED FROM PYTHON SOURCE LINES 59-68

.. code-block:: Python


    from tqdm import tqdm

    from joblib import Parallel, delayed

    p = Parallel(n_jobs=2, pre_dispatch=4, batch_size=1)
    out = p(delayed(task)(t) for t in tqdm(times))
    print(*out)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/7 [00:00<?, ?it/s]
     57%|█████▋    | 4/7 [00:00<00:00, 19.65it/s]
     86%|████████▌ | 6/7 [00:00<00:00,  7.48it/s]
    100%|██████████| 7/7 [00:00<00:00,  9.96it/s]
    7 2 3 5 6 4 1




.. GENERATED FROM PYTHON SOURCE LINES 69-81

As can be observed, the progress bar initially jumps from 0 to 4 tasks.
This is due to the ``pre_dispatch=4`` argument, which instructs
:class:`~joblib.Parallel` to dispatch 4 tasks at the start.
Afterwards, tasks are dispatched in chunks of ``n_jobs * batch_size``.
In this example, the progress bar advances in steps of 2 because
``n_jobs=2`` and ``batch_size=1``.

For large ``batch_size`` values, this progress bar becomes less precise
in reflecting how many tasks have started. Moreover, this solution reports
the number of dispatched tasks, whereas one might instead be interested in
the number of completed tasks. The following examples show how to track
completed tasks instead.

.. GENERATED FROM PYTHON SOURCE LINES 83-102

.. _solution_generator:

Using a generator
=================

.. note::

    This solution works for homogeneous tasks but updates progress based on
    task order. It fails to report progress when a later task completes
    while an earlier one is still running.

Since a :class:`~joblib.Parallel` call returns an iterable over task outputs
(by default, a list), one might try to wrap the returned iterable with
``tqdm``.

However, since the default return type is a ``list``, all tasks must
complete before the list becomes available. To observe progress as tasks
finish, we must use ``return_as='generator'``, which yields outputs as they
become available.

.. GENERATED FROM PYTHON SOURCE LINES 103-108

.. code-block:: Python


    p = Parallel(n_jobs=2, return_as="generator")
    out = p(delayed(task)(t) for t in times)
    print(*tqdm(out, total=len(times)))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/7 [00:00<?, ?it/s]
     14%|█▍        | 1/7 [00:00<00:04,  1.42it/s]
     57%|█████▋    | 4/7 [00:01<00:00,  4.63it/s]
     71%|███████▏  | 5/7 [00:01<00:00,  4.22it/s]
     86%|████████▌ | 6/7 [00:01<00:00,  5.03it/s]
    100%|██████████| 7/7 [00:01<00:00,  4.98it/s]
    7 2 3 5 6 4 1




.. GENERATED FROM PYTHON SOURCE LINES 109-121

The progress bar now updates as results are produced. However, we still
observe a jump from 1 task directly to 4 tasks.

This happens because when using ``return_as='generator'``, outputs are
yielded in the same order as the input tasks. The first yielded value
therefore corresponds to the task that takes 0.7 seconds to complete
(recall ``times = [7, 2, 3, 5, 6, 4, 1]``). When this first task finishes,
the second and third tasks have already completed as well, since
2 + 3 < 7. This explains the jump from 1/7 to 4/7.

Note that we must explicitly pass ``total=len(times)`` to ``tqdm`` because
the returned generator does not define a length, unlike a ``list``.

.. GENERATED FROM PYTHON SOURCE LINES 123-137

.. _solution_unordered_generator:

Using an unordered generator
============================

.. note::

    This solution does not preserve the tasks' input order in the results

If you do not care about the order of the outputs and want smoother
progress reporting, you can use ``return_as='generator_unordered'``.

In this case, results are yielded as soon as tasks complete, allowing the
progress bar to advance one step at a time.

.. GENERATED FROM PYTHON SOURCE LINES 138-143

.. code-block:: Python


    p = Parallel(n_jobs=2, return_as="generator_unordered")
    out = p(delayed(task)(t) for t in times)
    print(*tqdm(out, total=len(times)))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/7 [00:00<?, ?it/s]
     14%|█▍        | 1/7 [00:00<00:01,  4.91it/s]
     29%|██▊       | 2/7 [00:00<00:01,  3.82it/s]
     43%|████▎     | 3/7 [00:00<00:00,  4.26it/s]
     57%|█████▋    | 4/7 [00:01<00:00,  3.82it/s]
     71%|███████▏  | 5/7 [00:01<00:00,  3.66it/s]
     86%|████████▌ | 6/7 [00:01<00:00,  4.66it/s]
    100%|██████████| 7/7 [00:01<00:00,  4.98it/s]
    2 3 7 5 6 1 4




.. GENERATED FROM PYTHON SOURCE LINES 144-148

The progress bar now advances smoothly. Note how the order of the outputs
differs from the original input order.
If you need to preserve the original ordering, you could modify the task
function to also return the index of the task.

.. GENERATED FROM PYTHON SOURCE LINES 150-162

.. _solution_subclass:

Subclassing :class:`joblib.Parallel`
====================================

The :class:`~joblib.Parallel` class provides a method
:meth:`~joblib.Parallel.print_progress`, which is called whenever a task
completes. By default, this method prints varying amounts of information
depending on the value of the ``verbose`` parameter.

Here, we override this method in a custom subclass in order to update a
``tqdm`` progress bar instead.

.. GENERATED FROM PYTHON SOURCE LINES 163-180

.. code-block:: Python



    class ParallelTqdm(Parallel):
        def __call__(self, iterable, n_tasks):
            self.tqdm = tqdm(total=n_tasks)
            return super().__call__(iterable)

        def print_progress(self):
            self.tqdm.update(self.n_completed_tasks - self.tqdm.n)
            if self._is_completed():
                self.tqdm.close()


    p = ParallelTqdm(n_jobs=2)
    out = p((delayed(task)(t) for t in times), len(times))
    print(*out)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/7 [00:00<?, ?it/s]
     14%|█▍        | 1/7 [00:00<00:01,  4.92it/s]
     29%|██▊       | 2/7 [00:00<00:01,  3.84it/s]
     43%|████▎     | 3/7 [00:00<00:00,  4.31it/s]
     57%|█████▋    | 4/7 [00:01<00:00,  3.85it/s]
     71%|███████▏  | 5/7 [00:01<00:00,  3.65it/s]
     86%|████████▌ | 6/7 [00:01<00:00,  4.65it/s]
    100%|██████████| 7/7 [00:01<00:00,  4.98it/s]
    7 2 3 5 6 4 1




.. GENERATED FROM PYTHON SOURCE LINES 181-185

This approach provides accurate progress reporting for heterogeneous tasks
without changing the result ordering. However, it requires a more complex
interface, so the previous solutions may be preferable for simpler use
cases.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 5.650 seconds)


.. _sphx_glr_download_auto_examples_parallel_parallel_with_tqdm.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: parallel_with_tqdm.ipynb <parallel_with_tqdm.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: parallel_with_tqdm.py <parallel_with_tqdm.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: parallel_with_tqdm.zip <parallel_with_tqdm.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_