.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/parallel_memmap.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_parallel_memmap.py: =============================== NumPy memmap in joblib.Parallel =============================== This example illustrates some features enabled by using a memory map (:class:`numpy.memmap`) within :class:`joblib.Parallel`. First, we show that dumping a huge data array ahead of passing it to :class:`joblib.Parallel` speeds up computation. Then, we show the possibility to provide write access to original data. .. GENERATED FROM PYTHON SOURCE LINES 15-20 Speed up processing of a large data array ############################################################################# We create a large data array for which the average is computed for several slices. .. GENERATED FROM PYTHON SOURCE LINES 20-28 .. code-block:: Python import numpy as np data = np.random.random((int(1e7),)) window_size = int(5e5) slices = [slice(start, start + window_size) for start in range(0, data.size - window_size, int(1e5))] .. GENERATED FROM PYTHON SOURCE LINES 29-33 The ``slow_mean`` function introduces a :func:`time.sleep` call to simulate a more expensive computation cost for which parallel computing is beneficial. Parallel may not be beneficial for very fast operation, due to extra overhead (workers creations, communication, etc.). .. GENERATED FROM PYTHON SOURCE LINES 33-43 .. code-block:: Python import time def slow_mean(data, sl): """Simulate a time consuming processing.""" time.sleep(0.01) return data[sl].mean() .. GENERATED FROM PYTHON SOURCE LINES 44-45 First, we will evaluate the sequential computing on our problem. .. GENERATED FROM PYTHON SOURCE LINES 45-52 .. code-block:: Python tic = time.time() results = [slow_mean(data, sl) for sl in slices] toc = time.time() print('\nElapsed time computing the average of couple of slices {:.2f} s' .format(toc - tic)) .. rst-class:: sphx-glr-script-out .. code-block:: none Elapsed time computing the average of couple of slices 0.98 s .. GENERATED FROM PYTHON SOURCE LINES 53-55 :class:`joblib.Parallel` is used to compute in parallel the average of all slices using 2 workers. .. GENERATED FROM PYTHON SOURCE LINES 55-65 .. code-block:: Python from joblib import Parallel, delayed tic = time.time() results = Parallel(n_jobs=2)(delayed(slow_mean)(data, sl) for sl in slices) toc = time.time() print('\nElapsed time computing the average of couple of slices {:.2f} s' .format(toc - tic)) .. rst-class:: sphx-glr-script-out .. code-block:: none Elapsed time computing the average of couple of slices 0.71 s .. GENERATED FROM PYTHON SOURCE LINES 66-69 Parallel processing is already faster than the sequential processing. It is also possible to remove a bit of overhead by dumping the ``data`` array to a memmap and pass the memmap to :class:`joblib.Parallel`. .. GENERATED FROM PYTHON SOURCE LINES 69-89 .. code-block:: Python import os from joblib import dump, load folder = './joblib_memmap' try: os.mkdir(folder) except FileExistsError: pass data_filename_memmap = os.path.join(folder, 'data_memmap') dump(data, data_filename_memmap) data = load(data_filename_memmap, mmap_mode='r') tic = time.time() results = Parallel(n_jobs=2)(delayed(slow_mean)(data, sl) for sl in slices) toc = time.time() print('\nElapsed time computing the average of couple of slices {:.2f} s\n' .format(toc - tic)) .. rst-class:: sphx-glr-script-out .. code-block:: none Elapsed time computing the average of couple of slices 0.53 s .. GENERATED FROM PYTHON SOURCE LINES 90-93 Therefore, dumping large ``data`` array ahead of calling :class:`joblib.Parallel` can speed up the processing by removing some overhead. .. GENERATED FROM PYTHON SOURCE LINES 95-101 Writable memmap for shared memory :class:`joblib.Parallel` ############################################################################## ``slow_mean_write_output`` will compute the mean for some given slices as in the previous example. However, the resulting mean will be directly written on the output array. .. GENERATED FROM PYTHON SOURCE LINES 101-111 .. code-block:: Python def slow_mean_write_output(data, sl, output, idx): """Simulate a time consuming processing.""" time.sleep(0.005) res_ = data[sl].mean() print("[Worker %d] Mean for slice %d is %f" % (os.getpid(), idx, res_)) output[idx] = res_ .. GENERATED FROM PYTHON SOURCE LINES 112-113 Prepare the folder where the memmap will be dumped. .. GENERATED FROM PYTHON SOURCE LINES 113-116 .. code-block:: Python output_filename_memmap = os.path.join(folder, 'output_memmap') .. GENERATED FROM PYTHON SOURCE LINES 117-119 Pre-allocate a writable shared memory map as a container for the results of the parallel computation. .. GENERATED FROM PYTHON SOURCE LINES 119-123 .. code-block:: Python output = np.memmap(output_filename_memmap, dtype=data.dtype, shape=len(slices), mode='w+') .. GENERATED FROM PYTHON SOURCE LINES 124-126 ``data`` is replaced by its memory mapped version. Note that the buffer has already been dumped in the previous section. .. GENERATED FROM PYTHON SOURCE LINES 126-129 .. code-block:: Python data = load(data_filename_memmap, mmap_mode='r') .. GENERATED FROM PYTHON SOURCE LINES 130-131 Fork the worker processes to perform computation concurrently .. GENERATED FROM PYTHON SOURCE LINES 131-135 .. code-block:: Python Parallel(n_jobs=2)(delayed(slow_mean_write_output)(data, sl, output, idx) for idx, sl in enumerate(slices)) .. rst-class:: sphx-glr-script-out .. code-block:: none [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None] .. GENERATED FROM PYTHON SOURCE LINES 136-137 Compare the results from the output buffer with the expected results .. GENERATED FROM PYTHON SOURCE LINES 137-143 .. code-block:: Python print("\nExpected means computed in the parent process:\n {}" .format(np.array(results))) print("\nActual means computed by the worker processes:\n {}" .format(output)) .. rst-class:: sphx-glr-script-out .. code-block:: none Expected means computed in the parent process: [0.49941799 0.49951335 0.49946321 0.50006669 0.50002671 0.49990012 0.50010129 0.49998478 0.50010755 0.49971238 0.49999648 0.4998809 0.49999253 0.500088 0.50044859 0.50027714 0.500061 0.4997412 0.49946122 0.49986762 0.50001886 0.49990907 0.50016357 0.50033087 0.49997333 0.49975513 0.49998989 0.50028207 0.50061448 0.50068578 0.50082219 0.5006816 0.50053563 0.50033229 0.50021216 0.49996074 0.50052432 0.50075476 0.50050013 0.50044068 0.50071113 0.49997055 0.49950489 0.49935413 0.49932182 0.49899791 0.49943648 0.49957187 0.49981282 0.50048889 0.5005721 0.50009576 0.50005474 0.50021313 0.49997376 0.50042857 0.50050291 0.50092451 0.50044682 0.50004368 0.49986911 0.49960361 0.49915127 0.49921872 0.4992526 0.49875597 0.49920782 0.49929792 0.499483 0.4997265 0.49985992 0.50017824 0.49996324 0.50001763 0.49971817 0.49988047 0.49942196 0.49949806 0.49927049 0.49955793 0.49958791 0.49972976 0.49992339 0.50019055 0.49977305 0.4996532 0.49985402 0.49955604 0.49932991 0.49933411 0.49955789 0.49907223 0.49937178 0.49944696 0.49972926] Actual means computed by the worker processes: [0.49941799 0.49951335 0.49946321 0.50006669 0.50002671 0.49990012 0.50010129 0.49998478 0.50010755 0.49971238 0.49999648 0.4998809 0.49999253 0.500088 0.50044859 0.50027714 0.500061 0.4997412 0.49946122 0.49986762 0.50001886 0.49990907 0.50016357 0.50033087 0.49997333 0.49975513 0.49998989 0.50028207 0.50061448 0.50068578 0.50082219 0.5006816 0.50053563 0.50033229 0.50021216 0.49996074 0.50052432 0.50075476 0.50050013 0.50044068 0.50071113 0.49997055 0.49950489 0.49935413 0.49932182 0.49899791 0.49943648 0.49957187 0.49981282 0.50048889 0.5005721 0.50009576 0.50005474 0.50021313 0.49997376 0.50042857 0.50050291 0.50092451 0.50044682 0.50004368 0.49986911 0.49960361 0.49915127 0.49921872 0.4992526 0.49875597 0.49920782 0.49929792 0.499483 0.4997265 0.49985992 0.50017824 0.49996324 0.50001763 0.49971817 0.49988047 0.49942196 0.49949806 0.49927049 0.49955793 0.49958791 0.49972976 0.49992339 0.50019055 0.49977305 0.4996532 0.49985402 0.49955604 0.49932991 0.49933411 0.49955789 0.49907223 0.49937178 0.49944696 0.49972926] .. GENERATED FROM PYTHON SOURCE LINES 144-149 Clean-up the memmap ############################################################################## Remove the different memmap that we created. It might fail in Windows due to file permissions. .. GENERATED FROM PYTHON SOURCE LINES 149-156 .. code-block:: Python import shutil try: shutil.rmtree(folder) except: # noqa print('Could not clean-up automatically.') .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 2.680 seconds) .. _sphx_glr_download_auto_examples_parallel_memmap.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: parallel_memmap.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: parallel_memmap.py ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_