Serialization and memory-mapped loading#

Use case#

joblib.dump() and joblib.load() provide numpy-aware serialization optimized for memory-efficient loading in multiprocess environments. Large numpy arrays are stored as raw binary buffers that can be memory-mapped on load, so multiple worker processes share a single copy of the data instead of each holding a full in-memory duplicate.

The canonical use case is serving a scikit-learn model under a multiprocess server (e.g. gunicorn): dump the fitted model once, then each worker memory-maps the large coefficient arrays rather than pickling them across a pipe. This avoids duplicating potentially gigabytes of data per worker.

Security considerations#

Because joblib.load() is designed for controlled, internal pipelines — not for distributing models to end users — it only makes safety guarantees for files you produced yourself. It is intended for use cases such as caching intermediate results or sharing objects between worker processes, where the file was written by your own code or by collaborators you control.

Warning

joblib.dump() and joblib.load() are built on Python’s pickle protocol, which can execute arbitrary Python code during deserialization. Loading a file from an untrusted source is equivalent to running untrusted code. This is a fundamental property of pickle, not a joblib-specific bug.

Never call joblib.load() on a file you did not produce yourself or that came from a source you do not fully trust.

Scope of joblib’s security guarantees#

Joblib focuses on parallelism, caching, and distributed computation. It does not aim to be a safe model-exchange format. Consequently:

Joblib will not attempt to sandbox or restrict what pickle can deserialize — such restrictions are fragile by design and cannot be made reliable.
Security reports that require an attacker-controlled joblib file as their starting point are outside joblib’s threat model: if the file is malicious, the user should not have loaded it.
We may still fix behaviors that are surprising or unnecessarily dangerous (e.g. inconsistencies in how companion files are resolved), but these fixes aim for correctness, not for making untrusted-file loading safe.

A simple example#

First create a temporary directory:

>>> from tempfile import mkdtemp
>>> savedir = mkdtemp()
>>> import os
>>> filename = os.path.join(savedir, 'test.joblib')

Then create an object to be persisted:

>>> import numpy as np
>>> to_persist = [('a', [1, 2, 3]), ('b', np.arange(10))]

which is saved into filename:

>>> import joblib
>>> joblib.dump(to_persist, filename)
['...test.joblib']

The object can then be reloaded from the file:

>>> joblib.load(filename)
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]

Persistence in file objects#

Instead of filenames, joblib.dump() and joblib.load() functions also accept file objects:

>>> with open(filename, 'wb') as fo:
...    joblib.dump(to_persist, fo)
>>> with open(filename, 'rb') as fo:
...    joblib.load(fo)
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]

Compressed joblib pickles#

Setting the compress argument to True in joblib.dump() will allow to save space on disk:

>>> joblib.dump(to_persist, filename + '.compressed', compress=True)
['...test.joblib.compressed']

If the filename extension corresponds to one of the supported compression methods, the compressor will be used automatically:

>>> joblib.dump(to_persist, filename + '.z')
['...test.joblib.z']

By default, joblib.dump() uses the zlib compression method as it gives the best tradeoff between speed and disk space. The other supported compression methods are ‘gzip’, ‘bz2’, ‘lzma’ and ‘xz’:

>>> # Dumping in a gzip compressed file using a compress level of 3.
>>> joblib.dump(to_persist, filename + '.gz', compress=('gzip', 3))
['...test.joblib.gz']
>>> joblib.load(filename + '.gz')
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]
>>> joblib.dump(to_persist, filename + '.bz2', compress=('bz2', 3))
['...test.joblib.bz2']
>>> joblib.load(filename + '.bz2')
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]

The compress parameter of the joblib.dump() function also accepts a string corresponding to the name of the compressor used. When using this, the default compression level is used by the compressor:

>>> joblib.dump(to_persist, filename + '.gz', compress='gzip')
['...test.joblib.gz']

Note

Lzma and Xz compression methods are only available for python versions >= 3.3.

Compressor files provided by the python standard library can also be used to compress pickle, e.g gzip.GzipFile, bz2.BZ2File, lzma.LZMAFile:

>>> # Dumping in a gzip.GzipFile object using a compression level of 3.
>>> import gzip
>>> with gzip.GzipFile(filename + '.gz', 'wb', compresslevel=3) as fo:
...    joblib.dump(to_persist, fo)
>>> with gzip.GzipFile(filename + '.gz', 'rb') as fo:
...    joblib.load(fo)
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]

If the lz4 package is installed, this compression method is automatically available with the dump function.

>>> joblib.dump(to_persist, filename + '.lz4')
['...test.joblib.lz4']
>>> joblib.load(filename + '.lz4')
[('a', [1, 2, 3]), ('b', array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))]

Note

LZ4 compression is only available with python major versions >= 3

More details can be found in the joblib.dump() and joblib.load() documentation.

Registering extra compressors#

Joblib provides joblib.register_compressor() in order to extend the list of default compressors available. To fit with Joblib internal implementation and features, such as joblib.load() and joblib.Memory, the registered compressor should implement the Python file object interface.

Compatibility across python versions#

Compatibility of joblib pickles across python versions is not fully supported. Note that, for a very restricted set of objects, this may appear to work when saving a pickle with python 2 and loading it with python 3 but relying on it is strongly discouraged.

If you are switching between python versions, you will need to save a different joblib pickle for each python version.

Here are a few examples or exceptions:

Saving joblib pickle with python 2, trying to load it with python 3:

Traceback (most recent call last):
  File "/home/lesteve/dev/joblib/joblib/numpy_pickle.py", line 453, in load
    obj = unpickler.load()
  File "/home/lesteve/miniconda3/lib/python3.4/pickle.py", line 1038, in load
    dispatch[key[0]](self)
  File "/home/lesteve/miniconda3/lib/python3.4/pickle.py", line 1176, in load_binstring
    self.append(self._decode_string(data))
  File "/home/lesteve/miniconda3/lib/python3.4/pickle.py", line 1158, in _decode_string
    return value.decode(self.encoding, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 1024: ordinal not in range(128)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/lesteve/dev/joblib/joblib/numpy_pickle.py", line 462, in load
    raise new_exc
  ValueError: You may be trying to read with python 3 a joblib pickle generated with python 2. This is not feature supported by joblib.

Saving joblib pickle with python 3, trying to load it with python 2:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "joblib/numpy_pickle.py", line 453, in load
    obj = unpickler.load()
  File "/home/lesteve/miniconda3/envs/py27/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/lesteve/miniconda3/envs/py27/lib/python2.7/pickle.py", line 886, in load_proto
    raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3