bluepyparallel.parallel

Parallel helper.

Functions

init_parallel_factory(parallel_lib, *args, ...)

Return the desired instance of the parallel factory.

Classes

DaskDataFrameFactory([batch_size, ...])

Parallel helper class using dask.dataframe.

DaskFactory([batch_size, chunk_size, ...])

Parallel helper class using dask.

IPyParallelFactory([batch_size, chunk_size, ...])

Parallel helper class using ipyparallel.

MultiprocessingFactory([batch_size, ...])

Parallel helper class using multiprocessing.

NestedPool([processes, initializer, ...])

Class that represents a MultiProcessing nested pool.

NoDaemonProcess([group, target, name, args, ...])

Class that represents a non-daemon process.

ParallelFactory([batch_size, chunk_size])

Abstract class that should be subclassed to provide parallel functions.

SerialFactory([batch_size, chunk_size])

Factory that do not work in parallel.
class bluepyparallel.parallel.DaskDataFrameFactory(batch_size=None, chunk_size=None, scheduler_file=None, address=None, dask_config=None, **kwargs)

Bases: DaskFactory

Parallel helper class using dask.dataframe.

It is possible to pass a custom dask configuration in several ways. The simplest way is to pass a dictionary to the dask_config argument. Another way is to create a YAML file containing the configuration and then set the DASK_CONFIG environment variable to its path. Note that this environment variable must be set before dask is imported and can not be updated afterwards. Also, it is possible to use the SHMDIR or the TMPDIR environment variable to specify the directory in which the dask internals will be created. Note that this value will be overridden if a dask configuration is given. If no config is provided, the following is used:

{
    "distributed": {
        "admin": {
            "tick": {
                "limit": "1h"
            }
        },
        "worker": {
            "memory": {
                "pause": 0.8,
                "spill": false,
                "target": false,
                "terminate": 0.95
            },
            "profile": {
                "enabled": false
            },
            "use_file_locking": false
        }
    }
}
get_mapper(batch_size=None, chunk_size=None, **kwargs)

Get a Dask mapper.

If progress_bar=True is passed as keyword argument, a progress bar will be displayed during computation.

class bluepyparallel.parallel.DaskFactory(batch_size=None, chunk_size=None, scheduler_file=None, address=None, dask_config=None, **kwargs)

Bases: ParallelFactory

Parallel helper class using dask.

It is possible to pass a custom dask configuration in several ways. The simplest way is to pass a dictionary to the dask_config argument. Another way is to create a YAML file containing the configuration and then set the DASK_CONFIG environment variable to its path. Note that this environment variable must be set before dask is imported and can not be updated afterwards. Also, it is possible to use the SHMDIR or the TMPDIR environment variable to specify the directory in which the dask internals will be created. Note that this value will be overridden if a dask configuration is given. If no config is provided, the following is used:

{
    "distributed": {
        "admin": {
            "tick": {
                "limit": "1h"
            }
        },
        "worker": {
            "memory": {
                "pause": 0.8,
                "spill": false,
                "target": false,
                "terminate": 0.95
            },
            "profile": {
                "enabled": false
            },
            "use_file_locking": false
        }
    }
}
get_mapper(batch_size=None, chunk_size=None, **kwargs)

Get a Dask mapper.

shutdown()

Close the scheduler and the cluster if it was created by the factory.

class bluepyparallel.parallel.IPyParallelFactory(batch_size=None, chunk_size=None, profile=None, **kwargs)

Bases: ParallelFactory

Parallel helper class using ipyparallel.

get_mapper(batch_size=None, chunk_size=None, **kwargs)

Get an ipyparallel mapper using the profile name provided.

shutdown()

Remove zmq.

class bluepyparallel.parallel.MultiprocessingFactory(batch_size=None, chunk_size=None, processes=None, **kwargs)

Bases: ParallelFactory

Parallel helper class using multiprocessing.

get_mapper(batch_size=None, chunk_size=None, **kwargs)

Get a NestedPool.

shutdown()

Close the pool.

class bluepyparallel.parallel.NestedPool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, context=None)

Bases: Pool

Class that represents a MultiProcessing nested pool.

Process

alias of NoDaemonProcess

class bluepyparallel.parallel.NoDaemonProcess(group=None, target=None, name=None, args=(), kwargs={})

Bases: Process

Class that represents a non-daemon process.

property daemon

Get daemon flag.

class bluepyparallel.parallel.ParallelFactory(batch_size=None, chunk_size=None)

Bases: object

Abstract class that should be subclassed to provide parallel functions.

abstractmethod get_mapper(batch_size=None, chunk_size=None, **kwargs)

Return a mapper function that can be used to execute functions in parallel.

mappable_func(func, *args, **kwargs)

Can be used to add args and kwargs to a function before calling the mapper.

shutdown()

Can be used to cleanup.

class bluepyparallel.parallel.SerialFactory(batch_size=None, chunk_size=None)

Bases: ParallelFactory

Factory that do not work in parallel.

get_mapper(batch_size=None, chunk_size=None, **kwargs)

Get a map.

bluepyparallel.parallel.init_parallel_factory(parallel_lib, *args, **kwargs)

Return the desired instance of the parallel factory.

The main factories are: