7ThS r S r SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK
r
SSK r SSK r SSK
Jr SSKJrJrJr SSKJr SSKJr SSKJr S S
KJr S S KJ r J r J r J r J r J!r!J"r" S S K#J$r$J%r% S S
K&J'r' S SK(J)r)J*r* \"\!S.r+Sq,Sr-Sr.SS1r/\b \ \+S' S SK0J1r1 \ \+S' Sq,\ Rd " 5 r3S r4S\40r5\$" SS9\$" SS9\$" SS9\$" SS9\$" SS9\$" SS9\$" SS9\$" SS9S.r6Sr7Sr8S r9\6S \6S \6S 4S jr:\6S \6S \6S 4S! jr; " S" S#5 r< " S$ S%\<5 r=Sr>\?" \S&5 ( aF \R R S'S(5 R 5 =( d SrC\Cb \R " \CS)9r> " S* S+\E5 rFS,rGS-rHS.rIS8S/ jrJS0 rKS1 rL " S2 S3\E5 rMS8S4 jrNS9S5 jrO " S6 S7\)5 rPg):z+
Helpers for embarrassingly parallel code.
)divisionN) nullcontext)floorlog10sqrt) TimeoutError)Integral)uuid4 )mp)AutoBatchingMixinFallbackToBackend LokyBackendMultiprocessingBackendParallelBackendBaseSequentialBackendThreadingBackend) _Sentinel eval_expr)memstr_to_bytes)Loggershort_format_time) threading
sequentialr lokymultiprocessing)r c j SSK Jn [ SU 5 g! [ a nSn[ U5 UeSnAff = f)zDRegister Dask Backend if called with parallel_config(backend="dask")r )DaskDistributedBackenddaskzTo use the dask.distributed backend you must install both the `dask` and distributed modules.
See https://dask.pydata.org/en/latest/install.html for more information.N)_daskr register_parallel_backend ImportError)r emsgs G/var/www/auris/envauris/lib/python3.13/site-packages/joblib/parallel.py_register_daskr& G s> &1!&*@A &
# A%&s
2-2r )
default_value1Mr)backendn_jobsverbose temp_folder
max_nbytes mmap_modepreferrequire) processesthreadsN) sharedmemNc Z U [ U La U $ X [ U La X $ U R $ )zReturn the value of a parallel config parameter
Explicitly setting it in Parallel has priority over setting in a
parallel_(config/backend) context manager.
)default_parallel_configr' )paramcontext_configkeys r% _get_config_paramr: n s?
+C00 "9#">> "" r0 r1 r, c L [ XU5 u p4[ [ S US5 nX54$ )!Return the active default backendr+ )_get_active_backendr: r6 )r0 r1 r, r* configr+ s r% get_active_backendr@ s. *&7COG
6x @&(
SF ?r; c [ [ S[ 5 n[ [ S US5 n[ XS5 n [ XS5 n[ X#S5 nU [ ; a [ SU S[ 35 eU[ ; a [ SU S [ 35 eU S
:X a US :X a [ S 5 eS
nUc [ [ " SS9nSnUR n[ USS5 n[ USS5 nUS :H =( a U( + =( d! U( + =( a U S:H =( a U( + n U( + =( a U S
:H =( a Un
U ( ap [ [ " US9n US: aA U( a: [ SU R R SUR R S35 UR 5 n SU S' X4$ U
( a" [ [ " US9n
XR 5 4$ XC4$ )r= r? r* r0 r1 r, zprefer=z. is not a valid backend hint, expected one of zrequire=z4 is not a valid backend constraint, expected one of r2 r4 zJprefer == 'processes' and require == 'sharedmem' are inconsistent settingsTr
nesting_levelF uses_threadssupports_sharedmemr3
zUsing z as joblib backend instead of z8 as the latter does not provide shared memory semantics.r r+ )getattr_backendr6 r: VALID_BACKEND_HINTS
ValueErrorVALID_BACKEND_CONSTRAINTSBACKENDSDEFAULT_BACKENDrC DEFAULT_THREAD_BACKENDprint __class____name__copyDEFAULT_PROCESS_BACKEND)r0 r1 r, backend_configr* explicit_backendrC rD rE
force_threadsforce_processessharedmem_backend
thread_configprocess_backends r% r> r> s" Xx1HIN *NIG vx
@F CG CG
((fX 2 3
5
//wi 8 9
;
K!7 X
?+!< ))M7NE:L *>F +F4F0F I9!4I\9I +*Uv/DU O %%; <'
b= - *44==> ?--4->->-G-G,H IJJ
'++-
"#
h // ##:;-X 3 3 555 ""r; c
\ rS rSrSr\S 4\S \S \S \S \S \S \S
S S .S
jjrS rS rS r S r
Sr g )parallel_config a4 Set the default backend or configuration for :class:`~joblib.Parallel`.
This is an alternative to directly passing keyword arguments to the
:class:`~joblib.Parallel` class constructor. It is particularly useful when
calling into library code that uses joblib internally but does not expose
the various parallel configuration arguments in its own API.
Parameters
----------
backend: str or ParallelBackendBase instance, default=None
If ``backend`` is a string it must match a previously registered
implementation using the :func:`~register_parallel_backend` function.
By default the following backends are available:
- 'loky': single-host, process-based parallelism (used by default),
- 'threading': single-host, thread-based parallelism,
- 'multiprocessing': legacy single-host, process-based parallelism.
'loky' is recommended to run functions that manipulate Python objects.
'threading' is a low-overhead alternative that is most efficient for
functions that release the Global Interpreter Lock: e.g. I/O-bound
code or CPU-bound code in a few calls to native code that explicitly
releases the GIL. Note that on some rare systems (such as pyodide),
multiprocessing and loky may not be available, in which case joblib
defaults to threading.
In addition, if the ``dask`` and ``distributed`` Python packages are
installed, it is possible to use the 'dask' backend for better
scheduling of nested parallel calls without over-subscription and
potentially distribute parallel calls over a networked cluster of
several hosts.
It is also possible to use the distributed 'ray' backend for
distributing the workload to a cluster of nodes. See more details
in the Examples section below.
Alternatively the backend can be passed directly as an instance.
n_jobs: int, default=None
The maximum number of concurrently running jobs, such as the number
of Python worker processes when ``backend="loky"`` or the size of the
thread-pool when ``backend="threading"``.
This argument is converted to an integer, rounded below for float.
If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
``n_cpus`` is obtained with :func:`~cpu_count`.
For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
using ``n_jobs=-2`` will result in all CPUs but one being used.
This argument can also go above ``n_cpus``, which will cause
oversubscription. In some cases, slight oversubscription can be
beneficial, e.g., for tasks with large I/O operations.
If 1 is given, no parallel computing code is used at all, and the
behavior amounts to a simple python `for` loop. This mode is not
compatible with `timeout`.
None is a marker for 'unset' that will be interpreted as n_jobs=1
unless the call is performed under a :func:`~parallel_config`
context manager that sets another value for ``n_jobs``.
If n_jobs = 0 then a ValueError is raised.
verbose: int, default=0
The verbosity level: if non zero, progress messages are
printed. Above 50, the output is sent to stdout.
The frequency of the messages increases with the verbosity level.
If it more than 10, all iterations are reported.
temp_folder: str or None, default=None
Folder to be used by the pool for memmapping large arrays
for sharing memory with worker processes. If None, this will try in
order:
- a folder pointed by the ``JOBLIB_TEMP_FOLDER`` environment
variable,
- ``/dev/shm`` if the folder exists and is writable: this is a
RAM disk filesystem available by default on modern Linux
distributions,
- the default system temporary folder that can be
overridden with ``TMP``, ``TMPDIR`` or ``TEMP`` environment
variables, typically ``/tmp`` under Unix operating systems.
max_nbytes: int, str, or None, optional, default='1M'
Threshold on the size of arrays passed to the workers that
triggers automated memory mapping in temp_folder. Can be an int
in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
Use None to disable memmapping of large arrays.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
Memmapping mode for numpy arrays passed to workers. None will
disable memmapping, other modes defined in the numpy.memmap doc:
https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
Also, see 'max_nbytes' parameter documentation for more details.
prefer: str in {'processes', 'threads'} or None, default=None
Soft hint to choose the default backend.
The default process-based backend is 'loky' and the default
thread-based backend is 'threading'. Ignored if the ``backend``
parameter is specified.
require: 'sharedmem' or None, default=None
Hard constraint to select the backend. If set to 'sharedmem',
the selected backend will be single-host and thread-based.
inner_max_num_threads: int, default=None
If not None, overwrites the limit set on the number of threads
usable in some third-party library threadpools like OpenBLAS,
MKL or OpenMP. This is only used with the ``loky`` backend.
backend_params: dict
Additional parameters to pass to the backend constructor when
backend is a string.
Notes
-----
Joblib tries to limit the oversubscription by limiting the number of
threads usable in some third-party library threadpools like OpenBLAS, MKL
or OpenMP. The default limit in each worker is set to
``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
overwritten with the ``inner_max_num_threads`` argument which will be used
to set this limit in the child processes.
.. versionadded:: 1.3
Examples
--------
>>> from operator import neg
>>> with parallel_config(backend='threading'):
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
...
[-1, -2, -3, -4, -5]
To use the 'ray' joblib backend add the following lines:
>>> from ray.util.joblib import register_ray # doctest: +SKIP
>>> register_ray() # doctest: +SKIP
>>> with parallel_config(backend="ray"): # doctest: +SKIP
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
[-1, -2, -3, -4, -5]
r* r+ r, r- r. r/ r0 r1 N)r+ r, r- r. r/ r0 r1 inner_max_num_threadsc
[ [ S[ 5 U l U R " X40 U
D6nUUUUUUUUS.n U R R 5 U l U R R U R 5 V V
s0 s H u p[ U
[ 5 ( a M X_M sn
n 5 [ [ SU R 5 g s sn
n f )Nr? )r+ r, r- r. r/ r0 r1 r* ) rG rH r6 old_parallel_config_check_backendrR r\ updateitems
isinstancer setattr)selfr* r+ r, r- r. r/ r0 r1 r^ backend_params
new_configkvs r% __init__parallel_config.__init__j s $+8X?V#W %%gWW &$"
$ 7 7 < < > ## ( . . 0 Q 0da
1i8PTQT 0 Q
(D$8$89
Rs C
C
c U[ S L a Uc [ U5 S: a [ S5 eU$ [ U[ 5 ( a U[
; a U[ ; a [ U nU" 5 OpU[ ; a: [ R " SU S[ S3[ SS9 [
[ [
U' O,[ S U S
[ [
R 5 5 35 e[
U " S
0 UD6nO[ U5 S: a [ S 5 eUb7 UR R S 3nUR ( d U5 eX!l UR$ c0 U R& S nU[ S L a SnO UR$ nXql U$ )Nr* r zrinner_max_num_threads and other constructor parameters backend_params are only supported when backend is not None.joblib backend '3' is not available on your system, falling back to .
stacklevelInvalid backend: , expected one of zRConstructor parameters backend_params are only supported when backend is a string.z< does not accept setting the inner_max_num_threads argument. )r6 lenrJ rd strrL EXTERNAL_BACKENDSMAYBE_AVAILABLE_BACKENDSwarningswarnrM UserWarningsortedkeysrP rQ supports_inner_max_num_threadsr^ rC r` )rf r* r^ rg registerr$ parent_backendrC s r% ra parallel_config._check_backend s -i8 8$0C4G!4K 0
N gs # #h& // 0 9H J 88 MM*7) 488G7HK##$ )1 (AHW%$+G94F!(--/235
w'9.9G>"Q& :
! ,$$--. /2 2
99 >3 >9,A ) ( ! 5 5i @N !8!CC !
. < <
$1 !r; c U R $ N)r\ rf s r% __enter__parallel_config.__enter__ s ###r; c $ U R 5 g r )
unregister)rf typevalue tracebacks r% __exit__parallel_config.__exit__ s r; c : [ [ SU R 5 g )Nr? )re rH r` r s r% r parallel_config.unregister s (D$<$<=r; )r` r\ ) rQ
__module__ __qualname____firstlineno____doc__r6 rk ra r r r __static_attributes__rv r; r% r\ r\ sp IZ ( 2": 'x0' 2+M:*<8)+6&x0' 2 "":H8t$>r; r\ c 8 ^ \ rS rSrSr SU 4S jjrS rSrU =r$ )parallel_backendi a Change the default backend used by Parallel inside a with block.
.. warning::
It is advised to use the :class:`~joblib.parallel_config` context
manager instead, which allows more fine-grained control over the
backend configuration.
If ``backend`` is a string it must match a previously registered
implementation using the :func:`~register_parallel_backend` function.
By default the following backends are available:
- 'loky': single-host, process-based parallelism (used by default),
- 'threading': single-host, thread-based parallelism,
- 'multiprocessing': legacy single-host, process-based parallelism.
'loky' is recommended to run functions that manipulate Python objects.
'threading' is a low-overhead alternative that is most efficient for
functions that release the Global Interpreter Lock: e.g. I/O-bound code or
CPU-bound code in a few calls to native code that explicitly releases the
GIL. Note that on some rare systems (such as Pyodide),
multiprocessing and loky may not be available, in which case joblib
defaults to threading.
You can also use the `Dask `_ joblib
backend to distribute work across machines. This works well with
scikit-learn estimators with the ``n_jobs`` parameter, for example::
>>> import joblib # doctest: +SKIP
>>> from sklearn.model_selection import GridSearchCV # doctest: +SKIP
>>> from dask.distributed import Client, LocalCluster # doctest: +SKIP
>>> # create a local Dask cluster
>>> cluster = LocalCluster() # doctest: +SKIP
>>> client = Client(cluster) # doctest: +SKIP
>>> grid_search = GridSearchCV(estimator, param_grid, n_jobs=-1)
... # doctest: +SKIP
>>> with joblib.parallel_backend("dask", scatter=[X, y]): # doctest: +SKIP
... grid_search.fit(X, y)
It is also possible to use the distributed 'ray' backend for distributing
the workload to a cluster of nodes. To use the 'ray' joblib backend add
the following lines::
>>> from ray.util.joblib import register_ray # doctest: +SKIP
>>> register_ray() # doctest: +SKIP
>>> with parallel_backend("ray"): # doctest: +SKIP
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
[-1, -2, -3, -4, -5]
Alternatively the backend can be passed directly as an instance.
By default all available workers will be used (``n_jobs=-1``) unless the
caller passes an explicit value for the ``n_jobs`` parameter.
This is an alternative to passing a ``backend='backend_name'`` argument to
the :class:`~Parallel` class constructor. It is particularly useful when
calling into library code that uses joblib internally but does not expose
the backend argument in its own API.
>>> from operator import neg
>>> with parallel_backend('threading'):
... print(Parallel()(delayed(neg)(i + 1) for i in range(5)))
...
[-1, -2, -3, -4, -5]
Joblib also tries to limit the oversubscription by limiting the number of
threads usable in some third-party library threadpools like OpenBLAS, MKL
or OpenMP. The default limit in each worker is set to
``max(cpu_count() // effective_n_jobs, 1)`` but this limit can be
overwritten with the ``inner_max_num_threads`` argument which will be used
to set this limit in the child processes.
.. versionadded:: 0.10
See Also
--------
joblib.parallel_config: context manager to change the backend configuration.
c > [ TU ] " SUUUS.UD6 U R c S U l O#U R S U R S 4U l U R S U R S 4U l g )N)r* r+ r^ r* r+ rv )superrk r` old_backend_and_jobsr\ new_backend_and_jobs)rf r* r+ r^ rg rP s r% rk parallel_backend.__init__# s
"7
# # +(,D % ((3((2)D %
+ *%
!r; c U R $ r )r r s r% r parallel_backend.__enter__9 s (((r; )r r )N) rQ r r r r rk r r
__classcell__rP s @r% r r s Nb 9=
,) )r; r get_contextJOBLIB_START_METHOD )methodc 6 \ rS rSrSr S S jrS rS rS rSr g)
BatchedCallsiI zCWrap a sequence of (func, args, kwargs) tuples as a single callableNc [ U5 U l [ U R 5 U l X0l [ U[ 5 ( a Uu U l U l OUS sU l U l Ub X@l g 0 U l g r )
listrc rw _size_reducer_callbackrd tuplerH _n_jobs
_pickle_cache)rf iterator_slicebackend_and_jobsreducer_callback pickle_caches r% rk BatchedCalls.__init__L se .)
_
!1 & . .*: 'DM4< +;D 'DM4<-9-E\2r; c [ U R U R S9 U R VVVs/ s H
u po1" U0 UD6PM snnnsS S S 5 $ s snnnf ! , ( d f g = f)N)r* r+ )r\ rH r rc )rf funcargskwargss r% __call__BatchedCalls.__call__Z sQ T]]4<<
HDHJJOJ.@d&D$)&)JO I
HO I
Hs AA
AA
A'c U R b U R 5 [ U R U R U R 4S U R
44$ r )r r rc rH r r r s r%
__reduce__BatchedCalls.__reduce__` sL ! ! - " " $
ZZ$-- 6d>P>P Q
r; c U R $ r )r r s r% __len__BatchedCalls.__len__i s zzr; )rH r r r r rc )NN)
rQ r r r r rk r r r r rv r; r% r r I s" M UY N P
r; r DoneErrorPendingc : [ c g[ R " U S9$ )a Return the number of CPUs.
This delegates to loky.cpu_count that takes into account additional
constraints such as Linux CFS scheduler quotas (typically set by container
runtimes such as docker) and CPU affinity (for instance using the taskset
command on Linux).
Parameters
----------
only_physical_cores : boolean, default=False
If True, does not take hyperthreading / SMT logical cores into account.
r only_physical_cores)r r cpu_countr s r% r r v s
z >>.A BBr; c U( d gUS: a gU S:X a gSSU-
S- - n[ X- 5 n[ U S- U- 5 n[ U5 [ U5 :H $ ) zReturns False for indices increasingly apart, the distance
depending on the value of verbose.
We use a lag increasing as the square of index
TrF Fr g ? rq r )r int)indexr, scale
next_scales r% _verbosity_filterr sc 2 zR'\a''G !EuqyG+,J z?c%j ((r; c p ^ U 4S jn [ R " T 5 " U5 nU$ ! [ a U$ f = f)z6Decorator used to capture the arguments of a function.c > TX4$ r rv )r r functions r% delayed_function!delayed..delayed_function s %%r; ) functoolswrapsAttributeError)r r s ` r% delayedr sD &;$??845EF ;: ;s '
55c N \ rS rSrSrS rS rS rS rS r S r
S r S
r S r
S rg
)BatchCompletionCallBacki a Callback to keep track of completed results and schedule the next tasks.
This callable is executed by the parent process whenever a worker process
has completed a batch of tasks.
It is used for progress reporting, to update estimate of the batch
processing duration and to schedule the next batch of tasks to be
processed.
It is assumed that this callback will always be triggered by the backend
right after the end of a task, in case of success as well as in case of
failure.
c Xl X l X0l UR U l S U l S U l UR R ( d S U l g [ U l g r ) dispatch_timestamp
batch_sizeparallel_call_idparallel_call_id_completion_timeout_counterjobrH supports_retrieve_callbackstatus TASK_PENDING)rf r r r s r% rk BatchCompletionCallBack.__init__ sR "4 $
( 1 1 +/( ;; DK 'DKr; c Xl g)z)Register the object returned by `submit`.N)r )rf r s r% register_job$BatchCompletionCallBack.register_job s r; c N U R R nUR ( a U R 5 $ UR U R
US9n[
U[ S9nU R U5 U R 5 $ ! [ a n[
U[ S9n SnAN>SnAff = f)a Returns the raw result of the task that was submitted.
If the task raised an exception rather than returning, this same
exception will be raised instead.
If the backend supports the retrieval callback, it is assumed that this
method is only called after the result has been registered. It is
ensured by checking that `self.status(timeout)` does not return
TASK_PENDING. In this case, `get_result` directly returns the
registered result (or raise the registered exception).
For other backends, there are no such assumptions, but `get_result`
still needs to synchronously retrieve the result before it can
return it or raise. It will block at most `self.timeout` seconds
waiting for retrieval to complete, after that it raises a TimeoutError.
timeoutresultr N) r rH r _return_or_raiseretrieve_resultr dict TASK_DONE
BaseException
TASK_ERROR_register_outcome)rf r r* r outcomer# s r%
get_result"BatchCompletionCallBack.get_result s $ --(( - - ((* * 8 ,,TXXw,GF&;G
w'$$&& 8!J7G 8s (B
B$ B B$c n U R [ :X a U R eU R U ?$ ! U ?f = fr )r r _resultr s r% r (BatchCompletionCallBack._return_or_raise s/ {{j(ll"<< s +0 4c " Ub U R [ :w a U R $ [ R " 5 nU R c X l X R -
U: a' [ [ 5 [ S9nU R U5 U R $ )zGet the status of the task.
This function also checks if the timeout has been reached and register
the TimeoutError outcome when it is the case.
r )r r timer r r r r )rf r nowr s r%
get_status"BatchCompletionCallBack.get_status st ?dkk\ 9;; iik + + 3/2 , 22 2g =,.DG " "7 +{{r; c U R R R ( d U R 5 gU R R U R R
U R :w a
SSS5 gU R R ( a
SSS5 gU R " U0 UD6nSSS5 W( a U R 5 gg! , ( d f N'= f)z@Function called by the callback thread after a job is completed.N) r rH r
_dispatch_new_lockr r _aborting_retrieve_result)rf r r
job_succeededs r% r BatchCompletionCallBack.__call__! s }}%%@@
]]
}}%%)>)>>
!
}}&& !
! 1 14 B6 BM !" # !
s
&C< C!C
C"c [ R " 5 U R -
nU R R R U R
U5 U R R U R =R U R
-
sl U R R 5 U R R b U R R 5 SSS5 g! , ( d f g= f)z1Schedule the next batch of tasks to be processed.N) r r r rH batch_completedr r n_completed_tasksprint_progress_original_iterator
dispatch_next)rf this_batch_durations r% r %BatchCompletionCallBack._dispatch_newB s #iikD,C,C C
..t@ST ]]
MM + +t > + MM ( ( *}}//;
++- !
s
)A5C''
C5c U R R R U5 n[ [ US9nU R U5 US [ :g $ ! [
a nSUl [ U[ S9n SnANASnAff = f)zFetch and register the outcome of a task.
Return True if the task succeeded, False otherwise.
This function is only called by backends that support retrieving
the task result in the callback thread.
)r r Nr r ) r rH retrieve_result_callbackr r r
__traceback__r r )rf outr r r# s r% r (BatchCompletionCallBack._retrieve_resultP sr 8]]++DDSIF)F;G
w'x J..
8 "AO!J7G 8s 3A
A; A66A;c ( U R R U R [ S4; a
SSS5 gUS U l SSS5 US U l SU l U R [ :X a" SU R l SU R l U R R ( a gU R R U R R R U 5 SSS5 g! , ( d f N= f! , ( d f g= f)zdRegister the outcome of a task.
This method can be called only once, future calls will be ignored.
Nr r T) r r r r r r r
_exceptionr return_ordered_jobsappend)rf r s r% r )BatchCompletionCallBack._register_outcomef s ]]
{{<"66 !
"(+DK !
x( ;;* $'+DMM $&*DMM # == ' '
]]
MM & &t , !
) !
( !
s C2
C2&D2
D
D)r r r r r r r r N)rQ r r r r rk r r r r r r r r r rv r; r% r r s6 "'0"'H 0 !B . /, -r; r c * U[ U ' U( a U qgg)a Register a new Parallel backend factory.
The new backend can then be selected by passing its name as the backend
argument to the :class:`~Parallel` class. Moreover, the default backend can
be overwritten globally by setting make_default=True.
The factory can be any callable that takes no argument and return an
instance of ``ParallelBackendBase``.
Warning: this function is experimental and subject to change in a future
version of joblib.
.. versionadded:: 0.10
N)rL rM )namefactory make_defaults r% r! r! s HTN r; c P U S:X a g[ 5 u pU c Un UR U S9$ )a- Determine the number of jobs that can actually run in parallel
n_jobs is the number of workers requested by the callers. Passing n_jobs=-1
means requesting all available workers for instance matching the number of
CPU cores on the worker host(s).
This method should return a guesstimate of the number of workers that can
actually perform work concurrently with the currently enabled default
backend. The primary use case is to make it possible for the caller to know
in how many chunks to slice the work.
In general working on larger data chunks is more efficient (less scheduling
overhead and better use of CPU cache prefetching heuristics) as long as all
the workers have enough work to do.
Warning: this function is experimental and subject to change in a future
version of joblib.
.. versionadded:: 0.10
r )r+ )r@ effective_n_jobs)r+ r* backend_n_jobss r% r r s7 * { 0 2G
~ # #6 # 22r; c
^ \ rS rSrSr\S \S S\S SSS \S
\S \S \S
\S 4 U 4S jjrS rS rS r S r
S r S r S r
S rS rS rS rS rS rS rS rS rS rS! rS" rS# rS$ rS% r S& r S' r S(r U =r $ ))Paralleli aJ, Helper class for readable parallel mapping.
Read more in the :ref:`User Guide `.
Parameters
----------
n_jobs: int, default=None
The maximum number of concurrently running jobs, such as the number
of Python worker processes when ``backend="loky"`` or the size of
the thread-pool when ``backend="threading"``.
This argument is converted to an integer, rounded below for float.
If -1 is given, `joblib` tries to use all CPUs. The number of CPUs
``n_cpus`` is obtained with :func:`~cpu_count`.
For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For instance,
using ``n_jobs=-2`` will result in all CPUs but one being used.
This argument can also go above ``n_cpus``, which will cause
oversubscription. In some cases, slight oversubscription can be
beneficial, e.g., for tasks with large I/O operations.
If 1 is given, no parallel computing code is used at all, and the
behavior amounts to a simple python `for` loop. This mode is not
compatible with ``timeout``.
None is a marker for 'unset' that will be interpreted as n_jobs=1
unless the call is performed under a :func:`~parallel_config`
context manager that sets another value for ``n_jobs``.
If n_jobs = 0 then a ValueError is raised.
backend: str, ParallelBackendBase instance or None, default='loky'
Specify the parallelization backend implementation.
Supported backends are:
- "loky" used by default, can induce some
communication and memory overhead when exchanging input and
output data with the worker Python processes. On some rare
systems (such as Pyiodide), the loky backend may not be
available.
- "multiprocessing" previous process-based backend based on
`multiprocessing.Pool`. Less robust than `loky`.
- "threading" is a very low-overhead backend but it suffers
from the Python Global Interpreter Lock if the called function
relies a lot on Python objects. "threading" is mostly useful
when the execution bottleneck is a compiled extension that
explicitly releases the GIL (for instance a Cython loop wrapped
in a "with nogil" block or an expensive call to a library such
as NumPy).
- finally, you can register backends by calling
:func:`~register_parallel_backend`. This will allow you to
implement a backend of your liking.
It is not recommended to hard-code the backend name in a call to
:class:`~Parallel` in a library. Instead it is recommended to set
soft hints (prefer) or hard constraints (require) so as to make it
possible for library users to change the backend from the outside
using the :func:`~parallel_config` context manager.
return_as: str in {'list', 'generator', 'generator_unordered'}, default='list'
If 'list', calls to this instance will return a list, only when
all results have been processed and retrieved.
If 'generator', it will return a generator that yields the results
as soon as they are available, in the order the tasks have been
submitted with.
If 'generator_unordered', the generator will immediately yield
available results independently of the submission order. The output
order is not deterministic in this case because it depends on the
concurrency of the workers.
prefer: str in {'processes', 'threads'} or None, default=None
Soft hint to choose the default backend if no specific backend
was selected with the :func:`~parallel_config` context manager.
The default process-based backend is 'loky' and the default
thread-based backend is 'threading'. Ignored if the ``backend``
parameter is specified.
require: 'sharedmem' or None, default=None
Hard constraint to select the backend. If set to 'sharedmem',
the selected backend will be single-host and thread-based even
if the user asked for a non-thread based backend with
:func:`~joblib.parallel_config`.
verbose: int, default=0
The verbosity level: if non zero, progress messages are
printed. Above 50, the output is sent to stdout.
The frequency of the messages increases with the verbosity level.
If it more than 10, all iterations are reported.
timeout: float or None, default=None
Timeout limit for each task to complete. If any task takes longer
a TimeOutError will be raised. Only applied when n_jobs != 1
pre_dispatch: {'all', integer, or expression, as in '3*n_jobs'}, default='2*n_jobs'
The number of batches (of tasks) to be pre-dispatched.
Default is '2*n_jobs'. When batch_size="auto" this is reasonable
default and the workers should never starve. Note that only basic
arithmetic are allowed here and no modules can be used in this
expression.
batch_size: int or 'auto', default='auto'
The number of atomic tasks to dispatch at once to each
worker. When individual evaluations are very fast, dispatching
calls to workers can be slower than sequential computation because
of the overhead. Batching fast computations together can mitigate
this.
The ``'auto'`` strategy keeps track of the time it takes for a
batch to complete, and dynamically adjusts the batch size to keep
the time on the order of half a second, using a heuristic. The
initial batch size is 1.
``batch_size="auto"`` with ``backend="threading"`` will dispatch
batches of a single task at a time as the threading backend has
very little overhead and using larger batch size has not proved to
bring any gain in that case.
temp_folder: str or None, default=None
Folder to be used by the pool for memmapping large arrays
for sharing memory with worker processes. If None, this will try in
order:
- a folder pointed by the JOBLIB_TEMP_FOLDER environment
variable,
- /dev/shm if the folder exists and is writable: this is a
RAM disk filesystem available by default on modern Linux
distributions,
- the default system temporary folder that can be
overridden with TMP, TMPDIR or TEMP environment
variables, typically /tmp under Unix operating systems.
Only active when ``backend="loky"`` or ``"multiprocessing"``.
max_nbytes int, str, or None, optional, default='1M'
Threshold on the size of arrays passed to the workers that
triggers automated memory mapping in temp_folder. Can be an int
in Bytes, or a human-readable string, e.g., '1M' for 1 megabyte.
Use None to disable memmapping of large arrays.
Only active when ``backend="loky"`` or ``"multiprocessing"``.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, default='r'
Memmapping mode for numpy arrays passed to workers. None will
disable memmapping, other modes defined in the numpy.memmap doc:
https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
Also, see 'max_nbytes' parameter documentation for more details.
backend_kwargs: dict, optional
Additional parameters to pass to the backend `configure` method.
Notes
-----
This object uses workers to compute in parallel the application of a
function to many different arguments. The main functionality it brings
in addition to using the raw multiprocessing or concurrent.futures API
are (see examples for details):
* More readable code, in particular since it avoids
constructing list of arguments.
* Easier debugging:
- informative tracebacks even when the error happens on
the client side
- using 'n_jobs=1' enables to turn off parallel computing
for debugging without changing the codepath
- early capture of pickling errors
* An optional progress meter.
* Interruption of multiprocesses jobs with 'Ctrl-C'
* Flexible pickling control for the communication to and from
the worker processes.
* Ability to use shared memory efficiently with worker
processes for large numpy-based datastructures.
Note that the intended usage is to run one call at a time. Multiple
calls to the same Parallel object will result in a ``RuntimeError``
Examples
--------
A simple example:
>>> from math import sqrt
>>> from joblib import Parallel, delayed
>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
Reshaping the output when the function has several return
values:
>>> from math import modf
>>> from joblib import Parallel, delayed
>>> r = Parallel(n_jobs=1)(delayed(modf)(i/2.) for i in range(10))
>>> res, i = zip(*r)
>>> res
(0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5, 0.0, 0.5)
>>> i
(0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0, 4.0, 4.0)
The progress meter: the higher the value of `verbose`, the more
messages:
>>> from time import sleep
>>> from joblib import Parallel, delayed
>>> r = Parallel(n_jobs=2, verbose=10)(
... delayed(sleep)(.2) for _ in range(10)) #doctest: +SKIP
[Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 0.6s
[Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 0.8s
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 1.4s finished
Traceback example, note how the line of the error is indicated
as well as the values of the parameter passed to the function that
triggered the exception, even though the traceback happens in the
child process:
>>> from heapq import nlargest
>>> from joblib import Parallel, delayed
>>> Parallel(n_jobs=2)(
... delayed(nlargest)(2, n) for n in (range(4), 'abcde', 3))
... # doctest: +SKIP
-----------------------------------------------------------------------
Sub-process traceback:
-----------------------------------------------------------------------
TypeError Mon Nov 12 11:37:46 2012
PID: 12934 Python 2.7.3: /usr/bin/python
........................................................................
/usr/lib/python2.7/heapq.pyc in nlargest(n=2, iterable=3, key=None)
419 if n >= size:
420 return sorted(iterable, key=key, reverse=True)[:n]
421
422 # When key is none, use simpler decoration
423 if key is None:
--> 424 it = izip(iterable, count(0,-1)) # decorate
425 result = _nlargest(n, it)
426 return map(itemgetter(0), result) # undecorate
427
428 # General case, slowest method
TypeError: izip argument #1 must support iteration
_______________________________________________________________________
Using pre_dispatch in a producer/consumer situation, where the
data is generated on the fly. Note how the producer is first
called 3 times before the parallel loop is initiated, and then
called to generate new data on the fly:
>>> from math import sqrt
>>> from joblib import Parallel, delayed
>>> def producer():
... for i in range(6):
... print('Produced %s' % i)
... yield i
>>> out = Parallel(n_jobs=2, verbose=100, pre_dispatch='1.5*n_jobs')(
... delayed(sqrt)(i) for i in producer()) #doctest: +SKIP
Produced 0
Produced 1
Produced 2
[Parallel(n_jobs=2)]: Done 1 jobs | elapsed: 0.0s
Produced 3
[Parallel(n_jobs=2)]: Done 2 jobs | elapsed: 0.0s
Produced 4
[Parallel(n_jobs=2)]: Done 3 jobs | elapsed: 0.0s
Produced 5
[Parallel(n_jobs=2)]: Done 4 jobs | elapsed: 0.0s
[Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s remaining: 0.0s
[Parallel(n_jobs=2)]: Done 6 out of 6 | elapsed: 0.0s finished
r+ r* r r, Nz
2 * n_jobsautor- r. r/ r0 r1 c
> [ TU ] 5 Uc [ S n[ XUS9u pUR n[ XOS5 U l XPl X`l US; a [ SU S35 eX0l
US:g U l US:g U l 0 U
EU S 4US
4U
S 4U S 4U S
4US44 VVs0 s H u nnU[ UUU5 _M snnEU l
[ U R S [ 5 ( a% [! U R S 5 U R S ' [# SU R S S-
5 U R S' [$ b [$ U R S' O7[' [( S5 ( a" [( R* " 5 U R S' U[ S L d Uc UnO[ U[, 5 ( a UR c UUl O[' US5 ( a) [' US5 ( a X R S' [/ US9nOoU[0 ; aU U[2 ; aK [4 R6 " SU S[8 S3[: SS9 [0 [8 [0 U' [0 [8 " US9nO [0 U nU" US9n[ X S5 nUc URB n [E U5 nXl# U S :X a [I US S 5 ( d [ S!U- 5 eUS":X d [ U[J 5 ( a
US: a Xpl& O[ S#U- 5 e[ U[N 5 ( d U R ( a+ URP ( d [ S$RS X#5 5 e[T RV " 5 U l, [Z R\ " 5 U l/ [a 5 U l1 [e 5 U l3 [h Rj " 5 U l6 S U l7 X l8 S U l9 S U l: [w 5 Rx U l= S U l> g s snnf ! [< a4 n[ SU< S [? [0 RA 5 5 < 35 UeS nAff = f! [ a [ S 5 ef = f)%Nr+ )r0 r1 r, r, > r generatorgenerator_unorderedzlExpected `return_as` parameter to be a string equal to "list","generator" or "generator_unordered", but got z instead.r r& r. r- r/ r0 r1 r 2 contextr r* PoolLockrB rn ro rp rq rr rt ru z$n_jobs could not be converted to intr4 rE Fz)Backend %s does not support shared memoryr# z8batch_size must be 'auto' or a positive integer, got: %rz(Backend {} does not support return_as={})?r rk r6 r> rC r: r, r pre_dispatchrJ return_asreturn_generatorr _backend_kwargsrd rx r maxDEFAULT_MP_CONTEXThasattrr r r r rL rz r{ r| rM r} KeyErrorr~ r default_n_jobsr r+ rG r r r supports_return_generatorformatr RLockr collectionsdequer set _jobs_setr _pending_outputsqueueQueue_ready_batchesr rH _running_managed_backendr
hex_id _call_ref)rf r+ r* r, r, r r+ r r- r. r/ r0 r1 backend_kwargsactive_backendr8 rC r7 ri backend_factoryr# rP s r% rk Parallel.__init__ sM " >,X6F)<G*
& '44
()L ( H H BBK M
# )V 3 '+@ @
. -0 ,X& i( i(
!
!HE1 $UNA>>!
d**<8# > >1@$$\22D . +.a1E1Ei1PSU1U*V Y' ).@D +
R
'
'.0nn.>D + -i8 8GO$G
!4
5
5 $$,(5%
Wf
%
%''6*B*B /6 +,=IG
H
$4L)L MM"7) ,00?/@C
!) 9HW /mLG
"*7"3 &MBG"68D > ++F E[F k !'';OQV*W*W H7 RS S :j(#C#C
UV (O JZW
'#455$$W-N-N >EE # *DJ$**,DJ UDN$(FD !"'++-D %)D "
% 7;; M
j
x}} 79
E CD D Es* O( O. P/ .
P,8/P''P,/Qc B SU l SU l U R 5 U $ )NTF)r@ _calling_initialize_backendr s r% r Parallel.__enter__S s" $
" r; c SU l U R ( a! U R ( a U R 5 U R 5 g )NF)r@ r- rI _abort_terminate_and_reset)rf exc_type exc_valuer s r% r Parallel.__exit__Y s. % T]] KKM !!#r; c U R R " SU R U S.U R D6nU R bi U R R
( dN [ R " SR U R R R U R 5 5 U$ ! [ a, nUR U l U R 5 n SnAU$ SnAff = f)z?Build a process or thread pool and return the number of workers)r+ r NzThe backend class {!r} does not support timeout. You have set 'timeout={}' in Parallel but the 'timeout' parameter will not be used.rv )rH configurer+ r. r supports_timeoutr{ r| r5 rP rQ r r* rJ )rf r+ r# s r% rJ Parallel._initialize_backend_ s 0]],, {{T595I5IF ||'
0N0N
@@F
//88$,,A
! 0 IIDM--/F
0s B)B- -
C#7!C C#c p U R ( a% U R R U R 5 $ g)Nr )rH r r+ r s r% _effective_n_jobsParallel._effective_n_jobsu s% ====11$++> >r; c [ U R S5 ( a+ U R ( a U R R 5 SU l U R ( d U R R 5 g g )N stop_callF)r1 rH rI rZ r@ terminater s r% rN Parallel._terminate_and_resetz sN 4==+ . .4== MM # # %
$$ MM # # % %r; c J U R ( a g[ U5 nU =R U-
sl U =R S-
sl [ R " 5 n[ X2U 5 nU R
U5 U R R XS9nUR U5 g)zQueue the batch for computing, with or without multiprocessing
WARNING: this method is not thread-safe: it should be only called
indirectly via dispatch_one_batch.
Nr )callback)
r rw n_dispatched_tasksn_dispatched_batchesr r _register_new_jobrH submitr )rf batchr r
batch_trackerr s r% _dispatchParallel._dispatch s >> Z
:- !!Q&! !YY[/0BPTU
}- mm""5"A""3'r; c U R ( a U R R U5 g U R R U5 g r )r r r r: add)rf rd s r% ra Parallel._register_new_job s/ JJ m , NN } -r; c b U R U R 5 ( d SU l SU l gg)zDispatch more data for parallel processing
This method is meant to be called concurrently by the multiprocessing
callback. We rely on the thread-safety of dispatch_one_batch to protect
against concurrent consumption of the unprotected iterator.
FN)dispatch_one_batchr
_iteratingr s r% r Parallel.dispatch_next s0 &&t'>'>?? #DO&*D # @r; c
U R ( a gU R 5 nU R U R R SS9n[) U5 S:X a
SSS5 gU R= U5 SSS5 g! [
R Ga U R nX$- n [ [ R " X5 5 nO! [ a} n[ UR [
R 5 ( a SUl [ SX 5 nU R! U5 UR# [% U[& S95 SnA SSS5 gSnAff = f[) U5 S:X a
SSS5 gXR* L a+ [) U5 U: a [- S[) U5 SU- - 5 n O[- S[) U5 U- 5 n [/ S[) U5 U 5 H\ n
[1 XjX- U R2 R5 5 U R6 U R8 5 nU R R; U5 M^ U R R SS9n GN f = f! , ( d f g= f) a, Prefetch the tasks for the next batch and dispatch them.
The effective size of the batch is computed here.
If there are no more jobs to dispatch, return False, else return True.
The iterator consumption and dispatching is protected by the same
lock so calling this function should be thread safe.
F)blockNr r Tr rF ) r _get_batch_sizer r> getr< Empty_cached_effective_n_jobsr itertoolsislice Exceptionrd __context__ __cause__r ra r r r rw r r/ ranger rH get_nested_backendr r putre ) rf iteratorr tasksr+ big_batch_sizeru r# rd final_batch_sizeis r% rk Parallel.dispatch_one_batch s >>))+
ZZ7
= ++//e/<n 5zQ C ZF u%I Z ;; 5
= 66!+!4 !)"2"28"L MF
"!--==
'+ $;Az$PM**=9!33D*4UV G Z* v;!# M ZP 7 77CK.
I#I>%I $ CI
E A*E8I <IE I I'C I I
I I
I c n U R S:X a U R R 5 $ U R $ )z-Returns the effective batch size for dispatchr# )r rH compute_batch_sizer s r% rp Parallel._get_batch_size s- ??f $==335 5 ?? "r; c U R ( d gU R S: a [ R R nO[ R R nU" SU SU S35 g)z=Display the message on stout or stderr depending on verbosityNr' [z]:
)r, sysstderrwritestdout)rf r$ writers r% _printParallel._print sO || <<" ZZ%%FZZ%%F4&C5#$r; c U R U R :H =( a$ U R =( d U R ( + $ )z&Check if all tasks have been completed)r r_ rl r r s r%
_is_completedParallel._is_completed s6 %%)@)@@
OO -t~~F
r; c z U R ( d gU R b2 U R S: a" [ [ U R 5 5 S- nOSn[ R " 5 U R
-
nU R
5 ( aA U R SU R U S3 SU R U S3 S[ U5 S 35 gU R b [ U R U R 5 ( a gS
[ U5 3nU R nU R b, U R SXA S3 SU R U S3 S U 35 gS [ S 5 U- [ S
5 -
- nU R SXA S3 SU U 35 gU R nU R nUS:w a> Xd-
S- U R -
nX`R - S- nUS- U:H n U ( d Xx- ( a gU[! US5 - U R U-
- n
U R SXA S3 SXa S3 S[ U5 S[ U
5 35 g)zcDisplay the process of the parallel execution only a fraction
of time, controlled by self.verbose.
Nr r zDone dz out of z | elapsed: z finishedz | elapsed: zout of r} z tasks z remaining: )r, n_tasksr r r _start_timer r r r r r r` rw r_ _pre_dispatch_amountr/ ) rf width elapsed_timefmt_timer pad total_taskscursor frequency is_last_itemremaining_times r% r
Parallel.print_progress sN
|| << # q(8% -.2EEyy{T%5%55 KK..wai8))5'3<$\239>
$
$
0 !:!:DLLII$%6|%D$EFH**E||' E'+8DLL% 3J!H:V S^e 3c'l BC eE'#373%z JK**E 11Kz %,q043L3LL (LL 8A = $qyK 7 6#5 *S] :''%/ N
KKgQY'x G19/E F / = >l$^457
r; c SU l U R nU R ( d, [ US5 ( a U R nUR US9 SU l g )NTabort_everything) ensure_ready)r rH _abortedr1 r@ r )rf r* r s r% rM Parallel._abort_ sM --}}2D!E!E 00L $ $, $ ?
r; c SU l U R U5 ( a U R S LU l U R U5 ( a U R U5 ( a M US:X a SU l g g )NFall)rl rk r )rf r| r+ s r% _startParallel._startp sm " "8 , , " 5 5T ADO%%h// %%h// 5 $DO !r; c # ^ # [ R " 5 nSn U R X5 Sv U R R 5 U R 5 Sh vN SSS5 U R ( a / O U R n[ R " 5 U l [ 5 U l SU l U( d U R% 5 [/ U5 S : aJ UR1 5 nUR3 U R4 5 nU H nUv M [/ U5 S : a MI gg N! , ( d f N= f! [ Ga SU l U[ R " 5 :w a [ R " S5 SnU m " U 4S jS[ R 5 nU" SS9R 5 U R ( a / O U R n[ R " 5 U l [ 5 U l SU l U( d U R% 5 ggU R' 5 U R( ( a U R+ 5 e [, a SU l U R' 5 e f = f! U R ( a / O U R n[ R " 5 U l [ 5 U l SU l U( d U R% 5 f f = f7f)
z?Iterator returning the tasks' output as soon as they are ready.FNTa A generator produced by joblib.Parallel has been gc'ed in an unexpected thread. This behavior should not cause major -issues but to make sure, please report this warning and your use case at https://github.com/joblib/joblib/issues so it can be investigated.c " > \ rS rSrU 4S jrSrg)3Parallel._get_outputs.._GeneratorExitThreadi c > TR 5 TR ( a TR 5 TR 5 g r )rM r- _warn_exit_earlyrN )rf _parallels r% run7Parallel._get_outputs.._GeneratorExitThread.run s0 !((*$55 % 6 6 8!668r; rv N)rQ r r r r r )r s r% _GeneratorExitThreadr s
9 9r; r GeneratorExitThread)r r )r get_identr rH retrieval_context _retrieve
GeneratorExitr r{ r| Threadstartr r7 r8 r9 r: r? rN rM r- r r rw popleftr r )
rf r| r+ dispatch_thread_iddetach_generator_exitr _remaining_outputsbatched_resultsr r s
@r% _get_outputsParallel._get_outputs s< & 0 0 2 % C , KK /
002 >>+++ 3n (,DJJ $**,DJ UDN !DM())+$%) 0 8 8 :O - 8 8 FO) * $%){ , 32 * #DO "Y%8%8%::
' )-% 99+;+; 9 %*?@FFH$ (,DJJ $**,DJ UDN !DM())+ )%
KKM$$%%' "DO KKM (,DJJ $**,DJ UDN !DM())+ )s{ K /D? D. D, D.$D? ,B-Parallel._raise_error_fast..; s G
jjJ.F
s & &N)r r r r r )rf error_jobs r% r Parallel._raise_error_fast5 sO ZZ G
GI . ! Zs A
A!c " U R U R -
nU R 5 nSnU( a UU S3-
nU( d US-
nU( d UU R U R -
S3-
nU( a US-
n[ R
" U5 gg)z?Warn the user if the generator is gc'ed before being consumned.r z5 tasks have been successfully executed but not used.z Additionally, zK tasks which were still being processed by the workers have been cancelled.z` You could benefit from adjusting the input task iterator to limit unnecessary computation time.N)r r r r_ r{ r| )rf
ready_outputs is_completedr$ s r% r Parallel._warn_exit_earlyD s ..1B1BB
))+ /!VW
C (( **T-C-CCD E
C B
C
MM#
r; c # ^^# SU l Xl U R 5 mTS:w a% [ U5 m[ UU4S jS5 nS U 5 nSv U Hv u p4nU =R S-
sl U =R
S-
sl U" U0 UD6nU =R S-
sl U R 5 Uv U =R S-
sl Mx SU l
SU l SU l U R 5 g! [ a SU l
SU l SU l e f = f! SU l
SU l SU l U R 5 f = f7f)zSeparate loop for sequential output.
This simplifies the traceback in case of errors and reduces the
overhead of calling sequential tasks with `joblib`.
Tr c D > [ [ R " TT 5 5 $ r )r rt ru )r its r% 1Parallel._get_sequential_output..m s E)"2"22z"B Cr; rv c 3 6 # U H o H o"v M M g 7fr rv )r rc tasks r% r 2Parallel._get_sequential_output..o s Q.>U54D5D.>s NF)rl r rp r r` r_ r r
r r r r r r? ) rf iterableiterable_batchedr r r resr r s @@r% _get_sequential_output Parallel._get_sequential_output_ sA " " "DO&. # --/JQ(^#'CR$ R.>Q J '/"F))Q.)''1,'D+F+&&!+&##% !!Q&! '/ "DM #DO&*D # ! "DO !DN DM "DM #DO&*D # !s) ECC5 &E5!DD 'E Ec 6 [ U S[ 5 5 U R ( a! SnU R SL a US-
n[ U5 eSU l SSS5 SU l SU l SU l SU l SU l SU l
SU l g! , ( d f N@= f)z9Reset the counters and flags used to track the execution.r z+This Parallel instance is already running !Tz Before submitting new tasks, you must wait for the completion of all the previous tasks, or clean all references to the output generator.Nr F) rG r r? r- RuntimeErrorr` r_ r r r r r )rf r$ s r% _reset_run_tracking Parallel._reset_run_tracking s
T7KM
2}}C((D0>C
#3'' DM 3 %&!"# !"
A 3
2s :B
Bc ^ T R 5 [ US5 ( a [ U5 OST l [ R " 5 T l T R ( d T R 5 nOT R 5 nUS:X a: T R U5 n[ U5 T R ( a U$ [ U5 $ T R [ 5 R T l SSS5 UT l [% T R& [( 5 ( a
U 4S jnUT l UT l T R& R, R. nUS:X a [1 SU- 5 eT R3 SU SU S 35 [ T R& S
5 ( a T R& R5 5 S T l [9 U5 nT R: nUS :X a ST l ST l OoUT l [ US
5 ( a$ [A URC S[E U5 5 5 n[G U5 =T l n[H RJ " UT R> 5 n[M 5 T l' T RQ Xg5 n[R RT " U5 T l+ [ U5 T R ( a U$ [ U5 $ ! , ( d f GN= f)z)Main function to dispatch parallel tasks.r Nr c x > T R R R R T R 5 g r )rH _workers_temp_folder_managerset_current_contextrB r s r% _batched_calls_reducer_callback:Parallel.__call__.._batched_calls_reducer_callback s*
&&;;OOHHr; r z%s has no active worker.zUsing backend z with z concurrent workers.
start_callTr endswithr+ ),r r1 rw r r r r@ rJ rW r r r- r r r
rA r rs rd rH r r rP rQ r r r rI r r+ r r r replacerx r rt ru r r r weakrefrefrC )rf r r+ outputr backend_namer| r+ s ` r% r Parallel.__call__ s4
"(/)(D(Ds8}$ 99;$$--/F++-F Q; 00:F L ! 2 26 DV D ZZ !GKKDM
)/% dmm[ 1 1
&ED " )/%}}..77 Q; 9L HI I n\N&@TUV 4==, / / MM $ $ &
>(( 5 &*D #()D %&.D #|Z00 ( )=)=hF )T U 7:<7H HD % !''$2K2KLH "V""8: V,
V ..v@DL@_ Zs
J00
J?c R U R R < SU R < S3$ )Nz(n_jobs=))rP rQ r+ r s r% __repr__Parallel.__repr__ s "&.."9"94;;GGr; )$r r rH r. rs r rC rI r rB rl r r: r r@ r r r; r r r> r r? r r r r` r_ r+ r r+ r, r- r r r, )!rQ r r r r r6 rk r r rJ rW rN re ra r rk rp r r r
rM r r r r r r r r r r r r r s @r% r" r" s {~ 'x0' 2' 2!+M:*<8)+6&x0' 2V p $ ,
& (<. +T l#
%
=~ "$*M ^6D L
/ 6("T% NgARH Hr; r" )F)r )Qr
__future__r r7 r rt osr< r r r r{ r
contextlibr mathr r r r r numbersr uuidr
_multiprocessing_helpersr _parallel_backendsr
r r r r r r _utilsr r diskr loggerr r rL rM rN rS rz externalsr localrH r& ry r6 rI rK r: r@ r> r\ r r0 r1 environrq stripr r objectr r r r r r r r r! r r" rv r; r% r s,
" # # ( ( ) ! - "# $ -v6 >"8H
"HV O ??
&" N t,d+q) 40$/-d+t, 5 / ( #8 , #I . #I . #8 , #I . #I .O#dq> q>hh) h)`
2}
ZZ^^ 12
6
<
<
>
F$F
^^6 :!6 !J
C0)&
V-f V-t *3>]Hv ]Hr;