o ^Zho*@sddlZddlZddlZddlZddlmZmZddlm Z ddl m Z dgZ e ZdadadaddZ      dd dZGdddeZdS)N)ProcessPoolExecutorEXTRA_QUEUED_CALLS) cpu_count) get_contextget_reusable_executorcCs8tt}td7a|WdS1swYdS)zEnsure that each successive executor instance has a unique, monotonic id. The purpose of this monotonic id is to help debug and test automated instance creation. rN)_executor_lock_next_executor_id) executor_idr V/var/www/auris/lib/python3.10/site-packages/joblib/externals/loky/reusable_executor.py_get_next_executor_ids $r Fautor c Cs&tj|||||||||| d \} } | S)aReturn the current ReusableExectutor instance. Start a new instance if it has not been started already or if the previous instance was left in a broken state. If the previous instance does not have the requested number of workers, the executor is dynamically resized to adjust the number of workers prior to returning. Reusing a singleton instance spares the overhead of starting new worker processes and importing common python packages each time. ``max_workers`` controls the maximum number of tasks that can be running in parallel in worker processes. By default this is set to the number of CPUs on the host. Setting ``timeout`` (in seconds) makes idle workers automatically shutdown so as to release system resources. New workers are respawn upon submission of new tasks so that ``max_workers`` are available to accept the newly submitted tasks. Setting ``timeout`` to around 100 times the time required to spawn new processes and import packages in them (on the order of 100ms) ensures that the overhead of spawning workers is negligible. Setting ``kill_workers=True`` makes it possible to forcibly interrupt previously spawned jobs to get a new instance of the reusable executor with new constructor argument values. The ``job_reducers`` and ``result_reducers`` are used to customize the pickling of tasks and results send to the executor. When provided, the ``initializer`` is run first in newly spawned processes with argument ``initargs``. The environment variable in the child process are a copy of the values in the main process. One can provide a dict ``{ENV: VAL}`` where ``ENV`` and ``VAL`` are string literals to overwrite the environment variable ``ENV`` in the child processes to value ``VAL``. The environment variables are set in the children before any module is loaded. This only works with the ``loky`` context. ) max_workerscontexttimeout kill_workersreuse job_reducersresult_reducers initializerinitargsenv)_ReusablePoolExecutorr) rrrrrrrrrr _executor_r r r r%s4  cszeZdZ         dfdd Ze          dd d Zfd d Zd dZddZfddZ Z S)rNrr c s,tj||||||| | d||_||_dS)N)rrrrrrrr)super__init__r _submit_resize_lock) selfZsubmit_resize_lockrrrr rrrrr __class__r r ris  z_ReusablePoolExecutor.__init__rFrc  Cstt} |dur|dur| dur| j}nt}n |dkr%td|dt|tr.t|}|dur<|dkrz0_ReusablePoolExecutor._resize..MbP?csr<r8r=r?r r r rBrC)rr)r(Z_executor_manager_thread_wait_job_completionZ_processes_management_locklistZ _processesvaluessumrangeZ _call_queueputlenr1r%timesleepZ_adjust_process_countall)r rZ processesZnb_children_aliverr r r r3s<     "z_ReusablePoolExecutor._resizecCsF|jrtdttjd|jd|jr!t d|jsdSdS)z8Wait for the cache to be empty before resizing the pool.z\Trying to resize an executor with running jobs: waiting for jobs completion before resizing.z Executor z, waiting for jobs completion before resizingrDN) Z_pending_work_itemswarningswarn UserWarningr-r.r/r rLrM)r r r r rE s  z*_ReusablePoolExecutor._wait_job_completioncs4tt|j}d|t|_tj|||jddS)N)r2)maxrr(rr2r _setup_queues)r rrZmin_queue_sizer!r r rTs  z#_ReusablePoolExecutor._setup_queues) NNNrNNNr N NNrFrNNNr N) __name__ __module__ __qualname__r classmethodrr9r3rErT __classcell__r r r!r rhs6 `#rrU)rLrO threadingmultiprocessingr-Zprocess_executorrrZbackend.contextrbackendr__all__RLockrr rr0r rrr r r r s2   C