fTh1SSKrSSKrSSKJrJr SSKJr "SS\5r"SS\5r "S S \ 5r "S S \ 5r "S S\5r "SS\5r g)N)DatasetIterableDataset) ModelOutputc&\rSrSrSrSrSrSrg)PipelineDatasetc(XlX lX0lgNdatasetprocessparams)selfr rrs W/var/www/auris/envauris/lib/python3.13/site-packages/transformers/pipelines/pt_utils.py__init__PipelineDataset.__init__ s   c,[UR5$r lenr rs r__len__PipelineDataset.__len__4<<  rc^URUnUR"U40URD6nU$r r )riitem processeds r __getitem__PipelineDataset.__getitem__s,||ALL55 r)r rrN)__name__ __module__ __qualname____firstlineno__rrr __static_attributes__rrrrs !rrc6\rSrSrS SjrSrSrSrSrSr g) PipelineIteratorNc`XlX lX0lUS:XaSnX@lSUlSUlg)a Roughly equivalent to ``` for item in loader: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```N)loaderinferrloader_batch_size_loader_batch_index_loader_batch_data)rr-r.rr/s rrPipelineIterator.__init__s96    ! $ !2$( "&rc,[UR5$r )rr-rs rrPipelineIterator.__len__?s4;;rc:[UR5UlU$r iterr-iteratorrs r__iter__PipelineIterator.__iter__BT[[)  rc0^[TR[R5(a*TRTRR S5nGO,0nTRR 5GHup4[U[5(aUR5n[US[R5(a[U4SjU55X#'O9[US[R5(a[U4SjU55X#'MUS;a[U[5(av[US[R5(a[U4SjU55X#'O9[US[R5(a[U4SjU55X#'GM1UcSX#'GM;[UTR[R5(a#UTRR S5X#'GM[UTR[R5(a)[R"UTRS5X#'GMUTRX#'GM TRRU5nT=RS- slU$) zU Return item located at `loader_batch_index` within the current `loader_batch_data`. rc3^># UH"oTRRS5v M$ g7frNr0 unsqueeze.0elrs r 5PipelineIterator.loader_batch_item..U,1nfm`bT5M5M2N2X2XYZ2[2[fm*-c3l># UH)n[R"UTRS5v M+ g7fr>np expand_dimsr0rAs rrDrEW.1tlsfh"..DD\D\A]_`2a2als14> attentions hidden_statespast_key_valuesc3^># UH"oTRRS5v M$ g7fr>r?rAs rrDrE\rFrGc3l># UH)n[R"UTRS5v M+ g7fr>rIrAs rrDrE^rLrMNr,) isinstancer1torchTensorr0r@itemsrto_tupletuplerJndarrayrK __class__)rresultloader_batchedkelements` rloader_batch_item"PipelineIterator.loader_batch_itemFs d--u|| < <,,T-E-EFPPQRSF N"55;;= g{33%..0G!'!*ell;;,11nfm1n,n)#GAJ ;;,11tls1t,t)JJzZachOiOi!'!*ell;;,11nfm1n,n)#GAJ ;;,11tls1t,t)?(,N%(@(@ A5<<PP)00H0H(I(S(STU(VN%(@(@ A2::NN)+wt?W?W7XZ[(\N%)00H0H(IN%=>B,,66~FF   A%  rcURb*URUR:aUR5$[UR5nUR "U40UR D6nURb[U[R5(aUnO;[U[5(aUSnO [UR55SnX$n[U[5(a [U5nOURSnSUs=:aUR:a O OXPl[U[5(aUSOUUlSUlUR5$U$)Nr)r0r/r_nextr8r.rrSrTrUrXlistkeysrshaper1)rrr first_tensorkeyobserved_batch_sizes r__next__PipelineIterator.__next__us-  # # /D4L4LtOeOe4e))+ +DMM"JJt3t{{3  ! ! -)U\\22( Iu--(| 9>>+,Q/(~ ,--&),&7#&2&8&8&;#&?)?)??*=&6@E6R6RilXaD #'(D $))+ + r)r1r0r.r8r-r/rr ) r"r#r$r%rrr9r_rir&r'rrr)r)s%'N -^"rr)c8^\rSrSrSU4SjjrSrSrSrU=r$)PipelineChunkIteratorc&>[TU]XU5 g)a Roughly equivalent to ``` for iterator in loader: for item in iterator: yield infer(item, **params) ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item N)superr)rr-r.rr/rZs rrPipelineChunkIterator.__init__s$ /rcH[UR5UlSUlU$r )r7r-r8 subiteratorrs rr9PipelineChunkIterator.__iter__sT[[)  rcnURc6UR"[UR540URD6Ul[UR5nU$![ aN UR"[UR540URD6Ul[UR5nU$f=fr )rrr.rbr8r StopIteration)rrs rriPipelineChunkIterator.__next__s    # \#zz$t}}*=MMD  /T--.I / $zz$t}}*=MMD T--.I /sAAB43B4)r8rrr ) r"r#r$r%rr9rir& __classcell__)rZs@rrlrls0( rrlc$\rSrSrSrSrSrSrg)PipelinePackIteratorah Roughly equivalent to ``` packed = [] for item in loader: packed.append(item) if item["is_last"]: yield packed packed = [] ``` but it also handles cases where `item` are batched (meaning it's a dict of Tensor with first dimension > 1. In that case it does ``` packed = [] for batch in loader: # item is batched for item in batch: packed.append(item) if item["is_last"]: yield packed packed = [] ``` Arguments: loader (`torch.utils.data.DataLoader` or `Iterable`): The iterator that will be used to apply `infer` on. infer (any function): The function to apply of each element of `loader`. params (`dict`): The parameters passed to `infer` along with every item loader_batch_size (`int`, *optional*): If specified, the items of `loader` are supposed to come as batch, and are loader_batched here making it roughly behave as ``` for items in loader: for i in loader_batch_size: item = items[i] yield infer(item, **params) ```c:[UR5UlU$r r6rs rr9PipelinePackIterator.__iter__r;rcHSn/nURbURUR:aqURUR:aWUR5nURS5nUR U5 U(aU$URUR:aMWU(Gd}UR "[ UR540URD6nURGb[U[R5(aUnO [UR55SnXFn[U[5(a [U5nOURSnSUs=:aUR:a O OXplX@lSUlURUR:aWUR5nURS5nUR U5 U(aU$URUR:aMWO$UnURS5nUR U5 U(dGM}U$)NFis_lastr)r0r/r_popappendr.rbr8rrSrTrUrcrdrrer1)rr~ accumulatorrrrfrgrhs rriPipelinePackIterator.__next__s  # # /D4L4LtOeOe4e**T-C-CC--/((9-""4(&& **T-C-CC 4 #6F$++FI%%1i66#,Ly~~/03C#,>LlD11*-l*;'*6*<*rsd 5' g @@F*,*Za+aH ) ) ]W ]r