JThArSSKrSSKrSSKJrJr SSKJrJrJr SSK J r SSK J r SSK JrJr SSKJrJrJr SSKJrJr \ "5r\SLr/S Qr\"S 5r\"S S S 9r/SQr"SS\\5r"SS\\\S9r"SS\5r "SS\\\S9r!"SS5r""SS\"\5r#"SS\"\!5r$g)N)IterableIterator)CallableOptionalTypeVar) import_dill)_SnapshotState) _DataPipeMeta_IterDataPipeMeta)_deprecation_warning!_iter_deprecated_functional_names _map_deprecated_functional_names)DatasetIterableDataset) DataChunkDFIterDataPipe IterDataPipe MapDataPipe_T_T_coT) covariant)batchgroupby_dataframes_as_tuplestrace_as_dataframec~^\rSrSrS\\SS4U4SjjrS S\S\4SjjrS\ \4U4Sjjr S\ \4S jr S r U=r $) r'itemsreturnNcF>[U5n[TU] U5 XlgN)listsuper__init__r)selfr __class__s [/var/www/auris/envauris/lib/python3.13/site-packages/torch/utils/data/datapipes/datapipe.pyr$DataChunk.__init__(sU   indentcVUS-SRS[U555-S-$)N[z, c38# UHn[U5v M g7fr!)str).0is r' #DataChunk.as_str...s'C 1A s])joiniter)r%r*s r'as_strDataChunk.as_str-s)|dii'CT 'CCCcIIr)c#>># [TU]5ShvN gN7fr!)r#__iter__r%r&s r'r9DataChunk.__iter__0s7#%%%s c#8# URShvN gN7fr!rr%s r' raw_iteratorDataChunk.raw_iterator3s::s r=))__name__ __module__ __qualname____firstlineno__rrr$r.r6rr9r?__static_attributes__ __classcell__r&s@r'rr'sUhrlt JSJ#J&(2,&hrlr)rcv^\rSrSr%Sr0r\\\4\ S'Sr \ \\ S'Sr \ \\ S'Sr \ \\ S'Sr\ \\ S'Sr\ \\ S 'S r\\ S '\R&r\\ S 'Sr\ \\ S 'S\\4SjrSr\S5r\SSj5rSrU4Sjr\S5r\S5r Sr!Sr"U4Sjr#SSjr$Sr%U=r&$)r7a Iterable-style DataPipe. All DataPipes that represent an iterable of data samples should subclass this. This style of DataPipes is particularly useful when data come from a stream, or when the number of samples is too large to fit them all in memory. ``IterDataPipe`` is lazily initialized and its elements are computed only when ``next()`` is called on the iterator of an ``IterDataPipe``. All subclasses should overwrite :meth:`__iter__`, which would return an iterator of samples in this DataPipe. Calling ``__iter__`` of an ``IterDataPipe`` automatically invokes its method ``reset()``, which by default performs no operation. When writing a custom ``IterDataPipe``, users should override ``reset()`` if necessary. The common usages include resetting buffers, pointers, and various state variables within the custom ``IterDataPipe``. Note: Only `one` iterator can be valid for each ``IterDataPipe`` at a time, and the creation a second iterator will invalidate the first one. This constraint is necessary because some ``IterDataPipe`` have internal buffers, whose states can become invalid if there are multiple iterators. The code example below presents details on how this constraint looks in practice. If you have any feedback related to this constraint, please see `GitHub IterDataPipe Single Iterator Issue`_. These DataPipes can be invoked in two ways, using the class constructor or applying their functional form onto an existing ``IterDataPipe`` (recommended, available to most but not all DataPipes). You can chain multiple `IterDataPipe` together to form a pipeline that will perform multiple operations in succession. .. _GitHub IterDataPipe Single Iterator Issue: https://github.com/pytorch/data/issues/45 Note: When a subclass is used with :class:`~torch.utils.data.DataLoader`, each item in the DataPipe will be yielded from the :class:`~torch.utils.data.DataLoader` iterator. When :attr:`num_workers > 0`, each worker process will have a different copy of the DataPipe object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. :func:`~torch.utils.data.get_worker_info`, when called in a worker process, returns information about the worker. It can be used in either the dataset's :meth:`__iter__` method or the :class:`~torch.utils.data.DataLoader` 's :attr:`worker_init_fn` option to modify each copy's behavior. Examples: General Usage: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper, Mapper >>> dp = IterableWrapper(range(10)) >>> map_dp_1 = Mapper(dp, lambda x: x + 1) # Using class constructor >>> map_dp_2 = dp.map(lambda x: x + 1) # Using functional form (recommended) >>> list(map_dp_1) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> list(map_dp_2) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> filter_dp = map_dp_1.filter(lambda x: x % 2 == 0) >>> list(filter_dp) [2, 4, 6, 8, 10] Single Iterator Constraint Example: >>> from torchdata.datapipes.iter import IterableWrapper, Mapper >>> source_dp = IterableWrapper(range(10)) >>> it1 = iter(source_dp) >>> list(it1) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> it1 = iter(source_dp) >>> it2 = iter(source_dp) # The creation of a new iterator invalidates `it1` >>> next(it2) 0 >>> next(it1) # Further usage of `it1` will raise a `RunTimeError` functionsNreduce_ex_hook getstate_hookstr_hook repr_hook_valid_iterator_idr_number_of_samples_yielded_snapshot_state_fast_forward_iteratorrcU$r!r>s r'r9IterDataPipe.__iter__s r)c0U[R;a^U[;a[Un[S0UD6 [RUn[R "X05n[R "XCSS9 U$[SURRSU35eN__doc__wrapperwrappedassigned'z' object has no attribute 'rU) rrKr r functoolspartialupdate_wrapperAttributeErrorr&rBr%attribute_namekwargsffunctions r' __getattr__IterDataPipe.__getattr__s \33 3!BB:>J$.v.&&~6A ((1H  $ $X< XO DNN++,,GGWX r)c X RU'gr!rKcls function_namerhs r'register_functionIterDataPipe.register_function'/ m$r)c^TUR;a[STS35eU4Sjn[R"XBU5n[R"XRSS9 XPRT'g)N%Unable to add DataPipe function name  as it is already takenc>U"U/UQ70UD6n[U[5(a6U(d[U[5(aT[;aUR 5nU$r!) isinstancerrUNTRACABLE_DATAFRAME_PIPESr)rnenable_df_api_tracing source_dpargsrf result_piperos r'class_functionBIterDataPipe.register_datapipe_as_function..class_functionsPi9$9&9K+|44(Jy.,Q,Q$,FF&1&D&D&F  r)rYr[rK Exceptionr`rarb)rnrocls_to_registerryr}rhs ` r'register_datapipe_as_function*IterDataPipe.register_datapipe_as_functionsk CMM )7 F]^  $$ -B      (0 m$r)cjURn[Rb[RU5$U$z Serialize `lambda` functions when `dill` is available. If this doesn't cover your custom DataPipe's use case, consider writing custom methods for `__getstate__` and `__setstate__`, or use `pickle.dumps` for serialization. )__dict__rrMr%states r' __getstate__IterDataPipe.__getstate__s/   % % 1--e4 4 r)c>[Rb[RU5$[TU]"U0UD6$![a Nf=fr!)rrLNotImplementedErrorr# __reduce_ex__r%r{rfr&s r'rIterDataPipe.__reduce_ex__sP  & & 2 #22488w$d5f55'  9 AAcX[RbUb [S5eU[lgNz*Attempt to override existing getstate_hook)rrM RuntimeErrorrnhook_fns r'set_getstate_hookIterDataPipe.set_getstate_hooks'  % % 1g6IKL L%, "r)cX[RbUb [S5eU[lgNz+Attempt to override existing reduce_ex_hook)rrLrrs r'set_reduce_ex_hookIterDataPipe.set_reduce_ex_hooks'  & & 2w7JLM M&- #r)c|URbURU5$[URR5$r!rOr.r&rDr>s r'__repr__IterDataPipe.__repr__0 >> %>>$' '4>>..//r)c|URbURU5$[URR5$r!rNr.r&rDr>s r'__str__IterDataPipe.__str__0 == $==& &4>>..//r)cz>[[TU] 55[URR 55-$r!r"r#__dir__rKkeysr:s r'rIterDataPipe.__dir__,EGO%&dnn.A.A.C)DDDr)cg)a^ Reset the `IterDataPipe` to the initial state. By default, no-op. For subclasses of `IterDataPipe`, depending on their functionalities, they may want to override this method with implementations that may clear the buffers and reset pointers of the DataPipe. The `reset` method is always called when `__iter__` is called as part of `hook_iterator`. NrUr>s r'resetIterDataPipe.resetsr)rU)F)rN)'rBrCrDrErZrKdictr.r__annotations__rLrrMrNrOrPintrQr NotStartedrRrSrrr9ri classmethodrprrrrrrrrrrFrGrHs@r'rr7s!AF&(ItCM"')-NHX&-(,M8H%,#'Hhx '$(Ix!((, ,&''&4&?&?O^?15HX.5(5/ 00CH002 6-- .. 0 0 E  r)r) metaclassc\rSrSrSrSrg)rcg)NTrUr>s r' _is_dfpipeDFIterDataPipe._is_dfpipesr)rUN)rBrCrDrErrFrUr)r'rrsr)rc^\rSrSr%Sr0r\\\4\ S'Sr \ \\ S'Sr \ \\ S'Sr \ \\ S'Sr\ \\ S'S r\S 5r\S 5rS rU4S jr\S5r\S5rSrSrU4SjrSrU=r$)ra Map-style DataPipe. All datasets that represent a map from keys to data samples should subclass this. Subclasses should overwrite :meth:`__getitem__`, supporting fetching a data sample for a given, unique key. Subclasses can also optionally overwrite :meth:`__len__`, which is expected to return the size of the dataset by many :class:`~torch.utils.data.Sampler` implementations and the default options of :class:`~torch.utils.data.DataLoader`. These DataPipes can be invoked in two ways, using the class constructor or applying their functional form onto an existing `MapDataPipe` (recommend, available to most but not all DataPipes). Note: :class:`~torch.utils.data.DataLoader` by default constructs an index sampler that yields integral indices. To make it work with a map-style DataPipe with non-integral indices/keys, a custom sampler must be provided. Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.map import SequenceWrapper, Mapper >>> dp = SequenceWrapper(range(10)) >>> map_dp_1 = dp.map(lambda x: x + 1) # Using functional form (recommended) >>> list(map_dp_1) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> map_dp_2 = Mapper(dp, lambda x: x + 1) # Using class constructor >>> list(map_dp_2) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> batch_dp = map_dp_1.batch(batch_size=2) >>> list(batch_dp) [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]] rKNrLrMrNrOc0U[R;a^U[;a[Un[S0UD6 [RUn[R "X05n[R "XCSS9 U$[SURRSU35erX) rrKrr r`rarbrcr&rBrds r'riMapDataPipe.__getattr__s [22 2!AA9.I$.v.%%n5A ((1H  $ $X< XO DNN++,,GGWX r)c X RU'gr!rlrms r'rpMapDataPipe.register_function*rrr)cXR;a[SUS35eSn[R"X25n[R"XBSS9 X@RU'g)NrtrucU"U/UQ70UD6nU$r!rU)rnrzr{rfr|s r'r}AMapDataPipe.register_datapipe_as_function..class_function5si9$9&9K r)rYr[r)rnrorr}rhs r'r)MapDataPipe.register_datapipe_as_function.s^ MM )7 F]^  $$^E    (0 m$r)cjURn[Rb[RU5$U$r)rrrMrs r'rMapDataPipe.__getstate__?s/   $ $ 0,,U3 3 r)c>[Rb[RU5$[TU]"U0UD6$![a Nf=fr!)rrLrr#rrs r'rMapDataPipe.__reduce_ex__KsP  % % 1 "11$77w$d5f55'  rcX[RbUb [S5eU[lgr)rrMrrs r'rMapDataPipe.set_getstate_hookSs'  $ $ 0W5HKL L$+ !r)cX[RbUb [S5eU[lgr)rrLrrs r'rMapDataPipe.set_reduce_ex_hookYs'  % % 1g6ILM M%, "r)c|URbURU5$[URR5$r!rr>s r'rMapDataPipe.__repr___rr)c|URbURU5$[URR5$r!rr>s r'rMapDataPipe.__str__err)cz>[[TU] 55[URR 55-$r!rr:s r'rMapDataPipe.__dir__krr)rU)rBrCrDrErZrKrr.rrrLrrMrNrOrirrprrrrrrrrrFrGrHs@r'rrsB&(ItCM"')-NHX&-(,M8H%,#'Hhx '$(Ix!( 0000 6,, -- 0 0 EEr)rc,\rSrSrSrSrSrSrSrg)_DataPipeSerializationWrapperipcXlgr! _datapipe)r%datapipes r'r$&_DataPipeSerializationWrapper.__init__qs!r)cSn[R"UR5nX!4$![a2 [(a%[ RUR5nSnX!4$ef=f)NFT)pickledumpsrrHAS_DILLdill)r%use_dillvalues r'r*_DataPipeSerializationWrapper.__getstate__tsf LL0E   x 4>>2    s '6A#!A#cUup#U(aU5Ulg[R"U5Ulgr!)rloadsrr)r%rrrs r' __setstate__*_DataPipeSerializationWrapper.__setstate__s, !ZZ.DN#\\%0DNr)c[UR5$![a'n[[ U5R S35UeSnAff=f)Nz# instance doesn't have valid length)lenrr TypeErrortyperB)r%es r'__len__%_DataPipeSerializationWrapper.__len__sL t~~& & :&&''JK  s A"AArN) rBrCrDrEr$rrrrFrUr)r'rrps" !1r)rcN^\rSrSrS\\4U4SjjrSSjrS\4SjrSr U=r $) !_IterDataPipeSerializationWrapperirc2>[TU]U5 SUlgr!)r#r$_datapipe_iter)r%rr&s r'r$*_IterDataPipeSerializationWrapper.__init__s "9=r)rc:[UR5UlU$r!)r5rrr>s r'r9*_IterDataPipeSerializationWrapper.__iter__s"4>>2 r)cJURce[UR5$r!)rnextr>s r'__next__*_IterDataPipeSerializationWrapper.__next__s%""...D''((r))r)rr) rBrCrDrErrr$r9rrFrGrHs@r'rrs*>e!4>)%))r)rc\rSrSrSrSrg) _MapDataPipeSerializationWrapperic URU$r!r)r%idxs r' __getitem__,_MapDataPipeSerializationWrapper.__getitem__s~~c""r)rUN)rBrCrDrErrFrUr)r'rrs#r)r)%r`rcollections.abcrrtypingrrrtorch.utils._import_utilsr)torch.utils.data.datapipes._hook_iteratorr "torch.utils.data.datapipes._typingr r 'torch.utils.data.datapipes.utils.commonr r rtorch.utils.data.datasetrrrr__all__rrrxr"rrrrrrrrUr)r'rs ...1DO >} t   T]4( R  u ?5)5Fu p\ yE'%.MyEx@ )(E| )#'Dk#r)