o Zh@sfdZddlmZmZmZddlmZddlmZm Z m Z m Z m Z ddl mZGdddeZdgZd S) z Processor class for Bros. )ListOptionalUnion)ProcessorMixin) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypec"seZdZdZdgZdZd"fdd Z              d#d eee e ee e fd e d ee e e fd ee e efdeededeedee dee de de de de de deee efdef ddZddZddZed d!ZZS)$ BrosProcessora Constructs a Bros processor which wraps a BERT tokenizer. [`BrosProcessor`] offers all the functionalities of [`BertTokenizerFast`]. See the docstring of [`~BrosProcessor.__call__`] and [`~BrosProcessor.decode`] for more information. Args: tokenizer (`BertTokenizerFast`, *optional*): An instance of ['BertTokenizerFast`]. The tokenizer is a required input. tokenizer)Z BertTokenizerZBertTokenizerFastNc s |durtdt|dS)Nz"You need to specify a `tokenizer`.) ValueErrorsuper__init__)selfrkwargs __class__W/var/www/auris/lib/python3.10/site-packages/transformers/models/bros/processing_bros.pyr)szBrosProcessor.__init__TFrtextadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturncKs4|jd||||||||| | | | | ||d|}|S)z This method uses [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. )rrrrrrrrr r!r"r#r$r%r&Nr)r)rrrrrrrrrr r!r"r#r$r%r&rencodingrrr__call__/s(zBrosProcessor.__call__cO|jj|i|S)z This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderargsrrrrr+\zBrosProcessor.batch_decodecOr*)z This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder,rrrr/cr.zBrosProcessor.decodecCs|jj}tt|SN)rmodel_input_nameslistdictfromkeys)rZtokenizer_input_namesrrrr1jszBrosProcessor.model_input_namesr0)NTFNNrNNNFFFFTN)__name__ __module__ __qualname____doc__ attributesZtokenizer_classrrr r rboolstrrr rintr rr)r+r/propertyr1 __classcell__rrrrr sr         -r N)r8typingrrrZprocessing_utilsrZtokenization_utils_baserrr r r utilsr r __all__rrrrs   V