o Zh@sjdZddlZddlZddlmZmZddlmZddlm Z ddl m Z e e ZGd d d Zd gZdS) zTokenization classes for RAG.N)ListOptional) BatchEncoding)logging) RagConfigc@seZdZddZddZeddZddZd d Zd d Z d dZ ddZ      dde e dee e deedeede dee dedefddZdS) RagTokenizercCs||_||_|j|_dSN)question_encoder generatorcurrent_tokenizer)selfr r rW/var/www/auris/lib/python3.10/site-packages/transformers/models/rag/tokenization_rag.py__init__s zRagTokenizer.__init__cCsbtj|rtd|dtj|ddtj|d}tj|d}|j||j|dS)NzProvided path (z#) should be a directory, not a fileT)exist_okquestion_encoder_tokenizergenerator_tokenizer) ospathisfile ValueErrormakedirsjoinr save_pretrainedr )rZsave_directoryZquestion_encoder_pathZgenerator_pathrrrr#s  zRagTokenizer.save_pretrainedcKsZddlm}|dd}|durt|}|j||jdd}|j||jdd}|||dS)N) AutoTokenizerconfigr)rZ subfolderr)r r )Zauto.tokenization_autorpoprfrom_pretrainedr r )clsZpretrained_model_name_or_pathkwargsrrr r rrrr ,s    zRagTokenizer.from_pretrainedcOs|j|i|Sr )r rargsr"rrr__call__>szRagTokenizer.__call__cO|jj|i|Sr )r batch_decoder#rrrr'AzRagTokenizer.batch_decodecOr&r )r decoder#rrrr)Dr(zRagTokenizer.decodecC |j|_dSr )r r rrrr_switch_to_input_modeG z"RagTokenizer._switch_to_input_modecCr*r )r r r+rrr_switch_to_target_modeJr-z#RagTokenizer._switch_to_target_modeNlongestT src_texts tgt_texts max_lengthmax_target_lengthpaddingreturn_tensors truncationreturnc Kstdt|dur|jj}||fd||||d|} |dur"| S|dur*|jj}|d|d||||d|} | d| d<| S)Nu4`prepare_seq2seq_batch` is deprecated and will be removed in version 5 of 🤗 Transformers. Use the regular `__call__` method to prepare your inputs and the tokenizer under the `with_target_tokenizer` context manager to prepare your targets. See the documentation of your specific tokenizer for more detailsT)add_special_tokensr5r2r4r6)Z text_targetr8r5r4r2r6Z input_idslabelsr)warningswarn FutureWarningr Zmodel_max_length) rr0r1r2r3r4r5r6r"Z model_inputsr9rrrprepare_seq2seq_batchMsB   z"RagTokenizer.prepare_seq2seq_batch)NNNr/NT)__name__ __module__ __qualname__rr classmethodr r%r'r)r,r.rstrrintboolrr=rrrrr sB   r )__doc__rr:typingrrZtokenization_utils_baserutilsrZconfiguration_ragrZ get_loggerr>loggerr __all__rrrrs     _