o Zh(@sdZddlZddlmZmZddlmZddlmZddl m Z m Z m Z ddl mZmZmZmZdd lmZd d lmZeeZGd d d e ddZGddde ZdgZdS)zq Processor class for InstructBLIP. Largely copy of Blip2Processor with addition of a tokenizer for the Q-Former. N)ListUnion) BatchFeature) ImageInput)ProcessingKwargsProcessorMixinUnpack) AddedToken BatchEncodingPreTokenizedInput TextInput)logging) AutoTokenizerc @s*eZdZdddddddddd idZdS)InstructBlipProcessorKwargsTFr) add_special_tokenspaddingZstrideZreturn_overflowing_tokensZreturn_special_tokens_maskZreturn_offsets_mappingZreturn_token_type_idsZ return_lengthverbose) text_kwargs images_kwargsN)__name__ __module__ __qualname__ _defaultsrrg/var/www/auris/lib/python3.10/site-packages/transformers/models/instructblip/processing_instructblip.pyr&s rF)totalc seZdZdZgdZdgZdZdZdZdfdd Z    dd e d e e e ee ee fd eed efd dZddZddZeddZfddZefddZZS)InstructBlipProcessora Constructs an InstructBLIP processor which wraps a BLIP image processor and a LLaMa/T5 tokenizer into a single processor. [`InstructBlipProcessor`] offers all the functionalities of [`BlipImageProcessor`] and [`AutoTokenizer`]. See the docstring of [`~BlipProcessor.__call__`] and [`~BlipProcessor.decode`] for more information. Args: image_processor (`BlipImageProcessor`): An instance of [`BlipImageProcessor`]. The image processor is a required input. tokenizer (`AutoTokenizer`): An instance of ['PreTrainedTokenizer`]. The tokenizer is a required input. qformer_tokenizer (`AutoTokenizer`): An instance of ['PreTrainedTokenizer`]. The Q-Former tokenizer is a required input. num_query_tokens (`int`, *optional*):" Number of tokens used by the Qformer as queries, should be same as in model's config. )image_processor tokenizerqformer_tokenizernum_query_tokens)ZBlipImageProcessorZBlipImageProcessorFastrNc sPt|dstdddd|_|j|jgddn|j|_||_t|||dS)N image_tokenzFT) normalizedZspecial)Zspecial_tokens)hasattrr r#Z add_tokensr"super__init__)selfrr r!r"kwargs __class__rrr'Ps zInstructBlipProcessor.__init__imagestextr)returncKs|dur |dur td|jtfd|jji|}t}|durt|tr)|g}nt|ts9t|dts9td|d dd}|j|fi|dddi} ||dd<|j dur|duri} |j j |j } |j| gt |ddd } | D]} d d t| | | | D| | <qwn | } |durtd t| |d } || |j|fi|d}| d|d<| d|d<|dur|j|fi|d}|||S)a This method uses [`BlipImageProcessor.__call__`] method to prepare image(s) for the model, and [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. Args: images (`ImageInput`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`TextInput`, `PreTokenizedInput`, `List[TextInput]`, `List[PreTokenizedInput]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). Nz,You have to specify at least images or text.Ztokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringsrreturn_tensorsF)rr/cSsg|]\}}||qSrr).0Z img_encodingZ txt_encodingrrr sz2InstructBlipProcessor.__call__..aAExpanding inputs for image tokens in InstructBLIP should be done in processing. Please follow instruction here (https://gist.github.com/zucchini-nlp/e9f20b054fa322f84ac9311d9ab67042) to update your InstructBLIP model. Using processors without these attributes in the config is deprecated and will throw an error in v4.50.)Z tensor_typeZ input_idsZqformer_input_idsZattention_maskZqformer_attention_maskr) ValueErrorZ _merge_kwargsrr Z init_kwargsr isinstancestrlistpopr"r#contentlenziploggerZ warning_oncer updater!r)r(r,r-ZaudioZvideosr)Z output_kwargsencodingr/Z_text_encoding text_encodingZ image_tokensZimage_token_encodingkZqformer_text_encodingZimage_encodingrrr__call__YsV      zInstructBlipProcessor.__call__cO|jj|i|S)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoder(argsr)rrrrAz"InstructBlipProcessor.batch_decodecOr@)z This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r decoderBrrrrErDzInstructBlipProcessor.decodecCs"|jj}|jj}tt||SN)r model_input_namesrr5dictfromkeys)r(Ztokenizer_input_namesZimage_processor_input_namesrrrrGsz'InstructBlipProcessor.model_input_namesc stj|rtd|dtj|ddtj|d}|j|d|jv}|r/|j dt j|fi|}|rC|jdg7_|S)NzProvided path (z#) should be a directory, not a fileT)exist_okr!) ospathisfiler2makedirsjoinr!save_pretrained attributesremover&)r(Zsave_directoryr)Zqformer_tokenizer_pathZqformer_presentZoutputsr*rrrPs    z%InstructBlipProcessor.save_pretrainedc s>tj|fi|}t|tr|d}tj|dd}||_|S)Nrr!)Z subfolder)r&from_pretrainedr3tuplerr!)clsZpretrained_model_name_or_pathr) processorr!r*rrrSs  z%InstructBlipProcessor.from_pretrainedrF)NNNN)rrr__doc__rQZ valid_kwargsZimage_processor_classZtokenizer_classZqformer_tokenizer_classr'rrr r rr rrr?rArEpropertyrGrP classmethodrS __classcell__rrr*rr7s8  P  r)rWrKtypingrrZimage_processing_utilsrZ image_utilsrZprocessing_utilsrrr Ztokenization_utils_baser r r r utilsrautorZ get_loggerrr:rr__all__rrrrs      )