o Zh<]@s0dZddlmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZmZdd lmZmZdd lmZerOdd lZerVdd lZd ZGd ddeddZ GdddeddZ!GdddeddZ"d"ddZ#ddZ$ddZ%ddZ&ddZ'Gd d!d!eZ(d!gZ)d S)#z Processor class for IDEFICS. )CallableDictListOptionalUnion)urlparse) BatchFeature) ImageInput) ImagesKwargsProcessingKwargsProcessorMixin TextKwargsUnpack!_validate_images_text_input_order)PreTokenizedInput TextInput)is_tf_availableis_torch_available)deprecate_kwargNc@s^eZdZUeeed<eeeefed<ee e e e fed<ee e e e fed<dS)IdeficsImagesKwargsZ transform image_sizeZ image_meanZ image_stdN) __name__ __module__ __qualname__rr__annotations__rstrintrfloatrr r ]/var/www/auris/lib/python3.10/site-packages/transformers/models/idefics/processing_idefics.pyr.s rF)totalc@s&eZdZUeeed<eeed<dS)IdeficsTextKwargs add_eos_tokenadd_end_of_utterance_tokenN)rrrrboolrr r r r!r#5s r#c@s6eZdZUeed<eed<ddddiddidZd S) IdeficsProcessorKwargs text_kwargs images_kwargsFlongest)Zadd_special_tokenspaddingr$return_tensorspt)r(r)Z common_kwargsN)rrrr#rr _defaultsr r r r!r':s  r'cCs|dkr|dkrd|||k<n |dkrt||kd|}|dkr;|dk}d||<tjjj||d}d||ddf<|S|dkrct|d}t|d|}tj||d}t|d}t|t||}|S)Nr/r-tfr num_classes)depth) r0wheretorchnnZ functionalZone_hotequalZ expand_dimsZ zeros_like)Zincremental_maskr,r2Z negativesZ attn_maskZnegatives_expandedr r r!$incremental_to_binary_attention_maskIs$   r8cCs(|dkr t||S|dkrt||SdS)Nr-r0),image_attention_mask_for_packed_input_ids_pt,image_attention_mask_for_packed_input_ids_tf) input_ids tokenizerr,r r r!)image_attention_mask_for_packed_input_idscs   r=c Csvtj|dd}tj|dd}|t}|j}t|dD]6}d}d}t||D])\} } | |kr>|d7}|||| <d}n|||| <|rLd||| <| |krRd}q)qt|dD][}d}d}t||ddddD]-} ||| } | |kr|d7}|||| <d}n|||| <| |krd}|rd||| <qn||dk} ||| |8<||| d9<q[||fS)Nr/)Z fill_valuerFT)r5Z full_likeconvert_tokens_to_ids IMAGE_TOKEN eos_token_idrangesize enumerate) r;r<image_attention_masknext_image_attention_maskimage_token_id eod_token_id batch_idxcountseen_eodidxtoken_idZnon_negative_indicesr r r!r9jsL         r9cCs.|t}|j}t|d}tt|d}tt|d}t|D]m}d}d} t|d} t| dddD]W} ||| f} | |krc|d7}|| gg} |g}t|| |}t|| |}n| |kr|| s|d} d}|| gg} |g}t|| |}| r| |kr|| gg} dg}t|| |}q:q%||fS)Nrr/Fr>T) r?r@rAr0shapefillrBnumpytensor_scatter_nd_update)r;r<rGrHZ batch_sizerErFrIrJrKZ seq_lengthrLrMindicesupdatesr r r!r:s<       r:cCs$d|vrdSt|}t|j|jgS)zChecks if the passed string contains a valid url and nothing else. e.g. if space is included it's immediately invalidated the url F)rallschemenetloc)stringresultr r r!is_urlsrZcseZdZdZddgZddgZdZdZdfd d Ze d d ddd    dde e e e e e e e e e fde eee ee ee e ee e efdeedefddZddZddZeddZZS)IdeficsProcessorah Constructs a IDEFICS processor which wraps a LLama tokenizer and IDEFICS image processor into a single processor. [`IdeficsProcessor`] offers all the functionalities of [`IdeficsImageProcessor`] and [`LlamaTokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`IdeficsImageProcessor`): An instance of [`IdeficsImageProcessor`]. The image processor is a required input. tokenizer (`LlamaTokenizerFast`): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. image_size (`int`, *optional*, defaults to 224): Image size (assuming a square image) add_end_of_utterance_token (`str`, *optional*): The string representation of token representing end of utterance image_processorr<rr%ZIdeficsImageProcessorZLlamaTokenizerFastNc s|durtd|durtdt|||j|_t|dr#|jn|t|_|jj |jj |jj f|_ d|j j dgvrDd|_dSd|_dS)Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`. image_tokenZadditional_special_tokensTF) ValueErrorsuper__init__r\Zcurrent_processorhasattrrGr?r@Zimage_num_channelsrdefault_image_dimsr<Zspecial_tokens_mapget1tokenizer_was_trained_with_end_of_utterance_token)selfr\r<rr%kwargs __class__r r!rbs&zIdeficsProcessor.__init__promptsz5.0.0textT)Zold_nameversionnew_nameZraise_if_both_namesimagesrhreturnc. sF|dur |dur tdt||\}}|dur|}nM|durgt|ttfs(|g}t|tr0|g}t|ttfrCt|t|krCtdtdd|DsPtdt|dttfr`dd |D}tt||}|j t fd |j j i|}|d  d d }|d  dd} | dur|j} tdd|Ds|g}ddd} fdd} g} g} |D]|}|j j}g}d }d }t|D]L\}}|dkr|sdnd }t|tr|d}t|r|j|}|| |7}||d}q| r|r|| 7}||7}d }q|| |7}||d}q|r||j j7}|j|fi|d}| || |q|d  dd}|j | fi|d }|d}|d}tdd| D}td|}tdd| Ddk}g}g}g}t||| D]\}} }!|}"|"|j}#t|#|}$|!d|$}%t|%dkr|dkrtj|g|%ddR}&|%|&d|%d<nY|dkrt !|%dd}'t j"|g|'gdd }(t j|(|%j#d!}&t !|%d})t $t %|)d"}*|%}+t &|&|*|+}&n|dkrtj|g|j'R}&n|dkrt |g|j'R}&||&|dkr|t(|"|t(| qg|dkr.|t j)|"t j*d!|| qg|dkrEt+|}t+|}t+|}n|dkrYt +|}t +|}t +|}|rmt,||j |\},}-t-|,||d#},n,|dkrtj|j!d|j!ddtj.d!},n|dkrt j|j!d|j!ddft j.d!},t/||||,d$d%S)&aThis method takes batched or non-batched prompts made of text and images and converts them into prompts that the model was trained on and prepares the image pixel values for the model to process. Args: images (`Union[ImageInput, List[ImageInput], str, List[str], List[List[str]]]`): either a single image or a batched list of images - can be passed in when text contains only text prompts, in order to use the image-text-to-text behavior. text (`Union[List[TextInput], [List[List[TextInput]]]]`): either a single prompt or a batched list of prompts - see the detailed description immediately after the end of the arguments doc section. return_tensors (`str` or `TensorType`, *optional*, defaults to `TensorType.PYTORCH`): The type of tensors to return. Can be one of: - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. Returns: a dict with entries: `input_ids`, `attention_mask`, `pixel_values`, `image_attention_mask` which can be directly passed to `model.generate` Detailed explanation: Each entry in `text` is either a text to be passed as is or an image that will be processed. An image can be either an image object (`PIL.Image`) or a url from which the image can be retrieved. When the processor encounters an image it'll inject `` entry into the prompt. Example: ```python checkpoint = "HuggingFaceM4/idefics-9b" processor = AutoProcessor.from_pretrained(checkpoint) url = "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg" img = processor.image_processor.fetch_images([url])[0] prompts = [ "User:", img, "Describe this image. Assistant: An image of two kittens in grass. ", "User:", "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg", "Describe this image. Assistant:", ] inputs = processor(text=prompts, return_tensors="pt") generated_ids = model.generate(**inputs, max_length=100) generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` In this example the `prompts` will be converted into: ``` User:Describe this image. Assistant: An image of two kittens in grass. User:Describe this image. Assistant:' ``` and the two images will be massaged using [`IdeficsImageProcessor.__call__`] method and placed inside the `pixel_values` dict entry of the return value. This example also exemplifies that images can be passed as objects or as text urls. It can be seen that the first image is passed as object and the second one as a url. To do training do: ```python image_transform = transforms.Compose( [ transforms.RandomResizedCrop( (w, h), scale=(0.9, 1.0), interpolation=transforms.InterpolationMode.BICUBIC ), transforms.ToTensor(), transforms.Normalize(mean=self.image_mean, std=self.image_std), ] ) inputs = processor(text=prompts, transform=image_transform, return_tensors="pt") ``` In order to help debug prompt generation enable `debug=True` which will show you what's happening. Nz9You need to specify either `text` or `images` and `text`.aWhen providing both images and text arguments, the number of text prompts should be the same as the number of images.If you want to have several images per prompt, images should be nested as such: images=[[img1, img2], [img3, img4], ...] for text=[prompt1, prompt2, ...].css|]}t|tVqdSN) isinstancer.0ir r r! lsz,IdeficsProcessor.__call__..zQWhen using the image-text-to-text behavior, the prompts should only contain text.rcSsg|]}|gqSr r rsr r r! psz-IdeficsProcessor.__call__..Ztokenizer_init_kwargsr(r$Fr%css|] }t|ttfVqdSrq)rrlisttuplersr r r!rvszrr_cs|rSSrqr )last_was_imageZ fake_tokenr^r r! image_tokenss z/IdeficsProcessor.__call__..image_tokensTrTr)r,r-r;attention_maskcs|]}t|VqdSrqlenrtxr r r!rvr>csr~rqrrr r r!rvrr0)Zaxis)dtype)r/r>r1)r;r}Z pixel_valuesrE)data)0r`rrrrxryrrrUzipZ _merge_kwargsr'r<Z init_kwargspoprfanyZ bos_tokenrDstriprZr\Z fetch_imagesappendZ eos_tokenmaxsumrJrGminr5ZzerosrCr0rNconcatrZreshaperBrQrdZtensorZconvert_to_tensorZint32stackr=r8r&r ).rgrorlZaudioZvideosrhrkZ output_kwargsr$r%Zend_of_utterance_tokenr|Z all_promptsZ all_imagessample full_textZ image_objectsrzZ last_was_textruitemimager, text_encodingZ all_textsZall_attention_masksZmax_num_imagesZat_least_one_imageZoutput_input_idsZ output_imagesZoutput_attention_masksZ text_singler}Zextracted_imagesZpadded_input_idsZ image_countZlocal_max_num_imagesZcurrent_imagesZpadded_image_tensorZ image_shapeZ padded_shapeZ num_imagesrRrSrE_r r{r!__call__sb                                   zIdeficsProcessor.__call__cO|jj|i|S)z This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r< batch_decodergargsrhr r r!rzIdeficsProcessor.batch_decodecOr)z This method forwards all its arguments to LlamaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r<decoderr r r!rrzIdeficsProcessor.decodecCs"|jj}|jj}tt||Srq)r<model_input_namesr\rxdictfromkeys)rgZtokenizer_input_namesZimage_processor_input_namesr r r!rsz"IdeficsProcessor.model_input_names)Nr]N)NNNN)rrr__doc__ attributesZ valid_kwargsZimage_processor_classZtokenizer_classrbrrr rrrrrr'r rrrpropertyr __classcell__r r rir!r[sF     r[)r/)*rtypingrrrrr urllib.parserZfeature_extraction_utilsr Z image_utilsr Zprocessing_utilsr r r rrrZtokenization_utils_baserrutilsrrZutils.deprecationrr5Z tensorflowr0r@rr#r'r8r=r9r:rZr[__all__r r r r!s4      /!  c