o Zh^G @sdZddlmZmZmZddlZddlmZddl m Z m Z ddl m Z mZmZmZddlmZmZGd d d e d d ZGd dded d ZdeededeeefddZdeeeedeeedededejf ddZdedededefddZGdd d eZd gZdS)!zProcessor class for Mllama.)ListOptionalUnionN) BatchFeature) ImageInputmake_nested_list_of_images) ImagesKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc@seZdZUeeed<dS)MllamaImagesKwargsmax_image_tilesN)__name__ __module__ __qualname__rint__annotations__rr[/var/www/auris/lib/python3.10/site-packages/transformers/models/mllama/processing_mllama.pyrs rF)totalc@s"eZdZUeed<dddiiZdS)MllamaProcessorKwargs images_kwargsZ image_kwargsrN)rrrrr _defaultsrrrrr#s r input_idsimage_token_idreturncsfddt|D}t|dkrgSt|dkr |ddggSddt|dd|ddD}||dt|g|dd}|dddD]}|d|ddkrZ||d<|d}qJ|S)a Generate a cross-attention token mask for image tokens in the input sequence. This function identifies the positions of image tokens in the input sequence and creates a mask that defines which subsequent tokens each image token should attend to. Args: input_ids (List[int]): A list of token ids representing the input sequence. image_token_id (int): The id of the token used to represent images in the sequence. Returns: List[List[int]]: A list of [start, end] pairs, where each pair represents the range of tokens an image token should attend to. Notes: - If no image tokens are present, an empty list is returned. - For a single image token, it attends to all subsequent tokens until the end of the sequence. - For multiple image tokens, each attends to tokens up to the next image token or the end of the sequence. - Consecutive image tokens are treated as a group and attend to all subsequent tokens together. csg|] \}}|kr|qSrr).0itokenrrr Cz2get_cross_attention_token_mask..rcSsg|]\}}||gqSrr)r loc1loc2rrrr$LN) enumeratelenzipappend)rrZimage_token_locationsZ vision_masksZ last_mask_endZ vision_maskrr#rget_cross_attention_token_mask-s  $  r/cross_attention_token_mask num_tiles max_num_tileslengthc Cst|}tdd|D}tj||||ftjd}tt||D]5\}\}} tt|| D]'\} \} } t| dkrT| \} }t||}|dkrH|}d||| || d| f<q-q |S)a Convert the cross attention mask indices to a cross attention mask 4D array. This function takes a sparse representation of cross attention masks and converts it to a dense 4D numpy array. The sparse representation is a nested list structure that defines attention ranges for each image in each batch item. Args: cross_attention_token_mask (List[List[List[int]]]): A nested list structure where: - The outer list represents the batch dimension. - The middle list represents different images within each batch item. - The inner list contains pairs of integers [start, end] representing token ranges for each image. num_tiles (List[List[int]]): A nested list structure specifying the number of tiles for each image in each batch item. max_num_tiles (int): The maximum possible number of tiles. length (int): The total sequence length of the input. Returns: np.ndarray: A 4D numpy array of shape (batch_size, length, max_num_images, max_num_tiles) The array contains `1` where attention is allowed and `0` where it is not. Note: - Special handling is done for cases where the end token is -1, which is interpreted as attending to the end of the sequence. cSg|]}t|qSrr,)r masksrrrr${z@convert_sparse_cross_attention_mask_to_dense..)shapeZdtyper'r&N)r,maxnpZzerosZint64r+r-min)r0r1r2r3Z batch_sizeZmax_num_imagescross_attention_maskZ sample_idxZ sample_masksZsample_num_tilesZmask_idx locationsZmask_num_tilesstartendrrr,convert_sparse_cross_attention_mask_to_dense]s"   rAprompt bos_token image_tokencCsP||vr|Sd}||r|t|d}|d7}||s ||||S)a\ Builds a string from the input prompt by adding `bos_token` if not already present. Args: prompt (`str`): The input prompt string. bos_token (`str`): The beginning of sentence token to be added. image_token (`str`): The image token used to identify the start of an image sequence. Returns: str: The modified prompt string with the `bos_token` added if necessary. Examples: >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' >>> build_string_from_input("<|image|>Hello world", "", "<|image|>") '<|image|>Hello world' >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' rNr&) startswithr,)rBrCrDZnum_image_tokens_on_startrrrbuild_string_from_inputs  rFc seZdZdZddgZdgZdZdZdfdd Z    dd e e d e e e e ee ee fd eed efddZddZddZ dddZeddZZS)MllamaProcessora Constructs a Mllama processor which wraps [`MllamaImageProcessor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the image processor and tokenizer functionalities. See the [`~MllamaProcessor.__call__`] and [`~OwlViTProcessor.decode`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import MllamaProcessor from PIL import Image processor = MllamaProcessor.from_pretrained("meta-llama/Llama-3.2-11B-Vision") processor( images=your_pil_image, text=["<|image|>If I had to write a haiku for this one"], images_kwargs = {"size": {"height": 448, "width": 448}}, text_kwargs = {"padding": "right"}, common_kwargs = {"return_tensors": "pt"}, ) ``` Args: image_processor ([`MllamaImageProcessor`]): The image processor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer chat_templateZMllamaImageProcessorZPreTrainedTokenizerFastNcsbt|dsd|_||j|_n|j|_|j|_d|_||j|_|j|_tj|||ddS)NrDz <|image|>z<|python_tag|>)rJ) hasattrrDZconvert_tokens_to_idsrZ python_tokenZpython_token_idrCsuper__init__)selfrHrIrJ __class__rrrMs zMllamaProcessor.__init__imagestextkwargsrc s||dur |dur tdjtfdjji|}|d}d|d<|d}|d} i} |durt|tr8|g}nt|ttfrHt dd |DsLtd fd d |D} fd d |D}| dd} j|fi|} j || dgdfdd | dD}| | dg}|durt |}dd |D}|durtdd | Drt dd | Dstdt| dkr|| ks||kr|durtdd}t|t| kr|| krd}n||krd}td| d|d||durj|fi|}| d}| ||dur0|dur0fdd | dD}t||jjtd d | dDd!}|| d"<| dd}t| |d#}|S)$a& Main method to prepare text(s) and image(s) to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] if `text` is not `None` to encode the text. To prepare the image(s), this method forwards the `images` arguments to MllamaImageProcessor's [`~MllamaImageProcessor.__call__`] if `images` is not `None`. Please refer to the docstring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. TODO: add aspect_ratio_ids and aspect_ratio_mask and cross_attention_mask Nz'You must specify either text or images.Ztokenizer_init_kwargs text_kwargsreturn_tensorsr common_kwargscss|]}t|tVqdSN) isinstancestrr trrr sz+MllamaProcessor.__call__..zAInvalid input text. Please provide a string, or a list of stringscg|]}|jqSr)countrDrZrNrrr$r*z,MllamaProcessor.__call__..csg|] }t|jjqSr)rFrCrD)r Z text_itemr_rrr$ r%Z padding_sideimage)Z modalitiescr]r)r^rr Z token_idsr_rrr$$r*rrcSr4rr5)r samplerrrr$*r7css|]}|dkVqdS)rNr)r Z batch_imgrrrr\-zaIf a batch of text is provided, there should be either no images or at least one image per samplez@No image were provided, but there are image tokens in the promptzZMake sure to pass your images as a nested list, where each sub-list holds images per batchzhIf you activated truncation with `max_length`, increase the `max_length` so image tokens aren't cropped.z)The number of image tokens in each text (zA) should be the same as the number of provided images per batch (z). r1csg|]}t|jqSr)r/rrar_rrr$Ks css|]}t|VqdSrWr5)r rrrrr\Rrc)r1r2r3r=)dataZ tensor_type) ValueErrorZ _merge_kwargsrrIZ init_kwargsrXrYlisttupleallpopZ_check_special_mm_tokensupdateranysumrHrArr:r)rNrQrRZaudioZvideosrSZ output_kwargsrTrrVreZn_images_in_text_encodingZn_images_in_idsZn_images_in_imagesZ add_messageZimage_featuresr1r0r=rUZ batch_featurerr_r__call__s'           zMllamaProcessor.__call__cO|jj|i|S)z This method forwards all its arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. rI batch_decoderNargsrSrrrrs[zMllamaProcessor.batch_decodecOrq)z This method forwards all its arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rIdecodertrrrrwbrvzMllamaProcessor.decodeTFcKs|jj|f||d|S)a Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `List[str]`: The decoded text. )skip_special_tokensclean_up_tokenization_spacesrr)rNZgenerated_outputsrxryrSrrrpost_process_image_text_to_textisz/MllamaProcessor.post_process_image_text_to_textcCs0|jj}|jj}dd|D}t||dgS)NcSsg|]}|dkr|qS)r1r)r namerrrr$r*z5MllamaProcessor.model_input_names..r=)rImodel_input_namesrHrg)rNZtokenizer_input_namesZimage_processor_input_namesrrrr|sz!MllamaProcessor.model_input_namesrW)NNNN)TF)rrr__doc__ attributesZ valid_kwargsZimage_processor_classZtokenizer_classrMrrrrr rr rrrprsrwrzpropertyr| __classcell__rrrOrrGs4 w rG)r}typingrrrnumpyr;Zfeature_extraction_utilsrZ image_utilsrrZprocessing_utilsr r r r Ztokenization_utils_baser rrrrr/ZndarrayrArYrFrG__all__rrrrs2 " 0  0% ^