o ZhZM@sddlmZmZmZmZddlmZddlmZm Z m Z ddl m Z m Z mZddlmZmZmZddlmZer=ddlZGd d d e d d Zd ZddedDddedDZddZGddde ZdgZdS))ClassVarListOptionalUnion) BatchFeature) ImageInputis_valid_imagemake_flat_list_of_images)ProcessingKwargsProcessorMixinUnpack) AddedTokenPreTokenizedInput TextInput)is_torch_availableNc@s&eZdZddidddddidZd S) ColPaliProcessorKwargspaddinglongestZchannels_firstT)Z data_formatZdo_convert_rgbZreturn_tensorspt) text_kwargs images_kwargsZ common_kwargsN)__name__ __module__ __qualname__ _defaultsrr]/var/www/auris/lib/python3.10/site-packages/transformers/models/colpali/processing_colpali.pyr$s rF)totalzcCg|] }d|ddqS)z4>r.0irrr 2r$icCr)z3r rr!rrrr$2r%cCs|||||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`List[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_tokenZ image_seq_len image_tokenZ num_imagesrrrbuild_string_from_input5sr,c s\eZdZUdZddgZdgZdZdZdZe e e d<d Z e e e d < d,fd d Z d-dedeeeeeeefdeedefddZddZddZeddZede fddZ d.dedeedefddZdeeeefdeedefddZ !d/d"ed#ed#fd$ed#ed#fd%ed&ed'd(ed)e fdd#f d*d+Z Z!S)0ColPaliProcessora Constructs a ColPali processor which wraps a PaliGemmaProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColPaliProcessor`] offers all the functionalities of [`PaliGemmaProcessor`]. See the [`~PaliGemmaProcessor.__call__`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`LlamaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer chat_template)ZSiglipImageProcessorZSiglipImageProcessorFast)ZGemmaTokenizerZGemmaTokenizerFastzDescribe the image.visual_prompt_prefixz Question: query_prefixNc s|durtd|durtdt|dstd|j|_t|ds=ttddd}d |gi}|||t|_t|_n|j|_|j|_| t d|_ d|_ t j|||d dS) Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r+FT) normalizedZspecialZadditional_special_tokens)r0) ValueErrorhasattrr3r IMAGE_TOKENZadd_special_tokensZconvert_tokens_to_idsZimage_token_idr+Z add_tokens EXTRA_TOKENSZ add_bos_tokenZ add_eos_tokensuper__init__)selfr.r/r0kwargsr+Z tokens_to_add __class__rrr:ds&      zColPaliProcessor.__init__imagestextr<returnc snjtfdjji|}|ddd}|durdnd}|dur)|dur)td|dur5|dur5td|durt|rA|g}n$t|trMt|d rMnt|trat|d trat|d d setd j gt |} d d |D}fd d t | |D} t |}j |fi|dd} |ddddur|ddj7<j| fddi|d} i| d| i} |r| d| dd kd}| d|it| dS|dur5t|tr|g}nt|trt|d tstd|durjd}g}|D]}jjj|}||7}|d7}||q|ddd|dd<j|fddi|d}|SdS)a Main method to prepare for the model either (1) one or several texts, either (2) one or several image(s). This method is custom wrapper around the PaliGemmaProcessor's [`~PaliGemmaProcessor.__call__`] method adapted for the ColPali model. It cannot process both text and images at the same time. When preparing the text(s), this method forwards the `text` and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`]. When preparing the image(s), this method forwards the `images` and `kwargs` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. Ztokenizer_init_kwargsrsuffixNTFz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagescSsg|]}|dqS)RGB)convert)r"imagerrrr$sz-ColPaliProcessor.__call__..c s:g|]\}}t|jjjtt|trt|nddqS)r()r,r/r*r3r7 isinstancelistlen)r"r)Z image_listr;rrr$sr pixel_values max_lengthreturn_token_type_idsZ input_idsZtoken_type_idsilabels)dataz*Text must be a string or a list of strings r'2)Z _merge_kwargsrr/Z init_kwargspopr5r rGrHr1rIzipr r.getr3Z masked_fillupdaterstrquery_augmentation_tokenr*r2append)r;r?r@ZaudioZvideosr<Z output_kwargsrBrMZ texts_docZ input_stringsrKZinputsZ return_datarNZ texts_queryqueryZ batch_queryrrJr__call__s-(      zColPaliProcessor.__call__cO|jj|i|S)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r/ batch_decoder;argsr<rrrr\zColPaliProcessor.batch_decodecOr[)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r/decoder]rrrr` r_zColPaliProcessor.decodecCs"|jj}|jj}tt||SN)r/model_input_namesr.rHdictfromkeys)r;Ztokenizer_input_namesZimage_processor_input_namesrrrrbsz"ColPaliProcessor.model_input_namescCs|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r/Z pad_tokenrJrrrrWsz)ColPaliProcessor.query_augmentation_tokencK|jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`]. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. r?NrrZ)r;r?r<rrrprocess_images s!zColPaliProcessor.process_imagescKre)a Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`]. Args: text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). r@Nrrf)r;r@r<rrrprocess_queriesCs z ColPaliProcessor.process_queriesr&cpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec Cs@t|dkr tdt|dkrtd|dj|djkr"td|dj|djkr0td|dur9|dj}g}tdt||D]U}g}tjjjj ||||ddd} tdt||D]'} tjjjj || | |ddd} | t d | | j d d dj d d q`| tj|d d ||qCtj|dd S)aZ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColPali, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, List[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, List[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT)Z batch_firstZ padding_valuez bnd,csd->bcnsr)dimrF)rIr5ZdeviceZdtyperangetorchnnutilsZrnnZ pad_sequencerXZeinsummaxsumcatto) r;rjrkrlrmrnZscoresr#Z batch_scoresZ batch_queriesjZbatch_passagesrrrscore_retrievales2      "z ColPaliProcessor.score_retrieval)NNN)NNNNra)r&Nri)"rrr__doc__ attributesZ valid_kwargsZimage_processor_classZtokenizer_classr1rrV__annotations__r2r:rrrrrr rrrZr\r`propertyrbrWrgrhintrrz __classcell__rrr=rr-Ks| "    # & r-)typingrrrrZfeature_extraction_utilsrZ image_utilsrr r Zprocessing_utilsr r r Ztokenization_utils_baserrrrtrrrrr7rqr8r,r-__all__rrrrs    $ ]