o ZhY@sRdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddlmZmZddlmZmZmZmZdd lmZmZdd lmZmZmZdd lmZmZm Z ercd d l!m"Z"m#Z#m$Z$m%Z%erkddlm&Z&e'e(Z)erzddl*m*Z*ndZ*ddZ+ddZ,ddZ-GdddeddZ.GdddeddZ/GdddeZ0dgZ1dS)z Processor class for SmolVLM. N) timedelta) TYPE_CHECKINGDictListOptionalUnion) BatchFeature) ImageInputmake_nested_list_of_images) ImagesKwargsProcessingKwargsProcessorMixinUnpack) BatchEncoding TextInput)is_num2words_availableis_vision_availablelogging) VideoInput load_videomake_batched_videos)DEFAULT_MEDIA_OUTTRODEFAULT_VIDEO_INTROFRAME_TIMESTAMP_MESSAGEsmolvlm_sample_indices_fn)PreTokenizedInput) num2wordsc Cs~d}t|D]$}t|D]}||d|dd|dd||7}q |d7}q|d|||||7}|S)zKPrompt with expanded image tokens for when the image is split into patches.z )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_image_tokenZtext_split_imagesZn_hZn_wr)]/var/www/auris/lib/python3.10/site-packages/transformers/models/smolvlm/processing_smolvlm.py_prompt_split_image5s"  ( r+cCs|||||S)z5Prompt with expanded image tokens for a single image.r))r#r&r'r(r)r)r*_prompt_single_imageJsr,cCs2|dkr|dkrt||||dSt||||||S)Nr)r&r'r()r,r+)r$r%r#r&r'r(r)r)r*get_image_prompt_stringTs r-c@s.eZdZUeeed<eeeefed<dS)SmolVLMImagesKwargsreturn_row_col_infoZmax_image_sizeN) __name__ __module__ __qualname__rbool__annotations__rstrintr)r)r)r*r.cs r.F)totalc@s,eZdZUeed<ddddddidZdS)SmolVLMProcessorKwargs images_kwargsTF)Zadd_special_tokenspaddingZis_split_into_wordsr/) text_kwargsr9N)r0r1r2r.r4 _defaultsr)r)r)r*r8hs  r8c sNeZdZdZgdZddgZdZdZdZ  d,de de e ffd d Z d-d d Z    d.deeeeeeefdeedeeedfdedeedef ddZdeeee e fdeedeedeeee effddZddZddZed d!Z   " #d/d$ee d%fd&e e d'e e d(e d)e dejf d*d+Z Z!S)0SmolVLMProcessora} Constructs a SmolVLM processor which wraps a LLama tokenizer and SmolVLM image processor into a single processor. [`SmolVLMProcessor`] offers all the functionalities of [`SmolVLMImageProcessor`] and [`SmolVLMTokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`SmolVLMImageProcessor`): An instance of [`SmolVLMImageProcessor`]. The image processor is a required input. tokenizer (`PreTrainedTokenizerBase`): An instance of [`PreTrainedTokenizerBase`]. This should correspond with the model's text model. The tokenizer is a required input. video_processor (`SmolVLMImageProcessor`): n instance of [`SmolVLMImageProcessor`]. The video processor is a required input. image_seq_len (`int`, *optional*, defaults to 169): The length of the image sequence i.e. the number of tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)) chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. )image_processor tokenizervideo_processorr# chat_templateZSmolVLMImageProcessorZ AutoTokenizerNc st|dd|_t|dd|_||j|_t|dd|_t|dd|_||_|jd |_ |j |_ |j |_ |j d d |_|jd |_|jd |_tsNtdtj|||fd|i|dS)Nfake_image_tokenzr'zend_of_utterance_tokenzr(z video_sizedo_image_splittingF max_framesfpszbPackage `num2words` is required to run SmolVLM processor. Install it with `pip install num2words`.rA)getattrrCr'Zconvert_tokens_to_idsZimage_token_idrDr(r#Zvideo_samplingrEsize image_sizerFgetZdo_video_splittingdefault_max_frames default_fpsr ImportErrorsuper__init__)selfr>r?r@r#rAkwargs __class__r)r*rQs"     zSmolVLMProcessor.__init__Fc sV|dur fdd|D}dd|D}||f||d|} |dur'd| fS||kr6td|d|d| dd gt|g} | d d gt|g} g} t|| | D]P\} }}g}t||D]\}}t||jjjjd }| |qb| j}t|d krtd |d } t |D]\}}| |||d 7} q| | qV| | fS)Ncg|]}|jqSr)countr'.0samplerRr)r* z3SmolVLMProcessor.process_vision..cSsg|]}t|qSr))len)rZZsublistr)r)r*r]s)rFrJz!The number of images in the text z and images z should be the same.rowsrcols)r'r&r(z.The image token should be present in the text.r) ValueErrorpopr_zipr-r#r'rCr(appendsplit enumerate)rRtextimages output_kwargsrFimage_processor_size processorn_images_in_textZn_images_in_imagesZ image_inputsr$r%Zprompt_stringsr[Z sample_rowsZ sample_colsZimage_prompt_stringsZn_rowsZn_colsZimage_prompt_stringZ split_sampleir)r\r*process_visionsN    zSmolVLMProcessor.process_visionrirhrvideosrSreturnc  s|dur|dur|durtd|dur |du|duAr tdjtfdjji|}|durit|tr:|g}nt|tsJt|dtsJtdtfdd|D}|dkri|duri|duritd |d i}|durt |}j |||d j j j d \}} || n|durt|}j |||d j jjd \}} || |ddd} |durψj|fi|d} j|| dgd|| t|| dS)a, Processes the input prompts and returns a BatchEncoding. Example: ```python >>> import requests >>> from transformers import SmolVLMProcessor >>> from transformers.image_utils import load_image >>> processor = SmolVLMProcessor.from_pretrained("HuggingFaceM4/SmolVLM2-256M-Video-Instruct") >>> processor.image_processor.do_image_splitting = False # Force as False to simplify the example >>> url1 = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" >>> url2 = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg" >>> image1, image2 = load_image(url1), load_image(url2) >>> images = [[image1], [image2]] >>> text = [ ... "In this image, we see", ... "bla bla bla", ... ] >>> outputs = processor(images=images, text=text, return_tensors="pt", padding=True) >>> input_ids = outputs.input_ids >>> input_tokens = processor.tokenizer.batch_decode(input_ids) >>> print(input_tokens) ['<|begin_of_text|>(()*169) In this image, we see', '<|reserved_special_token_0|><|reserved_special_token_0|><|reserved_special_token_0|><|begin_of_text|>bla bla bla(()*169)'] ``` Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`, *optional*): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. If is of type `List[ImageInput]`, it's assumed that this is for a single prompt i.e. of batch size 1. text (`Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). Wherever an image token, `` is encountered it is expanded to `` + `` + `` * `image_seq_len` * `. videos (`List[PIL.Image.Image]`, `np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`, *optional*): The video or batch of videos to be prepared. Each video can be a list of PIL frames, NumPy array or PyTorch tensor. If is of type `List[VideoInput]`, it's assumed that this is for a single prompt i.e. of batch size 1. return_tensors (`Union[str, TensorType]`, *optional*): If set, will return tensors of a particular framework. See [`PreTrainedTokenizerFast.__call__`] for more information. Nz5You must provide one of `text`, `images` or `videos'.z4You must specify exactly one of `images` or `videos`Ztokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringscrVr)rWrYr\r)r*r]/r^z-SmolVLMProcessor.__call__..z We detected z4 tokens in the text but no images/videos were passedr9)rFrkrlZ videos_kwargsr;return_tensorsimage)Z modalities)Z tensor_type)rbZ _merge_kwargsr8r?Z init_kwargs isinstancer5listsumr rorFrKr>updaterrEr@rcZ_check_special_mm_tokensr ) rRrirhZaudiorprSrjrmZinputsZ vision_inputsrrZ text_inputsr)r\r*__call__s`7     zSmolVLMProcessor.__call__ conversations batch_images batch_videosbatch_video_metadatac Kst|}gg}}t||D]Q\}} t|| D]G\} } t| d} t| d} t| d}g}t| | D] \}}||}t|d}t|d}||dd|dq2|||t| qq|D]l}|D]g}d|vrnqgg}|dD]U}|dd kr|d }|d }t t| d }|d t j t |t |d dt|D]\}}|d tj |dd|ddiq|d tdqt||qt||d<qgqc|S)a  Used within `apply_chat_template` when a model has special way to process conversation history. For example, video models might want to specify in the prompt the duration of video or which frame indices at which timestamps were sampled. This information cannot be accessed before the video is loaded. For most models it is a no-op, must be overridden by model processors which require special processing. Args: conversation (`List[Dict, str, str]`): The conversation to process. Always comes in batched format. batch_images (`List[List[ImageInput]]`): Batch of images that were loaded from url/path defined in the conversation. The images are ordered in the same way as in the conversation. Comes in nested list format, one list of `PIL` images per batch. batch_videos (`List[List[ImageInput]]`): Batch of videos that were loaded from url/path defined in the conversation. The videos are ordered in the same way as in the conversation. Comes in nested list format, one list of 4D video arrays per batch. batch_video_metadata (`List[List[Dict[[str, any]]]]`): Batch of metadata returned from loading videos. That includes video fps, duration and total number of framer in original video. Metadata are ordered in the same way as `batch_videos`. Comes in nested list format, one list of 4D video arrays per batch. durationZframes_indicesrH<02d:contenttypevideor)secondsrh)Z frame_countZvideo_duration)rrh) timestamprs)copydeepcopyrdrIr6rer_rLrcrrformatrr5rgrr)rRryrzr{r|Zchat_template_kwargsZbatch_num_framesZbatch_timestampsZ metadata_listZ video_listmetadatarZ duration_secZ frames_idxrHZ timestampsidxZframe_npsecmmssZ conversationmsgZ new_contentblockZcurr_timestampsZcurr_num_framestdrntsr)r)r*#_process_messages_for_chat_templateUsR               #z4SmolVLMProcessor._process_messages_for_chat_templatecO|jj|i|}|S)z This method forwards all its arguments to SmolVLMTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r? batch_decode)rRargsrSZbatched_decode_outputr)r)r*rzSmolVLMProcessor.batch_decodecOr)z This method forwards all its arguments to SmolVLMTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r?decode)rRrrSZ decode_outputr)r)r*rrzSmolVLMProcessor.decodecCs"|jj}|jj}tt||S)N)r?model_input_namesr>rudictfromkeys)rRZtokenizer_input_namesZimage_processor_input_namesr)r)r*rsz"SmolVLMProcessor.model_input_namesopencvrr num_framesrHbackend skip_secsc  sN|dur|jn||dur|jn|fdd}t|||d\}}||fS)a Loads `video` to a numpy array. Args: video (`str` or `VideoInput`): The video to convert to the numpy array format. Can be a link to video or local path. num_frames (`int`, *optional*): Number of frames to sample uniformly. If not passed, the whole video is loaded. fps (`int`, *optional*): Number of frames to sample per second. Should be passed only when `num_frames=None`. If not specified and `num_frames==None`, all frames are sampled. backend (`str`, *optional*, defaults to `"opencv"`): The backend to use when loading the video. Can be any of ["decord", "pyav", "opencv", "torchvision"]. Defaults to "opencv". Returns: Tuple[`np.array`, Dict]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - Metadata dictionary. Ncst|fd|S)N)rG target_fpsr)r)rZ fn_kwargsrGrrr)r*sample_indices_fn_funcszFSmolVLMProcessor._load_video_for_model..sample_indices_fn_func)rZsample_indices_fn)rMrNr) rRrrrHrrrSrrr)rr*_load_video_for_models z&SmolVLMProcessor._load_video_for_model)rBN)FNN)NNNN)NNrr)"r0r1r2__doc__ attributesZ valid_kwargsZimage_processor_classZvideo_processor_classZtokenizer_classr6rr5rQrorr rrrrr8rrxranyrrrpropertyrnparrayr __classcell__r)r)rTr*r=ws|# 2 n W   r=)2rrdatetimertypingrrrrrnumpyrZfeature_extraction_utilsr Z image_utilsr r Zprocessing_utilsr r rrZtokenization_utils_baserrutilsrrrZ video_utilsrrrZvideo_processing_smolvlmrrrrrZ get_loggerr0loggerrr+r,r-r.r8r=__all__r)r)r)r*s8      v