fThYSrSSKrSSKJr SSKJrJrJrJrJ r SSK r SSK J r SSKJrJr SSKJrJrJrJr SS KJrJr SS KJrJrJr SS KJrJrJ r \"5(a S S K!J"r"J#r#J$r$J%r% \(aSSKJ&r& \RN"\(5r)\"5(aSSK*J*r* OSr*Sr+Sr,Sr-"SS\SS9r."SS\SS9r/"SS\5r0S/r1g)z Processor class for SmolVLM. N) timedelta) TYPE_CHECKINGDictListOptionalUnion) BatchFeature) ImageInputmake_nested_list_of_images) ImagesKwargsProcessingKwargsProcessorMixinUnpack) BatchEncoding TextInput)is_num2words_availableis_vision_availablelogging) VideoInput load_videomake_batched_videos)DEFAULT_MEDIA_OUTTRODEFAULT_VIDEO_INTROFRAME_TIMESTAMP_MESSAGEsmolvlm_sample_indices_fn)PreTokenizedInput) num2wordsc Sn[U5H7n[U5H nUUSUS-SUS-S3-UU--- nM" US- nM9 USU3U-UU--U-- nU$)zKPrompt with expanded image tokens for when the image is split into patches.z )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_image_tokentext_split_imagesn_hn_ws f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/smolvlm/processing_smolvlm.py_prompt_split_imager05sZ $C *+sQwiuS1WIQ/OOU`TaerRrr  % T! ! $ %&   " M] * +% % ' c&UU-UU--U-$)z5Prompt with expanded image tokens for a single image.)r&r)r*r+s r/_prompt_single_imager4Js6 # #   " M] * +% % 'r1cLUS:XaUS:Xa [UUUUS9$[X XXE5$)Nr)r)r*r+)r4r0)r'r(r&r)r*r+s r/get_image_prompt_stringr6Ts@Q:?# $;#1   :  r1c@\rSrSr%\\\S'\\\\ 4\S'Sr g)SmolVLMImagesKwargscreturn_row_col_infomax_image_sizer3N) __name__ __module__ __qualname____firstlineno__rbool__annotations__rstrint__static_attributes__r3r1r/r8r8cs!$'T#s(^,,r1r8F)totalc6\rSrSr%\\S'SSSS.SS0S.rSrg ) SmolVLMProcessorKwargsh images_kwargsTF)add_special_tokenspaddingis_split_into_wordsr:) text_kwargsrIr3N)r<r=r>r?r8rA _defaultsrDr3r1r/rGrGhs+&&#'#( "4 Ir1rGc ^\rSrSrSr/SQrSS/rSrSrSr S!S\ S\ \ 4U4Sjjjr S"S jrS#S \\\\\\\4S \\S \\\S 4S \S\\S\4 SjjrS\\\\ \ 4S\\S\\S\\\\ \44SjrSrSr\S5rS$S\\ S4S\ \ S\ \ S\ S\ S\R@4 Sjjr!S r"U=r#$)%SmolVLMProcessorwa9 Constructs a SmolVLM processor which wraps a LLama tokenizer and SmolVLM image processor into a single processor. [`SmolVLMProcessor`] offers all the functionalities of [`SmolVLMImageProcessor`] and [`SmolVLMTokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`SmolVLMImageProcessor`): An instance of [`SmolVLMImageProcessor`]. The image processor is a required input. tokenizer (`PreTrainedTokenizerBase`): An instance of [`PreTrainedTokenizerBase`]. This should correspond with the model's text model. The tokenizer is a required input. video_processor (`SmolVLMImageProcessor`): n instance of [`SmolVLMImageProcessor`]. The video processor is a required input. image_seq_len (`int`, *optional*, defaults to 169): The length of the image sequence i.e. the number of tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)) chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. )image_processor tokenizervideo_processorr& chat_templateSmolVLMImageProcessor AutoTokenizerc 4>[USS5Ul[USS5UlURUR5Ul[USS5Ul[USS5UlX@lURS Ul URUl URUl URRS S 5UlURS UlURS Ul["(d [%S5e[&TU]P"XU4SU0UD6 g)Nfake_image_tokenzr*zend_of_utterance_tokenzr+z video_sizedo_image_splittingF max_framesfpszbPackage `num2words` is required to run SmolVLM processor. Install it with `pip install num2words`.rU)getattrrYr*convert_tokens_to_idsimage_token_idrZr+r&video_samplingr[size image_sizer\getdo_video_splittingdefault_max_frames default_fpsr ImportErrorsuper__init__)selfrRrSrTr&rUkwargs __class__s r/rkSmolVLMProcessor.__init__s!( 3EGb c"9mYG'==d>N>NO&-i9QSg&h#"))5I>"Z*)88F).."1"D"D"1"@"@"D"DEY[`"a"1"@"@"N*99%@yt  _lTaleklr1c JUb*UVs/sHowRUR5PM nnUV s/sHn [U 5PM n n U"U4XES.UD6n UcSU 4$U W:wa[SUSU S35eU R SS/[U5-/5n U R SS/[U5-/5n /n[ XU 5Hupn/n[ UU5HMunn[ UUURURURURS9nURU5 MO URUR5n[U5S:Xa [S 5eUSn[U5HunnUUUUS --- nM URU5 M X4$s snfs sn f) N)r\rcz!The number of images in the text z and images z should be the same.rowsrcols)r*r)r+z.The image token should be present in the text.r) countr*len ValueErrorpopzipr6r&rYr+appendsplit enumerate)rltextimages output_kwargsr\image_processor_size processorsamplen_images_in_textsublistn_images_in_images image_inputsr'r(prompt_strings sample_rows sample_colsimage_prompt_stringsn_rowsn_colsimage_prompt_string split_sampleis r/process_visionSmolVLMProcessor.process_visions  MQRT6 T-=-= >T R:@A&wc'l&A  '9 Xe  <% % !1 134D3E\RdQeeyz "%%fsSY.?@ !%%fsSY.?@ 03Dj0Q ,F#% "%k;"?&=&& $ 0 0,0,A,A'+'>'> '#%++,?@#@"<<(8(89L< A% !QRR"!_F*34H*I&&- QU0CCC+J  ! !& )-1R0++U SAs $FF r|r{rvideosrmreturnc UcUcUc [S5eUcUSLUSL- (a [S5eUR"[4SURR0UD6nUb[ U[ 5(aU/nO8[ U[5(d#[ US[ 5(d [S5e[UVs/sHowRUR5PM sn5nUS:aUcUc[SUS35e0n UbT[U5nURUUUS URURURS 9up*U R!U 5 OVUbS[#U5nURUUUS URUR$UR&S 9up*U R!U 5 US R)S S5n Ub8UR"U40US D6n UR+X,S/S9 U R!U 5 [-XS9$s snf)a Processes the input prompts and returns a BatchEncoding. Example: ```python >>> import requests >>> from transformers import SmolVLMProcessor >>> from transformers.image_utils import load_image >>> processor = SmolVLMProcessor.from_pretrained("HuggingFaceM4/SmolVLM2-256M-Video-Instruct") >>> processor.image_processor.do_image_splitting = False # Force as False to simplify the example >>> url1 = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" >>> url2 = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg" >>> image1, image2 = load_image(url1), load_image(url2) >>> images = [[image1], [image2]] >>> text = [ ... "In this image, we see", ... "bla bla bla", ... ] >>> outputs = processor(images=images, text=text, return_tensors="pt", padding=True) >>> input_ids = outputs.input_ids >>> input_tokens = processor.tokenizer.batch_decode(input_ids) >>> print(input_tokens) ['<|begin_of_text|>(()*169) In this image, we see', '<|reserved_special_token_0|><|reserved_special_token_0|><|reserved_special_token_0|><|begin_of_text|>bla bla bla(()*169)'] ``` Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`, *optional*): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. If is of type `List[ImageInput]`, it's assumed that this is for a single prompt i.e. of batch size 1. text (`Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). Wherever an image token, `` is encountered it is expanded to `` + `` + `` * `image_seq_len` * `. videos (`List[PIL.Image.Image]`, `np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`, *optional*): The video or batch of videos to be prepared. Each video can be a list of PIL frames, NumPy array or PyTorch tensor. If is of type `List[VideoInput]`, it's assumed that this is for a single prompt i.e. of batch size 1. return_tensors (`Union[str, TensorType]`, *optional*): If set, will return tensors of a particular framework. See [`PreTrainedTokenizerFast.__call__`] for more information. Nz5You must provide one of `text`, `images` or `videos'.z4You must specify exactly one of `images` or `videos`tokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringsz We detected z4 tokens in the text but no images/videos were passedrI)r\r~r videos_kwargsrMreturn_tensorsimage) modalities) tensor_type)ru _merge_kwargsrGrS init_kwargs isinstancerBlistsumrsr*r rr\rdrRupdaterr[rTrv_check_special_mm_tokensr ) rlr|r{audiorrmr}rrinputs vision_inputsr text_inputss r/__call__SmolVLMProcessor.__call__s,n  m4 -/*),-A)P %M#&}#A&x< $X/?@ h. %(%;MC)CSBYBS2XB%%C"S&:; &< !'' 3 ''E 3$B*Q*L#C'  ^Eyy(G3*:*>*>q*A*:*>*>q*A'sURR"U0UD6nU$)z This method forwards all its arguments to SmolVLMTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )rS batch_decode)rlargsrmbatched_decode_outputs r/rSmolVLMProcessor.batch_decodes$ !% ; ;T LV L$$r1c>URR"U0UD6nU$)z This method forwards all its arguments to SmolVLMTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rSdecode)rlrrm decode_outputs r/rSmolVLMProcessor.decodes# --t>v> r1cURRnURRn[[R X!-55$)N)rSmodel_input_namesrRrdictfromkeys)rltokenizer_input_namesimage_processor_input_namess r/r"SmolVLMProcessor.model_input_namess< $ @ @&*&:&:&L&L#DMM"="UVWWr1rr num_framesr^backend skip_secsc z^^ ^ Uc UROUm Uc UROUm U UU 4Sjn[XUS9upX4$)aO Loads `video` to a numpy array. Args: video (`str` or `VideoInput`): The video to convert to the numpy array format. Can be a link to video or local path. num_frames (`int`, *optional*): Number of frames to sample uniformly. If not passed, the whole video is loaded. fps (`int`, *optional*): Number of frames to sample per second. Should be passed only when `num_frames=None`. If not specified and `num_frames==None`, all frames are sampled. backend (`str`, *optional*, defaults to `"opencv"`): The backend to use when loading the video. Can be any of ["decord", "pyav", "opencv", "torchvision"]. Defaults to "opencv". Returns: Tuple[`np.array`, Dict]: A tuple containing: - Numpy array of frames in RGB (shape: [num_frames, height, width, 3]). - Metadata dictionary. c$>[U4TTTS.UD6$)N)r] target_fpsr)r)r fn_kwargsr]rrs r/sample_indices_fn_funcFSmolVLMProcessor._load_video_for_model..sample_indices_fn_funcs(,%/JR[_h r1)rsample_indices_fn)rgrhr) rlrrr^rrrmrrr]rs ` @@r/_load_video_for_model&SmolVLMProcessor._load_video_for_modelsF81;0BT,, ),T%%#   %UOefr1) rhrgr\rfrZrYr+r&rdr*rar[)N)FNN)NNNN)NNopencvg)$r<r=r>r?__doc__ attributes valid_kwargsimage_processor_classvideo_processor_classtokenizer_classrCrrBrkrrr rrrrrGrrranyrrrpropertyrnparrayrrD __classcell__)rns@r/rPrPws*EJ#_5L3&O!'+ m  m  } m mFko.,dOSbf! l@j$z"2Dj9I4JJKl@I2DOTJ]E^^_l@  l@ /0 l@ l@\UDc3h01U:&U:& U #4S#X#78 Un%XX%)! %S,&'%SM%c] %  %  % %%r1rP)2rrdatetimertypingrrrrrnumpyrfeature_extraction_utilsr image_utilsr r processing_utilsr rrrtokenization_utils_baserrutilsrrr video_utilsrrrvideo_processing_smolvlmrrrrr get_loggerr<loggerrr0r4r6r8rGrP__all__r3r1r/rs ==4AVV?IIFF<   H %#I* -,e- -U q~qh  r1