o ZhC@sddlmZmZmZddlmZmZmZmZddl m Z m Z ddl m Z ddlmZmZGddded d ZGd d d ed d Zd ZGdddeZdgZdS))ListOptionalUnion) ImagesKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput) BatchFeature) ImageInputmake_flat_list_of_imagesc@s&eZdZUeeed<eeed<dS)Llama4ImagesKwargsZ max_patchesZresize_to_max_canvasN)__name__ __module__ __qualname__rint__annotations__boolrr[/var/www/auris/lib/python3.10/site-packages/transformers/models/llama4/processing_llama4.pyr"s rF)totalc@s"eZdZUeed<dddiiZdS)Llama4ProcessorKwargs images_kwargs text_kwargsZ padding_sideleftN)rrrrr _defaultsrrrrr's ra>{{- bos_token }} {%- if custom_tools is defined %} {%- set tools = custom_tools %} {%- endif %} {%- if not tools_in_user_message is defined %} {%- set tools_in_user_message = true %} {%- endif %} {%- if not date_string is defined %} {%- if strftime_now is defined %} {%- set date_string = strftime_now("%d %b %Y") %} {%- else %} {%- set date_string = "26 Jul 2024" %} {%- endif %} {%- endif %} {%- if not tools is defined %} {%- set tools = none %} {%- endif %} {#- This block extracts the system message, so we can slot it into the right place. #} {%- if messages[0]['role'] == 'system' %} {%- if messages[0]['content'] is string %} {%- set system_message = messages[0]['content']|trim %} {%- else %} {#- FIXME: The processor requires an array, always. #} {%- set system_message = messages[0]['content'][0]['text']|trim %} {%- endif %} {%- set messages = messages[1:] %} {%- set user_supplied_system_message = true %} {%- else %} {%- set system_message = "" %} {%- set user_supplied_system_message = false %} {%- endif %} {#- System message if the user supplied one #} {%- if user_supplied_system_message %} {{- "<|header_start|>system<|header_end|> " }} {%- if tools is not none %} {{- "Environment: ipython " }} {%- endif %} {%- if tools is not none and not tools_in_user_message %} {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }} {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }} {{- "Do not use variables. " }} {%- for t in tools %} {{- t | tojson(indent=4) }} {{- " " }} {%- endfor %} {%- endif %} {{- system_message }} {{- "<|eot|>" }} {%- endif %} {#- Custom tools are passed in a user message with some extra guidance #} {%- if tools_in_user_message and not tools is none %} {#- Extract the first user message so we can plug it in here #} {%- if messages | length != 0 %} {%- set first_user_message = messages[0]['content']|trim %} {%- set messages = messages[1:] %} {%- else %} {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }} {%- endif %} {{- '<|header_start|>user<|header_end|> ' -}} {{- "Given the following functions, please respond with a JSON for a function call " }} {{- "with its proper arguments that best answers the given prompt. " }} {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }} {{- "Do not use variables. " }} {%- for t in tools %} {{- t | tojson(indent=4) }} {{- " " }} {%- endfor %} {{- first_user_message + "<|eot|>"}} {%- endif %} {%- for message in messages %} {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %} {{- '<|header_start|>' + message['role'] + '<|header_end|> ' }} {%- if message['content'] is string %} {{- message['content'] }} {%- else %} {%- for content in message['content'] %} {%- if content['type'] == 'image' %} {{- '<|image|>' }} {%- elif content['type'] == 'text' %} {{- content['text'] }} {%- endif %} {%- endfor %} {%- endif %} {{- "<|eot|>" }} {%- elif 'tool_calls' in message and message.tool_calls|length > 0 %} {{- '<|header_start|>assistant<|header_end|> ' -}} {{- '<|python_start|>' }} {%- if message['content'] is string %} {{- message['content'] }} {%- else %} {%- for content in message['content'] %} {%- if content['type'] == 'image' %} {{- '<|image|>' }} {%- elif content['type'] == 'text' %} {{- content['text'] }} {%- endif %} {%- endfor %} {%- endif %} {{- '<|python_end|>' }} {%- for tool_call in message.tool_calls %} {{- '{"name": "' + tool_call.function.name + '", ' }} {{- '"parameters": ' }} {{- tool_call.function.arguments | tojson }} {{- "}" }} {%- endfor %} {{- "<|eot|>" }} {%- elif message.role == "tool" or message.role == "ipython" %} {{- "<|header_start|>ipython<|header_end|> " }} {%- if message.content is mapping or message.content is iterable %} {{- message.content | tojson }} {%- else %} {{- message.content }} {%- endif %} {{- "<|eot|>" }} {%- endif %} {%- endfor %} {%- if add_generation_prompt %} {{- '<|header_start|>assistant<|header_end|> ' }} {%- endif %} c seZdZdZddgZgdZdZdZdddd d d d d d ddef de de ffdd Z ddZ    d"de ede eeeeeeefdeedefddZddZddZed d!ZZS)#Llama4Processora Constructs a Llama4 processor which wraps a [`AutoImageProcessor`] and [`PretrainedTokenizerFast`] tokenizer into a single processor that inherits both the image processor and tokenizer functionalities. See the [`~Llama4Processor.__call__`] and [`~Llama4Processor.decode`] for more information. Args: image_processor ([`AutoImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`], *optional*): The tokenizer is a required input. patch_size (`int`, *optional*, defaults to 28): The size of image patches for tokenization. img_size (`int`, *optional*, defaults to 364): The size of the image to be tokenized. This should correspond to the size given to the image processor. image_token (`str`, *optional*, defaults to `"<|image|>"`): The token to be used to represent an image in the text. downsample_factor (`int`, *optional*, defaults to 1): The factor by which to scale the patch size. start_of_img_token (`str`, *optional*, defaults to `"<|START_OF_IMG|>"`): The token to be used to represent the start of an image in the text. end_of_img_token (`str`, *optional*, defaults to `"<|END_OF_IMG|>"`): The token to be used to represent the end of an image in the text. img_patch_token (`str`, *optional*, defaults to `"<|IMG_PATCH|>"`): The token to be used to represent an image patch in the text. img_line_break_token (`str`, *optional*, defaults to `"<|IMG_LINE_BREAK|>"`): The token to be used to represent a line break in the text. tile_token (`str`, *optional*, defaults to `"TILE"`): The token to be used to represent an image patch in the text. tile_global_token (`str`, *optional*, defaults to `"TILE_GLOBAL"`): The token to be used to represent the cover image in the text. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer) chat_template image_token patch_sizeZimg_sizeZdownsample_factorstart_of_img_tokenend_of_img_tokenimg_patch_tokenZimg_line_break_token tile_tokentile_global_tokenZAutoImageProcessorZ AutoTokenizerNg? <|image|><|image_start|> <|image_end|> <|patch|><|tile_x_separator|><|tile_y_separator|>r#pixel_shuffle_ratioc  sjtj||| dttd|d|_||_||_||_||j|_ ||_ ||_ | |_ | |_ | |_dS)N)r!g?)super__init__rrounddownsample_ratior#fake_image_tokenr"Zconvert_tokens_to_idsZimage_token_idr$r%r&r'r()selfrr r#r0r6r"Zstart_of_image_tokenZend_of_image_tokenZ patch_tokenZtile_x_separator_tokenZtile_y_separator_tokenr!kwargs __class__rrr3fs zLlama4Processor.__init__cCs|d}|\}}||dkr.t|D]}t|D]}|d|7}||dkr(|d7}q|d7}q|d7}|d|7}|d7}|S)z Create a structured string representation of image tokens Args: num_patches: Number of patches in the image Returns: String with appropriate image tokens r+r-r.r/r*r,)range)r7Z aspect_rationum_patches_per_chunkZ img_stringZratio_hZratio_wyyxxrrr_prompt_split_images        z#Llama4Processor._prompt_split_imageimagestextr8returnc s|durtdjtfdjji|}t|ttfs|g}i}|durt|}j dd|i|d}|ddj dd\}} t |j | j j } |d } tfd d |D} | t|krttd | d t|dd} g}|D]E}|j}|dkr||qz|j}g}t|D]\}}||||kr| | | }| d7} ||q|d|qz| t|krtd|}|ddd}j|fi|d}j||dgdti|||dS)au Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` and `kwargs` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] to encode the text. To prepare the vision inputs, this method forwards the `images` and `kwargs` arguments to Llama4ImageProcessor's [`~Llama4ImageProcessor.__call__`] if `images` is not `None`. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. NzYou have to specify text.Ztokenizer_init_kwargsrArZ pixel_valuesr aspect_ratiosc3s|] }|jVqdSN)countr6).0promptr7rr sz+Llama4Processor.__call__..zFound z) placeholders across the batch, but have z flattened images.r;zONumber of image placeholders in the prompt does not match the number of images.rreturn_tensorsimage)Z modalities)dataZ tensor_typer) ValueErrorZ _merge_kwargsrr Z init_kwargs isinstancelisttuplerrshaperr#r5popsumlenrGr6appendsplit enumerater@joinZ_check_special_mm_tokensr )r7rArBZaudioZvideosr8Z output_kwargsZ image_inputsZ image_heightZ image_widthr=rEZtotal_placeholdersZ image_indexZprocessed_textrIZplaceholder_countZ prompt_splitsZ new_promptZlocal_image_indexZ split_partZtokens_for_this_imagerMZ text_inputsrrJr__call__sj&        zLlama4Processor.__call__cO|jj|i|S)z This method forwards all its arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoder7argsr8rrrr^zLlama4Processor.batch_decodecOr])z This method forwards all its arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r decoder_rrrrb razLlama4Processor.decodecCs |jj}|jj}t|t|SrF)r model_input_namesrrR)r7Ztokenizer_input_namesZimage_processor_input_namesrrrrcsz!Llama4Processor.model_input_names)NNNN)rrr__doc__ attributesZ valid_kwargsZimage_processor_classZtokenizer_classr!rfloatr3r@rr rr r rrrr r\r^rbpropertyrc __classcell__rrr9rr3sR!  crN)typingrrrZtransformers.processing_utilsrrrrZ$transformers.tokenization_utils_baser r Zimage_processing_utilsr Z image_utilsr rrrr!r__all__rrrrs   e