o ZhY @s.ddlZddlmZddlmZmZmZmZddlm Z ddl m Z m Z ddl mZmZmZmZmZdd lmZmZerKdd lmZdd lmZmZerZdd lmZdd lmZee Z!dZ"Gdddej#Z$GdddZ%de&deee'ee'dedffddZ(eeddGdddeZ)dS)N)Iterable)DictListOptionalUnion)GenerationConfig)ProcessingKwargsUnpack)add_end_docstringsis_torch_availableis_vision_availableloggingrequires_backends)Pipelinebuild_pipeline_init_args)Image) load_images valid_images)*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES) KeyDatasetzc@seZdZdZdZdZdS) ReturnTyperrrN)__name__ __module__ __qualname__TENSORSNEW_TEXT FULL_TEXTrrX/var/www/auris/lib/python3.10/site-packages/transformers/pipelines/image_text_to_text.pyr/src @s>eZdZdZ ddedeeeeededffddZ dS) Chata This class is intended to just be used internally in this pipeline and not exposed to users. We convert chats to this format because the rest of the pipeline code tends to assume that lists of messages are actually a batch of samples rather than messages in the same conversation.Nmessagesimages Image.ImagecCs6|D]}d|vr d|vstdqt||}||_dS)NrolecontentzQWhen passing chat dicts as input, each dict must have a 'role' and 'content' key.) ValueErroradd_images_to_messagesr")selfr"r#messagerrr __init__:s   z Chat.__init__N) rrr__doc__rrrstrrr+rrrr r!5sr!r"r#r$cs|durg}n t|trt|tr|g}d}|D]_}|dD]Xtts&qd}|dkrPtfdddDsO|t|krK||d<|d 7}qtd q|d krvtd trrd d vrrdd<d d d<d =qtd qq|t|krtd |S)zS Retrieve and combine images from the chat and the images passed as input. Nrr&typeimagec3s|]}|vVqdSr,r).0keyr&rr Vsz)add_images_to_messages..)r0urlpathbase64rzlThe number of images in the chat messages should be the same as the number of images passed to the pipeline.Z image_urlr5zhWrong format for 'image_url' content type. The content should have an 'image_url' dict with a 'url' key.) isinstancerr.dictgetanylenr')r"r#Z idx_imagesr* content_typerr3r r(EsD        r(T)Z has_processorc seZdZdZdZdZdZdZdZe ddZ fddZ         dd e e fd d Z  dd eeeeeeeed ed eed eefdeeeeeeefffdd ZdddZdddZejdfddZZS)ImageTextToTextPipelinea2 Image-text-to-text pipeline using an `AutoModelForImageTextToText`. This pipeline generates text given an image and text. When the underlying model is a conversational model, it can also accept one or more chats, in which case the pipeline will operate in chat mode and will continue the chat(s) by adding its response(s). Each chat takes the form of a list of dicts, where each dict contains "role" and "content" keys. Unless the model you're using explicitly sets these generation parameters in its configuration files (`generation_config.json`), the following default values will be used: - max_new_tokens: 256 Example: ```python >>> from transformers import pipeline >>> pipe = pipeline(task="image-text-to-text", model="Salesforce/blip-image-captioning-base") >>> pipe("https://huggingface.co/datasets/Narsil/image_dummy/raw/main/parrots.png", text="A photo of") [{'generated_text': 'a photo of two birds'}] ``` ```python >>> from transformers import pipeline >>> pipe = pipeline("image-text-to-text", model="llava-hf/llava-interleave-qwen-0.5b-hf") >>> messages = [ >>> { >>> "role": "user", >>> "content": [ >>> { >>> "type": "image", >>> "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg", >>> }, >>> {"type": "text", "text": "Describe this image."}, >>> ], >>> }, >>> { >>> "role": "assistant", >>> "content": [ >>> {"type": "text", "text": "There is a dog and"}, >>> ], >>> }, >>> ] >>> pipe(text=messages, max_new_tokens=20, return_full_text=False) [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'}, {'type': 'text', 'text': 'Describe this image.'}]}, {'role': 'assistant', 'content': [{'type': 'text', 'text': 'There is a dog and'}]}], 'generated_text': ' a person in the image. The dog is sitting on the sand, and the person is sitting on'}] ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This image-text to text pipeline can currently be loaded from pipeline() using the following task identifier: "image-text-to-text". See the list of available models on [huggingface.co/models](https://huggingface.co/models?pipeline_tag=image-text-to-text). TF)max_new_tokenscs*tj|i|t|d|tdS)NZvision)superr+rZcheck_model_typer)r)argskwargs __class__rr r+s z ImageTextToTextPipeline.__init__NrCc Ks:i} i} i} | | |dur|| d<| dur| | d<|dur#|| d<|dur?d| vr/i| d<d| dvr9td|| dd<|durW|durW|durOtd|rTtjntj}|durb|durbtj}|durj|| d<| durr| | d<|durz|| d<|dur|jjj|d d }t |d krt d |d |d<| | | fS)Ntimeoutcontinue_final_messagegenerate_kwargsr@zp'max_new_tokens' is defined twice, once in 'generate_kwargs' and once as a direct parameter, please use only onez>`return_full_text` is mutually exclusive with `return_tensors` return_typeclean_up_tokenization_spacesF)Zadd_special_tokensrzStopping on a multiple token sequence is not yet supported on transformers. The first token of the stop sequence will be used as the stop sequence string in the interim.rZ eos_token_id) updater'rrrr processorZ tokenizerencoder<logger warning_once)r)r@rHrFZreturn_full_textreturn_tensorsrIrJZ stop_sequencerGrCZforward_kwargsZpreprocess_paramsZpostprocess_paramsZstop_sequence_idsrrr _sanitize_parameterssL       z,ImageTextToTextPipeline._sanitize_parametersr#r$textc sv|dur |dur tddd}||rGt|dtr(tjt||fi|S|dur3dgt|}ddt||D}tj|fi|S|durs||rst|dtrbtjt|fi|Sdd|D}tj|fi|S|dur|durt|s tj|fi|St |j d ddurt d |durtj|fi|S|durtd tj||d fi|S) a3 Generate a text given text and the image(s) passed as inputs. Args: images (`str`, `List[str]`, `PIL.Image, `List[PIL.Image]`, `List[Dict[str, Union[str, PIL.Image]]]`): The pipeline handles three types of images: - A string containing a HTTP(s) link pointing to an image - A string containing a local path to an image - An image loaded in PIL directly The pipeline accepts either a single image or a batch of images. Finally, this pipeline also supports the chat format (see `text`) containing images and text in this argument. text (str, List[str], `List[Dict[str, Union[str, PIL.Image]]]`): The text to be used for generation. If a list of strings is passed, the length of the list should be the same as the number of images. Text can also follow the chat format: a list of dictionaries where each dictionary represents a message in a conversation. Each dictionary should have two keys: 'role' and 'content'. 'role' should be one of 'user', 'system' or 'assistant'. 'content' should be a list of dictionary containing the text of the message and the type of the message. The type of the message can be either 'text' or 'image'. If the type is 'image', no text is needed. return_tensors (`bool`, *optional*, defaults to `False`): Returns the tensors of predictions (as token indices) in the outputs. If set to `True`, the decoded text is not returned. return_text (`bool`, *optional*): Returns the decoded texts in the outputs. return_full_text (`bool`, *optional*, defaults to `True`): If set to `False` only added text is returned, otherwise the full text is returned. Cannot be specified at the same time as `return_text`. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not to clean up the potential extra spaces in the text output. continue_final_message( `bool`, *optional*): This indicates that you want the model to continue the last message in the input chat rather than starting a new one, allowing you to "prefill" its response. By default this is `True` when the final message in the input chat has the `assistant` role and `False` otherwise, but you can manually override that behaviour by setting this flag. Return: A list or a list of list of `dict`: Each result comes as a dictionary with the following key (cannot return a combination of both `generated_text` and `generated_token_ids`): - **generated_text** (`str`, present when `return_text=True`) -- The generated text. - **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True`) -- The token ids of the generated text. - **input_text** (`str`) -- The input text. Nz0You must at least provide either text or images.cSs$t|tttfot|dtttfS)Nr)r8listtuplerr9)argrrr _is_chat=s$z2ImageTextToTextPipeline.__call__.._is_chatrcSsg|] \}}t||qSrr!)r1Zchatr0rrr Gsz4ImageTextToTextPipeline.__call__..cSsg|]}t|qSrrW)r1r0rrr rXQsZ chat_templatea The input data was not formatted as a chat with dicts containing 'role' and 'content' keys, even though this model supports chat. Consider using the chat format for better results. For more information, see https://huggingface.co/docs/transformers/en/chat_templatingz(You must provide text for this pipeline.)r#rR) r'r8r9rA__call__r!r<ziprgetattrrLrNrO)r)r#rRrCrVZchatsrDrr rYs6<z ImageTextToTextPipeline.__call__c Kst|tr'|dur|jdddk}|jj|j| ||jddd}||d<|St|tttfr6d}|}|}nt |d|d}|d}|d}t|ttfrYt |d krY| d d|jd |||jd |j |j d }||d<|S)Nr% assistantT)Zadd_generation_promptrGrPtokenizeZ return_dictrRr#)rFrpadding)r#rRrP)Zdtyper)r8r!r"rLZapply_chat_templateZ frameworkrSrTr.rr< setdefaulttoZ torch_dtype) r)ZinputsrFrGZprocessing_kwargs model_inputsr#rRZ inputs_textrrr preprocessns6  z"ImageTextToTextPipeline.preprocesscCsT|durin|}|d}d|vr|dn|d}|jjdi||}|||dS)NrR input_idsZdecoder_input_ids)generated_sequence prompt_textrdr)popmodelgenerate)r)rbrHrfrdrerrr _forwards   z ImageTextToTextPipeline._forwardc s|dtttfrgn|d|d}|tjkr+fddttDS|jjfi|}|jj|fi|}|tj tj hvryg}t ||D](\} } | | } d| kradkrqnn| | | t| dqN| | qN|}|tj krg} t |D]h\} }t| tr| |}nUt| tr|dur| jdd d k}|rt| jdd d}|d |7<t| jdd| jdd | jdd dd|gd g}n t| jd |d g}| |q| }ddt |D}|S)Nrfrerdcsg|] }||dqS)) input_textZgenerated_token_idsr)r1ireZ input_textsrr rXsz7ImageTextToTextPipeline.postprocess..rrr\r%r]r&rR)r%r&cSs*g|]\}}t|tr|jn||dqS))rkgenerated_text)r8r!r")r1rkrnrrr rXs )r8r.r!rrranger<rLZpost_process_image_text_to_textrrrZfindappendr"r9itemsrS)r)Z model_outputsrIrGZpostprocess_kwargsrdZgenerated_textsZdecoded_inputsZnew_generated_textsZtext_generatedZ decoded_inputZindex_input_textZ full_textsrfrnnew_textrecordsrrmr postprocesssV           z#ImageTextToTextPipeline.postprocess) NNNNNNNNN)NN)NNNr,)rrrr-Z_load_processorZ_load_image_processorZ_load_feature_extractorZ_load_tokenizerZ_pipeline_calls_generaterZ_default_generation_configr+r r rQrrr.rr9rYrcrjrrru __classcell__rrrDr r>tsZ=  G    p % r>)*enumcollections.abcrtypingrrrrZ generationrZprocessing_utilsr r utilsr r r rrbaserrZPILrZ image_utilsrrZmodels.auto.modeling_autorZpt_utilsrZ get_loggerrrNZ IMAGE_TOKENEnumrr!r9r.r(r>rrrr s0       /