o ZhMM@sdZddlZddlZddlmZmZmZddlZddl m Z ddl m Z m Z ddlmZmZmZmZddlmZmZmZmZmZmZmZdd lmZmZmZmZdd l m!Z!ergddl"Z"dd l#m$Z$m%Z%m&Z&ernddl'Z'e(e)Z*d Z+d dZ,         d'de-de.de-de-de.de.de.de.dee/dee-de$j$fddZ0 d(d ej1d!e-d"eee-e2ffd#d$Z3Gd%d&d&e Z4d&gZ5dS))z%Image processor class for Pix2Struct.N)DictOptionalUnion)hf_hub_download)BaseImageProcessor BatchFeature)convert_to_rgb normalizeto_channel_dimension_format to_pil_image)ChannelDimension ImageInputget_image_sizeinfer_channel_dimension_formatmake_list_of_imagesto_numpy_array valid_images) TensorTypeis_torch_availableis_vision_availablelogging)requires_backends)Image ImageDraw ImageFontzybelkada/fontscCsttdg|d}tjjj|||f||fd}||d|d||d}| ddddd|d||d||d||}|dS) a Utiliy function to extract patches from a given image tensor. Returns a tensor of shape (1, `patch_height`, `patch_width`, `num_channels`x `patch_height` x `patch_width`) Args: image_tensor (torch.Tensor): The image tensor to extract patches from. patch_height (int): The height of the patches to extract. patch_width (int): The width of the patches to extract. torchr)Zstrider) rtorch_extract_patches unsqueezernn functionalZunfoldreshapesizeZpermute)Z image_tensor patch_height patch_widthpatchesr*i/var/www/auris/lib/python3.10/site-packages/transformers/models/pix2struct/image_processing_pix2struct.pyr!4s    r!$blackwhitetext text_size text_colorbackground_color left_padding right_padding top_paddingbottom_padding font_bytes font_pathreturnc Csttdtjdd} | j|d} d| } |dur$| dur$t|} n | dur+| } ntt d} t j | d|d } t td d |}|d | | \}}}}|||}|||}td ||f|}t |}|j||f| || d |S)a Render text. This script is entirely adapted from the original script that can be found here: https://github.com/google-research/pix2struct/blob/main/pix2struct/preprocessing/preprocessing_utils.py Args: text (`str`, *optional*, defaults to ): Text to render. text_size (`int`, *optional*, defaults to 36): Size of the text. text_color (`str`, *optional*, defaults to `"black"`): Color of the text. background_color (`str`, *optional*, defaults to `"white"`): Color of the background. left_padding (`int`, *optional*, defaults to 5): Padding on the left. right_padding (`int`, *optional*, defaults to 5): Padding on the right. top_padding (`int`, *optional*, defaults to 5): Padding on the top. bottom_padding (`int`, *optional*, defaults to 5): Padding on the bottom. font_bytes (`bytes`, *optional*): Bytes of the font to use. If `None`, the default font will be used. font_path (`str`, *optional*): Path to the font to use. If `None`, the default font will be used. visionP)width)r0 Nz Arial.TTFzUTF-8)encodingr&RGB)rrrr)Zxyr0fillfont)r render_texttextwrap TextWrapperwrapjoinioBytesIOrDEFAULT_FONT_PATHrZtruetyperZDrawrnewZtextbboxr0)r0r1r2r3r4r5r6r7r8r9wrapperlinesZ wrapped_textrCZ temp_draw_ text_widthZ text_height image_width image_heightimageZdrawr*r*r+rDOs$ &        rDrSheaderinput_data_formatc Ksttdt||d}t|fi|}t|j|j}t|j||j}t|j||j}t d|||fd}| | ||fd| | ||fd|ft |}t |tjkrbt|tj}|S)a Renders the input text as a header on the input image. Args: image (`np.ndarray`): The image to render the header on. header (`str`): The header text. data_format (`Union[ChannelDimension, str]`, *optional*): The data format of the image. Can be either "ChannelDimension.channels_first" or "ChannelDimension.channels_last". Returns: `np.ndarray`: The image with the header rendered. r;)rUr@r.rAr)r render_headerr rDmaxr=intheightrrLZpasteresizerrr ZLASTr ) rSrTrUkwargsZ header_imageZ new_widthZ new_heightZnew_header_heightZ new_imager*r*r+rVs   rVcs@eZdZdZdgZ     ddeded eeee fd e d ed df fd d Z dde j d e d e deeeefd e j f ddZ  dde j deeeefdeeeefd e j fddZddddddejdfdedeedeedeed ee d eeee fdeeeefdedeeeefd efddZZS)Pix2StructImageProcessoraf Constructs a Pix2Struct image processor. Args: do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method. According to Pix2Struct paper and code, the image is normalized with its own mean and standard deviation. patch_size (`Dict[str, int]`, *optional*, defaults to `{"height": 16, "width": 16}`): The patch size to use for the image. According to Pix2Struct paper and code, the patch size is 16x16. max_patches (`int`, *optional*, defaults to 2048): The maximum number of patches to extract from the image as per the [Pix2Struct paper](https://arxiv.org/pdf/2210.03347.pdf). is_vqa (`bool`, *optional*, defaults to `False`): Whether or not the image processor is for the VQA task. If `True` and `header_text` is passed in, text is rendered onto the input images. flattened_patchesTNFdo_convert_rgb do_normalize patch_size max_patchesis_vqar:c sFtjdi||dur|nddd|_||_||_||_||_dS)N)rYr=r*)super__init__rar`r_rbrc)selfr_r`rarbrcr[ __class__r*r+rfs   z!Pix2StructImageProcessor.__init__rSrUc Kst|jdt|tj|}t|}|d|d}}t|tj\}} t ||||| } t t t | |||d} t t t | | ||d} t | |d} t | |d}tj jj|d| |fdddd d}t|||}|j}|d}|d }|d }||||g}t||dgd|||dg}t|d|g|d||dg}|d7}|d7}|tj}|tj}t|||gd }tj j|ddd|||g}t|}|S) a Extract flattened patches from an image. Args: image (`np.ndarray`): Image to extract flattened patches from. max_patches (`int`): Maximum number of patches to extract. patch_size (`dict`): Dictionary containing the patch height and width. Returns: result (`np.ndarray`): A sequence of `max_patches` flattened patches. rrYr=rrZbilinearFT)r&modeZ align_cornersZ antialiasr rr)rextract_flattened_patchesr r FIRSTrZ from_numpyrmathsqrtrWminfloorr#r$Z interpolater"Zsqueezer!shaper%Zarangerepeattofloat32catpadfloatr)rgrSrbrarUr[r'r(rRrQscaleZnum_feasible_rowsZnum_feasible_colsZresized_heightZ resized_widthr)Z patches_shaperowscolumnsdepthZrow_idsZcol_idsresultr*r*r+rksF   **  $z2Pix2StructImageProcessor.extract_flattened_patches data_formatcKsb|jtjkr |tj}t|}t|}t|dt t |j }t |f||||d|S)a Normalize an image. image = (image - image_mean) / image_std. The image std is to mimic the tensorflow implementation of the `per_image_standardization`: https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization Args: image (`np.ndarray`): Image to normalize. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. g?)meanstdr}rU) ZdtypenpZuint8astypertr~rrWrmrnprodrqr )rgrSr}rUr[r~rZadjusted_stddevr*r*r+r 5s    z"Pix2StructImageProcessor.normalizeimages header_textreturn_tensorsc  sd|dur|nj}|dur|nj}durnjdur!njj} | dddur3tdt|}t|s?td|rHdd|D}dd|D}durYt |d| rdurctd | d d| d dt t r{gt |fd dt|D}|rfd d|D}fdd|D}dd|D} t|| d|d} | S)a Preprocess an image or batch of images. The processor first computes the maximum possible number of aspect-ratio preserving patches of size `patch_size` that can be extracted from the image. It then pads the image with zeros to make the image respect the constraint of `max_patches`. Before extracting the patches the images are standardized following the tensorflow implementation of `per_image_standardization` (https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization). Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images. header_text (`Union[List[str], str]`, *optional*): Text to render as a header. Only has an effect if `image_processor.is_vqa` is `True`. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. max_patches (`int`, *optional*, defaults to `self.max_patches`): Maximum number of patches to extract. patch_size (`dict`, *optional*, defaults to `self.patch_size`): Dictionary containing the patch height and width. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Nr}z8data_format is not an accepted input as the outputs are zkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.cSg|]}t|qSr*)r .0rSr*r*r+ z7Pix2StructImageProcessor.preprocess..cSrr*)rrr*r*r+rrrz.A header text must be provided for VQA models.r8r9cs$g|]\}}t||dqS))r8r9)rV)rirS)r8r9rr*r+rscsg|] }j|dqS))rSrU)r r)rUrgr*r+rscsg|] }j|dqS))rSrbrarU)rkr)rUrbrargr*r+rs cSs$g|]}|jdddktjqS)r)Zaxisr)sumrrrtrr*r*r+rs$)r]Zattention_mask)dataZ tensor_type)r`r_rarbrcget ValueErrorrrrpop isinstancestrlen enumerater)rgrrr_r`rbrarr}rUr[rcZattention_masksZencoded_outputsr*)r8r9rrUrbrargr+ preprocess\sJ5     z#Pix2StructImageProcessor.preprocess)TTNr^FN)NN)__name__ __module__ __qualname____doc__Zmodel_input_namesboolrrrrXrfrndarraydictrr rkr rlrrr __classcell__r*r*rhr+r\s T *   r\) r,r-r.r/r/r/r/NNr)6rrIrmtypingrrrnumpyrZhuggingface_hubrZimage_processing_utilsrrZimage_transformsr r r r Z image_utilsr rrrrrrutilsrrrrZutils.import_utilsrrEZPILrrrrZ get_loggerrloggerrKr!rrXbytesrDrChildProcessErrorrVr\__all__r*r*r*r+s~ $     E *