o Zh)m @sddlZddlmZmZmZmZmZddlZddl m Z m Z ddl m Z mZmZmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZddlm Z m!Z!m"Z"e!r[ddl#m$Z$e"%e&Z'd eeefd d Z( dde)de)de)de)de)f ddZ*Gddde Z+dgZ,dS)N)DictIterableListOptionalUnion)BaseImageProcessor BatchFeature)convert_to_rgbpadresizeto_channel_dimension_format) OPENAI_CLIP_MEANOPENAI_CLIP_STDChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imageis_valid_imagemake_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments) TensorTypeis_vision_availablelogging)ImagereturncCsxt|ttfrt|dttfrt|ddrdd|DSt|ttfr.t|dr.|St|r5|gStd|)a  Accepts images in list or nested list format, and makes a list of images for preprocessing. Args: images (`Union[List[List[ImageInput]], List[ImageInput], ImageInput]`): The input image. Returns: list: A list of images. rcSsg|] }|D]}|qqSr ).0Zimg_listZimgr r ]/var/www/auris/lib/python3.10/site-packages/transformers/models/emu3/image_processing_emu3.py <z'make_batched_images..z#Could not make batched images from ) isinstancelisttupler ValueError)imagesr r r"make_batched_images0s0 r*@ Pheightwidthfactor min_pixels max_pixelscCs$||ks||krtd|d|d|t||t||dkr0tdt||t||t|||}t|||}|||krit|||}t||||}t||||}||fS|||krt|||}t||||}t||||}||fS)a)Rescales the image so that the following conditions are met: 1. Both dimensions (height and width) are divisible by 'factor'. 2. The total number of pixels is within the range ['min_pixels', 'max_pixels']. 3. The aspect ratio of the image is maintained as closely as possible. zheight:z or width:z must be larger than factor:z4absolute aspect ratio must be smaller than 200, got )r(maxminroundmathsqrtfloorceil)r.r/r0r1r2Zh_barZw_barbetar r r" smart_resizeGs$   r<c seZdZdZddgZdejdddddddddd f d ed ed ed ee e fdede ee e e fde ee e e fdedede de de ddffdd Z ddddddddejdf ded e ed ed e ed e e de ede ee e e fde ee e e fde ede ede eeeffddZ  d*de ejde e e de eeefde eeeffddZdddddddddddejdf ded e ed e eee fd ed e ed e e de ede ee e e fde ee e e fde eded!e eeefde ede eeeffd"d#Z      $ d+ded e ed e e de ede ee e e fde ee e e fd!eeefde eeeffd%d&Z d,d'ejdee ee fdee ee fde eeefdejf d(d)ZZS)-Emu3ImageProcessora  Constructs a Emu3 image processor that dynamically resizes images based on the original images. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image's (height, width) dimensions. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`): Resampling filter to use when resizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `[0.48145466, 0.4578275, 0.40821073]`): Mean to use if normalizing the image. This is a float or list of floats for each channel in the image. image_std (`float` or `List[float]`, *optional*, defaults to `[0.26862954, 0.26130258, 0.27577711]`): Standard deviation to use if normalizing the image. This is a float or list of floats for each channel in the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_pad (`bool`, *optional*, defaults to `True`): Whether to pad the image. If `True`, will pad the patch dimension of the images in the batch to the largest number of patches in the batch. Padding will be applied to the bottom and right with zeros. min_pixels (`int`, *optional*, defaults to `512 * 512`): The min pixels of the image to resize the image. max_pixels (`int`, *optional*, defaults to `1024 * 1024`): The max pixels of the image to resize the image. spatial_factor (`int`, *optional*, defaults to 8): The spatial downsample factor the image will be downsampled in feature extracting phase pixel_values image_sizesTgp?Nii do_resizeresample do_rescalerescale_factor do_normalize image_mean image_stddo_convert_rgbdo_padr1r2spatial_factorrc  s|tjdi| ||_||_||_||_||_|dur|nt|_|dur'|nt |_ | |_ | |_ | |_ | | d|_||_dS)N)r1r2r )super__init__rArBrCrDrErrFrrGr1r2rJsizerH)selfrArBrCrDrErFrGrHrIr1r2rJkwargs __class__r r"rLs  zEmu3ImageProcessor.__init__r) data_formatinput_data_formatc Cst|}| r dd|D}dd|D}t|dr!|r!td| dur+t|d} t|d| d\} } | | }}g}|D]=}|rZt| | |j|j|j d\}}t |||f|| d }|rd|j ||| d }|ro|j |||| d }t || | d }||q>t|}|S) a Preprocess an image or batch of images. Args: images (`ImageInput`): Image or batch of images to preprocess. Expects pixel values ranging from 0 to 255. If pixel values range from 0 to 1, set `do_rescale=False`. vision_info (`List[Dict]`, *optional*): Optional list of dictionaries containing additional information about vision inputs. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the `PILImageResampling` enums. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Mean to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Standard deviation to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. cSg|]}t|qSr )r r!imager r r"r#z2Emu3ImageProcessor._preprocess..cSrTr )rrUr r r"r#rWrzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.N)Z channel_dim)r0r1r2)rMrBrSscalerSrVmeanstdrSZinput_channel_dim)rrloggerZ warning_oncerrr<rJr1r2r rescale normalizer appendnparray)rNr)rArBrCrDrErFrGrHrRrSr.r/Zresized_heightZ resized_widthZprocessed_imagesrVr r r" _preprocesssF1      zEmu3ImageProcessor._preprocesscsDtdd|Dtdd|Dffddt||D}|S)au Pads images on the `num_of_patches` dimension with zeros to form a batch of same number of patches. Args: pixel_values (`List[np.ndarray]`): An array of pixel values of each images of shape (`batch_size`, `num_patches`, `image_in_3D`) image_sizes (`List[List[int]]`): A list of sizes for each image in `pixel_values` in (height, width) format. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: List[`np.ndarray`]: The padded images. cSg|]}|dqS)rr r!rMr r r"r#%rWz8Emu3ImageProcessor._pad_for_batching..cSre)r rfr r r"r#&rWc sDg|]\}}t|dd|dfdd|dffdqS)rrg)paddingrRrS)r )r!rVrMrRrSZ max_shaper r"r#(s&)r4zip)rNr>r?rRrSr rir"_pad_for_batchings z$Emu3ImageProcessor._pad_for_batchingrMreturn_tensorscCsd|dur|n|j}|dur|n|j}|dur|n|j}|dur!|n|j}|dur*|n|j}|dur3|n|j}|dur<|n|j}| durE| n|j} | durN| n|j} | durW| n|j } |durbt |}|durnt |snt dt |||| |||dg}|D]}|j|||||||| | | |d }||q}dd|D}| r|||}t|}t||d| dS) a Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. Shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input aspect ratio. resample (`int`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. do_pad (`bool`, *optional*, defaults to `True`): Whether to pad the image. If `True`, will pad the patch dimension of the images in the batch to the largest number of patches in the batch. Padding will be applied to the bottom and right with zeros. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. NzkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)rDrErFrGrArMrB) rArBrCrDrErFrGrRrHrScSsg|] }|jddqS)N)shaperUr r r"r#r$z1Emu3ImageProcessor.preprocess..)r>r?dataZ tensor_type)rArMrBrCrDrErFrGrHrIr*rr(rrdextendrkrbrcr )rNr)rArMrBrCrDrErFrGrHrIrlrRrSr>rVr?r r r" preprocess3s`B     zEmu3ImageProcessor.preprocessPIL.Image.Imagec CsV|dur|n|j}|durd|jn|}|dur|n|j}|dur#|n|j}|dur,|n|j}t|}t|dtjrGt|dkrC|S|dS|durQt |d}g} |D]C} t | } |rf|j | |||d} |rz|j | ||d} | ddtj} |r|r|dkrt| tj|d } | t| qU| | qUd | i} |dkr|nd}t| |d S) a% Postprocess an image or batch of images tensor. Postprocess is the reverse process of preprocess. The parameters should be same as in preprocess. Args: images (`ImageInput`): Image to postprocess. Expects a single or batch of images with pixel values ranging from -1 to 1. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Ng?rrg)rVrFrGrSrXrsr]r>ro)rCrDrErFrGrr%rlenrr unnormalizer_ZclipZastyperbZuint8r rZLASTraZ fromarrayrqr ) rNr)rCrDrErFrGrlrSr>rVrpr r r" postprocesss6'   zEmu3ImageProcessor.postprocessrVcCsd}t|trt||krtd|dt|n|g|}t|tr7t||kr6td|dt|n|g|}tddt||D}tdd|D}|j||||d}|S) a~ Unnormalizes `image` using the mean and standard deviation specified by `mean` and `std`. image = (image * image_std) + image_mean Args: image (`torch.Tensor` of shape `(batch_size, num_channels, image_size, image_size)` or `(num_channels, image_size, image_size)`): Batch of pixel values to postprocess. image_mean (`float` or `Iterable[float]`): The mean to use for unnormalization. image_std (`float` or `Iterable[float]`): The standard deviation to use for unnormalization. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. rzmean must have z$ elements if it is an iterable, got zstd must have css|] \}}| |VqdSNr )r!r[r\r r r" sz1Emu3ImageProcessor.unnormalize..css|]}d|VqdS)rgNr )r!r\r r r"ry srZ)r%rrur(r'rjr`)rNrVrFrGrSZ num_channelsZrev_image_meanZ rev_image_stdr r r"rvs"      zEmu3ImageProcessor.unnormalize)NN)NNNNNrsNrx)__name__ __module__ __qualname____doc__Zmodel_input_namesrZBICUBICboolrintfloatrrrLrZFIRSTrstrrdrbZndarrayrkrrrrrwrcrrv __classcell__r r rPr"r=fsH      !    d  0      }  Qr=)r+r,r-)-r7typingrrrrrnumpyrbZimage_processing_utilsrr Zimage_transformsr r r r Z image_utilsrrrrrrrrrrrrrutilsrrrZPILrZ get_loggerrzr^r*rr<r=__all__r r r r"s8<    D