o Zhs @sdZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z mZddlmZddlmZmZmZmZmZmZmZmZmZddlmZmZmZmZm Z erWddl!Z!er^ddl"Z"erodd l#m$Z$dd l%m&Z&m'Z'erwd d l(m)Z)e *e+Z, d1de eee j-fde.de.de/de/f ddZ0de j-de j-fddZ1de j-dee j-e j-ffddZ2d2de j-de j-de.de j-fd d!Z3de j-d"e j-d#e j-d$e j-de j-f d%d&Z4d'e/d(e j-d)e j-d*e j-fd+d,Z5d-d.Z6Gd/d0d0e Z7d0gZ8dS)3z"Image processor class for VitPose.N) TYPE_CHECKINGDictListOptionalTupleUnion)BaseImageProcessor BatchFeatureto_channel_dimension_format) IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDChannelDimension ImageInputinfer_channel_dimension_formatis_scaled_imagemake_list_of_imagesto_numpy_array valid_images) TensorTypeis_scipy_availableis_torch_availableis_vision_availablelogging)inv)affine_transformgaussian_filter)VitPoseEstimatorOutputi@?box image_width image_heightnormalize_factorpadding_factorc Cs|dd\}}}}||} tj||d||dgtjd} || |kr-|d| }n || |kr7|| }tj||||gtjd} | |} | | fS)a Encodes a bounding box in COCO format into (center, scale). Args: box (`Tuple`, `List`, or `np.ndarray`): Bounding box in COCO format (top_left_x, top_left_y, width, height). image_width (`int`): Image width. image_height (`int`): Image height. normalize_factor (`float`): Width and height scale factor. padding_factor (`float`): Bounding box padding factor. Returns: tuple: A tuple containing center and scale. - `np.ndarray` [float32](2,): Center of the bbox (x, y). - `np.ndarray` [float32](2,): Scale of the bbox width & height. N?Zdtype?)nparrayfloat32) r"r#r$r%r&Z top_left_xZ top_left_ywidthheightZ aspect_ratiocenterscaler2c/var/www/auris/lib/python3.10/site-packages/transformers/models/vitpose/image_processing_vitpose.pybox_to_center_and_scale8s$  r4bboxesreturncCsd|dddf|dddfd|dddf<|dddf|dddfd|dddf<|S)a Converts bounding boxes from the COCO format to the Pascal VOC format. In other words, converts from (top_left_x, top_left_y, width, height) format to (top_left_x, top_left_y, bottom_right_x, bottom_right_y). Args: bboxes (`np.ndarray` of shape `(batch_size, 4)): Bounding boxes in COCO format. Returns: `np.ndarray` of shape `(batch_size, 4) in Pascal VOC format. Nrrrr2)r5r2r2r3coco_to_pascal_vocds00r8heatmapsc Cst|tjs td|jdkrtd|j\}}}}|||df}t|d||df}t|d||df}t |d tj }|ddddd f||ddddd f<|dddddf||dddddf<t t |dd k|d}||fS) aGet keypoint predictions from score maps. Args: heatmaps (`np.ndarray` of shape `(batch_size, num_keypoints, height, width)`): Model predicted heatmaps. Returns: tuple: A tuple containing aggregated results. - coords (`np.ndarray` of shape `(batch_size, num_keypoints, 2)`): Predicted keypoint location. - scores (`np.ndarray` of shape `(batch_size, num_keypoints, 1)`): Scores (confidence) of the keypoints. zHeatmaps should be np.ndarrayr'z Heatmaps should be 4-dimensionalr7r)rrr7Nrg) isinstancer+ndarray ValueErrorndimshapereshapeZargmaxZamaxZtileastyper-where) r9 batch_size num_keypoints_r.Zheatmaps_reshapedidxscorespredsr2r2r3get_keypoint_predictionsxs  ,,rIcoordsbatch_heatmapskernelcs$|j\}}}}|jd}|dks||kstdt|ddtfdd|D}t|dd}t|}tj|d d d }|d d|d d|d} | |d|dt d|| d|7} | t dd} || } || d} || |d} || |d} || |d}|| d}|| d|}d| |}d| |}tj ||gdd}| ||dd}| d| |}| d| |}d| | | | | |||}tj ||||gdd}| ||dd}tj |ttjjtd}|td||8}|S)aDARK post-pocessing. Implemented by unbiased_data_processing. Paper references: - Huang et al. The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation (CVPR 2020). - Zhang et al. Distribution-Aware Coordinate Representation for Human Pose Estimation (CVPR 2020). Args: coords (`np.ndarray` of shape `(num_persons, num_keypoints, 2)`): Initial coordinates of human pose. batch_heatmaps (`np.ndarray` of shape `(batch_size, num_keypoints, height, width)`): Batched heatmaps as predicted by the model. A batch_size of 1 is used for the bottom up paradigm where all persons share the same heatmap. A batch_size of `num_persons` is used for the top down paradigm where each person has its own heatmaps. kernel (`int`, *optional*, defaults to 3): Gaussian kernel size (K) for modulation. Returns: `np.ndarray` of shape `(num_persons, num_keypoints, 2)` ): Refined coordinates. rrzQThe batch size of heatmaps should be 1 or equal to the batch size of coordinates.r7csg|] }fdd|DqS)cs g|] }t|dfddqS)g?rr)sigmaradiusZaxes)r).0ZheatmaprOr2r3 s zApost_dark_unbiased_data_processing...r2)rPr9rQr2r3rRsz6post_dark_unbiased_data_processing..gMbP?2)rrrTrrrUedge)mode).r).rr:rr(Zaxiszijmn,ijnk->ijmk)r?r=intr+r,Zcliplogpadflattenaranger@rAZ concatenateZlinalgrZfinfor-ZepseyeZeinsumsqueeze)rJrKrLrCrDr/r.Z num_coordsZbatch_heatmaps_padindexZi_Zix1Ziy1Zix1y1Zix1_y1_Zix1_Ziy1_ZdxZdyZ derivativeZdxxZdyyZdxyZhessianr2rQr3"post_dark_unbiased_data_processingsF    ,    $$rar0r1 output_sizecCs|jddvr tdt|dkrtdt|dkrtdt|dkr)td|d}|d|d d }|d |dd }t|}|d d d f||d |d d |d d d f<|d d df||d|dd |d d df<|S) aoGet final keypoint predictions from heatmaps and apply scaling and translation to map them back to the image. Note: num_keypoints: K Args: coords (`np.ndarray` of shape `(num_keypoints, ndims)`): * If ndims=2, corrds are predicted keypoint location. * If ndims=4, corrds are composed of (x, y, scores, tags) * If ndims=5, corrds are composed of (x, y, scores, tags, flipped_tags) center (`np.ndarray` of shape `(2,)`): Center of the bounding box (x, y). scale (`np.ndarray` of shape `(2,)`): Scale of the bounding box wrt original image of width and height. output_size (`np.ndarray` of shape `(2,)`): Size of the destination heatmaps in (height, width) format. Returns: np.ndarray: Predicted coordinates in the images. r)r7r'z5Coordinates need to have either 2, 4 or 5 dimensions.r7z9Center needs to have 2 elements, one for x and one for y.z,Scale needs to consist of a width and heightz2Output size needs to consist of a height and widthr rr*Nr()r?r=lenr+Z ones_like)rJr0r1rbscale_yscale_xZ target_coordsr2r2r3transform_predss    44rgtheta size_inputsize_dst size_targetcCst|}tjdtjd}|d|d}|d|d}t|||d<t| ||d<|d|dt|d|dt|d|d|d <t|||d <t|||d <|d|dt|d|dt|d|d|d <|S) a Calculate the transformation matrix under the constraint of unbiased. Paper ref: Huang et al. The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation (CVPR 2020). Source: https://github.com/open-mmlab/mmpose/blob/master/mmpose/core/post_processing/post_transforms.py Args: theta (`float`): Rotation angle in degrees. size_input (`np.ndarray`): Size of input image [width, height]. size_dst (`np.ndarray`): Size of output image [width, height]. size_target (`np.ndarray`): Size of ROI in input plane [w, h]. Returns: `np.ndarray`: A matrix for transformation. )r7rr)rrrTrMgr(rr7rrrUrr7)r+Zdeg2radzerosr-mathcossin)rhrirjrkmatrixrfrer2r2r3get_warp_matrix s 66rtcsfddtjdD}t|gdg}t|ddddd d f\d<d<d<d<d <d <fd d|D}tj|dd }|S) a[ This function implements cv2.warpAffine function using affine_transform in scipy. See https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.affine_transform.html and https://docs.opencv.org/4.x/d4/d61/tutorial_warp_affine.html for more details. Note: the original implementation of cv2.warpAffine uses cv2.INTER_LINEAR. csg|]}d|fqS).r2)rPi)srcr2r3rR4sz%scipy_warp_affine..r:)rrrrUrmrMrTrnrlcsg|] }t|ddqS)r)Z output_shapeorder)r)rPZchannel)M_invsizer2r3rRCsrX)ranger?r+Zvstackrstack)rvMryZchannelsZM_scipyZnew_srcr2)rxryrvr3scipy_warp_affine.s( r}cseZdZdZdgZ       d)dedeeee fded e e e fd ed ee e e e fd ee e e e fffd d Z  d*dejdee dee de deee fdeedee eefdejfddZddddddddejdf dede e e e ejfdeedeeee fdeed ee d eed ee e e e fd ee e e e fdee eefde eefdee eefdejjfddZ d+dejdejdejde fd d!Z   d,d"d#de e e e e ejfd$e d%ee d&e ee eff d'd(ZZS)-VitPoseImageProcessora\ Constructs a VitPose image processor. Args: do_affine_transform (`bool`, *optional*, defaults to `True`): Whether to apply an affine transformation to the input images. size (`Dict[str, int]` *optional*, defaults to `{"height": 256, "width": 192}`): Resolution of the image after `affine_transform` is applied. Only has an effect if `do_affine_transform` is set to `True`. Can be overridden by `size` in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `True`): Whether or not to apply the scaling factor (to make pixel values floats between 0. and 1.). rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do_normalize (`bool`, *optional*, defaults to `True`): Whether or not to normalize the input with mean and standard deviation. image_mean (`List[int]`, defaults to `[0.485, 0.456, 0.406]`, *optional*): The sequence of means for each channel, to be used when normalizing images. image_std (`List[int]`, defaults to `[0.229, 0.224, 0.225]`, *optional*): The sequence of standard deviations for each channel, to be used when normalizing images. pixel_valuesTNp?do_affine_transformry do_rescalerescale_factor do_normalize image_mean image_stdc  sptjdi|||_|dur|nddd|_||_||_||_|dur'|nt|_|dur0|nt |_ d|_ dS)N)r/r.r r2) super__init__rryrrrr rrrr%) selfrryrrrrrkwargs __class__r2r3ras  zVitPoseImageProcessor.__init__imager0r1rotation data_formatinput_data_formatr6c Cs|dur|n|}|d|df}t||dt|d|d}|tjkr'|nt|tj|}t|||d|dfd }t||tj}|S) a Apply an affine transformation to an image. Args: image (`np.array`): Image to transform. center (`Tuple[float]`): Center of the bounding box (x, y). scale (`Tuple[float]`): Scale of the bounding box with respect to height/width. rotation (`float`): Rotation angle in degrees. size (`Dict[str, int]`): Size of the destination image. data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format of the output image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format of the input image. Nr.r/g@r*r rr)rvr|ry)rtr+r,rZLASTr r}) rrr0r1rryrrZtransformationr2r2r3rvs   z&VitPoseImageProcessor.affine_transformimagesboxesreturn_tensorsc  s|dur|n|j}|dur|n|j}|dur|n|j}|dur!|n|j}|dur*|n|j}|dur3|n|j}| dur<| n|j} t|}t|sKt dt |t rft |t |krft dt |dt |t |t jrt ||jdkrt dt |d|jddd|D}t|dr|rtddurt|d|jrg} t||D](\}}|D]!}t||d |d |jd \}}|j|||d|d }| |qq| }g}|D]}|r|j||d }|r|j||| d}||qۇfdd|D}d|i}t|| d}|S)a Preprocess an image or batch of images. Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. boxes (`List[List[List[float]]]` or `np.ndarray`): List or array of bounding boxes for each image. Each box should be a list of 4 floats representing the bounding box coordinates in COCO format (top_left_x, top_left_y, width, height). do_affine_transform (`bool`, *optional*, defaults to `self.do_affine_transform`): Whether to apply an affine transformation to the input images. size (`Dict[str, int]` *optional*, defaults to `self.size`): Dictionary in the format `{"height": h, "width": w}` specifying the size of the output image after resizing. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image values between [0 - 1]. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use if `do_normalize` is set to `True`. return_tensors (`str` or [`~utils.TensorType`], *optional*, defaults to `'np'`): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **pixel_values** -- Pixel values to be fed to a model, of shape (batch_size, num_channels, height, width). NzkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.z%Batch of images and boxes mismatch : z != rcSsg|]}t|qSr2)rrPrr2r2r3rRsz4VitPoseImageProcessor.preprocess..zIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.r.r/)r#r$r%)rryr)rr1r)rmeanZstdrcsg|] }t|dqS))Zinput_channel_dimr rrrr2r3rR!s r)dataZ tensor_type)rryrrrrrrrr=r;listrdr+r<r?rloggerZ warning_oncerzipr4r%rappendZrescale normalizer )rrrrryrrrrrrrrZ new_imagesrZ image_boxesr"r0r1Ztransformed_imageZ all_imagesrZencoded_inputsr2rr3 preprocesssl8        z VitPoseImageProcessor.preprocess r9rLc Csb|j\}}}}t|\} } t| ||d} t|D]} t| | || || ||gd| | <q| | fS)aI Get final keypoint predictions from heatmaps and transform them back to the image. Args: heatmaps (`np.ndarray` of shape `(batch_size, num_keypoints, height, width])`): Model predicted heatmaps. center (`np.ndarray` of shape `(batch_size, 2)`): Center of the bounding box (x, y). scale (`np.ndarray` of shape `(batch_size, 2)`): Scale of the bounding box wrt original images of width and height. kernel (int, *optional*, defaults to 11): Gaussian kernel size (K) for modulation, which should match the heatmap gaussian sigma when training. K=17 for sigma=3 and k=11 for sigma=2. Returns: tuple: A tuple containing keypoint predictions and scores. - preds (`np.ndarray` of shape `(batch_size, num_keypoints, 2)`): Predicted keypoint location in images. - scores (`np.ndarray` of shape `(batch_size, num_keypoints, 1)`): Scores (confidence) of the keypoints. rL)r0r1rb)r?rIrarzrg) rr9r0r1rLrCrEr/r.rJrGrHrur2r2r3keypoints_from_heatmaps+s   &z-VitPoseImageProcessor.keypoints_from_heatmapsoutputsr kernel_size threshold target_sizesc$Cs<|jj\}}}}|dur|t|krtdtj|dftjd} tj|dftjd} ttj |} t |D]J} |durY|| d|| d} }t | || |g}| | || | <|j d|j d}}t | | ||d \}}|| | ddf<|| | ddf<q5|j|j| | |d \}}tj|d ftjd}| ddddf|ddddf<| ddddf|dddd f<t|}t|}td|}tt|}g}t|||}|D]<}g}|D]0}t|\}}} |}|}!|dur ||k}"||"}||"}|!|"}!|||!| d }#||#q||q|S) a Transform the heatmaps into keypoint predictions and transform them back to the image. Args: outputs (`VitPoseEstimatorOutput`): VitPoseForPoseEstimation model outputs. boxes (`List[List[List[float]]]` or `np.ndarray`): List or array of bounding boxes for each image. Each box should be a list of 4 floats representing the bounding box coordinates in COCO format (top_left_x, top_left_y, width, height). kernel_size (`int`, *optional*, defaults to 11): Gaussian kernel size (K) for modulation. threshold (`float`, *optional*, defaults to None): Score threshold to keep object detection predictions. target_sizes (`torch.Tensor` or `List[Tuple[int, int]]`, *optional*): Tensor of shape `(batch_size, 2)` or list of tuples (`Tuple[int, int]`) containing the target size `(height, width)` of each image in the batch. If unset, predictions will be resize with the default value. Returns: `List[List[Dict]]`: A list of dictionaries, each dictionary containing the keypoints and boxes for an image in the batch as predicted by the model. NzTMake sure that you pass in as many target sizes as the batch dimension of the logitsr7r)rrr.r/)r#r$rr')Z keypointsrGlabelsZbbox)r9r?rdr=r+ror-r itertoolschainrzr,ryr4rcpunumpytorchZtensorr]r8rnextr_r)$rrrrrrrCrDrEZcentersscalesZflattened_boxesrur#r$Z scale_factorr.r/r0r1rHrGZ all_boxesZposesrZ bboxes_xyxyresultsZpose_bbox_pairsZ image_bboxesZ image_resultsZposeZscoreZ bbox_xyxyZkeypoints_labelsZkeepZ pose_resultr2r2r3post_process_pose_estimationUsZ   $$       z2VitPoseImageProcessor.post_process_pose_estimation)TNTrTNN)NN)r)rNN)__name__ __module__ __qualname____doc__Zmodel_input_namesboolrrstrrYrfloatrrr+r,rrrZFIRSTrr<rPILZImagerrr __classcell__r2r2rr3r~Hs   5        .r~)r r!)r)9rrrptypingrrrrrrrr+Zimage_processing_utilsr r Zimage_transformsr Z image_utilsr rrrrrrrrutilsrrrrrrrZ scipy.linalgrZ scipy.ndimagerrZmodeling_vitposerZ get_loggerrrr<rYrr4r8rIrargrtr}r~__all__r2r2r2r3sT  ,     , "!(@ 0% f