o Zh42@sdZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z mZmZdd lmZd d lmZeeZeGd d d e ZeGddde ZdddZGddde jZGddde jZeddGdddeZddgZdS)zPyTorch VitPose model.) dataclass)OptionalTupleUnionN)nn)PreTrainedModel) ModelOutputauto_docstringlogging) load_backbone) VitPoseConfigc@sjeZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)VitPoseEstimatorOutputaQ Class for outputs of pose estimation models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Loss is not supported at this moment. See https://github.com/ViTAE-Transformer/ViTPose/tree/main/mmpose/models/losses for further detail. heatmaps (`torch.FloatTensor` of shape `(batch_size, num_keypoints, height, width)`): Heatmaps as predicted by the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossheatmaps. hidden_states attentions) __name__ __module__ __qualname____doc__rrtorchZ FloatTensor__annotations__rrrrrr[/var/www/auris/lib/python3.10/site-packages/transformers/models/vitpose/modeling_vitpose.pyr's rc@s>eZdZeZdZdZdZdee j e j e j fddfddZ dS) VitPosePreTrainedModelZvit pixel_valuesTmodulereturnNcCst|tjtjfr0tjj|jjt j d|j j d|jj |j_|jdur.|jjdSdSt|tjrE|jj|jjddSdS)zInitialize the weightsg)meanZstdNg?) isinstancerLinearConv2dinitZ trunc_normal_weightdatatorZfloat32configZinitializer_rangeZdtypebiasZzero_ LayerNormZfill_)selfrrrr _init_weightsJs    z$VitPosePreTrainedModel._init_weights)rrrrZ config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrr"r#r*r,rrrrrCs &rgaussian-heatmapc Cs|dvrtd|jdkrtd|j\}}}}d}|dkr7d}|dddddd f |dddddd f<||d |||}|}|D]$\} } |dd| d f|dd| d f<|dd| d f|dd| d f<qH|||||f}|d }|S) aFlip the flipped heatmaps back to the original form. Args: output_flipped (`torch.tensor` of shape `(batch_size, num_keypoints, height, width)`): The output heatmaps obtained from the flipped images. flip_pairs (`torch.Tensor` of shape `(num_keypoints, 2)`): Pairs of keypoints which are mirrored (for example, left ear -- right ear). target_type (`str`, *optional*, defaults to `"gaussian-heatmap"`): Target type to use. Can be gaussian-heatmap or combined-target. gaussian-heatmap: Classification target with gaussian distribution. combined-target: The combination of classification target (response map) and regression target (offset map). Paper ref: Huang et al. The Devil is in the Details: Delving into Unbiased Data Processing for Human Pose Estimation (CVPR 2020). Returns: torch.Tensor: heatmaps that flipped back to the original image )r-combined-targetz9target_type should be gaussian-heatmap or combined-targetzCoutput_flipped should be [batch_size, num_keypoints, height, width]r r.rN.) ValueErrorndimshapereshapeclonetolistflip) Zoutput_flipped flip_pairsZ target_type batch_sizeZ num_keypointsheightwidthZchannelsZoutput_flipped_backleftrightrrr flip_backYs" . " r>csDeZdZdZd fdd Zd dejdeejdejfdd ZZ S) VitPoseSimpleDecoderz Simple decoding head consisting of a ReLU activation, 4x upsampling and a 3x3 convolution, turning the feature maps into heatmaps. rNcsHtt|_tj|jddd|_tj|j j |j dddd|_ dS)NZbilinearF) scale_factormodeZ align_cornersrr  kernel_sizestridepadding) super__init__rReLU activationZUpsampler@ upsamplingr#backbone_config hidden_size num_labelsconvr+r( __class__rrrGs   zVitPoseSimpleDecoder.__init__ hidden_stater8cCs4||}||}||}|durt||}|SN)rIrJrNr>r+rRr8rrrrforwards    zVitPoseSimpleDecoder.forward)rNrS) rrrrrGrTensorrrU __classcell__rrrPrr?s* r?csBeZdZdZdeffdd Zd dejdeejfdd Z Z S) VitPoseClassicDecoderz Classic decoding head consisting of a 2 deconvolutional blocks, followed by a 1x1 convolution layer, turning the feature maps into heatmaps. r(csttj|jjdddddd|_td|_t |_ tjddddddd|_ td|_ t |_ tjd|jdddd|_dS) Nr/r F)rCrDrEr)rrB)rFrGrZConvTranspose2drKrLdeconv1Z BatchNorm2d batchnorm1rHrelu1deconv2 batchnorm2relu2r#rMrNrOrPrrrGs     zVitPoseClassicDecoder.__init__NrRr8cCs\||}||}||}||}||}||}||}|dur,t||}|SrS)r[r\r]r^r_r`rNr>rTrrrrUs        zVitPoseClassicDecoder.forwardrS) rrrrrrGrrVrrUrWrrrPrrXs$rXz? The VitPose model with a pose estimation head on top. )Z custom_introcseZdZdeddffdd Ze      ddejdeejdeejd eejd ee d ee d ee de e e ffd dZ ZS)VitPoseForPoseEstimationr(rNcs|t|t||_t|jjdstdt|jjds!tdt|jjds,td|jr3t|nt ||_ | dS)NrLz0The backbone should have a hidden_size attribute image_sizez0The backbone should have an image_size attribute patch_sizez/The backbone should have a patch_size attribute) rFrGr backbonehasattrr(r1Zuse_simple_decoderr?rXheadZ post_initrOrPrrrGs   z!VitPoseForPoseEstimation.__init__r dataset_indexr8labelsoutput_attentionsoutput_hidden_states return_dictcCsF|dur|n|jj}|dur|n|jj}|dur|n|jj}d}|dur(td|jj|||||d} |r:| jdn| dd} | jd} |jj j d|jj j d} |jj j d|jj j d} | ddd | d| | } |j| |d}|s|r|f| dd}n |f| dd}|dur|f|S|St||| j| jd S) ac dataset_index (`torch.Tensor` of shape `(batch_size,)`): Index to use in the Mixture-of-Experts (MoE) blocks of the backbone. This corresponds to the dataset index used during training, e.g. For the single dataset index 0 refers to the corresponding dataset. For the multiple datasets index 0 refers to dataset A (e.g. MPII) and index 1 refers to dataset B (e.g. CrowdPose). flip_pairs (`torch.tensor`, *optional*): Whether to mirror pairs of keypoints (for example, left ear -- right ear). Examples: ```python >>> from transformers import AutoImageProcessor, VitPoseForPoseEstimation >>> import torch >>> from PIL import Image >>> import requests >>> processor = AutoImageProcessor.from_pretrained("usyd-community/vitpose-base-simple") >>> model = VitPoseForPoseEstimation.from_pretrained("usyd-community/vitpose-base-simple") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> boxes = [[[412.8, 157.61, 53.05, 138.01], [384.43, 172.21, 15.12, 35.74]]] >>> inputs = processor(image, boxes=boxes, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> heatmaps = outputs.heatmaps ```NzTraining is not yet supported)rgrjrirkr0rr rZ)r8)rrrr)r(Zuse_return_dictrjriNotImplementedErrorrdZforward_with_filtered_kwargsZ feature_mapsr3rKrbrcZpermuter4 contiguousrfrrr)r+rrgr8rhrirjrkrZoutputsZsequence_outputr9Z patch_heightZ patch_widthroutputrrrrUs@( z VitPoseForPoseEstimation.forward)NNNNNN)rrrrrGr rrVrboolrtuplerrUrWrrrPrras4 ra)r-)r dataclassesrtypingrrrrZtorch.utils.checkpointrZmodeling_utilsrutilsr r r Zutils.backbone_utilsr Zconfiguration_vitposerZ get_loggerrloggerrrr>Moduler?rXra__all__rrrrs.       (& g