o Zh+@sdZddlmZddlmZmZddlZddlmZddlm Z ddl m Z m Z dd l mZd d lmZeGd d d e Ze Gddde ZGdddejZGdddejZGdddejZGdddejZGdddejZe ddGdddeZddgZdS)zPyTorch ViTMatte model.) dataclass)OptionalTupleN)nn)PreTrainedModel) ModelOutputauto_docstring) load_backbone)VitMatteConfigc@sbeZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)ImageMattingOutputa Class for outputs of image matting models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Loss. alphas (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Estimated alpha values. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossalphas hidden_states attentions) __name__ __module__ __qualname____doc__rrtorchZ FloatTensor__annotations__rrrrrr]/var/www/auris/lib/python3.10/site-packages/transformers/models/vitmatte/modeling_vitmatte.pyr s r c@s$eZdZeZdZdZgZddZdS)VitMattePreTrainedModel pixel_valuesTcCsDt|tjr|jjjd|jjd|jdur |jj dSdSdS)Ng)meanZstd) isinstancerConv2dweightdataZnormal_configZinitializer_rangebiasZzero_)selfmodulerrr _init_weights@s  z%VitMattePreTrainedModel._init_weightsN) rrrr Z config_classZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesr%rrrrr9s  rcs*eZdZdZdfdd ZddZZS) VitMatteBasicConv3x3zP Basic convolution layers including: Conv3x3, BatchNorm2d, ReLU layers. r csBttj||d||dd|_tj||jd|_t|_ dS)NrF) in_channels out_channels kernel_sizestridepaddingr")Zeps) super__init__rrconv BatchNorm2dZbatch_norm_eps batch_normReLUrelu)r#r!r(r)r+r, __class__rrr.Ls zVitMatteBasicConv3x3.__init__cCs"||}||}||}|SN)r/r1r3r#Z hidden_staterrrforwardYs   zVitMatteBasicConv3x3.forward)r'r rrrrr.r8 __classcell__rrr4rr&Gs r&c(eZdZdZfddZddZZS)VitMatteConvStreamzc Simple ConvStream containing a series of basic conv3x3 layers to extract detail features. cstd}|jdur|jj}|j}t|_|g||_t t |jdD]}|j|}|j|d}|j t |||q'dS)Nr ) r-r.Zbackbone_configZ num_channelsconvstream_hidden_sizesr ModuleListconvs conv_chansrangelenappendr&)r#r!r(r)iZin_chan_Z out_chan_r4rrr.fs     zVitMatteConvStream.__init__cCsJd|i}|}tt|jD]}|j||}dt|d}|||<q |S)NZdetailed_feature_map_0detailed_feature_map_r )rBrCr@str)r#rZout_dictZ embeddingsrEZname_rrrr8ys zVitMatteConvStream.forwardr9rrr4rr<as r<cr;)VitMatteFusionBlockz\ Simple fusion block to fuse features from ConvStream and Plain Vision Transformer. cs"tt|||ddd|_dS)Nr )r+r,)r-r.r&r/)r#r!r(r)r4rrr.s zVitMatteFusionBlock.__init__cCs4tjj|dddd}tj||gdd}||}|S)Nr'ZbilinearF)Z scale_factormodeZ align_cornersr )dim)rZ functionalZ interpolatercatr/)r#featuresZdetailed_feature_mapZupscaled_featuresoutrrrr8s zVitMatteFusionBlock.forwardr9rrr4rrHs rHcr;) VitMatteHeadzJ Simple Matting Head, containing only conv3x3 and conv1x1 layers. c sZt|jd}d}ttj||ddddt|tdtj|ddddd|_dS)Nrr )r*r+r,Tr) r-r.fusion_hidden_sizesrZ Sequentialrr0r2 matting_convs)r#r!r(Z mid_channelsr4rrr.s   zVitMatteHead.__init__cCs||}|Sr6)rRr7rrrr8s zVitMatteHead.forwardr9rrr4rrNs  rNcr;)VitMatteDetailCaptureModulezG Simple and lightweight Detail Capture Module for ViT Matting. c stt|jt|jdkrtd||_t||_|jj |_ t |_ |j g|j|_tt|jdD]}|j t||j||j |d |j|ddq8t||_dS)Nr z_The length of fusion_hidden_sizes should be equal to the length of convstream_hidden_sizes + 1.)r!r(r))r-r.rCrQr> ValueErrorr!r< convstreamrArr? fusion_blocksZ hidden_sizeZfusion_channelsrBrDrHrN matting_head)r#r!rEr4rrr.s&     z$VitMatteDetailCaptureModule.__init__cCs`||}tt|jD]}dtt|j|d}|j||||}q t||}|S)NrFr )rUrBrCrVrGrZsigmoidrW)r#rLrZdetail_featuresrEZdetailed_feature_map_namerrrrr8s z#VitMatteDetailCaptureModule.forwardr9rrr4rrSs rSzX ViTMatte framework leveraging any vision backbone e.g. for ADE20k, CityScapes. )Z custom_introc sbeZdZfddZe     d deejdeedeedeejdeef d d Z Z S) VitMatteForImageMattingcs2t|||_t||_t||_|dSr6)r-r.r!r backbonerSdecoderZ post_init)r#r!r4rrr.s    z VitMatteForImageMatting.__init__Nroutput_attentionsoutput_hidden_stateslabels return_dictc Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}d}|dur(td|jj|||d}|jd}|||} |sR| f|dd} |durP|f| S| St || |j |j dS)a8 labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth image matting for computing the loss. Examples: ```python >>> from transformers import VitMatteImageProcessor, VitMatteForImageMatting >>> import torch >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> processor = VitMatteImageProcessor.from_pretrained("hustvl/vitmatte-small-composition-1k") >>> model = VitMatteForImageMatting.from_pretrained("hustvl/vitmatte-small-composition-1k") >>> filepath = hf_hub_download( ... repo_id="hf-internal-testing/image-matting-fixtures", filename="image.png", repo_type="dataset" ... ) >>> image = Image.open(filepath).convert("RGB") >>> filepath = hf_hub_download( ... repo_id="hf-internal-testing/image-matting-fixtures", filename="trimap.png", repo_type="dataset" ... ) >>> trimap = Image.open(filepath).convert("L") >>> # prepare image + trimap for the model >>> inputs = processor(images=image, trimaps=trimap, return_tensors="pt") >>> with torch.no_grad(): ... alphas = model(**inputs).alphas >>> print(alphas.shape) torch.Size([1, 1, 640, 960]) ```NzTraining is not yet supported)r\r[rOr )rrrr) r!Zuse_return_dictr\r[NotImplementedErrorrYZforward_with_filtered_kwargsZ feature_mapsrZr rr) r#rr[r\r]r^rZoutputsrLroutputrrrr8s*)  zVitMatteForImageMatting.forward)NNNNN) rrrr.r rrZTensorboolr8r:rrr4rrXs&  rX)r dataclassesrtypingrrrrZmodeling_utilsrutilsrr Zutils.backbone_utilsr Zconfiguration_vitmatter r rModuler&r<rHrNrSrX__all__rrrrs.      #) Q