o Zh~@spdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZdd lmZd d lmZeeZeGdddeZGddde jZGddde jZ Gddde jZ!Gddde jZ"Gddde jZ#Gddde jZ$Gddde jZ%dEdd Z&Gd!d"d"e jZ'Gd#d$d$e jZ(Gd%d&d&e jZ)e j*j+dFd)e,d*e-fd+d,Z.Gd-d.d.e jZ/Gd/d0d0e jZ0Gd1d2d2e jZ1Gd3d4d4e jZ2Gd5d6d6e jZ3Gd7d8d8e jZ4Gd9d:d:e jZ5Gd;d<dd>e jZ7eGd?d@d@eZ8edAdBGdCdDdDe8Z9dDd@gZ:dS)GzPyTorch ZoeDepth model.N) dataclass)ListOptionalTupleUnion)nn)ACT2FN)DepthEstimatorOutput)PreTrainedModel) ModelOutputauto_docstringlogging) load_backbone)ZoeDepthConfigc@s|eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS) ZoeDepthDepthEstimatorOutputa Extension of `DepthEstimatorOutput` to include domain logits (ZoeDepth specific). Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. predicted_depth (`torch.FloatTensor` of shape `(batch_size, height, width)`): Predicted depth for each pixel. domain_logits (`torch.FloatTensor` of shape `(batch_size, num_domains)`): Logits for each domain (e.g. NYU and KITTI) in case multiple metric heads are used. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, num_channels, height, width)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosspredicted_depth domain_logits. hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrrr]/var/www/auris/lib/python3.10/site-packages/transformers/models/zoedepth/modeling_zoedepth.pyr$s rc>eZdZdZfddZdeejdeejfddZZ S)ZoeDepthReassembleStageaE This class reassembles the hidden states of the backbone into image-like feature representations at various resolutions. This happens in 3 stages: 1. Map the N + 1 tokens to a set of N tokens, by taking into account the readout ([CLS]) token according to `config.readout_type`. 2. Project the channel dimension of the hidden states according to `config.neck_hidden_sizes`. 3. Resizing the spatial dimensions (height, width). Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. c st|j|_t|_t|j|jD]\}}|j t |||dq|jdkrKt|_ |j }|jD]}|j t td||t|jq5dSdS)N)channelsfactorproject)super__init__ readout_typer ModuleListlayerszipneck_hidden_sizesZreassemble_factorsappendZoeDepthReassembleLayerreadout_projectsbackbone_hidden_size SequentialLinearr Z hidden_act)selfconfigZneck_hidden_sizer$ hidden_size_ __class__rr r(Vs     z ZoeDepthReassembleStage.__init__rreturnc Cs6|djd}tj|dd}|dddf|ddddf}}|j\}}}|||||}|dddd}|jdkrX|dd}|jdd |} t|| fd }n |jd krd||d }g} t |j |ddD])\} } |jdkr|j | | } | ddd|d ||} |j | | } | | qo| S) z Args: hidden_states (`List[torch.FloatTensor]`, each of shape `(batch_size, sequence_length + 1, hidden_size)`): List of hidden states from the backbone. rdimNrrr&r%)rr&radd)shapercatZreshapepermute contiguousr)flatten unsqueezeZ expand_as enumeratesplitr0r+r.) r4r patch_height patch_width batch_sizeZ cls_tokenZtotal_batch_sizesequence_lengthZ num_channelsZreadoutoutZ stage_idx hidden_staterrr forwardgs(&     zZoeDepthReassembleStage.forward rrrrr(rrTensorrM __classcell__rrr8r r"Fs &r"c$eZdZfddZddZZS)r/cst|j}tj||dd|_|dkr"tj||||dd|_dS|dkr-t|_dS|dkrBtj||dt d|dd|_dSdS)Nr) in_channelsZ out_channels kernel_sizerrSstridepaddingr) r'r(r1rConv2d projectionConvTranspose2dresizeZIdentityint)r4r5r#r$r6r8rr r(s "z ZoeDepthReassembleLayer.__init__cCs||}||}|SN)rXrZr4rLrrr rMs  zZoeDepthReassembleLayer.forwardrrrr(rMrPrrr8r r/s r/crQ)ZoeDepthFeatureFusionStagecs<tt|_tt|jD] }|jt |qdSr\) r'r(rr*r+rangelenr-r.ZoeDepthFeatureFusionLayer)r4r5r7r8rr r(s  z#ZoeDepthFeatureFusionStage.__init__cCsV|ddd}g}d}t||jD]\}}|dur||}n|||}||q|S)Nr=)r,r+r.)r4rZfused_hidden_statesZfused_hidden_staterLlayerrrr rMs   z"ZoeDepthFeatureFusionStage.forwardr^rrr8r r_s r_cs6eZdZdZfddZdejdejfddZZS)ZoeDepthPreActResidualLayerz ResidualConvUnit, pre-activate residual unit. Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. cst|j|_|jdur|jn|j }t|_tj|j |j ddd|d|_ t|_ tj|j |j ddd|d|_ |jrTtj |j |jd|_tj |j |jd|_dSdS)Nrr)rSrUrVbias)eps)r'r(Z!use_batch_norm_in_fusion_residualuse_batch_normuse_bias_in_fusion_residualrReLU activation1rWfusion_hidden_size convolution1 activation2 convolution2Z BatchNorm2dZbatch_norm_eps batch_norm1 batch_norm2)r4r5rhr8rr r(s8     z$ZoeDepthPreActResidualLayer.__init__rLr:cCsT|}||}||}|jr||}||}||}|jr&||}||Sr\)rjrlrgrormrnrpr4rLZresidualrrr rMs      z#ZoeDepthPreActResidualLayer.forward) rrrrr(rrOrMrPrrr8r rds "rdcs,eZdZdZdfdd Zd ddZZS) rba8Feature fusion layer, merges feature maps from different stages. Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. align_corners (`bool`, *optional*, defaults to `True`): The align_corner setting for bilinear upsample. Tcs@t||_tj|j|jddd|_t||_t||_ dS)NrT)rSre) r'r( align_cornersrrWrkrXrdresidual_layer1residual_layer2)r4r5rrr8rr r( s  z#ZoeDepthFeatureFusionLayer.__init__NcCst|dur#|j|jkrtjj||jd|jdfddd}|||}||}tjj|dd|jd}||}|S)Nr&rbilinearFsizemoderrZ scale_factorrxrr)r?r functional interpolatersrtrrrXrqrrr rMs    z"ZoeDepthFeatureFusionLayer.forward)Tr\rrrrr(rMrPrrr8r rbs  rbcr!) ZoeDepthNeckaO ZoeDepthNeck. A neck is a module that is normally used between the backbone and the head. It takes a list of tensors as input and produces another list of tensors as output. For ZoeDepth, it includes 2 stages: * ZoeDepthReassembleStage * ZoeDepthFeatureFusionStage. Args: config (dict): config dict. c szt||_|jdur|jjdvrd|_nt||_t|_ |j D]}|j tj ||j ddddq$t||_dS)N)Zswinv2rrF)rSrVre)r'r(r5Zbackbone_configZ model_typereassemble_stager"rr*convsr-r.rWrkr_ fusion_stage)r4r5Zchannelr8rr r(2s     zZoeDepthNeck.__init__rr:csvt|ttfs tdt|tjjkrtdjdur%|||}fddt |D} |}||dfS)z Args: hidden_states (`List[torch.FloatTensor]`, each of shape `(batch_size, sequence_length, hidden_size)` or `(batch_size, hidden_size, height, width)`): List of hidden states from the backbone. z2hidden_states should be a tuple or list of tensorszOThe number of hidden states should be equal to the number of neck hidden sizes.Ncsg|] \}}j||qSr)r).0ifeaturer4rr Ssz(ZoeDepthNeck.forward..r=) isinstancetuplelist TypeErrorrar5r- ValueErrorr~rEr)r4rrGrHfeaturesoutputrrr rMCs   zZoeDepthNeck.forwardrNrrr8r r}%s &r}cs:eZdZdZfddZdeejdejfddZZ S)#ZoeDepthRelativeDepthEstimationHeada Relative depth estimation head consisting of 3 convolutional layers. It progressively halves the feature dimension and upsamples the predictions to the input resolution after the first convolutional layer (details can be found in DPT's paper's supplementary material). cst|j|_d|_|jrtjdddddd|_|j}tj||ddddd|_tj ddd d |_ tj|d|j dddd|_ tj|j dddd d|_ dS) N)rr)rrrTr&rrruTryr)r'r( head_in_indexrXZadd_projectionrrWrkconv1ZUpsampleupsamplenum_relative_featuresconv2conv3)r4r5rr8rr r(bs z,ZoeDepthRelativeDepthEstimationHead.__init__rr:cCs||j}|jdur||}t|}||}||}||}t|}|}||}t|}|jdd}||fS)Nrr;) rrXrrirrrrsqueeze)r4rrrrrr rMqs           z+ZoeDepthRelativeDepthEstimationHead.forwardrNrrr8r r[s "rHz>cCsF||}||}|t||t|||t|||S)z%log(nCk) using stirling approximation)rlog)nkrfrrr log_binoms6rcs.eZdZdejffdd ZdddZZS) LogBinomialSoftmaxrcsht||_||_|jdtd|dddddd|jdt|jdgdddddddS) a7Compute log binomial distribution for n_classes Args: n_classes (`int`, *optional*, defaults to 256): Number of output classes. act (`torch.nn.Module`, *optional*, defaults to `torch.softmax`): Activation function to apply to the output. k_idxrrr=F) persistent k_minus_1N) r'r(ractZregister_bufferrarangeviewtensor)r4 n_classesrr8rr r(s $.zLogBinomialSoftmax.__init__?-C6?cCsz|jdkr |d}td||d}t||d}t|j|j|jt||j|jt|}|j||ddS)aCompute the log binomial distribution for probabilities. Args: probabilities (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Tensor containing probabilities of each class. temperature (`float` or `torch.Tensor` of shape `(batch_size, num_channels, height, width)`, *optional*, defaults to 1): Temperature of distribution. eps (`float`, *optional*, defaults to 1e-4): Small number for numerical stability. Returns: `torch.Tensor` of shape `(batch_size, num_channels, height, width)`: Log binomial distribution logbinomial(p;t). rrr;) ndimrDrclamprrrrr)r4 probabilities temperaturerfZone_minus_probabilitiesyrrr rMs   zLogBinomialSoftmax.forward)rr)rrrrsoftmaxr(rMrPrrr8r rsrcs*eZdZ  dfdd ZddZZS)%ZoeDepthConditionalLogBinomialSoftmaxrr&c s~t|||}ttj|||ddddttj|dddddt|_d|_|j |_ |j |_ t |t j d|_dS)aPer-pixel MLP followed by a Conditional Log Binomial softmax. Args: in_features (`int`): Number of input channels in the main feature. condition_dim (`int`): Number of input channels in the condition feature. n_classes (`int`, *optional*, defaults to 256): Number of classes. bottleneck_factor (`int`, *optional*, defaults to 2): Hidden dim factor. rrrTr)rN)r'r(rr2rWZGELUSoftplusmlpp_epsmax_tempmin_temprrrlog_binomial_transform)r4r5 in_featuresZ condition_dimrbottleneck_factor bottleneckr8rr r(s  z.ZoeDepthConditionalLogBinomialSoftmax.__init__cCs|tj||fdd}|dddddf|dddddf}}||j}|ddddf|ddddf|ddddf}||j}|ddddf|ddddf|ddddf}|d}|j|j||j}|||S)az Args: main_feature (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Main feature. condition_feature (torch.Tensor of shape `(batch_size, num_channels, height, width)`): Condition feature. Returns: `torch.Tensor`: Output log binomial distribution rr;Nr&.r)rrconcatrrDrrr)r4Z main_featureZcondition_featureZprobabilities_and_temperaturerrrrr rMs  6 6  z-ZoeDepthConditionalLogBinomialSoftmax.forward)rr&r^rrr8r rs %rcs&eZdZd fdd ZddZZS) ZoeDepthSeedBinRegressorrMbP? cst|j|_|j|_||_||_t|j|ddd|_ tj dd|_ t||ddd|_ |jdkr=tj dd|_dSt |_dS)adBin center regressor network. Can be "normed" or "unnormed". If "normed", bin centers are bounded on the (min_depth, max_depth) interval. Args: config (`int`): Model configuration. n_bins (`int`, *optional*, defaults to 16): Number of bin centers. mlp_dim (`int`, *optional*, defaults to 256): Hidden dimension. min_depth (`float`, *optional*, defaults to 1e-3): Min depth value. max_depth (`float`, *optional*, defaults to 10): Max depth value. rrTZinplacenormedN)r'r(bottleneck_featuresrbin_centers_type min_depth max_depthrrWrriact1rract2)r4r5n_binsmlp_dimrrr8rr r(s *z!ZoeDepthSeedBinRegressor.__init__cCs||}||}||}||}|jdkr\|d}||jddd}|j|j|}tj j |dd|jd}t j |dd }d |d d d d d f|d d dd d f}||fS||fS)z] Returns tensor of bin_width vectors (centers). One vector b for every pixel rrrTr<Zkeepdim)rrrrrrZconstant)rxvaluer;g?Nr=.) rrrrrsumrrrrzpadrZcumsum)r4x bin_centersZbin_widths_normedZ bin_widthsZ bin_edgesrrr rMs     0z ZoeDepthSeedBinRegressor.forward)rrrrr^rrr8r rsr,r&alphagammacCs|d|||S)a:Inverse attractor: dc = dx / (1 + alpha*dx^gamma), where dx = a - c, a = attractor point, c = bin center, dc = shift in bin center This is the default one according to the accompanying paper. Args: dx (`torch.Tensor`): The difference tensor dx = Ai - Cj, where Ai is the attractor point and Cj is the bin center. alpha (`float`, *optional*, defaults to 300): Proportional Attractor strength. Determines the absolute strength. Lower alpha = greater attraction. gamma (`int`, *optional*, defaults to 2): Exponential Attractor strength. Determines the "region of influence" and indirectly number of bin centers affected. Lower gamma = farther reach. Returns: torch.Tensor: Delta shifts - dc; New bin centers = Old bin centers + dc r)divpow)Zdxrrrrr inv_attractor5srcs0eZdZ    d fdd Zd d d ZZS) ZoeDepthAttractorLayerrrrFc st|j|_|j|_|j|_||_||_ ||_ ||_ ||_ |j }}t||ddd|_tjdd|_t||dddd|_tjdd|_dS)zq Attractor layer for bin centers. Bin centers are bounded on the interval (min_depth, max_depth) rrTrr&N)r'r(attractor_alpharZattractor_gammaZgemmaattractor_kindkind n_attractorsrrrmemory_efficientbin_embedding_dimrrWrrirrr r4r5rrrrrrrr8rr r(Js  zZoeDepthAttractorLayer.__init__NTcCs|dur|rtjj||jddddd}||}||}||}||}||}|d}|j\}}}} |||j d|| }|dddddd f} tjj||| fddd} |j sxt j t j d |j} | t| d| d d d } n,t j| | jd } t|j D]}| t| dd|d fd | 7} q|jdkr| |j } | | }|j|j||j} t j| d d \} }t | |j|j} || fS)ao The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape `(batch_size, prev_number_of_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `Tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers normed and scaled. NruTrxrrrr&r.meanrrr;devicer)rrzr{r?rrrrrrrrrrrrrD zeros_likerr`rrsortZclip)r4rprev_binprev_bin_embeddingr{ attractorsrIr7heightwidthZattractors_normedrfuncdelta_crbin_new_centersrrr rMis8    "&  zZoeDepthAttractorLayer.forward)rrrFNTr^rrr8r rIsrcs0eZdZ    d fdd Zd dd ZZS) ZoeDepthAttractorLayerUnnormedrrrTc st||_||_||_||_|j|_|j|_|j |_ ||_ |j }}t ||ddd|_t jdd|_t ||ddd|_t |_dS)zL Attractor layer for bin centers. Bin centers are unbounded rrTrN)r'r(rrrrrrrrrrrrrWrrirrrrrr8rr r(s  z'ZoeDepthAttractorLayerUnnormed.__init__Nc Cs,|dur|rtjj||jddddd}||}||}||}||}||}|jdd\}}tjj|||fddd}|js`t j t j d|j } | t |d|ddd } n,t j||jd } t|jD]} | t |dd| d fd|7} qm|j d kr| |j} || } | }| |fS) a The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape (batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape (batch_size, prev_num_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `Tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers unbounded. Two outputs just to keep the API consistent with the normed version. NrruTrrr&rr;r.r)rrzr{r?rrrrrrrrrrrDrrr`r) r4rrrr{rrrrrrrrrrr rMs.    "&  z&ZoeDepthAttractorLayerUnnormed.forward)rrrTrr^rrr8r rsrcs4eZdZdfdd ZdejdejfddZZS) ZoeDepthProjectorcsDtt||ddd|_tjdd|_t||ddd|_dS)aProjector MLP. Args: in_features (`int`): Number of input channels. out_features (`int`): Number of output channels. mlp_dim (`int`, *optional*, defaults to 128): Hidden dimension. rrTrN)r'r(rrWrrirr)r4r out_featuresrr8rr r(s zZoeDepthProjector.__init__rLr:cCs"||}||}||}|Sr\)rrrr]rrr rM   zZoeDepthProjector.forward)r)rrrr(rrOrMrPrrr8r rsrcsteZdZdZfddZdejdejfddZ  dd ejd ejd ejd eej dee de ejf ddZ Z S)ZoeDepthMultiheadAttentionzKEquivalent implementation of nn.MultiheadAttention with `batch_first=True`.cst||dkrtd|d|d||_t|||_|j|j|_t||j|_ t||j|_ t||j|_ t|||_ t ||_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ())r'r(rnum_attention_headsr[attention_head_size all_head_sizerr3querykeyrout_projDropoutdropout)r4r6rrr8rr r(s  z#ZoeDepthMultiheadAttention.__init__rr:cCs6|dd|j|jf}||}|ddddS)Nr=rr&rr)rwrrrrA)r4rZ new_x_shaperrr transpose_for_scores/s z/ZoeDepthMultiheadAttention.transpose_for_scoresNFquerieskeysvaluesattention_maskoutput_attentionscCs|||}|||}|||}t||dd} | t|j } |dur2| |} t j j | dd} | | } t| |} | dddd} | dd|jf} | | } || } |rm| | f} | S| f} | S)Nr=rr;rr&rr)rrrrrmatmulZ transposemathsqrtrrrzrrrArBrwrrr)r4rrrrrZ query_layerZ key_layerZ value_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsrrr rM4s$     z"ZoeDepthMultiheadAttention.forward)NF)rrrrr(rrOrrrboolrrMrPrrr8r rs&  rcs6eZdZd fdd Z d deejfddZZS) ZoeDepthTransformerEncoderLayer皙?relucst|j}|j}|j}t|||d|_t|||_ t ||_ t|||_ t ||_t ||_t ||_t ||_t||_dS)N)r)r'r(patch_transformer_hidden_sizeZ#patch_transformer_intermediate_sizeZ%patch_transformer_num_attention_headsr self_attnrr3linear1rrlinear2 LayerNormnorm1norm2dropout1dropout2r activation)r4r5rrr6Zintermediate_sizerr8rr r(]s      z(ZoeDepthTransformerEncoderLayer.__init__Nsrc_maskc Csn|}}|j||||dd}|||}||}|||||}|||}||}|S)N)rrrrr) r rr r rrr rr)r4srcrrrZsrc2rrr rMqs  z'ZoeDepthTransformerEncoderLayer.forward)rrr\) rrrr(rrrOrMrPrrr8r r\s rcs4eZdZfddZdejfddZddZZS)ZoeDepthPatchTransformerEncodercsLtj}tfddtjD|_tj|j dddd|_ dS)zViT-like transformer block Args: config (`ZoeDepthConfig`): Model configuration class defining the model architecture. csg|]}tqSr)rrr7r5rr rz.rrrTN) r'r(rrr*r`Znum_patch_transformer_layerstransformer_encoderrWrembedding_convPxP)r4r5rRr8rr r(s   z(ZoeDepthPatchTransformerEncoder.__init__cpuc Cstjd|||dd}tjd|d||dd}t|ttjd|d |}||} tjt| t| gdd} | jdd |dd} | S)zGenerate positional encodings Args: sequence_length (int): Sequence length embedding_dim (int): Embedding dimension Returns: torch.Tensor: Positional encodings. r)dtyperrr&g@rr;) rrrDexprrr@sincosrepeat) r4rIrJ embedding_dimrrpositionindexZdiv_termZ pos_encodingrrr positional_encoding_1ds $z6ZoeDepthPatchTransformerEncoder.positional_encoding_1dcCst||d}tj|d}|ddd}|j\}}}||j||||j|j d}t dD] }|j ||}q.|S)zForward pass Args: x (torch.Tensor - NCHW): Input feature tensor Returns: torch.Tensor - Transformer output embeddings of shape (batch_size, sequence_length, embedding_dim) r&)rrrr)rrr) rrCrrzrrAr?r#rrr`r)r4rZ embeddingsrIrJr rrrr rMs   z'ZoeDepthPatchTransformerEncoder.forward) rrrr(rZfloat32r#rMrPrrr8r rs rcs&eZdZdfdd ZddZZS)ZoeDepthMLPClassifierr:Ncs8t|}t|||_t|_t|||_dSr\)r'r(rr3r rirr )r4rrZhidden_featuresr8rr r(s  zZoeDepthMLPClassifier.__init__cCs"||}||}||}|Sr\)r rr )r4rLrrrr rMrzZoeDepthMLPClassifier.forward)r:Nr^rrr8r r$sr$cs(eZdZdZfddZddZZS)*ZoeDepthMultipleMetricDepthEstimationHeadszn Multiple metric depth estimation heads. A MLP classifier is used to route between 2 different heads. cstjjj|_j|_j}tj||dddd|_ t |_ t ddd|_ |jdkr5tn|jdkrszGZoeDepthMultipleMetricDepthEstimationHeads.__init__..rrrcs g|] }tjddqS)r&r,rrkrr*rr rszGZoeDepthMultipleMetricDepthEstimationHeads.__init__..rc s8i|]dtfddttDqS)r(cs(g|]}|dddqS)rrrrrrrr) Attractorr5 configurationrrr r szRZoeDepthMultipleMetricDepthEstimationHeads.__init__...)rr*r`ra)r)r0r5r)r1r r+ s  c s(i|]}|dt|dddqS)r(rr)r)rrr1)rr5last_inrr r+s)r'r(rnum_attractorsbin_configurationsrrrrWrrpatch_transformerr$mlp_classifierrrZ ModuleDictseed_bin_regressorsrseed_projectorr*r` projectorsrrconditional_log_binomial)r4r5rr8)r0rr5r3rr r(sJ          z3ZoeDepthMultipleMetricDepthEstimationHeads.__init__cs||}||dddddf}||}tj|jddddd}dd|jD} | tj|dd zfdd|jDd} Wnt yVt d d w| d } | d } |j } | |\}}|j d vrx|| | | }n|}||}|j}t|j||D]\}}}||}||||dd\}}|}|}q|}tjj||jddddd}tjj||jddddd}|j}|||}tj||ddd}||fS)NrTrr=r;cSsg|]}|dqSr(rr2rrr r5rzFZoeDepthMultipleMetricDepthEstimationHeads.forward..csg|] }|dkr|qSr<r)rr5Zbin_configurations_namerr r9szbin_configurations_name z! not found in bin_configurationssrrrZhybrid2r{rrurr)rr6r7rrrr5Zargmaxritem IndexErrorrr8rr9rr,r:rrzr{r?r;)r4outconv_activationrfeature_blocksrelative_depthrZ embeddingrZ domain_votenamesr)rrseed_bin_regressorr7seed_bin_centersrrr projector attractorr bin_embeddingbinrlastr;rKrr=r rM)s@          z2ZoeDepthMultipleMetricDepthEstimationHeads.forwardr|rrr8r r%s Tr%crQ)!ZoeDepthMetricDepthEstimationHeadcstjd}|d|d|djjj}|_|_||_j}t j ||dddd|_ |jdkr>t n|jdkrEt td |_t|d |_t fd d td D|_t fdd td D|_jd}t|d|_dS)NrrrrrrTrr'r.r&csg|] }tjdqS)r&r-rr*rr r}s z>ZoeDepthMetricDepthEstimationHead.__init__..rc s"g|] }|dqS))rrrrrr/)r0r5rrrrrr rs)r)r'r(r5rr4rrrrrrWrrrrrFrr9r*r`r:rrrr;)r4r5Zbin_configurationrrr3r8)r0rr5rrrrr r(^sL       z*ZoeDepthMetricDepthEstimationHead.__init__cCs2||}||\}}|jdvr||j|j|j}n|}||} t|j|j|D]\} } } | | } | | || dd\}}| }| } q,|}| d}t j j ||jddddd}tj||gdd}t j j | |jd dddd } ||| }t j j ||jd dddd }tj||ddd }|dfS) Nr>Tr?rr&rurvr;rrr)rrFrrrr9r,r:rclonerDrrzr{r?rr@r;r)r4rBrrCrDrr7rGrrrHrIrrJrKrrLZrelative_conditioningrKrrr rMs,      z)ZoeDepthMetricDepthEstimationHead.forwardr^rrr8r rM]s ;rMc@s$eZdZeZdZdZdZddZdS)ZoeDepthPreTrainedModelZzoedepth pixel_valuesTcCsxt|tjtjtjfr%|jjjd|jj d|j dur#|j j dSdSt|tj r:|j j |jj ddSdS)zInitialize the weightsg)rZstdNr)rrr3rWrYweightdataZnormal_r5Zinitializer_rangereZzero_r Zfill_)r4modulerrr _init_weightss   z%ZoeDepthPreTrainedModel._init_weightsN) rrrrZ config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrTrrrr rOs  rOzU ZoeDepth model with one or multiple metric depth estimation head(s) on top. )Z custom_introcsneZdZfddZe    d dejdeejdee dee dee d e e ej e ff d d ZZS) ZoeDepthForDepthEstimationcst|t||_t|jjdr&t|jjdr&|jjj|_|jjj|_nt dt ||_ t ||_ t|jdkr?t|nt||_|dS)Nr6 patch_sizezXZoeDepth assumes the backbone's config to have `hidden_size` and `patch_size` attributesr)r'r(rbackbonehasattrr5r6r1rVrr}neckr relative_headrar5r%rM metric_headZ post_init)r4r5r8rr r(s      z#ZoeDepthForDepthEstimation.__init__NrPlabelsroutput_hidden_states return_dictr:cCsRd}|dur td|dur|n|jj}|dur|n|jj}|dur$|n|jj}|jj|||d}|j}|j\} } } } |j } | | } | | }| || |\}}|g|}| |\}}|g|}|j |d|d|dd|d\}}|j dd}|s|dur||f|dd}n |f|dd}|dur|f|S|St||||j|jd S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth depth estimation maps for computing the loss. Examples: ```python >>> from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation >>> import torch >>> import numpy as np >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti") >>> model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # interpolate to original size >>> post_processed_output = image_processor.post_process_depth_estimation( ... outputs, ... source_sizes=[(image.height, image.width)], ... ) >>> # visualize the prediction >>> predicted_depth = post_processed_output[0]["predicted_depth"] >>> depth = predicted_depth * 255 / predicted_depth.max() >>> depth = depth.detach().cpu().numpy() >>> depth = Image.fromarray(depth.astype("uint8")) ```NzTraining is not implemented yet)r]rrrr&)rBrrCrDr;)rrrrr)NotImplementedErrorr5Zuse_return_dictr]rrWZforward_with_filtered_kwargsZ feature_mapsr?rVrYrZr[rrrr)r4rPr\rr]r^rrrr7rrrVrGrHrrKrDZ metric_depthrrrrr rMsF-   z"ZoeDepthForDepthEstimation.forward)NNNN)rrrr(r rrrZ LongTensorrrrrOr rMrPrrr8r rUs( rU)r)rr&);rr dataclassesrtypingrrrrrZtorch.utils.checkpointrZ activationsr Zmodeling_outputsr Zmodeling_utilsr utilsr r rZutils.backbone_utilsrZconfiguration_zoedepthrZ get_loggerrloggerrModuler"r/r_rdrbr}rrrrrZjitscriptfloatr[rrrrrrrr$r%rMrOrU__all__rrrr s\        !J?%6 ,,C8_TE$?c y