fTh|<SrSSKrSSKJrJrJrJrJrJ r J r SSK r SSK r SSK J r SSKJrJrJr SSKJr SSKJrJrJrJr SS KJrJr SS KJrJr SS KJ r J!r!J"r" SS K#J$r$ S SK%J&r& \!RN"\(5r)"SS\ RT5r+"SS\ RT5r,S>S\ RTS\ RZS\ RZS\ RZS\\ RZS\.S\.4Sjjr/"SS\ RT5r0"SS\ RT5r1"SS \ RT5r2"S!S"\ RT5r3S?S#\ RZS$\.S%\4S&\ RZ4S'jjr5"S(S)\ RT5r6"S*S+\ RT5r7"S,S-\ RT5r8"S.S/\ RT5r9"S0S1\ RT5r:\ "S2S3\55r;\ "S4S5\;55r<\ "S6S79"S8S9\;55r=\ "S:S79"S;S<\;\$55r>/S=Qr?g)@zPyTorch DINOv2 model.N)CallableDictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int) BackboneMixin) Dinov2Configc^\rSrSrSrS\SS4U4SjjrS\RS\ S \ S\R4S jr SS \RS \ \RS\R4S jjr Sr U=r$)Dinov2Embeddings%zE Construct the CLS token, mask token, position and patch embeddings. configreturnNc>[TU]5 [R"[R "SSUR 55UlUR(a:[R"[R"SUR 55Ul [U5Ul URRn[R"[R "SUS-UR 55Ul[R"UR 5UlUR$UlURUlXlg)Nr)super__init__r Parametertorchrandn hidden_size cls_tokenuse_mask_tokenzeros mask_tokenDinov2PatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_sizer!)selfr!r0 __class__s b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/dinov2/modeling_dinov2.pyr%Dinov2Embeddings.__init__*s ekk!Q8J8J&KL   ll5;;q&:L:L+MNDO 5f =++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++$33 embeddingsheightwidthcURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5nURn [RRUR[R5X4SS S 9RU S 9nURSSSS5RSSU5n[R "Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and interpolation at torch.float32 precision. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicF)sizemode align_cornersdtypedim)shaper1r'jit is_tracingr5rreshapepermuterFr functional interpolatetofloat32viewcat) r6r;r<r=r0 num_positionsclass_pos_embedpatch_pos_embedrH new_height new_widthsqrt_num_positions target_dtypes r8interpolate_pos_encoding)Dinov2Embeddings.interpolate_pos_encoding8s~!&&q)A- 0066q9A= yy##%%+*F6?++ +221bqb59221ab59r". __, &}c'9:)11!5G]`a)11!Q1=&,, --33   u}} -( 4 "<" *11!Q1=BB1b#Nyy/;CCr: pixel_valuesbool_masked_posc>URup4pVURRRRnURUR US95nUbjUR (aY[R"URS5URR UR5RS5U5nURRUSS5n [R"X4SS9nXRXU5-nURU5nU$)NrEr?rrrG)rIr/ projectionweightrFrPr+r'where unsqueezer-r*expandrSr[r4) r6r]r^ batch_size_r<r=rZr; cls_tokenss r8forwardDinov2Embeddings.forward`s'3'9'9$ v,,77>>DD **+>))"-t/A/A*BRBR/S/]/]^_/`blJ ^^**:r2> YY 7Q?  "?"? TY"ZZ \\*- r:)r*r!r4r-r/r5r1r+N)__name__ __module__ __qualname____firstlineno____doc__rr%r'Tensorintr[rrh__static_attributes__ __classcell__r7s@r8rr%s |  &D5<<&D&DUX&D]b]i]i&DPELL8ELLCYejeqeqr:rcf^\rSrSrSrU4SjrS\RS\R4SjrSr U=r $)r.vz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c>[TU]5 URURp2URUR pT[ U[RR5(aUOX"4n[ U[RR5(aUOX34nUSUS-USUS--nX lX0lX@lX`l [R"XEX3S9Ul g)Nrr) kernel_sizestride)r$r% image_sizer5 num_channelsr) isinstance collectionsabcIterabler0r Conv2dr`)r6r!rzr5r{r)r0r7s r8r%Dinov2PatchEmbeddings.__init__}s !'!2!2F4E4EJ$*$7$79K9Kk#-j+//:R:R#S#SZZdYq #-j+//:R:R#S#SZZdYq !!} 1 5*Q-:VW=:XY $$(&))L:ir:r]r"cURSnX R:wa[SURSUS35eURU5R S5R SS5nU$)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r@)rIr{ ValueErrorr`flatten transpose)r6r]r{r;s r8rhDinov2PatchEmbeddings.forwardsx#))!, ,, ,!../yaI __\2::1=GG1M r:)rzr{r0r5r`) rkrlrmrnror%r'rprhrrrsrts@r8r.r.vs. jELLU\\r:r.modulequerykeyvalueattention_maskscalingr4c [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr?)rHrF)ptrainingrr@) r'matmulrr rNsoftmaxrQrPrFr4r contiguous) rrrrrrr4kwargs attn_weights attn_outputs r8eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r:c ^\rSrSrS\SS4U4SjjrS\RS\R4SjrS S\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) Dinov2SelfAttentionr!r"Nc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rgFbias)r$r%r)num_attention_headshasattrrr!rqattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalr Linearqkv_biasrrrr6r!r7s r8r%Dinov2SelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=2O\M]r:) rrr!rrrrrrrNF)rkrlrmrnrr%r'rprrboolr rrhrrrsrts@r8rrs]|]](%ell%u||% bg!(0(>!Z^! uU\\5<</0% 2EE F!!r:rc^\rSrSrSrS\SS4U4SjjrS\RS\RS\R4S jr S r U=r $) Dinov2SelfOutputz The residual connection is defined in Dinov2Layer instead of here (as is the case with other models), due to the layernorm applied before each block. r!r"Nc>[TU]5 [R"URUR5Ul[R "UR5Ulgrj) r$r%r rr)denser2r3r4rs r8r%Dinov2SelfOutput.__init__sB YYv1163E3EF zz&"<"<= r:r input_tensorcJURU5nURU5nU$rjrr4)r6rrs r8rhDinov2SelfOutput.forwards$ =1  ]3 r:r) rkrlrmrnrorr%r'rprhrrrsrts@r8rrsI >|>> U\\RWR^R^r:rc ^\rSrSrS\SS4U4SjjrS\\SS4SjrS S\ RS \ \ RS \ S\ \\ R\ R4\\ R44S jjrS rU=r$)Dinov2Attentioni r!r"Nc>[TU]5 [U5Ul[ U5Ul[ 5Ulgrj)r$r%r attentionroutputset pruned_headsrs r8r%Dinov2Attention.__init__ s0 ,V4&v. Er:headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)NrrrG)lenrrrrrrrrrrrrunion)r6rindexs r8 prune_headsDinov2Attention.prune_headss u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r:rrrcfURXU5nURUSU5nU4USS-nU$)Nrr)rr)r6rrr self_outputsattention_outputrs r8rhDinov2Attention.forward"sC ~~m@QR ;;|A F#% QR(88r:)rrrr)rkrlrmrnrr%rrqrr'rprrr rrhrrrsrts@r8rr s"|"" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F  r:rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)Dinov2LayerScalei0r"c>[TU]5 [R"UR[ R "UR5-5Ulgrj) r$r%r r&layerscale_valuer'onesr)lambda1rs r8r%Dinov2LayerScale.__init__1s8 ||F$;$;ejjI[I[>\$\] r: hidden_statecXR-$rjrr6rs r8rhDinov2LayerScale.forward5sll**r:rr"N rkrlrmrnr%r'rprhrrrsrts@r8rr0s)^+ELL+U\\++r:rinput drop_probrr"cUS:XdU(dU$SU- nURS4SURS- --nU[R"X@RUR S9-nUR 5 URU5U-nU$)a* Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)rFdevice)rIndimr'randrFrfloor_div)rrr keep_probrI random_tensorrs r8 drop_pathr:sCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr:c^\rSrSrSrS S\\SS4U4SjjjrS\RS\R4Sjr S\ 4S jr S r U=r$) Dinov2DropPathiOzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrr"c.>[TU]5 Xlgrj)r$r%r)r6rr7s r8r%Dinov2DropPath.__init__Rs "r:rcB[XRUR5$rj)rrr)r6rs r8rhDinov2DropPath.forwardVs FFr:c8SRUR5$)Nzp={})formatrr6s r8 extra_reprDinov2DropPath.extra_reprYs}}T^^,,r:)rrj)rkrlrmrnrorfloatr%r'rprhstrrrrrsrts@r8rrOsQb#(5/#T##GU\\GellG-C--r:rcf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $) Dinov2MLPi]r"cz>[TU]5 UR=p#[URUR-5n[ R "X$SS9Ul[UR[5(a[URUl OURUl [ R "XCSS9Ul g)NTr)r$r%r)rq mlp_ratior rfc1r| hidden_actrr activationfc2r6r! in_features out_featureshidden_featuresr7s r8r%Dinov2MLP.__init__^s %+%7%77 f0063C3CCD99[E f'' - -$V%6%67DO$//DO99_Fr:rclURU5nURU5nURU5nU$rj)r rrrs r8rhDinov2MLP.forwardis2xx - |4 xx - r:)rr rrrrts@r8r r ]s) GELLU\\r:r cf^\rSrSrSU4SjjrS\R S\R 4SjrSrU=r $)Dinov2SwiGLUFFNipr"c$>[TU]5 UR=p#[URUR-5n[US-S- 5S-S-S-n[ R "USU-SS9Ul[ R "XCSS9Ulg)Nr@rTr) r$r%r)rqr r r weights_in weights_outrs r8r%Dinov2SwiGLUFFN.__init__qs %+%7%77 f0063C3CCD2Q67!;AAE))K_1D4P99_Nr:rcURU5nURSSS9up#[RR U5U-nUR U5$)Nr@r?rG)rchunkr rNsilur)r6rx1x2hiddens r8rhDinov2SwiGLUFFN.forwardzsQ|4 ##A2#.##B'",''r:)rrrrrts@r8rrps)O(ELL(U\\((r:rc ^\rSrSrSrS\SS4U4SjjrS S\RS\ \RS \ S\ \ \R\R4\ \R44S jjr S rU=r$) Dinov2LayerizCThis corresponds to the Block class in the original implementation.r!r"Nc>[TU]5 [R"URUR S9Ul[U5Ul[U5Ul URS:a[UR5O[R"5Ul[R"URUR S9UlUR (a[#U5UlO['U5Ul[U5Ulg)Nepsr)r$r%r LayerNormr)layer_norm_epsnorm1rrr layer_scale1drop_path_raterIdentityrnorm2use_swiglu_ffnrmlpr  layer_scale2rs r8r%Dinov2Layer.__init__s \\&"4"4&:O:OP (0,V4BHBWBWZ]B](=(=>cecncncp\\&"4"4&:O:OP  &v.DH (DH,V4r:rrrc>URURU5UUS9nUSnURU5nUSSnURU5U-nUR U5nUR U5nUR U5nURU5U-nU4U-nU$)N)rrr)rr-r.rr1r3r4)r6rrrself_attention_outputsrr layer_outputs r8rhDinov2Layer.forwards "& JJ} % /"0"  2!4,,-=>(,'78=H zz-0 xx - ((6 ~~l3mC /G+r:)rrr.r4r3r-r1r)rkrlrmrnrorr%r'rprrr rrhrrrsrts@r8r'r'sM5|55&-1"' ||ELL)  uU\\5<</0% 2EE F r:r'c^\rSrSrS\SS4U4SjjrS S\RS\\RS\ S \ S \ S\ \ \ 44 S jjr S rU=r$) Dinov2Encoderir!r"Nc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr) r$r%r!r ModuleListrangenum_hidden_layersr'layergradient_checkpointingr6r!rfr7s r8r%Dinov2Encoder.__init__sR  ]]vG_G_A`#aA`AK$7A`#ab &+#$bsA&rrroutput_hidden_states return_dictcU(aSOSnU(aSOSn[UR5HzupU(aXa4-nUbX(OSn UR(a0UR(aUR U R UU U5n O U "XU5n U SnU(dMrX{S4-nM| U(aXa4-nU(d[ SXU455$[UUUS9$)Nrrc3.# UH ocMUv M g7frjrG).0vs r8 (Dinov2Encoder.forward..sm$[q$[s )last_hidden_stater attentions) enumerater@rAr_gradient_checkpointing_func__call__tupler) r6rrrrDrEall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r8rhDinov2Encoder.forwards#7BD$5b4(4OA#$58H$H!.7.CilO**t}} $ A A ))!#% ! !-]M^ _ )!,M  &91=M[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul UR5 g)Nr)) r$r%r!rr;r;encoderr r+r)r, layernorm post_initrs r8r%Dinov2Model.__init__sW   *62$V, f&8&8f>S>ST r:r"c.URR$rjr;r/rs r8get_input_embeddings Dinov2Model.get_input_embeddings ///r:heads_to_pruneNcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr@rr)r6r{r@rs r8 _prune_headsDinov2Model._prune_heads#s< +002LE LL  u % / / ; ;E B3r:r]r^rrrDrEcUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR X0RR 5nURXS9nURUUUUUS9nUSn URU 5n U SS2SSS24n U(d X4n XSS-$[U U URURS9$)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Only relevant for pre-training. Nz You have to specify pixel_values)r^rrrDrErr)rM pooler_outputrrN) r!rrDuse_return_dictr get_head_maskr?r;rrrsrrrN) r6r]r^rrrDrEembedding_outputencoder_outputssequence_output pooled_output head_outputss r8rhDinov2Model.forward+s2C1N-TXT_T_TqTq$8$D $++JjJj &1% 4 'tn 4 d^4  u00 14 4 r:rpz Dinov2 Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. ) custom_introc^\rSrSrS\SS4U4Sjjr\SS\\RS\\RS\\RS \\ S \\ S \\ S\ \ \ 44S jj5rS rU=r$)Dinov2ForImageClassificationicr!r"Nc6>[TU]U5 URUl[U5UlURS:a.[ R "URS-UR5O[ R"5Ul UR5 g)Nrr@) r$r% num_labelsrpr\r rr)r0 classifierrtrs r8r%%Dinov2ForImageClassification.__init__jsy   ++!&) EKDUDUXYDYBIIf((1,f.?.? @_a_j_j_l  r:r]rlabelsrrDrEcUbUOURRnURUUUUUS9nUSnUSS2S4n USS2SS24n [R"XR SS9/SS9n UR U 5n Sn UGbURU R5nURRcURS:XaSURl OoURS:aNUR[R:XdUR[R:XaSURl OSURl URRS:XaI[5nURS:Xa&U"U R5UR55n OU"X5n OURRS:Xa=[!5nU"U R#S UR5UR#S 55n O,URRS:Xa[%5nU"X5n U(dU 4US S-nU bU 4U-$U$['U U UR(UR*S 9$) ab labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrrG regressionsingle_label_classificationmulti_label_classificationr?r@)losslogitsrrN)r!rr\r'rSr^rrPr problem_typerrFlongrqr squeezer rRr rrrN)r6r]rrrrDrErrr* patch_tokens linear_inputrrloss_fctrs r8rh$Dinov2ForImageClassification.forwardxs &1%->1->-E!FAN .  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#F3D))-JJ+- B @&++b/R))-II,./Y,F)-)9TGf$ Ev E$!//))   r:)rr\rr)rkrlrmrnrr%rrr'rprr rRrrhrrrsrts@r8rrcs |  04,0)-,0/3&*D u||,D ELL)D & D $D> D 'tn D d^D  u++ ,D D r:rzO Dinov2 backbone, to be used with frameworks like DETR and MaskFormer. c ^\rSrSrU4SjrS\4Sjr\S S\RS\ \ S\ \ S\ \ S\ 4 S jj5r S rU=r$) Dinov2Backboneicv>[TU]U5 [TU] U5 [URS-5Vs/sHo!R PM snUl[U5Ul[U5Ul [R"UR URS9UlUR5 gs snf)Nrr))r$r%_init_backboner>r?r) num_featuresrr;r;rrr r+r,rsrtrBs r8r%Dinov2Backbone.__init__s   v&9>v?W?WZ[?[9\]9\A//9\]*62$V, f&8&8f>S>ST ^sB6r"c.URR$rjrwrs r8rx#Dinov2Backbone.get_input_embeddingsrzr:r]rDrrEcUbUOURRnUbUOURRnUbUOURRnUR U5nUR USX4S9nU(a UR OUSnSn[URU5HupXR;dMURR(aURU 5n URR(ajU SS2SS24n URuppURRnU RXU-X-S5n U R!SSSS 5R#5n X4- nM U(dU(a U4USS-nU$U4US S-nU$[%UU(a UR OSU(aUR&S 9$SS 9$) a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base") >>> model = AutoBackbone.from_pretrained( ... "facebook/dinov2-base", out_features=["stage2", "stage5", "stage8", "stage11"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 16, 16] ```NT)rDrrErrGr?rrr@) feature_mapsrrN)r!rrDrr;rrrzip stage_namesrapply_layernormrsreshape_hidden_statesrIr5rLrMrrrN)r6r]rDrrErrrrstagerrerfr<r=r5rs r8rhDinov2Backbone.forwards@&1%>,#?L;;44#/12#6L4@3E3E0J6!%!7!7J#/#7#7 jDXZ_Zmoq#rL#/#7#71a#C#N#N#PL/ $H#&712;6M'712;6M%3G'//T->w))  EI  r:)r;rrrsr)NNN)rkrlrmrnr%r.rxrr'rprrrrhrrrsrts@r8rrs{ 0&;004,0&* G llG 'tnG $D> G d^ G  G G r:r)rrpr[r)r)rF)@rocollections.abcr}typingrrrrrrr r'torch.utils.checkpointr torch.nnr r r activationsrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrutilsrrrutils.backbone_utilsrconfiguration_dinov2r get_loggerrkrModulerr.rprrrrrrrrrr rr'r;r[rprr__all__rGr:r8rs8DDD AA!rrFQ771.   H %NryyNbBIIR% II% <<% % << % U\\* %  %%>;"));~ryy&$bii$N+ryy+U\\eTV[VbVb* -RYY - &(bii("0"))0h0 BII0 f&DO&D&DRM 'M M ` T #8T  T n Y *MY  Y x er: