fTh SrSSKrSSKJr SSKJrJrJrJrJ r SSK r SSK r SSK J r SSK Jr SSKJrJr SS KJr SS KJr SS KJrJrJrJr SS KJrJr SS KJrJ r J!r! SSK"J#r#J$r$J%r%J&r& SSK'J(r(J)r) \%RT"\+5r,\"SS\#55r-"SS\ R\5r/"SS\ R\5r0"SS\ R\5r1S\00r2"SS\ R\5r3"SS\ R\5r4"SS\ R\5r5"S S!\ R\5r6"S"S#\ R\5r7\$"S$S%\55r8"S&S'\ R\5r9"S(S)\ R\5r:SGS*\ R\S+\ RvS,\ RvS-\ RvS.\\ RvS/\"S4S5\ R\5r?"S6S7\ R\5r@"S8S9\ R\5rA\$"S:S;9"S<S=\855rB"S>S?\ R\5rC\$"S@S;9"SASB\855rD\$"SCS;9"SDSE\8\55rE/SFQrFg)HzPyTorch GIT model.N) dataclass)CallableListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging torch_int) GitConfigGitVisionConfigc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'S rg) GitVisionModelOutput2a Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions)__name__ __module__ __qualname____firstlineno____doc__r#rtorch FloatTensor__annotations__r$r%rr&__static_attributes__r'\/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/git/modeling_git.pyr!r!2sr*15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r1r!c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\ S\R4 S jjr S r U=r$) GitEmbeddingsPz;Construct the embeddings from word and position embeddings.c:>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5Ul[USS5UlUR#S[$R&"UR5R)S5SS9 g) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr:register_bufferr-arangeexpandselfconfig __class__s r2rBGitEmbeddings.__init__Ss !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  r1 input_idsr< inputs_embedspast_key_values_lengthreturncNUbUR5nOUR5SSnUSnUcURSS2XFU-24nUcURU5nOUnURS:XaUR U5nXx- nUR U5nUR U5nU$)Nr>rr;)sizer<rGr:rIrJrN) rTrXr<rYrZ input_shape seq_length embeddingsrIs r2forwardGitEmbeddings.forwardbs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  --i8J&J  ' ': 5"&":":<"H   -J^^J/ \\*- r1)rJrNr:rIrG)NNNr)r(r)r*r+r,rBrr- LongTensorr.intTensorrar0 __classcell__rVs@r2r4r4PsxE  "153759&' E,,-u//0  1 12  !$   r1r4c^\rSrSrSU4SjjrS\R S\R 4SjrSS\R S\\RS\\RS \\ S \\ S \\ S\ \R 4S jjr S rU=r$)GitSelfAttentionc,>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eX0lUc-[RSURRS35 URUl[URUR- 5Ul URUR-Ul [URRURR - S-S -5UlUR$bU=R"UR$-sl[&R("URUR5Ul[&R("URUR5Ul[&R("URUR5Ul[&R0"UR25UlU=(d [7US S 5UlUR8S :XdUR8S :XaHUR:Ul[&R<"SUR:-S - UR5Ulgg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr:r; relative_keyrelative_key_query) rArBrEnum_attention_headshasattr ValueError layer_idxlogger warning_oncerVr(rdattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingr LinearquerykeyvaluerLattention_probs_dropout_probrNrOr:rHrCdistance_embeddingrTrUr:rtrVs r2rBGitSelfAttention.__init__s)     : : :a ?PVXhHiHi#F$6$6#78 445Q8 #      !8!8 9:,,  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr1xr[cUR5SSURUR4-nURU5nUR SSSS5$)Nr>rrnrr )r]rqrwviewpermute)rTr new_x_shapes r2transpose_for_scores%GitSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$r1r%attention_mask head_maskpast_key_valueoutput_attentionspixel_values_presentc URU5nU(a UROSnURURU55n URUR U55n UbUR U SS2SS2US2SS24U SS2SS2US2SS24UR 5up[R"U SS2SS2SU2SS24U /SS9n [R"U SS2SS2SU2SS24U /SS9n URU5n [R"XRSS55nURS:XdURS:XGaU RSU RSnnUbB[R"US- [RURS 9R!SS5nO>[R""U[RURS 9R!SS5n[R""U[RURS 9R!SS5nUU- nUR%UUR&-S- 5nUR)U R*S 9nURS:Xa[R,"S U U5nUU-nOHURS:Xa8[R,"S U U5n[R,"S U U5nUU-U-nU[.R0"UR25- nUbX-n[4R6R9USS9nUR;U5nUbUU-n[R"UU 5nUR=SSSS 5R?5nURA5SSURB4-nUR!U5nU(aUU4OU4nUU4-nU$)Nrrndimr>rorprdtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr )"rr|rrrupdatertr-catmatmul transposer:shapetensorlongrrrQrrHtoreinsummathsqrtrwr functionalsoftmaxrNr contiguousr]rx)rTr%rrrrrmixed_query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_past query_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r2raGitSelfAttention.forwards!JJ}5,@((a--dhh}.EF // =0IJ  %/=/D/D!Q*+[Avw9I-JDNN0 ,N 9Q7F7A-=#>"OUVWI))[Aww1A%BDT$U[\]K//0AB !<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L)!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM]^--r1) rxrwrrNr|rrtrHrqr:rrNNNNNFF)r(r)r*r+rBr-rerrr.r boolrrar0rfrgs@r2riris uD%ell%u||%7;15*.,1/4J||J!!2!23JE--. J ! J $D> J'tnJ u|| JJr1ricz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) GitSelfOutputc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr8) rArBr r~rEdenserJrKrLrMrNrSs r2rBGitSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r1r% input_tensorr[cpURU5nURU5nURX-5nU$NrrNrJrTr%rs r2raGitSelfOutput.forward5 =1  ]3 }'CD r1rJrrN r(r)r*r+rBr-rerar0rfrgs@r2rr6> U\\RWR^R^r1reagerc^\rSrSrS U4SjjrSrSS\RS\\RS\\RS\\ S\\ S \\ S \ \R4S jjr S rU=r$) GitAttentioni c>[TU]5 [UR"XUS9Ul[ U5Ul[5Ulg)N)r:rt) rArBGIT_SELF_ATTENTION_CLASSES_attn_implementationrTroutputset pruned_headsrs r2rBGitAttention.__init__ sB .v/J/JK y  $F+ Er1c6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)lenrrTrqrwrrrrrrrrxunion)rTheadsindexs r2 prune_headsGitAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r1r%rrrrrr[cnURUUUUUU5nURUSU5nU4USS-n U $)Nrr)rTr) rTr%rrrrr self_outputsattention_outputrs r2raGitAttention.forward%sTyy        ;;|A F#% QR(88r1)rrrTrr)r(r)r*r+rBrr-rerr.r rrrar0rfrgs@r2rr s";*7;15*.,1/4||!!2!23E--.  !  $D> 'tn u|| r1rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)GitIntermediatei<c>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rArBr r~rEintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrSs r2rBGitIntermediate.__init__=s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r1r%r[cJURU5nURU5nU$rrrrTr%s r2raGitIntermediate.forwardEs& =1 00? r1rrrgs@r2rr<s(9U\\ellr1rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) GitOutputiLc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rArBr r~rrErrJrKrLrMrNrSs r2rBGitOutput.__init__Ms` YYv779K9KL f&8&8f>S>STzz&"<"<= r1r%rr[cpURU5nURU5nURX-5nU$rrrs r2raGitOutput.forwardSrr1rrrgs@r2rrLrr1rc^\rSrSrS U4SjjrSS\R S\\RS\\RS\\ S\\ S\\ S \ \R 4S jjr S r S rU=r$)GitLayeriZc>[TU]5 URUlSUl[ XS9Ul[ U5Ul[U5Ul g)Nr)rt) rArBchunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rTrUrtrVs r2rBGitLayer.__init__[sI '-'E'E$%fB+F3' r1r%rrrrrr[c URUUUUUUS9nUSnUSSn USn [URURURU5n U 4U -n X4-n U $)N)rrrrrr>)rrfeed_forward_chunkrr) rTr%rrrrrself_attention_outputsrrpresent_key_value layer_outputs r2raGitLayer.forwardcs"&   /)!5 "0" 2!4)2.2260  # #T%A%A4CSCSUe  /G+00r1cJURU5nURX!5nU$r)rr)rTrintermediate_outputrs r2rGitLayer.feed_forward_chunks)"//0@A{{#6I r1)rrrrrrr)r(r)r*r+rBr-rerr.r rrrarr0rfrgs@r2rrZs(7;15*.,1/4 || !!2!23 E--.  !  $D> 'tn  u||  Dr1rc(^\rSrSrU4SjrSS\R S\\RS\\RS\\ \ \ \ \R4S\\ S\\ S \\ S \\ S \\ S \ \ \R \ 44S jjrSrU=r$) GitEncoderic >[TU]5 Xl[R"[ UR 5Vs/sHn[X5PM sn5UlSUl gs snfNF) rArBrUr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rTrUirVs r2rBGitEncoder.__init__sR  ]]vG_G_A`#aA`AHV$7A`#ab &+#$bA&r%rrpast_key_values use_cacheroutput_hidden_statesr return_dictr[c ^UR(a/UR(aU(a[RS5 SnSn U(aP[ U[ 5(d;Sn Uc [ 5nO+[ R"U5n[RS5 U(aSOSn U(aSOSn Sn [UR5HupU(aX4-n UbX>OSnUR(a2UR(a!URURUUUUU5nO U"UUUUUU5nUSnU(aUSn U(dMU US4-n M U(aX4-n U(aU OSnU (aUR5nU (d[S UUU U 455$[UUU U S 9$) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)r'rr>rc30# UH nUcMUv M g7frr'.0vs r2 %GitEncoder.forward..s" A s r$rr%r&)rtrainingrurvrr r from_legacy_cache enumerater_gradient_checkpointing_func__call__to_legacy_cachetupler)rTr%rrrrrrrrreturn_legacy_cacheall_hidden_statesall_self_attentionsnext_decoder_cacher layer_modulelayer_head_mask layer_outputs next_caches r2raGitEncoder.forwards  & &4==##p" $ Z??"& &".."."@"@"Q##^ #7BD$5b4!(4OA#$58H$H!.7.CilO**t}} $ A A ))!"##% ! !-!"##%( ! *!,M%22%6"  &9]1=M 14D D +4'$ #335J "%'    '+&+*   r1)rUrr)NNNNFFFT)r(r)r*r+rBr-rerr.rr rrrrar0rfrgs@r2r r s,7;15SW$(,1/4/4&*Z ||Z !!2!23Z E--. Z "%uU5;L;L5M/N(N"OP Z D> Z $D>Z 'tnZ 'tnZ d^Z  uU\\"$;; <Z Z r1r c.\rSrSr\rSrSrSrSr Sr Sr g)GitPreTrainedModeligitTc[U[5(a[RR UR SUR RS9 [RR URRUR RS9 [RR URRUR RS9 [U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 UR b2URRUR R5 gg[U[R"5(aJURRR5 URRR%S5 gg)zInitialize the weights)meanstd)r9Ng?)rGitVisionEmbeddingsr initnormal_class_embeddingrUinitializer_rangepatch_embeddingweightposition_embeddingr~databiaszero_rCr7rJfill_)rTmodules r2 _init_weights GitPreTrainedModel._init_weightss f1 2 2 GGOOF22$++B_B_O ` GGOOF2299t{{?\?\O ] GGOOF55<<$++B_B_O ` fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r1r'N) r(r)r*r+r config_classbase_model_prefixsupports_gradient_checkpointing_supports_cache_class_supports_quantized_cacherGr0r'r1r2r4r4s$L&*#  $*r1r4c^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S S \RS\R4S jjr S r U=r $) r:i rUc>[TU]5 XlURUlUR UlUR Ul[R"[R"UR55Ul [R"URURUR UR SS9UlUR UR -S-UlURS-Ul[R""UR UR5UlUR'S[R("UR 5R+S5SS9 g)NF) in_channels out_channels kernel_sizestriderCrnrr<r=r?)rArBrUrE embed_dimrzr{r Parameterr-randnr=Conv2d num_channelsr? num_patches num_positionsrCrArPrQrRrSs r2rBGitVisionEmbeddings.__init__s  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr1r`heightwidthr[cURSS- nURRRS5nURSS- n[R R 5(d%XF:Xa X#:XaURUR5$USS2SS24nUSS2SS24nURSn X R-n X0R-n [US-5n URSXU 5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU 5n[R "Xx4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr>g?r rnbicubicF)r]mode align_cornersr)rrAr@ unsqueezer-jit is_tracingr<r{rreshaperr r interpolaterr) rTr`r\r]rYrArZclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r2interpolate_pos_encoding,GitVisionEmbeddings.interpolate_pos_encoding$si!&&q)A- !44;;EEaH*003a7 yy##%%+*F6?**4+<+<= =,QU3,QU3r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr1 pixel_valuesc ^URup4pVU(dJXPR:wdX`R:wa,[SUSUSURSURS3 5eURRR nURUR US95nURS5RSS5nURRUSS5n [R"X/SS 9n U(aXRXU5-n U $XRUR5-n U $) NzInput image size (*z) doesn't match model ().rrnrr>r)rrzrsr?r@rrflattenrr=rRr-rrlrAr<) rTrnrl batch_size_r\r] target_dtype patch_embeds class_embedsr`s r2raGitVisionEmbeddings.forwardMs$'3'9'9$ v'V-F%SbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C ##&C&CJX]&^^J$&=&=d>O>O&PPJr1) r=rUrTrzrYrZr?r{rAF)r(r)r*r+rrBr-rerdrlr.rar0rfrgs@r2r:r: siqq,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr1r:cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) GitVisionMLPi`c>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) rArBrUr r activation_fnr r~rErfc1fc2rSs r2rBGitVisionMLP.__init__asb  #F$5$5699V//1I1IJ99V55v7I7IJr1r%r[clURU5nURU5nURU5nU$r)r~r}rrs r2raGitVisionMLP.forwardhs4/ **=9 / r1)r}rUr~rrrgs@r2r{r{`s)KU\\ellr1r{rFrrrrscalingrNc [R"XRSS55U-nUbX-n[RR US[R S9RUR5n[RRXURS9n[R"X5n U RSS5R5n X4$)Nr>r)rr)pr#rrn) r-rrr rrfloat32rrrNr#r) rFrrrrrrNkwargs attn_weights attn_outputs r2eager_attention_forwardrps<<}}R'<=GL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r1c^\rSrSrSrU4SjrS S\RS\\RS\\RS\\ S\ \R\\R44 S jjr S r U=r $) GitVisionAttentioniz=Multi-headed attention from 'Attention Is All You Need' paperc >[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul URUl SUl [R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R"URUR5Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rqgF)rArBrUrErTrq num_headshead_dimrsscaleattention_dropoutrN is_causalr r~k_projv_projq_projout_projrSs r2rBGitVisionAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r1r%rcausal_attention_maskrr[c URupVnURU5nURU5n URU5n UR XVUR UR 5RSS5nU R XVUR UR 5RSS5n U R XVUR UR 5RSS5n URRS:waUbUbX#-nOUbUnO USLUl [n URRS:waTURRS:XaU(a[RS5 O[URRn U "UUU U UURURUR (dSO UR"S 9upU R%XVU5R'5n UR)U 5n U(dSn X4$) z#Input shape: Batch x Time x Channelrrnflash_attention_2Nrsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r7)rrrN)rrrrrrrrrUrrrrurvrrr#rNrerr)rTr%rrrrsr_rTquerieskeysvaluesattention_interfacerrs r2raGitVisionAttention.forwards-:,?,?) ++m,{{=)]+,,zt~~t}}U__`acdeyyOYYZ[]^_ZT^^T]]S]]^_abc ;; + +/B B).C.O!/!G&2!62$>DN(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! "))*)LWWY mmK0  L((r1) rUrNrTrrrrrrrr)NNF)r(r)r*r+r,rBr-rerrrrar0rfrgs@r2rrsGB.268<,1 5)||5)!.5) ( 5 5) $D> 5) u||Xell33 4 5)5)r1rc ^\rSrSrS\4U4SjjrS S\RS\RS\RS\\ S\ \R4 S jjr S r U=r$) GitVisionEncoderLayerirUc<>[TU]5 URUl[ U5Ul[ R"URURS9Ul [U5Ul [ R"URURS9Ul gr) rArBrErTr self_attnr rJrK layer_norm1r{mlp layer_norm2rSs r2rBGitVisionEncoderLayer.__init__sm +++F3<<F & u  ! &&r1rc^\rSrSrSrS\4U4SjjrS S\\RS\\RS\\ S\\ S \\ S \ \ \ 44 S jjrS rU=r$)GitVisionEncoderi z Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rUc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr) rArBrUr rrrrlayersr)rTrUrtrVs r2rBGitVisionEncoder.__init__sT  mmERXRjRjLk$lLkq%:6%BLk$lm &+#%mrrrrrrr[cLUbUOURRnUbUOURRnUbUOURRnU(aSOSnU(aSOSnUn [ UR 5HrupU(aXy4-nUR (a1UR(a URU RU UUU5n O U "U UUUS9n U Sn U(dMjXS4-nMt U(aXy4-nU(d[SXU455$[XUS9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr')rrrc3.# UH ocMUv M g7frr'rs r2r +GitVisionEncoder.forward..dse$Sq$Ss r$r%r&) rUrruse_return_dictr%rrr#r&r'r)r) rTrYrrrrrencoder_statesall_attentionsr%idx encoder_layerr0s r2raGitVisionEncoder.forwards8L2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]N$See e+Vd  r1)rUrr)NNNNN)r(r)r*r+r,rrBrr-rerrrrrar0rfrgs@r2rr s,,268<,0/3&*O !.O  ( 5 O $D> O 'tn O d^O  uo% &O O r1rc^\rSrSrS\4U4Sjjr\S S\\RS\\ S\\ S\\ S\\ S \ \ \ 44 S jj5rS rU=r$) GitVisionTransformerijrUc>[TU]5 XlURn[ U5Ul[ R"X!RS9Ul [U5Ul [ R"X!RS9Ul gr) rArBrUrEr:r`r rJrK pre_layrnormrencoderpost_layernorm)rTrUrTrVs r2rBGitVisionTransformer.__init__lsd  && -f5LL8M8MN'/  ll9:O:OPr1rnrrrlrr[cUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR XS9nUR U5nURUUUUS9nUSnURU5nU(d U4USS-$[UURURS9$)Nz You have to specify pixel_valuesrl)rYrrrrrr) rUrrrrsr`rrrrr%r&) rTrnrrrlrr%encoder_outputsr$s r2raGitVisionTransformer.forwardvs2C1N-TXT_T_TqTq$8$D $++JjJj &1%& 'tn & #+4. & d^ &  uo% && & r1rzY The vision model from CLIP, used in GIT, without any head or projection on top. ) custom_introc^\rSrSr\rSrS\4U4SjjrS\R4Sjr \ S S\ \ RS\ \S\ \S \S \ \S\\\44 S jj5rS rU=r$)GitVisionModelirnrUcd>[TU]U5 [U5UlUR 5 gr)rArBr vision_model post_initrSs r2rBGitVisionModel.__init__s'  08 r1r[cBURRR$r)rr`r?rTs r2get_input_embeddings#GitVisionModel.get_input_embeddingss  ++;;;r1rrrlrc^UbUOURRnURUUUUUS9$)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```)rnrrrlr)rUrr)rTrnrrrlrs r2raGitVisionModel.forwardsA8&1%# 'tn # #' # d^ #  uo% &# # r1rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $) GitProjectionirUc.>[TU]5 Xl[R"[R "UR RUR5[R"URUR RS95Ul gr) rArBrUr Sequentialr~ryrErJrKvisual_projectionrSs r2rBGitProjection.__init__sd  !# IIf**668J8J K LL++1E1E1T1T U" r1r`r[c$URU5$r)r)rTr`s r2raGitProjection.forwards%%j11r1)rUr) r(r)r*r+rrBr-rerar0rfrgs@r2rrs/ y 2%,,25<<22r1rzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states c^\rSrSrU4SjrSrSrSrS\S\ RS\ RS \ R4S jr SS jr\SS \\ RS \\ RS\\ RS\\ RS\\ RS\\ RS\\\\\ R(4S\\S\\S\\S\S\\S \\\ R\44Sjj5rSrU=r$)GitModelicr>^[TU]T5 TUl[T5Ul[ TR 5Ul[T5Ul [T5Ul TRb8[R"U4Sj[TR555UlUR#5 g)Nc3># UHEn[R"[R"SSTRR 55v MG g7f)rN)r rUr-zerosryrE)rrtrUs r2r $GitModel.__init__..s=;?A U[[Av/C/C/O/OPQQ?sA A)rArBrUr4r`rry image_encoderr rrrr}r ParameterListrimg_temperal_embeddingrrSs `r2rBGitModel.__init__s   '/+F,@,@A!&) !.v!6  * * 6*,*:*:;v>>?;+D ' r1c.URR$rr`rGrs r2rGitModel.get_input_embeddingss...r1c$XRlgrr)rTrs r2set_input_embeddingsGitModel.set_input_embeddingss */'r1cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rTheads_to_prunerrs r2 _prune_headsGitModel._prune_headss< +002LE LL  u % / / ; ;E B3r1r]rrr[c [R"[R"XX2S9SS9nURUS:H[ S55nU$)Nrrr)diagonal-inf)r-triuones masked_fillfloat)rTr]rrmasks r2_generate_future_maskGitModel._generate_future_masks=zz%**TLWXY 5=9 r1cnURSnURSnURnURn [R"Xw4XS9n [R "XvU-4[ S5URU S9n [R"Xg4U URS9n US:a?[R"URSURSU-4U URS9n[R"X4SS9n [R"XRU 54SS9n[R"X4SS9SSS24nUc2[R "URSURS4SUS9nUR[R:wa [S 5e[R"XQRS 9n[ S5UU'URURSXv-Xt-U-45nUR5nUSS2SS2SU24nUSS2SSS24nUU-USS2SS2SU24'USS2SSS2SS24nU$) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr-rfullrrrrrs zeros_likerRclone)rTtgtmemorytgt_maskrZmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrs r2create_attention_maskGitModel.create_attention_masks0))A,\\!_  ;; 7TJJ #99 : &M::  kk  !?? "A %{{"HNN1$58N$NOH yy(0a8 9kk%&89qA#ii 1=dAgF " *&+jj&,,q/6<>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer>z5You have to specify either input_ids or inputs_embedsrrrnrrz#pixel_values must be of rank 4 or 5)rXr<rYrZr)rrrrZ)tgt_len)rrrrrrrrr")%rUrrrrrs%warn_if_padding_and_no_attention_maskr]rr rget_seq_length get_head_maskrndimrr$rrappendr-rrr`rrrrepeatrrrrrrrr%r&)rTrXrr<rnrrYrrrrrlrr^r_rZprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr%rcombined_attention_maskexpanded_attn_maskrsequence_outputs r2raGitModel.forwardFs8J2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y Q#..*K  &',,.s3KTU U ^ "#  &"/599 "1%++A.$335 #&&y++2O2OP $(!  #  A%"&"4"4 #5### ""a'"$!&|'9'9!',>$Q 1a%78Sk-?-''**T-H-H-SS)#**+@A ">#())O"C!!FGG(,(>(>(O %??%'#9 +  % ,(- !''*A/?/E/Ea/HI&,,'..) %%>$D$D  ! !! $(A(F(Fq(I I1a% !  #<>N"OUVW --j:J:P:PRbRiRij#'"<"< ,#9 #=#   %"< 0 6 6 B"b!(() &)%71?U>U>VXY8Y%Z"'1{1~o.?+a.AR(RSWiiS,, 2+/!5#!-T!9'  *!,#%(;; ;&-+;;)77&11   r1)rUr`rrrrr) NNNNNNNNNNFN)r(r)r*r+rBrrrrdr-rrrerrrrrr rr.rrrrar0rfrgs@r2rrs &/0C#ekk5<<\a\h\h 0#d-115/3/3,004KO$(,0/3).&*c ELL)c !.c u||, c u||, c ELL) c   -c "%tE4E4E/F(F"GHc D>c $D>c 'tnc #'c d^c  uU\\"$>> ?c c r1rz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c ^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ \ \\ R4S\ \S\ \S\ \S\S\ \S\ \\ R\44Sjj5rSSjrSrSrU=r$)GitForCausalLMiz output.weightc>[TU]U5 [U5Ul[R "UR UR5UlUR5 gr) rArBrr5r r~rErDrrrSs r2rBGitForCausalLM.__init__sF  F#ii 2 2F4E4EF  r1cUR$rrrs r2get_output_embeddings$GitForCausalLM.get_output_embeddingss {{r1cXlgrr+)rTnew_embeddingss r2set_output_embeddings$GitForCausalLM.set_output_embeddingss$ r1rXrr<rnrrYlabelsrrrrrlrr[c U bU OURRn UbSn URUUUUUUUU U U U U S9 nUSnURU5nSnUbURRR SR RRnUSS2US2SS24R5nUSS2SS24R5nUR"URSURR5URS54SURR0UD6nU (dU4USS-nUbU4U-$U$[UUURURUR S9$) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr<rnrrYrrrrrlrrr>rrD)losslogitsrr%r&)rUrr5rrrrrTr|r loss_functionrrDrrr%r&)rTrXrr<rnrrYr2rrrrrlrrrr$r5r4num_image_tokensshifted_logitsrs r2raGitForCausalLM.forwardsj&1%IIKNAqrE]--/F%%##B (>(>? B ;;11 DY,F)-)9TGf$ Ev E%#33!//))   r1c UbCUR5nURSU:aUnOURSS- nUSS2US24nURnUcURU5nUUURSS5UUS.$)Nrrn)rXrrnrr)rrnew_onesget) rTrXrrrr past_lengthremove_prefix_lengthr^s r2prepare_inputs_for_generation,GitForCausalLM.prepare_inputs_for_generations  &)88:Kq!K/'2$(1q'9A'=$!!%9%:":;I oo  !&// # UH/oRSTRUR55v M1 g7f)rN) index_selectrr)r past_statebeam_idxs r2r 0GitForCausalLM._reorder_cache..s1ncmU_--aZ=N=N1OPPcms7:)r))rTrrEreordered_past layer_pasts ` r2_reorder_cacheGitForCausalLM._reorder_caches8)J ncmnn N*r1)r5r) NNNNNNNNNNNFN)NNN)r(r)r*r+_tied_weights_keysrBr,r0rrr-rerr rrrrrar?rIr0rfrgs@r2r'r's **%-115/3/3,004)-FJ$(,0/3).&*A ELL)A !.A u||, A u||, A ELL) A   -A &A "%tELL/A(A"BCA D>A $D>A 'tnA #'A d^A uU\\"$:: ;!A A HOS >r1r')r'rr4r)r7)Gr,r dataclassesrtypingrrrrrr-torch.utils.checkpointr activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrconfiguration_gitrr get_loggerr(rur!rr4rirrrrrrr r4r:r{rerrrrrrrrrr'__all__r'r1r2rZs  !99 !.)B Gll :   H % ?;? ?8-BII-`rryyrl BII   /299/f bii   .ryy.ba a H***:P"))Pf 299 .% II% <<% % << % U\\* %  %%,L)L)`/BII/f^ ryy^ B3 2993 l 2 '2  2 j 2BII 2 ~ !~  ~ B {'{  {| Qr1