fTh^SSKJrJrJrJr SSKrSSKJr SSKJ r SSK J r SSK J r Jr SSKJrJr SSKJr SS KJrJr S S KJrJrJrJrJrJrJr S S KJ r S S K!J"r"J#r# \RH"\%5r&"SS\ 5r'"SS\5r("SS\"5r)"SS\5r*"SS\5r+"SS\5r,"SS\5r-"SS\5r.\"SS\55r/"S S!\5r0/S"Qr1g)#)CallableOptionalTupleUnionN)PretrainedConfig)FlashAttentionKwargs)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModelCLIPVisionTransformer)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visioncR^\rSrSrSrSrSrSU4SjjrSrU=r $)MLCDVisionConfig*af This is the configuration class to store the configuration of a [`MLCDVisionModel`]. It is used to instantiate a MLCD vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the vision encoder of the MLCD [DeepGlint-AI/mlcd-vit-bigG-patch14-336](https://huggingface.co/DeepGlint-AI/mlcd-vit-bigG-patch14-336) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 1664): Dimensionality of the encoder layers and the pooler layer. intermediate_size (`int`, *optional*, defaults to 8192): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. projection_dim (`int`, *optional*, defaults to 1024): Dimensionality of text and vision projection layers. num_hidden_layers (`int`, *optional*, defaults to 48): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. num_channels (`int`, *optional*, defaults to 3): The number of input channels. image_size (`int`, *optional*, defaults to 336): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 14): The size (resolution) of each patch. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"quick_gelu"` are supported. layer_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the layer normalization layers. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (`float`, *optional*, defaults to 1.0): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configc >[TU]"S0UD6 XlX lX0lX@lXPlX`lXlXpl Xl Xl Xl Xl Xlg)N)super__init__ hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_groups num_channels patch_size image_sizeinitializer_rangeinitializer_factorattention_dropoutlayer_norm_eps hidden_act)selfr%r&r'r(r)r*r,r+r1r0r/r-r.kwargs __class__s ]/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/mlcd/modular_mlcd.pyr$MLCDVisionConfig.__init__ds`" "6"&!2!2#6 $8!($$!2"4!2,$) r/r1r%r,r.r-r&r0r(r*r'r)r+) ii 0riPgelugh㈵>g{Gz??) __name__ __module__ __qualname____firstlineno____doc__ model_typebase_config_keyr$__static_attributes__ __classcell__r4s@r5rr*sH4 l%J%O%%r7rc\rSrSrSrg)MLCDMLPr"N)r?r@rArBrFr"r7r5rJrJsr7rJc>\rSrSrS\S\S\R 4SjrSrg)MLCDRotaryEmbeddingnum_patches_heightnum_patches_widthreturnc~[R"XRRS9R S5R SU5n[R"X RRS9R S5R US5n[R "UR5UR5/SS9n[X5n[R"X`RRURRS9n[R"XpR5nXRS5n U $)aE Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicer:rdim)rSdtype) torcharangeinv_freqrS unsqueezeexpandstackflattenmaxrWouter) r2rOrPhpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embs r5forwardMLCDRotaryEmbedding.forwards LL+MM4H4H I S STU V ] ]^`bs t  LL*==3G3G H R RST U \ \]oqs t  ++x//183C3C3EFBO.B ll=1E1ET]]M`M`a#kk#}}=-5==a@r7r"N) r?r@rArBintrXTensorrhrFr"r7r5rMrMs ##%,,r7rMcj^\rSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $)MLCDVisionEmbeddingsconfigc(>[TU]U5 U?gN)r#r$position_embeddingr2ror4s r5r$MLCDVisionEmbeddings.__init__s   #r7 pixel_valuesrQcHURSnURRRnURUR US95nUR S5R SS5nURRUSS5n[R"XT/SS9nU$)Nr)rWrr:rTrU) shapepatch_embeddingweightrWtor^ transposeclass_embeddingr\rXcat)r2ru batch_size target_dtype patch_embeds class_embeds embeddingss r5rhMLCDVisionEmbeddings.forwards!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C r7r") r?r@rArBrr$rX FloatTensorrkrhrFrGrHs@r5rmrms2$/$ E$5$5 %,,  r7rmc^\rSrSrSrS\4U4SjjrS S\RS\ \R\R4S\ \RS\ \ S \ \R\ \R44 S jjr S rU=r$) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://arxiv.org/abs/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://arxiv.org/abs/2104.09864 rocT>[TU]U5 URUlSUlgNF)r#r$r) is_causalrss r5r$MLCDAttention.__init__s%  $*$?$?!r7 hidden_statesposition_embeddingsattention_maskr3rQc @URSSupVURU5RXVURUR45nUR U5RXVURUR45nUR U5RXVURUR45n USRS5R5n USRS5R5n [XxX5upxURSSSS5R5nURSSSS5R5nU RSSSS5R5n [n URRS:wadURRS:Xa-URSS 5(a[ R#S 5 O[$URRn U "UUUU U4UR&(dS O UR(UR*UR,S .UD6upU RSSSS5R5n U R/XeS5n UR1U 5n U RSSS5R5n X4$) NrTrr:rreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r=)dropoutscalingr)rwq_projreshape num_headshead_dimk_projv_projr[floatrpermute contiguousrro_attn_implementationgetlogger warning_oncer trainingrscalerviewout_proj)r2rrrr3r~ seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss r5rhMLCDAttention.forwardsj"/!4!4Sb!9 {{=199:SWSaSacgcpcp:qr [[/77QUQ_Q_aeanan8op {{=199:SWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|Y\#b $++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (? ;; + +w 6{{//69fjjI\^c>d>d##L '>dkk>^>^&_#$7      % $}}C$,,JJnn %  % ! "))!Q15@@B !&&zrB mmK0 !))!Q2==? ((r7)rr)rq)r?r@rArBrCrr$rXrkrrrr rhrFrGrHs@r5rrs/26 2)||2)#5<<#=>2)!. 2) -. 2) u||Xell33 4 2)2)r7rc ^\rSrSrS\4U4SjjrS S\RS\\R\R4S\ \RS\ \ S\\R4 S jjr S r U=r$) MLCDEncoderLayerrocD>[TU]U5 [U5Ulgrq)r#r$r self_attnrss r5r$MLCDEncoderLayer.__init__s  &v.r7rrrrrQcUnURU5nURUUUUS9upXQ-nUnURU5nURU5nXQ-nU4nU(aXv4- nU$)a; Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`Tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*, defaults to `False`): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. rrrr) layer_norm1r layer_norm2mlp)r2rrrrresidualroutputss r5rhMLCDEncoderLayer.forwards*!((7 &*nn' 3)/ '5' # !0  ((7 /  0  "   &Gr7)rr)r?r@rArBrr$rXrkrrboolrrhrFrGrHs@r5rrs///26,1 *||*#5<<#=>*!. * $D> * u  ! **r7rc^\rSrSrSrS\4U4SjjrSS\RS\ \R\R4S\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$) MLCDEncoderi2z Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig roc$>[TU]U5 g)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)r#r$rss r5r$MLCDEncoder.__init__;s  r7 inputs_embedsrrroutput_hidden_states return_dictrQcNUbUOURRnUbUOURRnUbUOURRnU(aSOSnU(aSOSnUn [ UR 5HrupU(aXy4-nUR (a1UR(a URU RU UUU5n O U "U UUUS9n U Sn U(dMjXS4-nMt U(aXy4-nU(d[SXU455$[U UUS9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`Tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr"rrr:c3.# UH ocMUv M g7frqr").0vs r5 &MLCDEncoder.forward..se$Sq$Ss )last_hidden_stater attentions) roruse_return_dictr enumeratelayersgradient_checkpointingr_gradient_checkpointing_func__call__tupler ) r2rrrrrrencoder_statesall_attentionsridx encoder_layer layer_outputss r5rhMLCDEncoder.forward?s:D%9$D $++JjJj &1%>Ne]N$See e+(%  r7r"NNNN)r?r@rArBrCrr$rXrrrkrrrr rhrFrGrHs@r5rr2s!/!26,0/3&*L ((L #5<<#=>L !. L $D> L 'tn L d^L  uo% &L L r7rc^\rSrSrS\4U4Sjjr\S S\\RS\\ S\\ S\\ S\ \ \ 44 S jj5rS rU=r$) MLCDVisionTransformeriroc >[TU]U5 [URUR-S-5Ul[ R"[R"SURUR-S-55Ul g)Nrr:) r#r$rMr%r(vision_rotary_embeddingnn ParameterrXrandn class_pos_embrss r5r$MLCDVisionTransformer.__init__sh  ':6;M;MQWQkQk;kop;p'q$\\%++a9K9KvOiOi9imn9n*opr7rurrrrQcUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR SURR -nUR SURR -nURXV5nURURR5n[R"URU/SS9n[R"Xw4SS9nUR5UR54n URU5n UR!U 5n UR#U U UUUS9n U Sn U SS2SSS24n UR%U 5n U(d X4U SS-$['U U U R(U R*S9$) Nz You have to specify pixel_valuesrTrrU)rrrrrr:)r pooler_outputrr)rorrr ValueErrorrwr+rrzrrSrXr}rrr pre_layrnormencoderpost_layernormr rr)r2rurrrrOrPrgembrrencoder_outputsr pooled_outputs r5rhMLCDVisionTransformer.forwards%9$D $++JjJj &1%/ 'tn / d^ / u00 1 / / r7rc.\rSrSr\rSrSrSrSr Sr Sr g)MLCDPreTrainedModelimlcdTcTURRn[U[5(aURRn[R R URSURS-U-S9 [R R URRURRU-S9 g[U[5(Ga URRnURS-SURR-S--U-nURS-U-n[R R URRUS9 [R R URRUS9 [R R UR RUS9 [R R UR"RUS9 g[U[$5(aURRnURR&S-SURR-S--U-nSURR&-S-U-n[R R UR(RUS9 [R R UR*RUS9 g[U[,5(avURRnURR&URR.-S-S-U-n[R R UR0SUS9 g[U[R25(aJUR4R6R95 URR6R;S5 g[U[R<5(a3UR4b%UR4R6R95 ggg)zInitialize the weightsr=g)meanstd)rrr>N)ror. isinstancermrinitnormal_r| embed_dimrxryr-rr'rrrrrJr%fc1fc2rr(r LayerNormbiasdatazero_fill_Linear)r2modulefactor in_proj_std out_proj_stdfc_std pos_emb_stds r5 _init_weights!MLCDPreTrainedModel._init_weightss// f2 3 3[[33F GGOOF22&BRBRTXBX[aBaO b GGOOF2299v}}?^?^ag?gO h  . .[[33F!++T1q6==;Z;Z7Z_c6cdgmmK",,d2f>> import requests >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```)rurrr)rorrr vision_model)r2rurrrs r5rhMLCDVisionModel.forwardsu>%9$D $++JjJj &1%( 'tn ( d^ ( u00 1 ( ( r7r)rrr)2typingrrrrrXtorch.nnrconfiguration_utilsrmodeling_flash_attention_utilsr modeling_outputsr r modeling_utilsr r processing_utilsrutilsrrclip.modeling_cliprrrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr?rrrJrMrmrrrrrr__all__r"r7r5r!s43 3BKF&,;[   H %Y%'Y%x g /D/$?)M?)D/'/dY +Y x6 16 r$%/$%$%N* o* Z r7