o Zh@sddlZddlmZddlmZmZmZmZmZddl Z ddl m m Z ddlZ ddl m Z ddlmZmZmZddlmZddlmZdd lmZdd lmZmZmZmZmZdd lm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&ddl'm(Z(m)Z)e#rddl*m+Z+ddl,m-Z-ddl.m/Z/ne0Z-e$1e2Z3GdddeZ4  dWde j5de j5dee j5dee j5dee j5e j5e j5e6ee j5ee j5ff ddZ7de j5de j5de6de6de j5f dd Z8Gd!d"d"e j9j:Z;  dWd#ee j5d$ee6fd%d&ZZ?Gd+d,d,e j>Z@Gd-d.d.e(ZA /dXd0d1d2e j5de j5d3e j5dee jBd4ee6e6fd5e6d6e6d7eeCdeee j5e j5fee j5ffd8d9ZDe jEfd0d1d2e j5d:e=d#e j5d$e6d4ee6e6fd5e6d6e6d;e jFdee j5fdd?ZHeGeDeHd@ZIGdAd1d1e j>ZJGdBdCdCe j>ZKe"GdDdEdEe ZLe"GdFdGdGeLZMGdHdIdIe j>ZNe"dJdKGdLdMdMeLZOe"dNdKGdOdPdPeLZPe"dQdKGdRdSdSeLZQe"GdTdUdUeLZRgdVZSdS)YN) nullcontext)DictLiteralOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)PretrainedConfig)_prepare_4d_attention_mask)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringis_flash_attn_2_availablelogging)is_triton_available)GemmaRotaryEmbeddingapply_rotary_pos_emb) flash_attn_varlen_qkvpacked_func)RotaryEmbedding) apply_rotarycseZdZdZdZdgZ                             d!dedffdd Zfdd ZZ S)"ModernBertConfiga This is the configuration class to store the configuration of a [`ModernBertModel`]. It is used to instantiate an ModernBert model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the ModernBERT-base. e.g. [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the ModernBert model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`ModernBertModel`] hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 1152): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 22): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu"` if not specified. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_cutoff_factor (`float`, *optional*, defaults to 2.0): The cutoff factor for the truncated_normal_initializer for initializing all weight matrices. norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. norm_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the normalization layers. pad_token_id (`int`, *optional*, defaults to 50283): Padding token id. eos_token_id (`int`, *optional*, defaults to 50282): End of stream token id. bos_token_id (`int`, *optional*, defaults to 50281): Beginning of stream token id. cls_token_id (`int`, *optional*, defaults to 50281): Classification token id. sep_token_id (`int`, *optional*, defaults to 50282): Separation token id. global_rope_theta (`float`, *optional*, defaults to 160000.0): The base period of the global RoPE embeddings. attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. global_attn_every_n_layers (`int`, *optional*, defaults to 3): The number of layers between global attention layers. local_attention (`int`, *optional*, defaults to 128): The window size for local attention. local_rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the local RoPE embeddings. embedding_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the embeddings. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the MLP layers. mlp_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the MLP layers. decoder_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in the decoder layers. classifier_pooling (`str`, *optional*, defaults to `"cls"`): The pooling method for the classifier. Should be either `"cls"` or `"mean"`. In local attention layers, the CLS token doesn't attend to all tokens on long sequences. classifier_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the classifier. classifier_bias (`bool`, *optional*, defaults to `False`): Whether to use bias in the classifier. classifier_activation (`str`, *optional*, defaults to `"gelu"`): The activation function for the classifier. deterministic_flash_attn (`bool`, *optional*, defaults to `False`): Whether to use deterministic flash attention. If `False`, inference will be faster but not deterministic. sparse_prediction (`bool`, *optional*, defaults to `False`): Whether to use sparse prediction for the masked language model instead of returning the full dense logits. sparse_pred_ignore_index (`int`, *optional*, defaults to -100): The index to ignore for the sparse prediction. reference_compile (`bool`, *optional*): Whether to compile the layers of the model which were compiled during pretraining. If `None`, then parts of the model will be compiled if 1) `triton` is installed, 2) the model is not on MPS, 3) the model is not shared between devices, and 4) the model is not resized after initialization. If `True`, then the model may be faster in some scenarios. repad_logits_with_grad (`bool`, *optional*, defaults to `False`): When True, ModernBertForMaskedLM keeps track of the logits' gradient when repadding for output. This only applies when using Flash Attention 2 with passed labels. Otherwise output logits always have a gradient. Examples: ```python >>> from transformers import ModernBertModel, ModernBertConfig >>> # Initializing a ModernBert style configuration >>> configuration = ModernBertConfig() >>> # Initializing a model from the modernbert-base style configuration >>> model = ModernBertModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```Z modernbertZpast_key_values gelu {Gz?@h㈵>FkjiAr @TclsNclassifier_poolingr2meanc$% stjd| || ||d|$||_||_||_||_||_||_||_| |_ | |_ | |_ ||_ ||_ ||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_|!|_|"|_|#|_|jdvrwt d|jddS)N) pad_token_id bos_token_id eos_token_id cls_token_id sep_token_idr5zQInvalid value for `classifier_pooling`, should be either "cls" or "mean", but is .)!super__init__ vocab_sizemax_position_embeddings hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsinitializer_rangeinitializer_cutoff_factornorm_eps norm_biasglobal_rope_thetaattention_biasattention_dropouthidden_activationglobal_attn_every_n_layerslocal_attentionlocal_rope_thetaembedding_dropoutmlp_bias mlp_dropout decoder_biasr4classifier_dropoutclassifier_biasclassifier_activationdeterministic_flash_attnsparse_predictionsparse_pred_ignore_indexreference_compilerepad_logits_with_grad ValueError)%selfr@rBrCrDrErMrArFrGrHrIr7r9r8r:r;rJrKrLrNrOrPrQrRrSrTr4rUrVrWrXrYrZr[r\kwargs __class__r=`/var/www/auris/lib/python3.10/site-packages/transformers/models/modernbert/modular_modernbert.pyr?sX'  zModernBertConfig.__init__cst}|dd|S)Nr[)r>to_dictpopr^outputr`r=rbrcs  zModernBertConfig.to_dict)#r!r"r#r$r%r&r'r(r)r*Fr+r,r-r-r,r.Fr/r r0r1r/Fr/Tr2r/Fr&FFr3NF) __name__ __module__ __qualname____doc__Z model_typeZkeys_to_ignore_at_inferencerr?rc __classcell__r=r=r`rbr 5sVgSr inputsattention_mask position_idslabelsreturncCs|jdtjd}tj|dd}t|}tjj tj |dtjdd}| dkr7||}n|j ^} } } | | } |j| g| R|}|durV||nd} |durb||nd}||||| |fS) a Remove padding from input sequences. Args: inputs: (batch, seqlen, ...) or (batch, seqlen) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. position_ids: (batch, seqlen), int, position ids labels: (batch, seqlen), int, labels Returns: unpadded_inputs: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) cu_seqlens: (batch + 1), the cumulative sequence lengths max_seqlen_in_batch: int unpadded_position_ids: (total_nnz) or None unpadded_labels: (total_nnz) or None dimdtypeF)as_tupler)rrN)sumtorchZint32Znonzeroflattenintmaxitemr functionalpadZcumsumrsshapeview)rlrmrnroZseqlens_in_batchindicesZmax_seqlen_in_batch cu_seqlensZunpadded_inputsbatchseqlenrestrZunpadded_position_idsZunpadded_labelsr=r=rb_unpad_modernbert_inputs  rrrrcCs|dkrtj|||j|jd}|||<|||}|S|j^}}tj||g|R|j|jd}|||<|j||g|R}|S)aQ Add padding to sequences. Args: inputs: (total_nnz, ...) or (total_nnz,), where total_nnz = number of tokens selected in attention_mask. indices: (total_nnz) batch: int, batch size seqlen: int, max sequence length Returns: padded_inputs: (batch, seqlen, ...) or (batch, seqlen) rv)rtdevice)rsrxZzerosrtrrr)rlrrrrfZ padded_inputs_rr=r=rb_pad_modernbert_output"s   "rc@s>eZdZe  ddeejdeefddZeddZ dS) ApplyRotaryEmbUnpadNr max_seqlenc Csd|}|j\}}}} |ddddf|d| } t| ||d||ddd||||||_|S)NrrqrFT)seqlen_offsetsrr interleavedinplace) contiguousrrrZsave_for_backwardr) ctxqkvcossinrr total_nnz_three_nheadsheaddimZqkr=r=rbforwardBs   zApplyRotaryEmbUnpad.forwardc Csn|j\}}}|}|j\}}}}|ddddf|d|} t| ||d||jdddd |ddddddfS)NrrqrFT)rrrrr conjugate)Z saved_tensorsrrrrr) rdorrrrrrrZdqkr=r=rbbackwardas  zApplyRotaryEmbUnpad.backwardNN) rgrhri staticmethodrrxTensorrzrrr=r=r=rbrAs rrrcCst|||||S)a Arguments: qkv: (total_nnz, 3, nheads, headdim) - input tensor for packed QKV. cos, sin: (seqlen_rotary, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). inplace: if True, apply rotary embedding in-place. seqlen_offsets: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. cu_seqlens: (batch + 1,) or None max_seqlen: int Return: out: (total_nnz, dim) rotary_dim must be <= headdim Apply rotary embedding to the first rotary_dim of x. )rapply)rrrrrr=r=rbapply_rotary_unpaddedxsrc seZdZdZ    ddededeedeejdeej f fd d Z dd ej d ej deed e ej e ej ej fffddZd efddZZS)!ModernBertUnpaddedRotaryEmbeddingzP The rotary position embeddings applied directly to unpadded sequences. r1NrsbaserrrtcsTtj||d|dd||_|dur$|dur&|dur(|j|||ddSdSdSdS)a max_seqlen: if max_seqlen, device, and dtype are provided, we precompute the cos_sin_cache up to max_seqlen. If the max_seqlen, device, or dtype during training/inference differ, the cos_sin_cache will be recomputed during the forward pass. TF)rsrZpos_idx_in_fp32rrNrrt)r>r?r_update_cos_sin_cache)r^rsrrrrtr`r=rbr?s   z*ModernBertUnpaddedRotaryEmbedding.__init__rrrpcCs6|dur|j||j|jdt||j|j||d}|S)z Apply rotary embedding *inplace* to qkv. qkv: (total_nnz, 3, nheads, headdim) cu_seqlens: (batch + 1,) cumulative sequence lengths max_seqlen: int max seq length in the batch Nrrr)rrrtrZ _cos_cachedZ _sin_cached)r^rrrr=r=rbrs z)ModernBertUnpaddedRotaryEmbedding.forwardcCsd|jd|jd|jS)Nzdim=z, base=z , scale_base=)rsrZ scale_baser^r=r=rb extra_reprsz,ModernBertUnpaddedRotaryEmbedding.extra_repr)r1NNNN)rgrhrirjrzfloatrrxrrtr?rrrrstrrrkr=r=r`rbrs8 rcspeZdZdZdeffdd Zejdddejdej fd d Z dde ejd e ej dej fd dZ Z S)ModernBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. configcsRt||_tj|j|j|jd|_tj |j|j |j d|_ t |j|_dS)N)Z padding_idxZepsbias)r>r?rrZ Embeddingr@rBr7tok_embeddings LayerNormrHrInormDropoutrQdropr^rr`r=rbr?s zModernBertEmbeddings.__init__TZdynamic input_idsrpcC||||Sr)rrr)r^rr=r=rbcompiled_embeddingssz(ModernBertEmbeddings.compiled_embeddingsN inputs_embedscCsH|dur|||}|S|jjr||n ||||}|Sr)rrrr[rr)r^rr hidden_statesr=r=rbrs zModernBertEmbeddings.forwardr)rgrhrirjr r?rxcompile LongTensorrrrrrkr=r=r`rbrs rcs<eZdZdZdeffdd ZdejdejfddZZ S) ModernBertMLPa6Applies the GLU at the end of each ModernBERT layer. Compared to the default BERT architecture, this block replaces :class:`~transformers.model.bert.modeling_bert.BertIntermediate` and :class:`~transformers.model.bert.modeling_bert.SelfOutput` with a single module that has similar functionality. rcsft||_tj|jt|jd|jd|_ t |j |_ t |j|_tj|j|j|jd|_dS)Nrr)r>r?rrLinearrBrzrCrRWir rMactrrSrWorr`r=rbr?s   zModernBertMLP.__init__rrpcCs2||jddd\}}|||||S)Nrrqrs)rchunkrrr)r^rinputZgater=r=rbrszModernBertMLP.forward) rgrhrirjr r?rxrrrkr=r=r`rbrsrc s6eZdZddedededeejffdd Z Z S) ModernBertRotaryEmbeddingNrrsrrcs.tj|||d|jd|||d\}|_dS)N)rr)rsr)r>r?Z rope_init_fnZattention_scaling)r^rrsrrZinv_freqr`r=rbr?sz"ModernBertRotaryEmbedding.__init__r) rgrhrir rzrrrxrr?rkr=r=r`rbrs.rFmoduleModernBertAttentionrsliding_window_maskrObsrsoutput_attentionsc Ks|j||d\} } |ddjdd\} } }t| | | | \} } |jd}t| | dd|}|dkr6|}||}tjj |dtj d  | j }tjj ||j|jd }t||}|dd}||d|}|rn||fS|fS) Nrnr rvrr࿩rqrqrqrr)ptraining) rotary_emb transposeunbindrhead_dimrxmatmulrr}ZsoftmaxZfloat32tortZdropoutrLrrr)rrrmrrnrOrrsr_kwargsrrquerykeyvaluescaleZ attn_weights attn_outputr=r=rbeager_attention_forwards    rr target_dtypec Ks||||d}|jtjtjfv} | r1|j} ||}t||||jr$|jnd|j|d} | | } nt||||jr;|jnd|j|d} | ||fS)Nrr/)rr dropout_pZ deterministicZ window_size) rtrxfloat16bfloat16rrrrLrXr) rrrrrrOrrsrrZ convert_dtypeZ orig_dtypeattnr=r=rbflash_attention_forward$s.   rcKs|j||d\} } |ddjdd\} } } t| | | | \} } |dkr%|}tj| | | |jr0|jnd|ddd}| |d |}|fS) Nrr rvrrrr/)rZ attn_maskrq) rrrrFZscaled_dot_product_attentionrrLrr)rrrmrrnrOrrsrrrrrrrr=r=rbsdpa_attention_forwardOs"  r)flash_attention_2eagersdpacsReZdZdZd dedeeffdd Z ddej d ee d ej fd d Z Z S)raPerforms multi-headed self attention on a batch of unpadded sequences. If Flash Attention 2 is installed, this module uses Flash Attention to improve throughput. If Flash Attention 2 is not installed, the implementation will use PyTorch's SDPA kernel, which requires padding and unpadding inputs, adding some overhead. See `forward` method for additional details. Nrlayer_idcs\t||_||_|j|jdkr td|jd|jd|j|_|j|_|j|_ |j|j|_ |j |j |_ t j |jd|j |jd|_||jdkr[|jd|jdf|_nd|_|j}|j}|jdkrt|jdurq|j}|j}|jd krt|j ||d |_n t||j |d |_t j |j|j|jd|_|jd krt |jnt |_t|_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r rrrr)rsrr)rrsrr/)r>r?rrrBrEr]rLrX num_headsr all_head_sizerrrKWqkvrNrOrJrArP_attn_implementationrrrrrIdentityout_dropsetZ pruned_heads)r^rrZ rope_thetarAr`r=rbr?s<       zModernBertAttention.__init__Frrrpc Ks||}|jd}|jjdkr|dd|j|j}n ||dd|j|j}t|jj|f||j|j ||j |d|}|d}| | |}|f|ddS)Nrrrqr )rrrOrrsrrv) rrrrrrrMODERNBERT_ATTENTION_FUNCTIONrrOrrr)r^rrr_rr attn_outputsr=r=rbrs(     zModernBertAttention.forwardrF) rgrhrirjr rrzr?rxrboolrrkr=r=r`rbrys +cseZdZddedeeffdd Zejdddej d ej fd d Z      ddej d eej deej deej deej deedee d ej fddZ ZS)ModernBertEncoderLayerNrrcspt||_|dkrt|_n tj|j|j|j d|_t ||d|_ tj|j|j|j d|_ t ||_dS)Nrr)rr)r>r?rrr attn_normrrBrHrIrrmlp_normrmlp)r^rrr`r=rbr?s  zModernBertEncoderLayer.__init__TrrrpcC|||Sr)rrr^rr=r=rb compiled_mlpz#ModernBertEncoderLayer.compiled_mlpFrmrrnrrrc Csf|j||||||||d}||d}|jjr||n|||} || }|f|ddS)Nrmrrnrrrrrv)rrrr[rrr) r^rrmrrnrrrrZ mlp_outputr=r=rbrs    zModernBertEncoderLayer.forwardr)NNNNNF)rgrhrir rrzr?rxrrrrrrrkr=r=r`rbrs6  rc seZdZeZdZdZddgZdZdZ dZ de j fddZ e  dd ed eejd eeeeeeffd effdd ZddZfddZZS)ModernBertPreTrainedModelmodelTrrFrcsv|jjdur ddtjdtffdd }|jj|jjtd|jj|jj|jj dd}t |t r?||j |d dSt |t rV||j|d ||j|d dSt |trm||j|d ||j|d dSt |tr|||j|d dSt |tr||j|d dSt |tttfr||j|d dSt |tjr|jjd |jdur|jjdSdSdS)Nr rstdcsRtjj|jd| ||dt|tjr%|jdur'tj|jdSdSdS)Nr/)r6rab)rinitZ trunc_normal_weight isinstancerrZzeros_)rrZ cutoff_factorr=rb init_weights  z.init_weightr)r)inout embedding final_outr r r r g?) rrGrModulerrFmathsqrtrDrBrrrrrrrrModernBertPredictionHeaddenseModernBertForMaskedLMdecoder#ModernBertForSequenceClassification ModernBertForTokenClassificationModernBertForQuestionAnswering classifierrrdataZfill_rZzero_)r^rrZstdsr=rrb _init_weightss@        z'ModernBertPreTrainedModel._init_weightsNuse_flash_attention_2 torch_dtype device_mapcheck_device_mapc s^|jdur#d|_z |j|tj|d|dWSttfy"d|_Ynwtj||tj||dS)NrF)rrZhard_check_onlyr)rrrr)Z_attn_implementation_internalZ_check_and_enable_flash_attn_2rxrr] ImportErrorr>_autoset_attn_implementation)r2rrrrrr`r=rbr0s(  z6ModernBertPreTrainedModel._autoset_attn_implementationcCs|jjdurdSt|dr!t|jdkr!|jjrtdd|j_|jjdkr4|jjr0tdd|j_|jjdkrG|jjrCtdd|j_|jjdurTt |j_dSdS) NF hf_device_maprvzqIf `accelerate` split the model across devices, `torch.compile` will not work. Falling back to non-compiled mode.Zmpsz|Compiling the model with `torch.compile` and using a `torch.mps` device is not supported. Falling back to non-compiled mode.cpuz|Compiling the model with `torch.compile` and using a `torch.cpu` device is not supported. Falling back to non-compiled mode.) rr[hasattrlenr logger warning_oncertyperrr=r=rb_maybe_set_compileQs.    z,ModernBertPreTrainedModel._maybe_set_compilecs<tj|i|}|jjdvr|jjrtdd|j_|S)N>TNzcResizing token embeddings with `torch.compile` is not supported. Falling back to non-compiled mode.F)r>resize_token_embeddingsrr[r$r%)r^argsr_Z model_embedsr`r=rbr(ps z1ModernBertPreTrainedModel.resize_token_embeddings)FNNT)rgrhrir Z config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attn_2Z_supports_sdpaZ_supports_flex_attnrr r classmethodrrrxrtrrrrzrr'r(rkr=r=r`rbrs2/ rc!seZdZdeffdd ZddZddZe             dd ee j d ee j d ee j d ee j d ee j dee j dee j dee dee dee dee dee dee deee j dfeffddZd e j de de j fddZZS)ModernBertModelrcsft|_t|_tfddtjD|_ tj j j j d|_d|_|dS)Ncsg|]}t|qSr=)r).0rrr=rb sz,ModernBertModel.__init__..rF)r>r?rr embeddingsrZ ModuleListrangerDlayersrrBrHrI final_normgradient_checkpointing post_initrr`r-rbr?s   zModernBertModel.__init__cCs|jjSrr/rrr=r=rbget_input_embeddingssz$ModernBertModel.get_input_embeddingscCs ||j_dSrr5)r^rr=r=rbset_input_embeddingss z$ModernBertModel.set_input_embeddingsNrrmrrnrrrr batch_sizeseq_lenroutput_hidden_states return_dictrp.c s| dur| n|jj} | dur| n|jj} | dur| n|jj} |du|duAr*td| r.dnd}| r4dnd}||durD|||durcdurc|durZ|jdd\n |jdd\|durj|jn|j}|dur|t j f|t j d}d}|jj dkrˆdur|dur|durd}|durt t||d ^}}}}Wdn1swYn#t||d ^}}}}n|durt j|d d }|j|| d \}}|j||d }|jD]?}| r||f}|jr|jr||j||||||| }n |||||||| d}|d }| r"t|dkr"||df}q| r+||f}||}|rMt|d}|durMtfdd|D}| s\tdd|||fDSt|||dS) sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. Nz:You must specify exactly one of input_ids or inputs_embedsr=rrFrT)rlrm)rr)r)rrrrvrlrrrc3s |] }t|dVqdS)r=N)r)r,hsr8rr9r=rb s  z*ModernBertModel.forward..css|] }|dur|VqdSrr=)r,vr=r=rbr@s)last_hidden_stater attentions)rrr:use_return_dictr]r'Z%warn_if_padding_and_no_attention_maskrrrxonesrrno_gradrarange unsqueeze_update_attention_maskr/r1r3rZ_gradient_checkpointing_func__call__r#r2rtupler)r^rrmrrnrrrrr8r9rr:r;Zall_hidden_statesZall_self_attentionsrZrepadrrZ encoder_layerZ layer_outputsr=r?rbrs!             zModernBertModel.forwardcCs|r#|jjdkrtdd|j_n|jjdkr#td|jjdt||j}t|jd d}t ||j }||jj dk d d |j}||t|jj}||fS)NrzOutputting attentions is only supported with the 'eager' attention implementation, not with "sdpa". Falling back to `attn_implementation="eager"`.rzZOutputting attentions is only supported with the eager attention implementation, not with zT. Consider setting `attn_implementation="eager"`. Setting `output_attentions=False`.rr)rrr$r%rrtrxrGrrHabsTrOrrZ masked_fillZ logical_notZfinfomin)r^rmrZglobal_attention_maskrowsZdistanceZ window_maskrr=r=rbrIs&    "z&ModernBertModel._update_attention_mask NNNNNNNNNNNNN)rgrhrir r?r6r7rrrxrrrzrrrrrrIrkr=r=r`rbr+}sb       "r+cs8eZdZdeffdd ZdejdejfddZZS)rrcsNt||_t|j|j|j|_t|j |_ tj |j|j |j d|_dS)Nr)r>r?rrrrBrVrr rWrrrHrIrrr`r=rbr?8s  z!ModernBertPredictionHead.__init__rrpcCrr)rrrrr=r=rbr?sz ModernBertPredictionHead.forward) rgrhrir r?rxrrrkr=r=r`rbr7srzd The ModernBert Model with a decoder head on top that is used for masked language modeling. )Z custom_introc"s eZdZdgZdeffdd ZddZdejfdd Z e j d d d e j d e j fddZ e              d!dee jdee j dee j dee j dee j dee j dee j dee j deedeedeedeedeedeed eee j effdd ZZS)"rzdecoder.weightrcs^t|||_t||_t||_tj|j |j |j d|_ |jj |_ |jj|_|dS)Nr)r>r?rr+rrheadrrrBr@rTrrYrZr4rr`r=rbr?Ks      zModernBertForMaskedLM.__init__cCs|jSrrrr=r=rbget_output_embeddingsXsz+ModernBertForMaskedLM.get_output_embeddingsnew_embeddingscCs ||_dSrrR)r^rTr=r=rbset_output_embeddings[s z+ModernBertForMaskedLM.set_output_embeddingsTrrfrpcCrr)rrQrer=r=rb compiled_head^rz#ModernBertForMaskedLM.compiled_headNrrmrrnrrorrrr8r9rr:r;cKsb|dur|n|jj}||jjdkr|dur|dur| dur| dur?| dur?|dur6|jdd\} } n |jdd\} } |durF|jn|j}|durXtj| | f|tjd}|durt t ||||d\}}}} }}Wdn1sywYnt ||||d\}}}} }}|j |||||||| | | | | |d }|d}|j r|dur| d}| |jdd}||jk}||}||}|jjr||n|||}d}|dur|j|||jjd }|jjdkr|jjs|durtnt t||| | d }Wdn 1swY|s'|f}|dur%|f|S|St|||j|jd S) r<Nrrr)rlrmrnro rrmrrnrrrrr8r9rr:r;rrq)r@r=losslogitsrrC)rrDr'rrrrxrErrFrrrYrrZr[rVrrQ loss_functionr@r\rrrrrC)r^rrmrrnrrorrrr8r9rr:r;r_routputsrBZ mask_tokensrZrYrfr=r=rbrbs~#      zModernBertForMaskedLM.forwardNNNNNNNNNNNNNN)rgrhriZ_tied_weights_keysr r?rSrrrUrxrrrVrrrrzrrrrrrkr=r=r`rbrCsj       rz` The ModernBert Model with a sequence classification head on top that performs pooling. c"eZdZdeffdd Ze              ddeejdeej deej deej d eej d eej d eej d eej d ee dee dee dee dee dee de e ej effddZZS)rrcs\t||j|_||_t||_t||_tj |j |_ t |j|j|_|dSr)r>r? num_labelsrr+rrrQrxrrrUrrrBrr4rr`r=rbr?s    z,ModernBertForSequenceClassification.__init__Nrrmrrnrrorrrr8r9rr:r;rpcKs|dur|n|jj}||j|||||||| | | | | |d }|d}|jjdkr3|dddf}n|jjdkrK||djdd|jdd d }||}||}| |}d}|dur|jj dur|j dkrpd |j_ n|j dkr|j t jks|j t jkrd |j_ nd |j_ |jj d krt}|j dkr|||}n+|||}n%|jj d krt}||d|j |d}n|jj d krt}|||}|s|f}|dur|f|S|St|||j|jdS)aB sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. NrWrr2r6rqrvrT)rsZkeepdimZ regressionZsingle_label_classificationZmulti_label_classificationrX)rrDr'rr4rHrwrQrrZ problem_typer_rtrxlongrzr squeezer rr rrrC)r^rrmrrnrrorrrr8r9rr:r;r_r\rBZ pooled_outputrZrYloss_fctrfr=r=rbrsn'        "       z+ModernBertForSequenceClassification.forwardr])rgrhrir r?rrrxrrrzrrrrrrkr=r=r`rbrs`      rzv The ModernBert Model with a token classification head on top, e.g. for Named Entity Recognition (NER) tasks. c"r^)rrcVt||j|_t||_t||_tj |j |_ t |j |j|_|dSrr>r?r_r+rrrQrxrrrUrrrBrr4rr`r=rbr?Us    z)ModernBertForTokenClassification.__init__Nrrmrrnrrorrrr8r9rr:r;rpcCs|dur|n|jj}||j|||||||| | | | | |d }|d}||}||}||}d}|durIt}||d|j |d}|s_|f|dd}|dur]|f|S|St |||j |j dS)a sliding_window_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding or far-away tokens. In ModernBert, only every few layers perform global attention, while the rest perform local attention. This mask is used to avoid attending to far-away tokens in the local attention layers when not using Flash Attention. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. indices (`torch.Tensor` of shape `(total_unpadded_tokens,)`, *optional*): Indices of the non-padding tokens in the input sequence. Used for unpadding the output. cu_seqlens (`torch.Tensor` of shape `(batch + 1,)`, *optional*): Cumulative sequence lengths of the input sequences. Used to index the unpadded tensors. max_seqlen (`int`, *optional*): Maximum sequence length in the batch excluding padding tokens. Used to unpad input_ids and pad output tensors. batch_size (`int`, *optional*): Batch size of the input sequences. Used to pad the output tensors. seq_len (`int`, *optional*): Sequence length of the input sequences including padding tokens. Used to pad the output tensors. NrWrrqrvrX) rrDr'rrQrrr rr_rrrC)r^rrmrrnrrorrrr8r9rr:r;r\rBrZrYrbrfr=r=rbrasD$   z(ModernBertForTokenClassification.forwardr])rgrhrir r?rrrxrrrzrrrrrrkr=r=r`rbrOs`      rc"seZdZdeffdd Ze             ddeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee dee dee dee de e eje ffddZZS)rrcrcrrdrr`r=rbr?s    z'ModernBertForQuestionAnswering.__init__Nrrmrrnstart_positions end_positionsrrrr8r9rr:r;rpcKs|dur|n|jj}||j||||||| | | | | |d }|d}||}||}||}|jddd\}}|d }|d }d}|dur_|dur_|j ||||fi|}|sv||f|dd}|durt|f|S|St ||||j |j dS)r<N) rmrrnrrrr8r9rr:r;rrvrqr)rY start_logits end_logitsrrC)rrDr'rrQrrsplitrarr[rrrC)r^rrmrrnrerfrrrr8r9rr:r;r_r\rBrZrgrhrYrfr=r=rbrsH#   z&ModernBertForQuestionAnswering.forwardrP)rgrhrir r?rrrxrrzrrrrrrkr=r=r`rbrs^      r)r r+rrrrrrr)Tr contextlibrtypingrrrrrrxZtorch.nn.functionalrr}rZtorch.utils.checkpointZtorch.nnr r r Z activationsr Zconfiguration_utilsrZmodeling_attn_mask_utilsrZmodeling_outputsrrrrrZmodeling_utilsrutilsrrrZutils.import_utilsrZgemma.modeling_gemmarrZflash_attn.flash_attn_interfacerZflash_attn.layers.rotaryrZflash_attn.ops.triton.rotaryrobjectZ get_loggerrgr$r rrzrrZautogradFunctionrrrr rrrrrrrrtrrrrrrr+rrrrr__all__r=r=r=rbs0          H$ ) ; 5   .   +  $P.:  wZ [