o Zhr@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZdd lmZmZm Z m!Z!ddl"m#Z#eruddl$m%Z%ddl&m'Z'e!(e)Z*Gdddej+Z,Gdddej+Z-Gdddej.Z/Gddde/Z0e/e0dZ1Gdddej.Z2eGdddeZ3eGdd d e3Z4ed!d"Gd#d$d$e3eZ5eGd%d&d&e3Z6ed'd"Gd(d)d)e3Z7gd*Z8dS)+zPyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availableis_torchdynamo_compilinglogging) BioGptConfig) BlockMask)make_flex_block_causal_maskcsReZdZdZdedeffdd Z  d dejd ed eej ffd d Z Z S) BioGptLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcsd|_t||j|dS)N)offsetsuper__init__)selfrr __class__Y/var/www/auris/lib/python3.10/site-packages/transformers/models/biogpt/modeling_biogpt.pyr"?sz)BioGptLearnedPositionalEmbedding.__init__rNattention_maskpast_key_values_length position_idscsV|dur"|}tj|dd||d}|dd|df}t||jS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)longtorchZcumsumtype_asr!forwardr )r#r(r)r*Z positionsr$r&r'r0Es  z(BioGptLearnedPositionalEmbedding.forwardrN) __name__ __module__ __qualname____doc__intr"r. LongTensorrTensorr0 __classcell__r&r&r$r'r:s rc sLeZdZdZd dedededeeffdd Zd ej ffd d Z Z S) BioGptScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?rr padding_idx embed_scalecst|||||_dSN)r!r"r=)r#rrr<r=r$r&r'r"^s z"BioGptScaledWordEmbedding.__init__ input_idscst||jSr>)r!r0r=)r#r?r$r&r'r0bsz!BioGptScaledWordEmbedding.forward)r;) r2r3r4r5r6rfloatr"r.r8r0r9r&r&r$r'r:Ys$r:cseZdZdZ      ddededed ed ed ed eed eeffdd Z      dde j dee j dee dee j dee j dedee j de e j ee j ee e j ffddZZS)BioGptAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfig layer_idxc st||_||_||_|||_||_|j||jkr*td|jd|d|jd|_||_ ||_ ||_ |durK|j rKt d|jjdtj|||d|_tj|||d|_tj|||d|_tj|||d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.rG)r!r"rCrDrEhead_dimrI ValueErrorscalingrFrHrJlogger warning_oncer%r2rLineark_projv_projq_projout_proj) r#rCrDrErFrGrHrIrJr$r&r'r"js0    zBioGptAttention.__init__ hidden_stateskey_value_statespast_key_valuer(layer_head_maskoutput_attentionscache_positionreturncCsv|du}|\} } } ||| d|j|jdd} | |j} |dur=t|tr;|j |j } |r7|j }n|j }n|}|rA|n|}|rX|durX| rX|j|j }|j|j }nE||}||}|| d|j|jdd}|| d|j|jdd}|dur|s|nd}||||j d|i\}}|rd|j |j <| |jd|jf}| j|} |j|}|j|}|d}t| |dd}|| |j| |fkrtd| |j| |fd||dur|ddddddd|jd f}|| |j| ||}|| |j| |}tjj|dd }|durN||jfkr3td |jfd||dddd|| |j| |}|| |j| |}|re|| |j| |}|| |j| |}nd}tjj||j|jd }t||}|| |j| |jfkrtd | |j| |jfd||| |j| |j}|dd}|| | |j}||}|||fS)#Input shape: Batch x Time x ChannelNrrr[Tz$Attention weights should be of size z , but is r+z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizerTviewrDrL transposerN isinstancer is_updatedgetrJcross_attention_cacheself_attention_cache key_cache value_cacherRrSupdatereshaper.ZbmmrMshaper functionalZsoftmaxrErbrCrU)r#rVrWrXr(rYrZr[is_cross_attentionbsztgt_len_ query_statesrgcurr_past_key_valuecurrent_states key_states value_statesZ proj_shapeZsrc_lenZ attn_weightsZattn_weights_reshapedZ attn_probs attn_outputr&r&r'r0s"             &  "    zBioGptAttention.forward)rBFTFNNNNNNFN)r2r3r4r5r6r@boolrrr"r.r8r rr0r9r&r&r$r'rAgs` * rAcseZdZ      d dejdeejdeedeejdeejded eejd eejeejeeejfffd d Z Z S)BioGptSdpaAttentionNFrVrWrXr(rYrZr[r\cs,|rtdtj||||||dS|du}|\} } } ||| d|j|j dd} |durKt |t rI|j |j} |rE|j}n|j}n|}|rO|n|}|rf|durf| rf|j|j}|j|j}nE||}||}|| d|j|j dd}|| d|j|j dd}|dur|s|nd}||||jd|i\}}|rd|j |j<d}|dur|ddddddd|jd f}| jjd kr|dur| } |}|}|jr|dur| dkrdnd }tjjj| ||||jr|j nd |d }| dd}|| | |j!}|"|}|d|fS)r]aBioGptModel is using BioGptSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` . Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rWrXr(rZr[Nr^rrr[Tr_cudaFrB)Z attn_maskZ dropout_prH)#rOrPr!r0rcrTrdrDrLrerfr rgrhrJrirjrkrlrRrSrmrodevicetype contiguousrHr.rrpZscaled_dot_product_attentionrbrErCrU)r#rVrWrXr(rYrZr[rqrrrsrtrurgrvrwrxry causal_maskrHrzr$r&r'r0sr  "       &   zBioGptSdpaAttention.forwardr{) r2r3r4r.r8rr r|rr0r9r&r&r$r'r}s0 r})eagersdpacseZdZddedeeffdd Z      ddejd eejd eejd ee d ee d ee deejde ej ee ej ej fffddZ ZS)BioGptDecoderLayerNrIrJcst|j|_t|j|j|j|jdd|d|_|j |_ t |j |_ |j|_t|j|_t|j|j|_t|j|j|_t|j|_dS)NT)rCrDrErFrHrJ)r!r" hidden_sizerCBIOGPT_ATTENTION_CLASSES_attn_implementationZnum_attention_headsZattention_probs_dropout_prob self_attnhidden_dropout_probrEr Z hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normrQZintermediate_sizefc1fc2final_layer_norm)r#rIrJr$r&r'r"us"  zBioGptDecoderLayer.__init__FTrVr(rYrXrZ use_cacher[r\c Cs|}||}|j||||||d\}} }tjj||j|jd}||}|}||}||}||}tjj||j |jd}| |}tjj||j|jd}||}|f} |r^| | f7} |re| |f7} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. )rVrXr(rYrZr[r`) rrrrprErbrrrrr) r#rVr(rYrXrZrr[ZresidualZself_attn_weightsoutputsr&r&r'r0s4        zBioGptDecoderLayer.forwardr>)NNNFTN)r2r3r4rrr6r"r.r8r r|r FloatTensorr0r9r&r&r$r'rts2 rc @seZdZeZdZdZdZdZdZ ddZ dde e j dfde j d e j d ed ef d d Zede j dedede jd e j def ddZdS)BioGptPreTrainedModelbiogptTcCst|tjr |jjjd|jjd|jdur|jj dSdSt|tj rC|jjjd|jjd|j durA|jj|j  dSdSt|tj rX|jj |jj ddSdS)zInitialize the weightsrB)meanZstdNr;)rfrrQweightdataZnormal_rIZinitializer_rangerGZzero_ Embeddingr<rZfill_)r#moduler&r&r' _init_weightss      z#BioGptPreTrainedModel._init_weightsFr(r input_tensorr[past_key_valuesrZc Cs:|jjdkr|dur|dkr|SdS|jjdkr&t|tjr$t|}|S|dur.|nd}|dur7|jnd}|jjdkrO|sO|sOt j ||||j drOdS|j }|j d} |r^|} nt|tjri|j d n|| d} |j|| | |||j dd } |jjdkr|dur|jjd vr|st|j} t | | } | S) NZflash_attention_2rBZflex_attentionrFr) inputs_embedsr)Z is_trainingrr^)sequence_length target_lengthdtyper[ batch_size)r~ZxpuZnpu)rIranyrfr.r8rget_seq_lengthZis_compileablerZ_ignore_causal_mask_sdparbrroZget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrfinfominZ_unmask_unattended) r#r(rr[rrZZpast_seen_tokensZusing_compilable_cacherrrr min_dtyper&r&r'_update_causal_masksT           z)BioGptPreTrainedModel._update_causal_maskrrrrc KsD|dur|dkr|}|St|j}tj||f|||jd}|dkr+tj|dd}|tj||jd|ddk9}|ddddddf |ddd}|dur| }|j d} |ddddddd| f|ddddddf |j} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N)Z fill_valuerrr)Zdiagonalrr^r)r,r.rrfullrZtriuarangernexpandclonerotoZ masked_fill) r(rrrr[rkwargsrrZ mask_lengthZ padding_maskr&r&r'r*s,  $ 6  zKBioGptPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionN)F)r2r3r4rZ config_classZbase_model_prefixZsupports_gradient_checkpointingZ_supports_sdpaZ_supports_cache_classZ_supports_static_cacherrr.r8r r|r staticmethodr6rrr&r&r&r'rsD  DrcseZdZdeffdd ZddZddZe           dd ee j d ee j d ee j d ee j d ee e e j deedee j deedeedeedee j dee effddZZS) BioGptModelrIcst|_j|_j|_j|_j|_ j r"t jnd}t j|j|j |d|_tj|j|_tfddtjD|_t|j|_d|_jdk|_|dS)Nr;)r=csg|]}t|dqS))rJ)r).0irIr&r' ssz(BioGptModel.__init__..Fr)r!r"rI layerdroprrErrC pad_token_idr<Zscale_embeddingmathsqrtr: vocab_size embed_tokensrZmax_position_embeddingsembed_positionsrZ ModuleListrangeZnum_hidden_layerslayersr layer_normgradient_checkpointingrZ _use_sdpa post_init)r#rIr=r$rr'r"es    zBioGptModel.__init__cC|jSr>rr#r&r&r'get_input_embeddings{z BioGptModel.get_input_embeddingscC ||_dSr>rr#valuer&r&r'set_input_embeddings~ z BioGptModel.set_input_embeddingsNr?r( head_maskrrrr*rZoutput_hidden_states return_dictr[r\c  Ks|dur|n|jj}| dur| n|jj} |dur|n|jj}| dur$| n|jj} |du|duAr4td|durA|d|jd}|durJ||}|j rY|j rY|rYt dd}d} |rnt |tsnd} t dt|}|dd\}}|dur|nd}| durtj||||jd} |durts||}tj|||jd}t |tr|jn|}|||| ||}|dur| d}|j|||d }||}tjj||j|j d }| rd nd}|rd nd}d}d}t|j D]h\}}| r||f7}|j rt!g}||j"krq|j r'|j r'|#|j$|||dur||ndd||| }n||||dur3||nd|||| d }|d}|rK||rHd nd}|rU||df7}q| r^||f7}|%|}|rh|nd}| rq|&}| st'dd|||||fDSt(|||||dS)NzDYou cannot specify both input_ids and inputs_embeds at the same timer^zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr)r*r`r&)r(rYrXrZrr[rrcss|] }|dur|VqdSr>r&)rvr&r&r' sz&BioGptModel.forward..)Zlast_hidden_staterrV attentionscross_attentions))rIrZrruse_return_dictrMrdrorrrbrOrPrfr r Zfrom_legacy_cachercrr.rrrZonesrjrZ unsqueezerrrprE enumeraterZrandrZ_gradient_checkpointing_func__call__rZto_legacy_cachetupler)r#r?r(rrrrr*rZrrr[rZreturn_legacy_cacherZ seq_lengthr)Zmask_seq_lengthZself_attn_cacherrVZall_hidden_statesZall_self_attnsZall_cross_attentionsZnext_decoder_cacheidxZ decoder_layerZdropout_probabilityZ layer_outputsZ next_cacher&r&r'r0s               zBioGptModel.forward NNNNNNNNNNN)r2r3r4rr"rrrrr.r7rrr8r|rrr0r9r&r&r$r'rcsR     rzR BioGPT Model with a `language modeling` head on top for CLM fine-tuning. )Z custom_introcseZdZdgZfddZddZddZe            dd ee j d ee j d ee j d ee j d ee e e j dee j deedee j deedeedeedee j dee effddZeddZZS)BioGptForCausalLMzoutput_projection.weightcs8t|t||_tj|j|jdd|_| dSNFrK) r!r"rrrrQrroutput_projectionrr#rIr$r&r'r"#s   zBioGptForCausalLM.__init__cCrr>rrr&r&r'get_output_embeddings,rz'BioGptForCausalLM.get_output_embeddingscCrr>r)r#Znew_embeddingsr&r&r'set_output_embeddings/rz'BioGptForCausalLM.set_output_embeddingsNr?r(rrrlabelsrr*rZrrr[r\c  Ks| dur| n|jj} |j|||||||| | | | d }|d}||}d}|dur7|j||fd|jji| }| sM|f|dd}|durK|f|S|St|||j|j|j |j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) r(rrrrr*rZrrr[rrr)losslogitsrrVrr) rIrrrZ loss_functionrrrrVrr)r#r?r(rrrrrr*rZrrr[rrZsequence_outputZprediction_scoresZlm_lossoutputr&r&r'r02sJ zBioGptForCausalLM.forwardcs.d}|D]}|tfdd|Df7}q|S)Nr&c3s$|] }|d|jVqdSr1)Z index_selectrr)rZ past_statebeam_idxr&r'rws"z3BioGptForCausalLM._reorder_cache..)r)rrZreordered_pastZ layer_pastr&rr'_reorder_cachers z BioGptForCausalLM._reorder_cache) NNNNNNNNNNNN)r2r3r4Z_tied_weights_keysr"rrrrr.r7rrr8r|rrr0rrr9r&r&r$r'rs^         ?rcseZdZfddZe           ddeejdeejdeejdeejdee e ej d eejd eejd ee d ee d ee dee de e e ffddZZS)BioGptForTokenClassificationcsjt||j|_t||_t|dr|jdur|j}n|j}t ||_ t |j |j|_ |dS)Nclassifier_dropout)r!r" num_labelsrrhasattrrrrZDropoutrErQr classifierr)r#rIrr$r&r'r"~s    z%BioGptForTokenClassification.__init__Nr?token_type_idsr(rrrrrrZrrr\c  Cs | dur| n|jj} |j||||||| | | d } | d} || } || }d}|duret}|durX|ddk}|d|j}t ||dt |j  |}|||}n ||d|j|d}| s{|f| dd}|dury|f|S|St ||| j| jdS) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr(rrrrZrrrr^rr)rrrVr)rIrrrErrrdrr.whereZtensorZ ignore_indexr/rrVr)r#r?rr(rrrrrrZrrtransformer_outputsrVrrloss_fctZ active_lossZ active_logitsZ active_labelsrr&r&r'r0sF   z$BioGptForTokenClassification.forwardr)r2r3r4r"rrr.r7rrr8r|rrr0r9r&r&r$r'r|sN      ra The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZdeffdd Ze          ddeejdeej deej dee e ej d eej d eejd ee d ee d ee dee de e effddZddZddZZS)BioGptForSequenceClassificationrIcs@t||j|_t||_tj|j|jdd|_| dSr) r!r"rrrrrQrscorerrr$r&r'r"s   z(BioGptForSequenceClassification.__init__Nr?r(rrrrrrZrrr\c  Cs| dur| n|jj} |j|||||||| | d } | d} || } |dur/|jdd\}}n |jdd\}}|jjdurAd}n"|durWt||jjdd | j }n d}t |j jd| tj|| j d|f}d}|dur|jjdur|jdkrd |j_n|jdkr|jtjks|jtjkrd |j_nd |j_|jjd krt}|jdkr|||}n+|||}n%|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| j| jd S) rNrrrr^rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rZ regressionZsingle_label_classificationZmulti_label_classification)rrrrVr)rIrrrrorr.nesumrrrOrPr%r2rZ problem_typerrr-r6rZsqueezerrdrrrrVr)r#r?r(rrrrrrZrrrrVrrrZ pooled_logitsrrrr&r&r'r0sn  $    "       z'BioGptForSequenceClassification.forwardcCs|jjSr>rrrr&r&r'r>sz4BioGptForSequenceClassification.get_input_embeddingscCs ||j_dSr>rrr&r&r'rAs z4BioGptForSequenceClassification.set_input_embeddings) NNNNNNNNNN)r2r3r4rr"rrr.r7rrr8r|rrr0rrr9r&r&r$r'rsL     Xr)rrrrr)9r5rtypingrrrr.Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Z cache_utilsr r Z generationr Zmodeling_attn_mask_utilsrZmodeling_outputsrrrrZmodeling_utilsrutilsrrrrZconfiguration_biogptrZ!torch.nn.attention.flex_attentionrZintegrations.flex_attentionrZ get_loggerr2rOrrr:ModulerAr}rrrrrrr__all__r&r&r&r'sZ         jX8\P j