fThrSrSSKrSSKJrJrJr SSKrSSKrSSKJr SSK J r J r J r SSK Jr SSKJrJr SS KJr SS KJr SS KJrJrJrJr SS KJr SS KJrJrJ r J!r! SSK"J#r# \"5(a SSK$J%r% SSK&J'r' \!RP"\)5r*"SS\RV5r,"SS\RV5r-"SS\R\5r/"SS\/5r0\/\0S.r1"SS\R\5r2\"SS\55r3\"SS \355r4\"S!S"9"S#S$\3\55r5\"S%S&\355r6\"S'S"9"S(S)\355r7/S*Qr8g)+zPyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)CacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_availableis_torchdynamo_compilinglogging) BioGptConfig) BlockMask)make_flex_block_causal_maskc^\rSrSrSrS\S\4U4SjjrS S\RS\S\ \R4U4S jjjr S r U=r $) BioGptLearnedPositionalEmbedding:zF This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcL>SUl[TU] XR-U5 g)N)offsetsuper__init__)selfr r! __class__s b/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/biogpt/modeling_biogpt.pyr&)BioGptLearnedPositionalEmbedding.__init__?s"  ++5}Eattention_maskpast_key_values_length position_idsc>UcSUR5n[R"USS9RU5U-R5S- nUSS2US24n[TU]X0R -5$)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)longtorchcumsumtype_asr%forwardr$)r'r,r-r. positionsr(s r)r6(BioGptLearnedPositionalEmbedding.forwardEsv  +002Nn!<DD^TWeekkmpqqI%Q(>(?%?@Lw|kk9::r+)r$rN)__name__ __module__ __qualname____firstlineno____doc__intr&r3 LongTensorrTensorr6__static_attributes__ __classcell__r(s@r)rr:s[FsF3F'(/3 ;((;!$;u||, ;;r+rc r^\rSrSrSrS S\S\S\S\\4U4SjjjrS\ R4U4S jjr S r U=r $) BioGptScaledWordEmbeddingYzT This module overrides nn.Embeddings' forward by multiplying with embeddings scale. r r! padding_idx embed_scalec2>[TU]XU5 X@lgN)r%r&rI)r'r r!rHrIr(s r)r&"BioGptScaledWordEmbedding.__init__^s  D&r+ input_idsc<>[TU]U5UR-$rK)r%r6rI)r'rMr(s r)r6!BioGptScaledWordEmbedding.forwardbswy)D,<,<<r?rfloatr&r3rAr6rBrCrDs@r)rFrFYsJ's'3'S'_ghm_n''===r+rFc^\rSrSrSrSS\S\S\S\S\S\S \\ S \\4U4S jjjr SS \ RS \\ RS\\ S\\ RS\\ RS\S\\ RS\\ R\\ R\\\ R44SjjrSrU=r$)BioGptAttentiongz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsdropout is_decoderbias is_causalconfig layer_idxc t>[T U]5 XlX lX0lX-UlXplUR U-UR:wa[SURSUS35eUR S-UlX@l X`l Xl Uc>UR(a-[RSURRS35 [ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ul[ R""XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.rZ)r%r&rVrWrXhead_dimr\ ValueErrorscalingrYr[r]logger warning_oncer(r:rLineark_projv_projq_projout_proj) r'rVrWrXrYrZr[r\r]r(s r)r&BioGptAttention.__init__js  "" !.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 4@ ii 4@ ii 4@  )TB r+ hidden_stateskey_value_statespast_key_valuer,layer_head_maskoutput_attentionscache_positionreturnc USLnUR5upn URU5RU SURUR5R SS5n XR -n Ub][U[5(aFURRUR5n U(a URnOURnOUnU(aUOUnU(a=Ub:W (a3WRURnURURnOUR!U5nUR#U5nURU SURUR5R SS5nURU SURUR5R SS5nUbNU(dUOSnWR%UUURSU05unnU(aSURUR'XR-SUR4nU R&"U6n UR&"U6nUR&"U6nURS5n[(R*"U UR SS55nUR5XR-U U4:wa.[-SXR-U U4SUR535eUb]USS2SS2SS2SUR.S 24nURXRU U5U-nURXR-U U5n[0R2R5USS 9nUbUR5UR4:wa*[-S UR4SUR535eURSSSS5URXRU U5-nURXR-U U5nU(a=URXRU U5nURXR-U U5nOSn[0R2R7UUR6UR8S 9n[(R*"UU5nUR5XR-XR4:wa7[-S XR-XR4SUR535eURXRXR5nUR SS5nUR'XUR:5nUR=U5nUUU4$)#Input shape: Batch x Time x ChannelNrr#rpTz$Attention weights should be of size z , but is r0z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizerhviewrWr` transposerb isinstancer is_updatedgetr]cross_attention_cacheself_attention_cache key_cache value_cacherfrgupdatereshaper3bmmrashaper functionalsoftmaxrXrxrVri)r'rkrlrmr,rnrorpis_cross_attentionbsztgt_len_ query_statesr}curr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r)r6BioGptAttention.forwards.T9',,.a{{=166sBPTP]P]^hhijlmn #ll2  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#-?)] ."<,66t~~FJ.::4>>JL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=NN*B > #++Z8 ''4 #++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6nn8LgW^7_6`a %%'(*   %+Aq!5Kz7G7G7K5K,KLN',,S..'7SVddL',,S>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVdVdfmov?wwL',,S>>-A7GTL  %1$5$5c>>7T[$\ !055cNN6JGU\]L$( !]]**<4<AAr+) r\rXrVr`r[rYrfr]rWrirhrbrg)FTFNNNNNNFN)r:r;r<r=r>r?rRboolrrr&r3rAr rr6rBrCrDs@r)rTrTgsYG  )-#'%C%C%C %C  %C  %C%C&%CC=%C%CT48*.1526"'15pB||pB#5<<0pB! pB !. pB "%,,/ pB pB!.pB u||Xell3XeELL>Q5RR SpBpBr+rTc8^\rSrSrS S\R S\\R S\\S\\R S\\R S\S\\R S \ \R \\R \\ \R 44U4S jjjr S r U=r $) BioGptSdpaAttentionirkrlrmr,rnrorprqc >U(a'[RS5 [TU] UUUUUUS9$USLnUR 5upn UR U5R U SURUR5RSS5n Ub][U[5(aFURRUR5n U(a URnOUR nOUnU(aUOUnU(a=Ub:W (a3WR"URnUR$URnOUR'U5nUR)U5nUR U SURUR5RSS5nUR U SURUR5RSS5nUbNU(dUOSnWR+UUURSU05unnU(aSURUR'SnUbUSS2SS2SS2SUR,S 24nU R.R0S :Xa3Ub0U R35n UR35nUR35nUR4(a UcU S:aSOS n[6R8R:R=U UUUUR>(a UR@OS US 9nURSS5R35nUR XURB5nUREU5nUSU4$)rsaBioGptModel is using BioGptSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` . Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rlrmr,rorpNrtrr#rpTrucudaFr) attn_mask dropout_pr[)#rcrdr%r6ryrhrzrWr`r{r|r r}r~r]rrrrrfrgrrdevicetype contiguousr[r3rrscaled_dot_product_attentionrxrXrVri)r'rkrlrmr,rnrorprrrrrr}rrrr causal_maskr[rr(s r)r6BioGptSdpaAttention.forwards    l 7?!1--"3- # .T9',,.a{{=166sBPTP]P]^hhijlmn  %.*=>>+66::4>>J %*8*N*N'*8*M*M'&4#-?)] ."<,66t~~FJ.::4>>JL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=  %(Aq2HJ4D4DR4H2H)HIK    # #v -+2I'224L#..0J'224L !NN{/BwQR{DX] hh))FF   !&*mmdll G "++Aq1<<> "&&sT^^D mmK0 D.00r+r) r:r;r<r=r3rArr rrr6rBrCrDs@r)rrs48*.1526"'15e1||e1#5<<0e1! e1 !. e1 "%,,/ e1 e1!.e1 u||Xell3XeELL>Q5RR Se1e1r+r)eagersdpacB^\rSrSrSS\S\\4U4SjjjrSS\RS\\RS\\RS\\ S \\ S \\ S \\RS \ \R\\ \R\R444S jjrSrU=r$)BioGptDecoderLayeritr\r]c >[TU]5 URUl[UR "URUR URSSUS9UlURUl [URUl URUl[R "UR5Ul[R$"URUR&5Ul[R$"UR&UR5Ul[R "UR5Ulg)NT)rVrWrXrYr[r])r%r& hidden_sizerVBIOGPT_ATTENTION_CLASSES_attn_implementationnum_attention_headsattention_probs_dropout_prob self_attnhidden_dropout_probrXr hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normreintermediate_sizefc1fc2final_layer_norm)r'r\r]r(s r)r&BioGptDecoderLayer.__init__us ++1&2M2MNnn0077  11 #F$5$56"(";";$&LL$@!99T^^V-E-EF99V55t~~F " T^^ <r+rkr,rnrmro use_cacherprqc XUnURU5nURUUUUUUS9upn[RR XRUR S9nX-nUnUR U5nURU5nURU5n[RR XRUR S9nURU5n[RR XRUR S9nX-nU4n U(aX4- n U(aX4- n U $)ay Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. )rkrmr,rnrorprv) rrrrrXrxrrrrr) r'rkr,rnrmrorrpresidualself_attn_weightsoutputss r)r6BioGptDecoderLayer.forwards38!11-@ <@>>'))+/) ||>!.>"%,,/ > ! > $D> >D>>!.> u  (51B1BEDUDU1U+V"WW X>>r+rc \rSrSr\rSrSrSrSr Sr Sr SS\ \ RS4S\ RS\ RS \S \4 S jjr\S\ RS \S \S\ R(S\ RS\4 Sj5rSrg)BioGptPreTrainedModelibiogptTc[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weightsr)meanstdNrQ)r|rreweightdatanormal_r\initializer_rangerZzero_ EmbeddingrHrfill_)r'modules r) _init_weights#BioGptPreTrainedModel._init_weightss fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r+r,r input_tensorrppast_key_valuesroc URRS:XaUbUS:HR5(aU$gURRS:Xa,[U[R 5(a [ U5nU$UbUR5OSnUb UROSnURRS:Xa5U(d.U(d'[R"UUUURS9(agURnURSn U(aUR5n O5[U[R 5(aURS OXi-S-n URUU U UUURSS 9n URRS:XaZUbWUR R"S ;a=U(d6[R$"U5R&n [R("X5n U $) Nflash_attention_2rflex_attentionrFr) inputs_embedsr- is_trainingrrt)sequence_length target_lengthdtyperp batch_size)rxpunpu)r\ranyr|r3rArget_seq_lengthis_compileabler_ignore_causal_mask_sdparxrrget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrrfinfomin_unmask_unattended) r'r,rrprropast_seen_tokensusing_compilable_cacherrrr min_dtypes r)_update_causal_mask)BioGptPreTrainedModel._update_causal_masks ;; + +/B B)~/D.I.I.K.K%% ;; + +/? ?.%,,77!<^!L! ! @O?Z?99;`aCRC^!?!?di ;; + +v 5>T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr+rrrrc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerrr)diagonalrrtr)r1r3rrfullrtriuarangerexpandclonerto masked_fill) r,rrrrprkwargsrr mask_length padding_masks r)rKBioGptPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_position*s}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r+rN)F)r:r;r<r=r config_classbase_model_prefixsupports_gradient_checkpointing_supports_sdpa_supports_cache_class_supports_static_cacherrr3rAr rr staticmethodr?rrrBrr+r)rrsL &*#N !*.#( BellK78BllB B  B  BH4 444{{ 4  4  44r+rc~^\rSrSrS\4U4SjjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ \ \ RS \ \S \ \ RS \ \S\ \S\ \S\ \ RS\\ \44Sjj5rSrU=r$) BioGptModelicr\c >[TU]U5 XlURUlURUlUR UlURUl UR(a [R"UR 5OSn[URURURUS9Ul[!UR"UR5Ul[&R("[+UR,5Vs/sH n[/XS9PM sn5Ul[&R2"UR5UlSUlUR8S:HUlUR=5 gs snf)NrQrP)r]Fr)r%r&r\ layerdroprrXrrV pad_token_idrHscale_embeddingmathsqrtrF vocab_size embed_tokensrmax_position_embeddingsembed_positionsr ModuleListrangenum_hidden_layersrlayersr layer_normgradient_checkpointingr _use_sdpa post_init)r'r\rIir(s r)r&BioGptModel.__init__es!   ))11 ++!..7=7M7Mdii 2 23SV 5   t~~t/?/?[  @@^@^`d`n`nommV[\b\t\tVu$vVuQR%7%LVu$vw ,,t~~6&+#44>  %ws E9cUR$rKrr's r)get_input_embeddings BioGptModel.get_input_embeddings{s   r+cXlgrKrr'values r)set_input_embeddings BioGptModel.set_input_embeddings~s!r+rMr, head_maskrrrr.rooutput_hidden_states return_dictrprqc UbUOURRnU bU OURRn UbUOURRnU bU OURRn USLUSL- (a [ S5eUbUR SURS5nUcURU5nUR(a/UR(aU(a[RS5 SnSn U(aB[U[5(d-Sn [RS5 [R "U5nUR#5SSupUbUR%5OSnU c#[&R("UUU-UR*S9n Uc4[-5(d%UU-n[&R."UUUR*S9n[U[5(a UR0OUnUR3UUU UU5nUcU R5S5nUR7UUUS 9nXG-n[8R:R=UUR<URS 9nU (aS OSnU(aS OSnSnSn[?UR@5HunnU (aUU4- nUR(a([&RB"/5nUURD:aMLUR(a<UR(a+URGURHUUUbUUOSSUUU 5nOU"UUUbUUOSUUUU S 9nUSnU(aUU(aS OSnU(dMUUS4- nM U (aUU4- nURKU5nU(aUOSnU (aURM5nU (d[OSUUUUU455$[QUUUUUS9$)NzDYou cannot specify both input_ids and inputs_embeds at the same timertzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rr)r.rvr)r,rnrmrorrpr#rc30# UH nUcMUv M g7frKr).0vs r) &BioGptModel.forward.. smAms )last_hidden_staterrk attentionscross_attentions))r\ror)ruse_return_dictrarzrrrrxrcrdr|r r from_legacy_cacheryrr3rrronesrr unsqueezerrrrX enumeraterrandr _gradient_checkpointing_func__call__rto_legacy_cachetupler)r'rMr,r(rrrr.ror)r*rprreturn_legacy_cacher seq_lengthr-mask_seq_lengthself_attn_cacherrkall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cacheidx decoder_layerdropout_probability layer_outputs next_caches r)r6BioGptModel.forwards 2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?I   --i8M  & &4==##p" $ Z??"&    \  2CCOTO!.!3!3!5cr!: ETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D*D4zAO"ZZ OML`L`aN/+>??  0 0   ..         )33A6L++NV u//0V $D>V 'tnV d^V !.V  u?? @V V r+r zR BioGPT Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introc^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ \ \ RS \ \ RS \ \S \ \ RS\ \S\ \S\ \S\ \ RS\\ \44Sjj5r\S5rSrU=r$)BioGptForCausalLMizoutput_projection.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 gNFr_) r%r&r rrrerroutput_projectionrr'r\r(s r)r&BioGptForCausalLM.__init__#sJ  !&) !#6+=+=v?P?PW\!] r+cUR$rKrRr s r)get_output_embeddings'BioGptForCausalLM.get_output_embeddings,s%%%r+cXlgrKrV)r'new_embeddingss r)set_output_embeddings'BioGptForCausalLM.set_output_embeddings/s!/r+rMr,r(rrlabelsrr.ror)r*rprqc  U bU OURRn URUUUUUUUU U U U S9 nUSnURU5nSnUb*UR"UU4SURR 0U D6nU (dU4USS-nUbU4U-$U$[ UUURURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) r,r(rrrr.ror)r*rprrr)losslogitsrrkr2r3) r\r4rrR loss_functionrrrrkr2r3)r'rMr,r(rrr]rr.ror)r*rprrsequence_outputprediction_scoreslm_lossoutputs r)r6BioGptForCausalLM.forward2s.&1%# UH/oRSTRUR55v M1 g7fr9) index_selectrr)r- past_statebeam_idxs r)r/3BioGptForCausalLM._reorder_cache..ws1ncmU_--aZ=N=N1OPPcms7:)r=)rrkreordered_past layer_pasts ` r)_reorder_cache BioGptForCausalLM._reorder_cachers8)J ncmnn N*r+)rrR) NNNNNNNNNNNN)r:r;r<r=_tied_weights_keysr&rWr[rrr3r@rrrArrrr6rrorBrCrDs@r)rOrOse 55&0156:1559@D-1$(37,0/3&*15= E,,-= !!2!23= E--. =   1 12 = "%ell(;"<= = ))*= D>= u//0= $D>= 'tn= d^= !.=  u77 8= = ~r+rOcj^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\ \ \RS\\RS \\RS \\ S \\ S \\ S \\ S\ \ \44Sjj5rSrU=r$)BioGptForTokenClassificationi|c>[TU]U5 URUl[U5Ul[ US5(aUR b UR nO URn[R"U5Ul [R"URUR5Ul UR5 g)Nclassifier_dropout)r%r& num_labelsr rhasattrrurrDropoutrXrer classifierr)r'r\rur(s r)r&%BioGptForTokenClassification.__init__~s   ++!&) 6/ 0 0V5N5N5Z!'!:!: !'!;!; zz"45 ))F$6$68I8IJ r+rMtoken_type_idsr,r(rrr]rror)r*rqc U bU OURRn URUUUUUUU U U S9 n U Sn URU 5n UR U 5nSnUb[ 5nUbUR S5S:HnUR SUR5n[R"UUR S5[R"UR5RU55nU"UU5nO2U"UR SUR5UR S55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)e labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr,r(rrror)r*rrtrr#)r_r`rkr2)r\r4rrXryrrzrvr3wheretensor ignore_indexr5rrkr2)r'rMr{r,r(rrr]rror)r*transformer_outputsrkr`r_loss_fct active_loss active_logits active_labelsres r)r6$BioGptForTokenClassification.forwardso*&1%= $D>= 'tn= d^=  u++ ,= = r+rsa The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c^^\rSrSrS\4U4Sjjr\SS\\RS\\RS\\RS\\ \ \RS\\RS \\RS \\ S \\ S \\ S \\ S\\ \44Sjj5rSrSrSrU=r$)BioGptForSequenceClassificationir\c>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 grQ) r%r&rvr rrrerscorerrSs r)r&(BioGptForSequenceClassification.__init__sS   ++!&) YYv114??O  r+rMr,r(rrr]rror)r*rqc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UbURSSupOURSSupURR cSnOUbV[ R"XRR 5RS5S- RU R5nO.Sn[RURRS35 U [ R"XRS9U4nSnUGbURR cUR"S:XaS URlOoUR"S:aNUR$[ R&:XdUR$[ R(:XaS URlOS URlURR S :XaJ[+5nUR"S:Xa&U"UR-5UR-55nOU"UU5nOURR S :Xa=[/5nU"UR1SUR"5UR1S55nO-URR S :Xa[35nU"UU5nU (dU4U SS-nUbU4U-$U$[5UUU R6U R8U R:S 9$) r}Nr~rr#rtrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classification)r_r`rrkr2)r\r4rrrr r3nesumrrrcrdr(r:r problem_typervrr2r?r squeezerrzrrrrkr2)r'rMr,r(rrr]rror)r*rrkr`rr pooled_logitsr_rres r)r6'BioGptForSequenceClassification.forwards(&1%s{{'''r+c$XRlgrKrr$s r)r&4BioGptForSequenceClassification.set_input_embeddingsAs #(  r+)rrvr) NNNNNNNNNN)r:r;r<r=rr&rrr3r@rrrArrrr6r!r&rBrCrDs@r)rrs1|156:15@D59-1$(,0/3&*V E,,-V !!2!23V E--. V "%ell(;"<= V   1 12 V ))*V D>V $D>V 'tnV d^V  u66 7V V p())r+r)rOrsrr r)9r>rtypingrrrr3torch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrmodeling_utilsrutilsrrrrconfiguration_biogptr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerr:rcrrrFModulerTrrrrr rOrsr__all__rr+r)rs )) AA!5) . / !!;J   H % ;r||;> = =ZBbiiZB|f1/f1T  UUpSOSSlt 't t n Y-Y  YxM #8M M `  g)&;g) g)T r+