fThP~SrSSKJrJrJr SSKrSSKrSSKJr SSK J r J r J r SSK Jr SSKJrJrJr SS KJr SS KJrJrJr SS KJrJr S S KJr \R<"\5r Sr!Sr"S"Sjr#"SS\RH5r%Sr&"SS\RH5r'\"SS\55r(\"SS\(55r)\"SS9"SS\(\55r*\"SS9"SS \(55r+/S!Qr,g)#zPyTorch CTRL model.)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcPS[R"SSUS--U- 5- nX-$)Nri')torchpow)posi d_model_size angle_ratess ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr 's-eiiQ!V 'DEEK  c[[R"U[RS9R U5R S5[R"U[RS9R U5R S5U5n[R "USS2SSS245n[R"USS2SSS245n[R"XE/SS9nU$)Ndtyperrrdim) r rarangeint64to unsqueezesincoscat)positionrr$ angle_radssinescosines pos_encodings rpositional_encodingr4,s XU[[144U;EEaH \588?II!LJ IIjADqD) *Eii 1add7+,G99e-26L r!c [R"XRSSSS55nURSnU[R "U5- nUb3UR S5UR S5pXX- U 2SU 24S-- nUbX-n[R"USS9n UbX-n [R"X5n X4$) Nrrr rr%gr&)rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask head_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputs rscaled_dot_product_attentionrK;s Q !Q1 56I B'"''"+5 (--b13J3O3OPR3SB" crc(9#:T#AA!"9"J &=2F-9 \\+ /F  $$r!cH^\rSrSrU4SjrSrSrSSjrSrU=r $)MultiHeadAttentionUch>[TU]5 X lXl[ XR- 5Ul[ R"X5Ul[ R"X5Ul [ R"X5Ul [ R"X5Ul [5Ul gN)super__init__ num_headsrintdepthrLinearWqWkWvdenseset pruned_heads)selfrrS __class__s rrRMultiHeadAttention.__init__Vsv "(67 ))L7))L7))L7YY|: Er!cURUR-n[U5S:Xag[XRX R5up[ UR U5Ul[ URU5Ul[ URU5Ul[ URUSS9Ul UR[U5- UlX R-UlURRU5Ulg)Nrrr&) rrSlenrr\rrWrXrYrZunion)r]headsattention_head_sizeindexs r prune_headsMultiHeadAttention.prune_headsds"//4>>A u:? 7~~Obduduv %TWWe4$TWWe4$TWWe4' EqA #e*4/..@ --33E:r!cxURUSURUR5nUR/SQ5$)Nr%rrrr )reshaperSrUr8)r]x batch_sizes rsplit_into_heads#MultiHeadAttention.split_into_headsus- IIj"dnndjj Ayy&&r!c jURSn URU5nURU5nURU5nUR X:5nUR X*5nUR X5nUb5USUSp[ R "X4SS9n[ R "X4SS9nUSLa[ R"X!45n OSn [X2XXg5nUSR/SQ5nUSnURU SUR5nURU5nX4nU (aUU4-nU$) Nrrr6r&TrPrir%) r9rWrXrYrmrr.stackrKr8rjrrZ)r]r@r?r>rA layer_pastrBrC use_cacheoutput_attentionsrlpast_key past_valuepresentrJscaled_attentionattnoriginal_size_attentionoutputss rforwardMultiHeadAttention.forwardys0WWQZ GGAJ GGAJ GGAJ  ! !! 0  ! !! 0  ! !! 0  !#-a=*Q-j 8-R0A :/r2A  kk1&)GG-aA^W!!9,,\:ay"2":"::r4K\K\"]34# 'Gr!)rXrWrYrrZrUrSr\NNNFF) __name__ __module__ __qualname____firstlineno__rRrfrmr{__static_attributes__ __classcell__r^s@rrMrMUs- ";"'((r!rMc[R"[R"X5[R"5[R"X55$rP)r SequentialrVReLU)rdffs rpoint_wise_feed_forward_networkrs- ==<5rwwy"))CB^ __r!c8^\rSrSrSU4SjjrSSjrSrU=r$) EncoderLayerc6>[TU]5 [X5Ul[ X5Ul[ R"USS9Ul[ R"USS9Ul [ R"U5Ul [ R"U5Ul g)Ngư>eps) rQrRrMmulti_head_attentionrffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)r]rrSrrater^s rrREncoderLayer.__init__sk $6|$O!2<E,,|>,,|> 4(  4( r!c  URU5nURUUUUUUUUUS9 n U Sn URU 5n X-n URU 5n UR U 5n UR U 5n X-n U 4U SS-nU$)NrqrBrCrrrsrr)rrrrrr)r]rkrArqrBrCrrrsnormed attn_outputs attn_outputout1out2 ffn_outputrzs rr{EncoderLayer.forwards#00    !)/1  #1o mmK0 t$XXd^ ]]:.  'L,,r!)rrrrrr)g?r})r~rrrrRr{rrrs@rrrs )qvr!rc"\rSrSr\rSrSrSrg)CTRLPreTrainedModel transformerc$[U[R[45(akURR R SURRS9 URb%URR R5 gg[U[R5(axURR R SURRS9 URb2URR URR5 gg[U[R5(aJURR R5 URR RS5 gg)zInitialize the weights.g)meanstdN?) isinstancerrVrweightdatanormal_configinitializer_rangebiaszero_ Embedding padding_idxrfill_)r]modules r _init_weights!CTRLPreTrainedModel._init_weightss fryy&1 2 2 MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r!N) r~rrrr config_classbase_model_prefixrrrr!rrrsL%*r!rc^\rSrSrU4SjrSrSrSr\SS\ \ RS\ \ \ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \S\ \S\ \S\ \S\\ \ R \44Sjj5rSrU=r$) CTRLModelc >[TU]U5 URUlURUl[ URUR[R5Ul [R"URUR5Ul[R"UR 5Ul[R$"['UR5Vs/sH9n[)URUR*UR,UR.5PM; sn5Ul[R2"URUR4S9UlUR95 gs snf)Nr)rQrRn_embdrn_layer num_layersr4 n_positionsrfloatr3rr vocab_sizewr embd_pdropdropout ModuleListrangern_headr resid_pdrophrlayer_norm_epsilon layernorm post_init)r]r_r^s rrRCTRLModel.__init__s  "MM ../0B0BDDUDUW\WbWbcf//?zz&"3"34 afgmguguav wav\]\&-- FDVDV Wav w fmm9R9RS  xs#AE.cUR$rPrr]s rget_input_embeddingsCTRLModel.get_input_embeddingss vv r!cXlgrPrr]new_embeddingss rset_input_embeddingsCTRLModel.set_input_embeddingssr!cUR5H-up#URURRU5 M/ g)zf Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrf)r]heads_to_prunelayerrcs r _prune_headsCTRLModel._prune_headss5+002LE FF5M . . : :5 A3r! input_idspast_key_valuesrBtoken_type_ids position_idsrC inputs_embedsrrrsoutput_hidden_states return_dictreturnc dU bU OURRn UbUOURRnU bU OURRn U bU OURRn UbUb [ S5eUbFUR X5 UR5n URSU S5nURSnO1Ub#UR5SSn URSnO [ S5eUb URO URnUc%Sn[S/[UR5-5nOUSSRS5nUc<[R"UU SU-[R US9nUR#S5nUbUS::a [ S5eURUS5nUR#S 5R#S 5nUR%UR&S 9nS U- [R("UR&5R*-nUR-X`RR.5nUbJURSU S5nUR1U5nU[2R4"UR65-nOSnUcUR1U5nU Sn[R8"[R:"UU-UU-5S 5R%U5nU[2R4"UR65-nUR<R%U5UlUR<USS24nUU-U-nUR?U5nU(aS OSnU (aS OSnU (aS OSn[A[CURU55HJununnU (aUU4-nU"UUUUUUUU S9nUSS unnUSLaUU4-nU (dMAUUS 4- nML UREU5nU (aUU4-nU (d[SUUUU455$[GUUUUS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer%rz5You have to specify either input_ids or inputs_embedsr6)r$devicez$batch_size has to be defined and > 0rrr#rrrTc3.# UH ocMUv M g7frPr).0r@s r $CTRLModel.forward..sr$`q$`s )last_hidden_stater hidden_states attentions)$rrsrrruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr<viewr9rtuplerarrr(longr+r*r$finfomin get_head_maskrrr:r;rtriuonesr3r enumerateziprr )r]rrrBrrrCrrrrsrrkwargs input_shaperlr past_lengthtoken_type_embedsseq_lenrA pos_embedsrpresentsall_hidden_statesall_attentionsrrrqrzrvs rr{CTRLModel.forward s4`2C1N-TXT_T_TqTq!*!6IDKKcd d  "  6 6y Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T  "K#TFS[$89O)!,Q/44R8K   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,55a8BB1EN,..TZZ.@N!N2ekk$**6M6Q6QQN&&y++2E2EF  %+00[_EN $~ 6  ):):!; ;  !    FF9-Mb/zz%**W{%:GkOQ_$`rr r&+$+%   r!)rrrrrr3r) NNNNNNNNNNN)r~rrrrRrrrrrr LongTensorr FloatTensorboolrTensorr r{rrrs@rrrsN& B15EI6:59371559$(,0/3&*^ E,,-^ "%e.?.?(@"AB^ !!2!23 ^ !!1!12 ^ u//0 ^ E--.^   1 12^ D>^ $D>^ 'tn^ d^^  uU\\"$;; <^ ^ r!rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc4^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ \ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \S\ \S\ \S\ \S\\ \ R \44Sjj5rSSjr\S\ \ \ R S\ R S\ \ \ R 4Sj5rSrU=r$)CTRLLMHeadModelizlm_head.weightc>[TU]U5 [U5Ul[R "UR URSS9UlUR5 g)NTr) rQrRrrrrVrrlm_headrr]rr^s rrRCTRLLMHeadModel.__init__sG  $V,yy0A0AM  r!cUR$rPrrs rget_output_embeddings%CTRLLMHeadModel.get_output_embeddingss ||r!cXlgrPrrs rset_output_embeddings%CTRLLMHeadModel.set_output_embeddingss% r!rrrBrrrCrlabelsrrrsrrrc  U bU OURRn URUUUUUUUU U U U S9 nUSnURU5nSnUb*UR"UU4SURR 0U D6nU (dU4USS-nUbU4U-$U$[ UUURURURS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N rrBrrrCrrrrsrrrrr)losslogitsrrr) rrrr loss_functionrr rrr)r]rrrBrrrCrrrrrsrrrtransformer_outputsr lm_logitsr!rJs rr{CTRLLMHeadModel.forwards v&1%# UHn[U4SjU55v M g7f)c3x># UH/oRSTRUR55v M1 g7f)rN) index_selectr*r)r past_stater+s rr;CTRLLMHeadModel._reorder_cache...Bs1j_iQ[))!X[[9J9J-KLL_is7:Nr)rrqr+s rr1CTRLLMHeadModel._reorder_cache..As' -  j_ij j j-s"%r2)rr+s `r_reorder_cacheCTRLLMHeadModel._reorder_cache8s -   r!)rr NNNNNNNNNNNNNN)r~rrr_tied_weights_keysrRrrrrrr rr r rrr r{r) staticmethodr4rrrs@rrrs++&15EI6:59371559-1$(,0/3&*a E,,-a "%e.?.?(@"ABa !!2!23 a !!1!12 a u//0 a E--.a   1 12a ))*a D>a $D>a 'tna d^a  uU\\"$:: ;a a Fd$  uU\\23  ?D||  uU\\" #    r!ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c^\rSrSrU4Sjr\SS\\RS\\ \ \RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\ \ \R\44Sjj5rSrU=r$)CTRLForSequenceClassificationiGc>[TU]U5 URUl[U5Ul[ R "URURSS9UlUR5 g)NFr) rQrR num_labelsrrrrVr classifierrrs rrR&CTRLForSequenceClassification.__init__SsR   ++$V,))FMM4??O r!rrrBrrrCrrrrrsrrrc U bU OURRn URUUUUUUUU U U U S9 n U SnURU5nUbURSSunnOURSSunnURR cUS:wa [ S5eURR cSnOUbXRR :gRUR[R5n[R"URSUR[RS9nUU-RS5nO.Sn[RURR S 35 U[R"UURS 9U4nSnUGbURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOU"UU5nOURR"S :Xa=[15nU"UR3SUR$5UR3S55nO-URR"S :Xa[55nU"UU5nU (dU4U SS-nUbU4U-$U$[7UUU R8U R:S9$)a" input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```Nr rrrz=Cannot handle batch sizes > 1 if no padding token is defined.r%)rr$z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)r!r"rr)rrrr>r9 pad_token_idrr*rrint32r(argmaxlogger warning_oncer^r~ problem_typer=r$rrTr squeezerrrrrr)r]rrrBrrrCrrrrrsrrr$rr"rlsequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsr!loss_fctrJs rr{%CTRLForSequenceClassification.forward\sP&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E' -;;*55   r!)r>r=rr6)r~rrrrRrrrr rr r rrrr{rrrs@rr;r;GsW15EI6:59371559-1$(,0/3&*p E,,-p "%e.?.?(@"ABp !!2!23 p !!1!12 p u//0 p E--.p   1 12p ))*p D>p $D>p 'tnp d^p  uU\\"$<< =p p r!r;)r;rrrr7)-__doc__typingrrrnumpyr:rrtorch.nnrrr generationr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_ctrlr get_loggerr~rGr r4rKModulerMrrrrrr;__all__rr!rr_s )) AA)ii-YY+   H % %4LL^`&299&R*/***@ #@ @ F S )?S  S l { $7{  { | cr!