o ZhP~@s^dZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZdd lmZdd lmZmZmZdd lmZmZd d lmZeeZ ddZ!ddZ"d&ddZ#Gdddej$Z%ddZ&Gdddej$Z'eGdddeZ(eGddde(Z)eddGd d!d!e(eZ*ed"dGd#d$d$e(Z+gd%Z,dS)'zPyTorch CTRL model.)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcCs$dtdd|d|}||S)Nri')torchpow)posi d_model_sizeZ angle_ratesrU/var/www/auris/lib/python3.10/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defn'srcCsttj|tjd|dtj|tjd|d|}t|dddddf}t|dddddf}tj||gdd}|S)Ndtyperrrdim) rrarangeZint64to unsqueezesincoscat)positionrr Z angle_radsZsinesZcosines pos_encodingrrrpositional_encoding,sr,c Cst||dddd}|jd}|t|}|dur7|d|d} } ||| | | d| fd7}|dur?||}tj|dd} |durN| |} t| |} | | fS) Nrrr rr!gr")rmatmulpermuteshapenpsqrtsizeZsoftmax) qkvmaskattention_mask head_maskZ matmul_qkZdkZscaled_attention_logitsndnsZattention_weightsoutputrrrscaled_dot_product_attention;s   r=cs@eZdZfddZddZddZ     d d d ZZS) MultiHeadAttentioncsjt||_||_t||j|_t|||_t|||_ t|||_ t|||_ t |_ dSN)super__init__ num_headsrintdepthrLinearWqWkWvdenseset pruned_heads)selfrrB __class__rrrAVs  zMultiHeadAttention.__init__cCs|j|j}t|dkrdSt||j||j\}}t|j||_t|j||_t|j||_t|j |dd|_ |jt||_||j|_|j ||_dS)Nrrr") rrBlenrrKrrFrGrHrIunion)rLheadsZattention_head_sizeindexrrr prune_headsds   zMultiHeadAttention.prune_headscCs"||d|j|j}|gdS)Nr!rrrr )reshaperBrDr/)rLx batch_sizerrrsplit_into_headsusz#MultiHeadAttention.split_into_headsNFc Cs|jd} ||}||}||}||| }||| }||| }|durE|d|d} } tj| |fdd}tj| |fdd}|durQt||f} nd} t||||||}|d gd}|d}| | d|j }| |}|| f}| r||f}|S) Nrrr-r"Tr?rTr!) r0rFrGrHrXrr)stackr=r/rUrrI)rLr6r5r4r7 layer_pastr8r9 use_cacheoutput_attentionsrWZpast_keyZ past_valuepresentr<Zscaled_attentionZattnZoriginal_size_attentionoutputsrrrforwardys.        zMultiHeadAttention.forwardNNNFF)__name__ __module__ __qualname__rArSrXr_ __classcell__rrrMrr>Us  r>cCs"tt||tt||Sr?)rZ SequentialrEZReLU)rdffrrrpoint_wise_feed_forward_networks"rfcs*eZdZdfdd Z d ddZZS) EncoderLayer皙?cs^tt|||_t|||_tj|dd|_tj|dd|_ t ||_ t ||_ dS)Ngư>Zeps) r@rAr>multi_head_attentionrfffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)rLrrBreZraterMrrrAs    zEncoderLayer.__init__NFc Cs|||}|j|||||||||d } | d} || } || } || } || } || } | | } | f| dd}|S)NrZr8r9r[r\rr)rmrjrprnrkrq)rLrVr7rZr8r9r[r\ZnormedZ attn_outputsZ attn_outputZout1Zout2Z ffn_outputr^rrrr_s*     zEncoderLayer.forward)rhr`)rarbrcrAr_rdrrrMrrgs rgc@seZdZeZdZddZdS)CTRLPreTrainedModel transformercCst|tjtfr"|jjjd|jjd|j dur |j j dSdSt|tj rE|jjjd|jjd|j durC|jj|j  dSdSt|tj rZ|j j |jjddSdS)zInitialize the weights.g)meanZstdN?) isinstancerrErweightdataZnormal_configZinitializer_rangebiasZzero_ EmbeddingZ padding_idxrlZfill_)rLmodulerrr _init_weightss     z!CTRLPreTrainedModel._init_weightsN)rarbrcrZ config_classZbase_model_prefixr~rrrrrss rscseZdZfddZddZddZddZe dd ee j d ee e e j d ee j d ee j dee j dee j dee j dee dee dee dee dee e jeffddZZS) CTRLModelcstj|_j|_tj|jtj |_ t j j|_t j|_t fddtjD|_t jjjd|_|dS)Ncs"g|] }tjjjjqSr)rgn_embdZn_headreZ resid_pdrop).0_rzrr s"z&CTRLModel.__init__..ri)r@rArrn_layerZ num_layersr,Z n_positionsrfloatr+rr| vocab_sizewroZ embd_pdropdropoutZ ModuleListrangehrlZlayer_norm_epsilon layernorm post_initrLrzrMrrrAs  zCTRLModel.__init__cC|jSr?rrLrrrget_input_embeddingszCTRLModel.get_input_embeddingscC ||_dSr?rrLZnew_embeddingsrrrset_input_embeddings zCTRLModel.set_input_embeddingscCs(|D] \}}|j|j|qdS)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrjrS)rLZheads_to_prunelayerrQrrr _prune_headsszCTRLModel._prune_headsN input_idspast_key_valuesr8token_type_ids position_idsr9 inputs_embedsr[r\output_hidden_states return_dictreturnc  KsZ| dur| n|jj} |dur|n|jj}| dur| n|jj} | dur$| n|jj} |dur4|dur4td|durP||||} |d| d}|j d}n|durb|dd} |j d}ntd|durm|j n|j }|durd}t dgt |j }n |ddd}|durtj|| d|tj|d}|d}|dur|dkrtd||d}|d d }|j|jd }d |t|jj}|||jj}|dur|d| d}||}|t|j9}nd}|dur||}| d}tt||||d |}|t|j9}|j||_|j|ddf}|||}||}|r7d nd}| r>d nd}| rEd nd}t t!|j |D]8\}\}}| r]||f}||||||||| d}|dd \}}|dur|||f}| r||d f7}qO|"|}| r||f}| st dd||||fDSt#||||dS)aE input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer!rz5You have to specify either input_ids or inputs_embedsr-)r devicez$batch_size has to be defined and > 0rrrrvrrrTcss|] }|dur|VqdSr?r)rr6rrr sz$CTRLModel.forward..)Zlast_hidden_stater hidden_states attentions)$rzr\r[ruse_return_dict ValueErrorZ%warn_if_padding_and_no_attention_maskr3viewr0rtuplerOrrr$longr&r%r ZfinfominZ get_head_maskrrr1r2rZtriuZonesr+r enumerateziprr )rLrrr8rrr9rr[r\rrkwargsZ input_shaperWr past_lengthZtoken_type_embedsZseq_lenr7Z pos_embedsrZpresentsZall_hidden_statesZall_attentionsrrrZr^r]rrrr_ s0       "       zCTRLModel.forward) NNNNNNNNNNN)rarbrcrArrrrrr LongTensorr FloatTensorboolrTensorr r_rdrrrMrrsT     rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). )Z custom_introcseZdZdgZfddZddZddZe            dd ee j d ee e e j d ee j d ee j d ee j dee j dee j dee j dee dee dee dee dee e jeffddZdddZed e e e jde jde e e jfddZZS)CTRLLMHeadModelzlm_head.weightcs8t|t||_tj|j|jdd|_| dS)NTr{) r@rArrtrrErrlm_headrrrMrrrAs   zCTRLLMHeadModel.__init__cCrr?rrrrrget_output_embeddingsrz%CTRLLMHeadModel.get_output_embeddingscCrr?rrrrrset_output_embeddingsrz%CTRLLMHeadModel.set_output_embeddingsNrrr8rrr9rlabelsr[r\rrrc  Ks| dur| n|jj} |j|||||||| | | | d }|d}||}d}|dur7|j||fd|jji| }| sM|f|dd}|durK|f|S|St|||j|j|j dS)a  input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N rr8rrr9rr[r\rrrrr)losslogitsrrr) rzrrtrZ loss_functionrr rrr)rLrrr8rrr9rrr[r\rrrtransformer_outputsrZ lm_logitsrr<rrrr_sH; zCTRLLMHeadModel.forwardcKs\|dur(|ddjd}|jd|kr|}n|jdd}|dd|df}|||dS)Nrrr)rrr[)r0)rLrrr[rrZremove_prefix_lengthrrrprepare_inputs_for_generation&s z-CTRLLMHeadModel.prepare_inputs_for_generationbeam_idxcstfdd|DS)a This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. c3s&|]}tfdd|DVqdS)c3s$|] }|d|jVqdS)rN)Z index_selectr%r)rZ past_staterrrrBs"z;CTRLLMHeadModel._reorder_cache...Nr)rrZrrrrAs  z1CTRLLMHeadModel._reorder_cache..r)rrrrr_reorder_cache8s zCTRLLMHeadModel._reorder_cache NNNNNNNNNNNNNN)rarbrcZ_tied_weights_keysrArrrrrrrrrrrr r_r staticmethodrrdrrrMrrsl       c  ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZfddZe            ddeejdeeeej deej deejdeejd eej d eej d eejd ee d ee dee dee de eej e ffddZZS)CTRLForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dS)NFr) r@rA num_labelsrrtrrEr classifierrrrMrrrASs   z&CTRLForSequenceClassification.__init__Nrrr8rrr9rrr[r\rrrc  CsJ| dur| n|jj} |j|||||||| | | | d } | d}||}|dur1|jdd\}}n |jdd\}}|jjdurH|dkrHtd|jjdurQd}n1|durv||jjk|jt j }t j |jd|jt j d}|| d}n d}t |jjd |t j ||jd |f}d}|dur|jjdur|jdkrd |j_n|jdkr|jt jks|jt jkrd |j_nd |j_|jjd krt}|jdkr|||}n,|||}n&|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dur|f|S|St||| j| jdS)a2 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```Nrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r!)rr z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)rZ regressionZsingle_label_classificationZmulti_label_classification)rrrr)rzrrtrr0Z pad_token_idrr%rrZint32r$ZargmaxloggerZ warning_oncerNraZ problem_typerr rrCrZsqueezerrrr rr)rLrrr8rrr9rrr[r\rrrrrrWZsequence_lengthZlast_non_pad_tokenZ non_pad_maskZ token_indicesZ pooled_logitsrZloss_fctr<rrrr_\sxh       "      z%CTRLForSequenceClassification.forwardr)rarbrcrArrrrrrrrrr r_rdrrrMrrGsT       r)rrrrsr)-__doc__typingrrrnumpyr1rrZtorch.nnrrrZ generationr Zmodeling_outputsr r r Zmodeling_utilsrZ pytorch_utilsrrrutilsrrZconfiguration_ctrlrZ get_loggerrarrr,r=Moduler>rfrgrsrrr__all__rrrrsF      O)D  ?