o Zh)@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddl mZddlmZmZmZddlmZmZmZdd lmZdd lmZmZmZdd lmZdd lm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&ddl'm(Z(e&)e*Z+ddZ,e-eeedZ.Gdddej/Z0Gdddej/Z1Gdddej/Z2Gdddej/Z3e%GdddeZ4eGddde$Z5e%Gdd d e4Z6e%d!d"Gd#d$d$e4eZ7e%d%d"Gd&d'd'e4Z8e%d(d"Gd)d*d*e4Z9gd+Z:dS),zPyTorch OpenAI GPT model.N) dataclass)AnyCallableDictOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu_newget_activationsilu)GenerationMixin)BaseModelOutputCausalLMOutputSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_conv1d_layer) ModelOutputauto_docstringlogging)OpenAIGPTConfigcsddl}ddldvrtjtdtdddd }t |}Wdn1s3wYtd ddd }t |}Wdn1sQwY fd d |D}fd d t d D} | d|dd} dd t| |D} dd | D} |jjj| djkrtd|jjjd| dj|jjj| djkrtd|jjjd| djt| d|jj_t| d|jj_|d| d| dt|| D]\} } | dd} | dddkrtd| d| dd} | d} |} | D]S} |d| r#| d| }n| g}|ddkr3t| d} n!|dd kr@t| d!} n|dd"krMt| d} nt| |d} t|d#kret|d}| |} q| j| jkr{td$| jd%| jd&td'| t| | _q|S)(zGLoad tf pre-trained weights in a pytorch model (from NumPy arrays here)rNz.ckptzLoading weights from z/parameters_names.jsonrzutf-8)encodingz/params_shapes.jsoncsg|]}|qS)prod).0shape)npr Y/var/www/auris/lib/python3.10/site-packages/transformers/models/openai/modeling_openai.py ;sz1load_tf_weights_in_openai_gpt..cs"g|] }d|dqS)z/params_z.npy)load)r"nr$openai_checkpoint_folder_pathr r%r&<s" cSsg|] \}}||qSr )Zreshape)r"paramr#r r r%r&>scSsg|]}|qSr )squeeze)r"Zarrr r r%r&Csrztokens_embed.weight.shape: z% does not match init_param[1].shape: zpositions_embed.weight.shape: z% does not match init_param[0].shape: z:0zLayer z does not end with :0/z [A-Za-z]+\d+z(\d+)gweightbbiaswzPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpyospathdirnameloggerinfoopenjsonr'ZcumsumrangesplitZ concatenatezip tokens_embedr3r# ValueErrorpositions_embedtorchZ from_numpydatapop fullmatchgetattrlenint)modelconfigr*r8Z names_handlenamesZ shapes_handleZshapesoffsetsZ init_paramsnamearrayZpointerZm_nameZ scope_namesnumr r)r%load_tf_weights_in_openai_gpt,sx             rU)ZrelurZgeluZswishcsLeZdZdfdd ZddZdddZd d Zdd d Zdd dZZ S) AttentionFcst|}||jdkrtd|d|j|jdtt||dd||dd|j|_||_ ||_ t |d||_ t |||_ t|j|_t|j|_t|_dS) NrzAttention n_state shape: z$ must be divisible by config.n_head r5rF persistentr )super__init__n_headrEregister_bufferrGZtrilZonesview split_sizescalerc_attnc_projr DropoutZ attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)selfnx n_positionsrOr_n_state __class__r r%rZ~s"   zAttention.__init__cCst|dkrdSt||j|j|j|j\}}t|||j|d|jg}t|j|dd|_t|j |dd|_ |j|j|jt||_|jt||_|j ||_dS)Nrr7rdim) rLrr[r^rgrGcatrr`raunion)rhheadsindexZ index_attnr r r% prune_headss  zAttention.prune_headsNc Cst||}|jr|t|d}|jddddd|dd|df}||dd|}|dur<||}tjj |dd}| |}|durQ||}t||g} |r_| || S)Nr,r0grrn) rGmatmulr_mathsqrtsizer5r Z functionalZsoftmaxrcappend) rhqkvattention_mask head_maskoutput_attentionsr6r4outputsr r r%_attns .  zAttention._attncCsD|dddd}|dd|d|df}|j|S)Nrr7rr r0r,)permute contiguousrxr])rhx new_x_shaper r r% merge_headss& zAttention.merge_headscCsT|dd|j|d|jf}|j|}|r"|ddddS|ddddS)Nr,rr7r r)rxr[r]r)rhrr{rr r r% split_headss & zAttention.split_headsc Cs||}|j|jdd\}}}||}|j|dd}||}|||||||}|d} || } || } || } | g|dd} | S)Nr7rnT)r{rr)r`rBr^rrrrare) rhrr}r~rquerykeyvalue attn_outputsarr r r%forwards      zAttention.forwardFNNF) __name__ __module__ __qualname__rZrtrrrr __classcell__r r rlr%rV}s  rVcs$eZdZfddZddZZS)MLPcsFt|j}t|||_t|||_t|j|_t |j |_ dSN) rYrZn_embdrc_fcraACT_FNSZafnactr rbrddropout)rhrkrOrirlr r%rZs    z MLP.__init__cCs$|||}||}||Sr)rrrar)rhrhZh2r r r%rs  z MLP.forwardrrrrZrrr r rlr%rs rcs(eZdZdfdd ZdddZZS) BlockFcsXt|j}t|||||_tj||jd|_t d|||_ tj||jd|_ dS)N)Zeps) rYrZrrVattnr LayerNormZlayer_norm_epsilonln_1rmlpln_2)rhrjrOr_rirlr r%rZs zBlock.__init__Nc CsV|j||||d}|d}|||}||}|||} | g|dd} | S)N)r}r~rrr)rrrr) rhrr}r~rrrr(mrrr r r%rs z Block.forwardrrrr r rlr%rsrcsJeZdZdZdeffdd Z d dejdeej dejfd d Z Z S) OpenAIGPTSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`OpenAIGPTConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rOcstt|dd|_|jdkrtt|_t|dr<|j r