o Zh@s0ddlmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZd d lmZmZmZmZd d lmZd dlmZd dlmZeeZ dZ!GdddeZ"Gddde Z#GdddeZ$Gddde eZ%GdddeZ&GdddeZ'GdddeZ(gdZ)dS) )OptionalTupleUnionN)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack) LossKwargslogging) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-Chat-0414c@ eZdZdS)Glm4MLPN__name__ __module__ __qualname__rrT/var/www/auris/lib/python3.10/site-packages/transformers/models/glm4/modular_glm4.pyr%rcseZdZdedeffdd Z       ddejdeejd eej d ee d ee d ee d eej dee ejejfde ede ejee ejejfffddZZS)Glm4DecoderLayerconfig layer_idxcsvt|j|_t||d|_t||_t|j|jd|_ t|j|jd|_ t|j|jd|_ t|j|jd|_ dS)N)r r!)Zeps) super__init__Z hidden_size Glm4Attention self_attnrmlprZ rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr r! __class__rrr#*s  zGlm4DecoderLayer.__init__NF hidden_statesattention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionposition_embeddingskwargsreturnc Ks|} ||}|jd||||||||d| \}} ||}| |}|} ||}||}||}| |}|f} |rB| | f7} | S)N)r.r/r0r1r2r3r4r5r)r'r%r)r(r&r*) r+r.r/r0r1r2r3r4r5r6ZresidualZself_attn_weightsZoutputsrrrforward5s2      zGlm4DecoderLayer.forward)NNNFFNN)rrrrintr#torchZTensorrZ LongTensorrboolrr rZ FloatTensorr8 __classcell__rrr,rr)s<   rc@r)r$Nrrrrrr$crr$c@r)KwargsForCausalLMNrrrrrr=gs r=cs2eZdZdeedeeefffdd ZZ S)Glm4ForCausalLM super_kwargsr7c stjdi|S)ar labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-Chat-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-Chat-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```Nr)r"r8)r+r?r,rrr8kszGlm4ForCausalLM.forward) rrrr r=rrr r8r<rrr,rr>js  r>c@r)Glm4ForSequenceClassificationNrrrrrr@rr@c@r)Glm4ForTokenClassificationNrrrrrrArrA)ZGlm4PreTrainedModelZ Glm4Modelr>r@rA)*typingrrrZtorch.utils.checkpointr:Z cache_utilsrZmodeling_flash_attention_utilsrZmodeling_layersrZmodeling_outputsr Zprocessing_utilsr utilsr r Zglm.modeling_glmrrrrZphi3.modeling_phi3rZconfiguration_glm4rZ modeling_glm4rZ get_loggerrloggerZ_CHECKPOINT_FOR_DOCrrr$r=r>r@rA__all__rrrrs,         :