o ZhD@sdZddlmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z dd lmZdd lmZd d lmZd d lmZmZmZmZmZmZmZddlmZeeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&gdZ'dS)zPyTorch BitNet model.)CallableOptionalTupleN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigc@ eZdZdS) BitNetRMSNormN__name__ __module__ __qualname__rrX/var/www/auris/lib/python3.10/site-packages/transformers/models/bitnet/modular_bitnet.pyr*rcs*eZdZdeffdd ZddZZS) BitNetMLPconfigcs"t|t|j|jd|_dSN)Zeps)super__init__rZintermediate_size rms_norm_eps ffn_sub_norm)selfr! __class__rrr$/s zBitNetMLP.__init__c Cs*|||||||}|S)N) down_projr&Zact_fnZ gate_projZup_proj)r'xr*rrrforward3s&zBitNetMLP.forward)rrrrr$r, __classcell__rrr(rr .sr cseZdZdedeffdd Z  ddejdeejejfde ejd e e d e ej d e e d eeje eje eejffd dZZS)BitNetAttentionr! layer_idxcs$t||t|j|jd|_dSr")r#r$rZ hidden_sizer% attn_sub_norm)r'r!r/r(rrr$9szBitNetAttention.__init__N hidden_statesposition_embeddingsattention_maskpast_key_valuecache_positionkwargsreturncKsR|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |durW| | |d}|| | |j |\} } t }|j j dkrw|j j dkrq| ddrqtd nt|j j }||| | | |f|jsd n|j|jd |\}}|jg|dR}||}||}||fS) Nrr )sincosr5eagerZsdpaZoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.g)Zdropoutscaling)shapeZhead_dimZq_projviewZ transposeZk_projZv_projrupdater/rr!Z_attn_implementationgetloggerZ warning_oncer ZtrainingZattention_dropoutr<Zreshape contiguousr0Zo_proj)r'r1r2r3r4r5r6Z input_shapeZ hidden_shapeZ query_statesZ key_statesZ value_statesr:r9Z cache_kwargsZattention_interfaceZ attn_outputZ attn_weightsrrrr,=sB       zBitNetAttention.forward)NN)rrrrintr$torchZTensorrrrZ LongTensorr rr,r-rrr(rr.8s& r.c@r)BitNetDecoderLayerNrrrrrrEqrrEc@r) BitNetModelNrrrrrrFurrFcs0eZdZdgZdZdZdeffdd ZZS)BitNetForCausalLMzlm_head.weightNr7c stjdi|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```Nr)r#r,)r'Z super_kwargsr(rrr,~szBitNetForCausalLM.forward) rrrZ_tied_weights_keysZ_tp_planZ_pp_planrr,r-rrr(rrGysrG)rGrFZBitNetPreTrainedModel)(__doc__typingrrrrDZ cache_utilsrZmodeling_flash_attention_utilsrZmodeling_outputsrZmodeling_utilsr Zprocessing_utilsr utilsr Zgemma.modeling_gemmar Zllama.modeling_llamarrrrrrrZconfiguration_bitnetrZ get_loggerrrArr r.rErFrG__all__rrrrs(        $  9 "