o Zh0@sddlmZddlmZmZmZmZerddlZddlmZer(ddl m Z er/ddl Z e eZGdddejjZGd d d ejZ     dd d Z     dddZdS))ACT2FN)is_accelerate_availableis_fbgemm_gpu_availableis_torch_availableloggingN)nn)init_empty_weightsc*eZdZejffdd ZddZZS)FbgemmFp8Linearcst|||||_||_tjtj||ftjd|_ tjtj|df|d|_ |j dtjdgtj ddd|rKtjtj|j|d|_ dSd|_ dS)Ndtypeinput_scale_ubF persistent)super__init__ in_features out_featurestorchr Parameterzeros float8_e4m3fnweight weight_scaleregister_bufferfloatbias)selfrrrZ weight_dtype __class__S/var/www/auris/lib/python3.10/site-packages/transformers/integrations/fbgemm_fp8.pyr!s zFbgemmFp8Linear.__init__cCsg|jdddR}tjjj|d|jd|jd\}}|j tj }tjjj ||j ||dd}|j dur?||j n|}| |j}||}~~|S)N)Zscale_ubTZuse_fast_accum)shaperopsfbgemmquantize_fp8_per_rowview contiguousrrtofloat32f8f8bf16_rowwiserrdevicereshape)rxZ output_shapeZ x_quantizedZx_scaleZweight_scale_float32outputr"r"r#forward/s    zFbgemmFp8Linear.forward__name__ __module__ __qualname__rr-rr3 __classcell__r"r"r r#r sr cr )FbgemmFp8Llama4TextExpertscst|j|_|j|_|j|_|j|_t|j|_ t j t j |j|jd|jft jd|_t j t j |jd|jdft jd|_t j t j |j|j|jft jd|_t j t j |j|jdft jd|_|jdt j dgt jddddS)Nrr rrFr)rrZnum_local_experts num_expertsZintermediate_size hidden_sizeZ expert_dimrZ hidden_actact_fnrrrrr gate_up_projr-gate_up_proj_scale down_projdown_proj_scalerr)rconfigr r r"r#rHs&  "z#FbgemmFp8Llama4TextExperts.__init__c Cs||jd|j}d}t|}t|jD]}||}|d|j}tjj |||j \}}|j j dd} |j tj} tjjj||j |ddd| || |dd| dddd} tjjj||j |dd| d|| |d| ddddd} | || } tjj | ||j \}}|jtj}tjjj||j|dd|||dddd}|||<q||j}|d|jS)z Args: hidden_states (torch.Tensor): (batch_size * token_num, hidden_size) Returns: torch.Tensor: (batch_size * token_num, hidden_size) r$NrrrTr%)r*r:r;rZ empty_likeranger0r'r(r)rr=r&r>r,r-r.Z transposer+r<r@r?r/)rZ hidden_statesZ num_tokensZ next_statesiZ expert_hiddenZexpert_hidden_reshapedZexpert_quantizedZ expert_scaleZsharded_expert_dimZgate_up_proj_scale_float32ZgateupZ activatedZactivated_quantizedZactivated_scaleZdown_proj_scale_float32Z expert_outputr"r"r#r3`sP   z"FbgemmFp8Llama4TextExperts.forwardr4r"r"r r#r9Gsr9Fc sddl}|dur g}|D]\} } || t| tjrp| |vrpd|tfdd|Dsptdd$| j } | j } t | | | j du|j | <d}|j | dWdn1s]wYtj|jgtjd |j | _| jjd kr| |vrd|tfd d|Dstddd||d d d<t|j|j | <Wdn1swYtj|jgtjd |j | _tt| dkrt| |||||||d\} }|dq||fS)z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successful or not. rN.c3$|] }|dvp |kVqdSrENr".0keyZcurrent_key_name_strr"r#  z2_replace_with_fbgemm_fp8_linear..T)Zinclude_buffersFr ZLlama4TextExpertsc3rFrGr"rHrKr"r#rLrMz\d+*z.down_proj_scale)has_been_replaced pre_quantizedrAtp_planr$)reZnamed_childrenappend isinstancerLinearjoinanyr rrr rZ_modulesZrequires_grad_rZtensorZactivation_scale_ubrrr!r5subr9Z text_configlenlistchildren_replace_with_fbgemm_fp8_linearpop)modelmodules_to_not_convertcurrent_key_namequantization_configrOrPrArQrRnamemodulerr_r"rKr#r\sh            r\c Cs`|durdgn|}|jdur||jtt|}t|||||||d\}}|s.td|S)a A helper function to replace all `torch.nn.Linear` modules by `FbgemmFp8Linear` modules. This will enable running your models using high performance fp8 kernel from FBGEMM library. The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no CPU/GPU memory is required to run this function. Each weight will be quantized along the channel. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`): Names of the modules to not convert in `FP8Linear`. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). NZlm_head)rPrArQzYou are loading your model using FP8 quantization but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)r_extendrZsetr\loggerwarning)r^r_r`rarPrArQrOr"r"r#replace_with_fbgemm_fp8_linears$     ri)NNNFFNN)NNNFNN)Z activationsrutilsrrrrrrZ accelerater Zfbgemm_gpu.experimental.gen_aiZ fbgemm_gpuZ get_loggerr5rgrUr Moduler9r\rir"r"r"r#s6    'X O