eThf=BSSKJrJrJr \"5(aSSKJr \"5(aSSKrSSKJr SSK Js J r \R"\ 5rSrS\R S\R 4Sjr\R$S \R S \R&S\R 4S j5r"S S \R*5r"SS\R.R05r"SS\R.R05r"SS\R65rSSjrSSjrg))is_accelerate_availableis_torch_availablelogging)init_empty_weightsNquantized_weightsreturncURnUS[-S- [-n[U5S:XaU4nO U/USSQ7nUS- n[R"X0R [R S9nUR[R 5n[[USU-S-5n[U5H0nXr-n[X-US5n USX- ===XXU SU---sss&M2 U$)a Packs a tensor of quantized weights into a compact format using 2 bits per value. Parameters: ----------- quantized_weights : torch.Tensor A tensor containing ternary quantized weights with values in {-1, 0, 1}. These values are adjusted to {0, 1, 2} before being packed. Returns: -------- torch.Tensor A packed tensor where each element stores 4 quantized values (each using 2 bits) in an 8-bit format. rNdevicedtyper) shapeVALUES_PER_ITEMlentorchzerosruint8tominrange) r original_shaperow_dimpacked_tensor_shapepackedunpackeditistartends X/var/www/auris/envauris/lib/python3.13/site-packages/transformers/integrations/bitnet.py pack_weightsr#s ',,Na ?2Q6?JG >a&j&<);< [[,5M5MUZU`U` aF ##EKK0H _~a0G;q@ AB 2Y %/>!#45#+8##6!a%#?? MrrctURn[U5S:XaUS[-nU4nOUS[-nU/USSQ7n[R"X@R [R S9n[[5H%nXbS-nXrS-nSSU--n X -SU-- XWU&M' URU5S- $)u Unpacks a tensor of quantized weights that were stored in a packed format using 2 bits per value. Parameters: ----------- packed : torch.Tensor A tensor containing packed weights where each element represents 4 quantized values (using 2 bits per value). dtype : torch.dtype The dtype of the returned Tensor Returns: -------- torch.Tensor A tensor of unpacked weights, where each value is converted from its packed 2-bit representation. Example: -------- packed = torch.tensor([[0b10100001, 0b00011000], [0b10010000, 0b00001010]], dtype=torch.uint8) # Unpack the values unpacked = unpack_weights(packed) # Resulting unpacked tensor print(unpacked) # Output: tensor([[ 0, -1], [-1, 1], [-1, 1], [-1, 1], [ 1, 0], [ 0, -1], [ 1, -1], [ 1, -1]]) Explanation of the example: --------------------------- Let's take the first value for example 0b10100001, we we will only focus on the first column, because every element is unpacked across the first dimension - First 2 bits: `01` → 0 at [0][0] - Second 2 bits: `00` → -1 at [0][2] - Third 2 bits: `10` → 1 at [0][4] - Fourth 2 bits: `10` → 1 at [0][6] the second value of the same row (0b10010000) will give the values for [0][1], [0][3], [0][5], [0][7] We subtract 1 because during the packing process, it's easier to work with values like 0, 1, and 2. To make this possible, we add 1 to the original ternary weights (which are typically -1, 0, and 1) when packing them. When unpacking, we reverse this by subtracting 1 to restore the original ternary values. r rNr r) rrrrrrrrr) rr packed_shapeoriginal_row_dimunpacked_shaperrr r!masks r"unpack_weightsr+9sb<\!"-=>{{>--u{{SH ? #O#1o%QU|%}!a%8s $ ;;u  !!r$c ^\rSrSrS S\S\S\S\S\4 U4Sjjjr\RS Sj5r \RS 5r S r S r U=r$) BitLinear~ in_features out_featuresbias use_rms_norm rms_norm_epsc >[T U]5 XPlXlX lUR S[ R"U[-U4[ RUS95 UR S[ R"SUUS95 U(a&UR S[ R"X%US95 OSUl SUl U(aSSK Jn U"XS9Ul gg) Nweightrr weight_scaler r1r LlamaRMSNormeps)super__init__rr/r0register_bufferrrrronesr1rms_normmodels.llama.modeling_llamar9) selfr/r0r1rrr2r3r9 __class__s r"r=BitLinear.__init__s  &(   KK0+>kk     JJ     lY_)` aDI  B(GDM r$cSUS- -*nSUS- -S- nXAR5RSSS9RRSS9- nX-R 5RX45nUR [ R5U4$)a Activation function : Performs symmetric, per-token quantization on the input activations. Parameters: ----------- x : torch.Tensor Input activations to be quantized. num_bits : int, optional (default=8) Number of bits to use for quantization, determining the quantization range. Returns: -------- result : torch.Tensor Quantized activation tensor, with values mapped to an `int8` range. scale : torch.Tensor The per-channel scaling factors used to quantize the tensor. rr Tdimkeepdimh㈵>r)absmaxvaluesclamproundrrint8)rBinputnum_bitsQnQpscaleresults r"activation_quantBitLinear.activation_quants$X\" # 8a< 1 $YY[__T_:AAGGDGQQ-&&(..r6yy$e++r$cXU-- nU$N)rBrR input_scaler7outs r"post_quant_processBitLinear.post_quant_processs\12 r$cURbURU5nURn[X RS9nUR U5upE[ R "URUR5U5nURX`RU5nURb-X`RRSS5RU5- nU$)Nrr rF) r@r5r+rrXFlinearrr_r7r1view expand_as)rBrRww_quant input_quantr]ys r"forwardBitLinear.forwards == $MM%(E KK **5#'#8#8#? HH[^^DJJ/ 9  # #A'8'8+ F 99 2&003 3Ar$)r1rr/r0r@)NNFư>))__name__ __module__ __qualname____firstlineno__intboolfloatr=rcompilerXr_rk__static_attributes__ __classcell__rCs@r"r-r-~s ""(H(H(H (H(H(H(HT ]],,. ]]  r$r-cV\rSrSrSr\\RS55r\S5r Sr g) WeightQuantz Implements a custom autograd function for weight quantization. This performs ternary quantization (-1, 0, 1) based on scaling by the mean absolute value of the weights. It uses the Straight-Through Estimator (STE) for the backward pass. cURnUR5nSUR5R5R SS9- nX-R 5R SS5U- nURU5$)Ng?rJrKrFr )rrurLmeanclamp_rPrOr)ctxr5rrVs r"rkWeightQuant.forwardsp fjjl'')00T0::.'')//A6>yyr$c&UR5nU$r[cloner grad_output grad_inputs r"backwardWeightQuant.backward &&( r$r\N rorprqrr__doc__ staticmethodrrvrkrrwr\r$r"r{r{s; ]]  r$r{cV\rSrSrSr\\RS55r\S5r Sr g)ActQuanta$ Implements a custom autograd function for activation quantization. This performs symmetric 8-bit quantization (to the range [-128, 127]) based on the maximum absolute value along the last dimension (per-token/row scaling). It uses the Straight-Through Estimator (STE) for the backward pass. cURnUR5nSUR5RSSS9RR SS9- nX-R 5RSS5U- nURU5$)NrFTrGrJrKi) rrurLrMrNrrPrOr)r activationrrVs r"rkActQuant.forwards  %%' jnn&**r4*@GGNNSWNXX (//177cBUJ }}U##r$c&UR5nU$r[rrs r"rActQuant.backwardrr$r\Nrr\r$r"rrs; ]]$$r$rc `^\rSrSrS S\S\S\S\S\S\4 U4SjjjrS rS r S r U=r $) AutoBitLinearir/r0r1 online_quantr2r3c >[T U]XU5 X`lSUlU(aSSKJn U "XS9UlU(dBUR S[R"SUUS95 URUR5 gg)Nrr8r:r7r r6) r<r=rr@rAr9r>rr?"_register_load_state_dict_pre_hook load_hook) rBr/r0r1rrrr2r3r9rCs r"r=AutoBitLinear.__init__su D9(  B(GDM   !   3 3DNN Cr$cUS-U;aQXS-RURR:wa([XS-URRS9XS-'U$)Nr5rb)rr5r+)rB state_dictprefixargskwargss r"rAutoBitLinear.load_hook"sb X * ,X# UHoSRT5;v M g7f).N)join).0keycurrent_key_names r" ._replace_with_bitnet_linear..Ts W@V#((#344@Vs # autobitlinearonline)r/r0r1rrrr2r3offlineFrm)r/r0r1rrr2r3Tr)modules_to_not_convertrquantization_confighas_been_replacedrF)named_childrenappendanyr isinstancennLinearr/r0 linear_classrr1r5rrquantization_moder2r3_modulesrequires_grad_r-rlistchildren_replace_with_bitnet_linearpop) modelrrrr pre_quantizednamemoduler/r0_s ` r"rr=s,,.   #! %W@VWWW#%fbii00T5W"("4"4K#)#6#6L*/B/O/OSb/b/<(3)5!'D!8#)==#7#7"(--"5"5*=*O*OS[*[)<)I)I)<)I)I 0t,/@@IM!NN40??F/8(3)5!'D!8#)==#7#7"(--"5"5M`)<)I)IfkM`)<)I)Ifj0t,t,;;EB(,%9&< tFOO%& '! +#>'=!1$7"3 $ A R ]/^  ##Q&%s F H;; I cUcS/OUnU(a(URbURUR5 [[U55n[ UUUUUS9upU(d[ R S5 U$)a) A helper function to replace all `torch.nn.Linear` modules by `BitLinear158` modules`. The function will be run recursively and replace all `torch.nn.Linear` modules except for the `lm_head` that should be kept as a `torch.nn.Linear` module. The replacement is done under `init_empty_weights` context manager so no CPU/GPU memory is required to run this function. Each weight will be quantized along the channel. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[`str`]`, *optional*, defaults to `["lm_head"]`): Names of the modules to not convert in `BitLinear`. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[`str`]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert (for instances modules that are offloaded to `cpu` or `disk`). lm_head)rzYou are loading your model using bitnet but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.)rextendrsetrloggerwarning)rrrrrrs r"replace_with_bitnet_linearrs2-C,Ji[Pf2IIU%%&9&P&PQ!#&<"=>: #  E   Lr$)NNNFF)NNNF)utilsrrr acceleraterrtorch.nnrtorch.nn.functional functionalrc get_loggerrorrTensorr#rvrr+Moduler-autogradFunctionr{rrrrrr\r$r"rsHH-##   H %#ELL#U\\#LA"5<<A" A" A"A"HT Tn%..)).u~~&&.7BII7x  @$J  ,r$