o Zh%@sddlZddlmZddlZddlmZmZddlmZee Z Gdddej Z Gdd d ej Z Gd d d ej ZGd d d ej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddej ZGdddeZideded d!d"fd#ed$e d%ed&d'ifd(e d)ed*ed+ejd,ed-ed.ed/ejd0ed1ejd2ejd3ejejejejd4ZeeZ d5d6Z!e!d%Z"e!d$Z#e!dZ$e!d#Z%e!d.Z&e!d3Z'e!d-Z(e!d,Z)dS)7N) OrderedDict)Tensornn)loggingc@"eZdZdZdedefddZdS)PytorchGELUTanha A fast C implementation of the tanh approximation of the GeLU activation function. See https://arxiv.org/abs/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. inputreturncCstjj|ddS)Ntanh)Z approximate)r functionalgeluselfr rG/var/www/auris/lib/python3.10/site-packages/transformers/activations.pyforward$szPytorchGELUTanh.forwardN__name__ __module__ __qualname____doc__rrrrrrrsrc@r)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 r r c Cs6d|dttdtj|dt|dS)N??@Hm?g@)torchr mathsqrtpipowrrrrr.s6zNewGELUActivation.forwardNrrrrrr(srcsLeZdZdZd deffdd Zdedefdd Zdedefd d ZZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 Fuse_gelu_pythoncs(t|r |j|_dStjj|_dSN)super__init__ _gelu_pythonactrr r )rr# __class__rrr&:s  zGELUActivation.__init__r r cCs |ddt|tdS)Nrrr)rerfrrrrrrr'As zGELUActivation._gelu_pythoncC ||Sr$r(rrrrrD zGELUActivation.forward)F) rrrrboolr&rr'r __classcell__rrr)rr"2s r"c@r)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r r cCs*d|dt|ddd||S)Nrrg3E?r)rr rrrrrMs*zFastGELUActivation.forwardNrrrrrr1Hr1c@r)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs r r cCs|td|S)NgZd;?)rsigmoidrrrrrVszQuickGELUActivation.forwardNrrrrrr3Qr2r3cs<eZdZdZdedeffdd Zdedefdd ZZS) ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415 minmaxcs8||krtd|d|dt||_||_dS)Nzmin should be < max (got min: z, max: )) ValueErrorr%r&r6r7)rr6r7r)rrr&gs   zClippedGELUActivation.__init__xr cCstt||j|jSr$)rZclipr r6r7)rr:rrrrozClippedGELUActivation.forward) rrrrfloatr&rrr0rrr)rr5Zs r5cs2eZdZdZfddZdedefddZZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cs ttdtj|_dS)N)r%r&rrr precomputed_constantrr)rrr&{s zAccurateGELUActivation.__init__r r c Cs,d|dt|j|dt|dS)Nrrr)rr r?r!rrrrrs,zAccurateGELUActivation.forward)rrrrr&rrr0rrr)rr=ss r=csDeZdZdZfddZdedefddZdedefdd ZZS) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish csttjj|_dSr$)r%r&rr mishr(r@r)rrr&s zMishActivation.__init__r r cCs|ttj|Sr$)rr rr Zsoftplusrrrr _mish_pythonr;zMishActivation._mish_pythoncCr,r$r-rrrrrr.zMishActivation.forward) rrrrr&rrDrr0rrr)rrBs  rBc@r)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. r r cCs|Sr$rrrrrrszLinearActivation.forwardNrrrrrrEr2rEc@seZdZdZdddZdS)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://arxiv.org/abs/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability 绹۞? ^/ ?cCs*|||td}ddt|S)Nrrr)divrrrr+)rr musigmarrrrszLaplaceActivation.forwardN)rGrHrrrrrrrrrrFsrFc@seZdZdZddZdS)ReLUSquaredActivationzX Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2 cCstj|}t|}|Sr$)rr relurZsquare)rr Z relu_appliedZsquaredrrrrs  zReLUSquaredActivation.forwardNrLrrrrrMs rMcseZdZfddZZS)ClassInstantiercs4t|}t|tr |n|if\}}|di|S)Nr)r% __getitem__ isinstancetuple)rkeycontentclskwargsr)rrrPs zClassInstantier.__getitem__)rrrrPr0rrr)rrOsrOr Zgelu_10i )r6r7 gelu_fastgelu_new gelu_pythonr#TZgelu_pytorch_tanhZ gelu_accurateZlaplaceZ leaky_reluZlinearrC quick_gelurNZrelu2Zrelu6r4silu)Zswishr ZprelucCs,|tvrt|Std|dtt)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)Zactivation_stringrrrget_activationsra)*r collectionsrrrrutilsrZ get_loggerrloggerModulerrr"r1r3r5r=rBrErFrMrOZ LeakyReLUZReLUZReLU6ZSigmoidZSiLUZTanhZPReLUZACT2CLSr]rarZrYr rXr[r\rCZ linear_actrrrrs