eTh%SSKrSSKJr SSKrSSKJrJr SSKJr \R"\ 5r "SS\R5r "SS \R5r "S S \R5r"S S \R5r"SS\R5r"SS\R5r"SS\R5r"SS\R5r"SS\R5r"SS\R5r"SS\R5r"SS\5r0S\_S\S S!S".4_S#\_S$\ _S%\S&S'04_S(\ _S)\_S*\_S+\R0_S,\_S-\_S.\_S/\R2_S0\_S1\R4_S2\R6_S3\R8_\R8\R:\R<S4.Er\"\5r S5r!\!"S%5r"\!"S$5r#\!"S5r$\!"S#5r%\!"S.5r&\!"S35r'\!"S-5r(\!"S,5r)g)6N) OrderedDict)Tensornn)loggingc*\rSrSrSrS\S\4SjrSrg)PytorchGELUTanha  A fast C implementation of the tanh approximation of the GeLU activation function. See https://arxiv.org/abs/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. inputreturnc>[RRUSS9$)Ntanh) approximate)r functionalgeluselfr s P/var/www/auris/envauris/lib/python3.13/site-packages/transformers/activations.pyforwardPytorchGELUTanh.forward$s}}!!%V!<<N__name__ __module__ __qualname____firstlineno____doc__rr__static_attributes__rrrr r s=V==rr c*\rSrSrSrS\S\4SjrSrg)NewGELUActivation(z Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 r r c SU-S[R"[R"S[R- 5US[R "US5---5--$)N??@Hm?g@)torchrmathsqrtpipowrs rrNewGELUActivation.forward.sPU{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrrNrrrrr!r!(s wVwwrr!c\^\rSrSrSrS S\4U4SjjjrS\S\4SjrS\S\4Sjr S r U=r $) GELUActivation2a Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://arxiv.org/abs/1606.08415 use_gelu_pythonc>[TU]5 U(aURUlg[R R UlgN)super__init__ _gelu_pythonactrrr)rr1 __class__s rr5GELUActivation.__init__:s/  ((DH}}))DHrr r cnUS-S[R"U[R"S5- 5--$)Nr$r%r&)r(erfr)r*rs rr6GELUActivation._gelu_pythonAs,s{cEIIediin.D$EEFFrc$URU5$r3r7rs rrGELUActivation.forwardDxxrr>)F) rrrrrboolr5rr6rr __classcell__r8s@rr/r/2sG***G&GVGVrr/c*\rSrSrSrS\S\4SjrSrg)FastGELUActivationHzu Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r r c^SU-S[R"US-SSU-U---5--$)Nr$r%g3E?r')r(rrs rrFastGELUActivation.forwardMs:U{cEJJu|/CsXX]M]`eMeGe/f$gghhrrNrrrrrErEHsiViirrEc*\rSrSrSrS\S\4SjrSrg)QuickGELUActivationQzj Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs r r c:U[R"SU-5-$)NgZd;?)r(sigmoidrs rrQuickGELUActivation.forwardVsu}}UU]333rrNrrrrrJrJQs4V44rrJcJ^\rSrSrSrS\S\4U4SjjrS\S\4SjrS r U=r $) ClippedGELUActivationZab Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://arxiv.org/abs/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://arxiv.org/abs/1606.08415 minmaxch>X:a[SUSUS35e[TU] 5 XlX lg)Nzmin should be < max (got min: z, max: )) ValueErrorr4r5rRrS)rrRrSr8s rr5ClippedGELUActivation.__init__gs8 9=cU'#aPQ Q rxr cl[R"[U5URUR5$r3)r(cliprrRrS)rrXs rrClippedGELUActivation.forwardos!zz$q'488TXX66r)rSrR) rrrrrfloatr5rrrrBrCs@rrPrPZs3 E77F77rrPc>^\rSrSrSrU4SjrS\S\4SjrSrU=r $)AccurateGELUActivationsz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cz>[TU]5 [R"S[R- 5Ulg)N)r4r5r)r*r+precomputed_constantrr8s rr5AccurateGELUActivation.__init__{s' $(IIa$''k$:!rr r c SU-S[R"URUS[R"US5---5--$)Nr$rr')r(rrbr,rs rrAccurateGELUActivation.forwardsEU{a%**T-F-F%RZ]b]f]fglno]pRpJp-q"rrssr)rb) rrrrrr5rrrrBrCs@rr^r^ss);tVtttrr^cP^\rSrSrSrU4SjrS\S\4SjrS\S\4SjrSr U=r $) MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://arxiv.org/abs/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish c`>[TU]5 [RRUlgr3)r4r5rrmishr7rcs rr5MishActivation.__init__s ==%%rr r cnU[R"[RR U55-$r3)r(rrrsoftplusrs r _mish_pythonMishActivation._mish_pythons%uzz"--"8"8"?@@@rc$URU5$r3r>rs rrMishActivation.forwardr@rr>) rrrrrr5rrprrrBrCs@rriris; &A&AVAVrric*\rSrSrSrS\S\4SjrSrg)LinearActivationzS Applies the linear activation function, i.e. forwarding input directly to output. r r cU$r3rrs rrLinearActivation.forwards rrNrrrrrurusVrruc"\rSrSrSrSSjrSrg)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://arxiv.org/abs/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability cX- RU[R"S5-5nSS[R"U5--$)Nr&r$r%)divr)r*r(r;)rr musigmas rrLaplaceActivation.forwards:  3!78cEIIe,,--rrN)g۞?g ^/ ?rrrrrrrrrrrzrzs .rrzc\rSrSrSrSrSrg)ReLUSquaredActivationzP Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2 cp[RRU5n[R"U5nU$r3)rrrelur(square)rr relu_appliedsquareds rrReLUSquaredActivation.forwards)}}))%0 ,,|,rrNrrrrrrs rrc(^\rSrSrU4SjrSrU=r$)ClassInstantiercl>[TU]U5n[U[5(aUOU04up4U"S0UD6$)Nr)r4 __getitem__ isinstancetuple)rkeycontentclskwargsr8s rrClassInstantier.__getitem__s7'%c*!+GU!;!;g'2 }V}rr)rrrrrrrBrCs@rrrs rrrgelu_10i )rRrS gelu_fastgelu_new gelu_pythonr1Tgelu_pytorch_tanh gelu_accuratelaplace leaky_relulinearrl quick_gelurrelu2relu6rMsilu)swishrpreluc U[;a [U$[SUS[[R5535e)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationrsBF"'((#4"55RSWX^XcXcXeSfRghiir)*r) collectionsrr(rrutilsr get_loggerrloggerModuler r!r/rErJrPr^rirurzrr LeakyReLUReLUReLU6SigmoidSiLUTanhPReLUACT2CLSrrrrrrrrrl linear_actrrrrsj #    H % =bii =w wRYY,ii4"))47BII72 tRYY t RYY"ryy . .BIIk  N  %s2'>? # !   N%6$=>     +   ",,    N %  BGG  "  RXX  rzz! " BGG# $WW GG XX) ,  !j]+ * %f ; ' L ) ff H % r