fThSrSSKrSSKJr SSKJr SSKJrJrJ r SSK r SSK r SSK J r SSK JrJrJr SS KJrJr \"5(aSS KJr SS KJr SS KJrJrJrJrJrJrJrJ r J!r! SS K"J#r# SSK$J%r% SSKJ&r& SSK'J(r( \&RR"\*5r+Sr,Sr-Sr."SS\ R^5r0"SS\ R^5r1"SS\ R^5r2"SS\ R^5r3"SS\ R^5r4"SS \ R^5r5"S!S"\ R^5r6"S#S$\ R^5r7"S%S&\ R^5r8"S'S(\ R^5r9"S)S*\ R^5r:"S+S,\ R^5r;"S-S.\ R^5r<"S/S0\ R^5r=\"S1S2\#55r>\"S3S4\55r?\"S5S6\>55r@\"S7S89"S9S:\>55rA\"S;S<\>55rB\"S=S89"S>S?\>55rC\"S@S89"SASB\>55rD\"SCSD\>55rE\"SESF\>55rF\"SGSH\>55rG/SIQrHg)JzPyTorch FNet model.N) dataclass)partial)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)auto_docstringis_scipy_available)linalg)ACT2FN) BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)logging) FNetConfigcURSnUSU2SU24nUR[R5n[R"SXU5$)z4Applies 2D matrix multiplication to 3D input arrays.rNzbij,jk,ni->bnk)shapetypetorch complex64einsum)xmatrix_dim_onematrix_dim_two seq_lengths ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/fnet/modeling_fnet.py_two_dim_matmulr*7sLJ#KZK*$<=N uA <<(!^ LLc[XU5$N)r*)r%r&r's r)two_dim_matmulr.@s 1n ==r+cUn[[UR5SS5H n[RR XS9nM" U$)z Applies n-dimensional Fast Fourier Transform (FFT) to input array. Args: x: Input n-dimensional array. Returns: n-dimensional Fourier transform of input n-dimensional array. rN)axis)reversedrangendimr"fft)r%outr0s r)fftnr6Es@ Cqvvqr*+iimmCm+, Jr+c6^\rSrSrSrU4SjrSSjrSrU=r$)FNetEmbeddingsUzGConstruct the embeddings from word, position and token_type embeddings.cj>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR UR 5Ul[R "UR"5UlUR'S[(R*"UR5R-S5SS9 UR'S[(R."UR0R35[(R4S9SS9 g) N) padding_idxeps position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsLinear projectionDropouthidden_dropout_probdropoutregister_bufferr"arangeexpandzerosr>sizelongselfconfig __class__s r)rEFNetEmbeddings.__init__Xs< !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>ST))F$6$68J8JKzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r+cbUbUR5nOUR5SSnUSnUcURSS2SU24nUcv[US5(a-URSS2SU24nUR USU5nUnO8[ R "U[ RURRS9nUcURU5nURU5n XI-n URU5n X- n URU 5n URU 5n URU 5n U $)Nr?rrArrCdevice)rZr>hasattrrArXr"rYr[rcrJrNrLrOrRrU) r] input_idsrAr> inputs_embeds input_shaper(buffered_token_type_ids buffered_token_type_ids_expandedrN embeddingsrLs r)forwardFNetEmbeddings.forwardns:  #..*K',,.s3K ^  ,,Q  ^ J": "66|D) ^^J/ __Z0 \\*- r+)rOrUrLrRrNrJ)NNNN) __name__ __module__ __qualname____firstlineno____doc__rErk__static_attributes__ __classcell__r_s@r)r8r8UsQ ,!!r+r8c4^\rSrSrU4SjrSrSrSrU=r$)FNetBasicFourierTransformcD>[TU]5 URU5 gr-)rDrE_init_fourier_transformr\s r)rE"FNetBasicFourierTransform.__init__s  $$V,r+cUR(d([[RRSS9UlgUR S::a[5(aURS[R"[R"UR5[RS95 URS[R"[R"UR5[RS95 [[UR UR"S9Ulg[$R&"S5 [Ulg[Ulg) N)rdimdft_mat_hiddenrB dft_mat_seq)r&r'zpSciPy is needed for DFT matrix calculation and is not found. Using TPU optimized fast fourier transform instead.)use_tpu_fourier_optimizationsrr"r4r6fourier_transformrKrrVtensorrdftrHr#tpu_short_seq_lengthr.rrrwarning)r]r^s r)ry1FNetBasicFourierTransform._init_fourier_transforms33%,UYY^^%HD "  + +t 3!##$$$ell6::f>P>P3QY^YhYh&i$$!5<< 6;V;V0W_d_n_n#o*1"43C3CTXTgTg*&**.&%)D "r+c>URU5RnU4$r-)rreal)r] hidden_statesoutputss r)rk!FNetBasicFourierTransform.forwards" ((7<<zr+)r) rmrnrorprEryrkrrrsrts@r)rvrvs-*.r+rvc.^\rSrSrU4SjrSrSrU=r$)FNetBasicOutputc~>[TU]5 [R"URUR S9UlgNr<)rDrErrOrHrPr\s r)rEFNetBasicOutput.__init__s, f&8&8f>S>STr+c,URX!-5nU$r-rOr]r input_tensors r)rkFNetBasicOutput.forwards|'CD r+rrmrnrorprErkrrrsrts@r)rrsUr+rc.^\rSrSrU4SjrSrSrU=r$)FNetFourierTransformcb>[TU]5 [U5Ul[ U5Ulgr-)rDrErvr]routputr\s r)rEFNetFourierTransform.__init__s& -f5 %f- r+cXURU5nURUSU5nU4nU$Nr)r]r)r]r self_outputsfourier_outputrs r)rkFNetFourierTransform.forwards1yy/ \!_mD!#r+)rr]rrts@r)rrs. r+rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)FNetIntermediatec>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr-) rDrErrQrHintermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr\s r)rEFNetIntermediate.__init__s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r+rreturncJURU5nURU5nU$r-rrr]rs r)rkFNetIntermediate.forwards& =1 00? r+r rmrnrorprEr"Tensorrkrrrsrts@r)rrs(9U\\ellr+rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) FNetOutputc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rDrErrQrrHrrOrPrSrTrUr\s r)rEFNetOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r+rrrcpURU5nURU5nURX-5nU$r-)rrUrOrs r)rkFNetOutput.forwards5 =1  ]3 }'CD r+)rOrrUrrts@r)rrs6> U\\RWR^R^r+rc4^\rSrSrU4SjrSrSrSrU=r$) FNetLayerc>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul gNr) rDrEchunk_size_feed_forward seq_len_dimrfourierr intermediaterrr\s r)rEFNetLayer.__init__sI '-'E'E$+F3 ,V4 ( r+cURU5nUSn[URURURU5nU4nU$r)rrfeed_forward_chunkrr)r]rself_fourier_outputsr layer_outputrs r)rkFNetLayer.forwardsO#||M:-a00  # #T%A%A4CSCSUc  /r+cJURU5nURX!5nU$r-)rr)r]rintermediate_outputrs r)rFNetLayer.feed_forward_chunks("//?{{#6G r+)rrrrr) rmrnrorprErkrrrrsrts@r)rrs) r+rc2^\rSrSrU4SjrSSjrSrU=r$) FNetEncoderi c>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) rDrEr^r ModuleListr2num_hidden_layersrlayergradient_checkpointing)r]r^_r_s r)rEFNetEncoder.__init__ sR  ]]uVE]E]?^#_?^!If$5?^#_` &+#$`sA&cbU(aSOSn[UR5H]upVU(aXA4-nUR(a.UR(aUR UR U5nOU"U5nUSnM_ U(aXA4-nU(d[ SX455$[XS9$)Nrc3.# UH ocMUv M g7fr-r).0vs r) &FNetEncoder.forward.."sX$Fq$Fs )last_hidden_stater) enumeraterrtraining_gradient_checkpointing_func__call__tupler)r]routput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r)rkFNetEncoder.forwards"6BD(4OA#$58H$H!**t}} $ A A,BWBWYf g ,] ; )!,M 5 14D D X]$FXX X``r+)r^rr)FTrrts@r)rr s, aar+rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) FNetPooleri(c>[TU]5 [R"URUR5Ul[R "5Ulgr-)rDrErrQrHrTanh activationr\s r)rEFNetPooler.__init__)s9 YYv1163E3EF '')r+rrc\USS2S4nURU5nURU5nU$r)rr)r]rfirst_token_tensor pooled_outputs r)rkFNetPooler.forward.s6+1a40 #56  6 r+)rrrrts@r)rr(s($ U\\ellr+rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)FNetPredictionHeadTransformi8cp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) rDrErrQrHrrrrrtransform_act_fnrOrPr\s r)rE$FNetPredictionHeadTransform.__init__9s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr+rrclURU5nURU5nURU5nU$r-)rrrOrs r)rk#FNetPredictionHeadTransform.forwardBs4 =1 --m< }5 r+)rOrrrrts@r)rr8s)UU\\ellr+rc8^\rSrSrU4SjrSrSSjrSrU=r$)FNetLMPredictionHeadiIcJ>[TU]5 [U5Ul[R "UR UR5Ul[R"[R"UR55Ul URURl gr-) rDrEr transformrrQrHrGdecoder Parameterr"rYbiasr\s r)rEFNetLMPredictionHead.__init__Jsi 4V<yy!3!3V5F5FG LLV->->!?@  II r+cJURU5nURU5nU$r-)rrrs r)rkFNetLMPredictionHead.forwardUs$}5  ]3 r+cURRRRS:XaURURlgURRUlg)Nmeta)rrrcr!r]s r) _tie_weights!FNetLMPredictionHead._tie_weightsZsC <<   # # ( (F 2 $ DLL  ))DIr+)rrr)rN) rmrnrorprErkr rrrsrts@r)rrIs & **r+rc.^\rSrSrU4SjrSrSrU=r$)FNetOnlyMLMHeadiccB>[TU]5 [U5Ulgr-)rDrEr predictionsr\s r)rEFNetOnlyMLMHead.__init__ds /7r+c(URU5nU$r-r)r]sequence_outputprediction_scoress r)rkFNetOnlyMLMHead.forwardhs ,,_=  r+rrrts@r)rrcs8!!r+rc.^\rSrSrU4SjrSrSrU=r$)FNetOnlyNSPHeadincn>[TU]5 [R"URS5UlgNr|)rDrErrQrHseq_relationshipr\s r)rEFNetOnlyNSPHead.__init__os'  " &*<*[TU]5 [U5Ul[R "UR S5Ulgr)rDrErrrrQrHrr\s r)rEFNetPreTrainingHeads.__init__zs4 /7 " &*<*.  - - KK   " " $ MM   $ $S ).r+rN) rmrnrorpr config_classbase_model_prefixsupports_gradient_checkpointingr5rrrr+r)r)r)sL&*#*r+r)c\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Srg) FNetForPreTrainingOutputiam Output type of [`FNetForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. Nlossprediction_logitsseq_relationship_logitsrr)rmrnrorprqr<rr" FloatTensor__annotations__r=r>rrrrrr+r)r;r;sd$)-D(5$$ %,59x 1 129;?Xe&7&78?8[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rDrEr^r8rjrencoderrpooler post_init)r]r^add_pooling_layerr_s r)rEFNetModel.__init__sK   (0"6* ,=j(4  r+c.URR$r-rjrJr s r)get_input_embeddingsFNetModel.get_input_embeddingss...r+c$XRlgr-rJ)r]values r)set_input_embeddingsFNetModel.set_input_embeddingss */'r+rerAr>rfrrrcUbUOURRnUbUOURRnUbUb [S5eUbUR 5nUupO&UbUR 5SSnUupO [S5eURR (a+U S::a%URR U :wa [S5eUb URO URn Ucr[URS5(a3URRSS2SU 24n U RX5n U nO$[R"U[RU S9nURUUUUS9n URU UUS 9nUS nUR bUR!U5OSnU(d UU4US S-$[#UUUR$S 9$) NzDYou cannot specify both input_ids and inputs_embeds at the same timer?z5You have to specify either input_ids or inputs_embedsrzThe `tpu_short_seq_length` in FNetConfig should be set equal to the sequence length being passed to the model when using TPU optimizations.rArb)rer>rArf)rrrr)r pooler_outputr)r^ruse_return_dict ValueErrorrZrrrcrdrjrArXr"rYr[rDrErr)r]rerAr>rfrrrg batch_sizer(rcrhriembedding_outputencoder_outputsrrRs r)rkFNetModel.forwards%9$D $++JjJj &1%cd d  "#..*K%0 "J  &',,.s3K%0 "J TU U KK 5 5d" 00J>;  &/%:!! @T@T  !t(899*.//*H*HKZK*X'3J3Q3QR\3i0!A!&[ SY!Z??%)' +  ,, !5#'  *!,8< 8O O4UY #]3oab6II I)-')77  r+)r^rjrDrE)T)NNNNNN)rmrnrorprqrErKrOr rr" LongTensorr?boolrrrrkrrrsrts@r)rBrBs /015593759/3&*C E,,-C !!1!12C u//0 C   1 12 C 'tn C d^C  uo% &C C r+rBz FNet Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. ) custom_introc:^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ S\ \ S\ \\44Sjj5rSrU=r$)FNetForPreTrainingicls.predictions.decoder.biascls.predictions.decoder.weightc>[TU]U5 [U5Ul[ U5UlUR 5 gr-)rDrErBr*r"clsrFr\s r)rEFNetForPreTraining.__init__&s4  f% '/ r+cBURRR$r-rarrr s r)get_output_embeddings(FNetForPreTraining.get_output_embeddings/xx##+++r+cXRRlURURRlgr-rarrrr]new_embeddingss r)set_output_embeddings(FNetForPreTraining.set_output_embeddings2*'5$$2$7$7!r+rerAr>rflabelsnext_sentence_labelrrrc UbUOURRnURUUUUUUS9n U SSupURX5upSnUbvUbs[ 5nU"U R SURR 5UR S55nU"U R SS5UR S55nUU-nU(dX4U SS-nUbU4U-$U$[UU U U RS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForPreTraining.from_pretrained("google/fnet-base") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```NrAr>rfrrr|r?)r<r=r>r) r^rSr*rar viewrGr;r)r]rerAr>rfrorprrrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r)rkFNetForPreTraining.forward6s+J&1%[TU]U5 [U5Ul[ U5UlUR 5 gr-)rDrErBr*rrarFr\s r)rEFNetForMaskedLM.__init__4  f% "6* r+cBURRR$r-rdr s r)re%FNetForMaskedLM.get_output_embeddingsrgr+cXRRlURURRlgr-rirjs r)rl%FNetForMaskedLM.set_output_embeddingsrnr+rerAr>rfrorrrc UbUOURRnURUUUUUUS9nUSn URU 5n Sn UbF[ 5n U "U R SURR 5UR S55n U(dU 4USS-n U bU 4U -$U $[XURS9$)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrrrr?r|r<logitsr) r^rSr*rar rsrGrr)r]rerAr>rfrorrrrrrvrurs r)rkFNetForMaskedLM.forwards"&1%[b[p[pqqr+ryNNNNNNN)rmrnrorpr{rErerlr rr"rrZrrrrkrrrsrts@r)r}r}|s8:Z[,8-115/304)-/3&*'rELL)'r!.'ru||, 'r   - 'r & 'r'tn'rd^'r un$ %'r'rr+r}zT FNet Model with a `next sentence prediction (classification)` head on top. c^\rSrSrU4Sjr\S S\\RS\\RS\\RS\\RS\\RS\\ S \\ S \ \ \ 44S jj5r S rU=r$)FNetForNextSentencePredictionic>[TU]U5 [U5Ul[ U5UlUR 5 gr-)rDrErBr*rrarFr\s r)rE&FNetForNextSentencePrediction.__init__rr+rerAr>rfrorrrc SU;a,[R"S[5 URS5nUbUOURR nUR UUUUUUS9n U Sn URU 5n Sn Ub2[5n U "U RSS5URS55n U(dU 4U SS-nU bU 4U-$U$[U U U RS9$) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForNextSentencePrediction.from_pretrained("google/fnet-base") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ```rpzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrrrr?r|r) warningswarn FutureWarningpopr^rSr*rar rsrr)r]rerAr>rfrorrkwargsrrseq_relationship_scoresrwrurs r)rk%FNetForNextSentencePrediction.forwards H !F * MM%  ZZ 56F%0%[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr- rDrE num_labelsrBr*rrSrTrUrQrH classifierrFr\s r)rE&FNetForSequenceClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ r+rerAr>rfrorrrc 4UbUOURRnURUUUUUUS9nUSn URU 5n UR U 5n Sn UGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5n UR S:Xa&U "U R5UR55n OU "X5n OURR S:Xa=[5n U "U RSUR 5URS55n O,URR S:Xa[5n U "X5n U(dU 4USS-n U bU 4U -$U $[!XUR"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationr?r|r)r^rSr*rUr problem_typerrCr"r[intr squeezer rsr rr)r]rerAr>rfrorrrrrr<rurs r)rk%FNetForSequenceClassification.forward's"&1%[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 gr) rDrErBr*rrSrTrUrQrHrrFr\s r)rEFNetForMultipleChoice.__init__fsV  f% zz&"<"<= ))F$6$6: r+rerAr>rfrorrrc UbUOURRnUbURSOURSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUS9n U Sn UR U 5n URU 5n U RSU5n Sn Ub[5nU"X5n U(dU 4U SS-nU bU 4U-$U$[XU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr?rrr|r) r^rSr rsrZr*rUrr rr)r]rerAr>rfrorr num_choicesrrrreshaped_logitsr<rurs r)rkFNetForMultipleChoice.forwardpsR&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) )%'!5#     ]3 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E(dZaZoZoppr+)rrUr*r)rmrnrorprEr rr"rrZrrrrkrrrsrts@r)rrds-115/304)-/3&*LqELL)Lq!.Lqu||, Lq   - Lq & Lq'tnLqd^Lq u// 0LqLqr+rc^\rSrSrU4Sjr\S S\\RS\\RS\\RS\\RS\\RS\\ S \\ S \ \ \ 44S jj5r S rU=r$)FNetForTokenClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr-rr\s r)rE#FNetForTokenClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ r+rerAr>rfrorrrc UbUOURRnURUUUUUUS9nUSn URU 5n UR U 5n Sn Ub<[ 5n U "U R SUR5UR S55n U(dU 4USS-n U bU 4U -$U $[XURS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrr?r|r) r^rSr*rUrr rsrrr)r]rerAr>rfrorrrrrr<rurs r)rk"FNetForTokenClassification.forwards&1%[TU]U5 URUl[U5Ul[ R "URUR5UlUR5 gr-) rDrErrBr*rrQrH qa_outputsrFr\s r)rE!FNetForQuestionAnswering.__init__sS   ++f% ))F$6$68I8IJ r+rerAr>rfstart_positions end_positionsrrrc UbUOURRnURUUUUUUS9n U Sn URU 5n U R SSS9upU R S5R 5n U R S5R 5n SnUbUb[UR55S:aUR S5n[UR55S:aUR S5nU RS5nURSU5nURSU5n[US9nU"X5nU"X5nUU-S- nU(dX4U SS-nUbU4U-$U$[XXRS9$) Nrrrrr?r}) ignore_indexr|)r< start_logits end_logitsr) r^rSr*rsplitr contiguouslenrZclampr rr)r]rerAr>rfrrrrrrrrrrt ignored_indexru start_lossend_lossrs r)rk FNetForQuestionAnswering.forwards&1%rs!)) AA7!   .6*   H %M>  :RYY:z# #Lbii 299  ryy    6a"))a>   "))"*299*4!bii!&bii& 9299 9*/**. ={= =2b #b b J V ,V  V r;r);r;r| R $7R  R j Fg$7Fg  FgRXq/XqXqv6d!46d6drA 2A A H r+