fThSrSSKrSSKJrJrJr SSKrSSKrSSKrSSKJ r SSK J r J r J r SSKJr SSKJrJr SS KJr SS KJr SS KJrJr SS KJrJrJrJr SS KJ r SSK!J"r"J#r#J$r$J%r%J&r& SSK'J(r(J)r) SSK*J+r+ \$"5(a SSK,J-r- SSK.J/r/ \"5(aSSKJ0r0 \&Rb"\25r3S\4S\4S\Rj4Sjr6\RnRpS5r9S\RjS\Rj4Sjr:S\RjS\RjS\RjS\Rj4Sjr;"S S!\ Rx5r="S"S#\=5r>\=\>S$.r?"S%S&\ Rx5r@"S'S(\ Rx5rA\#"S)S*\ 55rBS+rCS,rD\#"S-S.\B55rE\#"S/S09"S1S2\B\55rF\#"S3S09"S4S5\B55rG\#"S6S7\B55rH/S8QrIg)9zPyTorch GPT-J model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPast)PreTrainedModel)add_start_docstringsauto_docstringis_torch_flex_attn_availableis_torch_fx_proxylogging)assert_device_mapget_device_map) GPTJConfig) BlockMask)make_flex_block_causal_mask)_flash_attention_forwardnum_posdimreturnc SS[R"SUS[RS9U- -- n[R"S[R"U[RS9R 5U5R 5n[R "[R "U5[R"U54SS9$) N?i'rdtypez i , j -> i jrr$)torcharangeint64einsumfloatcatsincos)r#r$inv_freq sinusoid_inps ^/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/gptj/modeling_gptj.pycreate_sinusoidal_positionsr7>se QQekk JS PQRH<< WEKK0X0^0^0`bjkqqsL 99eii -uyy/FGQ OOctURUR5RURSSS5$Nrr)todevicerepeatshape)embed_positions position_idss r6get_embed_positionsrADs5   l11 2 9 9,:L:LQ:OQRTU VVr8xcUSS2SS2SS2SSS24nUSS2SS2SS2SSS24n[R"U*U4SS9nURS5$)Nr(rr+)r,stackflatten)rBx1x2s r6rotate_every_tworJIsS 1aCaC<B 1aADqD= B bS"I2&A 99R=r8tensorr2r3c[R"USS2SS2SSS24SS5n[R"USS2SS2SSS24SS5nX-[U5U--$)Nr(r )r,repeat_interleaverJ)rKr2r3s r6apply_rotary_pos_embrNPs\  ! !#aD!m"4a ;C  ! !#aD!m"4a ;C L-f5; <[TU]5 XlURn[R "UR 5Ul[R "UR5Ul SUl X l Uc-[RSURRS35 UR UlUR$UlUR"UR$-UlUR&UR$-UR":wa&[)SUR"SUR$S35e[*R,"[*R."UR&[*R0S95R3[*R4"55Ul[R8"UR"UR"SS 9Ul[R8"UR"UR"SS 9Ul[R8"UR"UR"SS 9Ul[R8"UR"UR"SS 9Ul URBUl!URB=(d UR"n[EX45Ul#g) NTzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.zEembed_dim must be divisible by num_attention_heads (got `embed_dim`: z and `num_attention_heads`: z).r)Fbias)$super__init__configmax_position_embeddingsrDropout attn_pdrop attn_dropout resid_pdrop resid_dropout is_causal layer_idxlogger warning_once __class____name__ hidden_size embed_dimnum_attention_headshead_dim ValueErrorr,sqrtrKfloat32r;get_default_dtype scale_attnLineark_projv_projq_projout_proj rotary_dimr7r?)selfrWr_ max_positions pos_embd_dimrbs r6rVGPTJAttention.__init__Ws  66 JJv'8'89ZZ(:(:;"      !8!8 9:,,   ++#)#=#= $*B*BB ==433 3t~~ EWX\XfXfWgh++/+C+C*DBH  **U\\$--u}}%UVYYZ_ZqZqZstiiUK iiUK iiUK  $..$..uM  ++8$.. :=Wr8c\UR5SSX#4-nURU5nU(aU$[UR5S:XaUR SSSSS5$[UR5S:XaUR SSSS5$[ S [UR535e) z? Splits hidden dim into attn_head_size and num_attention_heads NrDrrr r(3Input tensor rank should be one of [4, 5], but is: )sizeviewlenr>permuterh)rsrKrfattn_head_sizerotary new_shapes r6 _split_headsGPTJAttention._split_headszsKKM#2&*=)NN Y' M v||  !>>!Q1a0 0  ! #>>!Q1- -RSVW]WcWcSdRefg gr8c[UR5S:Xa$URSSSSS5R5nO][UR5S:Xa#URSSSS5R5nO![ S[UR535eUR 5SS X#-4-nUR U5$) zB Merges attn_head_size dim and num_attn_heads dim into hidden dim rxrrr r(ryrzNrE)r}r>r~ contiguousrhr{r|)rsrKrfrrs r6 _merge_headsGPTJAttention._merge_headss v||  !^^Aq!Q2==?F  ! #^^Aq!Q/::r functionalsoftmaxr*r[) rsquerykeyvalueattention_mask head_mask attn_weights causal_mask attn_outputs r6_attnGPTJAttention._attns'ffU]]#||E==R+@A #oo5  %(Aq/CIIbM/)ABK'5L}},,\r,B #u{{3 ((6   '3Lll<7 ((r8cURnURUR:wa!URUR5nX lURURSSS5$r:)r?r<r;r=r>)rsr@r?s r6_get_embed_positions"GPTJAttention._get_embed_positionss]..  ! !\%8%8 8-001D1DEO#2 %%l&8&8&;QBBr8 hidden_states layer_pastrr@r use_cacheoutput_attentionscache_positionr%.c 4URU5n URU5n URU5n URXRUR S5n URXRUR S5n URXRUR S5n [ U5(d#[RR5(a[URU5n OURU5n URS5RSSU RS5n [R "U SU 5n[R""XRSS-SS9unnUR$bU SS2SS2SS2SUR$24nU SS2SS2SS2UR$S24nU SS2SS2SS2SUR$24nU SS2SS2SS2UR$S24n['UUU5n['UUU5n[R("UU/SS9n [R("UU/SS9n O['XU5n ['XU5n U R+SSSS5n U R+SSSS5n Ub0UUUR$US .nUR-XUR.U5upUR1XXU5unnUR3UURUR 5nUR5U5nUR7U5nUU4nU(aUU4- nU$) NTFrDrr(r+rr r2r3partial_rotation_sizer)rprnrorrfrgrr,jit is_tracingrAr?r unsqueezer=r>gathersplitrrrNr1r~updater_rrrqr])rsrrrr@rrrrrrrr?repeated_position_idssincosr2r3k_rotk_passq_rotq_pass cache_kwargsrroutputss r6forwardGPTJAttention.forwards M*kk-( M*!!%)A)A4==RVW%=%=t}}dS!!%)A)A4==RWX \ * *eii.B.B.D.D2$2F2F UO"77 EO , 6 6r : A A!QH]H]^`Ha boq2GH;;v||B'71'<"ES ?? &1a!24??!223EAq$//"334F!Q#4T__#445E1aDOO$556F(S9E(S9E))UFO4CIIufo26E&s5C(S9Ekk!Q1% aAq)  !)-"0 L $**3t~~|TJC%)JJu5R[$\! \'' T5M5Mt}}] mmK0 ((5  +    &Gr8)r[rWrer?rgr^rnr_rfrqrpr]rrrlroN)NNNNNNFFN)rc __module__ __qualname____firstlineno__rVrrrrr, FloatTensorrr LongTensorboolrrTensorr__static_attributes__ __classcell__rbs@r6rPrPVs>!XF h &$ )>C'+6:3715$),159H((HUOH!!2!23 H u//0 H E--. HD>H$D>H!!1!12H  ellE%,,//0u||U5<<%8% c@Q:RRST V HHr8rPc^\rSrSrSrU4SjrSS\RS\\ S\\RS\\RS\\RS \\ S \\ S \\RS \ \ \R\ \R4\\ \R\ \R\ \RS 4444SjjrSrU=r$)GPTJFlashAttention2ia4 GPTJ flash attention module. This module inherits from `GPTJAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cD>[TU]"U0UD6 [5Ulgr)rUrVr_flash_attn_uses_top_left_mask)rsargskwargsrbs r6rVGPTJFlashAttention2.__init__s# $)&) /P.Q+r8rrrr@rrrrr%.c  URU5n URU5n URU5n URXRUR S5n URXRUR S5n URXRUR S5n [ U5(d#[RR5(a[URU5n OURU5n URS5RSSU RS5n [R "U SU 5n[R""XRSS-SS9unnUR$bU SS2SS2SS2SUR$24nU SS2SS2SS2UR$S24nU SS2SS2SS2SUR$24nU SS2SS2SS2UR$S24n['UUU5n['UUU5n[R("UU/SS9n [R("UU/SS9n O['XU5n ['XU5n U R+SSSS5n U R+SSSS5n Ub0UUUR$US .nUR-XUR.U5upU R+SSSS5R15n U R+SSSS5R15n U R+SSSS5R15n U R2nU[R4:Xa[R6"5(a[R8"5nOR[;UR<S 5(aUR<R>nO URR@R2n[BRES US 35 U RGU5n U RGU5n U RGU5n URH(aUR<RJOS nU RSn[MU U U UUUURNURPS9nURSURSURSURSURS-5nURUU5nURWU5nUU4nU(aUU4- nU$)NTFrDrr(r+rr r_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)dropoutr^use_top_left_mask),rprnrorrfrgrr,rrrAr?rrr=r>rrrrrNr1r~rr_rr*rjis_autocast_enabledget_autocast_gpu_dtypehasattrrWrweightr`rar;trainingrZr"r^rreshaperqr])rsrrrr@rrrrrrrr?rrr2r3rrrrr input_dtype target_dtypeattention_dropout query_lengthrrrs r6rGPTJFlashAttention2.forwardsP M*kk-( M*!!%)A)A4==RVW%=%=t}}dS!!%)A)A4==RWX \ * *eii.B.B.D.D2$2F2F UO"77 EO , 6 6r : A A!QH]H]^`Ha boq2GH;;v||B'71'<"ES ?? &1a!24??!223EAq$//"334F!Q#4T__#445E1aDOO$556F(S9E(S9E))UFO4CIIufo26E&s5C(S9E kk!Q1% aAq)  !)-"0 L $**3t~~|TJC kk!Q1%002 aAq)446 aAq)446kk %-- '((**$;;= &?@@#{{BB #{{1177    >$  HH\*E&&&CHH\*E6:mmDKK22{{1~ 0     %nn"AA  #**   q !<#5#5a#8,:L:LQ:OR^RdRdefRg:g mmK0 ((5  +    &Gr8)rr)rcrrr__doc__rVr,rrr rrrrrrrrrs@r6rrs$ R'+6:3715$),159~((~UO~!!2!23 ~ u//0 ~ E--. ~D>~$D>~!!1!12~  ellE%,,//0u||U5<<%8% c@Q:RRST V ~~r8r)eagerflash_attention_2ch^\rSrSrU4SjrS\\RS\R4SjrSr U=r $)GPTJMLPic >[TU]5 URn[R"X15Ul[R"X5Ul[URUl [R"UR5Ul gr) rUrVn_embdrrmfc_infc_outr activation_functionactrYr\r)rsintermediate_sizerWrerbs r6rVGPTJMLP.__init__s` MM YYy< ii 1= &445zz&"4"45 r8rr%cURU5nURU5nURU5nURU5nU$r)rrrr)rsrs r6rGPTJMLP.forwards@ =1 /  M2  ]3 r8)rrrr) rcrrrrVrr,rrrrrs@r6rrs16Xe.?.?%@UEVEVr8rcf^\rSrSrSU4SjjrSS\\RS\\S\\RS\\RS\\RS\\ S \\ S \\RS \ \ \R\\ \R\ \RS 4444S jjrSrU=r$) GPTJBlockic*>[TU]5 URb UROSUR-n[R "URUR S9Ul[UR"X5Ul [X15Ul g)Nryeps) rUrVn_innerrr LayerNormlayer_norm_epsilonln_1GPTJ_ATTENTION_CLASSES_attn_implementationattnrmlp)rsrWr_ inner_dimrbs r6rVGPTJBlock.__init__sj &,nn&@FNNa&--FW LLF4M4MN *6+F+FGZ 9-r8rrrr@rrrrr%.c Un URU5nURUUUUUUUUS9n U Sn U SSn URU5n X-U -nU(aU4U -n U $U4U SS-n U $)Nrrrr@rrrrrr)rrr)rsrrrr@rrrrresidual attn_outputsrrfeed_forward_hidden_statess r6rGPTJBlock.forwards! -0 yy'!)%/)!  #1o qr"%)XXm%<"#@8K $&0G%&4Gr8)rrrrr)rcrrrrVrr,rr rrrrrrrrrs@r6rrs.'+6:3715$),159" 1 12"UO"!!2!23 " u//0 " E--. "D>"$D>"!!1!12" uU\\"HU5<<uGXGXZ]G]A^3^-_$`` a""r8rc\^\rSrSr\rSrSrSrS/r Sr Sr Sr Sr SrSrU4SjrSrS rU=r$) GPTJPreTrainedModeli transformerTrpast_key_valuesFc&>[TU]"U0UD6 gr)rUrV)rsinputsrrbs r6rVGPTJPreTrainedModel.__init__s &+F+r8c[U[R45(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weights.r)meanstdNr') isinstancerrmrdatanormal_rWinitializer_rangerTzero_ Embedding padding_idxrfill_)rsmodules r6 _init_weights!GPTJPreTrainedModel._init_weightss fryyl + + MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r8)rcrrrr config_classbase_model_prefixis_parallelizablesupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_cache_class_supports_quantized_cache_supports_static_cache!_supports_param_buffer_assignmentrVr rrrs@r6rrsUL%&*#$ "3!  $!(-%,**r8ra This is an experimental feature and is a subject to change at a moment's notice. Uses a device map to distribute attention modules of the model across several devices. If no device map is given, it will evenly distribute blocks across all devices. Args: device_map (`Dict[int, list]`, *optional*): A dictionary that maps attention modules to devices. Note that the embedding module and LMHead are always automatically mapped to the first device (for esoteric reasons). That means that the first device should have fewer attention modules mapped to it than other devices. For reference, the GPT-J models have the following number of attention modules: - gpt-j-6B: 28 Example: ```python # Here is an example of a device map on a machine with 4 GPUs using gpt-j-6B, which has a total of 28 attention modules: model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B") device_map = { 0: [0, 1, 2, 3, 4, 5, 6], 1: [7, 8, 9, 10, 11, 12, 13], 2: [14, 15, 16, 17, 18, 19, 20], 3: [21, 22, 23, 24, 25, 26, 27], } model.parallelize(device_map) ``` a6 Moves the model to CPU from a model parallel state. Example: ```python # On a 4 GPU machine with gpt-j-6B: model = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B") device_map = { 0: [0, 1, 2, 3, 4, 5, 6], 1: [7, 8, 9, 10, 11, 12, 13], 2: [14, 15, 16, 17, 18, 19, 20], 3: [21, 22, 23, 24, 25, 26, 27], } model.parallelize(device_map) # Splits the model across several devices model.deparallelize() # Put the model back on cpu and cleans memory by calling torch.cuda.empty_cache() ``` c^\rSrSrU4Sjr\"\5SSj5r\"\5S5r Sr Sr \ SS\ \RS\ \\\\\R&4S \ \R(S \ \RS \ \RS \ \R(S \ \R(S\ \S\ \S\ \S\ \S\ \RS\\\44Sjj5rS S \\R&S4S\R&S\R&S\S\4 Sjjr\S \R&S\S\S\R6S\R&S\4 Sj5rSrU=r$)! GPTJModeli0c v>[TU]U5 URUlURUl[ R "URUR5Ul[ R"UR5Ul [ R"[UR5Vs/sH n[XS9PM sn5Ul[ R "URUR"S9UlSUlSUlSUlUR-5 UR.S:HUlgs snf)N)r_rFr)rUrVrre vocab_sizerr wterY embd_pdropdrop ModuleListrangen_layerrhrrln_fmodel_parallel device_mapgradient_checkpointing post_initr_use_flash_attention_2)rsrWirbs r6rVGPTJModel.__init__2s   ++<< 1 14>>BJJv001 fnnH]^H]1 & >H]^_LLV5N5NO $&+# &,&A&AEX&X# _s/D6c[R"S[5 UcD[[ UR 5[ [RR555OUUl [UR[ UR 55 SUl SURR5;aSO.S[[URR555-UlS[[#URR555-UlUR&R)UR 5UlURR+5HGup#UH<nS[U5-nUR UR)U5UR U'M> MI UR,R)UR$5Ulg)Na6`GPTJModel.parallelize` is deprecated and will be removed in v5 of Transformers, you should load your model with `device_map='balanced'` in the call to `from_pretrained`. You can also provide your own `device_map` but it needs to be a dictionary module_name to device, so for instance {'h.0': 0, 'h.1': 1, ...}Tcpucuda:)warningswarn FutureWarningrr}r%r#r,cuda device_countr(rr'keysstrmin first_devicemax last_devicerr;itemsr&)rsr(kvblock cuda_devices r6 parallelizeGPTJModel.parallelizeFsY     NXM_N3tvv;ejj.E.E.G(H Ieo  $//3tvv;7"%*doo.B.B.D%DE'TWX[\`\k\k\p\p\rXsTtJt"ST__-A-A-C)D%EE88;;t001OO))+DA%A. $u 0 0 =u , IILL!1!12 r8c[R"S[5 SUlSUlSUlSUlURRS5Ul[[UR55H.nURURS5URU'M0 URRS5Ul [RR5 g)N\Like `parallelize`, `deparallelize` is deprecated and will be removed in v5 of Transformers.Fr/)r1r2r3r'r(r9r;rr;r#r}r%r&r,r4 empty_cache)rsindexs r6 deparallelizeGPTJModel.deparallelize`s j  $! 88;;u%3tvv;'E FF5M,,U3DFF5M(IILL'   r8cUR$rrrss r6get_input_embeddingsGPTJModel.get_input_embeddingsps xxr8cXlgrrJrsnew_embeddingss r6set_input_embeddingsGPTJModel.set_input_embeddingsss!r8 input_idsrrtoken_type_idsr@r inputs_embedsrroutput_hidden_states return_dictrr%c  U bU OURRn U bU OURRn UbUOURRnU bU OURRn USLUSL- (a [ S5eUR (a/UR(aU(a[RS5 SnUcURU5nSn U(aP[U[5(d;Sn Uc [5nO+[R"U5n[RS5 URSnU c7UbUR!5OSn["R$"XU-UR&S 9n UcU R)S5nUR+X7XU 5nUR-X`RR.5nUnUb(UR1S U5nURU5nUU-nUR3U5nS UUR5S 54nSnU (aS OSnU (aS OSn[7UR85GHunnUR:(a["R<R?UR&5 UbTUR@RCUR&5Ul URDRCUR&5Ul"UbURCUR&5n[U["RF5(aURCUR&5nU (aUU4-nUR (a8UR(a'URIURJUSUUUUUU U 5 nOU"UUUUUUUU U S 9nUSnUSLaUSnU (aUUU(aS OS4-nUR:(dGMURLRO5HOunnUUS :XdMS[QU5-URR:wdM/URCS[QUS-5-5nMQ GM URUU5nUR1U5nU (aUU4-nU(aUOSnU (aURW5nU (d[YSUUUU455$[[UUUUS9$)u inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrr<rDrrr(r0c3.# UH ocMUv M g7frr).0r>s r6 $GPTJModel.forward..s^a^s )last_hidden_staterr attentions).rWrrVruse_return_dictrhr)rr`rarrr r from_legacy_cacher>get_seq_lengthr,r-r<r_update_causal_mask get_head_maskr$r|r!r{ enumerater%r'r4 set_device key_cacher; value_cacher_gradient_checkpointing_func__call__r(r<r7r;r&to_legacy_cachetupler)rsrSrrrTr@rrUrrrVrWrreturn_legacy_cache seq_lengthpast_key_values_lengthrrtoken_type_embeds output_shapenext_decoder_cacheall_self_attentionsall_hidden_statesr,r?rr=r> next_caches r6rGPTJModel.forwardvs,2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKL] &&y++2E2EF %  %+00Z@N $ 8 ),==M -0 J (:(:2(>? !$5b4"6BD!$&&)HAu"" %%m&:&:;#.0?0I0I0L0L]MaMa0bO-2A2M2M2P2PQ^QeQe2fO/*"-..1E1E"FKi66 ) ]-A-A BI#$58H$H!**t}};;NN! aL%"  "/.#.!-'l'&7#1 $AJMD %,QZ" &9W)QYZ=[<]&]#""" OO113DAqAbEzgA&6$:J:J&J(5(8(83q1u:9M(N 4g*n -0 %**<8  1]4D D +4'$ #335J):7HJ]^ '+&+*   r8r input_tensorc URRS:XaUbUS:HR5(aU$gURRS:Xa,[U[R 5(a [ U5nU$UbUR5OSnUb UROSnURRS:Xa5U(d.U(d'[R"UUUURS9(agURnURSn U(aUR5n O5[U[R 5(aURS OXi-S-n URUU U UUURSS 9n URRS:XaZUbWUR R"S ;a=U(d6[R$"U5R&n [R("X5n U $) Nrrflex_attentionrFsdpa)rUrp is_trainingrrD)sequence_length target_lengthr*r batch_size)r4xpunpu)rWranyrr,rr!rcis_compileabler_ignore_causal_mask_sdparr*r>get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionr<typefinfor8_unmask_unattended) rsrrxrrrpast_seen_tokensusing_compilable_cacher*r}r~r min_dtypes r6rdGPTJModel._update_causal_masks ;; + +/B B)~/D.I.I.K.K%% ;; + +/? ?.%,,77!<^!L! ! @O?Z?99;`aCRC^!?!?di ;; + +v 5>T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCK[Kr8r}r~r*rc UbUR5S:XaUnU$[R"U5Rn[R"X4XUR S9nUS:wa[R "USS9nU[R"X$R S9URSS5:-nUSSSS2SS24RUSSS5nUbUR5nURSn USS2SS2SS2SU 24USS2SSSS24RUR 5-n U S:Hn USS2SS2SS2SU 24RX5USS2SS2SS2SU 24'U$) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nry) fill_valuer*r<r)diagonalrZrDr)r$r,rr8fullr<triur-rexpandcloner>r; masked_fill) rr}r~r*rrrrr mask_length padding_masks r6r?GPTJModel._prepare_4d_causal_attention_mask_with_cache_positionas}>  %.*<*<*>!*C(K*' E*..I** 0Y\j\q\qK!##jjqA 5<< >S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c 6 Aq!\k\12r8) r+r(r!rer9r)r%r;r&r'rrr NNNNNNNNNNNN)F)rcrrrrVrPARALLELIZE_DOCSTRINGrADEPARALLELIZE_DOCSTRINGrGrLrQrrr,rrr rrrrrrrd staticmethodintr*rrrrs@r6rr0sEY(/0313212 !3 !"15NR6:59371559$(,0/3&*59c E,,-c "%uU5<<5H/I(I"JKc !!2!23 c !!1!12 c u//0 c E--.c   1 12c D>c $D>c 'tnc d^c !!1!12c  u-- .c c X#( BellK78BllB B  B  BH4 444{{ 4  4  44r8rzK The GPT-J Model transformer with a language modeling head on top. ) custom_introc v^\rSrSrS/rU4Sjr\"\5SSj5r\"\ 5S5r Sr Sr \ SS\\R S \\\\\\R(4S \\R*S \\R S \\R S \\R*S\\R*S\\R S\\S\\S\\S\\S\\R S\\\44Sjj5r\S \\\R(S\R(S\\\R(4Sj5rSrU=r$)GPTJForCausalLMizlm_head.weightc>[TU]U5 [U5Ul[R "UR UR5UlSUl SUl UR5 gNF) rUrVrrrrmrrlm_headr'r(r*rsrWrbs r6rVGPTJForCausalLM.__init__sV  $V,yy0A0AB $ r8c[R"S[5 UcN[[ UR R 5[[RR555OUUl [UR[ UR R 55 UR RUR5 URRUR R 5UlSUlg)NaT`GPTJForCausalLM.parallelize` is deprecated and will be removed in v5 of Transformers, you should load your model with `device_map='balanced'` in the call to `from_pretrained`. You can also provide your own `device_map` but it needs to be a dictionary module_name to device, so for instance {'transformer.h.0': 0, 'transformer.h.1': 1, ...}T)r1r2r3rr}rr%r#r,r4r5r(rrArr;r9r')rsr(s r6rAGPTJForCausalLM.parallelizes  -   ! 3t//112E%**:Q:Q:S4T U  $//3t/?/?/A/A+BC $$T__5||t'7'7'D'DE "r8c8[R"S[5 URR 5 URR S5UlUR R S5UlSUl[RR5 g)NrDr/F) r1r2r3rrGr;rr'r,r4rErKs r6rGGPTJForCausalLM.deparallelizesm j   &&(++..u5||u- #  r8cUR$rrrKs r6get_output_embeddings%GPTJForCausalLM.get_output_embeddingss ||r8cXlgrrrOs r6set_output_embeddings%GPTJForCausalLM.set_output_embeddingss% r8rSrrrTr@rrUlabelsrrrVrWrr%c U bU OURRn URUUUUUUUU U U U U S9 nUSnUR(ab[R R URR5 URURRR5nURU5R[R5nSnUb`URUR5nUR"UU4SURR0UD6nURUR5nU (dU4USS-nUbU4U-$U$[!UUUR"UR$UR&S9$)a inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) rrrTr@rrUrrrVrWrrrrlosslogitsrrr`)rWrarr'r,r4rgr9r;rrr<rj loss_functionrr*rrrr`)rsrSrrrTr@rrUrrrrVrWrrtransformer_outputsr lm_logitsroutputs r6rGPTJForCausalLM.forwards8&1%# UHn[U4SjU55v M g7f)c3x># UH/oRSTRUR55v M1 g7f)rN) index_selectr;r<)r\ past_staters r6r];GPTJForCausalLM._reorder_cache.../s1j_iQ[))!X[[9J9J-KLL_is7:Nrm)r\rrs r6r]1GPTJForCausalLM._reorder_cache...s' -  j_ij j j-s"%r)rrs `r6_reorder_cacheGPTJForCausalLM._reorder_cache%s -   r8)r(rr'rr) NNNNNNNNNNNNN)rcrrr_tied_weights_keysrVrrrArrGrrrrr,rrr rrrrrrrrrrrs@r6rrs ++ /0#1#$12 !3 !&15NR6:59371559-1$(,0/3&*59O E,,-O "%uU5<<5H/I(I"JKO !!2!23 O !!1!12 O u//0 O E--.O   1 12O ))*O D>O $D>O 'tnO d^O !!1!12O u,, -!O O b  uU\\23  ?D||  uU\\" #    r8ra The GPT-J Model transformer with a sequence classification head on top (linear layer). [`GPTJForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT, GPT-2, GPT-Neo) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c^\rSrSrU4Sjr\SS\\RS\\ \ \RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\ \ \44Sjj5rSrU=r$)GPTJForSequenceClassificationi4c>[TU]U5 URUl[U5Ul[ R "URURSS9UlSUl SUl UR5 g)NFrS) rUrV num_labelsrrrrmrscorer'r(r*rs r6rV&GPTJForSequenceClassification.__init__Csc   ++$V,YYv}}dooEJ $ r8rSrrrTr@rrUrrrrVrWr%c U bU OURRn URUUUUUUUU U U U S9 n U SnURU5nUbURSnOURSnURR cUS:wa [ S5eURR cSnOUbXRR :gRUR[R5n[R"URSUR[RS9nUU-RS5nO.Sn[RURR S35 U[R"UURS 9U4nSnUGbURUR5nURR"cUR$S:XaS URlOoUR$S:aNUR&[R(:XdUR&[R*:XaS URlOS URlURR"S :XaJ[-5nUR$S:Xa&U"UR/5UR/55nOU"UU5nOURR"S :Xa=[15nU"UR3SUR$5UR3S55nO-URR"S :Xa[55nU"UU5nU (dU4U SS-nUbU4U-$U$[7UUU R8U R:U R<S 9$)a inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N) rrrTr@rrUrrrVrWrrz=Cannot handle batch sizes > 1 if no padding token is defined.rD)r<r*z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rZ regressionsingle_label_classificationmulti_label_classificationr)rWrarrr> pad_token_idrhr;r<r,int32r-argmaxr`rarbrc problem_typerr*longrr squeezerr|rrrrr`)rsrSrrrTr@rrUrrrrVrWrrrrlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r6r%GPTJForSequenceClassification.forwardPs"4&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   YY}334F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r8)r(r'rrrr)rcrrrrVrrr,rrrrrrrrrrrs@r6rr4sL 15@D6:59371559-1$(,0/3&*c E,,-c "%ell(;"<=c !!2!23 c !!1!12 c u//0 c E--.c   1 12c ))*c D>c $D>c 'tnc d^c  u66 7c c r8rcr^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \ 44Sjj5rSrU=r$)GPTJForQuestionAnsweringic>[TU]U5 URUl[U5Ul[ R "URUR5UlSUl SUl UR5 gr) rUrVrrrrrmrd qa_outputsr'r(r*rs r6rV!GPTJForQuestionAnswering.__init__se   ++$V,))F$6$68I8IJ$ r8rSrrTr@rrUstart_positions end_positionsrrVrWr%c U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:a*UR S5RUR5n[UR55S:a*UR S5RUR5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS 9$) rYN)rrTr@rrUrrVrWrrrDr+) ignore_indexr()r start_logits end_logitsrr`)rWrarrrrrr}r{r;r<clamprrrr`)rsrSrrTr@rrUrrrrVrWrsequence_outputrrr total_loss ignored_indexr start_lossend_lossrs r6r GPTJForQuestionAnswering.forwards*&1%D 'tnD d^D  u22 3D D r8r)rrrrr)Jrr1typingrrrr,torch.fxtorch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrrmodeling_outputsrrrrmodeling_utilsrutilsrrrrrutils.model_parallel_utilsrrconfiguration_gptjr!torch.nn.attention.flex_attentionr integrations.flex_attentionr!r" get_loggerrcr`rrr7fxwraprArJrNModulerPrrrrrrrrrrr__all__rr8r6rs2)) AA!.)>h .L* !!;JJ   H %PP3P5<<P WW ==ELL=u||=X]XdXd= nBIInbM-Mb, bii&* *Z*/**B:(f#ffR  R )?R  R j  r $7r  r jS 2S S l r8