fTh'.SrSSKJrJr SSKrSSKJr SSKJ r "SS \RRR5r "S S \RRR5r"S S \RRR5rg)a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) shape_list) IdeficsConfigc^\rSrSrS\S\S\S\S\S\SS 4U4S jjrU4S jrS \RS\R4S jr Sr U=r $)TFIdeficsPerceiverResampler0config embed_dimdepthn_headshead_dim n_latentsreturnNc >[T U]"S 0UD6 X$XV4uUlUlUlUlUR RUl[URS5(dURS-OURRS-Ul /Ul [U5HnnURR[URURURURSUS3S9[!URUSUS3S9/5 Mp ["R$R&R)SSS 9Ulg ) a Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). r zblocks.z.0namez.1h㈵> layer_normepsilonrN)super__init__r rrrperceiver_configqk_layer_norms_perceiverqk_layer_normshasattr vision_configintermediate_dimblocksrangeappendTFIdeficsPerceiverAttention TFIdeficsMLPtfkeraslayersLayerNormalizationr) selfr r rrrrkwargsi __class__s `/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/idefics/perceiver_tf.pyr$TFIdeficsPerceiverResampler.__init__1s#( "6"FOZbFmC dmT^$55NN6//== NNQ %%//!3   uA KK  / dmmTEXEXahijhkkm_n!!6!6wqcQS_U  ((//<URURUR4SSSS9Ul[TU]U5 g)N random_normalTlatents)shape initializer trainabler) add_weightrr r6rbuild)r- input_shaper0s r1r;!TFIdeficsPerceiverResampler.build\s>>>4>>2[_fo'    k"r3contextc[R"URSS9n[R"U[R"U5SSS/5nUR Hup4U"X5U-nU"U5U-nM UR U5$)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsraxisr)r) expand_dimsr6tiler7r$r)r-r>r6attnffs r1call TFIdeficsPerceiverResampler.callcs{..A6'''BHHW$5a$8!Q#?@ HD7,w6GkG+G$w''r3) r$r rr#r6rrrr ) __name__ __module__ __qualname____firstlineno__rintrr;r)TensorrF__static_attributes__ __classcell__r0s@r1r r 0sn)^#)^03)^[TU]"S0UD6 XUsUlUlUlX@l[ RRRSSS9Ul [ RRRSSS9Ul UR (aZ[ RRRSSS9Ul [ RRRSSS9Ul URS-Ul[ RRRURUR-SS S 9Ul[ RRRURUR-SS S 9Ul[ RRRURUR-SS S 9Ul[ RRRUSS S 9Ulg)ziPerceiver Cross-Attention Module --> let long-form inputs be `context`, resampled embeddings be `latents`rcontext_layer_normrlatents_layer_norm q_layer_norm k_layer_normgFq_projuse_biasrk_projv_proj output_projNr)rrr rrr r)r*r+r,rTrUrVrWqk_scaleDenserXr[r\r])r-r rrr r.r0s r1r$TFIdeficsPerceiverAttention.__init__ps "6"6?(3 dm,"$((//"D"DTXl"D"m"$((//"D"DTXl"D"m    " B B4Vd B eD  " B B4Vd B eD  t+ hhoo++DLL4==,HSX_g+h hhoo++DLL4==,HSX_g+h hhoo++DLL4==,HSX_g+h 88??00UQ^0_r3r>r6c URU5nURU5n[U5up4nURU5nUR [ R "X/SS95nUR[ R "X/SS95nXgU4V s/sHSn [ R"[ R"XU RSURUR45/SQS9PMU sn upgnUR(a"URU5nURU5n[ R "SX`R"-U5n U [ R$"U SSS 9- n [ R&R)U SS9n [ R "S X5n UR+[ R"[ R"U /SQS9USURUR-455$s sn f) a Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`tf.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`tf.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `tf.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. r@r)rrr)permz... i d, ... j d -> ... i jT)rAkeepdimsz... i j, ... j d -> ... i d)rTrUrrXr[r)concatr\ transposereshaper7rrr rVrWeinsumr^ reduce_maxnnsoftmaxr])r-r>r6 batch_size seq_lengthr qkvxscoresstabilized_scoresrD resampleds r1rF TFIdeficsPerceiverAttention.calls))'2))'2,6w,?)  KK  KK 7"42> ? KK 7"42> ? AY  LLAAGGAJ dmm'\]dp q a   !!!$A!!!$A8!mm:KQO"R]]6T%RRuu}}.R}8II;TE  JJr||ILAJPRTXT`T`cgcpcpTpCq r   sAG,) rTr rrWr[rUrr]rVrXr r^r\) rHrIrJrKrLboolrr)rMrFrNrOrPs@r1r'r'os^`#``s`TX`gk`*+ BII+  + bii+ + r3r'cv^\rSrSrS\4U4SjjrS\\\RS\R4Sjr Sr U=r $)r(r c >[TU]"S 0UD6 URRUl[R R RSSS9Ul[R R RUSSS9Ul [R R RSS9Ul [R R RURSS S9Ul g ) z:Simple MLP block with intermediate_size and embedding sizerlnrFfcrYactrc_projNr)rrr"r r)r*r+r,r|r_r}ReLUr~r)r-intermediate_sizer r.r0s r1rTFIdeficsMLP.__init__s "6"--77((//44T4M((//''(9EPT'U88??''U'3hhoo++DNNUQY+Z r3 hidden_statesrcURU5nURU5nURU5nURU5nU$)N)r|r}r~r)r-rs r1rFTFIdeficsMLP.calls@ .  . /  M2 r3)r~rr r}r|) rHrIrJrKrrrrr)rMrFrNrOrPs@r1r(r(s;[-[(5+;"<r3r()__doc__typingrr tensorflowr)modeling_tf_utilsrconfiguration_ideficsrr*r+Layerr r'r(rr3r1rsj4 #+0<("((//"7"7<(~A "((//"7"7A H288??((r3