fTh$SrSSKJrJr SSKrSSKJr SSKJr "SS\R5r "SS \R5r "S S \R5r g) a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) IdeficsConfigc^\rSrSrS\S\S\S\S\S\SS 4U4S jjrS \RS\R4S jr S r U=r $)IdeficsPerceiverResampler0config embed_dimdepthn_headshead_dim n_latentsreturnNc ">[TU]5 X$XV4uUlUlUlUlUR RUl[R"[R"UR UR5SS9Ul [URS5(dURS-OURRS-Ul[R""[%U5Vs/sHcn[R""['URURURUR5[)UR U5/5PMe sn5Ul[R,"UR5Ulgs snf)a Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). T) requires_gradr N)super__init__r r rrperceiver_configqk_layer_norms_perceiverqk_layer_normsnn Parametertorchrandnlatentshasattr vision_configintermediate_dim ModuleListrangeIdeficsPerceiverAttention IdeficsMLPblocks LayerNorm layer_norm) selfr r r r rr_ __class__s ]/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/idefics/perceiver.pyr"IdeficsPerceiverResampler.__init__1s;( FOZbFmC dmT^$55NN||EKK$O_cd 6//== NNQ %%//!3  mmu &A  1$..$,,PTP]P]_c_r_rs"4#8#8&A &    ,,t~~6 s0A*F contextcURRURSSS5nURHup4U"X5U-nU"U5U-nM UR U5$)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsrr)rrepeatshaper%r')r(r-rattnffs r+forward!IdeficsPerceiverResampler.forward_se,,%%gmmA&61= HD7,w6GkG+G$w'') r%r rr rr'r rr) __name__ __module__ __qualname____firstlineno__rintrrTensorr3__static_attributes__ __classcell__r*s@r+rr0sa,7#,703,7[TU]5 XUsUlUlUlX@l[ R"UR5Ul[ R"UR5Ul UR (aJ[ R"UR5Ul [ R"UR5Ul URS-Ul [ R"URURUR-SS9Ul[ R"URURUR-SS9Ul[ R"URURUR-SS9Ul[ R"URUR-USS9Ulg)ziPerceiver Cross-Attention Module --> let long-form inputs be `context`, resampled embeddings be `latents`gFbiasN)rrr r rrrr&context_layer_normlatents_layer_norm q_layer_norm k_layer_normqk_scaleLinearq_projk_projv_proj output_proj)r(r r rrr*s r+r"IdeficsPerceiverAttention.__init__ms, 6?(3 dm,"$,,t~~">"$,,t~~">    " T]] ;D  " T]] ;D  t+ ii t}}0LSXY ii t}}0LSXY ii t}}0LSXY 99T\\DMM%A9SXYr5r-rc vURU5nURU5nURSSup4nURU5nUR [ R "X/SS95nUR[ R "X/SS95nXgU4V s/sHFoRX9RSURUR5RSS5PMH sn upgnUR(a"URU5nURU5n[ R"SX`R -U5n XR#SS S 9R%5- n U R'SS9n [ R"S X5n UR)U RSS5R+S55$s sn f) a Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`torch.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`torch.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `torch.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. N)dimrz... i d, ... j d -> ... i jT)rRkeepdimz... i j, ... j d -> ... i d)rDrEr0rJrKrcatrLreshaper r transposerrFrGeinsumrHamaxdetachsoftmaxrMflatten)r(r-r batch_size seq_lengthr qkvxscoresstabilized_scoresr1 resampleds r+r3!IdeficsPerceiverAttention.forwards))'2))'2,3MM"1,=)  KK  KK 7"4"= > KK 7"4"= > mnrsktuktfg99ZT\\4==Q[[\]_`aktua   !!!$A!!!$A;Q=NPQR"kkb$k&G&N&N&PQ ((R(0LL!>H  3 3Aq 9 A A" EFFvsA F6) rDr rrGrKrEr rMrFrJrrHrL) r6r7r8r9r:boolrrr;r3r<r=r>s@r+r#r#lscZ#ZZsZTXZ]aZ*(Gu||(Gell(Gu||(G(Gr5r#cv^\rSrSrS\4U4SjjrS\\\RS\R4Sjr Sr U=r $)r$r ch>[TU]5 URRUl[R "UR5Ul[R"URUSS9Ul[R"5Ul [R"XRSS9Ul g)z:Simple MLP block with intermediate_size and embedding sizeFrBN) rrrr rr&lnrIfcReLUactc_proj)r(intermediate_sizer r*s r+rIdeficsMLP.__init__sr --77,,t~~.))DNN,=EJ779ii 1>>N r5 hidden_statesrcURU5nURU5nURU5nURU5nU$)N)rlrmrorp)r(rss r+r3IdeficsMLP.forwards@ .  . /  M2 r5)rorpr rmrl) r6r7r8r9rrrrr FloatTensorr3r<r=r>s@r+r$r$s?O-OXeE4E4E.F%GEL]L]r5r$) __doc__typingrrrtorch.nnrconfiguration_ideficsrModulerr#r$r5r+r}sL4 # 09( 9(x>G >GBr5