o Zh'@szdZddlmZmZddlZddlmZddlm Z Gdd d ej j j Z Gd d d ej j j ZGd d d ej j j ZdS)a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) shape_list) IdeficsConfigcs\eZdZdededededededdffd d Zfd d Zd ejdejfddZ Z S)TFIdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc stjd i|||||f\|_|_|_|_|jj|_t |j ds'|jdn|j jd|_ g|_ t |D]#}|j t|j|j|j|jd|ddt|j |d|ddgq5tjjjddd |_d S) ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). r zblocks.z.0namez.1h㈵> layer_normepsilonrN)super__init__r r r rZperceiver_configZqk_layer_norms_perceiverqk_layer_normshasattr vision_configZintermediate_dimblocksrangeappendTFIdeficsPerceiverAttention TFIdeficsMLPtfkeraslayersLayerNormalizationr) selfr r r r r rkwargsi __class__rW/var/www/auris/lib/python3.10/site-packages/transformers/models/idefics/perceiver_tf.pyr1s"     z$TFIdeficsPerceiverResampler.__init__cs,|j|j|jfdddd|_t|dS)NZ random_normalTlatents)shapeZ initializerZ trainabler)Z add_weightrr r,rbuild)r&Z input_shaper)rr+r.\sz!TFIdeficsPerceiverResampler.buildcontextcCs`tj|jdd}t|t|dddg}|jD]\}}||||}|||}q||S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsraxisr)r"Z expand_dimsr,Ztiler-rr)r&r/r,attnffrrr+callcs  z TFIdeficsPerceiverResampler.call) __name__ __module__ __qualname__rintrr.r"Tensorr4 __classcell__rrr)r+r0s$ +rc sNeZdZdededededdf fdd Zd ejd ejdejfd d ZZ S) r r r r rrNc stjdi|||||_|_|_||_tjjj ddd|_ tjjj ddd|_ |jrBtjjj ddd|_ tjjj ddd|_ |jd|_tjjj|j|jdd d |_tjjj|j|jdd d |_tjjj|j|jdd d |_tjjj|dd d |_dS)ziPerceiver Cross-Attention Module --> let long-form inputs be `context`, resampled embeddings be `latents`rcontext_layer_normrlatents_layer_norm q_layer_norm k_layer_normgFq_projZuse_biasrk_projv_proj output_projNr)rrr r r rr"r#r$r%r;r<r=r>qk_scaleDenser?rArBrC)r&r r r rr'r)rr+rps z$TFIdeficsPerceiverAttention.__init__r/r,c s |}|}t|\}}|}tj||gdd}tj||gdd}fdd|||fD\}}}jrK |} |}t d|j |}|tj |ddd} tjj| dd} t d | |} ttj| gd d djjfS) a= Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`tf.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`tf.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `tf.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. r0c s8g|]}tjt||jdjjfgddqS)rrrrperm)r" transposereshaper-r r ).0xZ batch_sizer&rr+ s*z4TFIdeficsPerceiverAttention.call..z... i d, ... j d -> ... i jT)r1Zkeepdimsz... i j, ... j d -> ... i drGrI)r;r<rr?rAr"concatrBrr=r>ZeinsumrDZ reduce_maxnnZsoftmaxrCrLrKr r ) r&r/r,Z seq_lengthr qkvZscoresZstabilized_scoresr2Z resampledrrOr+r4s&       &z TFIdeficsPerceiverAttention.call) r5r6r7r8boolrr"r9r4r:rrr)r+r os"$r cs@eZdZdeffdd ZdeeejdejfddZ Z S)r!r c sttjd i||jj|_tjjjddd|_tjjj |ddd|_ tjjj dd|_ tjjj |jdd d|_ d S) z:Simple MLP block with intermediate_size and embedding sizerlnrFfcr@actrc_projNr)rrrr r"r#r$r%rXrErYZReLUrZr[)r&Zintermediate_sizer r'r)rr+rs  zTFIdeficsMLP.__init__ hidden_statesrcCs,||}||}||}||}|S)N)rXrYrZr[)r&r\rrr+r4s    zTFIdeficsMLP.call) r5r6r7rrrrr"r9r4r:rrr)r+r!s& r!)__doc__typingrrZ tensorflowr"Zmodeling_tf_utilsrZconfiguration_ideficsrr#r$ZLayerrr r!rrrr+s  ?D