fTh S r SSKrSSKJr SSKJrJrJrJ r SSK
r SSK r SSK
r SSK Jr SSKJr SSKJrJr SS KJrJr SS
KJr SS KJrJrJ r J r S S
K J r J r J!r! \ RD " \#5 r$S\ RJ S\ RJ 4S jr&S\ RJ S\ RJ 4S jr'S\ RJ S\(4S jr)SCS\ RJ S\*S\+S\(S\ RJ 4
S jjr,SDS jr-S r. " S S\R^ 5 r0 " S S \R^ 5 r1 " S S \R^ 5 r2\ " S S!\5 5 r3 " S" S#\R^ 5 r4 " S$ S%\R^ 5 r5 " S& S'\R^ 5 r6 " S( S)\R^ 5 r7 " S* S+\R^ 5 r8 " S, S-\85 r9 " S. S/\R^ 5 r: " S0 S1\R^ 5 r;\ " S2 S3\5 5 r< " S4 S5\R^ 5 r= " S6 S7\R^ 5 r> " S8 S9\R^ 5 r? " S: S;\<5 r@ " S< S=\R^ 5 rA " S> S?\<5 rB\ " S@ SA\<5 5 rC/ SBQrDg)EzPyTorch GroupViT model. N) dataclass)AnyOptionalTupleUnion)nn )ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging torch_int )GroupViTConfigGroupViTTextConfigGroupViTVisionConfiglogitsreturnc [ R R U [ R " [ U 5 U R S95 $ )Ndevice)r
functional
cross_entropytorcharangelenr )r s f/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/groupvit/modeling_groupvit.pycontrastive_lossr# ' s/
== & &vu||CKPVP]P]/^ __
similarityc X [ U 5 n[ U R 5 5 nX- S- $ )Ng @)r# t)r% caption_loss
image_losss r"
groupvit_lossr* , s* #J/L!*,,.1J % ,,r$ dimc U R U5 nUR USS9S n[ R " U [ R S9R XS5 nXBR
5 -
U- nU$ )NTkeepdimr
memory_format ?)softmaxmaxr
zeros_likelegacy_contiguous_formatscatter_detach)r r+ y_softindexy_hardrets r" hard_softmaxr< 2 sf
^^C
F JJsDJ )! ,E
fE4R4R
S
\
\]`il
mF
==?
"V
+C Jr$ tauhardc , [ R R R [ R " SU R
U R S9[ R " SU R
U R S95 nUR U R 5 nX- U- nUR U5 nU( a] UR USS9S n[ R " U [ R S9R X7S5 nXR 5 -
U- n U $ Un U $ )N )r dtyper1 Tr- r r/ )r
distributionsgumbelGumbeltensorr rA sampleshaper2 r3 r4 r5 r6 r7 )
r r= r> r+ gumbel_distgumbelsr8 r9 r: r; s
r" gumbel_softmaxrJ < s %%,,33
S fllC
S fllCK .G 3&G
__S
!F
3
-a0!!&8V8VW``admpq}}&/ J Jr$ c X- U R S - S- nX: a4 [ [ R " X$- 5 5 nU R S U- nO3[ [ R " X- 5 5 nU R S U- nU R S nU R S nU R XxXe5 n [
R R XU4SUS9n U $ )a
Args:
attentions (`torch.Tensor`): attention map of shape [batch_size, groups, feat_height*feat_width]
height (`int`): height of the output attention map
width (`int`): width of the output attention map
align_corners (`bool`, *optional*): the `align_corner` argument for `nn.functional.interpolate`.
Returns:
`torch.Tensor`: resized attention map of shape [batch_size, groups, height, width]
?r r bilinearsizemode
align_corners)rG intnproundreshaper r interpolate)
attentionsheightwidthrR scale
feat_width feat_height
batch_sizegroupss r" resize_attention_mapr` R s ^z / / 2
2s :E
~%-01
&&q)Z7 "((6>23 %%a(K7
!!!$J
a
F##J PJ **% z + J r$ c V / n[ R " 5 SnU Hj nUR SSS5 R 5 nUc UnOX4- n[ UR SSS5 R 5 /UQ76 nUR U5 Ml SSS5 US nU$ ! , ( d f N= f)a
Args:
attentions (`tuple(torch.FloatTensor)`: tuple of attention maps returned by `GroupViTVisionTransformer`
hw_shape (`tuple(int)`): height and width of the output attention map
Returns:
`torch.Tensor`: the attention map of shape [batch_size, groups, height, width]
Nr rL r )r no_gradpermute
contiguousr` append)rX hw_shape attn_mapsprev_attn_masks
attn_masks cur_attn_mapfinal_groupings r" get_grouping_from_attentionsrm p s I $J#++Aq!4??AJ &","1">/0G0G1a0P0[0[0]i`hiL \ * %
r]N !
s A3B
B(c 6 ^ \ rS rSrS\4U 4S jjrS rSrU =r$ )GroupViTCrossAttentionLayer configc > [ TU ] 5 [ U5 U l [ R
" UR UR S9U l [ U5 U l
[ R
" UR UR S9U l g Neps) super__init__GroupViTAttentionattnr LayerNorm hidden_sizelayer_norm_epsnorm2 GroupViTMLPmlp norm_postselfrq __class__s r" rw $GroupViTCrossAttentionLayer.__init__ sb
%f- \\&"4"4&:O:OP
v& f&8&8f>S>STr$ c UnX0R XS9S - nX0R U R U5 5 - nU R U5 nU$ )N)encoder_hidden_statesr ry r r} r )r querykeyxs r" forward#GroupViTCrossAttentionLayer.forward sK
% ;A> >
A ' ' NN1 r$ r ) __name__
__module__ __qualname____firstlineno__r rw r __static_attributes__
__classcell__r s @r" ro ro s U 3 U r$ ro c @ ^ \ rS rSrS\4U 4S jjrSS jrS rSrU =r $ )GroupViTAssignAttention rq c > [ TU ] 5 UR S- U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l UR U l
g )N ) rv rw r{ r[ r Linearq_projk_projv_projproj
assign_epsr s r" rw GroupViTAssignAttention.__init__ s
''-
ii 2 2F4F4FG ii 2 2F4F4FG ii 2 2F4F4FG IIf 0 0&2D2DE ++r$ c U( a U R ( a
[ USUS9nU$ U( a [ USS9nU$ [ R R USS9nU$ )N)r+ r> r+ )trainingrJ r< r r r2 )r ry rC r> s r" get_attn GroupViTAssignAttention.get_attn sX dmm!$BT:D #Db1 }},,Tr,: r$ c ` UnU R U5 nU R U5 nU R U5 nXR SS5 - U R - nU R U5 nU R USSS9nXUR
SSS9U R - - nXS- nU R U5 nXv4$ )Nr rb F)rC r> Tr+ r. ) r r r transposer[ r sumr r )r r r valueraw_attnry soft_attnouts r" r GroupViTAssignAttention.forward s E" kk# E" MM"b 11TZZ?}}X&MM(5uME xxBx5GHliin~ r$ )r r r r r[ r )TT)
r r r r r rw r r r r r s @r" r r s , 3 , r$ r c < ^ \ rS rSrS\4U 4S jjrS rS rSrU =r $ )GroupViTTokenAssign rq c V > [ TU ] 5 X0l [ R " UR
UR S9U l [ UR [ R R 5 ( a UR OUR UR 4nU Vs/ s H n[ XQR
- 5 PM snu pg[ XXc5 U l [ R " UR
UR S9U l [ R " UR
UR S9U l [% U5 U l [) U5 U l [ R " UR
UR S9U l [/ XR
XqR
5 U l g s snf rs )rv rw num_output_groupr rz r{ r| norm_tokens
isinstanceassign_mlp_ratio collectionsabcIterablerS GroupViTMixerMLP mlp_internorm_post_tokensnorm_xro pre_assign_attnr assign
norm_new_xr~ mlp_channels) r rq num_group_tokenr r r
tokens_dim channels_dimr s r" rw GroupViTTokenAssign.__init__ sD
0 <<(:(:@U@UV &11;??3K3KLL
# #))6+B+BC
JZ#ZIYAC,>,>(>$?IY#Z
)&:` " V-?-?VEZEZ [ ll6#5#56;P;PQ :6 B -f5 ,,v'9'9v?T?TU '0B0BLRdRd e $[s !F&c J U R U5 nU R U5 nU$ )z
Args:
group_tokens (torch.Tensor): group tokens, [batch_size, num_group_tokens, channels]
Returns:
projected_group_tokens (torch.Tensor): [batch_size, num_output_groups, channels]
)r r )r group_tokensprojected_group_tokenss r" project_group_token'GroupViTTokenAssign.project_group_token s+ "& != !%!6!67M!N %%r$ c U R U5 nU R U5 nU R U5 nU R X15 nU R X15 u pEXC-
nX@R U R
U5 5 - nXE4$ )z
Args:
image_tokens (`torch.Tensor`): image tokens, of shape [batch_size, input_length, channels]
group_tokens (`torch.Tensor`): group tokens, [batch_size, num_group_tokens, channels]
)r r r r r r r )r image_tokensr r new_image_tokens attentions r" r GroupViTTokenAssign.forward s '' 5 {{<0 !%!9!9,!G !%!5!56L![ &*kk2H&W#2+.?.?P`@a.bb **r$ ) r r r r r r r r r )
r r r r r rw r r r r r s @r" r r s! f 3 f* &+ +r$ r c : \ rS rSr% SrSr\\R \ S' Sr
\\R \ S' Sr \\R \ S' Sr \\R \ S' Sr
\\R \ S' Sr\\R \ S ' Sr\\ S
' Sr\\ S ' S \\ 4S
jrSrg)GroupViTModelOutputi a
Args:
loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`):
Contrastive loss for image-text similarity.
logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`):
The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text
similarity scores.
logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`):
The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image
similarity scores.
segmentation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels, logits_height, logits_width)`):
Classification scores for each pixel.
The logits returned do not necessarily have the same size as the `pixel_values` passed as inputs. This is
to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
The text embeddings obtained by applying the projection layer to the pooled output of
[`GroupViTTextModel`].
image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
The image embeddings obtained by applying the projection layer to the pooled output of
[`GroupViTVisionModel`].
text_model_output (`BaseModelOutputWithPooling`):
The output of the [`GroupViTTextModel`].
vision_model_output (`BaseModelOutputWithPooling`):
The output of the [`GroupViTVisionModel`].
Nlosslogits_per_imagelogits_per_textsegmentation_logits text_embeds image_embedstext_model_outputvision_model_outputr c J ^ [ U 4S jT R 5 5 5 $ )Nc 3 n ># U H* nUS ; a TU O[ TU5 R 5 v M, g7f))r r N)getattrto_tuple).0kr s r" /GroupViTModelOutput.to_tuple..1 s<
LLDGRYZ^`aRbRkRkRm m s 25)tuplekeysr s `r" r GroupViTModelOutput.to_tuple0 s#
YY[
r$ )r r r r __doc__r r r FloatTensor__annotations__r r r r r r r r r r r r r r$ r" r r s B )-D(5$$
%,48hu 0 01837OXe / /077;%"3"34;/3K%++,304L(5,,-448186:3:
%*
r$ r c ^ \ rS rSrSr S
S\S\\\\\4 4 S\S\4U 4S jjjrSS\ R S \ S
\ R 4S jjr S r
U =r$ )GroupViTPatchEmbeddingsi7 z
Image to Patch Embedding.
image_size
patch_size num_channels embed_dimc ` > [ TU ] 5 [ U[ R R
5 ( a UOX4n[ U[ R R
5 ( a UOX"4nUS US - US US - - nXl X l XPl [ R " X4X"S9U l g )Nr r ) kernel_sizestride) rv rw r r r r r r num_patchesr Conv2d
projection)r r r r r r r s r" rw GroupViTPatchEmbeddings.__init__< s #-j+//:R:R#S#SZZdYq
#-j+//:R:R#S#SZZdYq
!!}
1
5*Q-:VW=:XY $$& ))Lgr$ pixel_valuesinterpolate_pos_encodingr c > UR u p4pVU( dV XPR S :w d X`R S :w a2 [ SU SU SU R S SU R S S3 5 eU R U5 R S5 R SS5 nU$ )Nr r zInput image size (*z) doesn't match model ().rL )rG r
ValueErrorr flattenr )r r r r^ r rY rZ r s r" r GroupViTPatchEmbeddings.forwardM s 2>2D2D/
&' ++u8J/J (% 9 +,Adooa.@-AE
OOL ) 1 1! 4 > >q! Dr$ )r r r r ) r i F)r r r r r rS r r rw r Tensorboolr r r r s @r" r r 7 s 24 hh #uS#X./h h
h h" ELL D ]b]i]i r$ r c ^ \ rS rSrS\4U 4S jjrS\R S\S\S\R 4S jr S
S \R S
\
S\R 4S jjr S r U =r
$ )GroupViTVisionEmbeddingsiY rq c > [ TU ] 5 [ UR UR UR
UR S9U l U R R n[ R " [ R " SX!R 5 5 U l
[ R " UR 5 U l [ R " UR UR" S9U l UR U l Xl g )N)r r r r r rt )rv rw r r r r r{ patch_embeddingsr r Parameterr zerosposition_embeddingsDropoutdropoutrz r| layernormrq )r rq r r s r" rw !GroupViTVisionEmbeddings.__init__Z s
7 (( (( ,, (( !
++77 #%<< A{L^L^0_#` zz&..1 f&8&8f>S>ST ++ r$
embeddingsrY rZ r c , UR S nU R R S n[ R R 5 ( d XE:X a X#:X a U R $ U R nUR S nX R
- nX0R
- n [
US- 5 n
UR SXU5 nUR SSSS5 n[ R R UX4SSS 9nUR SSSS5 R SSU5 nU$ )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing and no class embeddings.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r rb rM r r rL bicubicFrO )
rG r
r jit
is_tracingr r rV rd r r rW view) r r rY rZ r
num_positionspatch_pos_embedr+
new_height new_widthsqrt_num_positionss r" r 1GroupViTVisionEmbeddings.interpolate_pos_encodingj s !&&q) 0066q9
yy##%%+*F6?++ + 22 r".
__, &}c'9 :)11!5G]`a)11!Q1= --33 ( 4
*11!Q1=BB1b#N r$ r r c UR u p4pVU R XS9nU R U5 nUR 5 u p8n U( a XpR XuU5 - nOXpR
- nU R
U5 nU$ )N)r )rG r r
rP r r
r )
r r r r^ r rY rZ r seq_len_s
r" r GroupViTVisionEmbeddings.forward s 2>2D2D/
&**<*k
^^J/
!+ !2
Q $#&C&CJX]&^^J#&>&>>J\\*-
r$ )rq r r
r r r
r )r r r r r rw r r rS r r r r r r s @r" r r Y sh 3 $ 5<< $ $ UX $ ]b]i]i $ LELL D ]b]i]i r$ r c ^ \ rS rSrS\4U 4S jjr S
S\\R S\\R S\\R S\R 4S jjr S r U =r
$ ) GroupViTTextEmbeddingsi rq c N > [ TU ] 5 UR n[ R " UR
U5 U l [ R " UR U5 U l U R S[ R " UR 5 R S5 SS9 g )N position_ids)r rb F)
persistent)
rv rw r{ r Embedding
vocab_sizetoken_embeddingmax_position_embeddingsposition_embeddingregister_bufferr r expandr rq r r s r" rw GroupViTTextEmbeddings.__init__ s
&& !||F,=,=y I "$,,v/M/My"Y
ELL)G)G H O OPW Xej
r$ input_idsr"
inputs_embedsr c < Ub UR S OUR S nU R R R S nXE: a [ SU SU 35 eUc U R S S 2S U24 nUc U R U5 nU R U5 nX6- nU$ )Nrb r r zRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rG r( weightr r" r& )r r- r" r.
seq_lengthmax_position_embeddingr
r s r" r GroupViTTextEmbeddings.forward s -6,AY__R(}GZGZ[]G^
!%!8!8!?!?!E!Ea!H . d , <=S [ TU ] 5 X l X@l US: a; [ R
" [ R " SXAR 5 5 U l OS U l [ R " [ U5 Vs/ s H n[ U5 PM sn5 U l
US: a [ UUUS9U l OS U l US: ab US: a\ [ R " [ R" " UR UR$ S9[' XUR S- U5 5 U l g S U l g s snf )Nr r )rq r r rt rL )rv rw r8 r r r r r r{ group_token
ModuleListrangeGroupViTEncoderLayerlayersr
downsample
Sequentialrz r| r group_projector)r rq r8 r9 r r r r s r" rw GroupViTStage.__init__ s
. Q !||EKK?L^L^,_ `D #D mm5QV<$X '
u ! ' 'r$ r7 c
^ \ rS rSr S S\S\\ S\\ S\\ 4U 4S jjjrS\R S\R 4S jr
S
r U =r $ ) r~ i* rq r{ intermediate_size output_sizec > [ TU ] 5 Xl [ UR U l Ub UO UR nUb UO UR nUb UOUn[ R " X#5 U l
[ R " X45 U l g rE ) rv rw rq r
hidden_act
activation_fnr{ r] r r fc1fc2)r rq r{ r] r^ r s r" rw GroupViTMLP.__init__+ s| #F$5$5 6%0% [ TU ] UR SS5 5 nUR SS5 $ Nr rL )rv r r )r r r s r" r GroupViTMixerMLP.forwardC s- GOAKK1 - .{{1a r$ r )r r r r r r r r s @r" r r B s
! !r$ r c F ^ \ rS rSrSrU 4S jrS\R S\S\4S jr SS\R S \
\R S
\
\R S \
\R S \
\ S
\
\R \
\R \
\
\R 4 4 S jjrSrU =r$ )rx iH z=Multi-headed attention from 'Attention Is All You Need' paperc > [ TU ] 5 Xl UR U l UR
U l U R U R - U l U R U R - U R :w a& [ SU R SU R S35 eU R S- U l UR U l [ R " U R U R 5 U l [ R " U R U R 5 U l [ R " U R U R 5 U l [ R " U R U R 5 U l g )Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: r r )rv rw rq r{ r num_attention_heads num_headshead_dimr r[ attention_dropoutr r r r r r out_projr s r" rw GroupViTAttention.__init__K s
++ 33 $..8
==4>> )T^^ ; MdnnM] ^NN#2'
]]D(
// ii? ii? ii? $..$..A
r$ rE r bszc UR X2U R U R 5 R SS5 R 5 $ ri )r rn ro r re )r rE r rs s r" _shapeGroupViTAttention._shape^ s5 {{3 GQQRSUVWbbddr$ rO rS rT r rQ r c UR 5 u pgnUSLn U R U5 U R - n
U ( aE U R U R U5 SU5 n U R U R U5 SU5 n ODU R U R U5 SU5 n U R U R U5 SU5 n X`R - SU R 4n
U R XU5 R " U
6 n
U R " U
6 n U R " U
6 n U R S5 n[ R " XR SS5 5 nUR 5 X`R - X~4:w a- [ SX`R - X~4 SUR 5 35 eUbv UR 5 USX~4:w a" [ SUSX~4 SUR 5 35 eUR X`R X~5 U- nUR X`R - X~5 nUbv UR 5 USX~4:w a" [ SUSX~4 SUR 5 35 eUR X`R X~5 U- nUR X`R - X~5 n[ R R USS9nU( a; UR X`R X~5 nUR X`R - X~5 nOSn[ R R! XR U R" S 9n[ R " UU 5 nUR 5 X`R - XpR 4:w a5 [ S
X`R XpR 4 SUR 5 35 eUR X`R XpR 5 nUR SS5 nUR% XgU5 nU R' U5 nUU4$ ) z#Input shape: Batch x Time x ChannelNrb r rL z$Attention weights should be of size z , but is z!Attention mask should be of size r )pr z `attn_output` should be of size )rP r r[ ru r r rn ro r r bmmr r r r r2 r r rV rq )r rO rS rT r rQ rs tgt_lenr is_cross_attention query_states
key_states value_states
proj_shapesrc_len attn_weightsattn_weights_reshaped
attn_probs attn_outputs r" r GroupViTAttention.forwarda s #0"4"4"6 i 2$ > {{=1DJJ> T[[1F%GSQJ ;;t{{3H'I2sSL T[[ %?SIJ ;;t{{='A2sKLNN*B
>
{{<#>CCZP __j1
#((*5 //!$ yy /C/CAq/IJ 3 #7"J J 6nn8Lg7_6` a %%'(*
! ,$))+Q/II 7a8R7S T-22457 (,,S..'SVkkL',,S>>-A7TL % ""$a(BB 7a8R7SS\]k]p]p]r\st (,,S..'SVddL',,S>>-A7TL}},,\r,B
%1$5$5c>>7$\ !055cNN6JG]L$( !]]**<<
R2
u||Xell 3XeELL>Q5RR SR2 R2r$ rx c ^ \ rS rSrS\4U 4S jjr S S\R S\R S\R S\\ S\
\R 4
S jjr S
r
U =r$ ) r> i rq c < > [ TU ] 5 UR U l [ U5 U l [ R " U R UR S9U l [ U5 U l [ R " U R UR S9U l g rs )
rv rw r{ r rx self_attnr rz r| layer_norm1r~ r layer_norm2r s r" rw GroupViTEncoderLayer.__init__ sm
++*62 <<F r> sk S~ S -2 & ||& & %|| &
$D> &
u !
& &r$ r> c & \ rS rSr\rSrSrS rSr g)GroupViTPreTrainedModeli groupvitTc b U R R n[ U[ R [ R
45 ( aV UR R R SUS9 UR b$ UR R R 5 Oh[ U[ R 5 ( aI UR R R 5 UR R R S5 U R R n[ U[ 5 ( ac UR R R R SUS- S9 UR R R R SUS- S9 g[ U[" 5 ( Ga U R R nUR$ S- SUR R& - S- - U- nUR$ S- U- n[ R( R UR* R US9 [ R( R UR, R US9 [ R( R UR. R US9 [ R( R UR0 R US9 g[ U[2 5 ( a U R R nUR R4 S- SUR R& - S- - U- nSUR R4 - S- U- n[ R( R UR6 R US9 [ R( R UR8 R US9 gg) zInitialize the weightsr@ )meanstdNr1 g{Gz?r rL )r ) rq initializer_ranger r r r r0 datanormal_biaszero_rz fill_initializer_factorr r& r( rx r num_hidden_layersinitr r r rq r~ r{ rb rc )r module
init_rangefactor in_proj_std out_proj_stdfc_stds r"
_init_weights%GroupViTPreTrainedModel._init_weights s [[22
fryy")) 4 5 5
MM & &CZ & @{{& &&(
-
- KK " " $ MM $ $S )// f 4 5 5 " " ) ) . . 6 6CVd] 6 S % % , , 1 1 9 9sQU
9 V
1
2
2[[33F!++T1q6==;Z;Z7Z_c6cdgmmK " , ,d 2f [ TU ] 5 Xl [ R " [ [
UR 5 5 Vs/ s HO n[ UUR U UR U UR U US: a UR US-
OSS9PMQ sn5 U l SU l g s snf )Nr r )rq r8 r r r9 F)
rv rw rq r r< r= r! depthsr7 num_group_tokensnum_output_groupsstagesgradient_checkpointing)r rq ir s r" rw GroupViTVisionEncoder.__init__ s
mm s6== 12
3A ! --*$*$;$;A$>%+%=%=a%@LMPQE)A)A!a%)HWX 3
',#
s AB0rO output_hidden_statesrQ return_dictc Ub UOU R R nUb UOU R R nUb UOU R R nU( a SOS nU( a SOS nS n[ U R
5 H= u pU( a XQ4- nU " XU5 n
U
S nU
S nU( d M- U
S c M5 XjS 4- nM? U( a XQ4- nU( d [
S XU4 5 5 $ [ XUS9$ )Nr r r rL c 3 . # U H oc M Uv M
g 7frE r r vs r" r 0GroupViTVisionEncoder.forward..E s g$Uq$U last_hidden_staterO rX )rq rQ r use_return_dict enumerater r r
) r rO r rQ r all_hidden_states
all_groupingsr r stage
layer_outputss r" r GroupViTVisionEncoder.forward# s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% %
d^ %
uo% &
%
%
r$ r c ^ \ rS rSrSrS\4U 4S jjr S
S\\R S\\R S\\
S\\
S \\
S
\ \ \
4 4 S jjrS rU =r$ )GroupViTTextEncoderiK z
Transformer encoder consisting of `config.num_hidden_layers` self-attention layers. Each layer is a
[`GroupViTEncoderLayer`].
Args:
config: GroupViTTextConfig
rq c > [ TU ] 5 Xl [ R " [ UR 5 Vs/ s H n[ U5 PM sn5 U l SU l g s snf rZ )
rv rw rq r r< r= r r> r? r )r rq r r s r" rw GroupViTTextEncoder.__init__T sT
mm5QWQiQiKj$kKja%9&%AKj$kl &+# %ls A&rS rT rQ r r r c L Ub UOU R R nUb UOU R R nUb UOU R R nU( a SOSnU( a SOSnUn [ U R
5 Hr u pU( a Xy4- nU R ( a1 U R ( a U R U R U UUU5 n O U " U UUUS9n U S n U( d Mj XS 4- nMt U( a Xy4- nU( d [ S XU4 5 5 $ [ XUS9$ )a
Args:
inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.
This is useful if you want more control over how to convert `input_ids` indices into associated vectors
than the model's internal embedding lookup matrix.
attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Causal mask for the text model. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
output_attentions (`bool`, *optional*):
Whether or not to return the attentions tensors of all attention layers. See `attentions` under
returned tensors for more detail.
output_hidden_states (`bool`, *optional*):
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
for more detail.
return_dict (`bool`, *optional*):
Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
Nr )rQ r r c 3 . # U H oc M Uv M
g 7frE r r s r" r .GroupViTTextEncoder.forward.. s e$Sq$Sr r ) rq rQ r r r r? r r _gradient_checkpointing_func__call__r r
)
r r. rS rT rQ r r encoder_statesall_attentionsrO idx
encoder_layerr s
r" r GroupViTTextEncoder.forwardZ s8 L 2C1N -TXT_T_TqTq$8$D $++JjJj &1% >Ne]N$See e +Vd
r$ )rq r r? )NNNNN)r r r r r r rw r r r r r r r
r r r r s @r" r r K s , 1 , 268<,0/3&*O
! .O
( 5 O
$D> O
'tn
O
d^O
uo% &O
O
r$ r c ^ \ rS rSrS\4U 4S jjr\ S
S\\R S\\R S\\R S\\
S\\
S \\
S
\ \ \
4 4S jj5 rS rU =r$ )GroupViTTextTransformeri rq c > [ TU ] 5 Xl UR n[ U5 U l [
U5 U l [ R " X!R S9U l UR U l g rs )
rv rw rq r{ r r r encoderr rz r| final_layer_norm eos_token_idr+ s r" rw GroupViTTextTransformer.__init__ s]
&& 08*62 " Y L
'tn
L
d^L
u00 1L
L
r$ r c ^ \ rS rSr\rS\4U 4S jjrS\R 4S jr S r
\ SS\ \
R S\ \
R S \ \
R S
\ \ S \ \ S \ \ S\\\4 4S
jj5 rSrU =r$ )GroupViTTextModeli rq c d > [ TU ] U5 [ U5 U l U R 5 g rE )rv rw r
text_model post_initr s r" rw GroupViTTextModel.__init__ s&
1&9 r$ r c B U R R R $ rE r r r& r s r" get_input_embeddings&GroupViTTextModel.get_input_embeddings s ))999r$ c 8 XR R l g rE r )r r s r" set_input_embeddings&GroupViTTextModel.set_input_embeddings s 5:""2r$ r- rS r" rQ r r c * U R UUUUUUS9$ )a
Examples:
```python
>>> from transformers import CLIPTokenizer, GroupViTTextModel
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTTextModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output # pooled (EOS token) states
```r- rS r" rQ r r r )r r- rS r" rQ r r s r" r GroupViTTextModel.forward s, 2 )% /!5#
r$ r r )r r r r r r rw r Moduler r r r r r r r r r r r r r s @r" r r s %L 1 :bii :; -115/3,0/3&*
ELL)
! .
u||,
$D>
'tn
d^
u00 1
r$ r c ^ \ rS rSrS\4U 4S jjr\ S S\\R S\\
S\\
S\\
S\ \ \
4 4
S jj5 rS
rU =r$ ) GroupViTVisionTransformeri: rq c > [ TU ] 5 Xl UR n[ U5 U l [
U5 U l [ R " X!R S9U l g rs ) rv rw rq r{ r r r r r rz r| r
r+ s r" rw "GroupViTVisionTransformer.__init__; sL
&& 26:,V4 i5J5JKr$ r r rQ r r c Ub UOU R R nUb UOU R R nUb UOU R R nUc [ S5 eU R U5 nU R
UUUUS9nUS nU R U5 nUR SS9nU( d Xx4USS - $ [ UUUR UR S9$ )Nz You have to specify pixel_values)rO r rQ r r r r r ) rq rQ r r r r r r
r r rO rX ) r r r rQ r rO r r r s r" r !GroupViTVisionTransformer.forwardD s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% '
d^ '
u00 1
'
'
r$ r c ^ \ rS rSr\rSrS\4U 4S jjrS\4S jr \
S S\ \ R S\ \ S\ \ S \ \ S\\\4 4
S
jj5 rS rU =r$ )
GroupViTVisionModelio r rq c d > [ TU ] U5 [ U5 U l U R 5 g rE )rv rw r vision_modelr r s r" rw GroupViTVisionModel.__init__s s'
5f = r$ r c B U R R R $ rE )r r r r s r" r (GroupViTVisionModel.get_input_embeddingsy s ++<<>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTVisionModel
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTVisionModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output # pooled CLS states
```r rQ r r r )r r rQ r r s r" r GroupViTVisionModel.forward| s( 8 % /!5# !
r$ r r )r r r r r r main_input_namerw r r r r r r r r r r r r r r s @r" r r o s 'L$O 3 =&= = 59,0/3&*
u 0 01
$D>
'tn
d^
u00 1
r$ r c > ^ \ rS rSr\rS\4U 4S jjr\ SS\\ R S\\ R S\\ R S\\ S\\ S \\ S
\ R 4S jj5 r
\ SS \\ R S\\ S\\ S \\ S
\ R 4
S
jj5 r\ SS\\ R S \\ R S\\ R S\\ R S\\ S\\ S\\ S\\ S \\ S
\\\4 4S jj5 rSrU =r$ )
GroupViTModeli rq c > > [ TU ] U5 [ UR [ 5 ( d" [ S[
UR 5 S35 e[ UR [ 5 ( d" [ S[
UR 5 S35 eUR nUR nUR U l UR U l
UR U l UR U l
[ U5 U l [! U5 U l [$ R& " [$ R( " U R U R SS9[$ R* " U R 5 [$ R, " SS9[$ R( " U R U R SS95 U l [$ R& " [$ R( " U R U R SS9[$ R* " U R 5 [$ R, " SS9[$ R( " U R U R SS95 U l [$ R2 " [4 R6 " U R8 R: 5 5 U l U R? 5 g )NzOconfig.text_config is expected to be of type GroupViTTextConfig but is of type .zSconfig.vision_config is expected to be of type GroupViTVisionConfig but is of type T)r )inplace) rv rw r text_configr TypeErrortype
vision_configr projection_dimprojection_intermediate_dimr{ text_embed_dimvision_embed_dimr r r r r rA r BatchNorm1dReLUvisual_projectiontext_projectionr r rE rq logit_scale_init_value logit_scaler )r rq r
r r s r" rw GroupViTModel.__init__ s
&,,.@AA++,-Q0
&..0DEE--./q2
(( ,,
$ 3 3+1+M+M( ) 5 5 - 9 9 1+> 5m D!# IId++T-M-MTX Y NN4;; < GGD ! IId668K8KRV W "
"}} IId))4+K+KRV W NN4;; < GGD ! IId668K8KRV W
<< T[[5W5W(XY
r$ r- rS r" rQ r r r c Ub UOU R R nUb UOU R R nUb UOU R R nU R UUUUUUS9nUS nU R U5 n U $ )aF
Returns:
text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by
applying the projection layer to the pooled output of [`GroupViTTextModel`].
Examples:
```python
>>> from transformers import CLIPTokenizer, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)
```r r )rq rQ r r r r )
r r- rS r" rQ r r text_outputsr
text_featuress
r" get_text_features GroupViTModel.get_text_features s 6 2C1N -TXT_T_TqTq$8$D $++JjJj &1%>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features(**inputs)
```r r )rq rQ r r r r )r r rQ r r vision_outputsr image_featuress r" get_image_features GroupViTModel.get_image_features s > 2C1N -TXT_T_TqTq$8$D $++JjJj &1% r$ return_lossoutput_segmentationc
Ub UOU R R nUb UOU R R nU( a SnUb UOU R R nU b U OU R R n U R UUUU S9n
U R
UUUUUU S9n U
S n U R U 5 n U S n
U R U
5 n
XR SSS9- n XR SSS9- n
U R R 5 n[ R " XR 5 5 U- nUR 5 nSnU( Gaf U
S nU R UR SUR S 5 5 nU( a U
S nOU
S
n[# UUR S
S 5 nUUR SSS9- n[ R " UU
R 5 5 U- nUR U R S SU
R S 5 R% SS
S5 nUR UR S UR S S5 n[ R " UU5 U- nUR UR S UR S UR S
UR S 5 nSnU( a [' U5 nU ( d Ub
UUUU
U U U
4nOUXXU
4nUb U4U- $ U$ [) UUUUU
U U U
S 9$ ) a
return_loss (`bool`, *optional*):
Whether or not to return the contrastive loss.
output_segmentation (`bool`, *optional*):
Whether or not to return the segmentation logits.
Examples:
```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(
... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
```NTr r r rb r r r rL )r r r r r r r r )rq rQ r' r r r r r r normr expr matmulr' rV rG rm rd r* r )r r- r rS r" r& rQ r r' r r" r r r r r r
seg_logitsimage_group_embedsrX groupinglogits_per_image_groupflatten_groupingr outputs r" r GroupViTModel.forward/ sc R 2C1N -TXT_T_TqTq#6#B HgHg $ $8$D $++JjJj &1%>RSCTVX Y &<>NOR]]J#++ #Z%5%5a%8(..:KX^^\]M^J 1D%$# " +O,ftu)-)9TGf$ Ev E" - + *#% * .
r$ ) r r r r r r r r r r r ) NNNNNNNNN)r r r r r r rw r r r r r r r r$ r5 r r r r r r r s @r" r r s !L)~ )V -115/3,0/3&*, ELL), ! ., u||, ,
$D> , 'tn
, d^,
, , \ 59,0/3&* . u 0 01. $D>. 'tn .
d^ .
. . ` 15481537&*,0/3.2&*N
E , ,-N
u 0 01N
! . N
u / /0 N
d^
N
$D>N
'tnN
&d^N
d^N
u)) *N
N
r$ r )r r r r )r Frb r )Er collections.abcr dataclassesr typingr r r r numpyrT r torch.utils.checkpointr activationsr
modeling_attn_mask_utilsr r modeling_outputsr
r modeling_utilsr utilsr r r r configuration_groupvitr r r
get_loggerr loggerr r# r* rS r< floatr rJ r` rm r ro r r r r r r r7 r~ r rx r> r r r r r r r r __all__r r$ r" rB s[ ! . . ! d K - D D \ \
H %
`U\\ `ell `
-ell -u|| - C 5<< e t RU _d_k_k ,<:
"))
- bii - `4+")) 4+n /
+ /
/
d bii DGryy GV%RYY %P[BII [| ")) 0!{ ! k2 k2^/299 /d $@o $@ $@N7
BII 7
t^
")) ^
BY
bii Y
x/
/ /
d2
2
j.
1 .
b ]
+ ]
]
@ cr$