o
ZhD% @ s d dl Z d dlmZ ddlmZ ddlmZ ddlmZm Z ddl
m Z m Z ddl
mZmZ d d
lmZ e rAd dlZd dlmZ d Zd Zd
ZdZdZ dddZG dd dZdS ) N)Image )_get_model_class)
AutoConfig) MODEL_FOR_PRETRAINING_MAPPING
MODEL_MAPPING)PROCESSOR_MAPPING_NAMES
AutoProcessor)TOKENIZER_MAPPING_NAMES
AutoTokenizer )is_torch_availablez[92mz[93mz[0mu ■u ⬚![]()
c
sp jdkrdddddf jdkr%ddddddf ttdd D }d}g }tD ]6\ } | krL|sL }d f< |dkrp| ksZ d krp d krd d7 d| | f< d}q:ddurfd d
tD d fd dtD }
|dur|dk} t| d
d |
| d
dt d|d |} tj | dt
t t dt t t d}
|d |
d |d dt|
d }dur|d7 }|| g }tD ]3\}}||f dkr|dd
tt|ttD q|tt|tt qtttt| }|D ] }|durO|d d d | d d | nd q4tD ]Z\ }t||}|v rpt | t n|}d fddtD }d}durd fddtD }|| dt d d | d| qXd|S )z
Generates an attention matrix from a given attention mask.
Optionally applies a sliding window mask (e.g., for Gemma2/3) and
marks regions where image tokens occur based on the specified `img_token`.
r N c s s | ] }t t|V qd S N)lenrepr).0word r V/var/www/auris/lib/python3.10/site-packages/transformers/utils/attention_visualizer.py 7 s z6generate_attention_matrix_from_mask..r r c s$ g | ] fd dt D qS )c s0 g | ]}d | krk rn ndnd qS )r r r r j)isliding_windowr r
H s 0 zBgenerate_attention_matrix_from_mask...)range)r )nr )r r r H s $ z7generate_attention_matrix_from_mask.. c 3 sX | ]'} d |f rt t t nd |kr t t t n d |f r&tntV qdS r NYELLOW BLACK_SQUARERESETGREEN WHITE_SQUAREr )maskr r r J s
)
boundariesz: i == j (diagonal) z: token_type_idszAttention MatrixzSliding Window Maskc S s g | ]
}t | t qS r )r# r% )r kr r r r h s z | c 3 sp | ]3}| v r |f r v r t t t n |kr*t t t n |f r2tntV qd S r r" r )r img_tokenr( wordsr r r w s $
c 3 s| | ]9}| v r$ v r$d f d |f kr$t t t n |kr0t t t n | r8tntV qdS r! r" r )r r. sliding_window_masktoken_type_bucketsr/ r r r s
z:
)intndimr max enumerater jointorchwhereZcumsumboolarangeZ bucketizer& r$ r% r# appendljustliststrrjustmapzipr )r/ r( r. r token_type_idsimage_seq_lengthZmax_word_lengthZ
first_img_idxoutputr, Z row_dummyZ
is_specialr+ Zlegendf_stringZvertical_headeridxr rowZ word_reprZ colored_wordZ row_displayZsliding_window_rowr )r r. r( r r r0 r1 r/ r #generate_attention_matrix_from_mask' s~
,"
$
.
rI c @ s: e Zd ZdefddZd defddZd defdd Zd
S ) AttentionMaskVisualizer
model_namec C s t |}d| _t| drd|_zt|t}W n ty' t|t }Y nw |d u r4t
d| d|| _ G dd d|t j
}|||| _| j|j || _|| _d S )Nr r r* z Model name z- is not supported for attention visualizationc @ s e Zd Zdd ZdS )z7AttentionMaskVisualizer.__init__.._ModelWrapperc S s$ t j| t dd| _|| _d S )Nr )nnModule__init__ZLinearZ dummy_moduleconfig)selfrO rK r r r rN s
z@AttentionMaskVisualizer.__init__.._ModelWrapper.__init__N)__name__
__module__ __qualname__rN r r r r
_ModelWrapper s rT )r from_pretrained image_tokenhasattrZget_text_configr r r Exceptionr
ValueError
mapped_clsrL rM modeltoZ torch_dtyperepo_idrO )rP rK rO rZ rT r r r rN s"
z AttentionMaskVisualizer.__init__r- input_sentencec C s | j ||d d S )N)suffix)visualize_attention_mask)rP r^ r_ r r r __call__ s z AttentionMaskVisualizer.__call__c C s | j }i }| jjtv rid}ttj|ddj}d}t j
| j |d}t |dr+|j
}n
|j|jgd }|r=|d|}||||d d
} |j|jgd | _
| d }
d | v r^| d |d < |j| d
d } n$| jjtv rt
| j } | |} | |d dd }
n
td|jj dd|j_| |jd |
|
| j jt |
j d d d| }
dt d| jj d| j d }
d}t!d|
t!dd| jj d| j d| j j" #t |
d | t!|
t$| |
| j
t%| jd d |d d |d }t!| t!|
d S ) Nzchttps://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg?download=trueT)streamr* )rD rV r r pt)Zimagestextr_ return_tensorsattention_maskrC Z input_ids)re z Model type z) does not support attention visualizationeagerr )rf Z input_tensorZcache_positionZpast_key_valuesz##z Attention visualization for z | r r2 z" Attention visualization for [1m:z[0m z r )r. r rC rD r )&r[ rO Z
model_typer r openrequestsgetrawr rU r] rW rV tokenizerZconvert_ids_to_tokensZimage_token_idreplacer
r tokenizerY Z_attn_implementationtrainZ_update_causal_maskr\ Zdtyper8 r; shaper: r rZ printrQ centerrI getattr)rP r^ r_ r[ kwargsZimgrD processorrV Zinputsrf tokensrm Ztop_bottom_borderZ side_borderrF r r r r` s|
z0AttentionMaskVisualizer.visualize_attention_maskN)r- )rQ rR rS r? rN ra r` r r r r rJ s rJ )r NNN) rj ZPILr Zmodels.auto.auto_factoryr Z models.auto.configuration_autor Zmodels.auto.modeling_autor r Zmodels.auto.processing_autor r Z models.auto.tokenization_autor
r Z import_utilsr
r8 Ztorch.nnrL r& r# r% r$ r' rI rJ r r r r s&
n