o
Zh$ @ s d dl mZmZ d dlZd dlmZ d dlmZmZmZ d dl m
Z
ddl m Z ddl
mZ ddlmZmZ d d
lmZmZmZ G d d d eZeG d
d deZG dd deeZeddG dd deeZg dZdS ) )OptionalUnionN)BCEWithLogitsLossCrossEntropyLossMSELoss) IJepaConfig )ImageClassifierOutput)PreTrainedModel)auto_docstring torch_int )
ViTEmbeddingsViTForImageClassificationViTModelc st e Zd Zddededdf fdd
Zdejd ed
edejfd d Z dd
ejde
ej dedejfddZ Z
S )IJepaEmbeddingsFconfiguse_mask_tokenreturnNc s6 t || | `| jj}ttd||j | _
d S )N ) super__init__Z cls_tokenpatch_embeddings num_patchesnn ParametertorchZrandnZ hidden_sizeposition_embeddings)selfr r r __class__ V/var/www/auris/lib/python3.10/site-packages/transformers/models/ijepa/modular_ijepa.pyr s zIJepaEmbeddings.__init__
embeddingsheightwidthc C s |j d }| jj d }tj s||kr||kr| jS | j}|j d }|| j }|| j } t|d }
|d|
|
|}|dddd}t j
j ||| fddd }|dddd dd|}|S )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r g ?r r r
ZbicubicF)sizemodeZ
align_corners)
shaper r Zjit
is_tracingZ
patch_sizer ZreshapeZpermuter Z
functionalZ interpolateview) r r# r$ r% r Z
num_positionsZpatch_pos_embeddimZ
new_heightZ new_widthZsqrt_num_positionsr! r! r" interpolate_pos_encoding s&
z(IJepaEmbeddings.interpolate_pos_encoding pixel_valuesbool_masked_posr- c C s |j \}}}}| j||d}|d ur1|j d } | j|| d}
|d|
} |d| |
| }|r=|| ||| }n|| j }| |}|S )N)r- r r& ?) r) r
mask_tokenexpandZ unsqueezeZtype_asr- r Zdropout) r r. r/ r- Z
batch_size_r$ r% r# Z
seq_lengthZ mask_tokensmaskr! r! r" forward> s
zIJepaEmbeddings.forward)F)NF)__name__
__module__ __qualname__r boolr r Tensorintr- r Z
BoolTensorr5
__classcell__r! r! r r" r s *r c @ sN e Zd ZeZdZdZdZddgZdZ dZ
de e j
e je jf ddfd d
ZdS ) IJepaPreTrainedModelijepar. Tr Z
IJepaLayermoduler Nc C s t |tjtjfr0tjj|jjt j
d| j j d|jj
|j_|jdur.|jj dS dS t |tjrE|jj |jjd dS t |trotjj|jjt j
d| j j d|jj
|j_|jdurq|jj dS dS dS )zInitialize the weightsg )meanZstdNr0 )
isinstancer LinearConv2dinitZ
trunc_normal_weightdatator Zfloat32r Zinitializer_rangedtypeZbiasZzero_ LayerNormZfill_r r r1 )r r? r! r! r"
_init_weightsc s0
z"IJepaPreTrainedModel._init_weights)r6 r7 r8 r Z config_classZbase_model_prefixZmain_input_nameZ supports_gradient_checkpointingZ_no_split_modulesZ_supports_sdpaZ_supports_flash_attn_2r r rB rC rI rJ r! r! r! r" r= Y s &r= c s, e Zd Zddededef fdd
Z ZS )
IJepaModelFr add_pooling_layerr c s$ t | || _t||d| _dS )z
add_pooling_layer (bool, *optional*, defaults to `True`):
Whether to add a pooling layer
use_mask_token (`bool`, *optional*, defaults to `False`):
Whether to use a mask token for masked image modeling.
)r N)r r r r r# )r r rL r r r! r" r { s zIJepaModel.__init__)FF)r6 r7 r8 r r9 r r< r! r! r r" rK z s $rK a
IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states)
e.g. for ImageNet.
Note that it's possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by
setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
position embeddings to the higher resolution.
)Z custom_introc s e Zd Zdef fdd Z ddeej deej deej dee d ee d
ee d ee d e e
e f fd
dZ Z
S )IJepaForImageClassificationr c s& t | t|dd| _| d S )NF)rL )r r rK r> Z post_init)r r r r! r" r s z$IJepaForImageClassification.__init__Nr. head_masklabelsoutput_attentionsoutput_hidden_statesr- return_dictr c C sv |dur|n| j j}| j||||||d}|d } | | jdd}
d} |dur||
j}| j jdu rX| jdkr>d| j _n| jdkrT|j t
j ksO|j t
j krTd| j _nd| j _| j jdkrvt
} | jdkrp| |
| } n+| |
|} n%| j jdkrt } | |
d | j|d } n| j jdkrt } | |
|} |s|
f|dd }
| dur| f|
S |
S t| |
|j|jd
S ) a
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
N)rN rP rQ r- rR r r )r, Z
regressionZsingle_label_classificationZmulti_label_classificationr& )losslogits
hidden_states
attentions)r Zuse_return_dictr> Z
classifierr@ rG ZdeviceZ problem_typeZ
num_labelsrH r longr; r Zsqueezer r+ r r rU rV )r r. rN rO rP rQ r- rR ZoutputsZsequence_outputrT rS Zloss_fctoutputr! r! r" r5 sP
"
z#IJepaForImageClassification.forward)NNNNNNN)r6 r7 r8 r r r r r: r9 r tupler r5 r< r! r! r r" rM s4
rM )r= rK rM )typingr r r Ztorch.nnr r r r Z-transformers.models.ijepa.configuration_ijepar Zmodeling_outputsr Zmodeling_utilsr
utilsr r Zvit.modeling_vitr r r r r= rK rM __all__r! r! r! r" s$ J
J