o Zh;@s2ddlmZmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZd d lmZmZmZmZmZmZd d lmZd d lmZeeZGdddeZGdddejZ GdddejZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&gdZ'dS))ListOptionalTupleUnionN)nn)ACT2FN)FlashAttentionKwargs)Unpack)is_torchdynamo_compilinglogging)KwargsForCausalLMLlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastLlavaPreTrainedModel)MistralRMSNorm)Mistral3Configc@ eZdZdS)Mistral3RMSNormN__name__ __module__ __qualname__rr\/var/www/auris/lib/python3.10/site-packages/transformers/models/mistral3/modular_mistral3.pyr(rcsBeZdZdZdeffdd Zdejdejdejfdd ZZ S) Mistral3PatchMergerz< Learned merging of spatial_merge_size ** 2 patches configcsLt||_|jj}|j|_|jjj|_tj||jd|dd|_ dS)Nr Fbias) super__init__r! vision_config hidden_sizespatial_merge_size patch_sizerLinear merging_layer)selfr!r' __class__rrr%1s   zMistral3PatchMerger.__init__image_features image_sizesreturnc sfdd|D}dd|D}|jd}g}t||D]7\}}||\}} ||| |dddd} tjjj | j j d} | |j dd } | | qtj |dd }|}|S) Ncs(g|]}|dj|djfqS)rr)r)).0Z image_sizer,rr ;sz/Mistral3PatchMerger.forward..cSsg|]\}}||qSrr)r2hwrrrr4?sr rr)Z kernel_sizeZstridedim)shape enumeratesplitviewZpermute unsqueezetorchrZ functionalZunfoldr(tappendcatr+) r,r/r0Ztokens_per_imagedZpermuted_tensorZ image_indexZ image_tokensr5r6Z image_gridgridrr3rforward:s"      zMistral3PatchMerger.forward) rrr__doc__rr%r?TensorrE __classcell__rrr-rr ,s$ r cs8eZdZdeffdd ZdejdejfddZZS)Mistral3MultiModalProjectorr!cstt|jj|jjd|_t||_ t |j t rdnt |j }tj|jj||jj|jd|_t|j|_tj|jj|jj|jd|_dS)N)Zepsrr")r$r%rr&r' text_configZ rms_norm_epsnormr patch_merger isinstancevision_feature_layerintlenrr*Zmultimodal_projector_biaslinear_1rZprojector_hidden_actactlinear_2)r,r!Znum_feature_layersr-rrr%Ss     z$Mistral3MultiModalProjector.__init__r/r0cCs8||}|||}||}||}||}|SN)rKrLrQrRrS)r,r/r0 hidden_statesrrrrEcs     z#Mistral3MultiModalProjector.forward) rrrrr%r?rGrErHrrr-rrIRsrIc@r)Mistral3CausalLMOutputWithPastNrrrrrrVlrrVc@r)Mistral3ModelOutputWithPastNrrrrrrWprrWc@seZdZddZdS)Mistral3PreTrainedModelcCst|jd|jj}t|tjr)|jjj d|d|j dur'|j j dSdSt|tj r>|jj d|j j dSt|trL|jj ddSdS)Ninitializer_rangeg)meanstdg?)getattrr!Zget_text_configrYrMrr*weightdataZnormal_r#Zzero_Z LayerNormZfill_r)r,moduler[rrr _init_weightsus    z%Mistral3PreTrainedModel._init_weightsN)rrrr`rrrrrXts rXc!@seZdZ ddejdejdeeee effddZ             ddej dejdeejd eej d ee ejd eejdeeee efd ee d ee dee dee deej dejde edeeeffddZdS) Mistral3ModelN pixel_valuesr0rNc s|dur|n|jj}dd|D}|j|f|dd|t|tr*j|}nfdd|D}tj|dd }| | d |}|S) aU Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layer (`Union[int, List[int]]`, *optional*): The index of the layer to select the vision feature. If multiple indices are provided, the vision feature of the corresponding indices will be concatenated to form the vision features. image_sizes (`torch.Tensor`, *optional*): Tensor containing the image sizes as returned by the processor. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). NcSsi|] \}}|dur||qSrTr)r2kvrrr sz4Mistral3Model.get_image_features..T)r0output_hidden_statescsg|]}j|qSr)rU)r2Z layer_idxZ image_outputsrrr4sz4Mistral3Model.get_image_features..r7r8r) r!rNitemsZ vision_towerrMrOrUr?rBZmulti_modal_projectorZsqueeze)r,rbr0rNkwargsZselected_image_featureZhs_poolr/rrgrget_image_featuress  z Mistral3Model.get_image_features input_idsattention_mask position_idspast_key_values inputs_embeds use_cacheoutput_attentionsrf return_dictcache_positionrir1c Ks| dur| n|jj} | dur| n|jj} | dur| n|jj} |dur$|n|jj}|du|duAr4td|dur@|dur@td|durJ||}|dur|j||| d}||jjk d}| | |j }t s|||kr||jjk}|jd|jd}td|d|| |j |j}|||}|jd |||||| | d | d |}t|j|j|j|j|dur|d Sdd S) Nz:You must specify exactly one of input_ids or inputs_embedszdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either one)rbrNr0r7rrz6Image features and image tokens do not match: tokens: z , features T) rlrmrnrorprqrfrrrs)last_hidden_staternrU attentionsimage_hidden_statesr)r!rqrfuse_return_dictrN ValueErrorZget_input_embeddingsrjZimage_token_idr>Z expand_astoZdevicer Znumelsumr:ZdtypeZmasked_scatterlanguage_modelrWrtrnrUru)r,rkrbrlrmrnrorNrprqrfrrrsr0rir/Zspecial_image_maskZn_image_tokensZn_image_featuresoutputsrrrrEsh     zMistral3Model.forwardrT) NNNNNNNNNNNNN)rrrr? FloatTensorrGrrrOrrj LongTensorboolr r rrWrErrrrrasj *       rac#@seZdZ              ddejdejdeejdeejdeeejdeejd eejd ee d ee d ee d ee deejde e ejfdeejde e de eeff ddZdS) Mistral3ForConditionalGenerationNrrkrbrlrmrnrolabelsrprqrfrrrslogits_to_keepr0rir1cKs| dur| n|jj} | dur| n|jj} | dur| n|jj} |jd|||||||| | d| |d |}|d}t| trBt| dn| }||dd|ddf}d}|durg|j d|||jj j d|}t |||j |j|j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Mistral3ForConditionalGeneration >>> model = Mistral3ForConditionalGeneration.from_pretrained("mistralai/Mistral-Small-3.1-24B-Instruct-2503") >>> processor = AutoProcessor.from_pretrained("mistralai/Mistral-Small-3.1-24B-Instruct-2503") >>> prompt = "[INST][IMG]What is the image?[/INST]" >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is the image?The image depicts two cats lying on a pink blanket." ```NT) rkrbrlrmrnrorprqrfrrrsr0r)logitsr vocab_size)lossrrnrUrurvr)r!rqrfrwmodelrMrOsliceZlm_headZ loss_functionrJrrVrnrUrurv)r,rkrbrlrmrnrorrprqrfrrrsrr0rir|rUZ slice_indicesrrrrrrEsL-  z(Mistral3ForConditionalGeneration.forward)NNNNNNNNNNNNrN)rrrr?r~r}rrGrrrrOr rrrVrErrrrrs`        r)rarXr)(typingrrrrr?rZ activationsrZmodeling_flash_attention_utilsr Zprocessing_utilsr utilsr r Zllava.modeling_llavarrrrrrZmistral.modeling_mistralrZconfiguration_mistral3rZ get_loggerrloggerrModuler rIrVrWrXrar__all__rrrrs(        &v Y