eThj^SSKJrJrJrJrJrJr SSKrSSK J r SSK J r SSK JrJrJrJr SSKJrJrJrJrJr SSKJrJrJrJrJrJrJrJ r J!r!J"r" SS K#J$r$ SS K%J&r& SS K'J(r(J)r)J*r* SS K+J,r,J-r- SS K.J/r/J0r0J1r1J2r2J3r3 SSK4J5r5 SSK6J7r7J8r8J9r9 SSK:J;r; SSKr>J?r?J@r@JArAJBrB SSKCJDrDJErEJFrFJGrG SSKHJIrI \3R"\K5rL\5"5(a SSKMrMSSKMJNrN SrO"SS\;5rP"SS\ 5rQ"SS\B5rR"SS\NR5rT"SS \NR5rU"S!S"\NR5rV"S#S$\5rW"S%S&\(S'S(9rX"S)S*\)5rY"S+S,\?5rZ"S-S.\NR5r["S/S0\NR5r\"S1S2\NR5r]"S3S4\=5r^\1"S5S6\&55r_"S7S8\A5r`"S9S:\@5ra"S;S<\$\/5rb"S=S>\_\>5rc"S?S@\D5rd"SASB\G5re"SCSD\F5rf\1"SESF9"SGSH\E55rg/SIQrhg)J)DictIterableListOptionalTupleUnionN)ACT2FN)PretrainedConfig)BaseImageProcessor BatchFeatureget_patch_output_sizeselect_best_resolution) PaddingModeconvert_to_rgbpadresizeto_channel_dimension_format) ChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imagemake_flat_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments)FlashAttentionKwargs)PreTrainedModel)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput) LossKwargs TensorTypeauto_docstringcan_return_tuplelogging)is_torch_available)CONFIG_MAPPING AutoConfig AutoTokenizer) LlamaConfig)LlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModelLlamaPreTrainedModel LlamaRMSNorm)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPast)divide_to_patches)nncURSnURSn[R"X4URURS9n[R "USS9n[R"S[R URS9n[R"Xv45n[URS5H.nXhn XhS-n X U n [R"XU5n XX&M0 U$)a  Compute the matrix multiplication (GEMM) for each expert sequentially. This approach is computationally inefficient, especially when dealing with a large number of experts. Args: token_states (torch.Tensor): Input tensor of shape (num_tokens, in_features). expert_weights (torch.Tensor): Weight tensor of shape (num_experts, in_features, out_features). tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). rdtypedevicedim) shapetorchzerosr@rAcumsumlongcatrangematmul) token_statesexpert_weightstokens_per_expert num_tokens out_featuresoutputcumsum_num_tokens zero_tensor expert_numstartendtokensouts ]/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/aria/modular_aria.pysequential_experts_gemmr[Cs##A&J!''+L [[9K9KT`TgTg hF %6A>++auzz:K:R:RSK ;"BCN0034 !-Q/C(ll6*#=>u 5 Mc V^\rSrSrSrSrSrS S\S\S\S\4U4S jjjrS r U=r $) AriaTextConfigba This class handles the configuration for the text component of the Aria model. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Aria [rhymes-ai/Aria](https://huggingface.co/rhymes-ai/Aria) architecture. This class extends the LlamaConfig to include additional parameters specific to the Mixture of Experts (MoE) architecture. Args: vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the LLaMA model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`LlamaModel`] hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 4096): The size of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `num_attention_heads`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 2048): The maximum sequence length that this model might ever be used with. Llama 1 supports up to 2048 tokens, Llama 2 up to 4096, CodeLlama up to 16384. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 2): Padding token id. bos_token_id (`int`, *optional*, defaults to 1): Beginning of stream token id. eos_token_id (`int`, *optional*, defaults to 2): End of stream token id. pretraining_tp (`int`, *optional*, defaults to 1): Experimental feature. Tensor parallelism rank used during pretraining. Please refer to [this document](https://huggingface.co/docs/transformers/main/perf_train_gpu_many#tensor-parallelism) to understand more about it. This value is necessary to ensure exact reproducibility of the pretraining results. Please refer to [this issue](https://github.com/pytorch/pytorch/issues/76232). tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE attention_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. head_dim (`int`, *optional*): The attention head dimension. If None, it will default to hidden_size // num_heads moe_num_experts (`int`, *optional*, defaults to 8): The number of experts in the MoE layer. moe_topk (`int`, *optional*, defaults to 2): The number of top experts to route to for each token. moe_num_shared_experts (`int`, *optional*, defaults to 2): The number of shared experts. aria_text text_configintermediate_sizemoe_num_expertsmoe_topkmoe_num_shared_expertsc Z>[TU]"SSU0UD6 XlX lX0lX@lg)N pad_token_id)super__init__rbrcrdre)selfrbrcrdrerg super_kwargs __class__s rZrjAriaTextConfig.__init__s1 ClClC!2. &<#r\)rbrcrerd)ir,r,r,) __name__ __module__ __qualname____firstlineno____doc__ model_typebase_config_keyintrj__static_attributes__ __classcell__rms@rZr^r^bsWhTJ#O"& &' = = = = !$ = =r\r^c p^\rSrSrSrSrSS0r\\S.r S S\ S\S \ \ S\ S \ 4 U4S jjjrS rU=r$) AriaConfiga This class handles the configuration for both vision and text components of the Aria model, as well as additional parameters for image token handling and projector mapping. Instantiating a configuration with the defaults will yield a similar configuration to that of the model of the Aria [rhymes-ai/Aria](https://huggingface.co/rhymes-ai/Aria) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vision_config (`AriaVisionConfig` or `dict`, *optional*): Configuration for the vision component. vision_feature_layer (`int`, *optional*, defaults to -1): The index of the layer to select the vision feature. text_config (`AriaTextConfig` or `dict`, *optional*): Configuration for the text component. projector_patch_to_query_dict (`dict`, *optional*): Mapping of patch sizes to query dimensions. image_token_index (`int`, *optional*, defaults to 9): Index used to represent image tokens. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated normal initializer for initializing all weight matrices. Attributes: model_type (`str`): Type of the model, set to `"aria"`. image_token_index (`int`): Index used to represent image tokens. projector_patch_to_query_dict (`dict`): Mapping of patch sizes to query dimensions. vision_config (`AriaVisionConfig`): Configuration for the vision component. text_config (`AriaTextConfig`): Configuration for the text component. ariaimage_token_idimage_token_index)ra vision_configvision_feature_layerraprojector_patch_to_query_dictinitializer_rangec ,>XPlUcSSS.nUR5VV s0sHup[U5[U 5_M sn nUl[ URR 55UlX l[U[5(aSUS'[US"S0UD6nOUc[S"5nXl X`l [U[5(aSU;a [S0UD6nO Uc [5nX0l[T U]@"S0UD6 gs sn nf)N)ii$idefics3_visionrurh)ritemsrwrmaxvalues'max_value_projector_patch_to_query_dictr isinstancedictr-rrr^rarirj) rkrrrarrrkwargskvrms rZrjAriaConfig.__init__ s"3 ) 0- )JgIlIlIn-oInc!fc!fnIn-o*7:4;];];d;d;f7g4$8! mT * **;M, '*=+FGX-XM  "*+<=?M*!2 k4 ( (\[-H(7;7K  (*K& "6"'.ps!D)rrrrrarr)Nr>NN g{Gz?)rprqrrrsrtru attribute_mapr^r. sub_configsrwrrfloatrjrxryrzs@rZr|r|s}"HJ-M#1:NK$&&*8[TU]5 [R"XSS9Ul[R"X#SS9Ul[ SUlg)NFbiasgelu_new)rirjr<Linear linear_in linear_outr act)rk in_featureshidden_features output_dimrms rZrjAriaProjectorMLP.__init__Es> ;eL))OeL*%r\chURURU55nURU5nU$Nrrr)rk hidden_statess rZforwardAriaProjectorMLP.forwardKs- !>?  6 r\r rprqrrrsrtrjrrxryrzs@rZrr8s & r\rcF^\rSrSrSrSS\S\4U4SjjjrS SjrSr U=r $) AriaCrossAttentioniQzb Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. config dropout_ratec.>[TU]5 URRnURRnX@l[ R"X3SS9Ul[ R"X3SS9Ul [ R"X3SS9Ul [ R"X4SS9Ul [ R"X35Ul [ R"U5Ul[ R "U5Ul[ R "U5Ulg)NFrT) batch_first)rirjr hidden_sizenum_attention_heads num_headsr<rq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)rkrrrrrms rZrjAriaCrossAttention.__init__Zs **66 ((<< "ii uE ii uE ii uE !33KX\]ii 9 zz,/ ,,{3\\+6r\cURURU55nURU5nURU5nUR U5nUR XEXcS9upxUR URU55nU$)ai Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rrrrrrrr) rkkey_value_statesrrquerykeyvalue attn_output_s rZrAriaCrossAttention.forwardks  DOOM:;--.>?kk*+ ,-,,U,T ll4;;{#;< r\) rrrrrrrrr)rr) rprqrrrsrtr|rrjrrxryrzs@rZrrQs*7z777"r\rcx^\rSrSrSrS\4U4SjjrS S\RS\ \R4Sjjr Sr U=r $) AriaProjectoriz Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. rc>[TU]5 URUlURR UlURRUlURR Ul URR Ul URR Ul [R"[R "UR"UR 55Ul['U5Ul[R*"UR 5Ul[/UR URUR5Ulgr)rirjrpatch_to_query_dictrrrrrkv_dimrarrr< ParameterrFrGrrr cross_attnrrr feed_forwardrkrrms rZrjAriaProjector.__init__s #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder\rrcJURSURSpCX@RR5;a*[SUSURR5S35eURUnURSUR S5R USS5nUbMURURS5nUR S5RSURS5S5nURXUS9nURURU55nU$) aH Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rrDzNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr>r)rErkeysKeyErrorr unsqueezerepeatrepeat_interleaverexpandsizerrr) rkrr batch_size num_patches query_numqueries attention_outrYs rZrAriaProjector.forwards9#3"8"8";=M=S=STU=VK 66;;= =$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r\) rrrrrrrrrrr) rprqrrrsrtr|rjrFTensorrrrxryrzs@rZrrs<ff( %,,AWr\rc ^\rSrSrSr/SQrSSSSSSSSS S\R4 S \\ \ S \\ \ S \ S \ S\\ \ \ \ 4S\\ S\\ S\ S\\ \ 4S\\ S\4U4SjjjrSSSSSSSSSSS\R"S4 S\\\ \4S \\\ \ \ 4S \\\ \ \ 4S \\ S \\ S\\ S\\ S\\ S\\ S\\ S\S\\\\4S\\S\\\\44SjjrS\R.S\S\S\R.4SjrS\R.S\S\S\R.4S jr\R8S!SS4S\R:S"\\ \ \ \ 4\\ \ \ 44S#\S$\\ \\ 4S\\\\4S\\\\4S\R:4S%jjrS\R.S&\ \ \ \ 4S'\ S\S\S\S\ \R.4S(jr S)r!U=r"$)*AriaImageProcessoria A vision processor for the Aria model that handles image preprocessing. Initialize the AriaImageProcessor. Args: image_mean (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Mean values for normalization. image_std (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Standard deviation values for normalization. max_image_size (`int`, *optional*, defaults to 980): Maximum image size. min_image_size (`int`, *optional*, defaults to 336): Minimum image size. split_resolutions (`list`, *optional*, defaults to a list of optimal,resolutions as tuples): The optimal resolutions for splitting the image. split_image (`bool`, *optional*, defaults to `False`): Whether to split the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. resample (PILImageResampling, *optional*, defaults to `BICUBIC`): The resampling filter to use if resizing the image.  pixel_values pixel_mask num_cropsNiPFTgp? image_mean image_stdmax_image_sizemin_image_sizesplit_resolutions split_imagedo_convert_rgb do_rescalerescale_factor do_normalizeresamplec  >[TU]"S0U D6 Uc/SQnUc/SQnX0lX@lXlX lX`lUc#/SQnUV s/sHoSS-U SS-4PM nn XPlXplXl Xl Xl Xl gs sn f)N)?rr))rDr,)rDr )rD)rD)rD)rD)rDro)r,r)r,r )r,r,)r,rD)r rD)r r,)rrD)rr,)rrD)rrD)rrD)rorDrrDrh) rirjrrrrrrrrrrr)rkrrrrrrrrrrrrelrms rZrjAriaImageProcessor.__init__s "6"  (J  'I,,$"&  $!y FW XFWQ%#+r!us{!;FW  X!2,$,( !Ys B ptimagesreturn_tensors data_formatinput_data_formatc nUbUO URnUbUO URnUbUO URnUbUO URnUbUO URnUbUO UR nUbUO UR nU bU O URn U bU O URn U bU O URn US;a [S5e[U5n[U5(d [S5e[U UUU UU S9 U(aUVs/sHn[U5PM nnUVs/sHn[U5PM nnU(a([!US5(a["R%S5 Uc['US5n/n/nSnUGHnU(aUR)UUR*UU UUS9nOU/nUb[-U5U:a [-U5nUGH.n[/U5unnU[1UU5- nUU:a[1[3UU-5U5U4nOU[1[3UU-5U54n[5UUU UUS 9nUUS- UUS - nn[7USU4SU44UUS9n[8R:"XD4[<S 9nS USUS2SUS 24'UR?U5 U(aURAUXS 9nU (a8URCUURURUUS9nU b [EUX5OUnUR?U5 GM1 GM [G[8RH"USS 9[8RH"USS 9US.U S9$s snfs snf)a Process a list of images. Args: images (ImageInput or list of ImageInput): The input image or a list of images. image_mean (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Mean values for normalization. image_std (`list`, *optional*, defaults to [0.5, 0.5, 0.5]): Standard deviation values for normalization. max_image_size (`int`, *optional*, defaults to `self.max_image_size` (980)): Maximum image size. min_image_size (`int`, *optional*, defaults to `self.min_image_size` (336)): Minimum image size. split_image (`bool`, *optional*, defaults to `self.split_image` (False)): Whether to split the image. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb` (True)): Whether to convert the image to RGB. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize` (True)): Whether to normalize the image. resample (PILImageResampling, *optional*, defaults to `self.resample` (BICUBIC)): The resampling filter to use if resizing the image. return_tensors (`str` or `TensorType`, *optional*, defaults to "pt"): The type of tensor to return. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: BatchFeature: A BatchFeature object containing: - 'pixel_values': Tensor of processed image pixel values. - 'pixel_mask': Boolean pixel mask. This mask is a 2D tensor of shape (max_image_size, max_image_size) where: - True (1) values indicate pixels that belong to the original resized image. - False (0) values indicate pixels that are part of the padding. The mask helps distinguish between actual image content and padded areas in subsequent processing steps. - 'num_crops': The maximum number of crops across all images. Nrrz(max_image_size must be either 490 or 980zkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)rrrrrrrzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.)rr)rrrrD)r@)imagescaler)axisrdata tensor_type)%rrrrrrrrrr ValueErrorrrrrrrlogger warning_oncerget_image_patchesrlenrrrwrrnprGboolappendrescale normalizerr stack)rkrrrrrrrrrrrrrrr r pixel_masksr crop_images crop_imagehwr new_sizecrop_image_resizedpadding_bottom padding_rightcrop_image_paddedrs rZ preprocessAriaImageProcessor.preprocesssR$.#9Zt !*!6IDNN +9+E4K^K^+9+E4K^K^%0%vay I    E"44**" 1&7 5  %g  C $4y$@ , ) %j11&Q26 #CE NN C^TH .CE NN0STH%+% 1&7 &"1?!0Ln_ghi_jNj $'&(1m*<= 1&7 %! XX~&FdS ;< =Xa[=-HQK-78"":.(, /~)5)% (,)$5*; )7)%'244E{f.& ##$56c*B "A > hh{;& '   iA=s ;L-L2r target_resolutionreturnc:[XU5upV[XU4X4S9nU$)a Resizes an image to a target resolution while maintaining aspect ratio. Args: image (np.array): The input image. target_resolution (tuple): The target resolution (height, width) of the image. resample (`PILImageResampling`): Resampling filter to use if resizing the image. input_data_format (`ChannelDimension` or `str`): The channel dimension format of the input image. Returns: np.array: The resized and padded image. rr)rr)rkr r(rr new_height new_width resized_images rZ_resize_for_patching'AriaImageProcessor._resize_for_patchings-&!6ePa b u9&=v r\cUupE[XU5upg[XW- S5up[XF- S5upURXX-4XU -44S9n U $)zE Pad an image to a target resolution while maintaining aspect ratio. r,)padding)rdivmodr) rkr r(r target_height target_widthr,r-paste_xr_xpaste_yr_y padded_images rZ_pad_for_patching$AriaImageProcessor._pad_for_patchingsi '8# 5ePa b l6: m8!< xx'-0H7^aTaJb/cxd r\r2modeconstant_valuesc^[U[5(d[U5S:wa [XX4XV5$Uc [ U5n[ R S[ RS[ RS[ RS0n[R"XXsUS9nUb[XU5nU$UnU$)a Pads the `image` with the specified `padding` and `mode`. Padding can be in the (`height`, `width`) dimension of in the (`num_patches`) dimension. In the second case an iterable if tuples is expected as input. Args: image (`np.ndarray`): The image to pad. padding (`int` or `Tuple[int, int]` or `Iterable[Tuple[int, int]]`): Padding to apply to the edges of the height, width axes. Can be one of three formats: - `((before_height, after_height), (before_width, after_width))` unique pad widths for each axis. - `((before, after),)` yields same before and after pad for height and width. - `(pad,)` or int is a shortcut for before = after = pad width for all axes. mode (`PaddingMode`): The padding mode to use. Can be one of: - `"constant"`: pads with a constant value. - `"reflect"`: pads with the reflection of the vector mirrored on the first and last values of the vector along each axis. - `"replicate"`: pads with the replication of the last value on the edge of the array along each axis. - `"symmetric"`: pads with the reflection of the vector mirrored along the edge of the array. constant_values (`float` or `Iterable[float]`, *optional*): The value to use for the padding if `mode` is `"constant"`. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use same as the input image. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. If unset, will use the inferred format of the input image. Returns: `np.ndarray`: The padded image. rconstantreflectedge symmetric)r>r?) rrwrrrrCONSTANTREFLECT REPLICATE SYMMETRICrr)rkr r2r>r?rrpadding_mode_mappings rZrAriaImageProcessor.pads` gs # #s7|q'8utk] ]  $ >u E   *     ! !6  ! !;   u,@,FXghR]Ri 'D-1+/!!=A&+)-,3'+'9'A'A"!T%[)"!DK("! "!  "! $DsCx$9: "!d^"!!"!"!c5j)"!tn"!%"!"!N;?9=(,(,&*)-%)*.'+'+;?2B2H2HDHD j$z"223D U5$u+#567D E%e"456 D ! D ! D d^D !D TND !D tnD %D !sJ!78D ./D $E#/?*?$@AD LXX27Vf 4XX27L\ ((009<>BDH@zz@sE#s(OXeCHo-FFG@ @ uhuo56 @ eC)9$9:; @$E#/?*?$@A@ @D0xx0U38_-0 0 % 0 & 0,0 bhh00r\rc>\rSrSrSS0SSS.\R S.rSrg) AriaProcessorKwargsitr2Fr)rr) text_kwargs images_kwargsrrhN)rprqrrrsr'PYTORCH _defaultsrxrhr\rZr`r`ts- u "  %,, Ir\r`F)totalc ^\rSrSrSrSS/rSS/rSrSrSS\ \ \ 4S\ \ S\ \ \ \\4\44U4S jjjrSS \ \\\\\\4S \ \S \\S \4SjjrSrSr\S5rSrU=r$) AriaProcessoria AriaProcessor is a processor for the Aria model which wraps the Aria image preprocessor and the LLama slow tokenizer. Args: image_processor (`AriaImageProcessor`, *optional*): The AriaImageProcessor to use for image preprocessing. tokenizer (`PreTrainedTokenizerBase`, *optional*): An instance of [`PreTrainedTokenizerBase`]. This should correspond with the model's text model. The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. size_conversion (`Dict`, *optional*): A dictionary indicating size conversions for images. image_processor tokenizer chat_templatesize_conversionrr/c,>UcSSS.nUR5VVs0sHupV[U5U_M snnUlURUlURUlUbUR cUR Ul[TU]!XUS9 gs snnf)Nrrr )rj) rrwrk image_tokenr pad_token unk_tokenrirj)rkrhrirjrkrrrms rZrjAriaProcessor.__init__s  "$'c2O6E6K6K6MN6MdaA 6MN$00'66  Y%8%8%@"+"5"5I  =Q OsBtextrrr)c UR"[4SURR0UD6n[ U[ 5(aU/nO8[ U[ 5(d#[ US[ 5(d [S5eUbUR"U40USD6nURURRSn/n URS5U-n UHQn U RURRURRU -5n U RU 5 MS O0nUn USRS S5n UR"U 40USD6n UR!XS /S 9 [#0U EUEU S 9$) a Main method to prepare for the model one or several sequences(s) and image(s). Args: text (`TextInput`, `PreTokenizedInput`, `List[TextInput]`, `List[PreTokenizedInput]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). images (`ImageInput`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **pixel_mask** -- Pixel mask to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringsNrbr,rrarr ) modalitiesr) _merge_kwargsr`ri init_kwargsrr[rPrrhrkrrEpopreplacermr_check_special_mm_tokensr )rkrqraudiovideosr output_kwargs image_inputstokens_per_imageprompt_stringsrsampler text_inputss rZ__call__AriaProcessor.__call__s<**  "&.."<"<  dC 6DD$'' 47C0H0H`a a  //0L $33L4M4M4S4STU4VW N$((58HHI(B(BDNND^D^ajDjk%%f- L!N&}599:JDQnn^T}]7ST  %%nwi%X!@K!@? RR*(, >^I0$y/4HYCZZ[>^$>^ ,- >^ >^@<6XXr\rgc4^\rSrSrSrS\4U4SjjrSrU=r$)AriaSharedExpertsMLPia Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. rc`>[TU]U5 URUR-Ulgr)rirjrbrers rZrjAriaSharedExpertsMLP.__init__ s) !'!9!9F[TU]5 XlX lX0l[ R "[R"X1U55Ul gr) rirjrrQgroupsr<rrFemptyweight)rkrrQrrms rZrjAriaGroupedExpertsGemm.__init__!s: &( ll5;;vL#QR r\cL[UURUR55$)a- Perform grouped matrix multiplication. Args: input (`torch.Tensor`): Input tensor of shape (num_tokens, in_features). tokens_per_expert (`torch.Tensor`): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). )r[rcpu)rkinputrOs rZrAriaGroupedExpertsGemm.forward(s''  KK  ! ! #  r\)rrrQrrrzs@rZrrs S  r\rc>^\rSrSrSrS\SS4U4SjjrSrSrU=r $) AriaGroupedExpertsMLPi<z~ Grouped MLP module for Mixture of Experts. Args: config (`AriaTextConfig`): Configuration object for the model. rr)Nc>[TU]5 Xl[URUR S-UR 5Ul[UR URUR 5Ulg)Nr,) rirjrrrrbrcfc1fc2rs rZrjAriaGroupedExpertsMLP.__init__Es_  )&*<*V>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor\cURX5n[R"USSS9upE[RR U5U-nUR X25nU$)z Forward pass of the Grouped MLP. Args: permuted_tokens (torch.Tensor): Permuted input tokens. tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor after passing through the MLP. r,r>rB)rrFchunkr< functionalsilur)rkpermuted_tokensrO fc1_output projectiongate fc2_outputs rZrAriaGroupedExpertsMLP.forwardKsTXXoA  ;;z1"= ]]'' 3d: XXj< r\)rrr) rprqrrrsrtr^rjrrxryrzs@rZrr<s(p~p$p r\rcn^\rSrSrSrS\4U4SjjrS\RS\R4Sjr Sr U=r $) AriaTextMoELayeri^z Aria Text Mixture of Experts (MoE) Layer. This layer applies a gating mechanism to route input tokens to different experts. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. rc>[TU]5 [R"URUR SS9Ul[U5Ul[U5Ul Xl gNFr) rirjr<rrrcrouterrexpertsrshared_expertsrrs rZrjAriaTextMoELayer.__init__isM ii 2 2F4J4JQVW ,V4 26: r\rr)cURnURSURS55nURU5n[R "X0R RSS9upE[RRUSS9nURn[R"UR5R[R5UR R SUR R S- S9RU5nUn U RS5n [R""U 5n UR%SXR R-5n UR'X5n [R("URSUR R-U RS54U RU R*S9nUR-SX5 URSUR RU RS55nXR/S5-R1SS9RU5nUR3URU55nUU-$)a Forward pass of the MoE Layer. Args: hidden_states (`torch.Tensor`): Input tensor of shape (batch_size, sequence_length, hidden_size). Returns: torch.Tensor: Output tensor after passing through the MoE layer. Process: 1. Route tokens to experts using the router. 2. Permute tokens based on routing decisions. 3. Process tokens through experts. 4. Unpermute and combine expert outputs. 5. Add shared expert output to the final result. r>rD)rrCrBr)binsminrr?)rEviewrrrFtopkrrdr<rsoftmaxr@histcflattentofloat32rcargsort index_selectrrGrA index_copy_rsumr)rkroriginal_shapelogits top_logits top_indicesscoresoriginal_dtyperOindicesflatten_indicessorted_indicesr expert_outputunpermuted_tokensrRshared_expert_outputs rZrAriaTextMoELayer.forwardqs $',,%**2}/A/A"/EF ]+"'**V{{7K7KQR"S &&zr&:$**!KK    ! $ $U]] 3,, ++a/  "^   ",,r*7'44Q++J^J^8^_ _H "KK \\!_t{{33 3]5G5G5J K%% ''  %%aG-222t{{7K7K]M_M_`aMbc#&6&6r&::??A?FKKN[ $22=3E3En3UV,,,r\)rrrr) rprqrrrsrtr^rjrFrrrxryrzs@rZrr^s4~9-U\\9-ell9-9-r\rc8^\rSrSrSrS\S\4U4SjjrSrU=r $)AriaTextDecoderLayeriaG Aria Text Decoder Layer. This class defines a single decoder layer in the language model, incorporating self-attention and Mixture of Experts (MoE) feed-forward network. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. layer_idx (`int`): Index of the layer. r layer_idxcD>[TU]U5 [U5Ulgr)rirjrmlprkrrrms rZrjAriaTextDecoderLayer.__init__s #F+r\)r) rprqrrrsrtr^rwrjrxryrzs@rZrrs  ,~,#,,r\rcB\rSrSr\rSrSS/rSrSr Sr Sr Sr Sr SrS rg ) AriaTextPreTrainedModelimodelrrTpast_key_valuesFc URRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(adUR R RSUS9 URb2UR R URR5 gg[U[5(a&UR R RS5 g[U[5(a%UR R RSUS9 gg)Nr=meanstd?)rrrr<rrrnormal_rzero_ Embedding padding_idxrfill_rrkmodulers rZ _init_weights%AriaTextPreTrainedModel._init_weightsskk++ fbii ( ( MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>.  0 0 MM   $ $S )  6 7 7 MM   & &CS & 98r\rhN)rprqrrrsr^ config_classbase_model_prefix_no_split_modulessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_class_supports_attention_backendrrxrhr\rZrrsA!L/1IJ&*#"3"N "& :r\rc*\rSrSr\rSrSrSrSr Sr g)AriaPreTrainedModeliFcURRn[U[R5(aWUR R RSUS9 URb%URR R5 gg[U[R5(aUR5 g[U[R5(aJUR R RS5 URR R5 g[U[5(a)[RR!UR"US9 gg)Nr=rr)r)rrrr<rrrrrrr_reset_parametersrrrinit trunc_normal_rrs rZr!AriaPreTrainedModel._init_weightsskk++ fbii ( ( MM   & &CS & 9{{&   &&('  5 5 6 6  $ $ &  - - MM   $ $S ) KK   " " $  . . GG ! !&,,C ! 8/r\rhN) rprqrrrsr|rr_supports_static_cacherrrxrhr\rZrrsL""'9r\rc0^\rSrSrS\4U4SjjrSrU=r$) AriaTextModelirc >[TU]U5 [R"[ UR 5Vs/sHn[ X5PM sn5UlSUlUR5 gs snf)NF) rirjr< ModuleListrKnum_hidden_layersrlayersgradient_checkpointing post_initrs rZrjAriaTextModel.__init__s_  mmFKFLdLdFe fFe !& 4Fe f  ',#  gsA1)r r )rprqrrrsr^rjrxryrzs@rZrrs~r\rc\rSrSrSrg)KwargsForCausalLMirhNrrhr\rZrrs3r\rcL^\rSrSrS/rS\4U4Sjjr\U4Sj5rSr U=r $)AriaTextForCausalLMizlm_head.weightrc>[TU]U5 [U5UlURUl[ R "URURSS9UlUR5 gr) rirjrr vocab_sizer<rrlm_headr rs rZrjAriaTextForCausalLM.__init__sU  "6*  ++yy!3!3V5F5FUS  r\c (>[TU]"U40UD6 gr)rir)rkrlrms rZrAriaTextForCausalLM.forward s - -r\)rrr) rprqrrrs_tied_weights_keysr^rjr(rrxryrzs@rZrrs,*+~..r\rc\rSrSrSrg)AriaCausalLMOutputWithPastirhNrrhr\rZrrrr\rc\rSrSrSrg)AriaModelOutputWithPastirhNrrhr\rZrrrr\rc^\rSrSrS\4U4SjjrSrSS\RS\ \RS\ 4Sjjr SS \RS\RS\RS \ \RS \ \RS \ \\RS \ \RS\ \S\ \S\ \S\ \S\ \RS\\S\\\44SjjrSrU=r$) AriaModelircD>[TU]U5 [U5Ulgr)rirjrmulti_modal_projectorrs rZrjAriaModel.__init__s  %26%:"r\c~UcgURSURRRURRRS9nURSURRRURRRS9nUR SS9S:R 5$)NrD) dimensionrstepr,)r>rBr)unfold vision_towerrrLrr)rkrpatches_subgrids rZ_create_patch_attention_mask&AriaModel._create_patch_attention_masks  $++""))44""))44,  *00""))44""))441   ###1A5;;==r\rrrcUbUOURRnURU5nURXSS9nSnUb'UR S5n[ R "U5nURUnURXS9n U $)a Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. pixel_mask (`torch.FloatTensor]`, *optional*): The tensors corresponding to the input image mask. vision_feature_layer (`Union[int, List[int]]`, *optional*): The index of the layer to select the vision feature. If multiple indices are provided, the vision feature of the corresponding indices will be concatenated to form the vision features. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). NT)patch_attention_maskoutput_hidden_statesrDr) rrr(r&rrF logical_notrr) rkrrrr+ image_outputsimage_attn_maskflattened_maskselected_image_featureimage_featuress rZget_image_featuresAriaModel.get_image_features-s,%9$D $++JjJj  $@@L)) Z^*   +199!)rrrz6Image features and image tokens do not match: tokens: z , features T) r6r7rr8r9r:r,r;r<)last_hidden_staterr attentionsimage_hidden_statesrh)rr:r,use_return_dictget_input_embeddingsrErFtensorrrIrArr expand_asrr3rrr@masked_scatterlanguage_modelrr>rrr?)rkr5rrr6r7rr8r9r:r,r;r<rspecial_image_maskn_image_tokens image_embedsr2n_imagesn_features_per_imagen_image_featuresoutputss rZrAriaModel.forwardRs 2C1N-TXT_T_TqTq$8$D $++JjJj &1%!(C %26O6O6QLL!;!;5::VcVjVjk7&"#5!9!9a!9!@!D!D!D!KA!N(KK,F,FF %1%;%;B%?%I%I-%X%[%[\i\p\p%q"".!3!3!3!:!>!>1!>!E!44)%%)[[%E%E5N .<-A-A!-DnFZFZ[\F]*H'*>> !11 L^L\\ghxgyz,..}/C/C]EXEXYN)889K^\M%%  )%+'/!5)    '%777@G33d!//))2>2J   QU   r\)r)Nr>) NNNNNNNNNNNN)rprqrrrsr|rjr(rF FloatTensorrrwr3 LongTensorrrrr#rrrrrrxryrzs@rZrrs;z;>&37$& #''#U../#" #N'+*.'+1537=A59$(,0/3&*59F ##F ''F $$ F !. F u//0 F "$u'8'8"9:F   1 12F D>F $D>F 'tnF d^F !!1!12F -.F  u-- .F F r\rz Aria model for conditional generation tasks. This model combines a vision tower, a multi-modal projector, and a language model to perform tasks that involve both image and text inputs. ) custom_introc%^\rSrSr\\SS\RS\RS\RS\ \RS\ \RS\ \ \RS\ \RS \ \RS \ \ S \ \ S \ \ S \ \ S\ \\R4S\ \RS\\S\ \\44 Sjj55rSU4SjjrSrU=r$)AriaForConditionalGenerationir5rrr6r7rr8labelsr9r:r,r;logits_to_keepr<rr)c 0U bU OURRn U bU OURRn U bU OURRn UR"SUUUUUUUU U U U US. UD6nUSn[ U [ 5(a [U *S5OU nURUSS2USS245nSnUb3UR"SUXRRRS.UD6n[UUURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or `model.image_token_id` (where `model` is your instance of `AriaForConditionalGeneration`). Tokens with indices set to `model.image_token_id` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import requests >>> import torch >>> from PIL import Image >>> from io import BytesIO >>> from transformers import AutoProcessor, AutoModel >>> from transformers.image_utils import load_image >>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible >>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg") >>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg") >>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg") >>> processor = AutoProcessor.from_pretrained("Rhymes-AI/Aria") >>> model = AutoModel.from_pretrained("Rhymes-AI/Aria", torch_dtype=torch.bfloat16, device_map="auto") >>> # Create inputs >>> messages = [ ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."}, ... {"type": "image"}, ... {"type": "text", "text": "What can we see in this image?"}, ... ] ... }, ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In which city is that bridge located?"}, ... ] ... } ... ] >>> prompts = [processor.apply_chat_template([message], add_generation_prompt=True) for message in messages] >>> images = [[image1, image2], [image3]] >>> inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt").to(model.device) >>> # Generate >>> generated_ids = model.generate(**inputs, max_new_tokens=256) >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> print(generated_texts[0]) Assistant: There are buildings, trees, lights, and water visible in this image. >>> print(generated_texts[1]) Assistant: The bridge is in San Francisco. ```N) r5rrr6r7rr8r9r:r,r;r<r)rrTr)lossrrrr?rh)rr:r,rArrrwslicer loss_functionrarrrrr?)rkr5rrr6r7rr8rTr9r:r,r;rUr<rrMr slice_indicesrrWs rZr$AriaForConditionalGeneration.forwardsD`2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W8W~ot4]k mA}a,?@A  %%f9P9P9[9[_eD*#33!//))   r\c X>[T U]"U4UUUUUS.U D6n USS:XaXJS'XZS'U $)N)rr8r6r<rUrrr)riprepare_inputs_for_generation) rkr5rr8rrr6r<rUr model_inputsrms rZr]:AriaForConditionalGeneration.prepare_inputs_for_generations\w<  +')))    !  !,8 ()3 &r\rh)NNNNNNNNNNNNrN)NNNNNNN)rprqrrrsr)r(rFrPrOrrrrrrwr#rrrrr]rxryrzs@rZrSrSs'+*.'+1537=A59-1$(,0/3&*3459u ##u ''u $$ u !. u u//0 u "$u'8'8"9:u   1 12u ))*u D>u $D>u 'tnu d^u c5<</0u !!1!12u *+!u " u00 1#u u tr\rS) r|r^rrgrSrrrrr)itypingrrrrrrnumpyr activationsr configuration_utilsr image_processing_utilsr r rrimage_transformsrrrrr image_utilsrrrrrrrrrrmodeling_flash_attention_utilsrmodeling_utilsr processing_utilsr!r"r#tokenization_utilsr$r%utilsr&r'r(r)r*utils.import_utilsr+autor-r.r/llama.configuration_llamar0llama.modeling_llamar1r2r3r4r5r6llava.modeling_llavar7r8r9r:&llava_next.image_processing_llava_nextr; get_loggerrprrFr<r[r^r|rModulerrrrr`rgrrrrrrrrrrrrrrS__all__rhr\rZrus@?!3uuee   C-HH>VV4<<3 G   H %>{=[{=|Q#!Q#h l ryy244n>BII>Bh+hV *% |XN|X~ Z8 Z ) RYY) XBIIDL-ryyL-^,,,$:o::69.9.J?,j>.13C." !<  6 @ @ FV#@VVr r\