fTh&SrSSKJrJr SSKJr SSKJr \R"\ 5r "SS\5r "SS \5r "S S \5r S /rg ) zDBRX model configuration)AnyOptional)PretrainedConfig)loggingc Z^\rSrSrSrSrS S\S\\S\S\S\ 4 U4S jjjr S r U=r $) DbrxAttentionConfiga/Configuration class for Dbrx Attention. [`DbrxAttention`] class. It is used to instantiate attention layers according to the specified arguments, defining the layers architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. clip_qkv (`float`, *optional*): If set, clip the queries, keys, and values in the attention layer to this value. kv_n_heads (`int`, *optional*, defaults to 1): For grouped_query_attention only, allow user to specify number of kv heads. rope_theta (`float`, *optional*, defaults to 10000.0): The base frequency for rope. attn_config attn_pdropclip_qkv kv_n_heads rope_thetakwargsc >[TU]"S0UD6 XlX lX0lX@lSHnXe;dM UR U5 M [U5S:wa[SU<35eg)N model_typeattn_implementationtransformers_version _commit_hash torch_dtyperFound unknown kwargs=) super__init__r r rrpoplen ValueError)selfr r rrrk __class__s c/var/www/auris/envauris/lib/python3.13/site-packages/transformers/models/dbrx/configuration_dbrx.pyrDbrxAttentionConfig.__init__.sj "6"$ $$mA{ 1 n v;! 5fY78 8 )r r rr)Ng@) __name__ __module__ __qualname____firstlineno____doc__base_config_keyfloatrintrr__static_attributes__ __classcell__r!s@r"r r s["$O $(# 995/9 9  9  99r$r cx^\rSrSrSrSrSS\\S\S\S\S\\ S \ S \\ S \ 4U4S jjjr S r U=r $) DbrxFFNConfigCa@Configuration class for Dbrx FFN. [`DbrxFFN`] class. It is used to instantiate feedforward layers according to the specified arguments, defining the layers architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: ffn_act_fn (`dict`, *optional*, defaults to `None`): A dict specifying activation function for the FFN. The dict should have a key 'name' with the value being the name of the activation function along with any additional keyword arguments. If `None`, then set to `{"name": "silu"}`. ffn_hidden_size (`int`, *optional*, defaults to 3584): The hidden size of the feedforward network. moe_num_experts (`int`, *optional*, defaults to 4): The number of experts in the mixture of experts layer. moe_top_k (`int`, *optional*, defaults to 1): The number of experts to use in the mixture of experts layer. moe_jitter_eps (`float`, *optional*, defaults to `None`): If not `None`, the jitter epsilon for the mixture of experts layer. moe_loss_weight (`float`, *optional*, defaults to 0.01): The loss weight for the mixture of experts layer. moe_normalize_expert_weights (`float`, *optional*, defaults to 1.0): The normalization factor for the expert weights. ffn_config ffn_act_fnffn_hidden_sizemoe_num_experts moe_top_kmoe_jitter_epsmoe_loss_weightmoe_normalize_expert_weightsrc >[T U]5 UcSS0nXlX lX0lX@lXPlX`lXplSHn X;dM URU 5 M [U5S:wa[SU<35eg)Nnamesilurrr) rrr6r7r8r9r:r;r<rrr) rr6r7r8r9r:r;r<rr r!s r"rDbrxFFNConfig.__init__Zs    &)J$..",.,H)mA{ 1 n v;! 5fY78 8 r$)r6r7r:r;r<r8r9)Nir&Ng{Gz?g?)r'r(r)r*r+r,rdictr.r-rrr/r0r1s@r"r3r3Cs(#O&*# *.!%8;9TN99 9  9 ! 99'/uo999r$r3c^\rSrSrSrSr\\S.rSSSSS .r SS\ S\ S\ S\ S \ S \ S \ S \ \S\ \S\ S\ S\ S\4U4SjjjrSrU=r$) DbrxConfigwa This is the configuration class to store the configuration of a [`DbrxModel`]. It is used to instantiate a Dbrx model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a different configuration to that of the [databricks/dbrx-instruct](https://huggingface.co/databricks/dbrx-instruct) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: d_model (`int`, *optional*, defaults to 2048): Dimensionality of the embeddings and hidden states. n_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. n_layers (`int`, *optional*, defaults to 24): Number of hidden layers in the Transformer encoder. max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. vocab_size (`int`, *optional*, defaults to 32000): Vocabulary size of the Dbrx model. Defines the maximum number of different tokens that can be represented by the `inputs_ids` passed when calling [`DbrxModel`]. resid_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability applied to the attention output before combining with residual. emb_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the embedding layer. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. ffn_config (`dict`, *optional*): A dictionary used to configure the model's FFN module. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. output_router_logits (`bool`, *optional*, defaults to `False`): Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss. See [here]() for more details. Example: ```python >>> from transformers import DbrxConfig, DbrxModel >>> # Initializing a Dbrx configuration >>> configuration = DbrxConfig(n_layers=2, d_model=256, n_heads=8, vocab_size=128) >>> # Initializing a model (with random weights) from the configuration >>> model = DbrxModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` dbrx)r r5n_headsd_modeln_layers max_seq_len)num_attention_heads hidden_sizenum_hidden_layersmax_position_embeddings vocab_size resid_pdrop emb_pdropr r5 use_cacheinitializer_rangeoutput_router_logitsrc  >Uc[5UlO,[U[5(a[S0UD6UlOXlU c[ 5UlO,[U [5(a[ S0U D6UlOXlXlX lX0lX@l XPl X`l Xpl Xl XlXlURR UlU R%SS5nU(a ['S5e[(TU]T"SSU0U D6 g)Ntie_word_embeddingsFz5tie_word_embeddings is not supported for DBRX models.r)r r isinstancerBr3r5rHrGrIrJrOrPrQrRrSrTrnum_key_value_headsrrrr)rrHrGrIrJrOrPrQr r5rRrSrTrrVr!s r"rDbrxConfig.__init__s  24D   T * *2A[AD *   +oDO  D ) )+9j9DO(O   &$&""!2$8!#'#3#3#>#> $jj)>F TU U K-@KFKr$) r rHrQr5rSrJrGrIrXrTrPrRrO) rZi}r%r%NNTg{Gz?F)r'r(r)r*r+rr r3 sub_configs attribute_mapr.r-rboolrrr/r0r1s@r"rDrDws4lJ"5]SK( '#0 M 59.2#'%*.L.L.L .L  .L  .L.L.L12.L]+.L.L!.L#.L.L.Lr$rDN)r+typingrrconfiguration_utilsrutilsr get_loggerr'loggerr r3rD__all__rr$r"rfs\ 3   H %&9*&9R19$19hnL!nLb .r$