o ZhX@sddlZddlmZmZddlmZmZmZmZddl Z ddl m Z ddl m Z ddl mZdefddZd eeee feeee ffd eeee feeee ffdeeeeffd d Zd e jdedeeee ffdee ge fdeee dfeedfeeeefffddZd e jdeee dfeedfeeeefffddZd e jdeee dfeedfeeeefffddZ dFd e jdeedee deddf ddZd e jdeeeefdee dee fdd Z dFd e jdeed!ee deddf d"d#Z $ $dGd%e jd&ee d'eed!ee d(eede jf d)d*Zd%e jfd+d,Zd%e jfd-d.ZGd/d0d0e jZ Gd1d2d2e jZ! dFd%e jd3edee!ee dfffd4d5Z" dFd%e jd3edee ee dfee dfffd6d7Z#d8eee dfdfdee dffd9d:Z$d;ee jdee ee dfee dfffddHd?e&e jd@eed$ee'fdAe j(j)fdBdCZ* $ >dHd?e&e jd@eed$ee'fdAe j(j)fdDdEZ+dS)IN)IterableSequence)AnyCallableNoReturnUnion)Tensor)NamedMemberAccessorreturncCstd)Na$make_functional(module): we don't yet support models that do parameter tying (also sometimes known as weight sharing). Please try to rewrite your model by replacing all instances of the tied parameter with another and/or comment your support in https://github.com/pytorch/functorch/issues/446) RuntimeErrorr r O/var/www/auris/lib/python3.10/site-packages/torch/_functorch/make_functional.pyraise_parameter_tying_errorsr named_paramstied_named_paramscCst|}t|}t|}t|}||sJi}|D] \}}|gf||<q!|D]\}}||vs:J||d|q0t|S)a[ named_params is a dictionary of tensors: {'A': A, 'B': B} tied_named_params is another dictionary of tensors {'A': A, 'B': B, 'B_tied': B} with potentially tied (or 'duplicated') tensors This function creates a mapping from the names in named_params to the names in tied_named_params: {'A': ['A'], 'B': ['B', 'B_tied']}. )dictsetkeysissubsetitemsappendvalues)rrZtensors_dict_keysZtied_tensors_dict_keysZtensor_to_mappingkeyZtensorr r r create_names_map!s     rmod named_members.subclassc Cst|dd}t|dd}t||}i}t|}|D]\}} | |vr.|tj| dd|| <|| } ||| qt|dkrDd\} } nt|\} } | | |fS)NF)Zremove_duplicateTmeta)devicerr r )tuplerr torchZ empty_like set_tensorlenzip) rrrZall_named_membersZunique_named_members names_mapmemoaccessornamep replacementnamesparamsr r r _extract_members=s      r.cCst||jtjS)aZ This function removes all the Parameters from the model and return them as a tuple as well as their original attribute names. The weights must be re-loaded with `load_weights` before the model can be used again. Note that this function modifies the model in place and after this call, mod.parameters() will be empty. )r.Znamed_parametersnn Parameterrr r r extract_weightsVs r2cCst||jddS)NcSs|SNr )xr r r gsz!extract_buffers..)r.Z named_buffersr1r r r extract_buffersdsr6Fr,r- as_paramscCs*t|}|r dd|D}|||dS)a  Reload a set of weights so that `mod` can be used again to perform a forward pass. Note that the `params` are regular Tensors (that can have history) and so are left as Tensors. This means that mod.parameters() will still be empty after this call. cSsg|]}t|qSr )r/r0).0r*r r r wsz load_weights..Nr Z set_tensors)rr,r-r7r(r r r load_weightsjs r;r&elemsc Csfg}t|}t||D]#\\}}}t|D]\}} |dkr)||| |q|| |qq |S)Nr)r r%r enumeraterZ swap_tensorr#) rr&r<resultr(_Z attr_nameselemiZ attr_namer r r _swap_state{srBbufferscCst|}|||dSr3r:)rr,rCr7r(r r r load_bufferssrDr modelweights weight_names buffer_namescCsPt|t|ks Jt|||t|dkr&t|t|ks Jt||||S)zload_state(model, weights, weight_names, buffers=(), buffer_names=()) -> model load_state takes `weights` and `buffers` and assigns them to the model. This is the inverse operation of `make_functional_deprecated_v1`. r)r$r;rD)rErFrGrCrHr r r load_states    rIcsFt}t|dkrtdt\}}fdd}||fS)amake_functional_deprecated_v1(model) -> weights, func, weight_names Given an nn.Module, make_functional_deprecated_v1 extracts the state (weights) and returns a functional version of the model, `func`. This makes it so that it is possible use transforms over the parameters of `model`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, func, _ = make_functional_deprecated_v1(model) func(weights, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, _, func = make_functional_deprecated_v1(model) grad_weights = grad(func)(weights, (x,)) ``` To put the state back into a model, use `load_state`. rzmake_functional_deprecated_v1(model): `model` has buffers. Please use make_functional_with_buffers_deprecated_v1(model) instead.cst}t||||Sr3)copydeepcopyr;)rFdata mutable_modelZ descriptorsrEr r funs  z*make_functional_deprecated_v1..fun)listrCr$r r2)rErCrFr?rOr rNr make_functional_deprecated_v1s   rQcs:t\}}t\}}fdd}|||fS)a`make_functional_with_buffers_deprecated_v1(model) -> weights, buffers, func, weight_names, buffer_names Given an nn.Module, make_functional_with_buffers_deprecated_v1 extracts the state (weights and buffers) and returns a functional version of the model, `func`. `func` can be invoked as follows: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) ``` And here is an example of applying the grad transform: ``` x = torch.randn(4, 3) model = nn.Linear(3, 3) weights, buffers, func, _, _ = make_functional_with_buffers_deprecated_v1(model) func(weights, buffers, (x,)) grad_weights = grad(func)(weights, buffers, (x,)) ``` To put the state back into a model, use `load_state`. cs*t}t||t||||Sr3)rJrKr;rD)rFrCrLrMZbuf_descriptorsrEZweight_descriptorsr r rOs   z7make_functional_with_buffers_deprecated_v1..fun)r2r6)rErFr?rCrOr rRr *make_functional_with_buffers_deprecated_v1srSc seZdZdZdejdeedfdeedfdeee efdeee efdd f fd d Z e dd ejde dedee dfee dfffddZdee dee defddZZS)FunctionalModuleWithBufferszW This is the callable object returned by :func:`make_functional_with_buffers`. stateless_model param_names.rHparam_names_mapbuffer_names_mapr Ncs6t||_||_||_t||_|j|dSr3)super__init__rUrVrHr all_names_mapupdate)selfrUrVrHrWrX __class__r r rZs  z$FunctionalModuleWithBuffers.__init__FrEdisable_autograd_trackingc CsTt|}t|\}}}t|\}}}|r|D]} | dqt|||||||fSNF)rJrKr2r6requires_grad_rT) rEr` model_copyr-rVrWrCrHrXparamr r r _create_from s   z(FunctionalModuleWithBuffers._create_fromr-rCc OsRt|j|jt|t|}z|j|i|Wt|j|j|St|j|j|wr3)rBrUr[r!)r]r-rCargskwargs old_stater r r forwards$z#FunctionalModuleWithBuffers.forwardF__name__ __module__ __qualname____doc__r/Moduler!strrrPrZ staticmethodboolrrerrri __classcell__r r r^r rTs>   rTc seZdZdZdejdeedfdeee efddffdd Z e dd ejd e dedee dfffd dZdee defddZZS)FunctionalModulezJ This is the callable object returned by :func:`make_functional`. rUrV.r&r Ncs t||_||_||_dSr3)rYrZrUrVr&)r]rUrVr&r^r r rZ3s  zFunctionalModule.__init__FrEr`cCs@t|}t|\}}}|r|D]}|dqt||||fSra)rJrKr2rbru)rEr`rcr-rVr&rdr r r re>s  zFunctionalModule._create_fromr-c OsFt|j|j|}z|j|i|Wt|j|j|St|j|j|wr3)rBrUr&)r]r-rfrgrhr r r riJs$zFunctionalModule.forwardrjrkr r r^r ru.s*    rur`cCs.t|}t|dkrtdtj||dS)amake_functional(model, disable_autograd_tracking=False) -> func, params Given a ``torch.nn.Module``, :func:`make_functional` extracts the state (params) and returns a functional version of the model, ``func``. This makes it so that it is possible use transforms over the parameters of ``model``. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) func(params, x) And here is an example of applying the grad transform over the parameters of a model. .. code-block:: python import torch import torch.nn as nn from functorch import make_functional, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params = make_functional(model) def compute_loss(params, x, t): y = func(params, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, x, t) If the model has any buffers, please use :func:`make_functional_with_buffers` instead. Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. rzdmake_functional(model): `model` has buffers. Please use make_functional_with_buffers(model) instead.r`)rPrCr$r rure)rEr`rCr r r make_functionalTs ; rwcCstj||dS)amake_functional_with_buffers(model, disable_autograd_tracking=False) -> func, params, buffers Given a ``torch.nn.Module``, make_functional_with_buffers extracts the state (params and buffers) and returns a functional version of the model ``func`` that can be invoked like a function. ``func`` can be invoked as follows: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers x = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) func(params, buffers, x) And here is an example of applying the grad transform over the parameters of a model: .. code-block:: python import torch import torch.nn as nn from functorch import make_functional_with_buffers, grad x = torch.randn(4, 3) t = torch.randn(4, 3) model = nn.Linear(3, 3) func, params, buffers = make_functional_with_buffers(model) def compute_loss(params, buffers, x, t): y = func(params, buffers, x) return nn.functional.mse_loss(y, t) grad_weights = grad(compute_loss)(params, buffers, x, t) Args: model (torch.nn.Module): Input model. disable_autograd_tracking (bool): Flag to disable gradients tracking for output parameters. The returned params are unrelated to the set of params from the original model. If False (default), the params will have ``requires_grad=True`` on them (aka they will be trackable with regular PyTorch autograd), matching the requires_grad-ness of the params from the original model. Otherwise, the returned params will have ``requires_grad=False``. Default, False. If you plan on using regular PyTorch autograd (e.g., if you want to call ``.backward()`` or ``torch.autograd.grad()``, then set ``disable_autograd_tracking=False``. Otherwise, if you're only planning on using functorch's gradient transforms, then please set ``disable_autograd_tracking=True`` to avoid unnecessarily tracking history with PyTorch autograd. rv)rTre)rEr`r r r make_functional_with_bufferss8rxtuple_of_tuple_of_tensorscCs"tt|}tdd|D}|S)Ncs|] }t|VqdSr3r"stackdetachr8Zshardsr r r s z"transpose_stack..)r!r%)ryresultsr r r transpose_stacks rmodelscst|dkr tdtdd|Ds tdd|Ds tdt|dtfdd|Ds5tdtd d |D\}}}t|}t|}|d||fS) a(combine_state_for_ensemble(models) -> func, params, buffers Prepares a list of torch.nn.Modules for ensembling with :func:`vmap`. Given a list of ``M`` ``nn.Modules`` of the same class, stacks all of their parameters and buffers together to make ``params`` and ``buffers``. Each parameter and buffer in the result will have an additional dimension of size ``M``. :func:`combine_state_for_ensemble` also returns ``func``, a functional version of one of the models in :attr:`models`. One cannot directly run ``func(params, buffers, *args, **kwargs)`` directly, you probably want to use ``vmap(func, ...)(params, buffers, *args, **kwargs)`` Here's an example of how to ensemble over a very simple model: .. code-block:: python num_models = 5 batch_size = 64 in_features, out_features = 3, 3 models = [torch.nn.Linear(in_features, out_features) for i in range(num_models)] data = torch.randn(batch_size, 3) fmodel, params, buffers = combine_state_for_ensemble(models) output = vmap(fmodel, (0, 0, None))(params, buffers, data) assert output.shape == (num_models, batch_size, out_features) .. warning:: All of the modules being stacked together must be the same (except for the values of their parameters/buffers). For example, they should be in the same mode (training vs eval). This API is subject to change -- we're investigating better ways to create ensembles and would love your feedback how to improve this. rz?combine_state_for_ensemble: Expected at least one model, got 0.css|]}|jVqdSr3Ztrainingr8mr r r r sz-combine_state_for_ensemble..css|]}|j VqdSr3rrr r r r szTcombine_state_for_ensemble: Expected all models to have the same training/eval mode.c3s|] }t|kVqdSr3)typerZ model0_typr r rzHcombine_state_for_ensemble: Expected all models to be of the same class.cSsg|]}t|qSr )rxr8rEr r r r9sz.combine_state_for_ensemble..)r$r allrr%r)rfuncsr-rCr rr combine_state_for_ensembles& ($   rcpu model_classensemble_shapercfdd}|S)Nc stdkr tdtdkri}t|Sd}|dkr.td|dtfddt|D}ti\}}}tdd|D}tt|}td d|D}|||fS) N,NYI: ensemble_shape with more than 1 elementr num_models  should be > 0c3$|] }iVqdSr3tor8r?rfrrgrr r r. z3functional_init..wrapped..css|] }t|dVqdS)rN)rQrr r r r2rcsrzr3r{r~r r r r4)r$ ValueErrorrrQr!ranger%) rfrgrE num_modelsrr?fnr,rFrrrrfrgr wrapped$s    z functional_init..wrappedr rrrrr rr functional_initsrcr)Nc stdkr tdtdkri}t|Sd}|dkr.td|dtfddt|D}ti\}}}}}ttdd|D\} } tt| } td d| D} tt| } td d| D} | | |||fS) Nrrrrrc3rr3rrrr r rIrz@functional_init_with_buffers..wrapped..css |] }t|ddVqdS)Nr)rSrr r r rTs  csrzr3r{r~r r r rZrcsrzr3r{r~r r r r\r)r$rrrQr!rrSr%) rfrgrErrr?rrGrHrFrCrrr r?s8     z-functional_init_with_buffers..wrappedr rr rr functional_init_with_buffers:s rrjr )r r),rJcollections.abcrrtypingrrrrr"Ztorch.nnr/rZ%torch.nn.utils._named_member_accessorr rrrqr!rPrrpr.r2r6rsr;rBrDrIrQrSrTrurwrxrrrinttypesZDevicerrr r r r s      ( ( (      *%8' G =    @