o +ѹg#@sddlmZddlZddlmZmZmZddlZgdZedddZ edd dZ d'd(ddZ d'd)ddZ d*d!d"Z d+d%d&ZdS),) annotationsN)OptionalTupleTypeVar)fuse_conv_bn_evalfuse_conv_bn_weightsfuse_linear_bn_evalfuse_linear_bn_weightsConvTztorch.nn.modules.conv._ConvNd)boundLinearTztorch.nn.LinearFconvbn%torch.nn.modules.batchnorm._BatchNorm transposeboolreturnc Csf|js|jr Jdt|}|jdur|jdusJt|j|j|j|j|j|j|j|\|_|_|S)a+Fuse a convolutional module and a BatchNorm module into a single, new convolutional module. Args: conv (torch.nn.modules.conv._ConvNd): A convolutional module. bn (torch.nn.modules.batchnorm._BatchNorm): A BatchNorm module. transpose (bool, optional): If True, transpose the convolutional weight. Defaults to False. Returns: torch.nn.modules.conv._ConvNd: The fused convolutional module. .. note:: Both ``conv`` and ``bn`` must be in eval mode, and ``bn`` must have its running buffers computed. Fusion only for eval!N) trainingcopydeepcopy running_mean running_varrweightbiaseps)r rrZ fused_convrY/Users/admin/.pyenv/versions/3.10.0/lib/python3.10/site-packages/torch/nn/utils/fusion.pyr s  rconv_w torch.Tensorconv_bOptional[torch.Tensor]bn_rmbn_rvbn_epsfloatbn_wbn_b-Tuple[torch.nn.Parameter, torch.nn.Parameter]cCs|j}|dur |jn|} |durt|}|durt|}|dur't|}t||} |r?ddgdgt|jd} nddgdgt|jd} ||| | j|d} ||| ||j| d} tj | |j tj | |j fS)aFuse convolutional module parameters and BatchNorm module parameters into new convolutional module parameters. Args: conv_w (torch.Tensor): Convolutional weight. conv_b (Optional[torch.Tensor]): Convolutional bias. bn_rm (torch.Tensor): BatchNorm running mean. bn_rv (torch.Tensor): BatchNorm running variance. bn_eps (float): BatchNorm epsilon. bn_w (Optional[torch.Tensor]): BatchNorm weight. bn_b (Optional[torch.Tensor]): BatchNorm bias. transpose (bool, optional): If True, transpose the conv weight. Defaults to False. Returns: Tuple[torch.nn.Parameter, torch.nn.Parameter]: Fused convolutional weight and bias. N)dtype) r,torch zeros_likeZ ones_likersqrtlenshapeZreshapetonn Parameter requires_grad)rr r"r#r$r&r'rZconv_weight_dtypeZconv_bias_dtypeZ bn_var_rsqrtr1Z fused_conv_wZ fused_conv_brrrr%s    rlinearcCsd|js|jr Jdt|}|jdur|jdusJt|j|j|j|j|j|j|j\|_|_|S)aFuse a linear module and a BatchNorm module into a single, new linear module. Args: linear (torch.nn.Linear): A Linear module. bn (torch.nn.modules.batchnorm._BatchNorm): A BatchNorm module. Returns: torch.nn.Linear: The fused linear module. .. note:: Both ``linear`` and ``bn`` must be in eval mode, and ``bn`` must have its running buffers computed. rN) rrrrrr rrr)r6rZ fused_linearrrrrTs  rlinear_wlinear_bc Csb|dur t|}|t||}||d}||||} tj||jtj| |jfS)aFuse linear module parameters and BatchNorm module parameters into new linear module parameters. Args: linear_w (torch.Tensor): Linear weight. linear_b (Optional[torch.Tensor]): Linear bias. bn_rm (torch.Tensor): BatchNorm running mean. bn_rv (torch.Tensor): BatchNorm running variance. bn_eps (float): BatchNorm epsilon. bn_w (torch.Tensor): BatchNorm weight. bn_b (torch.Tensor): BatchNorm bias. transpose (bool, optional): If True, transpose the conv weight. Defaults to False. Returns: Tuple[torch.nn.Parameter, torch.nn.Parameter]: Fused linear weight and bias. Nr*)r-r.r/Z unsqueezer3r4r5) r7r8r"r#r$r&r'Zbn_scaleZfused_wZfused_brrrr ks   r )F)r r rrrrrr )rrr r!r"rr#rr$r%r&r!r'r!rrrr()r6r rrrr )r7rr8r!r"rr#rr$r%r&rr'rrr() __future__rrtypingrrrr-__all__r r rrrr rrrrs     /