前情提要

大家好呀，VIsion Transformer已经到第六天了随着时间的推移，难度也是在也来越大的，相信大家已经从ResNet学习到了ViT Transformer再到Diet，今天更是学习到了2021的ICCV最佳论文Swin Transformer，Swin Transformer今年在各大磅单上更是直接屠榜，COCO验证集前8全是使用Swin刷榜，而Swin原本的论文才是第八，前七全是Swin的再创新，https://paperswithcode.com/sota/object-detection-on-coco-minival?p=end-to-end-semi-supervised-object-detection
。如果今天的笔记能够帮助到你那么绝对是我莫大的荣幸。

Swin 与 VIT 的区别

Swin 与 Vit 的区别可以用一张图来说明

相比较Vit直接下采样16倍，Swin使用了窗口的理念，并使用不同的下采样倍数，并且不同的窗口的计算是互不干扰的，这样相比较Vit，大大减少了计算量并有效的获得了性能上的提升

Patch层讲解

如果你已经学过ViT的话，你应该知道VIT拥有一个PatchEmbedding层它负责将一批RGB图像由四维映射成为三维

[B,C,H,W]->[B,H*W/N/N,C*N*N]

而Swin中PatchEmbedding 的职责由Patch Partition承担，由于朱老师的PPT暂未放出所以这里引用的B站up霹雳吧啦Wz的PPT。

由于Swin Transformer不像VIt一样，Swin Fransformer是多次下采样，第一次下采样使用的是4x4的卷积核

之后会通过一个全链接层也叫Linear Embedding层将通道数由48转换为C，而其全貌则为。

但是在Swin Transformer的代码实现中是直接将Patch Partition与Linear Embedding合而为一，直接使用一个卷积核4x4大小 stride为4的卷积核直接将通道数由3转换为C。

import paddle
import paddle.nn as nn

class PatchEmbed(nn.Layer):
    """
    2D Image to Patch Embedding
    """
    def __init__(self, patch_size=4, in_c=3, embed_dim=96, norm_layer=nn.LayerNorm):
        super().__init__()
        patch_size = (patch_size, patch_size)
        self.patch_size = patch_size
        self.in_chans = in_c
        self.embed_dim = embed_dim
        self.proj = nn.Conv2D(in_c, embed_dim, kernel_size=patch_size, stride=patch_size)
        self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()
    def forward(self, x):
        _, _, H, W = x.shape

        # # padding
        # # 如果输入图片的H，W不是patch_size的整数倍，需要进行padding
        # pad_input = (H % self.patch_size[0] != 0) or (W % self.patch_size[1] != 0)
        # if pad_input:
        #     # to pad the last 3 dimensions,
        #     # (W_left, W_right, H_top,H_bottom, C_front, C_back)
        #     x = F.pad(x, (0, self.patch_size[1] - W % self.patch_size[1],
        #                   0, self.patch_size[0] - H % self.patch_size[0],
        #                   0, 0))
         # 下采样patch_size倍
        x = self.proj(x)
        _, _, H, W = x.shape
        # flatten: [B, C, H, W] -> [B, C, HW]
        # transpose: [B, C, HW] -> [B, HW, C]
        x = paddle.transpose(x.flatten(2),(0,2,1))
        x = self.norm(x)
        print(x.shape)
        return x, H, W

model = PatchEmbed()
paddle.summary(model,(8,3,224,224))

[8, 3136, 96]
---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Conv2D-1      [[8, 3, 224, 224]]    [8, 96, 56, 56]         4,704     
  LayerNorm-1     [[8, 3136, 96]]       [8, 3136, 96]           192      
===========================================================================
Total params: 4,896
Trainable params: 4,896
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 4.59
Forward/backward pass size (MB): 36.75
Params size (MB): 0.02
Estimated Total Size (MB): 41.36
---------------------------------------------------------------------------






{'total_params': 4896, 'trainable_params': 4896}

PatchMerging讲解

由于Swin需要经理四次下采样，而第一次下采样是由patchemmedding
层进行处理，而之后还有三次就是由PatchMerging进行。PatchMerging会将特征图的宽和高再次下采样两倍，并且使得通道数Cx2。这里引用的图解也是引用的B站up霹雳吧啦Wz的PPT。

首先我们使用一个2x2大小的卷积核作为窗口，每个窗口都有四个像素，那么我就就把每个窗口相同位置的像素取出，并得到如第二个一样的四个特征矩阵，之后会对这四个特征矩阵在C纬度进行拼接也就得到了4xC，之后再进行一个LinearNorm层，最后通过一个Linear层进行一个线性映射变为2xC。这样
X的shape就变成了

[H/4,W/4,C]->[H/8,W/8,2*C]

class PatchMerging(nn.Layer):
    r""" Patch Merging Layer.
    Args:
        dim (int): Number of input channels.
        norm_layer (nn.Module, optional): Normalization layer.  Default: nn.LayerNorm
    """

    def __init__(self, dim, norm_layer=nn.LayerNorm):
        super().__init__()
        self.dim = dim
        self.reduction = nn.Linear(4 * dim, 2 * dim, bias_attr=False)
        self.norm = norm_layer(4 * dim)

    def forward(self, x, H=224//4, W=224//4):
        """
        x: B, H*W, C
        """
        B, L, C = x.shape
        print(x.shape)
        assert L == H * W, "input feature has wrong size"

        x = paddle.reshape(x,(B, H, W, C))

        # # padding
        # # 如果输入feature map的H，W不是2的整数倍，需要进行padding
        # pad_input = (H % 2 == 1) or (W % 2 == 1)
        # if pad_input:
        #     # to pad the last 3 dimensions, starting from the last dimension and moving forward.
        #     # (C_front, C_back, W_left, W_right, H_top, H_bottom)
        #     # 注意这里的Tensor通道是[B, H, W, C]，所以会和官方文档有些不同
        #     x = F.pad(x, (0, 0, 0, W % 2, 0, H % 2))

        x0 = x[:, 0::2, 0::2, :]  # [B, H/2, W/2, C]
        x1 = x[:, 1::2, 0::2, :]  # [B, H/2, W/2, C]
        x2 = x[:, 0::2, 1::2, :]  # [B, H/2, W/2, C]
        x3 = x[:, 1::2, 1::2, :]  # [B, H/2, W/2, C]
        x = paddle.concat([x0, x1, x2, x3], -1)  # [B, H/2, W/2, 4*C]
        x = paddle.reshape(x,(B, -1, 4 * C))  # [B, H/2*W/2, 4*C]

        x = self.norm(x)
        x = self.reduction(x)  # [B, H/2*W/2, 2*C]
        print(x.shape)
        return x

model = PatchMerging(96)
paddle.summary(model,(8,3136,96))

[8, 3136, 96]
[8, 784, 192]
---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
  LayerNorm-2     [[8, 784, 384]]       [8, 784, 384]           768      
   Linear-1       [[8, 784, 384]]       [8, 784, 192]         73,728     
===========================================================================
Total params: 74,496
Trainable params: 74,496
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 9.19
Forward/backward pass size (MB): 27.56
Params size (MB): 0.28
Estimated Total Size (MB): 37.03
---------------------------------------------------------------------------






{'total_params': 74496, 'trainable_params': 74496}

Swin TansFormer Block 讲解

与VIT TransFormer的Block层相似，Swin TransFormer与之不同的就是使用了S-MSA与SW-MSA来代替MSA，其他都没有改变

S-SMA与SW-MSA暂时有点难，咱们分到明天再去进行讲解，今天咱们先把整个Block模块进行实现

可以看到Block层是先经过一个LayerNorm然后经过S-MSA或者SW-MSA然后在进过一个Dropout或者是DropPath层，然后进行一个短接，然后再进行一个LayerNrom之后再进行一个Mlp再接一个Dropout或者是DropPath层，然后再来一个短接，这样一个Block层就实现了。

下面我先先来实现之前已经讲过Mlp层。

Mlp讲解

Mlp之前朱老师已经讲的很清楚了就是一个Linear层将通道数×4，然后经过GELU激活函数，然后再经过一个Dropout层再经过一个Linear层将通道数变回到原来的通道数，之后再接一个Dropout层

class Mlp(nn.Layer):
    """ MLP as used in Vision Transformer, MLP-Mixer and related networks
    """
    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
        super().__init__()
        out_features = out_features or in_features
        hidden_features = hidden_features or in_features

        self.fc1 = nn.Linear(in_features, hidden_features)
        self.act = act_layer()
        self.drop1 = nn.Dropout(drop)
        self.fc2 = nn.Linear(hidden_features, out_features)
        self.drop2 = nn.Dropout(drop)

    def forward(self, x):
        print(x.shape)
        x = self.fc1(x)
        x = self.act(x)
        x = self.drop1(x)
        x = self.fc2(x)
        x = self.drop2(x)
        print(x.shape)
        return x

model = Mlp(768)
paddle.summary(model,(8,197,768))

[8, 197, 768]
[8, 197, 768]
---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
   Linear-2       [[8, 197, 768]]       [8, 197, 768]         590,592    
    GELU-1        [[8, 197, 768]]       [8, 197, 768]            0       
   Dropout-1      [[8, 197, 768]]       [8, 197, 768]            0       
   Linear-3       [[8, 197, 768]]       [8, 197, 768]         590,592    
   Dropout-2      [[8, 197, 768]]       [8, 197, 768]            0       
===========================================================================
Total params: 1,181,184
Trainable params: 1,181,184
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 4.62
Forward/backward pass size (MB): 46.17
Params size (MB): 4.51
Estimated Total Size (MB): 55.29
---------------------------------------------------------------------------






{'total_params': 1181184, 'trainable_params': 1181184}

SwinTransformerBlock层实现

下面就是Block层的实现，这里有一个WindowAttention就是S-MSA与SW-MSA的实现，这里就先简单定义一下，没有任何内容。

class WindowAttention(nn.Layer):
    def __init__(self,dim, window_size, num_heads, qkv_bias,attn_drop, proj_drop):
        super().__init__()
        pass
    def forward(self,x):
        return x,x

class SwinTransformerBlock(nn.Layer):
    """ Swin Transformer Block.
        dim (int): Number of input channels.
        num_heads (int): Number of attention heads.
        window_size (int): Window size.
        shift_size (int): Shift size for SW-MSA.
        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
        qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
        drop (float, optional): Dropout rate. Default: 0.0
        attn_drop (float, optional): Attention dropout rate. Default: 0.0
        drop_path (float, optional): Stochastic depth rate. Default: 0.0
        act_layer (nn.Module, optional): Activation layer. Default: nn.GELU
        norm_layer (nn.Module, optional): Normalization layer.  Default: nn.LayerNorm
    """
    def __init__(self, dim, num_heads, window_size=7, shift_size=0,
            mlp_ratio=4., qkv_bias=True, drop=0., attn_drop=0., drop_path=0.,
            act_layer=nn.GELU, norm_layer=nn.LayerNorm):
        super().__init__()
        self.dim = dim
        self.num_heads = num_heads
        self.window_size = window_size
        self.shift_size = shift_size
        self.mlp_ratio = mlp_ratio
        assert 0 <= self.shift_size < self.window_size, "shift_size must in 0-window_size"

        self.norm1 = norm_layer(dim)
        self.attn = WindowAttention(
            dim, window_size=(self.window_size, self.window_size), num_heads=num_heads, qkv_bias=qkv_bias,
            attn_drop=attn_drop, proj_drop=drop)

        self.drop_path = nn.Dropout(0) #if drop_path > 0. else nn.Identity()
        self.norm2 = norm_layer(dim)
        mlp_hidden_dim = int(dim * mlp_ratio)
        self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)

    def forward(self, x):
        H, W = 56, 56
        B, L, C = x.shape
        assert L == H * W, "input feature has wrong size"

        shortcut = x
        x = self.norm1(x)
        x = paddle.reshape(x,(B, H, W, C))

        # # pad feature maps to multiples of window size
        # # 把feature map给pad到window size的整数倍
        # pad_l = pad_t = 0
        # pad_r = (self.window_size - W % self.window_size) % self.window_size
        # pad_b = (self.window_size - H % self.window_size) % self.window_size
        # x = F.pad(x, (0, 0, pad_l, pad_r, pad_t, pad_b))
        _, Hp, Wp, _ = x.shape

#--------------------------------------------------------------------------------------------------#
#                           S-MSA暂不处理，下次再说
#--------------------------------------------------------------------------------------------------#
        # # cyclic shift
        # if self.shift_size > 0:
        #     shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))
        # else:
        #     shifted_x = x
        #     attn_mask = None

        # # partition windows
        # x_windows = window_partition(shifted_x, self.window_size)  # [nW*B, Mh, Mw, C]
        # x_windows = x_windows.view(-1, self.window_size * self.window_size, C)  # [nW*B, Mh*Mw, C]

        # # W-MSA/SW-MSA
        # attn_windows = self.attn(x_windows, mask=attn_mask)  # [nW*B, Mh*Mw, C]

        # # merge windows
        # attn_windows = attn_windows.view(-1, self.window_size, self.window_size, C)  # [nW*B, Mh, Mw, C]
        # shifted_x = window_reverse(attn_windows, self.window_size, Hp, Wp)  # [B, H', W', C]

        # # reverse cyclic shift
        # if self.shift_size > 0:
        #     x = torch.roll(shifted_x, shifts=(self.shift_size, self.shift_size), dims=(1, 2))
        # else:
        #     x = shifted_x

        # if pad_r > 0 or pad_b > 0:
        #     # 把前面pad的数据移除掉
        #     x = x[:, :H, :W, :].contiguous()

#--------------------------------------------------------------------------------------------------#
        x = paddle.reshape(x,(B,H*W,C))

        x = shortcut + self.drop_path(x)
        x = x + self.drop_path(self.mlp(self.norm2(x)))

        return x

model = SwinTransformerBlock(96,3)
paddle.summary(model,(8,3136,96))


[8, 3136, 96]
[8, 3136, 96]
---------------------------------------------------------------------------
 Layer (type)       Input Shape          Output Shape         Param #    
===========================================================================
  LayerNorm-3     [[8, 3136, 96]]       [8, 3136, 96]           192      
   Dropout-3      [[8, 3136, 96]]       [8, 3136, 96]            0       
  LayerNorm-4     [[8, 3136, 96]]       [8, 3136, 96]           192      
   Linear-4       [[8, 3136, 96]]       [8, 3136, 384]        37,248     
    GELU-2        [[8, 3136, 384]]      [8, 3136, 384]           0       
   Dropout-4      [[8, 3136, 384]]      [8, 3136, 384]           0       
   Linear-5       [[8, 3136, 384]]      [8, 3136, 96]         36,960     
   Dropout-5      [[8, 3136, 96]]       [8, 3136, 96]            0       
     Mlp-2        [[8, 3136, 96]]       [8, 3136, 96]            0       
===========================================================================
Total params: 74,592
Trainable params: 74,592
Non-trainable params: 0
---------------------------------------------------------------------------
Input size (MB): 9.19
Forward/backward pass size (MB): 330.75
Params size (MB): 0.28
Estimated Total Size (MB): 340.22
---------------------------------------------------------------------------






{'total_params': 74592, 'trainable_params': 74592}

总结

今天我们已经把除了有关S-MSA与SW-MSA的所有知识都完成了，明天我们就要实现S-MSA与SW-MSA，以上都是个人理解，如有错误欢迎指正。

以上图片均使用自B站up主霹雳吧啦Wz，十分感谢。

我是老萌新

我在AI Studio上获得黄金等级，点亮5个徽章，来互关呀~ https://aistudio.baidu.com/aistudio/personalcenter/thirdview/553083

# 计算机视觉 # 目标检测 # paddlepaddle
Logo
百度飞桨AI Studio社区

学大模型，用大模型上飞桨星河社区！每天8点V100G算力免费领！免费领取ERNIE 4.0 100w Token >>>

更多推荐

cover

利用Amazon Bedrock畅玩Claude 3等多种领先模型，抢占AI高地(体验倒计时4小时)

avatar 百度飞桨星河社区
cover

RAPTOR：索引树状 RAG，使用树结构来捕捉文本的高级和低级细节

avatar 百度飞桨星河社区

RAG介绍

LlamaIndex是一个基于大型语言模型（LLM）的应用程序数据框架，旨在连接LLM与外部数据，特别是私有或特定领域的数据，以实现更准确的文本生成和知识增强。

avatar 百度飞桨星河社区